Hehe, well it wasn't exactly crystal clear, but I tried it and it worked. ;)
There are three prefs you can set:
pref | type | what it does | default value
kmeleon.plugins.layers.catchOpen | bool | catch windows events and apply them to layers | false
kmeleon.plugins.layers.catchClose | bool | catch windows events and apply them to layers | false
kmeleon.plugins.layers.catch | bool | sets both catchOpen and catchClose | not set
What I did was add the following line to Prefs.js:
Restarted the browser, and then after I clicked on a link that was set to open a new window, it opened and the layers.dll caught it and incorporated it into the existing frame--i.e., it opened in a new layer by default.