I think you should be able to use the cutting doors and windows anyway, as long as you copied the properties of the sample wall and used them. It's all about which layer the walls are on.

I cut this circle I drew with a regular cutting door symbol after I converted it to a path using "Line to path" in the right click menu of the Fractalize button. Looking back I can see I forgot to mention the conversion.

If you treat the smooth walls the same way they should also cut.

And in fact I've just tried it and discovered that you don't even need to convert them. Both Circle and smooth line should cut, but they must be on the same sheet and layer as the walls drawn by the regular wall drawing tools.

All you need to do is import the fill styles relevant to MS Overland.

Make a new map in the MS Overland style and save it as a temporary file, then come back to this file and use Draw->Insert file to import the temporary MS Overland file. Don't paste it. You don't need to. The fills will be there when you look again.

I had this issue with my own recent lit dungeon map, and solved it by duplicating the WALLS sheet, naming it WALL CAPS, and moving it above the lighting setup to hide the tops of the lit walls.

I then added an RGB Matrix Process to dim the colour of the copied walls to be the same as if it was still within the lighting setup. In this case I had used 35% shadow opacity in the Global Sun settings, so I set the Unity matrix to .35 instead of 1 all down the diagonal.

If I hide the WALLS CAP sheet I have the same issue as you do.

In fact I have it even more than you do, since the orange lights mix with each other over the tops of the walls and produce a strange green glow.

Only put the walls on the WALL CAPS sheet, since things like windows and doors are symbols and are not dimmed by the RGB Matrix Process. So they just look really odd.

It is easier to use the Color Key sheet effect.

Fortunately, I happen to be working on something similar, so I can show you some screen shots.

The visible floor here is on the FLOORS marine sheet. There is a Color Key sheet effect on this sheet.

The Color key knockout colour is magenta, so when the Color key is activated the magenta cuts a hole in the floor to reveal the underlying sheet - one called INLAY BRONZE. There's a polygon of a bronze texture on that sheet.

So all you would need to do to adapt this to your own map is to add the Color Key to the sheet where you want to cut the holes and draw the holes as separate shapes in the knockout colour.

In the Symbols menu you will find "Sort symbols in map".

Pick that option and select everything (right click, "all"), then press D for do it.