If you want to see how to build your layout using styrofoam (this is the stuff that is either blue, pink, or light green in colour - never use the white stuff) visit my website, click on DCC and then select how to build a module.
While styrofoam has been around for quite a few years now, it has not been readily accepted as a medium for making scenery. The standard still seems to be that old standy - plaster. Which is okay if you're not trucking your layout around all the time.
Since I belong to a "module" railroad club
http://www.hotrak.ca
it's very important that my modules are as light as possible. This is where styrofoam fits the bill. I use styrofoam for the deck and for making the hills and mountains. It comes in sheets that are usually 2' x 8' and from 1" to 4" thick. You can stack it and glue it together using either wood glue or a polyurethane glue (polyurethane glue is real neat!)
The old ways of carving the styrofoam told you to get one of those "surf-form" files. Don't!! This creates tons of electrically-charged crumbs of styrofoam that stick to everything. Which is probably why the use of styrofoam has never really caught on. There are, however, other ways of cutting the stuff from a simple hacksaw blade, a utility knife, a butcher knife (the "de-boning" knife is probably the best type), to "hot-wires".
If you're using knives to carve the styrofoam, be very careful! You can quickly cut yourself.
I like to use styrofoam because, like a sculptor, I can stand back and see what the carving looks like against my layout, cut off a slice here, a chunk there, glue a bit more there. And, if I don't like the results, it's very easy to start all over again.
Once I have my piece carved to my satisfaction, I glue it to the layout and blend the piece into the surrounding scenery with some ultra-light spachling compound. Once the compound has dried and I've built it up to match the surrounding contours, I take a very wet J-cloth and rub the spachling compound down to blend the new piece even more into the surrounding area (like plaster, spachling compound tends to dry in lumps and bumps).
I then paint the piece I've added with some flat brown latex paint (the kind you get from your building supply store) (my colour is "underbrush brown"). A couple more coats and all of the pink (or blue or light green) has disappeared. If I want to add a few more nooks and crannies to the whole thing, I simply pull out my carving knife, or other tools and carve away. And then cover up the pink with the flat brown latex paint.
From here on, I use the standard scenery techniques for the ground cover, ballast, trees, etc, etc.
My only suggestion (in addition to sharp carving tools) is to have a large garbage pail close by as you carve each slice. Clean up the mess as you carve.
And take your time to create your masterpieces.
PS - To use polyurethane glue, spread a bead or two on the bottom of the piece of styrofoam and spread it as thin as you can with a 1" putty knife. Spray the base of your deck where you're going to add this new creation with water (use one of those spray bottles). Press the carved piece onto the deck and hold it in place for about 15 minutes. If necessary, add some weights (a couple of 1lb sand bags, a hammer, or whatever is handy). In about an hour, the glue has set and it's ready for the spachling compound.