For mountains, i make a screen foundation out of screen mesh ( the plastic kind for screen doors ), then i apply paper towels that i have soaked in a soft slurry of pre made drywall compound (Spackle) . I apply several layers then dress it up with my dremel.
For the " solid" rock formations, i use the blue Styrofoam, scratch it up some, then apply drywall compound.. I then spray the formation with a water based latex paint , specially formulated for Styrofoam.. You can purchase this spray paint at most craft stores..
Oh, and by the way, have a good shop vac handy with you when you do any of the cutting of the drywall and Styrofoam..