Most smaller narrow gauge railroads, which were the majority, used dirt and gravel from the drainage ditches excavated alongside the track. I use paver sand to represent that gravel. I have an article at:
http://www.pacificcoastairlinerr.com/gravel/
Thank you if you visit
Harold