Even in a single geographic area can have a variety of colours. Well maintained ballast out on the main would be clean, high quality stuff, coloured only by what the railroad is carrying (as Jim noted).
In yards, the ballast might be practically hidden under a layer of sand (for traction), dirt, oil, grease, fuel, and garbage like newspapers, leaves, etc.
On infrequently used branch lines, the ballast might not be visible at all - covered completely by grass and weeds growing right up through the ballast beside and between the rails.
Andrew