Michael,
modern cat litter clumps, when wet - so you hardly can glue it down evenly with wetted water and white glue, or matte medium or other water based sticky stuff. On other, old-fashioned versions of Kitty Litter I have to second Vic: When you apply water it simply 'melts' into an oozy puddle of clay...
However, once I used a reddish version of that stuff on a diorama for the rip rap rocks at the foot of a cliff. To fix it in place, FIRST I applied white glue almost full strength (about 4 parts glue to 1 part water) at the base of the cliff. THEN I scattered the litter parts into the glue and pressed it down with the fingers. In some places, where more rocks were needed I added more glue, then litter. Looked quite fine. But for ballast it would have been much too coarse.
What I use for ballast is 'bird sand'. This is a fine grained granite sand, normally used to cover the floor plate of a bird cage. It is light gray colored and looks very good for well kept mainline ballast. I used it both on H0 and N scale layouts. And it is very cheap - a pound bag costs about 1-2$.
Only problem is: I live in Switzerland and I don't know if you can get this material in the US. Just try in a pet shop or even the pet department of a supermarket (that's where I got mine).
Happy ballasting!
Ron