What I would do, after much thinking and turning the box inside out, is separate the left side of the inner loop.You need to put in 4 sets of double gaps -- in both straight tracks and both cross tracks. It doesn't matter much where in these tracks they're placed, but someplace to the right of the turnouts, before the next turnout.
I would set the loop uo with two reverse switches -- right side and left side. The switches would be mounted on a panel and would give the direction on the end curves of the figure 8 (assume the power pack switch is set "forward"). You could mount them on a track diagram to make it clearer. Then for the loop one would be up, the other down. For the figure 8, both up or both down.
I'd also probably go for nice toggle switches rather than slide switches, both for strength and ease of use.
And another one for the outer loop.