Starting with the plan you started, I would make the very end of the permanent bench the harbor/fishing village area. Don't forget to locate a building along the docks there where refrigerated cars are loaded with fresh caught fish. If your modeling 40's-50's, you'll be running ice reefers, so put a second siding to the rear of the bench by the harbor with an icing dock. Also since you are hauling fish in your reefers, you want them kept colder than produce, so there should be bags of salt at the icing dock to salt the ice bunker to make the reefer colder. Because it is going to be a relatively small layout, you will be limited to fairly short trains. It might be interesting to make all of the fresh fish shipments in express reefers at the head end of your passenger trains. That would add the challenge of having the fish loaded and the cars iced in time to be cut into the head end of the passenger train before it leaves the station. We are probably talking about only 2-3 reefers total. That means you could bring the reefers down to the dock with a small switch engine or small steamer like a 2-6-0 mogul with a working coupler on the pilot. That means we get to the fish dock pulling our train in and we need to uncouple and run around behind the caboose to push the reefers into the fish brokers facility. Once loaded, you need to back the train out of the fish dock and push it into the icing dock to ice the bunkers. The total train length we are talking about here is a smallsteam loco, a tender (this could also be a great place to use a docksider type saddle tank locomotive eliminating the tender) 2-3 40 or 50 foot express reefers and a caboose. I'm visualising a train no longer than a typical computer keyboard. All of this could take place in the last 2-21/2 feet of the permanent bench.
Now I would model the passenger terminal as a flat along the wall adjacent to the fishing port. I think with your layout size I would run a 4-6-0 for passenger power with the aformentioned express reefers, a baggage car, mail car, a couple of coaches and an observation car. An 18 car passenger train pulled by a 4-8-4 or an ABBA set of f7's would chase it's tail around the layout. I think the passenger station should be condensed to about 2 tracks that are stubb ended. We'll operate the passenger station like Union Station in Los Angeles where the trains pulled into stubb ended sidings to offload/load paeesengers. Then when the train left, it backed out onto the wye and turned to head back out. We'll get to the "wye" later We have now used the back 8-12 inches of the next 8-10 feet of the permanent bench on the wall. In the area immediately in front of the terminal, will be the mainline/branch line going to the harbor. Put a runaround right in front of the terminal tracks to allow the reefers to be cut into the front of the passenger train. We have now used the first 10 or 12 feet of the permanent benchwork. There is a little bit of room left in the front for scenery and structures.
Now we get to the "wye." I would eliminat the crossover in the plan you drew completely, and put in a "wye". I think there is room for a "wye" with #6 turnouts and a 30 inch radius between the oval you drew and the permanent bench. That is where the passenger train will turn when it is backed out of the terminal.
The other thing to keep in mind is that even though you need a 30 inch radius for the passenger train, if you use small freight steam like 2-8-2 mikados for freight power and 40-50 foot freight cars, you can negotiate 18 inch radius curves without a problem. If you are using 18 inch radius, you have room to bring a siding to the inside fo the wye to a lumber yard and coal dealer. I think there is space to fit a run around track in the area in front of the wye to allow you to move the engine behind th train and push the cars into the lumber yard and coal dealer.
Now for your "red" branch line. Your branch line can be tighter radius than the mainline, so bring it off the inside of the mainline where you have the switch that leads to the crossover. Now you can have that branch wind its way up a hill toward the central backdrop, and around the back drop. On one side of the backdrop at the top of the hill put a part of a small logging operation. Most of it would be flats on the backdrop with just a little 2 or 3 track yard where log emptys can be dropped and log loads picked up. Around the other side of the backdrop you put your coal tipple with a couple of loading tracks, and a run around to an Atlas turntable where the locomotive can be turned, and the caboose put on the other end of the train. Now it can head back down the mountain with loads of coal, stop at the logging camp on the way back and pick up loaded log cars then head on down the branch. When the train gets back to the mainline, it continues around the mainline to the other side of the central backdrop. Put your sawmill between the mainline and the branch on the far side of the central backdrop between mainline and the branchline. You can then use a couple of sidings to create a "loads in-loads out" industry. In this case you would have loads of logs coming in and loads of lumber going out.
I'm afraid I'm not good with drawing, but I think I can draw a "word picture" that someone else could pick up and run with.