Yeh, me tooI always thought that little door was for throwing the spent cases out of... but yea, pistol port makes sense too.

In a mater of fact they find it a weak spot and deleted it in later production, but returned by demand., for a reverse use of what you thought

Here is a citate from Wikipedia:
"Being a ballistic weak spot, firing ports are often reinforced with additional armor, and in subsequent designs reduced in number (BFV), or deleted (Sherman and Tiger I [January 1944]) Other armor is improvised such as slat armor to stop shaped charges or chicken wire to stop grenades.
However due to strong tanker demand they are sometimes brought back as happened with the Sherman. This was in part due to its use during ammo resupply in the Sherman, eliminating the need for an additional crew member to pass ammo through the loader hatch, instead being able to simply pass the ammo from the ground through the firing port.