I agreed with everything until the last part.
"Above average" would be okay if it weren't for the fact that enemies are assumed to be anything BUT average.
Your average bandit, the mookiest of mook in most cases, usually has a class of rogue or warrior... not one of commoner class. A lifestyle of making your living attacking, stealing and killing others involves physical exertion and weapon expertise, for sure.
A level 1 bandit is a pushover. Yet a level 1 bandit is more than a match physically for a level 1 wizard... as it should be. The wizard has a lifestyle that requires hours, if not days, of intense study, with little physical activity or distraction. Sure, they get fresh air and exercise traipsing across the plains, but it pales in comparison to the physical prowess of enemies that make their living every day through physical attacks and violence, let alone animals, monsters or other beings whose very nature is more fit and powerful than humans.
Wizards aren't inherently wimps... but they shouldn't be able to easily wield deadly physical force against enemies that, by their very nature, should be more powerful than them.
I largely agree with this, but it's worth keeping in mind that in any realistic setting, not every combatant is a proper warrior. The people who live and breathe violence represent an extreme of the bell curve even among combatants, probably even professional soldiers. I don't want to downplay the importance of training, experience and constant exposure when it comes to physical combat, because it's crucial, but the fact of the matter is that even in the medieval times there simply wasn't enough
violence to go around for most people to get any good at it, even psychologically. Now compare this to what characters in RPGs go through - just on the way from Candlekeep to Nashkel Mines, Charname and his low-level buddies will probably have killed enough things to put to shame pretty much anybody short of Miyamoto Musashi or William Marshal. I've always tended to think of PC adventurers as being exceptional, not in the sense that they're "destined heroes" or whatever, but simply because the experiences they regularly go through are highly extreme by most human standards. So if in real life I was watching a street fight and had to choose between putting my money on your average meathead bandit or the starting PC wizard who's seen more dead things in a week than most people see in their lifetimes, I'd go with the wizard. If he's not a psychological wreck at that point, he's some kind of natural born killer.