Short Answer: Leadership. Mages get pitiful leadership, so your army is perhaps one fifth of that of a warrior, and one third of a paladin. You can shore some of that up with summoning spells though, but they tend to be expensive and not allow for dualcasting.
Mage leadership isn't that bad. IIRC its 100% leadership gain for warrior, 75% for paladin, 50% for mage. The bigger loss is in attack/defense points, which gives +3%/-3% damage respectively for each point compared to the attacker's stats up to a max of 3x normal damage attacks and 1/3rd normal damage taken at a 60 point differential. Warriors can usually expect around 20-30 extra in both attack and defense from skills, stats, and equipment, so thats a huge difference on top of the leadership.
Now, what mages CAN do to use their army is something like this: Take a good melee unit, ideally with base physical resistance. I'll consider Guard Droids since you can always get them pretty early and droids are by far the most imbalanced unit in the game. 20% physical resistance + 7% from medal + 10-25% from items (more is possible, this is what I would call average for mid/late game) = 37%-52% base physical resistance. Then you cast Stone Skin and this gets boosted to 77%-92% physical resistance. This ranges from being much better than a warrior stack to fucking invincible against almost everything in the game. Keep in mind that your defense reduces damage on top of this, even with your mediocre mage bonuses expect another 33% reduction or so. If that's not enough Divine Armor will give +50%ish damage reduction against ALL damage sources, though it will cost a ton and not last as long as Stone Skin. With your other spell for your first turn you use, say, Berserker, which adds +150% to the attack rating of the unit, counting both the unit's base stats and any stats your mage has, taking you from being 25 points below a warrior in attack to 25 points above them. If that's not enough, cast Hell Breath and multiply your damage by another 1.4x. Also cast Mana Spring on the second turn to both give the unit a defense bonus it doesn't need at all and give you +5 mana back every time it takes damage.
This also works marvelously for anything with Furious, which means you can just move your stack into the enemy units and watch as every one of their melee stacks attacks you for 50 damage then gets counter attacked for 50k and dies instantly. Royal Gryphons are superb, and Black Knights are insanely broken with the equipment that boosts them.
And yeah, otherwise, summons rock. Phantom with intelligence and the summoning boost can give you a duplicate of any stack you own at up to ~85%ish strength. Or double cast the 2nd tier phantom for 2 stacks at 70% strength. Did I mention the new unit can us all of their abilities again? You can phantom units that summon other units to summon a shit ton more units (which are still real units even if they come from phantoms). Against one of the bosses that summoned units every turn I simply covered the whole field in 5k Thorn stacks so nothing could come in. lols. Then you can abuse Paladins to AoE revive your units, give extra turns, and plain kill things. Inquisitors to spam your highest level dragon skills multiple times a turn. Archmages to mana shield shit everywhere for -50% damage, then spam their attack which reduces enemy initiative to one. Etc, etc.
Summoning spells that give leadership-based troops start to suck after a certain point, but the individual uber-units still kick ass. They get attack/defense bonuses from your attack/defense, your intelligence, then their base damage and health is increased based on both your intelligence and summoner skill. My end game infernal dragon had 3.5k health, 109/159 attack/defense, 60% crit and 700ish base damage per attack. Since that attack and defense is far beyond almost anything else in the game, you can consider him to have 10k health, 2k damage with a 60% chance to crit for nearly 4k. In addition to the special abilities, like always surviving the first lethal hit, regenerating 30% of it's health when under 10%, hitting multiple targets, lighting everything on fire, immunity to fire/magic/astral, 30%ish protection from physical, and transforming all killed enemies into juicy rage bonuses. Not going to win the game on it's own or be as impressive as a fully buffed well-placed phantom (though much better early game with lower leadership), but you can drop it right in the middle of the enemy on the first turn and by the time they've killed it your ranged attacks have killed off most of their force.