3AM musings.
Making 'duels' happen in a system that frequently has both guys casting spells at the same time makes things difficult, IMO. A duel tends to imply that there is action and reaction, whereas both actions applying simultaneously tends to bank on a literal rock-paper-scissors guessing game.
If I were to design a mage duel around a RTwP functionality, I'd give mages a 'charge' meter that increases/decreases based on spell usage. You would then have to give the mages a very diverse spellbook that is centered around elements of offense, defense, neutrality, and other things, with every spell either increasing or decreasing the bar. But most importantly, spells would have varied casting times as well as varied delivery times (time to get land on the enemy, for example). The overarching goal would be to implement strong resource management while trying to find an advantage over the opponent. I'd also slow down the speed at which spells become effective. The closer you get to (you go, I go, particularly) turn-based mechanics, the better. Magic bolts? Cheap, instant damage, but non-threatening. A giant fireball would give much more of a warning. You could technically cast a life-threatening spell early on, but without other things harrying your opponent, they could then just counter it with a defensive spell. The objective would be to 'punch your way in' through a mage's defenses, while making sure not to weaken yourself. Mage duels would end with spectacular combos that stretched their defense too thin instead of one-shot kills.
But again, it would be difficult to overcome the RTwP aspect, and it would also be difficult to implement in the context of a party-based game. Furthermore, the primary issue with 'mage duels' in games like these is that you essentially have all your spells from the getgo. To really add balance and for the sake of interesting gameplay, I wouldn't mind magecraft that had players 'graduating' their way through their spellbook instead of having it all available to them from the start like some kind of cheat sheet. For example, the charge bar. Every time it reaches its top, it resets, but spells cost a little bit cheaper to cast. So you could cast a giant fireball right off the bat, but it'll drain you - maybe a malus that leaves you in a state of slow spellcasting, for example. But if you cast it later in the fight, when you've topped the 'bar' a few times, it would be an easy, safe casting that had no effect on your ability to keep fighting. I'd like to see mages that reflected the combat sports - 'fast starters' are very threatening for a minute or two, but if you can survive the assault they always end up gassed and essentially slowed for the rest of the fight. There are other ways to influence spell usage, too, like casting the same spell repeatedly would earn you a malus, or perhaps available spells are somewhat randomized and 'reset' after each charge. Stuff to keep you on your toes...