Dragon Age on nightmare is actually quite challenging a lot of the time. Some of the fights are very tough and require a lot of preparation or understanding of the spells. I still find the Archdemon, High Dragon, Gaxkang, Branka + golems, and others to be very challenging encounters that require a good bit of reactivity and correct builds/equipment. However, it's also not "unfair" like some games can be. There is HP bloat, but more commonly enemies just become significantly more deadly and use better spells and combos more often.
There are also a lot of cheese tactics you can use. If you are smart or want to break combat, you will learn to love kiting with ranged party, Arcane Warrior, Glyph of Repulsion, Mana Clash, as well as the rather overpowered spell combos. You can also drop AoE nukes on the floor of most rooms before enemies aggro and watch as they die while they stand around in their non-detecting phase (wandering enemies would add a LOT to the game's combat).
For tips, this guy has a lot of good videos on specific fights as well as general tactics:
http://www.youtube.com/user/crusherven
There is more depth to the combat system than people here give it credit for, the main problem is that RTwP tends to always make stuff a bit cluster-fucky, and the game has too many exploitable tactics and "steamroll" options compared to Infinity Engine titles (especially Baldur's Gate 2 and Icewind Dale). If it wasn't for the game supporting typical popamole "run through and kill whatever comes willy-nilly" gameplay it would be much more interesting. I know I would appreciate the mechanics much more in a fully turn-based settings because the interactions (different elemental damage types interacting, fire spreading, spell combos, etc.) would really shine there.
As far as BioWare games go I would rate quality of combat as: Baldur's Gate 2 > Dragon Age > Baldur's Gate > Dragon Age 2, although Dragon Age only beats Baldur's Gate due to better bosses and special encounter design (the rules themselves are inferior). If you beat Dragon Age on Nightmare, getting into the Infinity Engine games should not be a problem.