I think a bunch of the Pillars of Eternity's systems are working together in conjunction to create rote, banal gameplay. It's not just one system but a multitude of systems together and how they interact.
I agree, it almost seems like Obsidian accidentally trapped PoE combat into boring repetitiveness by not understanding how the current iterations of their mechanics work together. I like the ideas behind many of their decisions regarding combat mechanics, but it is challenging to balance this system to not break at some joint and end up having banal cookie cutter optimal choices. However, I see potential in the ruleset and think its more of an iteration problem rather than the fundamental rules condemning PoE combat to tank and spank and other boring things.
Wearing better armor also comes at a cost, you pay in action speed for every point of base protection, meaning that the heavier armor you wear, your character will be less offensively effective in combat. This outcome in a broad sense is not necessarily bad, but it helps create the situation where characters who wear the heaviest armor are always taking aggro and sent in first because they are going to take way less damage than the other characters.
I like the idea of heavier armor giving more protection at the cost of recovery, but the armor options are currently balanced poorly and lead into huge gameplay problems like this whole naked or plate tank&spank -mess. For some reason the DR to recovery scale stops at brigandine already having 50% recovery slow and plate being just a boring +1 choice over it. Having naked at full speed and plate at 50% slow like now, but making all the medium choices between them have lesser speed reduction could make reasonable armor selection less binary.
Didn't PoE have flat and % dmg reduction in armors at one point of the beta? Do you know why they ended up choosing only flat reduction instead? Having % reduction and lower than current flat reduction on armor(and built in monster stats) would mean fast weapons not becoming so bad later in the game and potentially also promote the use of medium armors as you would not be able to have the heaviest armor you can find to make yourself take almost no dmg at all.
I think the Engagement system has a big impact on the moment to moment gameplay because it prevents simple movement based tactics that I used every encounter in the Infinity Engine games. I do think it has less of an impact on the gameplay than the way some of the other systems interact with eachother, but it's still worth mentioning as a problematic system.
I'm not yet sure what to think of engagement as a concept, but it is at least way too strong now. Weaker disengagement attacks would make potentially suffering them to get into a better position a trade one would take more often leading into less static and repetitive combat. I agree with you that moving in combat is discouraged way too much. This would also allow more interesting gameplay with smarter enemy AI that would break engagements when its reasonable rather than being contempt to hug your tank for eternity.
I like there being some kind of engagement and AoO, just so that there is potential for reasonable but not annoying enemy AI. Without disengagement attacks the optimal thing is to chase your squishies and run from your melee fighters for ever, which would be annoying as hell to face as the battle would soon be happening in multiple screens. The current overpowered disengagement attack does the opposite and is also bad.
I'd like there being incentives for staying in melee(not getting hit when running away) and moving away(better position to harm enemies, support allies or not to get beaten anymore), so something between IE games and current PoE. Staying in melee or running almost always being the optimal choice is boring and I think AoO:s have potential to make this kind of combat more interesting to play, even while I agree with you that AoO:s in real time are somewhat silly.
I also agree that there are problems with the attack resolution system and deflection, but I don't currently have ideas on how to improve on those. Haven't made up my mind about health systems and hard counters. I was initially against immunities, but something like a fire elemental not taking damage from fire is very reasonable and I now support that kind of things. I don't want the game to have spells making you immune to physical dmg and some other very binary hard counter situations, but it would be good if there was more reasons to react to enemy caused afflictions and powefull buffs. While I don't prefer mages dominating battles over other classes like in high level D&D, it would be good that they had buffs worth dispelling and would not be instantly dealt with some random ranged attacks.