First up, coming from a design theory perspective, there is a division between Perks and Advantages/Disadvantages systems. With Advantages/Disadvantages, every bonus taken must be counterbalanced by an equal negative. Which is a system that can still be min/maxed, but it is far more painful to do so, and it will often lead to making difficult choices. Or choices with consequences, in the parlance of the Codex. Perks, on the other hand, are straight up power tweaks to the character. The only consequence to the character is if you choose a lesser power tweak than a greater, thus making your character marginally less min/maxed.
But the key thing is not that these separate ideas coexist, but how implementing one or the other influences the nature of the game. Perks are part and parcel of the endless line of power-ups that is modern gaming design. The only negatives that remain anywhere in the character sheet are in dump stats, plus you're not allowed to dump very deep, and your dump stats don't really penalize you anyway, because they're 'dumps'. Everything upwards, all the way, no difficult character choices, next stop Demi-godhood.
From a modern player perspective, though, the only negative to Perks lies in the late game. Now, a Perks system that only altered non-gameplay situations would change nothing, would be quite fitting in story-based games. (Character background perks, for instance.) But that's not what people want - they want perks that directly tweak their character stats, so that they can further min/max. Which becomes a problem in the middle and end game as people who are good at games min/max their way beyond the power curve of the game, leading to a boring, overpowered slog at the end. At least, it's boring for those who actually want to be challenged in combat. Since most people don't want challenging combat, this aspect of Perks makes little difference to most players.
From an encounter design perspective, it often leads to one-trick-ponyism. Or characters who are are uber-maxed to do one thing. The encounter designer might have been trying to make players utilize a broad range of skills, but the one-trick-ponies will only ever do their one-trick. And they will be happy when it works, and very angry when it doesn't. Or the Drizzt-syndrome - always gotta use the mega-critted-out-specialized scimitars, never any other weapon. Thus, while Perks adds variety to the character sheet, it often narrows people's gameplay choices.
And finally, there is the issue of density. Adding Perks adds a whole slew of additional information that must be learned by players in order to successfully play the game, but does so for only incremental changes to the character sheet. So, if there's a 100 Perks, one must read and understand all 100, connect the synergies with other aspects of the character sheet, and then you get to choose only a single one, in order to get a +1 to some stat. That's a huge time investment for such an incremental and shallow change. Dense implementation is normally reserved for deep material, like spells, where the investment is equal to the payoff.