I've also played a ton of games with both gamepad and KB&M, and the difference is definitely not minor.
It depends on the genre and the game of course. Sure, there isn't much difference between those schemes if we're talking about TES or Fallout.
If we're talking about shooters with any real degree of difficulty though, the difference is night and day. Anyone who claims otherwise is in some serious denial.
It also depends on the skill level of the pad user, and the software implementation per game (deadzone, aim acceleration, sensitivity etc).
In my experience it is pretty minor. Noticeable always (I always disable aim assist in console games), but still minor. Only makes a real design-impacting difference at the competitive PvP level, where there for sure should be no aim assist.
I don't know why some people always make a big deal out of it. Remember PC FPS used to be played with
keyboard aiming, which is far inferior to gamepad aiming. Don't be dumb.