Warning: a wall of text.
The original game is quite dreadful. Combat, unlike most SMT games, is fought on a grid, which might sound like incline. Hold your horses, there's no Move command. Instead, you set several (4, I think) formations before combat and can only change between them. As all attacks and spells have very specific areas of effect, it's easy to end up in a situation where a character can't do anything because every potential target is out of range. It also has probably the worst plot in the history of SMT. On the brighter side, the fusion system is one of the most unique in the franchise, in that personas are fused out of demons you fight, not out of other personas. It's just as flawed as the combat is (you can't tell what skills the persona is going to inherit from the parent demons), but at least it offers an interesting change of pace. The alternate Snow Queen story branch is another good bit, in that it plays like a very old-school dungeon crawler that greatly limits your ability to save and grind. Overall, I don't think this game is worth anyone's time. Everything it does, Devil Survivor does better. If you do decide to take a look out of sheer curiosity, do yourself a favor and get the PSP version instead of the one for the original PlayStation. The translation is much better.
Persona 2 is two games: Innocent Sin (the Hitler one) and Eternal Punishment (the better one), with the latter being a sequel to the former. They play alike, so I'll cover both in this paragraph. Combat is much improved over the original: the grid is gone, fusion spells activated by several characters casting specific spells in a particular sequence are introduced. The demon negotiation system is probably the best in the series: every party member has 4 conversation options that provoke specific reactions in demons that depend on the demons' personality types. Up to 3 characters can join together to perform special actions during negotiations. The game's weakest part is complete lack of fusion. Talking to a demon gets you Tarot cards of the arcanum the demon's associated with, and these cards can be traded for personas of the same arcanum, with no skills being inherited in the process. I like these two games, although combat takes a little too long to get truly challenging and interesting. Finally, a piece of trivia: Eternal Punishment is the only Persona game that has a party composed almost entirely of adults.
Persona 3 is the edgy one. This is the game that introduced the dating sim mechanics and time management to the series. Instead of talking to demons (shadows, actually, demons aren't in this game), the main character spends time with his buddies. Each friend is associated with a certain Tarot arcanum. As you grow closer to people, personas of their arcanum get an increasingly high XP bonus when you fuse them. Fusion itself works the way it does in Nocturne and most SMT games. You trade two (or three, or more, in case of special fusions) personas for one that inherits some skills from the parents. Combat is similar to Nocturne and Digital Devil Saga, with one major difference: the extra actions a character earns by exploiting elemental weaknesses and scoring critical have to be spent by the same character. The PS2 versions of the game suffer greatly from inability to directly control your party members. Everyone except the main character is controlled by the AI, which is not very smart. Mitsuru's fondness for Marin Karin is especially legendary. There's a PSP version that introduces direct control, rebalances the game (generally for the better) and adds the ability to play a female character. Picking her changes the plot quite a bit and greatly tones down the emo. Another welcome change, in my opinion. The PSP version does lack The Answer, though. It's a sequel to the main story that was introduced in FES, a re-release of the original game for PS2. I think the PSP version is the best one, but there are those who prefer FES.
Persona 4 is mostly the same as 3, gameplay-wise, so I won't talk much about it. It's a little lighter on dungeon crawling and a little heavier on dialogue. Story is much more cheerful than in 3 and is probably my favorite in the series. There's a Vita port on the way that improves the graphics and adds new personas and some convenience features, such as being able to choose what skills are inherited during fusion, so you might want to wait for that one.