The system of
Dungeon Rats works pretty well. One of the rare examples where I love a game without magic nor science and one of the only example where I love a game with small scale battles (in my favourite games battles generally put you 7 vs 25, not 4 vs 6). Granted the closest to D&D in the list yet not like D&D, sure you can always make connections, character system is not completely opposite, trading blow works quite the same and the game relies on well on point status changes too, but HP balance and (non-)progression is different and there's no big area of effect stuff. Good encounters and resource managment help the game too but unique types of weapons and overall number balance, including points of action, initiative, etc. makes a very pleasant system.
Another one I consider very unlike D&D but love anyway is the one in
Voidspire Tactics and its sequels. Even trading blows work differently in this one, it's dodge + damage reduction stats, not something I would normally praise but he game works great anyway. Numbers are very low and HP progression more like in Dungeon Rats (well it increases but just a bit) and The Dark Eye than in D&D too. The system adapts very well to open exploration. Many good parts in it, amount of different, equally satisfying options is one (there are plenty of cool classes, some relatively original), interactions between elements (schocking an entire wet area ...) are another. Also this one can be played as a super balanced system where every encounter is a threat but some players here who are not me totally destroy it, that the game is breakable yet hard to break seems like the best possible compromise to me.
To be fair the system of
Divinity : Original Sin 1 is one which is totally unlike D&D and would work great with weapons which are not level-based and better overall itemization, these kill the game to some extent. It's breakable too, but
that is not a bad thing and part of why it's fun actually. There's a time and place for games where you do super strong things to try to deal with strong monsters earlier than you should.
The Earth Lords is an extreme case where you deal 10x10000 damages per turn at the end and that I enjoyed anyway, proof that low numbers are not everything, including in a game like this one where half of the quests can be do in any order (you have to do the other half in order). I guess
The Aethra Chronicles is another game which does not quite feel like D&D but is fun anyway, I could probably mention most of the Might&Magic series as well. All of these are quite non-linear and work, so their system can't be that bad.
One thing I absolutely hate despite it ironically appearing in games I love such as
Disciples of Steel and
Helherron is most XP gained via a character individually getting 1 (or 2) XP each time he performs an action instead of the party gaining shared XP based on dealing with obstacles and enemies.
