I am not sure what you are looking for exactly, but a few games that I think might be it:
Amber: A diceless game. Note that this doesn't mean there aren't rules, just that the rules don't use random elements. Players still build a character, but the GM is the ultimate arbiter of what happens as the result of each action. It might be a bit too heavy handed, especially if used specifically to avoid the kind of behavior you mentioned. But it might still be fun, and might help get your player out of that mindset.
Advanced Fighting Fantasy: Based on the fighting fantasy books, this game is really simple. In fact, so simple, it might be impossible to realy min max it. Instead, making play interesting depends almost entirely on going outside the system. If you happen to like the old gamebooks, this is a really good option.
Old AD&D with character stable: This probably isn't what you are looking for, but I am including it just in case. Older editions of AD&D had way less customization options, but plenty of character generation options. The game was built with the assumption that each player would go through several characters, starting from scratch when hil old character died. This is why it was so random as well; each character was a kind of gamble. If you hit really high stat numbers, you might use a more interesting class, like paladin, or assassin. Or a less common race, or both! If you had two very good attributes, you could try to "go the distance" with a high level dual-class human. If you play this right, your player could have plenty of min-maxing fun, but he probably wouldn't be able to break the campaign. Also, you could get to see the look on his face when his high-level character dies.
Sandbox Camapaign: This would work with pretty much any system, though some are more or less well suited to it (old AD&D again is well suited). Basically, just throw away notions of challenge rating, of encounter design, adventure design, etc. Instead, create a location according to some internal logic. Be sure to track stuff like food, rest, time, npc morale and whatnot. In this kind of game, min-maxing is still useful, but is far from being the most important thing. It doesn't matter if your character can dish out 100 points of damage per round. Heck, a powder keg probably could do that as well. But if you starve, or get a disease for sleeping in the open, or end up in a noble's shit list, that powerful attack won't matter any. Basically, this kind of game makes those things without very hard rules a lot more important. Consider things like search patrols, mass combat, the level of noise the players are making, monster lairs, etc. If the players don't play smartly, they will be overwhelmed, no matter how powerful they are. This mode of play is a bit hard on the GM, though.
Hackmaster: Basically, this is both earlier options mixed. And you play exactly like the guys in the Knights of the Dinner Table comic strips. It won't make the player stop min-maxing, but it will give him a lot to aim for with his min-maxing!
Traveller: The character creation is very random (although you do exert some control), and after created, PCs evolve very slowly or not at all. This is one of the first science fiction games ever published, and it has nice rules to generate random space sectors as you explore the galaxy (as well as several other random stuff).
Runequest: Another fantasy game, this uses the setting of Glorantha, the same one as the King of Dragon Pass game. Runequest's PCs never get to be as powerful as, say, D&D PCs. But what power they amass is usually linked to cults where they can learn to use magic as well as obtain temporal power. So, if your player wants to min-max, he can't just pick options from books. He must actually find people willing to teach him stuff and prove himself to those people. The system this game uses was the basis of several Chaosium games, including Call of C'thulhu.
sometimes it takes a few secession, but in the end I've never had any survivors.
