I only know that in TSR's times the imagination for new settings seemed boundless. Do capable writers care about D&D today?
I think it has a lot more to do with financial concerns than a lack of imagination. By the mid-90s TSR was publishing full-fledged product lines for ten different campaign settings (Forgotten Realms, Greyhawk, Dragonlance, Mystara, Planescape, Spelljammer, Birthright, Dark Sun, Ravenloft, Al-Qadim). This isn't even counting the small sub-lines within the Forgotten Realms brand (Mazitca, Kara-Tur, The Horde) and the D&D Gazetteers/Mystara (Hollow World, Savage Coast). Then there were a few one-off settings as well (each of the Historical Reference books would count, plus Council of Wyrms, etc).
The way I understand it, TSR was losing money on most of these. They were fragmenting their customer base with so many different product lines that were in competition with one another (to some degree; it's not like I didn't own stuff from most/all of them at the time). It was one poor financial decision among many made by TSR that sunk the company. Hence, after WotC bought out TSR, they axed most of the campaign settings and focused on books that were more generic in nature. Forgotten Realms was obviously the most popular and the most profitable, so they kept it and brought out the new book early in 3e's lifespan. I suspect that they kept Greyhawk more because people on the team wanted it than because it was especially popular or profitable at that point in time.
I agree that the settings of the early-mid 90s were fantastic (Dark Sun, Planescape, Birthright in particular), but they never seemed to get the traction as Forgotten Realms. And while I greatly dislike the FR, I do have a fondness for the original "grey box" campaign setting. Before the massive supplement bloat and reams of shitty novels.
Anyway, Dark Sun. Of all of TSR's settings, I would say that Dark Sun has the best chance of being brought back as a tabletop setting (which would make a CRPG based on it more likely). It's the only one from the list above that got an official 4e version. It's different enough from Forgotten Realms (and Greyhawk, Dragonlance, Mystara, Birthright) without being as weird as Planescape or Spelljammer. It would be a great candidate for a one-year focus from WotC: release a campaign setting book, an adventure path set in it, and then leave it alone.