On the subject of mentally unbalanced villains, the vast majority of Batman's rogue's gallery comes immediately to mind: The Joker, Two-Face, The Scarecrow, Victor Zsasz, The Ventriloquist, Poison Ivy, Killer Kroc, Mr. Freeze, Firefly, Maxie Zeus (and King Tut on that note), Hugo Strange, The Riddler, Black Mask II, Professor Pyg, etc, etc, etc. Most of those are certifiable Arkham Asylum regulars, and those who aren't at the very least have major psychological issues!
Sticking with DC, but moving beyond fiends considered to be primarily Batman villains: The Floronic Man, Post-Crisis Psycho Pirate, Post-Identity Crisis Jean Loring, Doctor Destiny, Post-Crisis Black Hand, and Hector Hammond are also notably certifiable.
On the Marvel side, Carnage naturally comes immediately to mind as their equivalent of Batman psychos like The Joker and victor Zsasz. In fact, I honestly consider Carnage to be Marvel's rather late answer to The Joker in much the same way as Norman Osbourne is Marvel's answer to Lex Luthour. I'm aware that their version of Arkham, Ravenloft, is filled with other notable villains--but such names are escaping me at the moment (I'm much more of a DC person). The Jackal certain had an unhealthy fixation with Gwen Stacy. Thanos certainly has a rather unhealthy infatuation with death, perhaps even surpassing DC's post-crisis Black Hand (vis-a-vis Darkest Night)!
Speaking of death, 2000 A.D.'s Dark Judges' belief in life as the ultimate crime is certain twisted by any normal, sane person's standard! On the subject of Judge Dredd, the "futsies" driven mad by life in the future are also notable (if not a specific character).
Those are the villains who immediately come to mind as especially unbalanced, but honestly it seems like a great many more--if not most--have at least a few notable major psychological issues like narcissism, megalomania, persecution complex, etc. (Add baldness and a massive fortune to that mix and you have Lex Luthor!) It's just part of the territory with comic book villainy! Mua-Ha-Ha!