The Doctor was originally a self-imposed exile from their culture as sort of a "renegade" after all. The Time Lords have often been somewhat antagonistic throughout the history of the show, though they became more so as time went on:
-At first they were shown to be mysterious, legalistic, immensely powerful, egotistical towards other races nearing xenophobia, and yet the majority held a vow to being observers only, in terms of events in the universe, unless it was a very dire situation. And that's when they were "neutral good."
-At the end of "The War Games" they forcibly had the Second Doctor regenerate into the Third and exiled on Earth in the 20th Century because he broke said "temporal non-interference laws," in particular his involvement in the conflict of the War Lords.
-Many "renegades" such as the Monk, Morbius, the Rani, and the Master obviously weren't exactly up to any good.
-Over time, delves into Time Lord culture and history showed signs of the society being more corrupt and decadent, if they weren't already as shown by Rassilon and Omega when they were "alive" (One being a dictator in his time and the other wanting to become a "God," even at the expense of the rest of the universe), in addition to the use of the Death Zone and possibly the prison planet of Shada.
-Depending on your judgement, the Time Lords' request for the Doctor to eliminate the Daleks throughout history by wiping them out before being created could be considered another sign of their darker nature.
-Borusa, one of the Doctor's friends, after becoming President of the High Council went mad with power and tried to gain immortality from the Ring of Rassilon to become a dictator and retain his position permanently. Naturally the Ring was a trap by Rassilon to put away those that might seek such power.
-In "Trial of a Time Lord," the Sixth Doctor was put on a "rigged" trial, with the Valeyard, a distillation of the Doctor's evil tendencies between his twelfth and final incarnations, acting as his prosecutor. He was originally charged with once again breaking the "non-interference law," though the Valeyard added the charge of committing genocide against the Vervoids. The true reason for the trial was to cover up the Ravolox affair (the High Council accidentally committed wide scale genocide by unsafely moving the entire planet's solar system, and evidence revealed it might have been a far future Earth), with the Doctor as a scapegoat. The Valeyard was promised the Doctor's remaining regenerations as payment if he succeeded.
Of course, as an "evil Doctor" he began to manipulate things beyond the High Council's control. He manipulated the evidence in the Matrix to further condemn the Doctor and even planned to take control of Gallifreyan society through chaos by murdering the High Council as well. This whole story led up to a pretty famous quote by the Doctor:
In all my travelling throughout the universe I have battled against evil, against power mad conspirators. I should have stayed here. The oldest civilisation: decadent, degenerate, and rotten to the core. Power mad conspirators, Daleks, Sontarans... Cybermen, they're still in the nursery compared to us. Ten million years of absolute power. That's what it takes to be really corrupt.
They weren't all evil, most of them still reflected the "neutral good" alignment and many were allies (Drax and Romana for example), albeit begrudgingly sometimes, but there is a seeming line of progression towards what finally occurred in the last moments of the Time War. Bringing back Rassilon, or at least having a Time Lord who assumed his name, and having him take control of the High Council is a pretty big indicator at how badly things had gone wrong.
