Your question goes to a single factor: nostalgia and rose-tinted classes. In many series not eclusing Japan's , there are generation gaps whom say THEIR version is THE version and anything else is heresy and should be burned. Showa fans of most fandoms (aka first gen) tend to hate on the second gen for X reason no matter how ridiculous it would be or legit, just because it is newer and therefore seen as "inferior". Heisei fans (aka the second gen) deplore much of Showa perhaps due to its age or the costumes, effects or style, but nonetheless they accept it begrudgingly if a part of the canon if they don't dislike it. Overall, the Showa fanbase is by far generally snobbish in their views, but only generally, liek the other two.
For Sentai, it has evolved due to its times. In the 80's there was Star Wars, Marvel had some influence over the series , and the tone became to be dystopian. The Sci-finess of the franchise reflected this and its why Bioman , Shaider , Black and many of the series at the time were fairly dark. The economy began to improve and the media reflected the unsureness in oneself as a Japanese citizen. You know, the Japanese film industry and by extnesiion their economy suffered huge blows like most areas in the world and the move to television serials reflected that. Films like Godzilla came back after a decade and we see this resurgence. Anone remember Black and the Little Mermaid?
People had the nostalgia for older products and a revival of the Kamen Rider series and by extension a renewal of Sentai invigorated both series. But by Fiveman, it went stale. So with a better economy, you get more optimistic stories aka light hearted scifi or fantasy as you will, and series which rode on the unsureness of the people like Rider and the Metal Hero faded gradually as they struggled and failed to reinvent themselves for a new generation.
As seen in the 90's, the tone of the Sentai were serious to comic, depending on the franchise's and country's state. Ohranger was dark at the wrong time ( the explosion) and didn't stay with its tone and Carranger came to its rescue. People like consistency , and in bad times, well they love things to bring hope. Look at Gobusters for example.
--------
Power Rangers was the opposite because the series became more and more serious and didn't have a breather , plus change is a ***** to many. Once Turbo embraced the comic, it got far better reception but it took In Space to secure the franchise for posterity due to ratings.
-----
I'm not sure of the specifics of the 2000's , but I believe the shift was from the fall of the economy in the late 90's , which like the previous saw a return to much darker stories and themes. GARO anyone?
---
In short, Sentai evolves like anything else and for the times it was made, people loved it. Seeing as we don't have much of a generation gap any longer mostly anywhere, people can have the open mind to enjoy older series and media. Japan was never like America in this aspect with gen gaps , but you understand. It largely depends on the viewer and their tastes.
Sure its kidy right now in places, but hell, its not the end of the world. Anyone can enjoy any story so long as it is decently written ,executed and acted. And as long as a story accomplishes those three things, I mind not light hearted or dark as long as there's a balance. Too much dark is depressing and almost corny while camp can be played to its advantage. ITs how you handle a material rather than how its tone sucks or otherwise.
Stop drinking the kool aid,nostalgic men and women. Sentai has been generally the same: five multi-colored heroes with robots that give children lessons and hope and happiness.