The downsides are that you'll need to tinker with it much more to keep it all working, and that you compromise security of your phone to an extent (apps with root permission can do whatever the fuck they want without you ever noticing, including installing rootkits etc). You'll get less "it just works" and more "oh it stopped working with this update so now I'll have to find/wait for a solution".
My opinion is that the attractiveness of rooting has been fading for years now. If you look at Android several years ago some of it's UI choices were pretty bad and needed fixing, but Android has become much much better over the years. I've rooted and custom ROMd all of my phones so far, but the number of modifications I've made has been steadily declining. A month ago I bought a phone that doesn't have a rootkit yet and no custom ROM support and so far I haven't really missed it.
And now for something completely different: