The only downside is something I can not substantiate and that is that by using warmers, you increase the tire's life by reducing heat cycles.
This is VERY true.
There are 2 big reasons why i use tirewarmers on trackdays.
1. Reduces number of heatcycles, the tire lasts longer. Also eliminates cold tearing (when the surface is hot, but carcass is cold)
2. Amount of riding time.. I don´t have to take it slowly the first 1-2 maybe 3 laps before pouring it on. Most trackdays in Sweden have 15-20 minute sessions. Why spend 4-5 minutes warming the tire up every session? Thats 1/3 of my total riding time gone because i was heating my tires.
That said..
The Slick compound tires are HORRIBLE when cold. tirewarmers are absolutly essential unless the ambient temps are high AND you leave the bike in the sun. A slick compound tire works best at a temp of roughly 80-90 degrees celcius/175-195 degrees fahrenheit. Those temps are close to impossible to acheive without tirewarmers. Using slick compound tires on the street is entirly possible (i do it), but your not pushing the envelope when riding to work on a sunny morning. But it´s only after the slick passes the roadtire in temp that it will give more grip.
Silica or "road" compound tires have amazing grip and are actually better for most intermidiate riders. The reason for this is simple, the tire works best at around 50 degrees celcius/125 degress Fahrenheit but retains good grip at lower temps aswell. The tire works well in damp and even wet conditions (Well.. not like a WET tire, but good enough). Even at 8-10 degrees (~45 degress F) and wet roads, it´ll work OK. A slick would be bloddy awful at that temp..
My BEST recommendation would be to talk to a GOOD local tire guy about what riding you do and let him recommend a good tire.. OR buy the BEST or top of the line ROAD sports tire from whatever manufacturer you like. I´m partial to Pirelli, but Michelin would be my personal second choice.
Point is that.. If you feel like you need the extra level of grip that having the absolute best road sports tire on the bike is needed and that your on the level that it takes the absolute best one to keep you on the track, you might aswell make the switch to a slick compound tire and get some tirewarmers..

Just don´t ride it to work on a cold november morning with the rain coming down..
2c
//amullo