Dirt rubs off really easily. It can even wash off with rain.
But then we either need to very frequently (and very thoroughly) clean the roadways or accept that their effectiveness (of the solar panels and LEDs, at least) is going to be very variable. I'm convinced that the solar panels would be a relatively insignificant contribution to the grid anyway, so let's not worry about that; but do you really want to accept road markings that may turn out to be unreliable?
@Lorddave (sorry, too lazy to quote specific posts), fair enough, if we assume that the power to do all that stuff doesn't come from the solar panels themselves (my assumption that this would be the case comes from the Solar Roadways guy saying that if we covered all roads in mainland USA with solar panels, we could power the entire country solely off of that, as well as other claims on their page such as "we won't have to rely on foreign oil anymore!"), then everything is possible and merely highly impractical; but if that's the case, dare I ask why we even bother with the solar panels in the first place? A lit and heated road would be much cheaper, and the solar panel output is largely insignificant anyway. Why introduce an expensive and fragile element to the project if it doesn't serve much of a purpose?
How much does it cost to realize that the great outdoors are dirty and dirt doesn't do solar panels any favors?
As opposed to putting solar panels inside walls where there is no sun?
Well, there is also the option of putting them above the ground (e.g. on the roofs of houses). Sure, there will still be some dirt, but it's nothing compared to what's going on on a roadway.