One thing that hasn't been mentioned in this discussion is insurance coverage requirements. I wonder if an insurer could deny coverage in an accident based on the fact that you had removed a factory safety feature. For example, if someone sideswipes you and the insurer claims you could have seen them if the mirror was still there...I remember Dad installing a passenger side mirror on my Mom's brand new 1963 Dodge Dart. Of course, back then even the heater was an option on the lower models.
From what I've seen, the law has one of two options: either the interior mirror looking out the rear window, or a passenger side exterior mirror. But every state has their own laws, and I haven't read every one. I think most manufacturers put mirrors on both sides mostly for the look, keeping it symmetric.
I have the same thought about people who have removed the noisemaker that sounds at low speeds. If you hit a pedestrian and they say they couldn't hear you, you could be in trouble.
I actually worry more about insurance than the legal aspect. In the unlikely event a cop noticed and decided to write you up, at worst you get a fix-it ticket that costs you a few bucks. If your insurance company denies coverage in an accident, you could be in for a world of hurt. Neither is particularly likely, but the risk reward seems not so great.