The more voters the better, that's why I think even citizens who are felons should be allowed to vote. Two states allow prisoners to vote from prison; 21 restore voting rights upon completion of their sentence. You break the social contract then you suffer specific consequences. You repay that debt to society and you get your voting privileges back.
As I've said many times there are two things I'd do to improve the current system, and neither involves taking away the right to vote. One: ranked voting. Two: taxes are revenue only. No spoils to contributors in the form of tax breaks.
Edit: Holy crap, this thread has wrought another miracle: I just saw an ad for Libertarian T-shirts.
“Life, liberty, and property do not exist because men have made laws. On the contrary, it was the fact that life, liberty, and property existed beforehand that caused men to make laws in the first place.” (The Law, p.6) Frederic Bastiat