Anti-aliasing basically smooths over edges. The higher resolution you run in, the less noticeable anti-aliasing becomes because the pixels themselves are smaller. So if you're running 640x480 it's a lot easier to notice rough edges when you only have 640 across, than in 1600x1200 when you have 1600 across. Generally speaking, the higher the anti-aliasing the "better" it will look, but the lower performance you'll get. So settle for a setting that doesn't make your FPS crawl. At ~1600x1200 (or the corresponding wide screen resolution), 4x-8x is usually more than enough, at least for me.
When you say strange glitter, do you mean white pixels? It's a graphical artifact, I believe with ATI cards?
In any case, if you post screenshots it might be easier to figure out what's happening
Use Print Screen to take a shot, it will tell you in-game where it's saved. If you don't see it (some Vista users don't have their keys work), rebind it to another and then it'll work.