What you are seeing is normal.
If you could think of your screen as a grid, the way that you had it before split your screen into a grid that was 640x480 units in area. When you upped the resolution, it split that same area into a grid that was 800x600 units, so each of those units was necessarily smaller. With the resolution up, you can see more things at once, but they will be smaller.
What I am guessing is that your monitor's circuitry is only designed to be able to split the screen into so fine of a grid. Once you try to exceed that limitation, the monitor doesn't know how to handle it, and it goes blank. Many games anymore are set to a default resolution of 1024x768 or more, so if your monitor can't handle that, it could exhibit the behavior you describe. What game is it?
As another note, you will probably not get great performance out of that card on that system. It will be better than it was in some ways, but that video card is so powerful that the rest of the system will have a hard time keeping up with it. It's kind of like throwing a header with 3" collectors on a Honda. I'm guessing that you will get jerky performance at some of it's higher settings once you get it working because the card will be waiting on the Athlon to send it info. More RAM will help (and it's cheap anymore), but you should think about upgrading the 550 if you want to get the most out of that card. I've seen tests of those GeForce 4's that indicate that even 1.7Ghz processors have a hard time feeding it enough data.
Hope this helps.
- Freed
