The most obvious problem with a simple, classical picture of virtual particles is that this sort of behavior can't possibly result in attractive forces. If I throw a ball at you, the recoil pushes me back; when you catch the ball, you are pushed away from me. How can this attract us to each other? The answer lies in Heisenberg's uncertainty principle.
Suppose that we are trying to calculate the probability (or, actually, the probability amplitude) that some amount of momentum, p, gets transferred between a couple of particles that are fairly well- localized. The uncertainty principle says that definite momentum is associated with a huge uncertainty in position. A virtual particle with momentum p corresponds to a plane wave filling all of space, with no definite position at all. It doesn't matter which way the momentum points; that just determines how the wavefronts are oriented. Since the wave is everywhere, the photon can be created by one particle and absorbed by the other, no matter where they are. If the momentum transferred by the wave points in the direction from the receiving particle to the emitting one, the effect is that of an attractive force.
The moral is that the lines in a Feynman diagram are not to be interpreted literally as the paths of classical particles. Usually, in fact, this interpretation applies to an even lesser extent than in my example, since in most Feynman diagrams the incoming and outgoing particles are not very well localized; they're supposed to be plane waves too.