Justin Horn of the When Will Apple? blog ran some tests two years ago and found that his iPhone 3 charged in 90 minutes when plugged in to a wall outlet and in 140 minutes when charging over USB. He recently ran similar tests on the iPhone 4, and his results are described in a post on Justin's blog and shown in the following graph, which Justin gave me permission to republish here:
In his test, it took Justin an extra 30 minutes (130 minutes versus 160 minutes) to get a full charge on USB versus plugging it into a wall outlet. And perhaps just as importantly, his tests show that you get more charge along the way using a wall outlet, so if you only have 30 minutes to spare, in Justin's test you'll go from zero to 25% over USB and zero to almost 35% with a wall outlet.
Different computers have different USB charging capacity, and the latest Apple computers provide more power over USB than earlier models. Thus, I expect that if you expanded this chart to cover different Mac and PC computers, you'd see different results charging over different USB ports. Having said that, I think that it is fair to say that if you want the fastest charge, use a power outlet.