Did the transaction involve many to one or one to many addresses? Large transactions (in bytes, meaning the number of addresses involved, not the amount of coins transferred) will take longer unless you pay a higher fee. The standard fee is .0005 BTC, but if you're consolidating 20 addresses into 1, you should pay more like .0015 BTC or it will take longer. I've seen it take 12 hours to get 6 confirmations without a fee.