There's one situation where you need the buyer's public key -- if the buyer signs the message as well as encrypts it, some (or most, or all -- not really sure) software chokes and doesn't decrypt the message instead of just decrypting it before telling you it can't verify the signature as authentic.