Right, obviously you need the private key that corresponds to the public key that I used to encrypt that message. If you delete your key or use a new PGP program without importing it, you won't be able to decrypt messages encrypted with that public key.