Except you lose your PGP key unless you backed it up somewhere. You can enable a persistent volume with the Ubuntu Live CD, but it won't be encrypted by default. It may be possible to do some LUKS/dm-crypt magic on it, but I'm not sure. I think it's easier just to use Tails.