Since you store the private keys, it all boils down to trusting you. What's to stop you from snooping on people's messages, getting their personal info and blackmailing them?