Well the idea is that only the recipient has that key. Just like a bitcoin wallet. But you're right, if a centralized server had it or something, that would be bad.