One thing concerns me a little.. "F" in your payload description. I think I get why it's there (so you can know which of your friends has already seen it),
Yes, because of a few reasons. For one, because we don't want to waste bandwidth pointing people to things they already know about. For two, in the case of private messages between multiple parties, we want each party to know who the other communicators are, so they can tag responses to the message for each of them as well, and keep the coversation synchronized.
F. Information allowing the people who obtain the message to determine which of their other contacts have been pointed to the message
1. It can't be an actual list, because the payload stays static after its written.
That is true. We could have it be an initial list of people who received the message, but if more people are pointed to the message there is no way to update the initial list. We need to keep tabs of who knows about which messages, otherwise people will waste bandwidth pointing their friends to messages they have already been pointed to a thousand times. Also, nobody will know who can already view a message they make a reply to, and thus things will become totally disorganized.
2. So it has to be something that your client can use to derive search keys and loop through your contacts, seeing who has received a link to
it via more searches. The problem is, what are they searching for? Are clients going to post a read-receipt?
Alternatively, we could make it so only the initiator of a post can authorize new people to see the resulting thread. Except what if somebody made a reply to the initial post that they don't want new people to see? We would need to add another level of control, public, deferred, private. Public posts can be pointed to by anyone, deferred posts defer to the starter of the thread in regards to who can be pointed to them in the future, and private posts cannot be pointed to by anybody. However, we really don't want to give users so many options because it will just make a confusing system for them. Nobody wants to mark every single post they make as [Public, Deferred, Private] and [Introduce, Hide]. But if only the initiator of the thread can invite new people to it, then we can have the initial list of people the message is sent to, and any new people who are invited can be included in a message sent by the person who started the thread.
But this will not work for public posts, because by their very nature we need to let anybody point people to them. But we do not want to have a hundred people point somebody to a post that they already know about, simply because it is a waste of bandwidth to upload and download the metadata packet, as well as a waste of processing power for the PIR-like servers etc.
What we could do for public posts is this. First of all let's remove section F from the payload and make it part of the metadata packet. When Alice makes a public post, she sends it to Bob and Carol. So she includes Bob and Carols name in section F (or perhaps a bloom filter or something will be better, we can think about the technical details of how to do this part next). At this point Bob and Carol both get the message from Alice, and they can determine that the message is viewable by Alice, Bob, Carol. Now Bob is having a conversation with Doug about the same subject, so he merges Alice's post perceptually with Dougs post to make a 'weave', but since Alice's post is marked public Bob decides to socially merge the weave together into a single thread. Now Bob points Doug to the original message, and his metadata packet includes that Alice, Carol and Bob can see it. At this point Bob needs to point Alice and Carol to the previous post by Doug as well, and when he does this he sends them the metadata packet including the pointer to the post from Doug as well as the information that Bob, Alice, Carol and Doug can see it. Now Doug likes this new thread and wants to include his contact Earl in the conversation. Since Doug already knows that the message can be seen by Bob, Alice, Carol, and Doug, when he points Earl to the messages making up the thread he includes in the metadata packet that the previously mentioned names can see these posts, and when Earl makes a response to the thread Dougs client points Alice, Carol, Bob, and Doug to the post by Earl, including Alice, Carol, Bob, Doug, and Earl in the metadata packet.
Something to that extent anyway, I kind of confused myself writing that honestly. Organization of multi party communications without a centralized administration or even forum, or even shared perception of what a 'weave' is, is going to be tricky, and is obviously something that still needs thought given to it. But I don't think it will be harder than it has been to implement all the crytpo crap we have done, or to implement all the crypto crap that we still need to do.
And any time you only require (something from the Payload) and (a list of contacts) to pull that list of who has links, an adversary should be able to loop through *all* known contacts in the system and see if each has seen it.
Anybody who is able to decrypt a post needs to know who else has been able to decrypt the post so that they know to point them to any responses made to the post if they want to. On the other hand, we don't want people to learn that people they do not know have seen the post. There are cryptographic solutions to this problem. The problem being: Alice sends a message to Bob and Carol, and she wants Bob to know to send replies he makes to Carol if he knows who Carol is, but doesn't want Bob to learn anything at all about Carol if he doesn't already know who she is.
It prevents you from resending the links to people who have seen the payload, but most of the downside of accidentally resending it to someone who's seen it (deep copies) is negated by the tiny size of the metadata object. You'll know which of your friends were on the threads.
Yes the metadata can be quite small compared to the payload. However if 100 people point somebody to something that he already knows about, it could still add up and be a big waste of bandwidth and processing capacity of the PIR-like server as well. But even more importantly, there will be no way to organize threads if nobody knows who is a part of them. If Alice sends a message to Bob and Carol, Bob and Carol need to know that each other can see the message to be able to make replies to the original message that both of them can see. Otherwise Bob will only respond to Alice and Carol will only respond to Alice, and there is no group messaging taking place but rather a running conversation between Alice and Bob and between Alice and Carol about the same subject matter.
How would a public forum (aka newsgroup, whatever) work? You'd need a way to search for all metadata objects tagged as belonging to that group, grab them, apply WoT/etc, then download payload for whatever you actually want. But public forums aren't one-to-many messaging, they're one-to-ANY messaging, and that's actually significantly different from your described use case.
There are not group tags but rather individual tags between each user. Picture it more like E-mail I imagine. Okay, using usenet as example. Alice sends a message to a usenet newsgroup, and it is encrypted and tagged. This is the message on the PIR-like server. But now Alice needs to let her two contacts, Bob and Carol, know about the message. So she sends an E-mail to Bob with the tag of the message and the key to decrypt it, and a different E-mail to carol with the tag of the message and the key to decrypt it. She also says in her E-mail to Bob that she also sent the message to Carol, and in her E-mail to Carol that she also sent the message to Bob. So now Bob goes to the newgroup and finds the message with the tag concatenated to it, then he decrypts it and reads the message. Now when Bob responds to the message he posts a brand new message to the usenet group, and now he can either only tell Alice about the message if he does not want Carol to read it, or he can tell both Alice and Carol about the message. And the communication carries on like this, each participant makes a new post to the Usenet group and then E-mails everybody they want to see the post telling them its tag as well as the key to decrypt it. There is not a 'group' that Alice and Bob and Carol are part of, and there is no tag that is between them, rather they act like a group and can communicate like a group because they all know each others E-mail addresses and can inform each other about the posts they make in the 'thread' that they all know the other knows about. Does that make sense?
The main reason that I think that public forums are so important to anonymous communities is that they're one of the few ways for new people to join in discussion. Otherwise, while you might end up with your own slowly-growing circle of trusted friends, and that works great for you. But the new guy can only talk to himself. Plus, you need a plausible way to introduce your sockpuppets to your friends.
Yes there is some difference between this and a truly public forum like SR, it will be harder to bootstrap into a system like this. Hopefully not by much. If you can think of a better design please share it, right now we still need to finish the fundamental cryptographic systems before we even really start on the forum that uses them. Next on the list is implementing a PIR like system for message retrieval. Already done with mix code for forward messages, including a pretty sophisticated cryptographic packet format (will post my code in a week or so after I discuss this with some others who have contributed code as well, but I do hope to bring you and Astor and SS on board and release all the current code publicly)
Mix packet format implemented: https://research.microsoft.com/en-us/um/people/gdane/papers/sphinx-eprint.pdf
Also have modified Sphinx to support Alpha Mixing which is implemented: http://freehaven.net/doc/alpha-mixing/alpha-mixing.pdf
Also have modified Sphinx to use bloom filter to protect from a potential timing attack: https://en.wikipedia.org/wiki/Bloom_filter
So forward anonymity is totally done at this point, as well as a bunch of other stuff (IE: have the related ECC algorithms wrapped very nicely, Tor interfaced with, etc). Right now I am looking for the PIR-like system to implement. This looks interesting:
www.cs.berkeley.edu/~dawnsong/papers/2009%20new%20techniques%20a16-bethencourt.pdf
so does this
https://www.usenix.org/conference/foci12/one-way-indexing-plausible-deniability-censorship-resistant-storage
Both of these look like they will be a challenge for me to implement, thankfully some others helping have more skill than I do. Do you think you would be able to help implement either of these, after looking through the paper briefly? (in the case of the berkeley paper note that we would need to use the dictionaryless system, as we will not have a premade dictionary of words to search for, and in the case of One Way Indexing some modifications will need to be made as well). The berkley one at least needs a homomorphic encryption algorithm, thankfully there is this already done: http://hms.isi.jhu.edu/acsc/libpaillier/ , also there is a rough implementation of that PSS already implemented in I think C++ but it is not production worthy and I would probably just use it as a reference. However, something more similar to the one way indexing is probably better as it looks like we can prevent the PIR-like server from censoring specific content.
As an example (of a new guy, not sockpuppetry), I just showed up on SRF because I enjoyed browsing some of the security-related threads. I'm probably not the target market here. Don't buy/sell on SR, haven't seen anything for sale there that fits my boring lifestyle. But this place has an impressively diverse, open-minded crowd, and one that's actively *applying* anonymity technology.
Indeed
.