Silk Road forums
Discussion => Security => Topic started by: astor on September 05, 2013, 02:01 pm
-
Just published: https://blog.torproject.org/blog/how-to-handle-millions-new-tor-clients
[tl;dr: if you want your Tor to be more stable, upgrade to a Tor Browser Bundle with Tor 0.2.4.x in it, and then wait for enough relays to upgrade to today's 0.2.4.17-rc release.]
Starting around August 20, we started to see a sudden spike in the number of Tor clients. By now it's unmistakable: there are millions of new Tor clients and the numbers continue to rise:
Where do these new users come from? My current best answer is a botnet.
Some people have speculated that the growth in users comes from activists in Syria, Russia, the United States, or some other country that has good reason to have activists and journalists adopting Tor en masse lately. Others have speculated that it's due to massive adoption of the Pirate Browser (a Tor Browser Bundle fork that discards most of Tor's security and privacy features), but we've talked to the Pirate Browser people and the downloads they've seen can't account for this growth. The fact is, with a growth curve like this one, there's basically no way that there's a new human behind each of these new Tor clients. These Tor clients got bundled into some new software which got installed onto millions of computers pretty much overnight. Since no large software or operating system vendors have come forward to tell us they just bundled Tor with all their users, that leaves me with one conclusion: somebody out there infected millions of computers and as part of their plan they installed Tor clients on them.
It doesn't look like the new clients are using the Tor network to send traffic to external destinations (like websites). Early indications are that they're accessing hidden services — fast relays see "Received an ESTABLISH_RENDEZVOUS request" many times a second in their info-level logs, but fast exit relays don't report a significant growth in exit traffic. One plausible explanation (assuming it is indeed a botnet) is that it's running its Command and Control (C&C) point as a hidden service.
My first observation is "holy cow, the network is still working." I guess all that work we've been doing on scalability was a good idea. The second observation is that these new clients actually aren't adding that much traffic to the network. Most of the pain we're seeing is from all the new circuits they're making — Tor clients build circuits preemptively, and millions of Tor clients means millions of circuits. Each circuit requires the relays to do expensive public key operations, and many of our relays are now maxed out on CPU load.
There's a possible dangerous cycle here: when a client tries to build a circuit but it fails, it tries again. So if relays are so overwhelmed that they each drop half the requests they get, then more than half the attempted circuits will fail (since all the relays on the circuit have to succeed), generating even more circuit requests.
So, how do we survive in the face of millions of new clients?
Step one was to see if there was some simple way to distinguish them from other clients, like checking if they're using an old version of Tor, and have entry nodes refuse connections from them. Alas, it looks like they're running 0.2.3.x, which is the current recommended stable.
Step two is to get more users using the NTor circuit-level handshake, which is new in Tor 0.2.4 and offers stronger security with lower processing overhead (and thus less pain to relays). Tor 0.2.4.17-rc comes with an added twist: we prioritize NTor create cells over the old TAP create cells that 0.2.3 clients send, which a) means relays will get the cheap computations out of the way first so they're more likely to succeed, and b) means that Tor 0.2.4 users will jump the queue ahead of the botnet requests. The Tor 0.2.4.17-rc release also comes with some new log messages to help relay operators track how many of each handshake type they're handling.
(There's some tricky calculus to be done here around whether the botnet operator will upgrade his bots in response. Nobody knows for sure. But hopefully not for a while, and in any case the new handshake is a lot cheaper so it would still be a win.)
Step three is to temporarily disable some of the client-side performance features that build extra circuits. In particular, our circuit build timeout feature estimates network performance for each user individually, so we can tune which circuits we use and which we discard. First, in a world where successful circuits are rare, discarding some — even the slow ones — might be unwise. Second, to arrive at a good estimate faster, clients make a series of throwaway measurement circuits. And if the network is ever flaky enough, clients discard that estimate and go back and measure it again. These are all fine approaches in a network where most relays can handle traffic well; but they can contribute to the above vicious cycle in an overloaded network. The next step is to slow down these exploratory circuits in order to reduce the load on the network. (We would temporarily disable the circuit build timeout feature entirely, but it turns out we had a bug where things get worse in that case.)
Step four is longer-term: there remain some NTor handshake performance improvements that will make them faster still. It would be nice to get circuit handshakes on the relay side to be really cheap; but it's an open research question how close we can get to that goal while still providing strong handshake security.
Of course, the above steps aim only to get our head back above water for this particular incident. For the future we'll need to explore further options. For example, we could rate-limit circuit create requests at entry guards. Or we could learn to recognize the circuit building signature of a bot client (maybe it triggers a new hidden service rendezvous every n minutes) and refuse or tarpit connections from them. Maybe entry guards should demand that clients solve captchas before they can build more than a threshold of circuits. Maybe we rate limit TAP handshakes at the relays, so we leave more CPU available for other crypto operations like TLS and AES. Or maybe we should immediately refuse all TAP cells, effectively shutting 0.2.3 clients out of the network.
In parallel, it would be great if botnet researchers would identify the particular characteristics of the botnet and start looking at ways to shut it down (or at least get it off of Tor). Note that getting rid of the C&C point may not really help, since it's the rendezvous attempts from the bots that are hurting so much.
And finally, I still maintain that if you have a multi-million node botnet, it's silly to try to hide it behind the 4000-relay Tor network. These people should be using their botnet as a peer-to-peer anonymity system for itself. So I interpret this incident as continued exploration by botnet developers to try to figure out what resources, services, and topologies integrate well for protecting botnet communications. Another facet of solving this problem long-term is helping them to understand that Tor isn't a great answer for their problem.
-
Also, from the first comment: http://blog.fox-it.com/2013/09/05/large-botnet-cause-of-recent-tor-network-overload/
Large botnet cause of recent Tor network overload
Recently, Roger Dingledine described a sudden increase in Tor users on the Tor Talk mailinglist. To date there has been a large amount of speculation as to why this may have happened. A large number of articles seem to suggest this to be the result of the recent global espionage events, the evasion of the Pirate Bay blockades using the PirateBrowser or the Syrian civil war.
At the time of writing, the amount of Tor clients actually appears to have more than quintupled already. The graph shows no signs of a decline in growth, as seen below:
An alternative recurring explanation is the increased usage of botnets using Tor, based on the assertion that the increase appears to consist of mostly new users to Tor that apparently are not doing much given the limited impact on Tor exit performance. In recent days, we have indeed found evidence which suggests that a specific and rather unknown botnet is responsible for the majority of the sudden uptick in Tor users. A recent detection name that has been used in relation to this botnet is “Mevade.A”, but older references suggest the name “Sefnit”, which dates back to at least 2009 and also included Tor connectivity. We have found various references that the malware is internally known as SBC to its operators.
Previously, the botnet communicated mainly using HTTP as well as alternative communication methods. More recently and coinciding with the uptick in Tor users, the botnet switched to Tor as its method of communication for its command and control channel. The botnet appears to be massive in size as well as very widespread. Even prior to the switch to Tor, it consisted of tens of thousands of confirmed infections within a limited amount of networks. When these numbers are extrapolated on a per country and global scale, these are definitely in the same ballpark as the Tor user increase.
Thus one important thing to note is that this was an already existing botnet of massive scale, even prior to the conversion to using Tor and .onion as command and control channel.
As pointed out in the Tor weekly news, the version of Tor that is used by the new Tor clients must be 0.2.3.x, due to the fact that they do not use the new Tor handshake method. Based on the code we can confirm that the version of Tor that is used is 0.2.3.25.
The malware uses command and control connectivity via Tor .onion links using HTTP. While some bots continue to operate using the standard HTTP connectivity, some versions of the malware use a peer-to-peer network to communicate (KAD based).
Typically, it is fairly clear what the purpose of malware is, such as banking, clickfraud, ransomware or fake anti-virus malware. In this case however it is a bit more difficult. It is possible that the purpose of this malware network is to load additional malware onto the system and that the infected systems are for sale. We have however no compelling evidence that this is true, so this assumption is merely based on a combination of small hints. It does however originate from a Russian spoken region, and is likely motivated by direct or indirect financial related crime.
This specific version of the malware, which includes the Tor functionality, will install itself in:
%SYSTEM%\config\systemprofile\Local Settings\Application Data\Windows Internet Name System\wins.exe
Additionally, it will install a Tor component in:
%PROGRAMFILES%\Tor\Tor.exe
A live copy for researchers of the malware can be found at:
hxxp://olivasonny .no-ip .biz /attachments/tc.c1
This location is regularly updated with new versions.
Related md5 hashes:
2eee286587f76a09f34f345fd4e00113 (August 2013)
c11c83a7d9e7fa0efaf90cebd49fbd0b (September 2013)
Related md5 hashes from non-Tor version:
4841b5508e43d1797f31b6cdb83956a3 (December 2012)
4773a00879134a9365e127e2989f4844 (January 2013)
9fcddc45ae35d5cdc06e8666d249d250 (February 2013)
b939f6ef3bd292996f97aa5786757870 (March 2013)
47c8b85a4c82ed71487deab68de196ba (March 2013)
3e6eb9f8d81161db44b4c4b17763c46a (April 2013)
a0343241bf53576d18e9c1329e6a5e7e (April 2013)
Thank you to our partners for the help in investigating this threat.
-
It is kind of retarded that someone with such a botnet would use Tor to try to hide its' CNC, lol.
-
Actually let me rephrase that. It is kind of retarded that Botnet masters still use CNC servers instead of CNC signature keys and P2P.
-
So astor and kmf pretty much nailed the botnet aspect of this event. Once reports of a zero day worm (or trojan?) that connects to tor began to circulate, it was pretty obvious what it was lol
Then again, no one really knows for sure what the botnet is doing on tor. If they're actually running a CnC server as a hidden service rather than using their own (much larger) network of bots, that's how you know that tor means business. After all, the network did work even during the peak of this "attack" and before any measures were put in place to mitigate the load. Shit, even the developers of tor were surprised it was still working. Even though it doesn't look like this "attack" was meant to disrupt tor, you still gotta hand it to the tor team.
-
I did say a few days ago that it was likely to be a vanilla botnet run by Russian/Eastern European hackers. The geographic location may turn out to be wrong, but there are 100 million PCs in the world that are infected with malware. Knowing nothing else about the surge, that was the most likely explanation based on the baseline probability.
All this talk about Israel and intelligence agencies and deanonymization attacks on hidden services seemed unlikely, given the massive fallout that would result if LE infected millions of computers with malware. Not worth it to find Silk Road. They tried pretty hard to keep Stuxnet off regular users' computers, and the Iranian nuclear program is a much bigger deal than SR.
-
In all fairness, the Israeli Intelligence Agency (Mossad) quite likely doesn't give a flying fuck what you or anybody else thinks about them using a botnet to do whatever the fuck they want. Russian FSB certainly doesn't, they have used botnets to DDoS entire countries.
-
In all fairness, the Israeli Intelligence Agency (Mossad) quite likely doesn't give a flying fuck what you or anybody else thinks about them
Yet they wrote Stuxnet with mechanisms to prevent widespread infection. ;)
-
In all fairness, the Israeli Intelligence Agency (Mossad) quite likely doesn't give a flying fuck what you or anybody else thinks about them
Yet they wrote Stuxnet with mechanisms to prevent widespread infection. ;)
Well Stuxnet did cause widespread infection, it just cleaned itself off infected computers after it determined they were not the targets. Also I think Stuxnet is probably more from the NSA/CIA than Mossad.
-
Forgive my ignorance on the subject at hand, but could controlling this high a percentage of the overall clients on the tor network be leveraged in some way to deanonymize a hidden service/tor user?
-
Forgive my ignorance on the subject at hand, but could controlling this high a percentage of the overall clients on the tor network be leveraged in some way to deanonymize a hidden service/tor user?
I don't know if creating large numbers of connections can exploit a bug in some applications to deanonymize a hidden service, since there are so many applications that can listen for network connections, but at the network layer, no. You need to control or find the entry guard. An attacker running many relays would be far more dangerous with respect to deanonymization by attacks on the Tor network. Many clients just clog it up.
-
selectively clogging up parts of the network can force clients toward nodes you control, but largely yeah it is better to have relays than clients if you try to attack the network. Unless you have some external position to view network links you cannot even deanonymize anybody regardless of how many clients you have.
-
selectively clogging up parts of the network can force clients toward nodes you control
Which still requires nodes you control. Clients alone can't really deanonymize you.
-
Thank you astor, kmfkewm. One further question: if we were to assume an adversary has control over some number of entry guards (which the government likely does), and has control over a similarly sized botnet as to what has been observed on the tor network currently (which the government likely could), would there be any advantage to using a botnet in conjunction with the entry guard exploit over just the entry guard exploit alone?
-
Thank you astor, kmfkewm. One further question: if we were to assume an adversary has control over some number of entry guards (which the government likely does), and has control over a similarly sized botnet as to what has been observed on the tor network currently (which the government likely could), would there be any advantage to using a botnet in conjunction with the entry guard exploit over just the entry guard exploit alone?
An attacker with this many nodes could do substantial damage to Tor. They could quickly trace hidden service to its entry guards (especially since they could add several relays without being detected, and they don't need entry flag to detect entry guards), the biggest bottleneck would be the number of relays they have on the network as well as how many circuit requests the hidden service can manage before it locks up or the Tor infrastructure it uses locks up. After tracing to the entry guards, the attacker could DDoS them easily with this many nodes (this attacker could easily DDoS the entire Tor network several times over). This forces the hidden service to pick new entry nodes (unless strict entry guards are set in torrc, in which case it will just be unreachable). The attacker can then trace to the new entry guards and once again DDoS them. They can continue to do this until the hidden service selects one of their entry guards, at which point they have deanonymized it.
The attacker could also make a large number of the botnet nodes relays, although none of them would immediately get the entry flag, and if the attacker added too many at once they would all be black listed. Not sure how closely the Tor devs are monitoring the directory authority servers but my guess is they are hesitant to let new nodes be added right now. So it is possible this attacker could become the overwhelming majority of Tor middle and exit nodes, but they wouldn't become the majority of entry nodes, all the nodes they add would be banned almost immediately, and there are probably automatic systems in place to prevent them from actually adding many nodes at all, all at once anyway.
The attacker could also carry out congestion attacks, how effective this would be would depend on how many entry guards they own. But this attacker could easily make it so that it is much more probable that their entry guard or exit node is selected by a user of Tor, simply by overloading some percentage of the nodes that they do not own.
So this attacker is what would be considered pretty strong. If they wanted to they could DDoS the entire Tor network indefinitely. They could also try to brute force their way to hidden services and they could probably deanonymize most of the ones that don't have strict entry guards set, although how much of the Tor network they would need to bring down first would vary, they wouldn't have any trouble bringing it all down if they needed to though. This attacker could also trivially censor any hidden service simply by brute force and becoming every HSDIR node. They could also carry out the inverse trawling for hidden service attack after doing this, but again this would require them to actually get some of their botnet nodes on the network as Tor relays and particularly entry guards. They could also flash flood middle and exit nodes, but would be detected doing so and prevented from it, either automatically after triggering some limit at the directory authority servers or shortly after doing so by the owners of the DA servers. They could slowly add nodes and get a large number of entry guards over time, but it would be a pretty slow process for them to do so without getting all of them blacklisted. On the other hand they could also prevent new people from adding nodes by flash flooding 1k or so nodes per day, which would result in all being blacklisted in addition to any legitimate new nodes added. They could flood 1k new nodes per day for over eight years with a botnet this size, which would effectively make it so that either Tor cannot let arbitrary volunteers add nodes anymore or they need to let the owner of this Botnet gain a massive internal presence in the network. Also they could increase the probability that clients use any entry guards or exit nodes that they do own, by congesting the ones they don't own.
So yeah pretty powerful attacker who could at best take Tor completely down or make it much harder for the network to grow (probably requiring node operators to be individually authenticated as real people in the future), and at worst could trace hidden services (probably now), eventually own enough of the network to carry out large scale deanonymizing attacks (probably not yet but over time is possible), and use congestion attacks to make it so they actually need to own fewer routing nodes to deanonymize people (they can use their non-relay Botnet nodes to increase the probability that their malicious Tor relays are used by targets).
I cannot really say this attacker is internal since we don't know how many if any relay nodes they operate, but they definitely have a big enough Botnet that they could potentially be (in that they have the potential to be) one of the most powerful internal attackers in the world. Of course a powerful external attacker could be even more dangerous though.
-
Thanks, that's exactly the kind of response I was looking for. I guess the remaining question is, has this botnet shown any malicious intent other than the strain it's existence has put on the tor network?
edit: After thinking on it, one benefit of this botnet is that it has greatly increased the noise to signal ratio anyone tracking legitimate tor users would need to sift through. A silkroad user in a podunk town had a high likelihood of being the one of the few in his town using tor (if not the only one), with an indiscriminate piece of malware running around which creates tor traffic is like creating a jungle for us e-druggies to hide in.
-
Thank you astor, kmfkewm. One further question: if we were to assume an adversary has control over some number of entry guards (which the government likely does), and has control over a similarly sized botnet as to what has been observed on the tor network currently (which the government likely could), would there be any advantage to using a botnet in conjunction with the entry guard exploit over just the entry guard exploit alone?
An attacker with this many nodes could do substantial damage to Tor. They could quickly trace hidden service to its entry guards (especially since they could add several relays without being detected, and they don't need entry flag to detect entry guards), the biggest bottleneck would be the number of relays they have on the network as well as how many circuit requests the hidden service can manage before it locks up or the Tor infrastructure it uses locks up. After tracing to the entry guards, the attacker could DDoS them easily with this many nodes (this attacker could easily DDoS the entire Tor network several times over). This forces the hidden service to pick new entry nodes (unless strict entry guards are set in torrc, in which case it will just be unreachable). The attacker can then trace to the new entry guards and once again DDoS them. They can continue to do this until the hidden service selects one of their entry guards, at which point they have deanonymized it.
The attacker could also make a large number of the botnet nodes relays, although none of them would immediately get the entry flag, and if the attacker added too many at once they would all be black listed. Not sure how closely the Tor devs are monitoring the directory authority servers but my guess is they are hesitant to let new nodes be added right now. So it is possible this attacker could become the overwhelming majority of Tor middle and exit nodes, but they wouldn't become the majority of entry nodes, all the nodes they add would be banned almost immediately, and there are probably automatic systems in place to prevent them from actually adding many nodes at all, all at once anyway.
The attacker could also carry out congestion attacks, how effective this would be would depend on how many entry guards they own. But this attacker could easily make it so that it is much more probable that their entry guard or exit node is selected by a user of Tor, simply by overloading some percentage of the nodes that they do not own.
So this attacker is what would be considered pretty strong. If they wanted to they could DDoS the entire Tor network indefinitely. They could also try to brute force their way to hidden services and they could probably deanonymize most of the ones that don't have strict entry guards set, although how much of the Tor network they would need to bring down first would vary, they wouldn't have any trouble bringing it all down if they needed to though. This attacker could also trivially censor any hidden service simply by brute force and becoming every HSDIR node. They could also carry out the inverse trawling for hidden service attack after doing this, but again this would require them to actually get some of their botnet nodes on the network as Tor relays and particularly entry guards. They could also flash flood middle and exit nodes, but would be detected doing so and prevented from it, either automatically after triggering some limit at the directory authority servers or shortly after doing so by the owners of the DA servers. They could slowly add nodes and get a large number of entry guards over time, but it would be a pretty slow process for them to do so without getting all of them blacklisted. On the other hand they could also prevent new people from adding nodes by flash flooding 1k or so nodes per day, which would result in all being blacklisted in addition to any legitimate new nodes added. They could flood 1k new nodes per day for over eight years with a botnet this size, which would effectively make it so that either Tor cannot let arbitrary volunteers add nodes anymore or they need to let the owner of this Botnet gain a massive internal presence in the network. Also they could increase the probability that clients use any entry guards or exit nodes that they do own, by congesting the ones they don't own.
So yeah pretty powerful attacker who could at best take Tor completely down or make it much harder for the network to grow (probably requiring node operators to be individually authenticated as real people in the future), and at worst could trace hidden services (probably now), eventually own enough of the network to carry out large scale deanonymizing attacks (probably not yet but over time is possible), and use congestion attacks to make it so they actually need to own fewer routing nodes to deanonymize people (they can use their non-relay Botnet nodes to increase the probability that their malicious Tor relays are used by targets).
I cannot really say this attacker is internal since we don't know how many if any relay nodes they operate, but they definitely have a big enough Botnet that they could potentially be (in that they have the potential to be) one of the most powerful internal attackers in the world. Of course a powerful external attacker could be even more dangerous though.
Wow very in depth explanation. The timing of this botnet is quite weird as well considering the recent research paper "Users Get Routed: Traffic Correlation on Tor by Realistic Adversaries". I gave that a read and from what I understand and this is not my area of expertise but with this botnet out on the loose doesn't that mean that more streams will be open? Lets assume it's not the NSA or any intelligence agency for a second. With more streams wouldn't it be easier to hide in with all the fake traffic activity? I know this was talked about a couple of years ago by the Tor developers in regards to generating fake traffic but it was dismissed as that technique didn't warrant the enormous hit that the network would take. Since the idea was first brought up I think the Tor network has matured and the fact that it is still up even with this straining amount of traffic is a testament to that. But then again having your traffic routed through a path your adversary determines is far from ideal and I think at this point we really don't know the motive or goal of this attack.
Also I know a lot of ppl have speculated that if SR where ever to be taken down then there would be two options:
1) Attack the Tor network infrastructure
2) Attack bitcoins
It seems to me that #2 has been already implemented or tried to be implemented as we all know it's still fairly easy to get bitcoins. I guess with all that being said I wonder what others opinions are in regards to the motive and/or identity of the attacker/attackers...