66
« on: February 15, 2014, 11:06:44 pm »
I am deeply thankful for the hundreds of supportive messages I've received. I am consistently blown away by how the core of this community come together in times of strife.. To the multitude of you who lost faith in Silk Road, I still am glad I've heard your words.
I am still devastated that I have failed you. And even I have fallen victim to this attack – the majority of my personal earnings are completely gone. My life is not in danger though, and my identity is still protected – I’m thankful for this.
So now, it’s time to share with you our plan to move forward.
I realize you have no way of knowing whether I'm telling the truth, on anything. Trying to communicate the authenticity of my true character on the darknet is difficult to say the least, and I do not expect any of you to take me at my word. But my actions over the next few weeks will speak louder than words ever could, and if you stick around you’ll realize this.
Silk Road is Not Dead
We are here to stay, and for the most part, I believe our values are the same as yours – the community’s. And even though I represent the latest idiot to make a security mistake, I am unlike those who simply give in, and am determined to use lessons learned to fight for this community's long term security.
I'm focusing on multi-signature transactions as our only safe trading mechanism long term. Additionally, I am ready and willing to collaborate with other darknet administrators and utilize our talented penetration testing team to help them to regularly audit their security.
And although the tinfoil club will probably hate this and continue to tear me to shreds without regard for my actions in December, and whether you choose to believe them or not, here are the facts:
1. I live a very busy and complicated life. I take OPSEC to extreme levels which consume many of my hours per day. Any time you see me here, it means I have completed a checklist which requires over an hour of preparation for me to reposition and reconfigure many things to ensure I leave the most minimal trail possible and can "safely" connect to this place. This job will never be safe, and it is impossible to be perfectly covert. I know what I am facing, both from LE and from vendors who do not share my opinions of non-violence.
2. All released announcements were my true plans, and all delays were due to unanticipated external factors upon which I cannot elaborate. My staff has been very frustrated at my lack of ability to keep the dev team on time. Again, this is completely my fault and something for which I take complete responsibility. I do not trust any developers, so I meticulously review anything the dev team sends to me. Many have criticized me for this, and I listened to them. Had I reviewed the code EVEN MORE meticulously and delayed the fix to speed up slow withdrawals last month, transaction malleability would not have affected us at all. This is completely my fault for listening to critics instead of my own sense of caution.
3. The public has known that auto-finalize has been disabled for a long time, and has known that the escrow balance is massive.
4. Any smart person could deduce that re-enabling of auto-finalize requires me to move a vast sum from cold storage to the hot storage wallet to ensure adequate fund availability (which admittedly was too large and one of my biggest failures this month). The attack was surgical and timed perfectly.
The reason such a high quantity of coins was stolen was because we were preparing to implement auto-finalization and had therefore moved a huge volume from the cold storage to the hot storage to ensure the coins from all auto-finalized orders could be withdrawn without a hitch.
Here are as many details on the attack as I can provide. I am still investigating, and do not want to leak anything which could compromise any other markets. Transparency in this issue is key, however, you deserve as granular an explanation as possible as to why this happened.
SR2 Attack Investigation Results as of Feb 15
1. When withdrawal delays were happening last month, we wrote a custom bitcoin implementation using several available libraries a reference, and built it on a shared infrastructure to load balance across accounts and provide risk mitigation if ever one server was compromised.
2. As this implementation was being written by behind-the-scenes dev staff, a lynch mob was getting exponentially worse in the forums. Our moderator staff became more and more insistent that it was crucial to fix this problem immediately, relaying to me many concerns, and threats, from the community.
3. We cut a corner, deciding to watch transactions. Many community members were reporting missing withdrawals, weeks after sending them. We implemented a liberal check to the "Refresh Deposit Balances" link on the account page - in essence, we allowed it to search deeply for any missing coins, rescanning and rescanning. One of our team members was familiar with MtGox's procedures and source code and vouched for the idea. Our thoughts were that if the largest bitcoin exchange was using this approach to find missing coins, it must be commonplace. (I received her/his permission to publish this.)
4. The transaction malleability bug was published last week at a time when I was disconnected from the internet for an extended period for OPSEC reasons. I did not become aware of the threat until the attack was already occurring. This was a failure on my part to trust more staff with critical access to stop withdrawals. The people who did have access to disable withdrawals were terrified of doing it without my permission, due to how negatively the community reacts when withdrawals are turned off. They knew if they disabled withdrawals, then re-enabled them, there would be a run on the bank and I would need to be online to deposit cold storage into hot storage. In effect, this was my failure to emphasize to staff that it is ALWAYS okay to lean on the side of caution and hit killswitches as they deem necessary. It was also my failure to communicate expectations to the community that sometimes killswitches will have to be hit, and it just means we're playing it safe. This month it is clear we did not play it safe. Killswitches are pointless if the only person confident enough to hit them is the boss.
5. The attacker used an account first registered in early Jan to try exploiting the vulnerability. Looking back at the account's history indicates that this account has been used to pentest many aspects of our system unsuccessfully. Every previous attempt to find an attack vector failed, due to hardening done by our penetration testing team. I had previously seen these attempts in the logs as something to celebrate: "Look, our hardening is working, this hacker would have won, but we put security over features!" - Several people suggested disabling the hacker's accounts, but that is pointless in our business as anyone can simply re-register.
6. We are still unclear as to the exact approach used to cause our servers' perception of transactions to mismatch with reality, because we have largely been focusing on preparing for relaunch and building communication to share with you, the community, over the past day. To the best of our understanding, the attacker deposited increasingly larger amounts of bitcoins, and withdrew them while flooding the blockchain with DDoS of similar transactions.
7. Our system reflected his withdrawals, and his account showed as zero.
8. The attacker rapidly clicked "Refresh deposits" link until our code assumed that the transaction failed (all indications point to malleability as the root case). Our system re-credited his account with the X BTC it judged "unsuccessfully" withdrawn. This, again, due to the liberal rescanning mechanisms added last month (see point 3).
9. In reality, our servers perceived the transactions inaccurately and the BTC were successfully withdrawn. As a result our system inaccurately represented the user's balance as larger than it truly was. This allowed the attacker to withdraw coins again.
10. The attacker began depositing larger amounts, such as 50 BTC at a time, in order to exploit the vulnerability quickly.
11. They then drained our entire hot storage in one bitcoin server shard, and tried the attack from one of their other accounts. The other account was on a different shard, and allowed that hot storage to be drained.
12. The attacker continued using different accounts until virtually all of our hot storage shards were drained.
Due to the type of vulnerability, our accounting system had an inaccurate perception of bitcoins on hand and therefore did not trigger any alerts until real user withdrawals began failing unexpectedly. This was a failure on our part to build a new server alert to check each shard of the new shared bitcoin infrastructure, and another symptom of how rushed we were when implementing the new system during a forum lynch-mob about slow withdrawals.
Lessons learned so far:
1. More staff should have access to killswitches and should not be afraid to use them. Even if it's just because of some news post about on clearnet about some "Transaction Malleability" thing. Killswitches should be used liberally and regularly.
2. A community needs to be comfortable with regular downtime - especially during periods when community safety and the safety of its money could be threatened. A good staff will be responsibly paranoid and hitting killswitches as they deem necessary, and should not be afraid of their community panicking each time precaution is exercised.
3. There’s more to web security than rigorous penetration testing. It also requires the constant review of external services, systems and infrastructures.
4. Never make assumptions when implementing protocols you did not design. Had we not trusted MtGox's staff procedures, we would have reviewed our approaches with much more rigor. Just because a large company's procedures are successful does not make them are safe.
5. Accept that your community will hate you regardless. In the darknet, you will be crucified by your community if anything ever goes wrong. Psychologically prepare for everyone to hate you, because you inevitably will fail at something, and you will need to stay determined no matter what. Don't let it get to your head, and don't let it affect your rigor with security. We cut corners with our deposit/withdrawal speed optimizations because we were afraid of the lynch-mob. We were foolish in this regard. We should have taken the entire site offline for as long as it took to implement a high-volume fault-tolerant bitcoin server in a well-tested, rigorous manner. This cost us an unspeakable amount of trust that we had slaved to build, some of us for years.
This is where we're at, with as much transparency as this medium allows:
The worst that could happen did not happen.
The worst thing that could happen for our community's finances, however, did.
We hate that it occurred under our watch.
There is nothing we can do to change this. All we can do is be transparent about where we are at, and move forward.
To those who will respond here for dramatic effect, please consider contributing your energy to the reconstruction of our community. If you cannot bring yourself to do this, then please take your negativity elsewhere. This is simple. If you have lost faith in Silk Road, I don't blame you in the least. I also do not want to read about it in perpetuity.
Trolling is easy. Trolling during a disaster is cowardice. Choosing to strengthen rather than troll, when trolling is the popular move - that's noble. I know that I speak for all Silk Road staff, seen and unseen, when I speak these words: We are going nowhere. We will remain, rebuild, and move forward toward the greater goal for which we strive by our running of this marketplace: Personal freedom and foundational human rights.
Silk Road will always rise again.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
I am deeply thankful for the hundreds of supportive messages I've received. I am consistently blown away by how the core of this community come together in times of strife.. To the multitude of you who lost faith in Silk Road, I still am glad I've heard your words.
I am still devastated that I have failed you. And even I have fallen victim to this attack – the majority of my personal earnings are completely gone. My life is not in danger though, and my identity is still protected – I’m thankful for this.
So now, it’s time to share with you our plan to move forward.
I realize you have no way of knowing whether I'm telling the truth, on anything. Trying to communicate the authenticity of my true character on the darknet is difficult to say the least, and I do not expect any of you to take me at my word. But my actions over the next few weeks will speak louder than words ever could, and if you stick around you’ll realize this.
Silk Road is Not Dead
We are here to stay, and for the most part, I believe our values are the same as yours – the community’s. And even though I represent the latest idiot to make a security mistake, I am unlike those who simply give in, and am determined to use lessons learned to fight for this community's long term security.
I'm focusing on multi-signature transactions as our only safe trading mechanism long term. Additionally, I am ready and willing to collaborate with other darknet administrators and utilize our talented penetration testing team to help them to regularly audit their security.
And although the tinfoil club will probably hate this and continue to tear me to shreds without regard for my actions in December, and whether you choose to believe them or not, here are the facts:
1. I live a very busy and complicated life. I take OPSEC to extreme levels which consume many of my hours per day. Any time you see me here, it means I have completed a checklist which requires over an hour of preparation for me to reposition and reconfigure many things to ensure I leave the most minimal trail possible and can "safely" connect to this place. This job will never be safe, and it is impossible to be perfectly covert. I know what I am facing, both from LE and from vendors who do not share my opinions of non-violence.
2. All released announcements were my true plans, and all delays were due to unanticipated external factors upon which I cannot elaborate. My staff has been very frustrated at my lack of ability to keep the dev team on time. Again, this is completely my fault and something for which I take complete responsibility. I do not trust any developers, so I meticulously review anything the dev team sends to me. Many have criticized me for this, and I listened to them. Had I reviewed the code EVEN MORE meticulously and delayed the fix to speed up slow withdrawals last month, transaction malleability would not have affected us at all. This is completely my fault for listening to critics instead of my own sense of caution.
3. The public has known that auto-finalize has been disabled for a long time, and has known that the escrow balance is massive.
4. Any smart person could deduce that re-enabling of auto-finalize requires me to move a vast sum from cold storage to the hot storage wallet to ensure adequate fund availability (which admittedly was too large and one of my biggest failures this month). The attack was surgical and timed perfectly.
The reason such a high quantity of coins was stolen was because we were preparing to implement auto-finalization and had therefore moved a huge volume from the cold storage to the hot storage to ensure the coins from all auto-finalized orders could be withdrawn without a hitch.
Here are as many details on the attack as I can provide. I am still investigating, and do not want to leak anything which could compromise any other markets. Transparency in this issue is key, however, you deserve as granular an explanation as possible as to why this happened.
SR2 Attack Investigation Results as of Feb 15
1. When withdrawal delays were happening last month, we wrote a custom bitcoin implementation using several available libraries a reference, and built it on a shared infrastructure to load balance across accounts and provide risk mitigation if ever one server was compromised.
2. As this implementation was being written by behind-the-scenes dev staff, a lynch mob was getting exponentially worse in the forums. Our moderator staff became more and more insistent that it was crucial to fix this problem immediately, relaying to me many concerns, and threats, from the community.
3. We cut a corner, deciding to watch transactions. Many community members were reporting missing withdrawals, weeks after sending them. We implemented a liberal check to the "Refresh Deposit Balances" link on the account page - in essence, we allowed it to search deeply for any missing coins, rescanning and rescanning. One of our team members was familiar with MtGox's procedures and source code and vouched for the idea. Our thoughts were that if the largest bitcoin exchange was using this approach to find missing coins, it must be commonplace. (I received her/his permission to publish this.)
4. The transaction malleability bug was published last week at a time when I was disconnected from the internet for an extended period for OPSEC reasons. I did not become aware of the threat until the attack was already occurring. This was a failure on my part to trust more staff with critical access to stop withdrawals. The people who did have access to disable withdrawals were terrified of doing it without my permission, due to how negatively the community reacts when withdrawals are turned off. They knew if they disabled withdrawals, then re-enabled them, there would be a run on the bank and I would need to be online to deposit cold storage into hot storage. In effect, this was my failure to emphasize to staff that it is ALWAYS okay to lean on the side of caution and hit killswitches as they deem necessary. It was also my failure to communicate expectations to the community that sometimes killswitches will have to be hit, and it just means we're playing it safe. This month it is clear we did not play it safe. Killswitches are pointless if the only person confident enough to hit them is the boss.
5. The attacker used an account first registered in early Jan to try exploiting the vulnerability. Looking back at the account's history indicates that this account has been used to pentest many aspects of our system unsuccessfully. Every previous attempt to find an attack vector failed, due to hardening done by our penetration testing team. I had previously seen these attempts in the logs as something to celebrate: "Look, our hardening is working, this hacker would have won, but we put security over features!" - Several people suggested disabling the hacker's accounts, but that is pointless in our business as anyone can simply re-register.
6. We are still unclear as to the exact approach used to cause our servers' perception of transactions to mismatch with reality, because we have largely been focusing on preparing for relaunch and building communication to share with you, the community, over the past day. To the best of our understanding, the attacker deposited increasingly larger amounts of bitcoins, and withdrew them while flooding the blockchain with DDoS of similar transactions.
7. Our system reflected his withdrawals, and his account showed as zero.
8. The attacker rapidly clicked "Refresh deposits" link until our code assumed that the transaction failed (all indications point to malleability as the root case). Our system re-credited his account with the X BTC it judged "unsuccessfully" withdrawn. This, again, due to the liberal rescanning mechanisms added last month (see point 3).
9. In reality, our servers perceived the transactions inaccurately and the BTC were successfully withdrawn. As a result our system inaccurately represented the user's balance as larger than it truly was. This allowed the attacker to withdraw coins again.
10. The attacker began depositing larger amounts, such as 50 BTC at a time, in order to exploit the vulnerability quickly.
11. They then drained our entire hot storage in one bitcoin server shard, and tried the attack from one of their other accounts. The other account was on a different shard, and allowed that hot storage to be drained.
12. The attacker continued using different accounts until virtually all of our hot storage shards were drained.
Due to the type of vulnerability, our accounting system had an inaccurate perception of bitcoins on hand and therefore did not trigger any alerts until real user withdrawals began failing unexpectedly. This was a failure on our part to build a new server alert to check each shard of the new shared bitcoin infrastructure, and another symptom of how rushed we were when implementing the new system during a forum lynch-mob about slow withdrawals.
Lessons learned so far:
1. More staff should have access to killswitches and should not be afraid to use them. Even if it's just because of some news post about on clearnet about some "Transaction Malleability" thing. Killswitches should be used liberally and regularly.
2. A community needs to be comfortable with regular downtime - especially during periods when community safety and the safety of its money could be threatened. A good staff will be responsibly paranoid and hitting killswitches as they deem necessary, and should not be afraid of their community panicking each time precaution is exercised.
3. There’s more to web security than rigorous penetration testing. It also requires the constant review of external services, systems and infrastructures.
4. Never make assumptions when implementing protocols you did not design. Had we not trusted MtGox's staff procedures, we would have reviewed our approaches with much more rigor. Just because a large company's procedures are successful does not make them are safe.
5. Accept that your community will hate you regardless. In the darknet, you will be crucified by your community if anything ever goes wrong. Psychologically prepare for everyone to hate you, because you inevitably will fail at something, and you will need to stay determined no matter what. Don't let it get to your head, and don't let it affect your rigor with security. We cut corners with our deposit/withdrawal speed optimizations because we were afraid of the lynch-mob. We were foolish in this regard. We should have taken the entire site offline for as long as it took to implement a high-volume fault-tolerant bitcoin server in a well-tested, rigorous manner. This cost us an unspeakable amount of trust that we had slaved to build, some of us for years.
This is where we're at, with as much transparency as this medium allows:
The worst that could happen did not happen.
The worst thing that could happen for our community's finances, however, did.
We hate that it occurred under our watch.
There is nothing we can do to change this. All we can do is be transparent about where we are at, and move forward.
To those who will respond here for dramatic effect, please consider contributing your energy to the reconstruction of our community. If you cannot bring yourself to do this, then please take your negativity elsewhere. This is simple. If you have lost faith in Silk Road, I don't blame you in the least. I also do not want to read about it in perpetuity.
Trolling is easy. Trolling during a disaster is cowardice. Choosing to strengthen rather than troll, when trolling is the popular move - that's noble. I know that I speak for all Silk Road staff, seen and unseen, when I speak these words: We are going nowhere. We will remain, rebuild, and move forward toward the greater goal for which we strive by our running of this marketplace: Personal freedom and foundational human rights.
Silk Road will always rise again.
-----BEGIN PGP SIGNATURE-----
iQIcBAEBCgAGBQJS//M9AAoJECRwq0imSkC0rEgP/iH0+H6k3WkgYO69WBy/PQkv
SRYa18N1dXctQeo6x3aS6KlWS9ncdGXt/HK0l0M3+R82UmowSluY0K8EU1I3odIo
df9sNbZcDW4S1lahqCoObJ8ITYJKVvQVOREm7AqaKixA+atYoiJ42yAsl+zS4Hd7
uw2uxxKm6LfDQ8CYOTrtEp3Lhp0Y/GTgM/7gIXvQy9HYEJTG5f3aYpnVSbSC4yjR
8Y9lsznn0NWoXMR9i7SAP2WxCEdnZni/m7uqmVLu41JNDdbjBeRCAnKFYmZTEKXi
/0Oesl97Ni7/qiAQehZeo/qbQxgRLp+2Voa16LXBVC9UttmpzyRaMGK8raHOx//c
LHRWqRKmSgOHLCcDEsWuDbJD9Nrfq6Ih7Sn1rYe6gwVE2hwV6xHcmr5LVBHzaKwE
U4tYxH/TmcfoSBUP8OFtz5GfEz/XHZv50t388CRWBEDqtpPE+g9vIxbYBBFrfr0P
aDKkZ5MeVsjs3wO146Adv27tK5i8i5J1Lb8SGPh0y8B9jxHFail0AbuVhppwzjho
m1eN8hjx7xnROIBZejhCFWSADE6vfZ6ttmATmGrjpuRmV93Ce25aj1t+VsXa2jim
ff7+PYn/dM0tBLW+mxCklvw10jjfKDexNBIGQ7lTPu75WvUCUFtQVqzTAY7nL2Mt
p2gJK+Oftq1t+0o/esa7
=G++m
-----END PGP SIGNATURE-----
Show Posts