Show Posts
316
Silk Road discussion / Down-time: summary and reflections
« on: November 17, 2012, 08:10 am »
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hey gang,
What a week! How do I even begin trying to summarize all of this? A couple of weeks ago, we started seeing the accessibility and speed of Silk Road start to drop, especially around peak hours. Monitoring the number of incoming connections, pageviews and registrations showed record breaking numbers, so the obvious conclusion was that we were getting an influx of new members that was overloading our current infrastructure. We immediately went to work identifying and overcoming the bottlenecks in the system. However, as each was overcome, demand for resources would fill in and run up against the next bottleneck. In the past week we've literally been through every layer of our entire system from hardware, to software, to application making it more efficient and streamlined. We did not rule out the possibility that the increased usage was due to a hacking attempt (known as denial of service, or DoS) and took this time to implement several security measures that had been in the works anyway.
Then the site went completely off-line. There are a few things that I don't trust to anyone else, and that's the wallets, and rebooting the system. I was away from my computer when the site went down, and came back a couple of days later to utter chaos on the forums. I've pretty much been in communication since, but basically we did a full redeploy of the entire system including the new security and performance measures which turned out to be quite challenging in the end. We ran into a few stumbling blocks, but were able to get past them. I have to take a moment here to thank my team, both the customer support staff and the technical team. These guys were right there with me the whole time and I don't think I could've gotten through it without them, certainly not as quickly.
So, now we are back. What have I learned from all of this? First off, it helped me realize just how much the site means to me and all of you. These forums were like a battleground of opinion, with everyone expressing their concern. By far the worst part of this whole thing was putting everyone through that uncertainty. I knew nothing I could say would make any difference until you had access to your funds and were able to use the site again. So mostly I just focused on getting us back up as quickly as I could without cutting any corners or taking any chances. I also learned a ton about the limitations of the technology we rely on and the direction we need to go to keep Silk Road secure and scalable.
Thank you to everyone who encouraged and supported me and the team this week. Your kind words were our fuel. To anyone who doubted us, or suspected us of betrayal, again, I am so sorry to put you through that uncertainty. It is not something I want to repeat ever again and you can rest assured I'll do everything I can to prevent and limit any future down time. I know this whole market is based on the trust you put in me and I don't take that lightly. It's an honor to serve you and though you don't know who I am, and have no recourse if I were to betray you, I hope that as time goes on I will have more opportunities to demonstrate that my intentions are genuine and no amount of money could buy my integrity.
Especially you old hats that have been around since the beginning, but this goes for everyone, you all are like family to me. Sure we have some crazy cousins floating around, but they just add character, right? Doesn't matter though, I love you all. Of all the people in the world, you are the ones who are here, in the early stages of this revolution. You are the ones getting this thing off the ground and driving it forward. It is a privilege to have you by my side. Thank you for your trust, faith, camaraderie and love.
Sincerely,
Dread Pirate Roberts
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)
iQEcBAEBAgAGBQJQp8UyAAoJEAIiQjtnt/olyusIALRRhZM49sth+5wudmHXeN4Z
HrhuUzIUQdNVYvK51moKFugB1oH2Fv+X7sp+2FRbUy1Ra5PqtvQBf9qo9xFsdlUj
QplvN20eXZ3+bRcjUqW8wKSDxO+BB+Q0twkE04uQj4igMgyWZjzdM5IFTxnIWmkg
c/HYH8hkS3BB2objHogiwQJhvX8F0A4ei1+K8id89owfv8RucP37aQxVPmpFEAD7
vnbE9KH5hHfw8gOeRKU5Vch4NfF0s6Zr2yFOBgx8tHsmARlsOjktXWVh0VfRCn1v
PFhQeypgblSPOmSSROCIfBxwei6sniC/kXi7/lJK5A7fqqjfge2HqBk8bBuflM8=
=mi48
-----END PGP SIGNATURE-----
Hash: SHA1
Hey gang,
What a week! How do I even begin trying to summarize all of this? A couple of weeks ago, we started seeing the accessibility and speed of Silk Road start to drop, especially around peak hours. Monitoring the number of incoming connections, pageviews and registrations showed record breaking numbers, so the obvious conclusion was that we were getting an influx of new members that was overloading our current infrastructure. We immediately went to work identifying and overcoming the bottlenecks in the system. However, as each was overcome, demand for resources would fill in and run up against the next bottleneck. In the past week we've literally been through every layer of our entire system from hardware, to software, to application making it more efficient and streamlined. We did not rule out the possibility that the increased usage was due to a hacking attempt (known as denial of service, or DoS) and took this time to implement several security measures that had been in the works anyway.
Then the site went completely off-line. There are a few things that I don't trust to anyone else, and that's the wallets, and rebooting the system. I was away from my computer when the site went down, and came back a couple of days later to utter chaos on the forums. I've pretty much been in communication since, but basically we did a full redeploy of the entire system including the new security and performance measures which turned out to be quite challenging in the end. We ran into a few stumbling blocks, but were able to get past them. I have to take a moment here to thank my team, both the customer support staff and the technical team. These guys were right there with me the whole time and I don't think I could've gotten through it without them, certainly not as quickly.
So, now we are back. What have I learned from all of this? First off, it helped me realize just how much the site means to me and all of you. These forums were like a battleground of opinion, with everyone expressing their concern. By far the worst part of this whole thing was putting everyone through that uncertainty. I knew nothing I could say would make any difference until you had access to your funds and were able to use the site again. So mostly I just focused on getting us back up as quickly as I could without cutting any corners or taking any chances. I also learned a ton about the limitations of the technology we rely on and the direction we need to go to keep Silk Road secure and scalable.
Thank you to everyone who encouraged and supported me and the team this week. Your kind words were our fuel. To anyone who doubted us, or suspected us of betrayal, again, I am so sorry to put you through that uncertainty. It is not something I want to repeat ever again and you can rest assured I'll do everything I can to prevent and limit any future down time. I know this whole market is based on the trust you put in me and I don't take that lightly. It's an honor to serve you and though you don't know who I am, and have no recourse if I were to betray you, I hope that as time goes on I will have more opportunities to demonstrate that my intentions are genuine and no amount of money could buy my integrity.
Especially you old hats that have been around since the beginning, but this goes for everyone, you all are like family to me. Sure we have some crazy cousins floating around, but they just add character, right? Doesn't matter though, I love you all. Of all the people in the world, you are the ones who are here, in the early stages of this revolution. You are the ones getting this thing off the ground and driving it forward. It is a privilege to have you by my side. Thank you for your trust, faith, camaraderie and love.
Sincerely,
Dread Pirate Roberts
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)
iQEcBAEBAgAGBQJQp8UyAAoJEAIiQjtnt/olyusIALRRhZM49sth+5wudmHXeN4Z
HrhuUzIUQdNVYvK51moKFugB1oH2Fv+X7sp+2FRbUy1Ra5PqtvQBf9qo9xFsdlUj
QplvN20eXZ3+bRcjUqW8wKSDxO+BB+Q0twkE04uQj4igMgyWZjzdM5IFTxnIWmkg
c/HYH8hkS3BB2objHogiwQJhvX8F0A4ei1+K8id89owfv8RucP37aQxVPmpFEAD7
vnbE9KH5hHfw8gOeRKU5Vch4NfF0s6Zr2yFOBgx8tHsmARlsOjktXWVh0VfRCn1v
PFhQeypgblSPOmSSROCIfBxwei6sniC/kXi7/lJK5A7fqqjfge2HqBk8bBuflM8=
=mi48
-----END PGP SIGNATURE-----