Silk Road forums

Discussion => Silk Road discussion => Topic started by: StExo on May 29, 2013, 09:47 pm

Title: Scraping the Road (Vendor Backup Pages) - 1239 Vendors and counting!
Post by: StExo on May 29, 2013, 09:47 pm
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


Link: http://5uvrgtrgtwkkxsgw.onion
Press the button "Vendor Backups" to arrive to the relevant section


Dear all - I'd like to introduce you all to one of my latest ventures. Over the past few months SilkRoad has had a pretty rough time and as a result, there has been a huge effort by DPR & Co. to keep this place afloat amid a myriad of attacks on all fronts and I'm sure we all applaud them for that heroic effort.

However, the question has been recently - what if? Say SilkRoad goes down tomorrow, do you have all your vendors saved? Unlikely, that's probably what most users will face and the daunting prospect of gathering unreliable information from anonymous sources, scamming would be rife. So, I have went through the monumental effort (those working closely with me know exactly how long this took to get started) of putting together a new hidden service, featuring the saved HTML page of each vendors page.

This is NOT to be used to conduct business outside of SilkRoad, nor for that matter to be used as an active part of the marketplace as it has no official links to SilkRoad other than DPR merely confirming we are permitted to scrape the pages for this purpose. However, in the event SilkRoad goes down for an extended period of time, I hope it is useful for everybody to get in touch with your vendors again, some of whom rely on SilkRoad sales for their entire income stream.


As a word of fair warning and to answer some probable questions, I must lay forth the following:

- - This is not a system designed to be perfectly secure, it is not possible for us to currently do that with our setup right now. We are hosted on Freedom Hosting who are well known by the hidden service community - but this means they can access and alter our content without our knowledge. Whenever a site update occurs, I re-upload the entire site content so any modified content is removed and replaced with the copies I have stored locally. Freedom Hosting for now have a solid reputation, but this does not imply their trust is on par with SilkRoad staff or any trust at all so if you spot something unusual, let me know right away and I'll look into it.

- - The list is only trustworthy to the extent that I myself am trusted by those using it. I do not consider myself superior to any other member of this forum, but I feel I've been around long enough to be a temporary measure until we can come together and form a better solution to this if need be. If you do not trust me to maintain this list - I do not oblige you to, I am merely trying to aid those who do so please do not post criticism of this factor unless it is constructive in some manner in which case I welcome it with open arms.

- - This is the page where I host my own stuff too, feel free to browse but it is not officially connected to the marketplace in any way and so ideas, suggestions and content hosted there is separate from anything on SilkRoad unless otherwise stated. DPR has cleared me to scrape the marketplace pages as I have done, but has not given any kind of official endorsement and is not under his direct control in any manner, although I will respect his wishes on the matter, so do not consider this any branch of the SilkRoad staff.

- - The page uses basic Javascript for aesthetic reasons and unfortunately if I remove it from the content template, the page stops working for some  unknown reason, yet the page will still function with javascript disabled. I encourage you to use NoScript to block such javascript, but this is not any kind of malicious attempt and a quick glance at the source code can reveal it is merely page effects and structure.

- - I will update the backup pages as often as I can, but it is an extensively time consuming effort to do so. Information on the pages may be up to several days old in the event SilkRoad goes down so still err on the side of caution and encrypt all information to them for if SilkRoad is compromised, any vendor not taking proper precautions may also have been compromised.

- - It is the vendors responsibility to inform me if they make a drastic change to their information such as both an e-mail and PGP change. If the marketplace went down shortly after, very few people will notice this error. If you make a big change then just send me a friendly reminder and I'll update it immediately for you, otherwise it may be several days before your specific page is updated.

- - A .zip file is not available to download all of the content from the site just yet until it has been properly audited to contain none of my personal information. This is a temporary measure but I will introduce such an option as soon as possible assuming the security experts of the forum have no objections/thoughts on the matter.

- - YES I will host specific content if you ask me, however each request is individually considered as I will not host any kind of pornographic content (inc. child porn), malware, phishing, scams etc. If the content is server intensive, then make clear what kind of resources the script requires to run.

- - NO I will not send you a Freedom Hosting invite for the simple reason I don't have any left to give away and I already have several people asking for one who are close associates.

- - I am in dire need of some technical assistance on this project as manually sourcing all these pages is an absolute pain and takes extraordinary man hours to collate. If you can offer any help, please just message me and I may throw a few coins your way if it's implemented and working :) .

- - I know, the site is rudimentary, that's being considered and adjusted as soon as possible. We'll get a more fancy look as soon as the core elements are polished off as we require function over fashion.



That's all from me. A notice can be found at the bottom of the page which indicates the last time the site itself was updated and a notice just above the vendor list indicates the last time that whole list was updated (it will not be reset if less than half of all vendor pages are updated at that time).

Suggestions/constructive criticism/ideas are welcomed!

Link: http://5uvrgtrgtwkkxsgw.onion
Press the button "Vendor Backups" to arrive to the relevant section

======================
Credits to astor for helping me out and brain-storming with me on this & Jack N Hoff for doing some pre-release auditing for me.

More features WILL be added to the site as time goes on such as image uploading and possibly more advanced features still in the works. Send me your suggestions!



-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (MingW32)

iQIcBAEBAgAGBQJRpnhYAAoJEO0W4z0WMXVatHwP/0ORzQctNBTvIXkX7yykKy21
gG6/n9VvECfBVZnQolMeu3LbuV7DEhsD/mie24VfYfmpJ4/csuI54LwxYK84rA3m
xb6Ad9DT1ptmSPYwWx9kl3SCTMnyV4ThN/pyjCVDxjKejjwCTzhPqzqia0X676F+
fyOO2kvX2ev6vpsiqG5l86uaboF06jm0cet3CXg1EFdtULCt+nGPm7UXdzFADtxN
Co8si6dAhSXUCYFKcmAFMTylz8ir8UICAsGenAnYRU4q7t7U88TGBJCr5YrbZFUh
NT2FDI8sM/qP+AUClOgVnd0AtYmMW4SWz7hVvoZZoOI0VInIpkLSfMOxTSNUrxki
ssCPf9MwbbH0BFG/kRRafT6Wd5DedFHzV1+uGHlGeat3ZxW0mhEefgYpC62AojFY
vEQKEdalCMU0CJYKV1dJIuxkmLSxKPMqNuZ4/gOUVE6wY2ZxFj6qMRUYVNIC3R+Z
XMF7Eq/+PfiNy+wDWpdZVvz1SED0wWoYkRaUMrrKsGfZBZKxFzYA4BPHVt+aFWXs
nLQa2DOvCKMuYZ+lyymyRgiAfVanciMBNFY/d+9D1lF6XRpiyJM3RRrU3nv5+q3+
G876JGWVvNehYKvPHBuA1K3vJF42s9F06zFP0SaKtEaGh8/vD8XIsORDZTiJZdKD
gnmeQyqWcU0H2kh/2pTriQIcBAEBAgAGBQJRpnhYAAoJEPu9CfR0ll5ptHwP+waU
I/8EsxzDUCfWYQUpjl2Ws+Ctxw865IrxjdPxy3Bi/DTnpMFrsm3SZpHNc+rwWgdw
j0kTdktxnCwNkIxVg+VN+U+lfgXtgepjSjgW59M5pmsWq00p0qIyOfuG38HLB+zn
lAnlzqvqQoXqkIoFSC7qlthJW5O3MfnLxZsxrf+IpXw13X1AKAhJo/rTfDHH4eIg
QP5N9Vt5GRgeQDy3EJC9nd0oBvJVr5TkqhmCAnjTfGBZAOMuL/+ZAwMWuZm9m2PN
XYnnBoiVCCVzkNC1UgU08pDEZyfcoT48+EpnoRNKJDQfGp25XpqW+gNMEMf7rVjj
s74GjxGZnfSRjf1CeTnXRC6+aKaHMKIqKJHVQAOU0GCEoa7s1tMrtsOkgkz/ycEt
jiUazgB/J/oVwUvvr6EkyHCBs+jtdDNV2Vc643jiOqs0Lz2brMQypTJGeI+5gUcR
D9C+SgH/mKujHzgmKF5BSKffqBw/NE60mQ9kmfvZcJGLHqnqPMOWUX6KLunFOTiL
FADgYdW/R9jSKR3igHQ0doQmD0qFtOw2iSZvown9JWA+BU08F7qvyEgcvEAnxXrE
cDODH6ozx4PY5iEJHfJ04eHmI6mnudN4iNjHJeJhZyDFUJjymzP3pgBo0eTfFEHP
W/VIzfXjH54O3HF2ln00Kxhvr366DmRZeUDT0PmA
=A0LJ
-----END PGP SIGNATURE-----


Updates (all will be signed with both of my keys):

http://dkn255hz262ypmii.onion/index.php?topic=165858.msg1187817#msg1187817 (01.06.2013)
Title: Re: Scraping the Road (Vendor Backup Pages)
Post by: astor on May 29, 2013, 10:12 pm
This is a hell of an effort, StExo, nice work. :)

TBH, I knew about this a little while ago and it allowed me to do something that I've been wanting to do for a while, which is to find all vendor keys and put them in a master file.
Title: Re: Scraping the Road (Vendor Backup Pages)
Post by: StExo on May 29, 2013, 10:14 pm
This is a hell of an effort, StExo, nice work. :)

TBH, I knew about this a little while ago and it allowed me to do something that I've been wanting to do for a while, which is to find all vendor keys and put them in a master file.

Many thanks. Have a peek at the mail I just sent and we'll throw it all up there too on both sites.
Title: Re: Scraping the Road (Vendor Backup Pages)
Post by: scout on May 29, 2013, 10:19 pm
Impressive work, StExo!
Title: Re: Scraping the Road (Vendor Backup Pages)
Post by: JackieChan on May 29, 2013, 10:36 pm
Wow that's amazing.

It's kind of chilling to know there are under 1000 vendors. I've never even tried to estimate before.
Title: Re: Scraping the Road (Vendor Backup Pages)
Post by: astor on May 29, 2013, 11:17 pm
1000 vendors with public profiles in the time period that he scraped the site. If he did it again tomorrow, he may get hundreds of different vendors who weren't public/active the first time.

I think it's kind of awesome to know that at any time you can get drugs from a thousand different sources.
Title: Re: Scraping the Road (Vendor Backup Pages)
Post by: StExo on May 30, 2013, 12:19 am
Impressive work, StExo!

Many thanks :) My way of contributing back to the community!


Wow that's amazing.

It's kind of chilling to know there are under 1000 vendors. I've never even tried to estimate before.

I think there are at least 200 more vendors not covered in this as it was a 1 pass scrape so those who had their listings disabled will appear in the next one. At least you appear on it :)
Title: Re: Scraping the Road (Vendor Backup Pages)
Post by: Jack N Hoff on May 30, 2013, 02:07 am
The work that you've done for the community is appreciated brother.

Wow that's amazing.

It's kind of chilling to know there are under 1000 vendors. I've never even tried to estimate before.

I told StExo earlier that I was quite surprised to find out there is such a small number of vendors.  I really thought that there was more.
Title: Re: Scraping the Road (Vendor Backup Pages)
Post by: StExo on May 30, 2013, 02:57 am
I think the number is lower is because of vendors not listing themselves publicly and/or the listings changing. As I had to browse them over several hours undoubtedly the order of the listings would change over that time scale so as I do page 91 for example, a vendor on page 92 may jump to page 90 and I will miss them as I noticed I came across some vendors more than 20 times. Very annoying but I guess over time the list will slowly build and become more complete for all of us with enough community support and people making sure their profiles are correct.

As a side note to all, myself and astor discussed before about vendors having PGP keys listed on their profile and found a worrying number of vendors not listing PGP keys or having invalid/dysfunctional PGP keys. We've notified some of the concerned parties but I imagine if we scanned the keys manually too, we'd find many more notes of concern unfortunately.

Vendors - CHECK, CHANGE, CONFIRM your PGP keys!
Title: Re: Scraping the Road (Vendor Backup Pages)
Post by: HEATFan on May 30, 2013, 04:21 am
This is an excellent contribution to the community! Thank you.

Now, I can think of two people already who are not on that list, one of them is in stealth mode and the other does not have any listings up which is probably why they were missed. What is the best way for us to help contribute towards this? Post here or PM you the names of vendors who are not currently on that list? The ones I have spotted off the top of my head are JackBlue and EastCoastCollective.

Quote
As a side note to all, myself and astor discussed before about vendors having PGP keys listed on their profile and found a worrying number of vendors not listing PGP keys or having invalid/dysfunctional PGP keys. We've notified some of the concerned parties but I imagine if we scanned the keys manually too, we'd find many more notes of concern unfortunately.

That is truly a shame and I while I'm all for minimal regulation I think that using PGP should be a requirement to become a vendor. There is a really low bar set to become a vendor and with that job comes a HUGE responsibility where you are being trusted with a task that could potentially ruin someone's life. Lots of  buyers and unfortunately even some vendors have a very little concern about what they are doing here and that's definitely not a good attitude to have when you are knowingly breaking the law.
Title: Re: Scraping the Road (Vendor Backup Pages)
Post by: StExo on May 30, 2013, 08:30 am
Indeed. From making the list I also got to see some pretty worrying features in some profiles. Some instances are extremely rare such as one vendor openly blackmailing his customers and admitting they keep customer details on record for "dispute purposes" - this vendor I've noticed has been banned since I made the list and his listing removed accordingly.

Another unsettling discovery was how many vendors use weak PGP keys, some using 512-bit keys! I noticed it straight away and they changed their key and was updated accordingly before I published the first publicly available copy of the vendor pages hence the 1 day delay between page date and publishing. But yes I agree HEATFan, a vendor has a duty of care as such to his/her clients and the precautions taken by some are impressive, whilst others are shocking although fortunately this only seems the case with small vendors probably because any large vendor who done something like that would have a mark against their name from security conscious users.
Title: Re: Scraping the Road (Vendor Backup Pages)
Post by: BlackIris on May 30, 2013, 08:57 am
Wow, this is really a monumental effort (and even more it is an ONGOING monumental effort), you are really a staple of this community.

+1.

You have all my sincere gratitude.

P.S.: +1 also for the V.V.V.V.V. sig  ;) 
Title: Re: Scraping the Road (Vendor Backup Pages)
Post by: astor on May 30, 2013, 09:18 am
Another unsettling discovery was how many vendors use weak PGP keys, some using 512-bit keys! I noticed it straight away and they changed their key and was updated accordingly before I published the first publicly available copy of the vendor pages hence the 1 day delay between page date and publishing. But yes I agree HEATFan, a vendor has a duty of care as such to his/her clients and the precautions taken by some are impressive, whilst others are shocking although fortunately this only seems the case with small vendors probably because any large vendor who done something like that would have a mark against their name from security conscious users.

Yes, I noticed this when I was active in PGP Club, before it was moved to the Newbie section. It's roughly the same stats.

But we have a chance now to help everyone improve their keys. :)

Title: Re: Scraping the Road (Vendor Backup Pages)
Post by: StExo on May 30, 2013, 09:33 am
512-bit keys can be cracked right, as in it's not actually beyond technology but has been actually achieved?
Title: Re: Scraping the Road (Vendor Backup Pages)
Post by: caerus on May 30, 2013, 10:19 am
512-bit keys can be cracked right, as in it's not actually beyond technology but has been actually achieved?

The truth is if a vendor is using a 512bit key they are most likely doing a lot of other dumb shit. No need to involve the NSA to bust them  ;)

Thanks a lot for taking time for this project.

+1
Title: Re: Scraping the Road (Vendor Backup Pages)
Post by: Bungee54 on May 30, 2013, 11:04 am
Subbed & Liked very much !  8)


Superb Job!
Title: Re: Scraping the Road (Vendor Backup Pages)
Post by: Dingo Ate My Drugs on May 30, 2013, 04:54 pm
Hey,

I am not on that list.
My vendor account has had listings removed for the last 12 days, but I wasn't in vacation mode. Just taking some days off from orders.
I would like to be added to the list.

And thanks for the effort, and +1.

Dingo :-)
Title: Re: Scraping the Road (Vendor Backup Pages)
Post by: Dingo Ate My Drugs on May 30, 2013, 05:00 pm
I'm not criticizing at all, but there is a lot of vendors missing.
Some big vendors such as powerade, supertrips and MDUK are missing from the list.
They're just vendors i remember off the top of my head.
Title: Re: Scraping the Road (Vendor Backup Pages)
Post by: Jack N Hoff on May 30, 2013, 05:13 pm
I'm not criticizing at all, but there is a lot of vendors missing.
Some big vendors such as powerade, supertrips and MDUK are missing from the list.
They're just vendors i remember off the top of my head.

I didn't even notice that.  So many vendors, it's like sensory overload.  I would guess that there are probably at least a few hundred missing.  If people keep pointing out missing ones then we'll get a lot of them added though. :)
Title: Re: Scraping the Road (Vendor Backup Pages)
Post by: heatcheck on May 30, 2013, 05:25 pm
Subbed.

Someone wrote a blog post using this list as a source for how many vendors were on the road. Its posted on the SR Reddit.  I wonder how the number of vendors on SR compares to a site like Etsy. Maybe one day we will have more.
Title: Re: Scraping the Road (Vendor Backup Pages)
Post by: StExo on May 30, 2013, 05:50 pm
Subbed.

Someone wrote a blog post using this list as a source for how many vendors were on the road. Its posted on the SR Reddit.  I wonder how the number of vendors on SR compares to a site like Etsy. Maybe one day we will have more.

What is the source of this? I never go on Reddit as I hear there is an abundance of stupidity there which I think rubs off on oneself if you spend too long around that level of intelligence. There are far more vendors than on my list because as I explained above, there are a whole host of limiting factors as to how accurate this is and only DPR & Co will have any solid figures on this and he has made clear his position of not being involved in this as it's a community project so I'm respecting that and won't bug him too much.

I'm working with another forum member to find a better solution than the one I currently have so hopefully the next update will be far better.

@Dingo - Will add you in a moment!
Title: Re: Scraping the Road (Vendor Backup Pages)
Post by: heatcheck on May 30, 2013, 06:03 pm
What is the source of this? I never go on Reddit as I hear there is an abundance of stupidity there which I think rubs off on oneself if you spend too long around that level of intelligence. There are far more vendors than on my list because as I explained above, there are a whole host of limiting factors as to how accurate this is and only DPR & Co will have any solid figures on this and he has made clear his position of not being involved in this as it's a community project so I'm respecting that and won't bug him too much.

I'm working with another forum member to find a better solution than the one I currently have so hopefully the next update will be far better.

@Dingo - Will add you in a moment!

Your list is the source that this blog post is referencing  ;).

It was actually posted in the r/drugs section, rather than the r/silkroad section.

Here is the link to the blog post: [CLEARNET]http://weirderweb.com/2013/05/29/silk-road-has-likely-topped-1000-vendors/[CLEARNET]

Here is the link to the reddit post: [CLEARNET] http://www.reddit.com/r/Drugs/comments/1fb4en/silk_road_now_has_about_1000_vendors_selling_at/ [CLEARNET]

Title: Re: Scraping the Road (Vendor Backup Pages)
Post by: Jack N Hoff on May 30, 2013, 06:17 pm
Quote
The site has 976 active vendors at this very moment according to a new crawl of the vendor’s pages from helpful SR member StExo. In response to recent DDoS attacks on Silk Road, StExo archived the html from every vendor page in an effort to make it easier to contact them during blackouts.

Way to go StExo!


Weirderweb loves writing articles about Silk Road.  They are so infatuated with this place.
Title: Re: Scraping the Road (Vendor Backup Pages)
Post by: MaidMandy on May 30, 2013, 06:26 pm
Hats off to a very impressive bit of work :)
Title: Re: Scraping the Road (Vendor Backup Pages)
Post by: StExo on May 30, 2013, 06:48 pm
Your list is the source that this blog post is referencing  ;).

It was actually posted in the r/drugs section, rather than the r/silkroad section.

Here is the link to the blog post: [CLEARNET]http://weirderweb.com/2013/05/29/silk-road-has-likely-topped-1000-vendors/[CLEARNET]

Here is the link to the reddit post: [CLEARNET] http://www.reddit.com/r/Drugs/comments/1fb4en/silk_road_now_has_about_1000_vendors_selling_at/ [CLEARNET]

Admittedly, I don't mind a few compliments or thanks for the work, but I never set this up as any kind of study or otherwise something for those outside of the forum as there will be plenty of Reddit, from having a quick browse, who are totally clueless on how to maintain their security and safety when ordering products from the road. But oh well, I'll enjoy the limelight eh whilst I still can.

Behind the scenes as I've said, there is far more going on. Like the deep web itself, the front cover is only 10%. However, it's not my job to reveal some of the darker things on the road, or to disclose anything I have been asked not to by individuals who I respect. For what information I did release to astor, we found some extremely disturbing results so whilst we won't go into great detail on the subject now, you should all be aware that it is not only me scraping these pages, but law enforcement around the world who have probably done this for months now. Remember that "modify" button on your post or the option for vendors to change their profile may change how the surface appears, but it defenceless against historical copies of the site.

Anyway on a more positive note, with this project, it will continue for as long as I can possibly maintain it and if I had to state the future of it and some of the discussions I've been having with fellows members: You ain't seen nothing yet ;).
Title: Re: Scraping the Road (Vendor Backup Pages)
Post by: StExo on May 30, 2013, 06:50 pm
Hats off to a very impressive bit of work :)

The hats come off to the vendors, staff and buyers of the market who are the soul of the revolution my friend.
Title: Re: Scraping the Road (Vendor Backup Pages)
Post by: Okamiyasha on May 30, 2013, 08:39 pm
EPIC you NEED A MEDAL ;D ;D ;D ;D ;D ;D
Title: Re: Scraping the Road (Vendor Backup Pages)
Post by: joywind on May 30, 2013, 09:04 pm
i don't see 3jane
Title: Re: Scraping the Road (Vendor Backup Pages)
Post by: RoxiPal on May 30, 2013, 09:11 pm
Cheers to OP and all of those involved, Long Live SR
Title: Re: Scraping the Road (Vendor Backup Pages)
Post by: StExo on May 30, 2013, 09:25 pm
In the process of updating and adding some more vendors guys, bare with me.
Title: Re: Scraping the Road (Vendor Backup Pages)
Post by: astor on May 30, 2013, 09:25 pm
I just saw that article posted on /r/onions so I came here and realized it had already been posted here.

This forum pays a lot of bloggers' salaries, it seems.
Title: Re: Scraping the Road (Vendor Backup Pages)
Post by: wicked420 on May 30, 2013, 09:27 pm
I have a hidden service that is multihomed and not hosted on freedom.  I wouldnt mind helping to host the backup information as well.

Currently my hidden service is just hosting guides on GPG encryption, and some other basic tools related to remaining secure on Tor.

Let me know if you need further hosting needs.

http://p3lr4cdm3pv4plyj.onion/
Title: Re: Scraping the Road (Vendor Backup Pages)
Post by: SealTeam6 on May 30, 2013, 09:39 pm
That's dope!
Title: Re: Scraping the Road (Vendor Backup Pages)
Post by: StExo on May 30, 2013, 11:26 pm
I have a hidden service that is multihomed and not hosted on freedom.  I wouldnt mind helping to host the backup information as well.

Currently my hidden service is just hosting guides on GPG encryption, and some other basic tools related to remaining secure on Tor.

Let me know if you need further hosting needs.

http://p3lr4cdm3pv4plyj.onion/

Many thanks for the offer but not right now, we still need to get our heads around the 1 host first, but we'll be in touch when we're a bit more stable.
Title: Re: Scraping the Road (Vendor Backup Pages)
Post by: Crazy Eights on May 30, 2013, 11:35 pm
Good job. it appears you put time in - so +1

please do not include us in your page - Thanks!

ce
Title: Re: Scraping the Road (Vendor Backup Pages)
Post by: Dingo Ate My Drugs on May 31, 2013, 01:31 am
Subbed.

Someone wrote a blog post using this list as a source for how many vendors were on the road. Its posted on the SR Reddit.  I wonder how the number of vendors on SR compares to a site like Etsy. Maybe one day we will have more.

What is the source of this? I never go on Reddit as I hear there is an abundance of stupidity there which I think rubs off on oneself if you spend too long around that level of intelligence. There are far more vendors than on my list because as I explained above, there are a whole host of limiting factors as to how accurate this is and only DPR & Co will have any solid figures on this and he has made clear his position of not being involved in this as it's a community project so I'm respecting that and won't bug him too much.

I'm working with another forum member to find a better solution than the one I currently have so hopefully the next update will be far better.

@Dingo - Will add you in a moment!
Thank you
Title: Re: Scraping the Road (Vendor Backup Pages)
Post by: Miah on May 31, 2013, 02:01 am
Wow very nice contribution. I can offer my help be that as it may. I do web programming but never touched a site on the ToR network although I'm quite curious to find out myself what they differences are and learn more about. I'm here if you need me  8)
Title: Re: Scraping the Road (Vendor Backup Pages)
Post by: BenJesuit on May 31, 2013, 02:31 am
+1 Nicely done.
Title: Re: Scraping the Road (Vendor Backup Pages)
Post by: BYSD on May 31, 2013, 02:47 am
this is amazing.. so much love sent to you for this project.
good going stExo!  ;D
Title: Re: Scraping the Road (Vendor Backup Pages)
Post by: StExo on May 31, 2013, 02:35 pm
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1



Quick Update:

1. For the last 12 hours or so Freedom Hosting has had a few timeouts so I have updated as per all the requests above but just cannot upload it just yet as it has a fairly big file size with so many pages.

2. I've been in talks with a few people now and one solution has been proposed to not just backup vendor pages but SilkRoad itself, every product, the Wiki etc and somebody has a good idea on how to go about this so I'll be testing it out in due course. A fully functioning offline SilkRoad would be nice, but I am not sure how useful it would be beyond the vendor backup pages since it'll increase the storage requirements by 10.

Let me know your thoughts on point 2.

StExo

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (MingW32)

iQIcBAEBAgAGBQJRqLUyAAoJEO0W4z0WMXVa80EP/i19MP1Z97pZoCeuinIjRZhs
wNELD/cdpm5lKMKgEXzazhteYeWeb+B0cZCjHMid2scVLWZgO4YKwERY1fQozI1z
tlZBqEapKT4UN1AbZkyQLphbb5J24C+KAVakYbipV3P5yrzRzo5IKBJA5S0I1NOs
R8hJ9BiiQuyIZ5nMTIgBYviQFENddHcSmUhQTKVBHUGKsMuIWhCXOcPO+eCmh/iJ
Aj0sY8msPX56sGy1ZOefCZpOCcfjuEv/tAlR2/pWwoDJuOfuuCo3PLwqUWfcub85
L6sXzSS9JLbXc017xAwkHoupleFrydkun61fiIhZwdb8d5Q7DDfrkn6nV58f5Ouh
Py5GlsaZEGCxP6FdWkFu9zGXxKRLDEM9Tri3EeXUERbs9rD7uTlCaTSqhuTKEqRn
zhqYkTkncKLtyKz6mgY81C1HF2wq2k5UoLd6Z4GNho/ErciZGvoZSlZIcMDngOjG
O3HupxiFH5y5oTMPeUDvWPzXBiRvIlbvyPiVlUw4gcvZfp3+akIYe5jaTtlVf5Vh
zskH+25LVidcXXKF9tQ23xsOAy/BxRnNcvupmrvPY67HcFLvzlh4rrB0qhL+T/+W
FK9a8oTLZ66F3s7IrYOZJJuRgzCwRvIprGXRvreF6rFACBvYVdm1L8hQMDuOcKhj
MKSswcA0Rex7Wb/cLVExiQIcBAEBAgAGBQJRqLUyAAoJEPu9CfR0ll5p80EP/1C0
U9VeA65IynmQZVEn2W6Uu6twrGovi4/6qpwS2U2/s9SfKan2boVCKRT7WuCHOEoK
PnMvGXYFLS/InB3iHWEw6xma1tFCLrqV9VROOuO8ZwjORvKe8wT2FOo3mQ5HrE66
9NmDZ5jaGRrf203oovoXQEeBBbuzz8CFmS3D/FwIHEmgqQJzwvZplCppB2txa2bm
R/x0QZ+giGrVSKSvQbFbTtNTuvFo7GMjoNWaunM49A68W9dvNQ2kivSQnm/ndkxF
b/bDPcjJflWjiPIxgd0vWQ4I68uai6aIimCP3iyQHlAoDgTeHjG6cZSBf6gNS3Se
NjsFgoatG2zEXH2mMyYfjU+owcQ2dsVnx/MXHaE+xcd55L7oFYEGGv8aLOXG9eeO
D8HvmRRhtgOgwNhtMjZUweTL+C4riebw6nemyob31BeBPTGmQVOzHzIJ8XKmOFVK
xH7XQAp1S1HH4tsJ7+pngaRhtmlxISpLmOB7Sf+Lj2IFoEuKopyde+nMHk0db5Y8
kLkh2HN3TPR63bDRzgymxNgZUYb1ETJg7E5t3oFjvxQ90MeeP1VmsGmLH2rXocEK
OlHXIXRa1UmAjsPTy6BMBbsi1kqcmW7iSW8CZZbs1ITPwpEkXcq2KjuERvXRBU1+
gFh9HxFtoFVgfy4hh6SGnqudBn0Rxlv22jkJ0CUr
=CkVL
-----END PGP SIGNATURE-----
Title: Re: Scraping the Road (Vendor Backup Pages)
Post by: Jack N Hoff on May 31, 2013, 04:09 pm
Quote
I've been in talks with a few people now and one solution has been proposed to not just backup vendor pages but SilkRoad itself, every product, the Wiki etc and somebody has a good idea on how to go about this so I'll be testing it out in due course. A fully functioning offline SilkRoad would be nice, but I am not sure how useful it would be beyond the vendor backup pages since it'll increase the storage requirements by 10.

Is this really feasible?  How would it continuously update?
Title: Re: Scraping the Road (Vendor Backup Pages)
Post by: StExo on May 31, 2013, 04:37 pm
Quote
I've been in talks with a few people now and one solution has been proposed to not just backup vendor pages but SilkRoad itself, every product, the Wiki etc and somebody has a good idea on how to go about this so I'll be testing it out in due course. A fully functioning offline SilkRoad would be nice, but I am not sure how useful it would be beyond the vendor backup pages since it'll increase the storage requirements by 10.

Is this really feasible?  How would it continuously update?

It's feasible to do yes but it just changes the method of collecting pages. So instead of downloading pages for www.xxxx.com/xxxx it just lands on the homepage and starts following every link it can see, just like how a search engine does. It means updates are done quicker and will index more vendors but at the same time, increases storage requirements ten fold.
Title: Re: Scraping the Road (Vendor Backup Pages)
Post by: Jack N Hoff on May 31, 2013, 04:58 pm
Quote
I've been in talks with a few people now and one solution has been proposed to not just backup vendor pages but SilkRoad itself, every product, the Wiki etc and somebody has a good idea on how to go about this so I'll be testing it out in due course. A fully functioning offline SilkRoad would be nice, but I am not sure how useful it would be beyond the vendor backup pages since it'll increase the storage requirements by 10.

Is this really feasible?  How would it continuously update?

It's feasible to do yes but it just changes the method of collecting pages. So instead of downloading pages for www.xxxx.com/xxxx it just lands on the homepage and starts following every link it can see, just like how a search engine does. It means updates are done quicker and will index more vendors but at the same time, increases storage requirements ten fold.

Ah.  I have a program that does that.  I forget what it's called but it follows every link and saves the pages.  You can set it to only save pages within that domain or not.  Surely this program would have to be running continuously to be able to keep up with the ever changing items, prices and vendors on the marketplace correct?
Title: Re: Scraping the Road (Vendor Backup Pages)
Post by: SelfSovereignty on May 31, 2013, 05:01 pm
Quote
I've been in talks with a few people now and one solution has been proposed to not just backup vendor pages but SilkRoad itself, every product, the Wiki etc and somebody has a good idea on how to go about this so I'll be testing it out in due course. A fully functioning offline SilkRoad would be nice, but I am not sure how useful it would be beyond the vendor backup pages since it'll increase the storage requirements by 10.

Is this really feasible?  How would it continuously update?

It's feasible to do yes but it just changes the method of collecting pages. So instead of downloading pages for www.xxxx.com/xxxx it just lands on the homepage and starts following every link it can see, just like how a search engine does. It means updates are done quicker and will index more vendors but at the same time, increases storage requirements ten fold.

Ah.  I have a program that does that.  I forget what it's called but it follows every link and saves the pages.  You can set it to only save pages within that domain or not.  Surely this program would have to be running continuously to be able to keep up with the ever changing items, prices and vendors on the marketplace correct?

To be honest, just about every Linux distribution has a program that can do that:
Code: [Select]
wget --mirror -e robots=off --page-requisites
Title: Re: Scraping the Road (Vendor Backup Pages)
Post by: Jack N Hoff on May 31, 2013, 05:09 pm
To be honest, just about every Linux distribution has a program that can do that:
Code: [Select]
wget --mirror -e robots=off --page-requisites

I don't know how to do anything on linux other than browse the net, email, pgp and crack wifi.  I'm not very proficient on it.  I usually can't remember what I ate for breakfast so the whole command line thing isn't my cup of tea.

This would have to run 24/7 because the market is always changing right?
Title: Re: Scraping the Road (Vendor Backup Pages)
Post by: StExo on May 31, 2013, 05:25 pm
This would have to run 24/7 because the market is always changing right?

Maybe not 24/7 no but I'm sure running it most of the day will let the whole thing stay up to date. Unfortunately we can't get a frozen picture of SilkRoad as it is always active and changing so simply reducing the timespan the captures are made over will be good.

I'm also not on linux and pretty useless with commands for that matter. But at least with some kind of indexing, it takes out 99% of the work required and I just need to do a bit of manual checking for the vendor list itself and provide an alternative link if somebody wishes to enter the actual full index.
Title: Re: Scraping the Road (Vendor Backup Pages)
Post by: SelfSovereignty on May 31, 2013, 07:37 pm
Oh; sorry.  I just assumed whatever machine you'd be using to perform this continual ripping would be running something like Tails.  It really doesn't matter, that just seemed like the easiest way to me.  Hard to argue with what works though, right :)
Title: Re: Scraping the Road (Vendor Backup Pages)
Post by: Jack N Hoff on June 01, 2013, 03:16 am
There should be roughly 1100 vendors because there is eleven vendors in the top one percentile.
Title: Re: Scraping the Road (Vendor Backup Pages)
Post by: HEATFan on June 01, 2013, 03:23 am
There should be roughly 1100 vendors because there is eleven vendors in the top one percentile.

I was thinking this same thing and I can't quite tell why yet (because I'm dumb) but that number seems wrong to me.
Title: Re: Scraping the Road (Vendor Backup Pages)
Post by: Limetless on June 01, 2013, 03:30 am
That's very impressive StExo. Kudos. :)
Title: Re: Scraping the Road (Vendor Backup Pages)
Post by: StExo on June 01, 2013, 03:52 am
There should be roughly 1100 vendors because there is eleven vendors in the top one percentile.

That sounds about right I had the same calculation which helped make my prediction. However I guess a deeper index will be interesting.


That's very impressive StExo. Kudos. :)

Can't afford to lose contact with you, in the last stages of planning my next move we spoke of in mail so stopping it now will be pretty pricey for me :(

=======================

General note:

More technical issues I'm afraid. 24-48 hour delay until I can get some more info on solving it.
Title: Re: Scraping the Road (Vendor Backup Pages)
Post by: phoboss on June 01, 2013, 05:29 am
I always thought there was at least 100,000 vendors due to everything on offer or am I living in cloud cuckoo land again oh dear I must wake up one day
Title: Re: Scraping the Road (Vendor Backup Pages)
Post by: StExo on June 01, 2013, 05:36 am
I always thought there was at least 100,000 vendors due to everything on offer or am I living in cloud cuckoo land again oh dear I must wake up one day

With around 12,000 drug listings and off the top of my head 20,000 total listings, I think you were out by a small margin. By a small margin I mean that's like describing the world population as 70 million...

Had to be said :) However, I hope you're enjoying your trip (please excuse the pun).
Title: Re: Scraping the Road (Vendor Backup Pages)
Post by: Jack N Hoff on June 01, 2013, 05:39 am
I always thought there was at least 100,000 vendors due to everything on offer or am I living in cloud cuckoo land again oh dear I must wake up one day

9,028 drug listings lol.  Definitely in a cloud cuckoo land again.  Hopefully we'll have 100,000 vendors in the future. :D
Title: Re: Scraping the Road (Vendor Backup Pages)
Post by: RxKing on June 01, 2013, 06:33 am
All I can say is.....................................Holly shit this is impressive1!! The amount of work and time put into this by yourself/Astor/jacknhoff for the sole purpose of helping others on this site is really awesome. Good work.

Do you mind telling us just how much time this actually took ...Just curious?

Also if there was ever a time for someone to have a donation address in the signature ...you are the first that should do it and deserve it and I will make a donation. So please put an address up. Even though you do not have me listed  :-[


 Thanks
Title: Re: Scraping the Road (Vendor Backup Pages)
Post by: StExo on June 01, 2013, 11:39 am
All I can say is.....................................Holly shit this is impressive1!! The amount of work and time put into this by yourself/Astor/jacknhoff for the sole purpose of helping others on this site is really awesome. Good work.

Do you mind telling us just how much time this actually took ...Just curious?

Also if there was ever a time for someone to have a donation address in the signature ...you are the first that should do it and deserve it and I will make a donation. So please put an address up. Even though you do not have me listed  :-[

 Thanks

To compile the data, about 36 work hours of manual page downloads, another 5 hours naming and labelling them, 12-24 hours to set the website up and ensure all personal data in the files was removed (some programs insert metadata unfortunately) and another 8 hours to filter out the pages we didn't want to display in public as we were shocked by the information on vendor pages and simply could not publish it for the safety of the vendor so we've let those concerned know (it affects about 10% of all vendors) and asked them to correct it.

But you definitely are in this next update, just done a quick search and I've got your page :)
Title: Re: Scraping the Road (Vendor Backup Pages)
Post by: RxKing on June 01, 2013, 02:07 pm
.................pages we didn't want to display in public as we were shocked by the information on vendor pages and simply could not publish it for the safety of the vendor so we've let those concerned know (it affects about 10% of all vendors) and asked them to correct it.

Wow.Impressive.That is a lot of time for this project. Also your page is very interesting and is a great place to go if you need or want to know more information on the dark web. Also it has links so you can search the dark web. It is a very cool site and basically a must go to for people that are new to SR and even those of us that have been here a while.  Whoever has not gone should go to http://5uvrgtrgtwkkxsgw.onion/


Without giving away the actual information OBVIOUSLY.....can you tell us the kind of information that "shocked" you?
Title: Re: Scraping the Road (Vendor Backup Pages)
Post by: oooooner on June 01, 2013, 02:15 pm
This is absolutely fantastic.  I cannot even express my thanks as I truly am living off my income from Silk Road right now so I am so in awe at your abilities and at the same time so grateful for putting this into place.  Amazing, my gosh.  Thank you.
Title: Re: Scraping the Road (Vendor Backup Pages)
Post by: StExo on June 01, 2013, 02:28 pm
Wow.Impressive.That is a lot of time for this project. Also your page is very interesting and is a great place to go if you need or want to know more information on the dark web. Also it has links so you can search the dark web. It is a very cool site and basically a must go to for people that are new to SR and even those of us that have been here a while.  Whoever has not gone should go to http://5uvrgtrgtwkkxsgw.onion/


Without giving away the actual information OBVIOUSLY.....can you tell us the kind of information that "shocked" you?

Thanks for the kind words :) But I'm afraid the information we can't release yet until we've sorted the problem out. If we did publish it, it's like handing an automatic prison sentence to a lot of vendors (yep, that bad).

Been hit by technical issues over the past 2 days since SilkRoad isn't designed for being crawled and the Tor connection to do so also means it is hard to do it with any speed. Progress is being made though and 1 last barrier and it shouldn't be too hard to get daily copies going almost automatically.
Title: Re: Scraping the Road (Vendor Backup Pages)
Post by: StExo on June 01, 2013, 06:25 pm
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1



Quick update for everyone!

The lastest index of SilkRoad obtained more results than I previously predicted. Having filtered some of the vendor pages for their own security, we now have indexed a whopping 1,239 vendors who are actively selling at any given moment.

This latest index of vendors will be published later tonight and I'll keep you up to date on developments as they occur. Right now the job which needs to be done is sorting it all into alphabetical order and then tying each vendor code (the last few digits of their vendor page in the URL) to the vendor username. Once we've done that last step, it'll be uploaded.

Many thanks to SelfSovereignty who has been wonderful in helping me get down a methodology of collecting the pages quicker and more efficiently, the rate of accidentally downloading the same vendor page twice was 1.3% as opposed to the first attempt which was 81.5%.

Finally, let me express my word of thanks for the kind words thus far for the project. I am always open to ideas on how to expand it and add new services to my site, not just for the vendor backups but anything which could be useful to us at all!

URL: http://5uvrgtrgtwkkxsgw.onion/

StExo


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (MingW32)

iQIcBAEBAgAGBQJRqjyMAAoJEO0W4z0WMXVabWUP/0JaALHLC45djWaHLIq5/Hfl
B7dahsg2VT5C/1CPqSEe9VuerSgiA0+RuKLfdEFa3yhVh6nN62SUVKXSJ1TyYaOm
jODE2wLTVX7M3b7KdTFXp1JuuTle6uQnrCwj0uTVoy2QSvFWlSB/j6xplSfxcYL7
YKpuFk/aZry7g4eWiDAwUV4NnhqNqXesd8TF4mVcOckEnJ8EYnuTTWgEZdVi6PFs
4T++xLhef51WtLeZ3+xL86d5rNzJPgrjSgcm4Y0tE5h1/E0GzJvinZAA+1Pf9rpG
g4y7tsJLI1SUQXqOAThUqK21OAYh2IPYfPDLHATfI7GbSR62KPAkgdEvFoC9f2Dg
wszqB2gmZse0tgSXdSjwXxpoYfOELSM9srVr7qhwRaPKREfsrwKpcEcXN6tY9/oU
L55I+DEtjt3e+l8L8vZ8CV3fw8tiA9ID9dul53Fj7MY1KtKVmL/tK3TWIuRGYYJ4
dumBNXXPsEPh8+hqFtY2nt85aOGCaigaDneO38wRBA6bRwHFZG/K5UKcE5UY9Mgf
KHyouz9AFVSkfnva7Or1oVhDoFGH6JB+BgaKo46roUstbgMUZk4dY0nzbQdwgonu
iYK9jrBj1cYyo0bSKNlZMFtbe3v1G8x55DexjLx8AG6X22aN26bX4noeVvZV+Jgu
xjPn2J+c9U+gPST/xvSriQIcBAEBAgAGBQJRqjyMAAoJEPu9CfR0ll5pbWUQAKOV
L+3fDsv9fbL8be+a2zrTtoHopGuooFzSJKjHEGSIO4Tmn79iSYX7893sqvVw5DHz
QsGuJS7DmuMPsAVrEHLdes5Utt2mxjHDLxRlmX4RnkxAf4YEiAUXeb7tccePp/QU
q7FvCZ8fb86a9wPgimpvgzLpSdSnATaJjLjaQ4H3L3+46YiV42JcIFCfg6dIJgje
1+v9XNacpEFtu6+hRlBJjFhLXF00tBM+veePGRj/lcP/Gf4QX6z9PJ2RKz95OYtN
xb3Z1QuKexHEKznHZnOuuV0i779ucIZm2NGxzaMacM1bPHfjSNqn1f1YGtc6sxhB
5oPlSKqfe1jz8zIxuWExvH0W83vsicUgBDdAXYDGXF5mjDW3kCxIuJNoP9JCoySW
qVuVFidtR78va+gyJ+Ag+SxhBWHZq1FEjr/VsygcHSz73VNwWalnxHVKiC1fAmZK
ugitdUHMiFQeH+kWkcQHeSyj9lfR9Cgvjqc4IAw+xSjnGXotDjtUZ3g79ImBm0//
DTL0i3jZasY5Y9+5kJuwpY+rsYH5zUiWmTmoKtE+hFXZi5vNowpbXDA1gTdlnf/S
jAyMvbGxWvEkcB+/GRxJqMRCpab8LAwKMRmrxZLhJAC8qX2hVndPerPTNTrdOdzF
LDw3BpDA1l13NCIjE1znY9G/q7xaGvN+WAnvRcuc
=wq2K
-----END PGP SIGNATURE-----
Title: Re: Scraping the Road (Vendor Backup Pages) - 1239 Vendors and counting!
Post by: StExo on June 03, 2013, 05:51 am
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

@WeirderWeb:

Thanks for sharing this again, gave it a mention on my site too. Please check your inbox as I think my e-mail was caught in the spam filters.

StExo
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (MingW32)

iQIcBAEBAgAGBQJRrC7zAAoJEO0W4z0WMXVa5RAP/0lCKj2502Tq88D+iERCmCa+
8yrJN16CP3r7u1Hf+u17qTcjBJMo4WrC7iRgV9neUn9MyGhzJ8Fov6YKqyvyA6Bd
fcR9xsdIdYnRgYcSTzDBH5UbbjdjH4RdcxmiGFHbAnZPDUSwP7JoQc32EaqTvKzQ
v81OqzvyyST07y3l9LOfEmnCseuPUoxHXD5YZzPOFVB38toWcDGgsOOX0eQePnDW
b8c0iMi3GA9iuHbhkerC+qWSU/BACYV01oMOQ64AIFmGw3+3P5urzs8APUYqrNag
7lan+m/72LPr5u9VltMjmuXihSIxcu0lmiElhGxKd0JQ5ui+eXXSmbcoeYNPhGqj
qEuaCtNjBBCfhgz4Kl1MRpuIktfX4Crl180lAa/1doDcTrA8JyxSey5NJk0Fnlvb
QGWN7lAnhrk6bB8j0V4zpUJhbarD2+wu0dkT54qe5mli5dSeXO3uCcnk79K7aKI/
UBTarysaqJ3fzxYRJlmL8cFqj1rFmMSRrpwJPwrXEbu1kaHl66VPn3N8TQY/8elm
jXdIwI9A7YXG0Nm5lhJlp2Sfcz+lNF6ikTBMAAo6B80g+wIbOp5osTmM78ipAaew
1zSUT/E19Lguiqur3ntKnap99FgjvKGm1Cmh+0cqflYgTZJIj9Wj6zgBmdqW2/UT
SUEcBFxNcvg0AFoPCWRFiQIcBAEBAgAGBQJRrC7zAAoJEPu9CfR0ll5p5RAP/i5f
8XnVLSWa3ZUm0t0gd5Lj0p5YDMHfA3fRldIEXS/CtvipNdGFqrGSNs7QP0qYPnjx
C03GYRA6/05VK6YTt1QvLKrC3xi/RPwDpviibuMkrlVewS5boQpcV68fEwGGAgzm
PxFPp6BdElnxN0PHAlopu50IRukIB305rqipdue9suRvtuDpGr6MZf2LC4q9V/P/
cpMrOZwVUqkRjzZLGsTwyOGHelxvsH34oJJDB5Gv1wnDXLR8DMDWuseQ1b5VPZPe
hILWITBt38ldQ48pE6N0pqN9uVdAnzmc3wsrvE9k9l4gY1Xa/qmQaIbQO8+O34wE
4SXpJNdnszQgaetsu80P8wmyIVNqXXBnI7dpATFXn4YccUpXsj/0lYe/yl3s++5A
wSEx8vWxPFNXC9HwE79zccyYc/m2y7+nK0gc4THNZrK6ip5RIEHtVzQX9y7sOz2I
pXjGC54wFfTHWIrwk6C4QAf0MYfAaYDiDxvGQt0Vy9HDA6PJECTLnPIlM8E0TlVy
UjiZXqjjkdyJhK5Cx2U4hBxwWRXOX/rMJBnpLvNPeGsLdQb9PjWLDU/0NKDBSlWV
2GiPNlKT7Y2d7N0AU+R8XBRdHgFD8dyPuunP0i8S4I6CbZclaLH9a5wL97ro69a/
jEcTR28xcuUkNcStg0PNbER45U7LzUHjx68essvh
=XQsc
-----END PGP SIGNATURE-----
Title: Re: Scraping the Road (Vendor Backup Pages) - 1239 Vendors and counting!
Post by: StExo on June 03, 2013, 01:16 pm
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

The latest version containing all crawled vendors (except those who opted out) is now online!

http://5uvrgtrgtwkkxsgw.onion/#vendorbackups
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (MingW32)

iQIcBAEBAgAGBQJRrJc1AAoJEO0W4z0WMXVaq6cP/A2Ou4Tt5RV/8bfuB5vkFQ59
dEMiF2Lni/7u2c4ti3UTFZuE9QYbRkzKuDDbm0DpDr6gIL3vQ5XpGhAYVKYWMNza
Um6PNALNwM7f9+1CyBPlmbsyh9E7b0OjFAIgtUmtYzfUBCHGfWWd0X++pgdFGLBu
T4NX7Zu0NHy6aZYiz3gqv/Al+ektN8IbCbJ7IsD8LkR+ls4SatDBJHuwWvWkfCWC
4XaWVdo7GGC3n3DsndFFaoYYVARS0VHC7+h0XuskDZc4VBW+6Z+5nu1DQqUM+XZB
VaiOUlQKeafKxtSg/2tpehQzT5Vb+0ySsXzbuuyRz4YKpJg7Sj43DgH/SSefXVeJ
Iz0fz8nuw+ha4+VNzvChcoiOcmGcvwBFoYG0LAlYPKY7VKhlbIY6QdGFlKZcMqLO
QckMsQnwa/JGLh2lD5jItkKBv0ZH0G1O1a7aZBPyl1MPniPEhpNc1XaU4iSMS9ws
XudflJ0iEmJ1OxrDN7iwyM87PTxiurc5kV4bOmngZiaGsOH+jI/1M219MdzsMk5+
O2v/GCO3Z4grV1IL/NQ+5b0pgbFhjXnwU7QgS1yRpPUQN6VPgsPbkga/0oe6WDLb
gn9fwpr0cM9rO10K1w1nnRRy8CNUw637/MiegB+ujIPuZtu71Vbrki9ixx9EVre8
sBpeodcpvcsLWkz4iM7OiQIcBAEBAgAGBQJRrJc1AAoJEPu9CfR0ll5pq6cP+wVS
ncEFaBp+WOtEabcUZVPg5zdbGBc5pUs563T3N2aWp/69iWGQJREEBtLahiyrsRdp
EzHnZRTh+8Kgpk1hdH/Usqs+aIQ+eSY5HP0M/jhYjNaq43SMi3dxg62+Psp1zPDM
2vn+T1xPW9BbD09GS6juZxltipT2D7LB7/Y+WYQq44voMxAc9MFm0jhyb1PC+ROc
M2HOtdMhGzhbJJb+Rb9qtNDyHgVsLkF7uiFQxooWfbH8I2Xau8Plul2XxwJqPLJm
7ECHnoLAM3BZj/XTmi/t4hJmFnqO/0GyNyHFKiKY9yMSpb9CkxCD4jcEkK3R+KvY
ahLOV2/3s1gJzuqT7mZl/5+fZjUDGyO/fQkFJHTWaYR3JuG3QMtZKyzZqY3Lrikk
yl48L/122ZK9Ct/EcYV1OhHQW6VO8kVQ5FjsS1nTzrfLA2/VrlaQ9hqhMVaUIuiI
YPTcsP6ySsuP6i+d8NMIDEVoOQnZukTcg0R2zvyGPUrRlrK0CcaBSCwH+F+qBj6M
EcJ6ueXtf7nY6yPIkM44WJRFSs7sFMqoOihY37MoBYhzFOWZ5OHkAoJjjgsLcBGF
fd+nnWRktFv377PfXZXVxpyv1yEhbga5AlBFizIHYdJVVHTzl2DPD1J2kjrWW2kk
PkrNDQnXH8kOzcUlXsmyKPX0jeM+hYAQbvRAtPwj
=Hd1t
-----END PGP SIGNATURE-----
Title: Re: Scraping the Road (Vendor Backup Pages) - 1239 Vendors and counting!
Post by: Vatican on June 03, 2013, 03:43 pm
Thanks for your work man, much appreciated!
Title: Re: Scraping the Road (Vendor Backup Pages) - 1239 Vendors and counting!
Post by: pakchoi23 on June 03, 2013, 05:11 pm
Subbing, bumping, props, karma etc. Amazing work, better than my paltry private effort.
Title: Re: Scraping the Road (Vendor Backup Pages) - 1239 Vendors and counting!
Post by: VersacePandaEgg on June 03, 2013, 09:43 pm
Subbed

Tried to give ya another +1 StExo, but it wouldn't let me! Haha

Great work!
Title: Re: Scraping the Road (Vendor Backup Pages) - 1239 Vendors and counting!
Post by: goblin on June 03, 2013, 11:23 pm
Hey, thanks StExo, terrific snapshots of all those vendors' profile pages. And thanks for getting me in there too, as I am still a small-time operator!

goblin
Title: Re: Scraping the Road (Vendor Backup Pages) - 1239 Vendors and counting!
Post by: CiscoYankerStuck on June 04, 2013, 08:18 am
+1'd and subbed.
Title: Re: Scraping the Road (Vendor Backup Pages) - 1239 Vendors and counting!
Post by: sharonneedles on June 04, 2013, 09:10 am
Thanks, Saint Exo! +1 buddy
Title: Re: Scraping the Road (Vendor Backup Pages) - 1239 Vendors and counting!
Post by: empireofthestate on June 04, 2013, 10:17 pm
Once i can give some karma you ll be first.. youve done the community a service..thank you sir
Title: Re: Scraping the Road (Vendor Backup Pages) - 1239 Vendors and counting!
Post by: MarleysMainMan on June 04, 2013, 11:57 pm
MMM
--------------------------------------------------------

Hello fellow cloud nine participants,We here at Marleys just want to pay our rent doing something we love and have been a part of for about 12yrs now. We know that we will be a shining star here due to our impeccable customer service along with quality and speed. We will always make sure you're more than satisfied with any order placed with us! With that being said, here are a few things you may want to know about us below...

SHIPPING

WE ONLY SHIP IN THE US. AND WE SHIP FROM THE EAST COAST FAR AWAY FROM CALI AND COLORADO.

Every precaution (as stated earlier) has been taken so in case of failure there are a few reasons instead of people trying to steal your money(which is not what were about). If there are any problems don't hesitated to contact me anytime:) All packages are shipped in standard business style envelopes with an inner lining of bubble wrap. after that there are 2 vacuum sealed bags surrounding your package and is concealed inside a container that holds no suspicion. you also wont have to worry about signing for your package. all packages should arrive within 3-5 business days.



MIXED ORDERS
If one strain just doesn't cut it for you then feel free to mix it up. we just ask that the requests are for 1/2 oz's and higher. please specify on any mixed orders in the address box when you order, we want your order to be correct just as much as you do.


SAFETY

We all the know the stress that these simple transactions cause so to help you calm yourself a bit, everything received on paper is burned daily, and as far as computers go our computers are encrypted with FIPS 140-2 Rev. 2 compliant data at rest encryption with multiple, strong cascading ciphers, proven hashing algos, and wipe all data immediately with excellent PRNG entropy generation. In other words you're info is safe with us. We do our best to make sure you get your package on time and safely! That is the most important thing here is safety.



COMMUNICATION

If anything goes wrong like failure to receive even with a DCN saying delivered were sorry but there are NO REFUNDS, but before typing up the worst thing you can possibly think of please try and get a hold of me so we can work out some kind or mutual agreement that leaves both parties happy. Especially if it was intercepted, we always can work something out so we're both happy :) We check our messages everyday so you usually get a response within 24hrs, (except weekends of course):)



-We do not process orders over the weekend. They will be processed Sunday night and shipped out first thing Monday morning.

LINKS AND INFO

EMAIL

MarleysMainMan00@tormail.org

BMR Backup

http://5onwnspjvuk7cwvk.onion/index.php?p=viewUser&id=120525


FORUM REVIEW THREAD

http://dkn255hz262ypmii.onion/index.php?topic=30010.0





PUBLIC PGP KEY BELOW GUYS:)

-----BEGIN PGP PUBLIC KEY BLOCK-----
Version: GnuPG v2.0.17 (MingW32)

mQENBE/7euQBCACfA0Hf05ZtwmsuYm5oszpNRJx4H5vNyhUitWmrNSGS4pHFzlV/
64xv3a9qpX8njvFQAAEFMNzepRArXsrC0OnZFDfumjYAoPk8IVGb/9GgNVJ1X7AZ
xgnLeaw0gzsW9BcA/6FXAsi6um9LoEA1sRAb2NmkqQAZQ/mlVogFvE25XOk1QSFL
EYs7oaVQJVkBJNvh8rGcF25pXfhdSqJN4rP7klHm3AFSOEuR6eiz0YnqEZ5eQOAs
0cyLdG6FIBMMBuyglnopwPwcIVWQni819AoC1rveBjy2ltU65PFvRtDuraeTmTs8
IKzctBPCreSN1ccok3VlprwgryuLFa42Fo/1ABEBAAG0SU1hcmxleXNNYWluTWFu
IChLZWVwIG9uIGEgc21va2luIG1lIGJhYnkuLi4pIDxNYXJsZXlzTWFpbk1hbkB0
b3JtYWlsLm9yZz6JATgEEwECACIFAk/7euQCGw8GCwkIBwMCBhUIAgkKCwQWAgMB
Ah4BAheAAAoJELMrRUcLqyqgvHgH/1Mrb9Z6C2iROrbNbK4JwN5H0FqzAMhxZkjT
9ONXzSGXJhYduTkidathL7TMzEapLQACMCnLZXOC3rPkdC2BVOSBB9OfAsDkbwJ1
pyJMGjqBBGN5tPvgArGYsXK86n7X69mcmE2PrhJvycB3yH1vB+28ljjRRzJzoov7
rAqlbsCMGczkErqjKon1yGIK2Dc6s/hfGCCmwBb79zbhG9M0FWDABX/3x0upKxy3
HoRk2fv3Qxf0U5U7124H+ATuls/aaCMJ7WazfR1WGMixBk09/QHse7bRE0mI/39D
80YZ56T+W1R+/IfPnOIVASioDyqdO4AfCMvfIIbKj+0LdQjrQYg=
=zYoZ
-----END PGP PUBLIC KEY BLOCK-----


*********Remember to STAY SAFE out there guys!************


MMM

P.s. we will be offering free grams to all who post a review or something in our review thread, just let us know you posted in our review thread before you place your next order and we'll add it to it. Apparently no one has time for the forums and we're hoping to create a little incentive to post their experience with us! MMM
Title: Re: Scraping the Road (Vendor Backup Pages) - 1239 Vendors and counting!
Post by: Jack N Hoff on June 04, 2013, 11:59 pm
What the fuck are you doing advertising in StExo's thread?!  :o 

You really thought that this was the proper place to advertise? :o

-1
Title: Re: Scraping the Road (Vendor Backup Pages) - 1239 Vendors and counting!
Post by: StExo on June 05, 2013, 12:40 am
@MarleysMainMan - You're already on the vendor backups? If you specifically want me to update yours due to a PGP or email change then PM me, don't copy the whole thing in here :/

Your page: http://5uvrgtrgtwkkxsgw.onion/vendors/fcfe3de881.htm
Title: Re: Scraping the Road (Vendor Backup Pages) - 1239 Vendors and counting!
Post by: jase00 on June 05, 2013, 07:36 am
I'll be honest, when I first saw this I was like eeh whats the point. *edit* just saw your other thread detailing issues you found in profiles.. so was pretty stupid of me to think that originally */edit*
But I actually used it for legit purposes the other day and now see the value in it :)
I'm just some random so you probs don't really care either way hehe but just thought I would say thanks heaps for providing this. It really is pretty cool and much appreciated.

I placed a largish order the other day.. didn't arrive yet so PM'd the vendor and got a bit worried. I noticed the vendors profile now said "no refunds blah blah" so I used your site to confirm my suspicions that they only changed the profile to say no refunds very very recently. Package turned up all good, but it was great to be able to check the feedback page from a week or two ago.

Any chance you could please archive the data for a couple of months if you are not already ? It would be very useful in situations like mine above where you need to see historical data. Perhaps have it searchable by vendor name as you have, but also the month ?
Title: Re: Scraping the Road (Vendor Backup Pages) - 1239 Vendors and counting!
Post by: StExo on June 05, 2013, 08:54 am
I'll be honest, when I first saw this I was like eeh whats the point. *edit* just saw your other thread detailing issues you found in profiles.. so was pretty stupid of me to think that originally */edit*
But I actually used it for legit purposes the other day and now see the value in it :)
I'm just some random so you probs don't really care either way hehe but just thought I would say thanks heaps for providing this. It really is pretty cool and much appreciated.

I placed a largish order the other day.. didn't arrive yet so PM'd the vendor and got a bit worried. I noticed the vendors profile now said "no refunds blah blah" so I used your site to confirm my suspicions that they only changed the profile to say no refunds very very recently. Package turned up all good, but it was great to be able to check the feedback page from a week or two ago.

Any chance you could please archive the data for a couple of months if you are not already ? It would be very useful in situations like mine above where you need to see historical data. Perhaps have it searchable by vendor name as you have, but also the month ?

The original purpose of the data was actually to simply be a backup so in the event SilkRoad went down you can still get in touch with vendors. Many vendors here earn a large portion or their entire income stream from here too so it's giving them a bit of a lifeline.

I will be providing historical data so for as long as I keeping taking copies (another is being collected as I speak and will be sorted tonight) so you'll (for now) be able to get your hands on the datasets for every 2-4 days and once my technique has been nailed, maybe even daily. All downloaded files are kept by the day they are downloaded too so in the future when it can be even more precise, I maybe will be able to extract specific data from the pages and obtain long lists of feedback for individual products and timestamp the downloaded page.
Title: Re: Scraping the Road (Vendor Backup Pages) - 1239 Vendors and counting!
Post by: astor on June 05, 2013, 01:38 pm
I'll be honest, when I first saw this I was like eeh whats the point. *edit* just saw your other thread detailing issues you found in profiles.. so was pretty stupid of me to think that originally */edit*

Vendors got a free security audit before LE gave them one. Really they should be thanking their lucky stars... and making a donation to StExo.
Title: Re: Scraping the Road (Vendor Backup Pages) - 1239 Vendors and counting!
Post by: StExo on June 05, 2013, 02:23 pm
I'll be honest, when I first saw this I was like eeh whats the point. *edit* just saw your other thread detailing issues you found in profiles.. so was pretty stupid of me to think that originally */edit*

Vendors got a free security audit before LE gave them one. Really they should be thanking their lucky stars... and making a donation to StExo.

All donations forward to your nearest children's/homeless charity please :) I have a roof over my head and food on the plate so I'm sure to other people that money would be far better spent - but thank you for the gesture.
Title: Re: Scraping the Road (Vendor Backup Pages) - 1239 Vendors and counting!
Post by: PoolPlaya on June 07, 2013, 06:37 am
thanks for doing this, StExo!  I know full well how much work is put into projects like this, especially in the beginning stages where most of the work is manual and mind numbingly boring.   But one continues to look for ways to automate tasks that take up the most time, until you got some process down that kicks ass.  Sounds like you're making progress in that area already.

So thanks for coming up with the idea and following thru with it.  I'm sure when you grasped the enormity of the task at hand, you might have had a few second thoughts about whether it was all going to be worth it.  I have found that just barreling right into it and investing blood, sweat and tears, makes it easier to make the commitment to finish it out :)  Once you've got 10 hours of grunt work in, you don't want that to all go to waste. ;D

I would be interested in hearing about your procedure for doing this (you can pm if you like).  Maybe I can research some solutions to any issues you have, or maybe come up with something that will cut down on manual work.  I have been in IT for the past few decades, and have always enjoyed finding ways to automate things... mostly in software rollouts or collecting information about pc's thru the network.  Anything that involved someone manually having to perform a task over and over, I would always be trying to find a away to automate it.  Work SMART, not HARD is my motto! 8)

Just wanted to say thanks a lot to you, and anyone else that has worked on this.  I feel it important to make sure you know that there are many of us that appreciate the time and effort that was put into something that can benefit all of us.
Title: Re: Scraping the Road (Vendor Backup Pages) - 1239 Vendors and counting!
Post by: StExo on June 07, 2013, 02:06 pm
thanks for doing this, StExo!  I know full well how much work is put into projects like this, especially in the beginning stages where most of the work is manual and mind numbingly boring.   But one continues to look for ways to automate tasks that take up the most time, until you got some process down that kicks ass.  Sounds like you're making progress in that area already.

So thanks for coming up with the idea and following thru with it.  I'm sure when you grasped the enormity of the task at hand, you might have had a few second thoughts about whether it was all going to be worth it.  I have found that just barreling right into it and investing blood, sweat and tears, makes it easier to make the commitment to finish it out :)  Once you've got 10 hours of grunt work in, you don't want that to all go to waste. ;D

I would be interested in hearing about your procedure for doing this (you can pm if you like).  Maybe I can research some solutions to any issues you have, or maybe come up with something that will cut down on manual work.  I have been in IT for the past few decades, and have always enjoyed finding ways to automate things... mostly in software rollouts or collecting information about pc's thru the network.  Anything that involved someone manually having to perform a task over and over, I would always be trying to find a away to automate it.  Work SMART, not HARD is my motto! 8)

Just wanted to say thanks a lot to you, and anyone else that has worked on this.  I feel it important to make sure you know that there are many of us that appreciate the time and effort that was put into something that can benefit all of us.

The trick is trying to load every single page on SilkRoad without needing to manually click through them. Once you can do that it's easy because you can then just grab all the page URL's, filter them so only the ones which contain /user/ remain and then make it a mass URL download. So other minor niggles like making sure it doesn't download the "report vendors" or pages like that since it'll trigger the option but they're easy once you've got the basics hammered. The next version is downloading this very moment but it does take hours to finish.
Title: Re: Scraping the Road (Vendor Backup Pages) - 1239 Vendors and counting!
Post by: StExo on June 07, 2013, 08:32 pm
Thanks, Saint Exo! +1 buddy

It doesn't actually stand for Saint :) It's the first 2/3 letters of words in a latin phrase which effectively forms my life code.