131
« on: August 11, 2013, 12:39 am »
UPDATE (08/18/2013 2213 UTC): Wow! So much feedback and ideas for ratings. I think this will be a major focus for a while until we can really flesh everything out so both vendors and buyers can have all of the information and tools they need to decide who they want to work with. At this point, there are so many things to do, we have to start thinking about what order to do things in and how to transition. Everyone responds to change either positively or negatively, and I'll keep doing my best to keep the negative to a minimum.
An idea I had not yet mentioned, but want to implement is moving discussion to the main site. What I am thinking is that there will be a "discuss" link on every vendor page, item page, and even the category pages. This link will go to a thread with the latest posts about the vendor, item, or category. I'm thinking we'll require a minimum spent of maybe 1 btc to post. Each post will include the user's account age, total purchased, number of transactions, and number of vendors they've bought from. For the item and vendor pages, we can display whether the user has bought that item or purchased from that vendor, and we'll highlight posts from the vendor themselves when they post on their own thread or a thread of one of their items. I'll want to enforce a strong policy of courteous conduct and will have a link for you to flag posts for admin review, and we'll give admins the ability to suspend posting privileges. Also, we'll want you to have the option to post anonymously, so I think we'll put a "discussion name" field on the settings page, so you don't have to post under your account name.
My hope is that this feature will fill in any unexpected gaps that might be created by changes in the rating system, so at worst, free and open discussion between buyers and sellers can be looked to when making decisions.
UPDATE (2028 UTC): Ok, I've rolled out the first set of major changes. We've done away with the percent based rating score entirely and are now displaying the ratings as a bar chart that shows the relative weight of each rating category (1 through 5) and a total average. This chart shows up next to each listing when browsing, and on the vendors page. I outlined the change to the vendor, buyer, and rating weights, but will put it here again:
The rating algorithm has been changed to the following:
A vendor's score, which determines their rank and factors into the weight of buyers, is found by adding up all of their sales with each sale multiplied by a factor that starts at 100% now and tapers off to 0% after 8 months.
A buyer's score, which affects how heavily their feedback is weighted when determining a vendor's rating (not rank), is found by multiplying the price of their purchases by the vendor score of the vendor they purchased from, and then adding them all together
A rating/review's score, which determines it's weight when averaging with other ratings, is found by multiplying the buyer weight of the buyer leaving the rating, the volume of the purchase, and the same dampening factor used for the vendor's score.
---
Normal protocol dictates that updates don't get talked about until they are ready for release. This is because if they get delayed or put on the backburner everyone won't be disappointed and the less info that is released the better in general. It also keeps the developers unconstrained as they work on the update. I want to try a different approach for an update we are working on. I want to include the community in the process, get feedback as we go, and pull back the curtain just a little bit.
The update we are working on is an overhaul to the feedback, rating and ranking system. The current system has been in place for a long time and served us well, but I believe we can learn from how it has performed over the past couple of years and make some much needed improvements. This system is integral to the proper functioning of Silk Road and affects everyone, which is partly why I want to make this development process public so I don't overlook it's affect on all of the various people involved.
So, here's what we have so far...
I want to rework how both buyers and vendors are rated. We've identified 6 critical dimensions that I think affect how SR users should be judged. They are:
buyer weight
total amount spent
total purchases
number of vendors purchased from
purchase distribution across vendors
weight of vendors purchased from
account age
vendor weight
total sales volume
total orders processed
total number of customers
sales distribution across customers
weight of customers serviced
account age
Some of these are self-explanatory, but I'll go through the ones that may not be.
"Purchase distribution across vendors" is a measure of how evenly distributed a buyers purchases are across the vendors they have purchased from. Let's say two users have spent $1000 on 10 items from 5 different vendors. If one of them bought 2 items from each of the 5 vendors and spent $100 on each item and the other bought 6 items from one vendor and 1 from the other 4 and spent $900 at one vendor, I think we should judge these two buyers differently. I think the feedback from the one with an even distribution is more valuable, all else being equal. The same is true for a vendor looking at "sales distribution across customers". If most of a vendors sales are to just a few customers, this is not as impressive at a vendor who has good feedback across a large distribution of customers.
"weight of vendors purchased from" and "weight of customers serviced" is a kind of feedback loop connecting the weights of both vendors and buyers. Basically with a vendor who has serviced buyers with a high weight as measured across the 6 dimensions defined above, we should weight them more highly than a vendor who has only serviced low weight buyers. Conversely buyers who've made purchases from high weight vendors should be weighted more highly themselves.
I think the other 4 dimensions are self-evident. Part of what must be considered is how much to emphasize the 6 dimensions relative to one another, and how to weight high values in each dimension relative to low values in the same dimension. So for example, how much should account age influence a buyer's weight compared to how much they've spent? Should a user who has been around for a year and spent $1000 be weighted higher or lower than a user that has been around one month and spent $5000? Looking at a single dimension, should a vendor who has completed 100 transactions be weighted twice as much as a vendor who's completed 50, or more, or less?
These are questions that are hard to answer without looking at past data and experimenting with different settings and seeing how they affect things. This is something we'll be doing and sharing with you in this thread.
Another thing I'd like to change is how feedback is displayed on vendor and item pages. Right now it just shows up as it is left in a continuous stream of feedback. Instead I'd like to push the higher quality reviews to the top and filter out the low quality ones. There are a few dimensions I can think of to judge the quality of a review:
buyer weight (as defined above)
amount spent on the purchase
age of feedback
length of review (just thought of this one. not too long not too short?)
Looking at it this way, recent feedback from high quality buyers who've spent good money will be seen first. With the age factor included, no one feedback will stick around forever even if a very high quality buyer spent a lot of money. This will also have the added benefit of obscuring how much business a particular vendor is doing. As it is, one can simply count the feedback being left and get a good estimate of how many sales a vendor is doing.
There are a couple of other changes I have in mind, but this post is already getting long and complex, so let's leave it at that. I look forward to your hearing your feedback.
Show Posts
