You are right actually, I was thinking if the domain is dkn255hz262ypmii.onion.to that it would check onion.to/robots.txt but it takes the subdomain into consideration so it would load http://dkn255hz262ypmii.onion.to/robots.txt /me embarrassed