Nah, the attacker would have to be watching the other end of your Tor circuit, either at the exit node, or at specific points between you and a hidden service, namely the hidden service directories, intro points, or the hidden service's entry guards. If an attacker is merely a local observer of all your home connections, that tells him nothing about what you are doing on the other end of a Tor circuit. Well, there are about 3 million people worldwide who use Tor every month for dozens of reasons, and about 250,000 in the United States, so flagging all of them would be pretty useless, which is why I don't think anyone is going on fishing expeditions against Tor users.