The scary thing is that there aren't many people searching for asteroids that could be headed for Earth. And even if we find that one is headed for Earth, we don't have the technology to do anything about it. Trying to destroy the asteroid could make matters worse since shooting something at it to break the asteroid into smaller pieces could cause more destruction than anything. So what are we to do?
The answer is that their's nothing I can do about it, so there's no point in worrying about it, especially since it probably won't happen in yours or my lifetime anyway.