Unless you have some kind of really weird problem, keep it. The number of IPv6 sites is very small, but there are some and it will let you get to them even if you're at an IPv4 only location.
If it is causing you a problem, it's best to fix it. I've seen a number of people recommending removing it to solve problems. However, they're not actually solving the root cause of the issue. In all the cases I've seen, removing Teredo just happens to cause a side-effect that fixes their problem... :)