I have 6 meg download speed and I never see rebuffering (I used to with a slower stream) but I haven't seen it for months and months.
You should confirm your ISP speeds and then look through the many threads on this forum on this subject. There is discussion on how to trace routes, what it means, how to address/understand the issues.
Again, I actually have recently used 3 different streams. Uverse at 6 meg, ATT DSL at 2.6 or so, and Time Warner (Hawaii) at 12meg. I watch almost daily and, again, havent seen any buffering issues for at least 6 months. The service has been amazing stable and totally satisfactory.