Remote voice control fails about every hour

My remote control works fine after a system refresh or pairing. After about an hour the button needs to be pushed at least four times before I can give a command. Xfinity sent me a new remote that fails the same way.