Xfinity Remote Volume Control Needs Time To Warm Up?

I've had Xfinity for some time.  

But, every time I turn on the TV with the Xfinity remote, we have to wait a good 2-3 minutes before we are able to adjust the volume with the remote.

All of the other features work immediately(channel changing, DVR access, etc), but the volume never works immediately.

Is there a fix for this?