Latency spikes on Xbox One

The last 6 months I've been experiencing high latency while gaming online with my Xbox One. It will jump anywhere between 40mbps up to 170mbps. I've had Comcast technicians come out to my house several times only to tell me there's nothing wrong with it. I've tried port forwarding and it didn't help. I'm currently using the XB7 and I've replaced the modem several times with a new one. Any suggestions on to what could be my problem.