Rise in latency every 10-12 minutes

Every 10-12 minutes our internet's latency spikes upwards towards 2000ms. We are using the XB6. Have reset it numerous times. Changed cords and even tried different cable outlets. Still the same result. Internet speed is fine but when it comes to gaming it is very difficult. I even turned off the wifi and left only my xbox and laptop connected via ethernet to make sure we weren't overwhelming it. Still happened. This started happening when we move into a new house back in July. We kept the same equipment and plan at the old house but this was never an issue.