horrendous latency issues (peak of 8 seconds to google.com) - need help

I have a 200mbps plan, and I have been experiencing some terrible latency when using *WIRED* connections. I ran mtr over a period of several hours pinging google.com, and observed that:

1. The latency to the router is around 2ms

2. The latency to the next hop after the modem is ... 4-8 SECONDS... I reset both modem/router and got the same issue. This happened today again (it has happened in multiple days). The standard deviation for latency during the day is around 1 second these days, which is unacceptable. 

3. The packet drop rate is around 0.1% across the entire day (I guess this has happened multiple times during my meetings).

4. Throughput is not alwyas an issue but sometimes it is. I saw it go from 200mbps to .... 500kbps sometimes. This is not repeatable, though, and it's not always (though it does coincide sometimes) at the same time as the latency spikes.

For reference, I have a Arris SB8200 modem, and tried TWO routers which didn't seem to matter. One was Asus rt-ax88u, and the other was a Netgear Nighthawk X4 R7500.

I have another house nearby to where this is happening (~1 mile away), and I am NOT seeing these issue there with the same equipment.

It's driving me nuts because when I am in a video call, SOMETIMES I will get hit by the 8 second latency, and at that time it feels like I am talking to someone from the moon due to the insane latency.

What is mega-frustrating is that I got to speak to some Xfinity specialist on the phone previously about this and the summary of the conversation was that I have a problem with "milliseconds". As an outcome of that a tech came by and mentioned that maybe the issue is happening is with the node to which the modem is connected to and that he would open a ticket, but I haven't heard back.

I am getting desperate because due to COVID-19 I am forced to work from home, and I am in meetings >75% of the day, and currently with the connection I have it's getting really really uncomfortable to meet anyone (dropped audio/super-huge lag as I described above).