Unloaded and loaded download latency extremely different.

I'm getting ~10 ms unloaded and 150-250 ms loaded latency. I've tried restarting my modem/router. I've tried shutting down and disconnecting other electronic devices on my network. 

This is making it very difficult to maintain connection during videoconferences via Zoom and also while playing online videogames. 

Please assist.