Xfinity browsing internet impossible in the evenings

For the past few weeks, browing the internet on my Xfinity becomes near impossible in the evenings. Ironically, I am able to watch UHD Netflix just fine at the same time (i.e. if I wanted to). I have a 1Gbps connection and recommended modem.

Things I have tried:

- Connecting various devices directly to the modem via ethernet cable, this doesn't change anything

- Restarting the whole setup

- Things work great during the day with the exact same setup

- The same devices work just fine when connecting using my phone's internet

I am out of ideas.  My theory is that somehow during the evening the video/netflix load is very high in the region, and xfinity prioritizes those streams for good user experience, however this causes other internet traffic to ground to a halt.