Omega slow internet throughout most of the day

Currently paying for maximum speed of 2GB.
Xfinity XB8 Modem

But even with only 1 device connected, the download speeds are always less than 50Mbps, while the upload speed is way higher. Only time the I've seen higher than 100Mbps downloads is for 1 hour when I first installed it or during the hours 12AM to 6AM (CST). The other 18 hours throughout the day is literally terrible. 

And most of the time, my connection is through ethernet (which is Cat6 with 1GB transfer). I even personally turned down my transfer rate from 1GB to 100Mpbs, and the results are still >50Mpbs.

Like why am I paying for 2GB if even combined (me and my roommates) won't ever be able to reach that? Like it's terrible with even just ONE device connected.

Edit: 
Also, am I not allowed to play games? Like I'm actively seen my downloads speeds getting lowered as soon as I connect to a Minecraft server. Literally goes from 40Mpbs to fukin 15-25Mpbs. Should've listened to my friends when they told me Xfinity sucked balls.