latency spikes during peak hour

Hello,

Ever since I moved to our new construction home 2 yrs ago, I've been having latency spikes and packet losses every day during peak hours (4pm-10pm). I've been monitoring the latency for a few days and it seems like the pattern is very consistent, with little to no latency spike during the rest of the day. It's a single family home and I'm the only user of the internet. The computer is running on ethernet cable, and I've been getting the same result when connecting directly to the modem. Any suggestion?