Slow Internet Speeds Using Remote Desktop

I've been attempting to use the remote desktop application on a laptop while at home connected to my xfinity Wi-Fi. Once I am in the remote connection I experience very slow interent speeds with all of my applications (Chrome, Outlook, etc.) . I experience better speeds during non-peak hours of the day but anytime from 8am - 5pm it's always very slow. Are there any weird settings with xfinity that may be causing this or something else? 

I'm using an ethernet connection to my gateway router. I'm also using a VPN connection through my company. I have the 500 Mbps plan with xfinity. 

Please let me know if anyone has a solution to this problem. 

Thanks,

Alex