Download latency issues

I'm having significant download latency issues on my direct Ethernet wired PC which is now over 60 milliseconds, and even worse on Wi-Fi over my phone which exceeds 200 milliseconds. This is beginning to cause problems with a couple of my apps and I am desperate to find a solution.