Starting at approx. 4PM EDT today, I began to see high packet loss on 3 of 4 connections I'm monitoring. Attached are a couple example RRD graphs from smokeping, showing loss to 184.108.40.206 (Google Public DNS) and a perfectly normal connection to a third party VPN service's IP that appears to be unaffected.
Could someone please look into this issue and see if there's anything wrong with connections in Westland, MI? A friend of mine who is connected to the same node is also running smokeping and sees even more loss than I do. I can provide some MTRs if needed as well.
Solved! Go to Solution.
If you are using a flavor of Windows, could you run some native Windows OS traces please. And don't trace to any DNS servers as many, if not most of them de-prioritize / rate limit ICMP packet based ping probes.
Sorry, I don't have any Windows boxes available to test from. As for the trace to Google DNS, I was seeing the same packet loss there as I was on other traces to a couple of my webservers, so I highly doubt their server is simply rejecting ICMP packets. The loss has at this point subsided, but I have a feeling that it will return later today. Please see the attached graphs showing a period of loss to two of my webservers, 1 in Dearborn, MI and 2 in Quebec, and another graph showing loss over the same period to 220.127.116.11.
I'm going to be keeping an eye on this throughout the day. Yesterday, the loss began promptly at 4PM. I'm interested to see if the loss shows back up.
My friend did contact Comcast support, and was able to speak with a higher level technician, who did indeed see issues on the side of Comcast's network. Unfortunately, the technician also said that until both my friend and I have a truck rolled out to our apartments, Comcast won't look into the issue upstream from us. If you could forward this on to some senior engineers, I'd appreciate it.
I live just a couple miles away from Alan in Westland, MI, and my internet traffic is having the exact same issues.
Below is a series of smokeping graphs that were taken from my apartment. The first one is to Google DNS. The second is to the same server based in Montreal, QC, Canada. The third is to a VPN server in NYC. Notice how the packet loss is significantly less to the NYC server.
EG - May I ask why a traceroute from a Windows box is preferred? I've always found the MTR tools in Linux-based operating systems to be more detailed, and I don't run any Windows-based systems at all, only Mac and Linux systems. I work for a datacenter and major web hosting company and we utilize MTR to assist with the diagnosis of network and internet issues if they ever arise.
I will be adding additional smokeping monitors to my setup to get a better picture of what is going on. I did speak with a Tier 2 Technician with Comcast and he did state that there was some sort of interface failure reported on the CMTS locally in Westland, MI, but was unable to provide me any details on whether there is already a fix planned. Additionally, a CMTS failure doesn't really make sense given the circumstances, it seems that the issues happen when the traffic reaches Chicago (350 Cermak) according to MTRs. The Comcast technician did schedule a technician to come out to my residence, but I'm confident that the issue lies beyond my residence in Comcasts equipment.
Looks like it's time for an evening update. I took a traceroute this morning to 18.104.22.168, which showed no loss on the way there, or to the IP itself. I've begun to see minor packet loss on the same trace at this time. Looks like hops 4-6 are starting to see high load. Assuming things go the same as they did last night, I should start to see an increase in loss as more traffic starts to flow through those hops. Timing seems to be consistent with when folks would be getting home from school and work. Attached are some screenshots.
Attached are some more smokeping graphs. Looks like it's now prime time for packet loss. Notice how some tests, like to comcast.com and a VPN server in New York, see little to no packet loss. The issue continues to be present on only some routes. I can upload MTRs to each of these addresses if requested.