Xfinity plant
Community Forum
Xfinity globe

Packet loss on some routes - Westland MI

Frequent Visitor

Packet loss on some routes - Westland MI

Hello,

 

Starting at approx. 4PM EDT today, I began to see high packet loss on 3 of 4 connections I'm monitoring. Attached are a couple example RRD graphs from smokeping, showing loss to 8.8.8.8 (Google Public DNS) and a perfectly normal connection to a third party VPN service's IP that appears to be unaffected.

 

Could someone please look into this issue and see if there's anything wrong with connections in Westland, MI? A friend of mine who is connected to the same node is also running smokeping and sees even more loss than I do. I can provide some MTRs if needed as well.

 

Best,

Alan

MullvadNY_last_10800.png
GoogleDNS_last_10800.png

Accepted Solutions
Frequent Visitor

Re: Packet loss on some routes - Westland MI

I’m mobile right now, so no fancy screenshots, but last night at about 10PM EDT, I noticed that the loss cut off, which differed from the tapering off I’d seen on nights prior. I checked things out, and it looks like Comcast added a new hop immediately after the Westland OTN. I haven’t had issues since. So if one of the engineers who got that fixed up is reading this, thank you, and you’re awesome.

All Replies
Frequent Visitor

Re: Packet loss on some routes - Westland MI

I got bored, so here's some MTR deliciousness.

Screen Shot 2018-03-15 at 8.13.37 PM.png
Expert

Re: Packet loss on some routes - Westland MI

If you are using a flavor of Windows, could you run some native Windows OS traces please. And don't trace to any DNS servers as many, if not most of them de-prioritize / rate limit ICMP packet based ping probes.

Frequent Visitor

Re: Packet loss on some routes - Westland MI

Hi EG,

 

Sorry, I don't have any Windows boxes available to test from. As for the trace to Google DNS, I was seeing the same packet loss there as I was on other traces to a couple of my webservers, so I highly doubt their server is simply rejecting ICMP packets. The loss has at this point subsided, but I have a feeling that it will return later today. Please see the attached graphs showing a period of loss to two of my webservers, 1 in Dearborn, MI and 2 in Quebec, and another graph showing loss over the same period to 8.8.8.8.

 

I'm going to be keeping an eye on this throughout the day. Yesterday, the loss began promptly at 4PM. I'm interested to see if the loss shows back up.

 

My friend did contact Comcast support, and was able to speak with a higher level technician, who did indeed see issues on the side of Comcast's network. Unfortunately, the technician also said that until both my friend and I have a truck rolled out to our apartments, Comcast won't look into the issue upstream from us. If you could forward this on to some senior engineers, I'd appreciate it.

 

Thanks 

GoogleDNS.png
Web-Server-Quebec.png
Web-Server-Dearborn.png
New Poster

Re: Packet loss on some routes - Westland MI

I live just a couple miles away from Alan in Westland, MI, and my internet traffic is having the exact same issues.

 

Below is a series of smokeping graphs that were taken from my apartment. The first one is to Google DNS. The second is to the same server based in Montreal, QC, Canada. The third is to a VPN server in NYC. Notice how the packet loss is significantly less to the NYC server.

 

EG - May I ask why a traceroute from a Windows box is preferred? I've always found the MTR tools in Linux-based operating systems to be more detailed, and I don't run any Windows-based systems at all, only Mac and Linux systems. I work for a datacenter and major web hosting company and we utilize MTR to assist with the diagnosis of network and internet issues if they ever arise.

 

I will be adding additional smokeping monitors to my setup to get a better picture of what is going on. I did speak with a Tier 2 Technician with Comcast and he did state that there was some sort of interface failure reported on the CMTS locally in Westland, MI, but was unable to provide me any details on whether there is already a fix planned. Additionally, a CMTS failure doesn't really make sense given the circumstances, it seems that the issues happen when the traffic reaches Chicago (350 Cermak) according to MTRs. The Comcast technician did schedule a technician to come out to my residence, but I'm confident that the issue lies beyond my residence in Comcasts equipment.

Screen Shot 2018-03-16 at 7.56.49 AM.pngScreen Shot 2018-03-16 at 7.57.27 AM.pngScreen Shot 2018-03-16 at 7.58.01 AM.png

New Poster

Re: Packet loss on some routes - Westland MI

Apparently the images aren't displaying using the built-in inline image tool, I've attached the images below

Screen Shot 2018-03-16 at 7.58.01 AM.png
Screen Shot 2018-03-16 at 7.57.27 AM.png
Screen Shot 2018-03-16 at 7.56.49 AM.png
New Poster

Re: Packet loss on some routes - Westland MI

So when I attached the images, it reversed the order. First is NYC VPN, second is Montreal, third is Google DNS.

Expert

Re: Packet loss on some routes - Westland MI

O/k  

Frequent Visitor

Re: Packet loss on some routes - Westland MI

Fantastic! Thank you very much, EG. I look forward to their response.

Expert

Re: Packet loss on some routes - Westland MI

Good luck with it !

Frequent Visitor

Re: Packet loss on some routes - Westland MI

Looks like it's time for an evening update. I took a traceroute this morning to 8.8.8.8, which showed no loss on the way there, or to the IP itself. I've begun to see minor packet loss on the same trace at this time. Looks like hops 4-6 are starting to see high load. Assuming things go the same as they did last night, I should start to see an increase in loss as more traffic starts to flow through those hops. Timing seems to be consistent with when folks would be getting home from school and work. Attached are some screenshots.

 

 

 

 

Screen Shot 2018-03-16 at 7.26.49 AM.png
Screen Shot 2018-03-16 at 5.13.07 PM.png
Frequent Visitor

Re: Packet loss on some routes - Westland MI

Attached are some more smokeping graphs. Looks like it's now prime time for packet loss. Notice how some tests, like to comcast.com and a VPN server in New York, see little to no packet loss. The issue continues to be present on only some routes. I can upload MTRs to each of these addresses if requested.

Screen Shot 2018-03-16 at 9.52.11 PM.png
Screen Shot 2018-03-16 at 9.50.26 PM.png
Screen Shot 2018-03-16 at 9.50.19 PM.png
Screen Shot 2018-03-16 at 9.50.11 PM.png
Screen Shot 2018-03-16 at 9.50.04 PM.png
Screen Shot 2018-03-16 at 9.49.55 PM.png
Screen Shot 2018-03-16 at 9.49.46 PM.png
Screen Shot 2018-03-16 at 9.49.30 PM.png
New Poster

Re: Packet loss on some routes - Westland MI

Recently Experiencing up to 22% peak loss while gamming using my 100mbs Xfinity. (Southwest michigan based)
Frequent Visitor

Re: Packet loss on some routes - Westland MI

I’m mobile right now, so no fancy screenshots, but last night at about 10PM EDT, I noticed that the loss cut off, which differed from the tapering off I’d seen on nights prior. I checked things out, and it looks like Comcast added a new hop immediately after the Westland OTN. I haven’t had issues since. So if one of the engineers who got that fixed up is reading this, thank you, and you’re awesome.