Large SCP transfers blocked

I am finding that large file transfers (about 1.9GB in this case) via scp (secure copy using ssh) fail between my system and multiple servers at various sites from my Comcast internet. When I take the laptop to my other home where I have Verizon FIOS, scp of this size file works first time, every time (I tested to several different remote servers at different locations).

Is there a problem with Comast reliability or is the system intentionally injecting bad packets or dropping data to prevent this sort of use of consumer internet?

Responses

XfinityRay

Official Employee

•

2.7K Messages

3 years ago

Hello, @rboudrie. I am sorry to hear that you are seeing a problem when transferring large files. We have a reliable network and want you to be able to use the service without problems. Did you run a traceroute to see where the file is getting hung up?

0

R

rboudrie

Visitor

•

4 Messages

3 years ago

I'll give it another try when I am at the alternate residence where I have Comcast. I did not run traceroute, but the file did start the transfer upload severanl hundred MB before getting the error, and did so with repeated attempts (stopping at a different point each time). Since it stared without error, and I am also able to open an SSH connection in a command line windows (via putty) without problem, I don't expect traceroute to do anything other than show me a working path.

I know Comcast used to use Sandvine to inject bad packets to kill large transfers - are you certain that it, or a similar product, is not running on your network?

(edited)

7

0

CCBrandon1

Problem Solver

•

770 Messages

Thank you for the additional info, and sharing your experience here. I am not seeing this listed as something we would limit based on the Acceptable Use Policy for Xfinity Internet.

I did find a prior thread, with some info that may help your case here. However, please let us know what you experience when you return to your other home, we would love to help get this resolved!

I no longer work for Comcast.

3 years ago

0

C

coreyt429

Visitor

•

3 Messages

I've been struggling with this same issue, and can add some more information though a traceroute likely won't help you as it would go over my VPN, though if it helps fix the issue, I can spin up an AWS server and do some tests.

What I do know is, this mostly seems to be a wifi issue, and it seems to kill all TCP sessions to/from the same IP at least in my experience, I can test further to see what happens with different IP's.

Here is my experience to give you an idea.

I have 4 ssh sessions opened to 4 different servers (same VPN, though, so same IP flow from the router's perspective).

When I try to scp a large file to one of the servers, it runs for a random amount of time then gets connection closed. At the same time, the other 4 ssh sessions also get connection closed.

That is when connected via wifi.

When wired, the "connection closed" issue does not occur as often, but still can occur.

When tethered to by phone to bypass the xfinity router, it never occurs.

Also of note, from watching packet captures of this behavior, it seems that the connection is not being closed in the other direction, the remote server continues to retransmit the last TCP message until it times out.

3 years ago

0

C

coreyt429

Visitor

•

3 Messages

Here is an example traceroute and scp session: (Note, this did not kill ssh session to other servers, so that behavior was likely just killing every session to the vpn address)

PS C:\Users\coreyt429\Downloads> tracert ec2-18-217-195-167.us-east-2.compute.amazonaws.com

Tracing route to ec2-18-217-195-167.us-east-2.compute.amazonaws.com [18.217.195.167]
over a maximum of 30 hops:

2 10 ms 14 ms 10 ms 100.92.156.3
3 11 ms 10 ms 15 ms 68.85.205.141
4 13 ms 10 ms 14 ms po-200-xar02.tuscaloosa.al.tusca.comcast.net [162.151.22.57]
5 18 ms 16 ms 20 ms 68.85.227.73
6 31 ms 32 ms 32 ms ae-100-ar01.b0atlanta.ga.atlanta.comcast.net [68.85.109.21]
7 24 ms * * 162.151.96.105
8 30 ms * 27 ms be-33011-cs01.56marietta.ga.ibone.comcast.net [96.110.43.65]
9 24 ms 24 ms 27 ms be-2104-pe04.56marietta.ga.ibone.comcast.net [96.110.37.98]
10 28 ms 23 ms 23 ms 50.208.232.30
11 34 ms 24 ms 24 ms 54.239.105.95
12 * * * Request timed out.
13 * * * Request timed out.
14 49 ms 48 ms 48 ms 52.93.239.97
15 89 ms 51 ms 55 ms 52.95.2.107
16 47 ms 48 ms 47 ms 52.95.2.106
17 * * * Request timed out.
18 51 ms 62 ms 48 ms 15.230.134.143
19 57 ms 51 ms 49 ms 108.166.252.39
20 * * * Request timed out.
21 54 ms 47 ms 47 ms 108.166.252.35
22 * * * Request timed out.
23 * * * Request timed out.
24 * * * Request timed out.
25 * * * Request timed out.
26 * * * Request timed out.
27 * * * Request timed out.
28 * * * Request timed out.
29 * * * Request timed out.
30 * * * Request timed out.

Trace complete.
PS C:\Users\coreyt429\Downloads> scp .\PB.as.22.0.1123.pb20220501.Linux-x86_64.zip ec2-user@ec2-18-217-195-167.us-east-2.compute.amazonaws.com:
Warning: Permanently added 'ec2-18-217-195-167.us-east-2.compute.amazonaws.com,18.217.195.167' (ECDSA) to the list of known hosts.
PB.as.22.0.1123.pb20220501.Linux-x86_64.zip 12% 17MB 3.2MB/s 00:35 ETAclient_loop: send disconnect: Connection reset
./PB.as.22.0.1123.pb20220501.Linux-x86_64.zip: Broken pipe
PS C:\Users\coreyt429\Downloads>

3 years ago

0

R

rboudrie

Visitor

•

4 Messages

@coreyt429 Wifi works with these large transfers from my other home, however, I will try it hardwired the next time I am at the location with Comcast internet.

3 years ago

0

CCJennifer

Problem Solver

•

743 Messages

Hello @coreyt429! We appreciate the extra details on the issue you're experiencing sending files. Have you had the chance to try other IP addresses yet, or have you had this issue outside of VPN at all? Also, have any settings been altered to connect to your VPN?

I no longer work for Comcast.

3 years ago

0

GameDr04

Visitor

•

1 Message

3 years ago

I have sunk an inordinate amount of time into this issue, myself...to an obsessive level.

I'll spare you most of the gruesome details of the journey.

TL;DR: I can confirm that SSH-based uploads of large files are indeed being shuttered by Comcast. Download/inward transfers are unaffected.

I have transferred a single 20 gigabyte file:

• to OneDrive via the web interface with zero interruptions (multiple times)

• to a remote server via FTP with zero interruptions (multiple times)

• to a WebDAV server (port 80) with zero interruptions (multiple times)

As soon as I try to copy that file via scp, rsync, sftp, or even tar-tar, the transfer is guaranteed to be interrupted.

I even changed one of my remote servers' SSH listening port to 443 (SSL) and the connection was still interrupted, so we know it's not port-based.

All of these tests were run from

• my home (Xfinity)

• my sister's home (Xfinity)

• four different friends' homes (Xfinity, Xfinity, AT&T Fiber, Verizon FiOS)

Xfinity connections fail every time. AT&T and Verizon succeed every time.

I've done all of these things with and without routers in the chain (my friends are real troopers), spread across 13 of my own servers at seven different locations on Earth, and the results are always the same.

I can confirm that the Xfinity connection is being flooded with packets flagged with RST (reset), none of which originate from either endpoint, milliseconds before the time of death of the connection.

Whether or not packet sniffers are perfect, the number of connection killing packets sometimes amounts to whole megabytes of data that neither endpoint can account for.

When I open a VPN (either Nord or a friend across the country with her own OpenVPN server) and run an SSH-based transfer, it is guaranteed to succeed. All packets respect the DSCP class configured at the endpoints (AF11 and AF22, in this case).

Without a VPN, outbound packets have their DSCP class replaced with CS1.

Comcast is absolutely shaping traffic.

That, in and of itself, is 100% understandable. They have a huge network to manage and what good would it be if one hog ruined it for everybody?

I genuinely don't blame them for that.

What I don't understand is why they straight-up kill the connection. Why not throttle it?

And why SSH? Is it because BitTorrent started using SSH as a "bypass" for the filters?

Most importantly: why now? It's worked for me for years, until very recently. Was there a change made to the whole network? If so, was it intentional? If not, how can we get it fixed? Is there a network engineer at Comcast/Xfinity to whom we can speak about this?

I'd be annoyed to find out that it was intentional but at least we'd all have closure.

I have considered switching to Comcast Business because of this except I don't want to be force-fed a modem.

To anybody else having this issue: get a VPN or use a while[1] script (see below) for all your rsync needs. The CRC doesn't get interrupted so the transfer will complete eventually.

Hopefully Comcast can offer some non-hacky resolution, someday.

#!/bin/bash
while [ 1 ]; do
    rsync -rvPt SOURCE DESTINATION:/path
    if [ "$?" = "0" ]; then
        exit;
    else
        echo "RETRYING"
        sleep 10
    fi
done

1

tusc00

Regular Visitor

•

2 Messages

3 years ago

If you can install tailscale at both endpoints, you might be interested in using it to encapsulate traffic using the wireguard protocol. They recently released support for SSH authentication as well: https://tailscale.com/blog/tailscale-ssh/

0

Your Home Network

rboudrie

Large SCP transfers blocked

XfinityRay

rboudrie

CCBrandon1

coreyt429

coreyt429

rboudrie

CCJennifer

GameDr04

tusc00

New to the Community?