7 Messages
High latency node (part deux)
Hello again,
I was once again experiencing a frustrating and inconsistent issue with my backups. I use both Restic and BorgBackup, which use scp/sftp tunnels to backup.
Unfortunately, that means they are sensitive to disconnects and unstable connections. This prompted an entire day of debugging and network testing my LAN, the servers to which I'm backing up, and testing various network configurations.
I have tested many things, including but not limited to:
- Disabling ipv6 and only using ipv4
- Using non DFS channels
- Ratelimiting my upload
- Using 2.4ghz only, not 5 or 6ghz channels
- Factory resetting my modem/router
- Rebooting the modem/router
- Uploading to a BorgBase US server
- Uploading to a BorgBase EU server
- Uploading to a Hetzner DE server
- Uploading to a Hetzner Fin Server
- Changing the upload network buffer
- Scp'ing and rsyncing the files over SSH instead of Borg/Restic
- Altering the SSH keepalive settings (enabled or disabled; changing ServerKeepAlive{,Max} values to recommended settings)
Not to be deterred, I then suspected that my network is fine, but there is a hop somewhere that is causing rapid inconsistent disconnects.
And wouldn't you know it, ye olde node from this thread (https://forums.xfinity.com/conversations/your-home-network/high-latency-node/66ad86a759bf556396a1e032) has reappeared as a high-loss node, consistently.
That thread mentions be-300-arsc1.sacramento.ca.ccal.comcast.net.
I see two extremely high loss nodes (see image)
- be-300-arsc1.sacramento.ca.ccal.comcast.net has ~90% loss on this route
- lag-39.ear3.sjo1.sp.lumen.tech is even worse
Please please please investigate this this time. This does not appear to be an issue on my end.
stephenSRM
7 Messages
4 days ago
If there is some other reason why SCP ssh tunnels would stochastically fail repeatedly, to the point where it is impossible to upload for more than 30 seconds, for 2 different servers in two different countries, I am open to it. But I cannot find any issues within my network itself. I welcome any and all ideas.
1
0
stephenSRM
7 Messages
4 days ago
Ok - I am able to back up successfully to both Borgbase and Hetzner if, and only if, I have changed my MTU from 1500 to a lower value.
I was able to successfully step this MTU value down to 1200, instead of 1000.
1200 is suspiciously close to the minimum ipv6 MTU of 1280.
This makes me wonder whether there is an ipv6 switch at one of these hops that 1) Has the MTU set to the *minimum* of 1280 and 2) ICMP is disabled, so it is unable to communicate the need for fragmentation 3) Therefore dropping packets.
It is also possible that one of the listed hops above is the culprit, because *if* ICMP is disabled somewhere, we would see that loss pattern -- ICMP ping packets not being returned 90+% of the time. Those ICMP packets are the same ones that communicate the need for fragmentation.
(edited)
0
0
XfinityKei
Official Employee
•
1.4K Messages
2 days ago
@stephenSRM Thanks for bringing your connection concerns to our attention. I know having your connection drop is never ideal. Our team is here to help resolve this for you. Can you tell me more about your setup and what's connected? Outside your modem do you have a separate router, splitter, Wi-Fi extenders, amplifier or a power strip connected?
2
0