DevHeads.net

Problem with ssh disconnecting

Running CentOS 7 on workstation and having a problem with ssh disconnects. My ssh_config contains:

Host *
TCPKeepAlive yes
ServerAliveInterval 30
ServerAliveCountMax 300

and sshd_config on the server contains:

TCPKeepAlive yes
ClientAliveInterval 60
ClientAliveCountMax 300

Have I missed any setting needed to prevent these random disconnects? I don't think there is anything wrong with the network card, the driver, or the cable, since if I am on a VPN connection via another server, the VPN and any ssh connection stay up indefinitely.

Thanks.

Comments

Re: Problem with ssh disconnecting

By Liam O'Toole at 02/12/2018 - 20:24

On 2018-02-12, H < ... at meddatainc dot com> wrote:
Another poster has provided some possible reasons for the
disconnections. Whatever the cause, autossh (from the epel repo) is a
good workaround.

Re: Problem with ssh disconnecting

By Hakan at 02/12/2018 - 21:13

On 02/12/2018 07:24 PM, Liam O'Toole wrote:

Re: Problem with ssh disconnecting

By Liam O'Toole at 02/13/2018 - 06:54

On 2018-02-13, H < ... at meddatainc dot com> wrote:
In that case sftp with the '-a' option would be a better choice.

Re: Problem with ssh disconnecting

By Kay Diederichs at 02/15/2018 - 08:42

On 02/13/2018 11:54 AM, Liam O'Toole wrote:
or rsync --partial

Re: Problem with ssh disconnecting

By Earl Ramirez at 02/12/2018 - 22:12

On Mon, 2018-02-12 at 20:13 -0500, H wrote:
I don't know if this would help but I had a similar issue and it turned
out that there was a custom script in /etc/profile.d/ that contain
TMOUT 900.

You can also check in /etc/profile, usually, the security logs has
something about the disconnects, of you can use wireshark or a similar
tool to capture and analyse the packets.

Re: Problem with ssh disconnecting

By Hakan at 02/19/2018 - 22:05

On 02/12/2018 09:12 PM, Earl Ramirez wrote:
I think that I forgot to mention that the problem is not the common "write failed: broken pipe" but "packet_write_wait: broken pipe".

I don't think the problem is router related since another computer, albeit running CentOS 6, does not have similar issues.

Does the information above suggest any other reason for my problem?

Re: Problem with ssh disconnecting

By Stephen John Smoogen at 02/12/2018 - 19:34

There are usually 2 different reasons for this:
1. The VPN is UDP and times out/drops keeps alives so that they no longer
function properly. [The UDP connection will make it look like you have a
new SSH connection which of course the system will drop because that would
allow for security problems.]

2. A firewall in the chain of things (system you are on, the system you are
going to, or somewhere in between) has session flushing issues. If you have
the firewall set up to only accept NEW port 22 connections and then just
looks to see if the ESTABLISHED, RELATED tables are accepted elsewhere then
if the session somehow ages out or is flushed due to usage, the ssh
connection can get dropped.

The solution to one is to see if a TCP VPN fixes the problem. The second
one is to either make the iptables kernel tables larger or to have all port
22 accepted even if it is not ESTABLISHED.

These aren’t the only ways the problem you see can occur but they are some
of the most common I have run into.

Re: Problem with ssh disconnecting

By Hakan at 02/12/2018 - 21:05

On 02/12/2018 06:34 PM, Stephen John Smoogen wrote:
I should also explain that:

- I am on a workstation (located in the US), ssh-ing into server 1 (located in the US).

- From server 1 I use scp to transfer large files from server 2 (located in Europe) to server 1 (in the US).

The above randomly disconnects.

However, when:

- I use a VPN connection to server 3 (also located in Europe).

- From the same workstation as above, do exactly as above, connections are rock-solid.

Re: Problem with ssh disconnecting

By Marcelo Ricardo... at 02/19/2018 - 23:09

On Mon, Feb 12, 2018 at 08:05:18PM -0500, H wrote:
...
You have two ssh sessions here. AFAICT, it's the first one
(workstation->server 1) that randomly disconnects, right?

It would seem to me that something between your workstation and server
1 is possibly mishandling TCP options (being it sack, timestamps or
both). Is there a router or a firewall in between?

Try doing a traffic capture between both and see why it hangs. If
you're hesitant to post the binary traffic capture, post the text
version of it with anonymized IP addresses. Just please be sure to
disable ssh protocol so it would include TCP details on it then.

Marcelo

Re: Problem with ssh disconnecting

By Hakan at 02/20/2018 - 10:33

On 02/19/2018 10:09 PM, Marcelo Ricardo Leitner wrote:
Let's reduce the problem to a simpler case: the workstation running CentOS 7 is behind a router, behind the same router is the server running CentOS 6. For various reasons, I access the server via its external IP address, not the local one. Access is via ssh.

When using straight ssh, the connection times out after random times, always less than 20 minutes or so, though, with the error message "packet_write_wait: broken pipe", not the common "write failed: broken pipe".

However, when I am in a ssh session on a VPN connection (which terminates in an outside server far away from my network), the ssh connection does /not/ time out.

Further, accessing the same server as first mentioned, in the identical fashion, from another computer running CentOS 6, I never encounter the "packet_write_wait" error and disconnection.

My conclusion is thus that this is not a router issue but could conceivable be a bug in CentOS 7. I am also ruling out a cable issue or a physical network card issue since the VPN connection is stable.

Re: Problem with ssh disconnecting

By Hakan at 02/21/2018 - 20:19

On 02/20/2018 09:33 AM, H wrote:
No messages in secure or message logs.