Re: USB-serial adapter suggestions needed

On Jan 10, 2024, at 15:55, bob prohaska <fbsd@www.zefox.net> wrote:
> 
> On Wed, Jan 10, 2024 at 11:30:28AM -0800, Mark Millard wrote:
>> On Jan 10, 2024, at 11:21, Mark Millard <marklmi@yahoo.com> wrote:
>> 
>>> On Jan 10, 2024, at 10:16, bob prohaska <fbsd@www.zefox.net> wrote:
>>> 
>>>> On Tue, Jan 09, 2024 at 05:03:42PM -0800, Mark Millard wrote:
>>>>> . . .
>>>> 
>>>> In general the tip session is used only for observation or
>>>> troubleshooting. Ssh connections are used for other activity, 
>>>> including OS build/install cycles, poudriere, etc. They are
>>>> usually placed in the background, writing to log files so that
>>>> accidental disconnects from the workstation don't stop them.
>>> 
>>> Are you using:
>>> 
>>> NAME
>>>    nohup ??? invoke a utility immune to hangups
>>> 
> 
> The OS build commands end with &, which I gather does nohup 
> among other things (not sure what).  

"&" creates background processes that are still killed when
their parent tty or controlling process goes away. nohup
avoids that kill.

man signal reports:

     Num   Name             Default Action       Description
     1     SIGHUP           terminate process    terminal line hangup

https://en.wikipedia.org/wiki/SIGHUP describes it via:

QUOTE
With the decline of access via serial line, the meaning of SIGHUP has changed somewhat on modern systems, often meaning a controlling pseudo or virtual terminal has been closed. If a command is executed inside a terminal window and the terminal window is closed while the command process is still running, it receives SIGHUP.[1]

If the process receiving SIGHUP is a Unix shell, then as part of job control it will often intercept the signal and ensure that all stopped processes are continued before sending the signal to child processes (more precisely, process groups, represented internally by the shell as a "job"), which by default terminates them.[2]

. . . Firstly, the Single UNIX Specification describes a shell utility called nohup, which can be used as a wrapper to start a program and make it ignore SIGHUP by default. . . .
END QUOTE

(I omitted an alternate way that does not apply to FreeBSD.)

nohup avoids using that "which by default terminates them" status.

But I'll also note that:

https://www.gnu.org/software/libc/manual/html_node/Operation-Error-Signals.html

reprots:

QUOTE
Macro: int SIGPIPE

Broken pipe. If you use pipes or FIFOs, you have to design your application so that one process opens the pipe for reading before another starts writing. If the reading process never starts, or terminates unexpectedly, writing to the pipe or FIFO raises a SIGPIPE signal. If SIGPIPE is blocked, handled or ignored, the offending call fails with EPIPE instead.

Pipes and FIFO special files are discussed in more detail in Pipes and FIFOs.

Another cause of SIGPIPE is when you try to output to a socket that isn’t connected. See Sending Data.
END QUOTE


>>> That only ssh sessions that in turn run tip fail suggests that the
>>> tip session gets the initial problem and then things propagate. I
>>> want more than a suggestion. For example: direct tip runs that
>>> are not in any ssh session: still get some form of failures?
>> 
> 
> As it happens, the tip connection between nemesis.zefox.com (terminal
> server) and ns2.zefox.net (console host) dropped. The display reported:
> 
> login: Jan 10 09:42:46 ns2 sshd[48003]: error: Fssh_kex_exchange_identification: Connection closed by remote host
> Jan 10 10:51:54 ns2 sshd[48135]: error: PAM: Authentication error for illegal user shell from 185.11.61.234
> Jan 10 10:53:03 ns2 sshd[48138]: error: Fssh_kex_exchange_identification: Connection closed by remote host
> Jan 10 10:55:47 ns2 sshd[48152]: error: Fssh_kex_exchange_identification: Connection closed by remote host
> Jan 10 11:22:26 ns2 sshd[48203]: error: PAM: Authentication error for illegal user test from 85.209.11.226
> Jan 10 12:24:21 ns2 sshd[48300]: error: Fssh_kex_exchange_identification: Connection closed by remote host
> Jan 10 13:12:06 ns2 sshd[48380]: error: Fssh_kex_exchange_identification: Connection closed by remote host
> Jan 10 13:14:06 ns2 sshd[48381]: fatal: Timeout before authentication for 59.56.110.106 port 50300
> Jan 10 13:34:34 ns2 sshd[926]: error: beginning MaxStartups throttling
> Jan 10 14:12:41 ns2 sshd[48506]: error: PAM: Authentication error for illegal user shutdown from 185.11.61.234
> 
> 
> FreeBSD/arm (ns2.zefox.net) (ttyu0)
> 
> login: client_loop: send disconnect: Broken pipe

Note: "client_loop" is a function in:

/usr/main-src/crypto/openssh/clientloop.c

"send disconnect" is reported after a signal stops the
looping:

        /* Terminate the session. */
        
        /* Stop watching for window change. */
        ssh_signal(SIGWINCH, SIG_DFL);
         
        if ((r = sshpkt_start(ssh, SSH2_MSG_DISCONNECT)) != 0 ||
            (r = sshpkt_put_u32(ssh, SSH2_DISCONNECT_BY_APPLICATION)) != 0 ||
            (r = sshpkt_put_cstring(ssh, "disconnected by user")) != 0 ||
            (r = sshpkt_put_cstring(ssh, "")) != 0 ||   /* language tag */
            (r = sshpkt_send(ssh)) != 0 ||
            (r = ssh_packet_write_wait(ssh)) != 0)
                fatal_fr(r, "send disconnect");

Note that the code just above is executing on "pi4 RasPiOS workstation"
trying to write to the pipe involved --but it ends up reporting the
"send disconnect".

fatal_fr uses ssh_err(...), which in turn can use strerror(...) that
can get the text "Broken pipe" (SIGPIPE).

The "Broken pipe" indicates the type of signal that stopped
client_loop, indicating that on nemesis.zefox.com there was
no longer any process around set up to read the data from
the pipe.

What was running on nemesis.zefox.com was its side of the
ssh that in turn was attached to the shell that in turn
was running tip --until those exited/were-killed on
nemesis.zefox.com .

But there is no information here about which of those was the
one to start the failure on nemesis.zefox.com :

A) Was it the tip process?
B) Was it the shell process?
C) Was it the nemesis.zefox.com side of the ssh?

We only know that the end result was lack of anything
reading the pipe on nemesis.zefox.com : by then
the ssh side on nemesis.zefox.com had stopped being
set up to read the pipe.

> bob@raspberrypi:~ $ 
> 
> Interestingly, there's a MaxStartups warning. I discovered the failure
> somewhat before 15:17 hours. The link had been up for a couple of
> days IIRC.
> 
> I logged in on nemsis.zefox.com's hdmi console and usb keyboard (the only Pi
> in my collection set up for video console logins), opened x11, started an
> xterm, su'd to root and ran tip to connect to ns2.zefox.net.

Did you then look at /var/logs/messages on ns2.zefox.net ? What,
if anything, did such have from around the failure time frame?

Did you look at /var/logs/messages on nemsis.zefox.com ? What,
if anything, did such have from around the failure time frame?

Did you look at /var/logs/messages [or the analogous linux
place(s)] on "pi4 RasPiOS workstation"?  What, if anything, did
such have from around the failure time frame?

I'll note that /var/logs/messages (or the like) need not be an
exact match to what the console reports.

> The connection was normal, no problem with needing to power-cycle the 
> usb-serial adapter using usbconfig; tip connected and displayed a brief
> string of mixed printable and non-printing characters, ending with a
> login prompt on the same line. Next came two reports from ns2 of 
> "... kex protocol error...."  
> 
> 
> 
> 
> 
>>>            Specifies the maximum number of concurrent unauthenticated
>>>            connections to the SSH daemon.  Additional connections will be
>>>            dropped until authentication succeeds or the LoginGraceTime
>>>            expires for a connection.  The default is 10:30:100.
>>> 
>>>            Alternatively, random early drop can be enabled by specifying the
>>>            three colon separated values start:rate:full (e.g. "10:30:60").
>>>            sshd(8) will refuse connection attempts with a probability of
>>>            rate/100 (30%) if there are currently start (10) unauthenticated
>>>            connections.  The probability increases linearly and all
>>>            connection attempts are refused if the number of unauthenticated
>>>            connections reaches full (60).
> 
> If sshd mixed up new with existing sessions it might kill one running tip,
> but it'd likely kill others from time to time. So far that hasn't happend,
> at least not often.
> 
>>> 
>>> It does suggest that testing isolated from the source(s) of
>>> unauthenticated sessions could be worth while in case handling
>>> the load from such sessions when already heavily loaded with
>>> buildworld/builkernel or the like leads to other problems (and
>>> denial of service consequences?).
>>> 
>>> I do not expect that this issue is all that likely but
>>> expectations are not evidence of their own accuracy/inaccuracy.
> 
> It's my (very) subjective impression the ssh/tip problems are more
> frequent when ssh attacks are more abundant. Nemesis.zefox.com is
> public and sees a certain amount of doorknocking but for some 
> reason considerably less than www.zefox.org.  
> 
> As an aside, during some stages of testing with pelorus.zefox.org
> the host was on the LAN and so not subject to attack. It certainly
> didn't prevent failure of tip sessions. Now pelorus.zefox.org is
> on a public IP and seeing at least sporadic doorknocking.


===
Mark Millard
marklmi at yahoo.com