Unwanted auto-assertion of DTR & RTS on serial port open

From: Mychaela Falconia <mychaela.falconia_at_gmail.com>
Date: Mon, 23 May 2022 21:38:50 UTC
Hello,

I am not a current FreeBSD user (I use Linux at the moment), but I am
asking here in connection with this recent feature addition to FreeBSD:

https://reviews.freebsd.org/D20031

The main purpose of this inquiry is to find out if this newly added
FreeBSD feature really works as it seems at first glance, or if it is
not quite there yet.

There was a poor design decision made back in 1970s in Original UNIX
in relation to serial port handling, this bad design decision has been
codified in POSIX and other standards, and it persists to this day in
all or most current or "modern" Unix-style OSes.  FreeBSD _may_ have
fixed it recently - but please read onward.  The bad design is as
follows: whenever a serial port is opened under Ancient UNIX or under
any "modern" POSIX-compliant OS, that open syscall right there and
then causes the physical DTR and RTS signals to be asserted, without
giving userspace any opportunity to say "no, please don't do it".  How
is it bad?  Answer: it wreaks havoc on custom special-purpose hardware
in which either or both signals (DTR and/or RTS) have been repurposed
for some non-standard uses.

My specific hardware application is as follows: I have a GSM cellphone
with a built-in USB-serial computer interface (FT2232D, providing two
UART channels behind one USB device), and one of the two UART channels
behind the FT2232D chip (the second one, Channel B going by FTDI chip
doc convention) has its DTR and RTS outputs repurposed.  (The second
UART channel on the GSM baseband chip is data leads only, no modem
control or even hw flow control, hence the signals were otherwise
unused and available for creative repurposing.)  DTR in particular is
wired to drive a superdeep reset to the baseband chipset, useful for
regaining control from runaway code, but obviously not to be asserted
in normal phone usage.

I need to be able to perform the following sequence of operations, and
do so _without_ causing DTR to become asserted, not even for one
microsecond:

1) Connect the USB cable between the phone and my PC or laptop;
2) Open the serial port corresponding to the second UART channel
behind FT2232D;
3) Do some serial communication (send and receive bytes) with the GSM
processor behind that UART channel.

The hardware aspect works just fine: I have pull-up resistors on the
lines running from FT2232D BDBUS2 (RTS) and BDBUS4 (DTR) outputs to my
74LVC2G07 OD buffer, thus no erroneous reset will get triggered until
and unless FT2232D actively drives low on its output - tristate is
fine.  Actual physical hw tests confirm that no erroneous resets happen
on USB plug/unplug operations, with USB VBUS and the host connection
appearing and disappearing.  Instead the problem is software: on Linux
what I seek is currently impossible without applying some kernel
patches, as the action of opening the serial port immediately asserts
DTR and RTS.  When I do apply the necessary patch, everything works
like a charm - but so far I have had no success with getting any of
the necessary patches mainlined, neither my narrow proposal nor a more
generalized patch by another author.

Now my question to this list: how would FreeBSD fare with such custom
hardware?  I don't have a spare machine to install FreeBSD on just for
this test, so I thought I would ask.  If someone plugs an FT2232D (two
UART channels) USB device into a FreeBSD box, one would get ttyU0 and
ttyU1 devices, right?  (And presumably also cuaU0 and cuaU1?)  Reading
the description of D20031 linked above, it appears to me that I would
need to do something like this:

stty -f /dev/ttyU1.init -rtsdtr

and then open /dev/ttyU1 normally to do my desired serial
communication.  But here is the critical question, coming from a
FreeBSD-ignorant person (those ttyXX.init devices don't exist in
Linux): won't the action of opening /dev/ttyU1.init for the purpose of
doing the ioctl (what stty does) in itself cause DTR and RTS to become
asserted briefly?

My hardware cannot tolerate ANY spurious/unwanted assertions of DTR
and RTS, not even for a moment.  I did an oscilloscope experiment: if
I open the serial port via unpatched Linux kernel and then immediately
close it, I see a 4 ms pulse on the o'scope, i.e., DTR goes asserted
for 4 ms.  This 4 ms pulse is certainly long enough to kill a running
phone.

But coming from Linux and having no current experience with FreeBSD, I
don't understand how /dev/ttyXX.init devices work.  I understand their
intended purpose: to set termios flags that will apply as initial
state to subsequent opens of the "regular" tty device.  But what is
the nitty-gritty?  Can one open a ttyXX.init or ttyXX.lock device and
perform actual serial I/O (beyond just termios settings) from that
open file descriptor?  The "normal" actions that happen when a regular
/dev/ttyXX or /dev/cuaXX device is opened, "normal" actions that
include automatic assertion of DTR & RTS - do these implicit actions
still happen when one opens /dev/ttyXX.init instead of /dev/ttyXX, or
are they skipped/suppressed?

Why am I asking these questions despite not being a currently active
FreeBSD user?  For two reasons:

1) If someone out in the world ever wishes to use one of my FreeCalypso
GSM phones with FreeBSD instead of Linux, I would like to support that
prospective user.  In that case it would be important to know if the
recent addition of CNO_RTSDTR termios flag on initial-state devices
already solves the problem in FreeBSD, or if more work is needed to
really fix the problem.

2) At some point I will need to get back to the battle with Linux
kernel gatekeepers^Wmaintainers, to get them to implement/accept *some*
workable fix for the original 1970s design bug - and knowing whether
or not FreeBSD got it right will be an important point for argument.

TIA for any feedback,
Mother Mychaela of FreeCalypso