openbgpds not talking each other since 8.2-STABLE upgrade

[J David]

To help understand what's going on and test some of this stuff, I
hacked up a TCP-MD5-aware echo server and tried various things.

The first thing I found was that setting
net.inet.tcp.signature_verify_input to 0 does not stop the listener
socket from setting TCP_MD5SIG.  So, setting this is not a way to
"mask" the availability of TCP signatures if the kernel supports them.

Ultimately, the behavior of connections depends on four factors:
- the server's net.inet.tcp.signature_verify_input setting
- the client's net.inet.tcp.signature_verify_input setting
- whether the server socket sets TCP_MD5SIG
- whether the client socket sets TCP_MD5SIG

(In all test cases the correct SA's are present.)

Unfortunately, this creates sixteen test cases, so it took some time
to try them all.

1. sverify = 0, cverify = 0, smd5 = n, cmd5 = n
5. sverify = 0, cverify = 1, smd5 = n, cmd5 = n
9. sverify = 1, cverify = 0, smd5 = n, cmd5 = n
13. sverify = 1, cverify = 1, smd5 = n, cmd5 = n

These all work as expected.  No MD5 signatures are requested, used or checked.

2. sverify = 0, cverify = 0, smd5 = n, cmd5 = y
10. sverify = 1, cverify = 0, smd5 = n, cmd5 = y

These work.  MD5 signatures generated client->server only.

6. sverify = 0, cverify = 1, smd5 = n, cmd5 = y
14. sverify = 1, cverify = 1, smd5 = n, cmd5 = y

These work, but not properly.  Connects and echoes, but client gets
stuck in LAST_ACK, client seems to ignore server's ACK.  MD5
signatures client->server only.

3. sverify = 0, cverify = 0, smd5 = y, cmd5 = n

Works.  MD5 signatures generated server->client only.

4. sverify = 0, cverify = 0, smd5 = y, cmd5 = y
12. sverify = 1, cverify = 0, smd5 = y, cmd5 = y

Works.  Signatures generated in both directions.

7. sverify = 0, cverify = 1, smd5 = y, cmd5 = n

Times out.  Client ignores SYN-ACK from server.  Signatures generated
server->client.

8. sverify = 0, cverify = 1, smd5 = y, cmd5 = y
16. sverify = 1, cverify = 1, smd5 = y, cmd5 = y

Works, but not properly.  Connects and echoes, but client gets stuck
in LAST_ACK again, client seems to ignore server's final ACK.  MD5
signatures in both directions.

11. sverify = 1, cverify = 0, smd5 = y, cmd5 = n

Does not work.  Server goes to SYN_RCVD, client goes to ESTABLISHED.
Server appears to ignore client's ACK.  MD5 signatures server->client
only.

15. sverify = 1, cverify = 1, smd5 = y, cmd5 = n

Times out. Server goes to SYN_RCVD, client sticks at SYN_SENT, appears
to ignore SYN-ACK.  Signatures server->client only.

While this was a bit time-consuming, it seems worthwhile because it
pointed up a couple of apparent problems that may have ramifications
outside openbgpd.

For openbgpd to work properly with both clients that do and don't use
TCP-MD5, cases 11 and 16 both need to work.  Currently neither do.

That case 16 doesn't work suggests that openbgpd probably has
hard-to-observe problems leaving sockets in LAST_ACK if it is the
MD5-using client.  Probably this has no effect in practice, because it
only appears to happen as the client, which would use a different port
to establish a new connection.  But it does generate a lot of
unnecessary retransmits so it would be nice to track this behavior
down.  Also, this is the actual as-designed/useful use of TCP-MD5
(generating and checking) so it seems the most important one to ensure
is 100% correct.

Case 11 certainly explains why OpenBGPd gets so confused; one side
thinks the connection is established, the other doesn't.  In order for
this to work, there must be some method of "turning off" the
expectation of receiving MD5-signed packets on a particular
connection.  In this context, the TCP_MD5SIG seems to be doing "double
duty" in that it is telling the connection to both send and expect MD5
signatures, although the tcp(4) man page refers only to outgoing
traffic.  (Though again that may be a documentation error.)

So there may be some bugs here on the FreeBSD side that may make it
hard to get openbgpd working properly.  There are a couple of other
things that may be worth looking at, but if someone knows the way
forward from here, any guidance would be appreciated.

Thanks!