[Bug 270285] Network issue with very small frames (tcp, padded)

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=270285

            Bug ID: 270285
           Summary: Network issue with very small frames (tcp, padded)
           Product: Base System
           Version: 12.3-STABLE
          Hardware: Any
                OS: Any
            Status: New
          Severity: Affects Only Me
          Priority: ---
         Component: kern
          Assignee: bugs@FreeBSD.org
          Reporter: mhaarmann@midoco.de

Hi experts,

we are using freebsd under the hood of a pfsense firewall (latest CE release,
which builds on 12.3 Stable).

Recently, we activated a bunch of servers in a new very fast (virtualized)
environment, which is connected to the internal network with 10GBit. The
pfsense firewall has 1GBit NICs on LAN and WAN side and is a physical machine.
We now encountered an issue which is probably resulting from an overload
situation or MTU issue (while all MTU values are 1500 everywhere):
When transmitting a medium large file over http (apache http server running on
Ubuntu on the server side), the traffic sometimes, not always contains a very
small frame which is corrupting the output.

We can see in the packet capture taken on the LAN side of the firewall that
normally all data frames are 1514 bytes, but all of a sudden, a 60 byte frame
(marked as PSH,ACK) arrives which contains a very small tcp segment data of 4
bytes only. This frame is padded with two 0 bytes at the end AFTER the payload.

Extract from wireshark:
Transmission Control Protocol, Src Port: 80, Dst Port: 37710, Seq: 553466, Ack:
188, Len: 4Frame 136159: 60 bytes on wire (480 bits), 60 bytes captured (480
bits)
Ethernet II, Src: e2:84:72:d3:14:5c (e2:84:72:d3:14:5c), Dst: IETF-VRRP-VRID_04
(00:00:5e:00:01:04)
    Destination: IETF-VRRP-VRID_04 (00:00:5e:00:01:04)
    Source: e2:84:72:d3:14:5c (e2:84:72:d3:14:5c)
    Type: IPv4 (0x0800)
    Padding: 0000

Internet Protocol Version 4, Src: 192.168.25.41, Dst: 80.xxx.xxx.xxx
    0100 .... = Version: 4
    .... 0101 = Header Length: 20 bytes (5)
    Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)
    Total Length: 44
    Identification: 0x594c (22860)
    Flags: 0x40, Don't fragment
    ...0 0000 0000 0000 = Fragment Offset: 0
    Time to Live: 64
    Protocol: TCP (6)
    Header Checksum: 0x37c2 [validation disabled]
    [Header checksum status: Unverified]
    Source Address: 192.168.25.41
    Destination Address: 80.xx.xx.xx
Transmission Control Protocol, Src Port: 80, Dst Port: 37710, Seq: 553466, Ack:
188, Len: 4
    Source Port: 80
    Destination Port: 37710
    [Stream index: 77]
    [Conversation completeness: Complete, WITH_DATA (47)]
    [TCP Segment Len: 4]
    Sequence Number: 553466    (relative sequence number)
    Sequence Number (raw): 2934325866
    [Next Sequence Number: 553470    (relative sequence number)]
    Acknowledgment Number: 188    (relative ack number)
    Acknowledgment number (raw): 2019009064
    0101 .... = Header Length: 20 bytes (5)
    Flags: 0x018 (PSH, ACK)
    Window: 42153
    [Calculated window size: 42153]
    [Window size scaling factor: -2 (no window scaling used)]
    Checksum: 0xc1c8 [unverified]
    [Checksum Status: Unverified]
    Urgent Pointer: 0
    [Timestamps]
    [SEQ/ACK analysis]
    TCP payload (4 bytes)
    [Reassembled PDU in frame: 137331]
    TCP segment data (4 bytes)

0000   00 00 5e 00 01 04 e2 84 72 d3 14 5c 08 00 45 00
0010   00 2c 59 4c 40 00 40 06 37 c2 c0 a8 19 29 50 XX
0020   XX XX 00 50 93 4e ae e6 42 6a 78 57 a2 28 50 18
0030   a4 a9 c1 c8 00 00 ee 86 11 a2 00 00

The real payload is only 4 bytes, (0xee, 0x86, 0x11, 0xa2), After that, two
bytes are appended (sort of padding, resulting in a total of 60 bytes).
We do not know why this padding occurs, but it is copied into the forwarded
frame, leading to a currupted output.


Any idea is welcome how to solve this. The padding seems to be the underlying
reason of the data congestion, but we do not know if there is a way to prevent
the padding on the sending side or if simply there is a bug in the tcp code of
Freebsd which forwards this padded bytes as content.
Traffic between the internal machines is not affected for whatever reason
(mostly Linux). 

The congestion issues do not occur on each transfer, but only sometimes. 
Most of the time, those small frames are not sent for whatever reason.
I found in the cap file that there have been TCP window full / TCP window
update  messages, but these are 
We have tested always with the same file, but the frame sizes vary on each
transfer.
Padding seems to be a well-known process in order to make small packets have
the minimum length of 64 bytes, as I have read in some articles.
So the tcp code should be able to handle this from my point of view.
I can of course provide the .cap file with the defect flow if that makes sense.
But my C knowledge is very limited, so I am not able to debug the kernel (and
as always, this happens in production, is not reproducable in testing, and we
do not have the identical network setup for test).

Best regards,

Marcus

-- 
You are receiving this mail because:
You are the assignee for the bug.