cpsw drops packets when stressed on BBB and 11.0-STABLE

[Olavi Kumpulainen]

Hi,

I'm running a snapshot build of FreeBSD-11,
FreeBSD beaglebone 11.0-STABLE FreeBSD 11.0-STABLE #0 r317153: Thu Apr 20 09:21:26 UTC 2017     

on a BBB.
I see that cpsw drops outgoing packets when stressed.  

Out of some reason, dev.cpswss.0.stats.RxStartOfFrameOverruns increments when packets are dropped which may be a hint on what’s going on. 

The fact that RxStartOf... increases is confusing, because the packets seem to be dropped in transmission. 

Anyway - I’ve found a simple way to reproduce the problem, namely by sending long pings.

On the BBB:

# tcpdump -ni cpsw0 icmp& 

Initial state of RxStartOfFrameOverruns in BBB after playing around a bit:

# sysctl dev.cpswss.0.stats.RxStartOfFrameOverruns 
dev.cpswss.0.stats.RxStartOfFrameOverruns: 86                     

# ping -c 1 -s 14000 192.168.0.3

PING 192.168.0.3 (192.168.0.3): 14000 data bytes

11:36:57.965980 IP 192.168.0.158 > 192.168.0.3: ICMP echo request, id 53762, seq 0, length 1480
11:36:57.966658 IP 192.168.0.158 > 192.168.0.3: ip-proto-1
11:36:57.966826 IP 192.168.0.158 > 192.168.0.3: ip-proto-1
11:36:57.966923 IP 192.168.0.158 > 192.168.0.3: ip-proto-1
11:36:57.967009 IP 192.168.0.158 > 192.168.0.3: ip-proto-1
11:36:57.967090 IP 192.168.0.158 > 192.168.0.3: ip-proto-1
11:36:57.967173 IP 192.168.0.158 > 192.168.0.3: ip-proto-1
11:36:57.967254 IP 192.168.0.158 > 192.168.0.3: ip-proto-1
11:36:57.967336 IP 192.168.0.158 > 192.168.0.3: ip-proto-1
11:36:57.967414 IP 192.168.0.158 > 192.168.0.3: ip-proto-1 

(10 packets has supposedly been put into the tx ring in BBB) 

Looking at RxStartOfFrameOverruns in the BBB, I see an increment by 5…

#sysctl dev.cpswss.0.stats.RxStartOfFrameOverruns 
dev.cpswss.0.stats.RxStartOfFrameOverruns: 91

 

I've set up a tcpdump on the target machine:

$ sudo tcpdump -ni eth2 icmp
13:52:42.603199 IP 192.168.0.158 > 192.168.0.3: ICMP echo request, id 53762, seq 0, length 1480
13:52:42.604697 IP 192.168.0.158 > 192.168.0.3: ip-proto-1 

(Eight fragments lost!)

Without tcpump in BBB, more packets seem to go through (showing tcpdump on target);

13:56:08.396553 IP 192.168.0.158 > 192.168.0.3: ICMP echo request, id 55554, seq 0, length 1480
13:56:08.397781 IP 192.168.0.158 > 192.168.0.3: ip-proto-1
13:56:08.399029 IP 192.168.0.158 > 192.168.0.3: ip-proto-1
13:56:08.400157 IP 192.168.0.158 > 192.168.0.3: ip-proto-1
13:56:08.401409 IP 192.168.0.158 > 192.168.0.3: ip-proto-1 

(Five packets lost) 

Again, there's an increment in RxStartOfFrame...:

# sysctl dev.cpswss.0.stats.RxStartOfFrameOverruns 
dev.cpswss.0.stats.RxStartOfFrameOverruns: 96              

I added a printf in tx_enqueue() in an attempt to see what’s going on, but doing so “fixed the bug” – obviously by adding a delay in the forwarding code. Maybe we have a timing/race between the driver and the cpsw hardware?

Also, I tested sending 14k pings from the standard-installed Linux in the BBB and that worked just fine. So the packets aren't lost between the hosts (the machines are connected via the same switch).

Any ideas?

Cheers -

/Olavi