Re: 60+% ping packet loss on Pi3 under -current and stable-13
Date: Sun, 01 May 2022 19:58:45 UTC
On 2022-May-1, at 12:15, Mark Millard <marklmi@yahoo.com> wrote:
> On 2022-May-1, at 11:12, bob prohaska <fbsd@www.zefox.net> wrote:
>
>> On Sat, Apr 30, 2022 at 06:39:57PM -0700, Bakul Shah wrote:
>>> On Apr 29, 2022, at 7:12 PM, bob prohaska <fbsd@www.zefox.net> wrote:
>>>>
>>>> Since about December of 2021 I've been noticing problems with
>>>> wired network connectivity on a pair of raspberry pi 3 machines
>>>> using wired network connections. One runs stable-13.1, the other
>>>> runs -current, both are up to date as of a few days ago.
>>>>
>>>> Essentially both machines fail to respond to inbound network
>>>> connections via ssh or ping after reboot. If I get on the
>>>> serial console and start an outbound ping to anywhere, both
>>>> machines respond to incoming pings with about a 65% packet
>>>> loss.
>>
>>> Suggest running tcpdump on the rpi3 to see what is going on
>>> when connected to the public vs private net.
>>>
>>
>> Public net first, since that's where the machine is now. Gateway.zefox.net
>> is the name of my router's public interface, dcn.org belongs to my isp and
>> fusionbroadband is their service provider..
>>
>> While on the -current Pi3 serial console (with no outbound ping running)
>> and no inbound traffic from my hosts I see after a couple minutes:
>>
>> root@www:/mnt # tcpdump
>> tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
>> listening on ue0, link-type EN10MB (Ethernet), capture size 262144 bytes
>> 10:39:40.887853 ARP, Request who-has www.zefox.org tell gateway.zefox.net, length 46
>> 10:39:40.887929 ARP, Reply www.zefox.org is-at b8:27:eb:71:46:4e (oui Unknown), length 28
>> 10:39:40.893220 ARP, Request who-has 50-1-20-1.dsl.static.fusionbroadband.com tell www.zefox.org, length 28
>> 10:39:40.915469 ARP, Reply 50-1-20-1.dsl.static.fusionbroadband.com is-at 00:1b:90:d2:4a:c4 (oui Unknown), length 50
>> 10:39:40.915529 IP www.zefox.org.50714 > spoke.dcn.davis.ca.us.domain: 51409+ PTR? 28.20.1.50.in-addr.arpa. (41)
>> 10:39:40.943602 IP spoke.dcn.davis.ca.us.domain > www.zefox.org.50714: 51409 1/3/6 PTR www.zefox.org. (265)
>> 10:39:40.945416 IP www.zefox.org.15986 > spoke.dcn.davis.ca.us.domain: 44966+ PTR? 31.20.1.50.in-addr.arpa. (41)
>> 10:39:40.973487 IP spoke.dcn.davis.ca.us.domain > www.zefox.org.15986: 44966 1/3/6 PTR gateway.zefox.net. (266)
>> 10:39:40.975037 IP www.zefox.org.57611 > spoke.dcn.davis.ca.us.domain: 31749+ PTR? 1.20.1.50.in-addr.arpa. (40)
>> 10:39:46.288219 IP www.zefox.org.49710 > wheel.dcn.davis.ca.us.domain: 31749+ PTR? 1.20.1.50.in-addr.arpa. (40)
>> 10:39:46.316239 IP wheel.dcn.davis.ca.us.domain > www.zefox.org.49710: 31749 1/3/6 PTR 50-1-20-1.dsl.static.fusionbroadband.com. (291)
>> 10:39:46.318267 IP www.zefox.org.17061 > spoke.dcn.davis.ca.us.domain: 37579+ PTR? 2.253.150.168.in-addr.arpa. (44)
>> 10:39:46.346851 IP spoke.dcn.davis.ca.us.domain > www.zefox.org.17061: 37579* 1/2/2 PTR spoke.dcn.davis.ca.us. (145)
>> 10:39:46.348674 IP www.zefox.org.40440 > spoke.dcn.davis.ca.us.domain: 20572+ PTR? 1.253.150.168.in-addr.arpa. (44)
>> 10:39:51.420705 IP www.zefox.org.64019 > wheel.dcn.davis.ca.us.domain: 20572+ PTR? 1.253.150.168.in-addr.arpa. (44)
>> 10:39:51.448850 IP wheel.dcn.davis.ca.us.domain > www.zefox.org.64019: 20572* 1/2/2 PTR wheel.dcn.davis.ca.us. (145)
>> 10:40:40.147603 ARP, Request who-has 50-1-20-1.dsl.static.fusionbroadband.com tell ns1.zefox.net, length 46
>> 10:40:40.148844 IP www.zefox.org.46127 > spoke.dcn.davis.ca.us.domain: 12186+ PTR? 29.20.1.50.in-addr.arpa. (41)
>> 10:40:40.176486 IP spoke.dcn.davis.ca.us.domain > www.zefox.org.46127: 12186 1/3/6 PTR ns1.zefox.net. (262)
>> 10:40:57.688225 ARP, Request who-has www.zefox.org tell gateway.zefox.net, length 46
>> 10:40:57.688305 ARP, Reply www.zefox.org is-at b8:27:eb:71:46:4e (oui Unknown), length 28
>> 10:42:14.488727 ARP, Request who-has www.zefox.org tell gateway.zefox.net, length 46
>> 10:42:14.488804 ARP, Reply www.zefox.org is-at b8:27:eb:71:46:4e (oui Unknown), length 28
>> 10:42:43.761226 ARP, Request who-has 50-1-20-1.dsl.static.fusionbroadband.com tell www.zefox.com, length 46
>> 10:42:43.762522 IP www.zefox.org.56181 > spoke.dcn.davis.ca.us.domain: 28779+ PTR? 26.20.1.50.in-addr.arpa. (41)
>> 10:42:43.790361 IP spoke.dcn.davis.ca.us.domain > www.zefox.org.56181: 28779 1/3/6 PTR www.zefox.com. (265)
>> 10:43:31.289103 ARP, Request who-has www.zefox.org tell gateway.zefox.net, length 46
>> 10:43:31.289181 ARP, Reply www.zefox.org is-at b8:27:eb:71:46:4e (oui Unknown), length 28
>>
>> If I now start an inbound ping from one of my hosts it gets no reply and
>> tcpdump reports no additional traffic. With an outbound ping running there's
>> at least a sparse reply.
>>
>> ^C
>> 28 packets captured
>> 28 packets received by filter
>> 0 packets dropped by kernel
>> root@www:/mnt #
>>
>> The "oui unknown" looks like some sort of failure.....
>> Can you ping www.zefox.org? I have no outside vantage point.
>> There is still no outbound ping running and I would expect
>> you'll get no or very sparse reply.
>>
>>
>> Thus far only the two Pi3s suffer from connectivity problems; Pi2s and a Pi4 have
>> no difficulty on the same address block. Is there a switch for tcpdump that will
>> limit records to relevant traffic? Otherwise it's a flood.
>>
>> These results were obtained after standing idle overnight and
>> are rather different (in ways I don't understand) from behavior
>> immediately after reboot, I'll have to repeat as I learn more.
>
> I wonder if there is a notable difference between
> monitoring traffic from 2 places:
>
> A) from the machine seeing the problem
> vs.
> B) from a machine not having problems but
> connected were all the traffic would be
> on the wire it is connected to.
>
> It may be that monitoring from both and
> comparing/contrasting the reported traffic
> from the two provides additional evidence.
>
> There may be modes of monitoring that are
> relevant for this. But I'm not familiar
> with any detail here.
>
>
> For reference:
>
> # ping www.zefox.org
> PING www.zefox.org (50.1.20.28): 56 data bytes
> ^C
> --- www.zefox.org ping statistics ---
> 32 packets transmitted, 0 packets received, 100.0% packet loss
>
> I found the command traceroute and it reports:
>
> # traceroute www.zefox.org
> traceroute to www.zefox.org (50.1.20.28), 64 hops max, 40 byte packets
> 1 192.168.1.1 (192.168.1.1) 0.697 ms 0.486 ms 1.277 ms
> 2 172.30.26.66 (172.30.26.66) 30.019 ms
> 172.30.26.67 (172.30.26.67) 41.720 ms
> 172.30.26.66 (172.30.26.66) 28.645 ms
> 3 68.85.243.125 (68.85.243.125) 8.967 ms
> 68.85.243.77 (68.85.243.77) 11.462 ms
> 68.85.243.125 (68.85.243.125) 10.254 ms
> 4 24.124.129.106 (24.124.129.106) 7.510 ms
> 96.216.60.165 (96.216.60.165) 10.176 ms
> 24.124.129.106 (24.124.129.106) 8.945 ms
> 5 68.85.243.197 (68.85.243.197) 10.837 ms
> 96.216.60.165 (96.216.60.165) 10.252 ms
> 68.85.243.197 (68.85.243.197) 16.036 ms
> 6 68.85.243.197 (68.85.243.197) 14.660 ms
> be-36211-cs01.seattle.wa.ibone.comcast.net (68.86.93.49) 14.629 ms
> 68.85.243.197 (68.85.243.197) 8.849 ms
> 7 be-2412-pe12.seattle.wa.ibone.comcast.net (96.110.34.142) 14.607 ms
> be-36221-cs02.seattle.wa.ibone.comcast.net (68.86.93.53) 14.122 ms
> be-2212-pe12.seattle.wa.ibone.comcast.net (96.110.34.134) 13.877 ms
> 8 be-2412-pe12.seattle.wa.ibone.comcast.net (96.110.34.142) 14.133 ms * 13.663 ms
> 9 be2075.ccr21.sfo01.atlas.cogentco.com (154.54.0.233) 30.176 ms *
> be3717.ccr22.sfo01.atlas.cogentco.com (154.54.86.209) 29.002 ms
> 10 be3717.ccr22.sfo01.atlas.cogentco.com (154.54.86.209) 28.477 ms
> be2430.ccr31.sjc04.atlas.cogentco.com (154.54.88.186) 27.203 ms
> be2075.ccr21.sfo01.atlas.cogentco.com (154.54.0.233) 28.515 ms
> 11 38.104.141.82 (38.104.141.82) 29.820 ms
> be2430.ccr31.sjc04.atlas.cogentco.com (154.54.88.186) 28.605 ms
> 38.104.141.82 (38.104.141.82) 33.735 ms
> 12 38.104.141.82 (38.104.141.82) 27.160 ms
> 0.xe-0-3-0.scrm-gw1.scrmca01.sonic.net (135.180.179.146) 32.336 ms
> 38.104.141.82 (38.104.141.82) 31.867 ms
> 13 0.xe-0-0-0.cr1.scrmca13.sonic.net (135.180.179.166) 31.761 ms
> 0.xe-0-3-0.scrm-gw1.scrmca01.sonic.net (135.180.179.146) 29.864 ms
> 0.xe-0-0-0.cr1.scrmca13.sonic.net (135.180.179.166) 31.711 ms
> 14 0.xe-0-0-0.cr1.scrmca13.sonic.net (135.180.179.166) 30.373 ms
> gig1-1-1.gw.wscrca11.sonic.net (50.1.36.106) 35.567 ms
> 0.xe-0-0-0.cr1.scrmca13.sonic.net (135.180.179.166) 31.146 ms
> 15 gig1-1-1.gw.davsca11.sonic.net (50.1.36.110) 31.513 ms
> gig1-1-1.gw.wscrca11.sonic.net (50.1.36.106) 31.203 ms
> gig1-1-1.gw.davsca11.sonic.net (50.1.36.110) 31.354 ms
> 16 gig1-1-1.gw.davsca11.sonic.net (50.1.36.110) 30.125 ms * 31.996 ms
> 17 * * *
> 18 * * *
> 19 * * *
> 20 * * *
> 21 * * *
> 22 * * *
> 23 * * *
> 24 * * *
> 25 * * *
> 26 * * *
> 27 * * *
> 28 * * *
> 29 * * *
> 30 * * *
> ^C
>
> (There did not seem to be much point in having it continue.)
I found and built a port called net/mtr-nox11
("My traceroute") and tried it, letting it just
run. The initial try eventually got a connection
but reported a 99.2% packet loss as of when I
captured the below:
My traceroute [v0.95]
amd64_ZFS (192.168.1.120) -> www.zefox.org (50.1.20.28) 2022-05-01T12:40:22-0700
Keys: Help Display mode Restart statistics Order of fields quit
Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. 192.168.1.1 0.0% 135 0.4 0.8 0.1 3.1 0.4
2. 172.30.26.66 0.0% 134 28.2 26.1 9.3 132.7 18.1
3. 68.85.243.77 0.0% 134 8.6 9.0 7.5 11.2 0.8
4. 24.124.129.106 0.0% 134 10.2 9.1 7.6 13.4 0.9
5. 96.216.60.165 0.0% 134 9.0 9.1 7.8 14.3 0.9
6. 68.85.243.197 0.0% 134 14.4 13.6 9.2 44.3 5.4
7. be-36241-cs04.seattle.wa.ibone.comcast.net 0.0% 134 16.8 14.9 13.0 22.6 1.1
8. be-2412-pe12.seattle.wa.ibone.comcast.net 0.0% 134 13.5 15.0 12.8 46.4 3.2
9. (waiting for reply)
10. be2075.ccr21.sfo01.atlas.cogentco.com 0.0% 134 29.3 29.0 26.7 54.1 2.9
11. be2379.ccr31.sjc04.atlas.cogentco.com 0.0% 134 28.0 28.7 27.1 40.3 1.3
12. 38.104.141.82 0.0% 134 28.0 33.8 26.6 114.8 16.5
13. 0.xe-0-3-0.scrm-gw1.scrmca01.sonic.net 0.0% 134 30.9 31.0 29.0 33.7 0.8
14. 0.xe-0-0-0.cr1.scrmca13.sonic.net 0.0% 134 31.1 32.3 29.3 93.2 6.7
15. gig1-1-1.gw.wscrca11.sonic.net 0.0% 134 31.3 34.9 29.5 330.4 26.5
16. gig1-1-1.gw.davsca11.sonic.net 0.0% 134 32.8 32.1 29.9 44.1 1.7
17. (waiting for reply)
18. (waiting for reply)
19. www.zefox.org 99.2% 134 74.9 74.9 74.9 74.9 0.0
I stopped and restarted it and so far no connection
-- waiting even longer than that first time: Snt
is now over 600. Rows 18 and 19 have not shown up,
the last is 17.
. . . (some more time goes by) . . .
I have now stopped it, avoiding the extra load on the
machines and network.
Looks like there is some problem getting past
gig1-1-1.gw.davsca11.sonic.net .
===
Mark Millard
marklmi at yahoo.com