From nobody Thu Mar 16 21:14:25 2023 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Pd0QD5dpvz3ybnx for ; Thu, 16 Mar 2023 21:14:44 +0000 (UTC) (envelope-from yvesguerin@yahoo.ca) Received: from sonic309-21.consmr.mail.gq1.yahoo.com (sonic309-21.consmr.mail.gq1.yahoo.com [98.137.65.147]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Pd0QB6R6Dz4KJr for ; Thu, 16 Mar 2023 21:14:42 +0000 (UTC) (envelope-from yvesguerin@yahoo.ca) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.ca header.s=s2048 header.b=lVAxIjHy; spf=pass (mx1.freebsd.org: domain of yvesguerin@yahoo.ca designates 98.137.65.147 as permitted sender) smtp.mailfrom=yvesguerin@yahoo.ca; dmarc=pass (policy=reject) header.from=yahoo.ca DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.ca; s=s2048; t=1679001280; bh=qrHtO9YOk8HWB/fSr9UsEg0IRQkzKlHOmL7US7coS84=; h=Date:From:Reply-To:To:In-Reply-To:References:Subject:From:Subject:Reply-To; b=lVAxIjHyAkxU/fXxDF1/lP5iFMGP/iSH62Onm517/dD7hwl9sKY1I47raYmGe0DE5B/rL9biZXmrfalM9F2/i7e7i6DQMGgkrE3iQiNPuPvntmDFOo24K4n+zgW7GFbgaISD9q6MZjb4sx4RAz5Z15/rIJfNvyGRQpbLuKGhN9Wl8IixHr1lwxdg2O9rq4ZFPdAhuBS6vFgva5WzHzIWXTUFbQAfYbcH6ONTmVFvDhjWdccU12GEjO3TU0vw/L0MYumpUxBKudSZIQa+IEw1LrFjbzVwO7qGdXIQcn1ZnPQlN2RI2KV8MKUykMDmjl4LHd1x3vRRtAOEb/l6ee+kxQ== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1679001280; bh=QKsl/5zu+Pze3tIcTSkQ/PtkZy1CJ2jmYZVzr7E+YNJ=; h=X-Sonic-MF:Date:From:To:Subject:From:Subject; b=RTTs1eGE40EBnCxtqXLX8O2G9v6z9lFKKNYoQuIWCp72uf6oNbEzWLLVD2DwRU1OY37xdsLnVNJeQOPL9g3kF9MqgAQUWFBSIRHWTpP1L80bbeTKXNI0Z8vfwis1kOltXtCirR71ejc/eH2Qf7KCTNWswkjKLKyjdRb1hHiWKVFSYGru6QibCChote4OLLvGnoOxL8NpWF4B54gn9E6XwjH/CB9oShMPzkQIkjwlJHHxPnyRlsoPgmPLyRqw5rHzGDWZIg8ugHvGN1+HntKP8fp/w6yHJ4FV66Yawg+p2HtTWQTUKyL3fxlzZeF00+vaG4I+1x/rq2EQzrm3UX9uKg== X-YMail-OSG: gzRtlt0VM1lLVGZqnSvDaHvoC2E6c4u8YLoeCNyHNEzQqEVZJQrW9djfljF9Hbx 0HcCEBfO0pUrDMW6LXx5MMROlYhvzU3.l.nMk9MySAp_aTMSJH2xFrq02n8wwmOvL.g65z086JZB 1LhBRvlPENHPPlQJSQ6kiTPESsrA9DabUauQ3rWk1q6nWCury299BsEh8SK0azpUU42jabwS2jsi QOY2R7EwLFOR2Uju0H1pVrQuoVt3CqAIfSAHADEOaVtpnsJrmvlwg8bmdn_E14j2GqgfjG2lzFBt .1rGgVJ975ZL_WnAJguuh5ISUuhM6N84SFGSTnEc0UHAVD0H0v5JtO9Dng_MM7Qsiy_Y3BB9kYUo Pxtml_qVYN4e7_CsVpol1Gdu5hd5f1tjl4ewnHwXQOqjVgZEugFWnE1n8j7S7OLbMlfzxEo_KLKk HUUM0R_4fPKlQ8tx8SkfHX1d_Vpt1tExF1CQ3LgMTvCouggIjQkF6JQSdRTKjwnmeGnUIkiIoGtQ ZogijwFFs..izgRqBIRZkbaqWVN8i6giV3oqx5zeEjqP3.tGzYxy2VTunlZV9qha47YaYtMPdy25 g0lTv009zxGL47h4PuPl.Bzs1dI9s1F1n3eAUHP1V7MFsA2_oSN0LTWHikaDrF4YFDZA3h2.DbBi bCha5miaNcEq2IiZ7lBQnGfQItce9UtXyrv0AcImjT6sg8A1jj.fG8QTcTpv9MOOqY7C3lm2NaNV 5gpIoM6Q6rzzByOqHYrMwHA9ia5Hx42G0UySX_5szG7OgcIzm0w27IEK89fEqAz7fRV4zQbMhVPt RcvdVqv4xV7qLa5tHKp9V9ALguW3qvOpWSuTUPnz6SnVL5Nkeodv_oQugQqdDw9aYMKxHcYZJukb HZJRaUUhmeXa3bzTc3JChndFkLpYR48eM6CP0wnFQQEnV.QnqZYeets4l68khFY7ATFUefAuVZ5W jLjswez46nyIBqqWqxdayP0OmtctM6j3R69Bx4i8CuMptg7S8m8oyWXsr6IP0U3LiZ6lOcIffKpV XnLGdHpjqS7hW2ZyncqVMA7qn9hyFaW4Olmdzm_LkPogX9B3hVa9ZPCTo7usaMeSadRPxwj.jnuY auyJzgS2pJvJg9NNJNHLery6dXtxqfT6BaaDSyGkyLUUxDApH_MkXOEpXd4u7FRCh0v0qHiuUV6B ETEGpC.JkzC1aZZTewtGNPGg8z6Gl1.JPtsjizLQ.FJWWSThhrIZAokVaxdbcIZFpR_Dk2U2xrsE kGTllNf0ZWNjFV1PVT3CGJzVHlaFAsXgCGn6acW.YUs1qxAqRqhNgZtiwd30f2pv6wrbk1kxKmst hLeD7DkbYmGUdTJLgrJq2Q35RzGp1OJgOkpnZnvr4ZzsojybkjsQX34gtl1LbSfpED2b5iuu_gq2 4O.s0s0usL06Si.jt6byUlAPvH.U2tzbs0PeEesOyAeaVEuQQdMJKLEuvR44tnKdXPsLgPOnuMEq 94lCeTHp2eKE_ofXTaZpF2Lv766LMBldR1NBZ1ENxSA3YhAec44iKxI4J41dQKrwfDV4Nx8VhZGt ALOqHM3noERlP1duemAwFc2SH2inME3JlWD7RwPY04SDt.ghHN.oLNghEYb127Zbgrn_cyrr.Eln GOnIdq7A8by._vh5qoc93Hrn0mN70C.iGYWJ1HTLWx7hPEprNqOVA10Vsbr25sGwCHIJBnrLy.Ar DoDLXt2YrBVkjPwTCs2HkO28q3exjxQJjMQHJCTG2b90fU6UTxn.wCuxr.gwyWSLGS69e8LoOIvI 49SSLxEGaEv7H7hV69R7CvfNk9vmXDaBs_8k_.nT8UMQiTdtRMCgvzJUA1hMdVcid57DDwKEgg_u H5bp5lTuh4by1G.6cqCpI3KOdQnie.GeoENhBGhdPTDWlHRrAPdXJN4pzFwNPmA5GmSAcZ1j6x5W OMKGmmLzMwQN1VdtiNQeU5H9uzqyfsk_iYcMPrjwheAB3.ZXhO_dmgr1bVWk7OVbv9Rbrxf6Ml39 CO7WMgKymIy2_xyg8kKgMXVMoPPg7F46fcYTQ396t7T2y.vu7Z0W37SK2d6rhmhF4S1HgzSpOCR. G5QBEB1q.tzyy433q5r0yoAKprnn8jS9hHwyAALZaX9AQTz7CEsY5iaiBWKSTM4OdiOy.r1Q7Zz9 .yOLE9Q.8vJarv25HynYBDJIMeHqAuGfWBJbhlLqAO40GFq4tDYHZF9yjMPX1pyS1XBHO5WSllhA gcDvwOnFmLoXAHgIgiS4ogyNMFh9a4E8q6mFxllUwlJ2mqIXWCrHIM8C3Ll75sWBO X-Sonic-MF: X-Sonic-ID: 133a61de-c0d4-48af-83d7-158a5353ad0d Received: from sonic.gate.mail.ne1.yahoo.com by sonic309.consmr.mail.gq1.yahoo.com with HTTP; Thu, 16 Mar 2023 21:14:40 +0000 Date: Thu, 16 Mar 2023 21:14:25 +0000 (UTC) From: =?UTF-8?Q?Yves_Gu=C3=A9rin?= Reply-To: =?UTF-8?Q?Yves_Gu=C3=A9rin?= To: "freebsd-stable@freebsd.org" , Attila Nagy Message-ID: <132303943.191443.1679001265318@mail.yahoo.com> In-Reply-To: References: Subject: Re: Kernel DHCP unpredictable/fails (PXE boot), userspace DHCP works just fine List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_191442_1545221852.1679001265315" X-Mailer: WebService/1.1.21311 YMailNorrin X-Spamd-Result: default: False [0.50 / 15.00]; NEURAL_SPAM_MEDIUM(0.87)[0.865]; DMARC_POLICY_ALLOW(-0.50)[yahoo.ca,reject]; NEURAL_SPAM_SHORT(0.35)[0.352]; NEURAL_SPAM_LONG(0.28)[0.284]; R_DKIM_ALLOW(-0.20)[yahoo.ca:s=s2048]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; FROM_HAS_DN(0.00)[]; REPLYTO_EQ_FROM(0.00)[]; TAGGED_RCPT(0.00)[]; HAS_REPLYTO(0.00)[yvesguerin@yahoo.ca]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FREEMAIL_REPLYTO(0.00)[yahoo.ca]; RCVD_IN_DNSWL_NONE(0.00)[98.137.65.147:from]; ARC_NA(0.00)[]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; FREEMAIL_FROM(0.00)[yahoo.ca]; DKIM_TRACE(0.00)[yahoo.ca:+]; TO_DN_SOME(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; MLMMJ_DEST(0.00)[freebsd-stable@freebsd.org]; RCPT_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_TO(0.00)[freebsd.org,gmail.com]; FREEMAIL_ENVFROM(0.00)[yahoo.ca]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Queue-Id: 4Pd0QB6R6Dz4KJr X-Spamd-Bar: / X-ThisMailContainsUnwantedMimeParts: N ------=_Part_191442_1545221852.1679001265315 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Dear Attila, May be I will add some noise to your thread, sorry in advance, I am just a = sysadmin and I faced the same problem with one of my old hp g7 the network = card was broken (malfunctionning) , sometime it works and sometime not when= I used pxe and dhcpd (take to much time to answer to the dhcp so the mothe= rboard decided to reboot, etc. (infinite loop)).=C2=A0 The card works perfe= ctly when it's setup by an OS. May be it's a stupid question or two: do you check the network cable ?=C2= =A0 (I faced some defective cables and it ruin my day...) in the same way w= hat about the hub/router attached to this server (configuration, etc.), Do = you switched a good one by a bad one ? (same network cable, hub/router, etc= .) I spend too much nights in the lab... Regards,=20 Yves Guerin=20 Le jeudi 16 mars 2023 =C3=A0 16:44:49 UTC=E2=88=924, Attila Nagy a =C3=A9crit : =20 =20 Hi, As this is super annoying, I'm willing to pay a $500 bounty for solving thi= s issue (whomever is first, however I don't anticipate a big competition :)= Having an invoice would be best, but I'm willing to accept individuals as = well).I can't give remote access, but can run debug builds with serial cons= ole. stable/13 branch. I have a bunch of netbooted machines, one set in a cluster is older (HP DL8= 0 G9, 2x8C, Intel I350 -igb- NICs), the other set is newer (HP XL225n G10, = AMD EPYC2x16C, BCM57412 -bnxt- NICs).All of these boot from the network, wh= ich is basically:- get IP and options with DHCP with the help of the NIC's = PXE stack- get the loader and kernel, start it- do another round of DHCP fr= om the kernel (bootp_subr.c)- mount the root via NFS and let everything wor= k as usual The problem is that the newer machines take an indefinite time to boot. The= older ones (with igb NIC) work reliably, they always boot fast. The process of getting an IP address via DHCP (bootpc_call from bootp_subr.= c) either succeeds normally (in a few seconds), or takes a lot of time.Comm= on (measured) times to boot range from 10s of minutes to anywhere between a= few hours (1-6).Sometimes it just gets stuck and couldn't get past bootpc_= call (getting the DHCP lease). What I've already tried:- we have a redundant set of DHCP servers which off= er static leases (so there are two DHCPOFFERs), so I tried to turn off one = of them, nothing has changed - tried to disable SMP, the effect is the same - tried to see whether it's a network issue. The NIC's PXE stack always get= s the lease quickly and booting FreeBSD from an ISO and issuing dhclient on= the same interface is also fast. After the machines have booted, there are= no network issues, they work reliably (since more than a year for 20+ mach= ines, so not just a few hours) This issue wasn't so bad previously (only a few mins to tens of minutes del= ay), but recently it got pretty unbearable, even making some machines unboo= table for days... First I thought it might be a packet loss (or more exactly packet delivery = from the DHCP server to the receiving socket), either in the network or in = the NIC/kernel itself, so I placed a few random printfs into bootp_subr.c a= nd udp_usrreq.c. After spending some time trying to understand the problem it feels like a r= ace condition in=20 bootpc_call, but I don't know the code well enough to effectively verify th= at. Here are the modified bootp_subr.c and udp_usrreq.c:https://gist.githubuser= content.com/bra-fsn/128ae9a3bbc0dbdbb2f6f4b3e2c5157a/raw/a8ade8af252f618c84= a46da2452d557ebc5078ac/bootp_subr.chttps://gist.github.com/bra-fsn/128ae9a3= bbc0dbdbb2f6f4b3e2c5157a/raw/a8ade8af252f618c84a46da2452d557ebc5078ac/udp_u= srreq.c(modified from stable/13 branch from a few weeks earlier) This is the output with the always working DL80 (igb) machine:https://gist.= github.com/bra-fsn/128ae9a3bbc0dbdbb2f6f4b3e2c5157a/raw/a8ade8af252f618c84a= 46da2452d557ebc5078ac/DL80%2520igb%2520good.txt This is the console output from a working boot for the XL225n (bnxt) machin= e:https://gist.github.com/bra-fsn/128ae9a3bbc0dbdbb2f6f4b3e2c5157a/raw/a8ad= e8af252f618c84a46da2452d557ebc5078ac/XL225n%2520bnxt%2520good.txtas you can= see, it's much slower than the DL80 (which also isn't that fast...) And this one is a longer output, without success to that point (2 minutes w= ithout completing the DHCP flow):https://gist.github.com/bra-fsn/128ae9a3bb= c0dbdbb2f6f4b3e2c5157a/raw/a8ade8af252f618c84a46da2452d557ebc5078ac/XL225n%= 2520bnxt%2520long.txt For the latter, here's an excerpt from the DHCP log: https://gist.githubusercontent.com/bra-fsn/128ae9a3bbc0dbdbb2f6f4b3e2c5157a= /raw/a8ade8af252f618c84a46da2452d557ebc5078ac/dhcp_log.txt It seems the DHCP state always gets reset to IF_DHCP_UNRESOLVED even if the= re's answers from the DHCP server. Here's another, longer console log, which succeeded after spending 236 seco= nds in the loop: https://gist.github.com/bra-fsn/128ae9a3bbc0dbdbb2f6f4b3e2c5157a/raw/a77f52= f5e83c699b38a7c2d3acdc52d26ceeba71/XL225n%2520bnxt%2520long%2520good.txt Any ideas about this? =20 ------=_Part_191442_1545221852.1679001265315 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Dear Attila,

May be I will add some noise to your thread, sorry in advance, I am ju= st a sysadmin and I faced the same problem with one of my old hp g7 the net= work card was broken (malfunctionning) , sometime it works and sometime not= when I used pxe and dhcpd (take to much time to answer to the dhcp so the = motherboard decided to reboot, etc. (infinite loop)).  The card works = perfectly when it's setup by an OS.

May be it's a stupid = question or two: do you check the network cable ?  (I faced some defec= tive cables and it ruin my day...) in the same way what about the hub/route= r attached to this server (configuration, etc.), Do you switched a good one= by a bad one ? (same network cable, hub/router, etc.)

I = spend too much nights in the lab...

Regards,

Yves Guerin


=20
=20
Le jeudi 16 mars 2023 =C3=A0 16:44:49 UTC=E2=88=924, At= tila Nagy <nagy.attila@gmail.com> a =C3=A9crit :


Hi,

As thi= s is super annoying, I'm willing to pay a $500 bounty for solving this issu= e (whomever is first, however I don't anticipate a big competition :) Havin= g an invoice would be best, but I'm willing to accept individuals as well).=
I can't give remote access, but can run debug builds with serial= console. stable/13 branch.

I have a bunch of = netbooted machines, one set in a cluster is older (HP DL80 G9, 2x8C, Intel = I350 -igb- NICs), the other set is newer (HP XL225n G10, AMD EPYC2x16C, BCM= 57412 -bnxt- NICs).
All of these boot from the network, which is = basically:
- get IP and options with DHCP with the help of the NI= C's PXE stack
- get the loader and kernel, start it
- d= o another round of DHCP from the kernel (bootp_subr.c)
- mount th= e root via NFS and let everything work as usual

Th= e problem is that the newer machines take an indefinite time to boot. The o= lder ones (with igb NIC) work reliably, they always boot fast.
The process of getting an IP address via DHCP (bootpc_call from bootp_sub= r.c) either succeeds normally (in a few seconds), or takes a lot of time.
Common (measured) times to boot range from 10s of minutes to anywh= ere between a few hours (1-6).
Sometimes it just gets stuck and c= ouldn't get past bootpc_call (getting the DHCP lease).

=
What I've already tried:
- we have a redundant set of DHCP s= ervers which offer static leases (so there are two DHCPOFFERs), so I tried = to turn off one of them, nothing has changed
- tried to disab= le SMP, the effect is the same
- tried to see whether it's a = network issue. The NIC's PXE stack always gets the lease quickly and bootin= g FreeBSD from an ISO and issuing dhclient on the same interface is also fa= st. After the machines have booted, there are no network issues, they work = reliably (since more than a year for 20+ machines, so not just a few hours)=

This issue wasn't so bad previously (only a f= ew mins to tens of minutes delay), but recently it got pretty unbearable, e= ven making some machines unbootable for days...

Fi= rst I thought it might be a packet loss (or more exactly packet delivery fr= om the DHCP server to the receiving socket), either in the network or in th= e NIC/kernel itself, so I placed a few random printfs into bootp_subr.c and= udp_usrreq.c.

After spending some time trying to = understand the problem it feels like a race condition in
boo= tpc_call, but I don't know the code well enough to effectively verify that.=

Here are the modified bootp_subr.c and udp_u= srreq.c:
(modified from stable/13 branch from a few weeks earli= er)

This is the output with the always working= DL80 (igb) machine:
https://gist.github.com/bra-fsn/128ae9a3bbc0dbdbb2f6f4b3e2c5157a/r= aw/a8ade8af252f618c84a46da2452d557ebc5078ac/DL80%2520igb%2520good.txt

This is the console output from a working boot for = the XL225n (bnxt) machine:
as you can see, it's much slower than the DL80 (which = also isn't that fast...)

And this one is a longer = output, without success to that point (2 minutes without completing the DHC= P flow):

For the latt= er, here's an excerpt from the DHCP log:

It seems the DHCP state always = gets reset to IF_DHCP_UNRESOLVED even if there's answers from the DHCP serv= er.

Here's another, longer console log, which = succeeded after spending 236 seconds in the loop:

Any ideas about this?

------=_Part_191442_1545221852.1679001265315--