NFS v3 - FreeBSD 12 server connection getting timed out on CentOS 8 client

[parv/freebsd]

On Sun, Aug 22, 2021 at 8:05 PM parv/freebsd wrote:

(I have rearranged my quoted email)

[CentOS 8 NFS v3 client sometimes prints ...]

> Aug 21 08:31:04 [CLIENT] kernel: watchdog: BUG: soft lockup - CPU#0 stuck
> for 22s! [kworker/u24:0:672075]
>
> Aug 21 08:31:04 [CLIENT] kernel: watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u24:10:672632]
>
> ...

> On the [FreeBSD 12.2] server, there is not much CPU load; drive activity
> does not match the
> connection being timed. On [CentOS 8] client, CPU getting stuck messages
> are being printed
> to everyone when trying to write to the NFS server
>
...

> On the server side, I had not seen any output -- from smartctl or nfsd --
> which
> could indicate such a problem.
>
...

> On the server, I have set "vfs.nfsd.debuglevel=2" to see if there would be
> any
> output when the client would complain after connection getting timed out.
>
> How do I go about investigating or solve this?
>
...

I have collected tcpdump output on both server & client ...

  https://github.com/parv/centos8-nfs3-issue/

... but am unable to differentiate between error and normal operation in
the output,
unlike of strace. :-(

When I ran rsync, CentOS sent the message "CPU stuck" around 21:20, so links
around to that time ...

  client output (215 KB),

https://github.com/parv/centos8-nfs3-issue/blob/main/centos8-client.tcpdump.2118-2120#L1623

  server output (414 KB),

https://github.com/parv/centos8-nfs3-issue/blob/main/freebsd12-server.tcpdump.2118-2120#L2621


- parv