ZFS perfomance regression in FreeBSD 12 APLHA3->ALPHA4

[Subbsd]

Hi,

I'm seeing a huge loss in performance ZFS after upgrading FreeBSD 12
to latest revision (r338466 the moment) and related to ARC.

I can not say which revision was before except that the newver.sh
pointed to ALPHA3.

Problems are observed if you try to limit ARC. In my case:

vfs.zfs.arc_max="128M"

I know that this is very small. However, for two years with this there
were no problems.

When i send SIGINFO to process which is currently working with ZFS, i
see "arc_reclaim_waiters_cv":

e.g when i type:

/bin/csh

I have time (~5 seconds) to press several times 'ctrl+t' before csh is executed:

load: 0.70  cmd: csh 5935 [arc_reclaim_waiters_cv] 1.41r 0.00u 0.00s 0% 3512k
load: 0.70  cmd: csh 5935 [zio->io_cv] 1.69r 0.00u 0.00s 0% 3512k
load: 0.70  cmd: csh 5935 [arc_reclaim_waiters_cv] 1.98r 0.00u 0.01s 0% 3512k
load: 0.73  cmd: csh 5935 [arc_reclaim_waiters_cv] 2.19r 0.00u 0.01s 0% 4156k

same story with find or any other commans:

load: 0.34  cmd: find 5993 [zio->io_cv] 0.99r 0.00u 0.00s 0% 2676k
load: 0.34  cmd: find 5993 [arc_reclaim_waiters_cv] 1.13r 0.00u 0.00s 0% 2676k
load: 0.34  cmd: find 5993 [arc_reclaim_waiters_cv] 1.25r 0.00u 0.00s 0% 2680k
load: 0.34  cmd: find 5993 [arc_reclaim_waiters_cv] 1.38r 0.00u 0.00s 0% 2684k
load: 0.34  cmd: find 5993 [arc_reclaim_waiters_cv] 1.51r 0.00u 0.00s 0% 2704k
load: 0.34  cmd: find 5993 [arc_reclaim_waiters_cv] 1.64r 0.00u 0.00s 0% 2716k
load: 0.34  cmd: find 5993 [arc_reclaim_waiters_cv] 1.78r 0.00u 0.00s 0% 2760k

this problem goes away after increasing vfs.zfs.arc_max