pw operations slow under zfs load

[javocado]

Hi,

We have a pretty busy ZFS pool running on an 8.3 AMD system. We are
noticing that when the pool is busy pw-related operations seem to  take a
long time to complete:

# time pw unlock 1000
0.007u 0.036s 0:39.72 0.0%      45+1953k 0+113io 0pf+0w

# time pw lock 1000
0.032u 0.022s 1:09.63 0.0%      24+1132k 0+114io 0pf+0w

Wile the command is running, we note that the process is locked in the D
state:

root  85051  0.0  0.0  5832   960   0  D+    1:53PM   0:00.02
/usr/sbin/pwd_mkdb -u 1000 /etc/master.passwd

We also note that there is next to 0 disk activity on the boot volume:

# gstat -f ad

dT: 1.005s  w: 1.000s  filter: ad
 L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w   %busy Name
    0      0      0      0    0.0      0      0    0.0    0.0| ad6
    0      0      0      0    0.0      0      0    0.0    0.0| ad8
    0      0      0      0    0.0      0      0    0.0    0.0| ad10
    0      0      0      0    0.0      0      0    0.0    0.0| ad12

And plenty of free mem:

Mem: 400M Active, 3391M Inact, 128G Wired, 1935M Cache, 14G Buf, 6055M Free

So, what's going on here? How does a busy pool with it's own set of drives
(which operate off an HBA) affect the speed of operations involving the
boot volume (a UFS-formatted SSD connected to the mobo)?