Poor ZFS performance

[Alex Povolotsky]

Hello

I'm writing lots of (tens of millions) relatively small files, hashing 
them out in three-level directory, 100 entries per level.

Speed is quite unsatisfying, about 2 millions files per day writing in 
20 processes, that is, about one file per second for process.

CPU is not loaded (la 1.34, 0.4%sys, 0.2% intr, 5.7% user)

Disk is used (sysat gives 0.45 MB/s, 89% busy)

atime is off, even sync is disabled.

FreeBSD insta1.inf.sudo.su 11.0-RELEASE-p1 FreeBSD 11.0-RELEASE-p1 #0 
r306420: Thu Sep 29 01:43:23 UTC 2016 
root at releng2.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC  amd64

CPU: Intel(R) Xeon(R) CPU           E5530  @ 2.40GHz (2400.14-MHz 
K8-class CPU)
   Origin="GenuineIntel"  Id=0x106a5  Family=0x6  Model=0x1a  Stepping=5

Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>

Features2=0x9ce3bd<SSE3,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,SSE4.2,POPCNT>
   AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
   AMD Features2=0x1<LAHF>
   VT-x: PAT,HLT,MTF,PAUSE,EPT,VPID
   TSC: P-state invariant, performance statistics
real memory  = 34363932672 (32772 MB)
avail memory = 33271255040 (31729 MB)
Event timer "LAPIC" quality 400
ACPI APIC Table: <080312 APIC1521>
FreeBSD/SMP: Multiprocessor System Detected: 16 CPUs
FreeBSD/SMP: 2 package(s) x 4 core(s) x 2 hardware threads

No L2ARC.

ARC Size:                               60.73%  18.29   GiB
         Target Size: (Adaptive)         60.84%  18.32   GiB
         Min Size (Hard Limit):          12.50%  3.76    GiB
         Max Size (High Water):          8:1     30.12   GiB

ARC Size Breakdown:
         Recently Used Cache Size:       93.56%  17.14   GiB
         Frequently Used Cache Size:     6.44%   1.18    GiB


ARC Efficiency:                                 1.60b
         Cache Hit Ratio:                65.87%  1.05b
         Cache Miss Ratio:               34.13%  544.82m
         Actual Hit Ratio:               62.58%  999.04m

         Data Demand Efficiency:         99.88%  283.23m
         Data Prefetch Efficiency:       31.76%  109.26m

(I've disabled prefetch yesterday with no visible result)

Attempt to restart program with only two processes yields 10 files per 
second in several minutes falling to 5, to 3 and so on.

Zpool status for the pool in question

   pool: Z840TZPB
  state: ONLINE
   scan: none requested
config:

         NAME            STATE     READ WRITE CKSUM
         Z840TZPB        ONLINE       0     0     0
           gpt/Z840TZPB  ONLINE       0     0     0

errors: No known data errors

smartctl

=== START OF INFORMATION SECTION ===
Model Family:     Seagate Archive HDD
Device Model:     ST8000AS0002-1NA17Z
Serial Number:    Z840TZPB
LU WWN Device Id: 5 000c50 093201557
Firmware Version: AR17
User Capacity:    8,001,563,222,016 bytes [8.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5980 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2, ACS-3 T13/2161-D revision 3b
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Sat Jan 21 11:41:30 2017 MSK
SMART support is: Available - device has SMART capability.
SMART support is: Enabled


disk is fresh (420 hours), no errors.

What can I do to improve writing performance?

Alex