ZFS regimen: scrub, scrub, scrub and scrub again.

[Zaphod Beeblebrox]

Ok... here's the existing data:

There are 3,236,316 files summing to 97,500,008,691 bytes.  That puts the
"average" file at 30,127 bytes.  But for the full breakdown:

512 : 7758
1024 : 139046
2048 : 1468904
4096 : 325375
8192 : 492399
16384 : 324728
32768 : 263210
65536 : 102407
131072 : 43046
262144 : 22259
524288 : 17136
1048576 : 13788
2097152 : 8279
4194304 : 4501
8388608 : 2317
16777216 : 1045
33554432 : 119
67108864 : 2

I produced that list with the output of ls -R's byte counts, sorted and
then processed with:

(while read num; do count=$[count+1]; if [ $num -gt $size ]; then echo
$size : $count;size=$[size*2]; count=0; fi; done) <imapfilesizelist

... now the new machine has two 2T disks in a ZFS mirror --- so I suppose
it won't waste as much space as a RAID-Z ZFS --- in that files less than
512 bytes will take 512 bytes?  By far the most common case is 2048 bytes
... so that would indicate that a RAID-Z larger than 5 disks would waste
much space.

Does that go to your recomendations on vdev size, then?   To have an 8 or 9
disk vdev, you should be storing at smallest 4k files?