Anyone using freebsd ZFS for large storage servers?

[Dennis Glatting]

On Thu, 31 May 2012, Oscar Hodgson wrote:

> The subject is pretty much the question.  Perhaps there's a better
> place to be asking this question ...
>
> We have (very briefly) discussed the possibility of using FreeBSD
> pizza boxes as a storage heads direct attached to external JBOD arrays
> with ZFS.  In perusing the list, I haven't stumbled across indications
> of people actually doing this.  External JBODs would be running 24 to
> 48TB each, roughly.  There would be a couple of units.  The pizza
> boxes would be used for computational tasks, and nominally would have
> 8 cores and 96G+ RAM.
>
> Obvious questions are hardware compatibility and stability.  I've set
> up small FreeBSD 9 machines with ZFS roots and simple mirrors for
> other tasks here, and those have been successful so far.
>
> Observations would be appreciated.
>


mc:

real memory  = 120259084288 (114688 MB)
FreeBSD/SMP: Multiprocessor System Detected: 64 CPUs
FreeBSD/SMP: 4 package(s) x 16 core(s)

mc>  zpool list
NAME     SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
disk-1  14.5T  4.95T  9.55T    34%  1.00x  ONLINE  -
disk-2   270G   297M   270G     0%  1.00x  ONLINE  -

disk-1, RAIDz1, uses Hitachi 4TB drives.



iirc:

real memory  = 68719476736 (65536 MB)
FreeBSD/SMP: Multiprocessor System Detected: 32 CPUs
FreeBSD/SMP: 2 package(s) x 16 core(s)

iirc> zpool list
NAME     SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
disk-1  18.1T  6.70T  11.4T    36%  1.00x  ONLINE  -
disk-2  5.44T  3.05G  5.43T     0%  1.00x  ONLINE  -

disk-1, RAIDz1, uses a bunch of 2TB drives


I have another storage server named bd3 that has a RAIDz2 array of 2.5T 
drives (11 of them, IIRC) but it is presently powered down for 
maintenance.


btw:

real memory  = 25769803776 (24576 MB)
FreeBSD/SMP: Multiprocessor System Detected: 12 CPUs
FreeBSD/SMP: 1 package(s) x 6 core(s) x 2 SMT threads

btw> zpool list
NAME     SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
disk-1  9.06T  97.3G  8.97T     1%  1.00x  ONLINE  -
disk-2  9.06T  5.13T  3.93T    56%  1.00x  ONLINE  -

Those are smaller RAIDz1 arrays of 1TB and 2TB drives, IIRC.


I also have three other systems, over clocked to 4GHz with 16GB of RAM and 
presently powered off, each with 3 or 4 2TB disks RAIDz1.


None of these systems have external arrays. The storage systems use common 
technologies, such as NFS, to export their space but their primary mission 
is manipulating (sort-of) big data and crypto attacks, though one is being 
converted to a Hadoop node for experimentation.

I have only had four issues over the past year and a half:

1) It is important to keep your ZFS patches up to date and the firmware in 
you controllers up to date. Failure to do this results in a => :(

2) Under heavy I/O my systems freeze for a few seconds. I haven't looked 
into why but they are completely unresponsive. Note I am also using 
compressed volumes (gzip), which puts a substantual load on the kernel.

3) I have had a number of disk failures -- not too many and not too few. 
These are merely an annoyance with no loss of data.

4) In two systems I use OCZ Revo drives. After several months of operating 
they go Tango Uniform, requiring a system boot where they return from the 
dead. None of my other SSD technologies exhibit the same problem.