Slow nfsd write performance, tweaks needed

[Mehmet Erol Sanliturk]

On Thu, Nov 13, 2014 at 5:00 AM, Rick Macklem <rmacklem at uoguelph.ca> wrote:

> Mark Schouten wrote:
> > Hi,
> >
> >
> > I am in the process of switching from a ZFS On Linux-based NFS-server
> > to a FreeBSD-based NFS-server. The FreeBSD implementation of ZFS is
> > way superiour over ZoL, and the box serves as storage for a
> > virtualizationplatform, so stability is welcome. :)
> >
> >
> > The box is stable, but performs terribly. Surely, I'm doing something
> > wrong, but I would like some tips and tricks to speed things up.
> >
> >
> >
> >
> > Here's my setup:
> > CPU: Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz (HyperThreading is
> > enabled)
> > RAM: 64GB
> > NIC: 2x igb in lagg0 (loadbalancing)
> Oops, I didn't see this before my last post. igb had problems with
> the 64K TSO issue and I'd try to get rid of lagg as well.
> (You might be much better off just using a single net interface
>  without lagg.)
>
> Again, good luck with it, rick
>
> > Disks:
> >
> > export1     1.81T   914G   942G    49%  1.00x  ONLINE  -
> >   mirror     928G   457G   471G         -
> >     da0         -      -      -         -
> >     da1         -      -      -         -
> >   mirror     928G   457G   471G         -
> >     da2         -      -      -         -
> >     da3         -      -      -         -
> >   mirror    9.94G   173M  9.77G         -
> >     da4p1       -      -      -         -
> >     da5p1       -      -      -         -
> > cache           -      -      -      -      -      -
> >   da4p2      223G   223G     8M         -
> >   da5p2      223G   223G     8M         -
> >
> >
> > da0-3 are 1TB WDs
> > da4-5 are 240GB Samsung SSD 840s
> >
> >
> > Here's (related) info from rc.conf.
> >
> > nfs_server_enable="YES"
> > nfs_server_flags="-u -t -n 128"
> > rpcbind_enable="YES"
> > mountd_enable="YES"
> > rpc_lockd_enable="YES"
> > rpc_statd_enable="YES"
> >
> >
> >
> >
> > I have compression enabled on all the ZFS-filesystems, and
> > jumboframes are enabled on the nics.
> >
> >
> > As soon as one of the (Linux) clients start to do some IO, NFS
> > responsetimes go up bigtime (yesterday up to 13 seconds), while the
> > hardware is pretty much idle, I must be doing something very wrong.
> > I'm mostly a Linux-guy, so any hit with a FreeBSD cluebat is
> > appreciated.
> >
> >
> > Regards,
> >
> >
> > --
> > Kerio Operator in de Cloud? https://www.kerioindecloud.nl/
> > Mark Schouten  | Tuxis Internet Engineering
> > KvK: 61527076 | http://www.tuxis.nl/
> > T: 0318 200208 | info at tuxis.nl
>



A terrible write performance may come from Linux mount options in fstab :


There is sync / async for the NFS connection :

I was set sync : A file requiring 30 ( thirty ) SECONDs to write become 30
MINUTEs to write .
After working to solve this problems by trying diffrerent parameter setting
over many days , there only remained sync / async .
When I selected async , writes turned to NORMAL .

I do not know effect of compression ( my opinion is that it will not be
"terrible" ) , but my suggestion is to check as a possible trouble point :

sync / async in Linux computer ( in fstab or mount statement )


Thank you very much .

Mehmet Erol Sanliturk