NFS Performance issue against NetApp
Rick Macklem
rmacklem at uoguelph.ca
Sun May 12 12:48:06 UTC 2013
Marc G. Fournier wrote:
> 'k, here is on Linux ... this is right after rebooting the server,
> doing
> a mount and running the startup once:
>
> Client rpc stats:
> calls retrans authrefrsh
> 40602 0 40609
>
> Client nfs v3:
> null getattr setattr lookup access readlink
> 0 0% 13000 32% 5 0% 6140 15% 6741 16%
> 0 0%
> read write create mkdir symlink mknod
> 3556 8% 6711 16% 3743 9% 307 0% 0 0%
> 0 0%
> remove rmdir rename link readdir readdirplus
> 1 0% 0 0% 0 0% 0 0% 16 0%
> 380 0%
> fsstat fsinfo pathconf commit
> 0 0% 2 0% 1 0% 0 0%
>
> One thing to note is that both Linux/FreeBSD have
> "rsize=65536,wsize=65536" ... but there are 63x as many reads / 34x as
> many writes on FreeBSD as on Linux ... ?
>
> Just noticed this on the FreeBSD stats:
>
> Rpc Info:
> TimedOut Invalid X Replies Retries Requests
> 0 0 0 0 818479
>
> 818k Retries? Is that normal ... ?
>
> Also, the NetApp volumes being used here are not shared ... there are
> no
> other clients mounting these, and the Linux/FreeBSD volumes are
> seperate
> ... same size, same jboss install, same configuration, same war file
> ...
> I could mount /vol/linux_jboss onto the FreeBSD, or /vol/freebsd_jboss
> onto the Linux, and they would load the same way ... in fact, the
> jboss
> install itself was done onto the FreeBSD and copied over to the Linux
> ... and both are using OpenJDK7 ... I tried to make it as identical as
> I
> could ...
>
>
> On 2013-05-11 7:27 PM, Marc G. Fournier wrote:
> >
> > With
> >
> > vfs.nfs.noconsist=3 ... 385595ms
> >
> > nfsstat -z before startup, nfsstat -c after:
> >
> > Client Info:
> > Rpc Counts:
> > Getattr Setattr Lookup Readlink Read Write Create
> > Remove
> > 332594 5 17238 0 224426 231137
> > 3743 1
> > Rename Link Symlink Mkdir Rmdir Readdir
> > RdirPlus Access
> > 0 0 0 307 0 71 0 8447
> > Mknod Fsstat Fsinfo PathConf Commit
> > 0 509 0 0 0
> > Rpc Info:
> > TimedOut Invalid X Replies Retries Requests
> > 0 0 0 0 818479
> > Cache Info:
> > Attr Hits Misses Lkup Hits Misses BioR Hits Misses BioW
> > Hits Misses
> > 608296 332596 526200 17245 -95425 224426 13178
> > 231137
> > BioRLHits Misses BioD Hits Misses DirE Hits Misses Accs
> > Hits Misses
> > 0 0 1050 55 502 7
> > 543340 8448
> >
Ok, so disabling the mtime based cache consistency doesn't make
much difference. Forget about that one.
I've attached another patch (which you probably shouldn't use for
a production system either) to be tried instead of the last one.
(This one is basically "work in progress" by Alexander Kabaev for
better performance during file linking. I hope he doesn't mind
me posting it.)
rick
> >
> > ============
> >
> > vfs.nfs.noconsist=2 ... 392201ms
> >
> > Client Info:
> > Rpc Counts:
> > Getattr Setattr Lookup Readlink Read Write Create
> > Remove
> > 332557 5 17228 0 224421 231131
> > 3743 1
> > Rename Link Symlink Mkdir Rmdir Readdir
> > RdirPlus Access
> > 0 0 0 307 0 72 0 8430
> > Mknod Fsstat Fsinfo PathConf Commit
> > 0 502 0 0 0
> > Rpc Info:
> > TimedOut Invalid X Replies Retries Requests
> > 0 0 0 0 818395
> > Cache Info:
> > Attr Hits Misses Lkup Hits Misses BioR Hits Misses BioW
> > Hits Misses
> > 607834 332557 525801 17231 -95401 224421 13178
> > 231131
> > BioRLHits Misses BioD Hits Misses DirE Hits Misses Accs
> > Hits Misses
> > 0 0 1028 56 502 0
> > 542925 8431
> >
> >
> > ============
> > vfs.nfs.noconsist=0 ... 391622ms
> >
> >
> > Client Info:
> > Rpc Counts:
> > Getattr Setattr Lookup Readlink Read Write Create
> > Remove
> > 236122 5 17221 0 230575 230823
> > 3743 1
> > Rename Link Symlink Mkdir Rmdir Readdir
> > RdirPlus Access
> > 0 0 0 307 0 71 0 8425
> > Mknod Fsstat Fsinfo PathConf Commit
> > 0 516 0 0 0
> > Rpc Info:
> > TimedOut Invalid X Replies Retries Requests
> > 0 0 0 0 727799
> > Cache Info:
> > Attr Hits Misses Lkup Hits Misses BioR Hits Misses BioW
> > Hits Misses
> > 711860 236124 526549 17225 -101525 230490 13178
> > 230823
> > BioRLHits Misses BioD Hits Misses DirE Hits Misses Accs
> > Hits Misses
> > 0 0 1057 55 516 0
> > 543709 8425
> >
> >
> > I checked a second time with nonconsist=0, and the nfsstat -c values
> > seem to come out pretty much the same ...
> >
> > I'm going to head down to the office and try again with Solaris (I'd
> > have to re-install, since I used that system for the Solaris), and
> > see
> > what nfsstat -c results I get out of that ... will post a followup
> > on
> > this when completed ...
> >
> >
> >
> > On 2013-05-10 5:32 PM, Rick Macklem wrote:
> >> Marc G. Fournier wrote:
> >>> FYI … I just installed Solaris 11 onto the same hardware and ran
> >>> the
> >>> same test … so far, I'm seeing:
> >>>
> >>> Linux @ ~30s
> >>> Solaris @ ~44s
> >>>
> >>> OpenBSD @ ~200s
> >>> FreeBSD @ ~240s
> >>>
> >>> I've even tried FreeBSD 8.3 just to see if maybe its as 'newish'
> >>> issue
> >>> … same as 9.x … I could see Linux 'cutting corners', but
> >>> Oracle/Solaris too … ?
> >>>
> >> The three client implementations (BSD, Linux, Solaris) were
> >> developed
> >> independently and, as such, will all implement somewaht different
> >> caching algorithms (the RFCs specify what goes on the wire, but say
> >> little w.r.t. client side caching).
> >>
> >> I have a attached a patch that might be useful for determining if
> >> the client side buffer cache consistency algorithm in FreeBSD is
> >> causing the slow startup of jboss. Do not run this patch on a
> >> production system, since it pretty well disables all buffer cache
> >> coherency (ie. if another client modifies a file, the patched
> >> client
> >> won't notice and will continue to cache stale file data).
> >>
> >> If the patch does speed up startup of jboss significantly, you can
> >> use the sysctl:
> >> vfs.nfs.noconsist
> >> to check for which coherency check is involved by decreasing the
> >> value for the sysctl by 1 and then trying a startup again. (When
> >> vfs.nfs.noconsist=0, normal cache coherency will be applied.)
> >>
> >> I have no idea if buffer cache coherency is a factor, but trying
> >> the attached patch might determine if it is.
> >>
> >> Note that you have never posted updated "nfsstat -c" values.
> >> (Remember that what you posted indicated 88 RPCs, which seemed
> >> bogus.) Finding out if FreeBSD does a lot more of certain RPCs
> >> that Linux/Solaris might help isolate what is going on.
> >>
> >> rick
> >>
> >>> On 2013-05-03, at 04:50 , Mark Felder <feld at feld.me> wrote:
> >>>
> >>>> On Thu, 02 May 2013 18:43:17 -0500, Marc G. Fournier
> >>>> <scrappy at hub.org> wrote:
> >>>>
> >>>>> Hadn't thought to do so with Linux, but …
> >>>>> Linux ……. 20732ms, 20117ms, 20935ms, 20130ms, 20560ms
> >>>>> FreeBSD .. 28996ms, 24794ms, 24702ms, 23311ms, 24153ms
> >>>> Please make sure both platforms are using similar atime settings.
> >>>> I
> >>>> think most distros use ext4 with diratime by default. I'd just do
> >>>> noatime on both platforms to be safe.
> >>>> _______________________________________________
> >>>> freebsd-fs at freebsd.org mailing list
> >>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
> >>>> To unsubscribe, send any mail to
> >>>> "freebsd-fs-unsubscribe at freebsd.org"
> >>> _______________________________________________
> >>> freebsd-fs at freebsd.org mailing list
> >>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
> >>> To unsubscribe, send any mail to
> >>> "freebsd-fs-unsubscribe at freebsd.org"
> >
> > _______________________________________________
> > freebsd-fs at freebsd.org mailing list
> > http://lists.freebsd.org/mailman/listinfo/freebsd-fs
> > To unsubscribe, send any mail to
> > "freebsd-fs-unsubscribe at freebsd.org"
-------------- next part --------------
A non-text attachment was scrubbed...
Name: nfs.diff
Type: text/x-patch
Size: 2682 bytes
Desc: not available
URL: <http://lists.freebsd.org/pipermail/freebsd-fs/attachments/20130512/b46a753a/attachment.bin>
More information about the freebsd-fs
mailing list