Problems Re-Starting mountd

[Andrey Simonenko]

On Fri, Jan 04, 2013 at 08:43:32AM -0800, Tim Gustafson wrote:
> > Can you give example of two lines for two users (four lines in total).
> 
> /export/home/abc -network=1.2.3.4/22
> /export/home/abc -network=5.6.7.8/23
> 
> /export/home/def -network=1.2.3.4/22
> /export/home/def -network=5.6.7.8/23
> 
> > How many file systems are mounted on your system?
> 
> Around 1,400.
> 
> > What are types of these file systems?
> 
> All ZFS.

As I understood each /export/home/* pathname from /etc/exports is
a mount point for ZFS file system.

> 
> > If NFS export settings on your system have -mapall or -maproot,
> > then tell which type of database is used for users and groups names.
> 
> They do not.
> 
> > Give the content of /etc/nsswitch.conf.
> 
> group: files ldap
> passwd: files ldap
> hosts: files dns
> networks: files
> shells: files
> services: compat
> services_compat: nis
> protocols: files
> rpc: files
> 

I created 2000 file systems on ZFS file system backed by vnode md(4)
device.  The /etc/exports file contains 4000 entries like your example.

On 9.1-STABLE mountd spends ~70 seconds in flushing current NFS exports
in the NFS server, parsing data from /etc/exports and loading parsed
data into the NFS server.  ~70 seconds is not several minutes.  Most of
time mountd spends in nmount() system call in "zio->io_cv" lock.

Can you show the output of "truss -fc -o /tmp/output.txt mountd"
(wait wchan "select" state of mountd and terminate it by a signal).
If everything is correct you should see N statfs() calls, N+M nmount()
calls and something*N lstat() calls, where N is the number of /etc/exports
lines, M is the number of mounted file systems.  Number of lstat() calls
depends on number of components in pathnames.

Since truss does not support all needed system calls, I modified it
(src/usr.bin/truss/):

--- syscalls.c.orig	2012-12-10 13:54:44.000000000 +0200
+++ syscalls.c	2013-01-08 16:19:40.000000000 +0200
@@ -194,6 +194,10 @@ struct syscall syscalls[] = {
 	  .args = { { Int, 0 } } },
 	{ .name = "nanosleep", .ret_type = 0, .nargs = 1,
 	  .args = { { Timespec, 0 } } },
+	{ .name = "nmount", .ret_type = 0, .nargs = 3,
+	  .args = { { Ptr, 0 }, { Int, 1 }, { Int, 2 } } },
+	{ .name = "statfs", .ret_type = 0, .nargs = 2,
+	  .args = { { Name | IN, 0 }, { Ptr, 1 } } },
 	{ .name = "select", .ret_type = 1, .nargs = 5,
 	  .args = { { Int, 0 }, { Fd_set, 1 }, { Fd_set, 2 }, { Fd_set, 3 }, { Timeval, 4 } } },
 	{ .name = "poll", .ret_type = 1, .nargs = 3,