+12K ZFS file systems or snapshots

[Artis Caune]

Hello FreeBSD-fs,

We want to make ZFS file system for each user:
    # zfs create tank/user_x
    # zfs set quota=1G tank/user_x

We also want to create snapshots for every user:
    user_x at month_ago
    user_x at week_ago
    user_x at day_ago



We have run bonnie tests on amd64, 6G RAM IBM box with 3 SAS drives in 
raidz1 (7.0-BETA3).
Tests failed with "kmem_map too small".

Raising kmem_size to 1500M and limiting arc_max to 512M passed bonnie 
test for 4 days.
'vmstat -m' solaris MemUse is ~600M - 700M


We could not create more than 12K file systems with 'zfs create'.
ZFS panics with "kmem_map too small" when 'vmstat -m' solaris MemUse 
reaches 1500M (kmem_size).

Solaris recommend 1G for every 10K file systems or snapshots, each fs or 
snapshot requires 64K of kmem.


It looks like kmem_size is u_int and is limited to 4G, but we can boot 
only with 1500M. With 2G kmem_size console is full with panics.



We want +/- 10K users on each box. We can:
* create 10K file systems (users), use tank snapshots, not per-user 
snapshots
* create 3K file systems (users) and 3 snapshots per file system
+room for zfs cache


If kmem_size is limited to ~2G, then there is no need for more than 2G 
RAM for ZFS file server?