Some gruesome moments with performance of FreeBSD at over 20K interfaces

[Vladislav Prodan]

Dear Colleagues!

I had a task, using FreeBSD 10.0-STABLE:
1) Receive 20-30 Q-in-Q VLAN (IEEE 802.1ad ), inside of which 2k-4k vlan (IEEE 802.1Q). Total ~60K vlan
2) To every vlan interface assign ipv4 and ipv6 addresses, define routes to ipv4 and ipv6 addresses on another side of vlan (ip unnumbered), and also prescribe ipv6 network /64 by size through ipv6 address on another side of vlan.
3) Perform routing from the world to all of these ipv4/ipv6 addresses и ipv6 networks inside ~60K vlan



To accomplish the 1st task I have no alternatives to using Netgraph.
I noticed incorrect behavior of ngctl(8) after addition of 560th vlan (bin/187835)
Than speed of addition 4k, 8k, 12k vlans was damnably slow:
	10 minutes for first 4k vlans
	18 minutes for first 5k vlans
	28 minutes for first 6k vlans
	52 minutes for first 8k vlans
Than I added more 4к vlans
	20 minutes - 9500 vlans
	33 minutes - 10500 vlans
	58 minutes - 12к vlans 

In total speed of addition of 4k, 8k, 12k vlans was subsequently 10m/52m/110m
It’s hard to imagine, how many time is needed to add ~60K vlan :(
Process was accelerated a little by shooting off devd, bsnmpd, ntpd services, but it found another problems and limitations.

For example,
a) Service ntpd refuse to start at 12K interfaces:
ntpd[2195]: Too many sockets in use, FD_SETSIZE 16384 exceeded
I remind, that in files /usr/src/sys/sys/select.h and /usr/include/sys/select.h FD_SETSIZE value is only 1024U

b) Service bsnmpd started at 12K interfaces, but immediately loaded CPU at 80-100%

last pid: 64011;  load averages:  1.00,  0.97,  0.90                                             up 0+05:25:39  21:26:36
58 processes:  3 running, 54 sleeping, 1 waiting
CPU: 68.2% user,  0.0% nice, 30.6% system,  1.2% interrupt,  0.0% idle
Mem: 125M Active, 66M Inact, 435M Wired, 200K Cache, 525M Free
ARC: 66M Total, 28M MFU, 36M MRU, 16K Anon, 614K Header, 2035K Other
Swap: 1024M Total, 1024M Free

  PID USERNAME    THR PRI NICE   SIZE    RES STATE    TIME    WCPU COMMAND
63863 root          1  96    0   136M   119M RUN     35:31  79.98% bsnmpd
...

c) Size of fields during output of command netstat(1) - netstat -inW is unsufficient (bin/188153)

d) If indicate in command netstat of interface it’s impossible to understand, which ipv4/ipv6 neworks are indicated here.

# netstat -I ngeth123.223 -nW
Name      Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll
ngeth12  1500 <Link#8187>   08:00:27:cd:9b:8e        0     0     0        1     5     0
ngeth12     - 172.18.206.13 172.18.206.139           0     -     -        0     -     -
ngeth12     - fe80::a00:27f fe80::a00:27ff:fe        0     -     -        1     -     -
ngeth12     - 2001:570:28:1 2001:570:28:140::        0     -     -        0     -     -

e) Very low output of command arp:
# ngctl list | grep ngeth | wc -l
   12003
# ifconfig -a | egrep -e 'inet ' | wc -l
   12007
# time /usr/sbin/arp -na > /dev/null
150.661u 551.002s 11:53.71 98.3%        20+172k 1+0io 0pf+0w


More info at http://freebsd.1045724.n5.nabble.com/arp-8-performance-use-if-nameindex-instead-of-if-indextoname-td5898205.html

After using of patch, speed became acceptable:

# time /usr/sbin/arp -na > /dev/null
0.114u 0.090s 0:00.14 142.8%    20+170k 0+0io 0pf+0w 

I suspect, that output of standard network stack will be too low to accomplish a 3rd task, routing  of ~60K vlan
I have no idea, how to use netmap(4) in this situation :(
Please, help me in fulfillment of assigned task.

P.S. 
Colleague-Linuxoid is adjusting the same task and bragging: 
At Debian, in test (kernel 3.13), 80K vlans arose in 20 minutes. It takes 3 GB RAM. And deleting of these vlans also took 20 minutes.

-- 
Vladislav V. Prodan 
System & Network Administrator 
http://support.od.ua 
+380 67 4584408, +380 99 4060508
VVP88-RIPE