Scalability of ALTQ

[Bernhard Schmidt]

Hi,

we're currently using an old and overloaded packeteer packetshaper for
IP-over-Satellite deployment. Since this business will be expanded in
the next weeks to a level the packeteer cannot cope with anymore I'm
looking for alternatives in the unix-world, especially with pf and ALTQ
since it is impressively easy to configure and maintain.

There currently are four encapsulating units that have 30-40 Mbps IP
bandwidth each. Every unit has up to ten first-level customers with
about 20 second-level customers behind them. Every customer (first and
second level) has a commited rate and a burstable rate. There is no
oversubscribing in this business with regard to the commited rates,
while burstable may be used (up to a paid amount) as long as noone else
needs it.

The address ranges behind those encapsulators tend to be very large (/18
or bigger), with very many concurrent sessions. Due to the transmission
technique used the packet shapers cannot see the traffic coming back
from the customers, sometimes these packets don't even flow through our
site.

a) Is pf/kernel capable of that many queues? The only way I could think
   of copying that commited/burstable model into pf were two queues for
   each level, something like

   altq on fxp0 cbq bandwidth 40Mb queue { cust1, cust2, ... }

   queue cust1 on fxp0 bandwidth 5Mb cbq { cust1_commit }
   queue  cust1_commit on fxp0 bandwidth priority 2 10Mb cbq(borrow) { cust1_sub1, cust1_sub2 }
   queue   cust1_sub1 on fxp0 bandwidth 10Mb cbq { cust1_sub1_commit }
   queue    cust1_sub1_commit on fxp0 priority 2 bandwidth 2Mb cbq(borrow) 
   queue   cust1_sub2 on fxp0 bandwidth 0Mb cbq { cust1_sub2_commit }
   queue    cust1_sub2_commit on fxp0 priority 2 bandwidth 10Mb cbq(borrow)

   and so on, which should simulate the following ruleset

   Encapsulator 1 (40Mb max rate)
    Customer1 (10Mb commited + 5Mb burstable)
     Subcustomer 1 (2Mb commited + 10Mb burstable)
     Subcustomer 2 (10Mb commited (+ 0Mb burstable))

   and so on. Subcustomer1 could even have subcustomers on their own.
   For 200 subcustomers per encapsulation unit this makes more than 400
   queues per box, not talking about handling several encap. units on
   one box. And if one wants to use the nifty pf feature to use another
   queue for interactive traffic we're at twice the size. What about
   adding RED/ECN in this environment, adding additional need for
   resources (I guess).

   Another problem that might occur (I haven't tested it yet, so it is
   just speculation).... assuming the ruleset above, I guess packets
   "borrowing" from their parent class still get the attributes of their
   native class. With sub2 doing 10Mbps commited traffic and sub1 10Mbps
   (2Mbps commited + 8Mbps burst) there would be 20Mbps of traffic
   fighting for 15Mbps bandwidth of the cust1 queue. With all having
   prio 2, sub2 might be dropped below his 10Mbps commited rate.

   After reading up the manpage I believe hsfc could be the solution,
   with something like

   queue cust1 on fxp0 hsfc(realtime(10Mb), upperlimit(15Mb)) { cust1_sub1, cust1_sub2 }
   queue  cust1_sub1 on fxp0 hsfc(realtime(2Mb), upperlimit(12Mb))
   queue  cust1_sub2 on fxp0 hsfc(realtime(10Mb))

   would this help me? Any better ideas?

b) Is pf/kernel capable of handling that many states? I'm not able to
   access the packeteer at the moment, but I think we're way over 10000
   concurrent sessions.

Any idea whether pf would be able to scale at that level? I was thinking
about boxes in the P4 2GHz class, best would be if one could handle
at least two encapsulators.

Even better would be if pf could do this whole thing in a bridged
environment, but this might be too much to ask.

I'm glad to hear any experience with deployments of that size.

Bernhard