[patch] ipfw interface tracking and opcode rewriting

[Luigi Rizzo]

On Wed, Apr 24, 2013 at 11:50:48PM +0400, Alexander V. Chernikov wrote:
> On 24.04.2013 23:09, Luigi Rizzo wrote:
> > On Wed, Apr 24, 2013 at 08:46:01PM +0400, Alexander V. Chernikov wrote:
> >> On 24.04.2013 20:23, Luigi Rizzo wrote:
...
> >> Well, actually I'm thinking of the next 2 steps:
> >> 1) making kernel rule header more compact (20 bytes instead of 48) and
> >> making it invisible for userland.
> >> This involves rule counters to be stored separately (and possibly as
> >> pcpu-based ones).
> >> 2) since ruleset is now nearly readonly and more or less compact we can
> >> try to store it in
> >> contiguous address space to optimize cache line usage.
> > certainly a worthwhile goal (also using gleb's new counters)
> > but i suspect that compacting rules are a second order effect.
> > I a bit skeptical they make a big difference on the in-kernel
> > version of ipfw. You might see some difference in the
> My current numbers are ~5mpps of IPv4 forwarding with ipfw turned on (1 
> rule) for vlans over ixgbe, with 60% cpu usage (2xE5646).

yes, but that means about 1mpps per core. the userspace version,
i have been told, reached 14.2mpps with one core when running
on netmap interfaces (single rule). So the impact of 20-30ns
per rule is much higher in the second case.

Speaking of which -- it would be better to report performance
in terms of ns per packet (or per rule), rather than Mpps, because
it is much easier to compare results.
Saying "20% better" or "300Kpps more" requires a lot more
context to understand how much things improved.

> For lagg with 2x ixgbe it is ~7mpps with the same 60% usage.
> (And, say, 70% of CPU usage on our production is ipfw, despite low 
> number of rules).
> > userspace version, which runs on top of netmap.
> We are preparing to move forward in this direction (and thinking of 
> 20-30mpps as our goal).
> (And I hope some changes of kernel-based version can migrate to userland 
> one :))

yes, my goal is to have a single source tree that builds both in kernel
and userspace. The diffs are very small, I just have a little bit
of glue code to hook the packet path and the control socket.

cheers
luigi