Interrupt performance

[Slawa Olhovchenkov]

On Sat, Jan 29, 2011 at 07:52:11AM +1100, Bruce Evans wrote:

> >> there are profiling tools that you may decide to run.
> >
> > What tools I can use on amd64?
> >
> > I boot kernel configured with 'config -p'.
> > Most time in spinlock_exit and acpi_cpu_c1.
> 
> Normal profiling works poorly (I see you found my old mail about high
> resolution profiling).  Linux might be misreporting the overhead for
> exactly the same reasons that normal profiling works poorly:
> - the profiling clock frequency of ~1 KHz was adequate for 5 MHz machines
>    in 1998, but is now too slow.  Statistics clocks are even slower (128
>    Hz in FreeBSD, and possibly 100 Hz (?) jiffies in Linux).
> - the statistics clock might be too synchronized with other interrupts.
>    The above spinlock_exit and acpi_cpu_c1 times indicate that the
>    statistics clock almost always fires on exit from another spinlock
>    and/or inside ACPI, for waking up from idle for the latter.  Seeing
>    lots of exits from spinlocks may indicated that spinlocks are being
>    used too much.
> But FreeBSD will report interrupt times and system for non-fast-interrupts
> to an accuracy of about 1 microsecond, since it doesn't use the
> statistics clock much for this.  OTOH, for fast interrupts it is typical
> behaviour in FreeBSD and Linux to not see them at all from the statistics
> clock interrupt, since they mask all interrupts so they mask the
> statistics clock interrupt in particular.  In FreeBSD, lots of time
> apparently spent in spinlock_exit is a typical result of this, or at
> least similar things, since spinlock_enter masks all interrupts (except
> in my version of course).  Linux doesn't have fast interrupts in the
> same way that FreeBSD does, but at least in old versions almost all of
> its interrupts masked other interrupts a lot.

I do some more test and build kernel with KTR.
Now I don't think that inetrrupt overhead on FreeBSD weight: I try
polling and don't see any difference.

I see many reported by netperf send errors. I found this
http://docs.freebsd.org/cgi/mid.cgi?E1Aice9-0002by-00.

After insert into src/nettest_bsd.c usleep(1000) if ENOBUF I see 53%
idle and ./loop 2000000000 "Elapsed 15188006 us" -- this near to linux
(Elapsed 14107670 us).

10% of difference may be by more weight network stack (only 32104
ticks from 126136 in interrupt handler and task switching, and 94032
-- UDP processing in network stack and passing datagram to driver).
May be weight SOCKBUF_LOCK/SOCKBUF_UNLOCK and/or
INP_INFO_RUNLOCK/INP_RUNLOCK.