Interrupt performance

[Stefan Lambrev]

Hi,

On Feb 1, 2011, at 1:37 PM, Slawa Olhovchenkov wrote:

> On Sat, Jan 29, 2011 at 07:52:11AM +1100, Bruce Evans wrote:
> 
>>>> there are profiling tools that you may decide to run.
>>> 
>>> What tools I can use on amd64?
>>> 
>>> I boot kernel configured with 'config -p'.
>>> Most time in spinlock_exit and acpi_cpu_c1.
>> 
>> Normal profiling works poorly (I see you found my old mail about high
>> resolution profiling).  Linux might be misreporting the overhead for
>> exactly the same reasons that normal profiling works poorly:
>> - the profiling clock frequency of ~1 KHz was adequate for 5 MHz machines
>>   in 1998, but is now too slow.  Statistics clocks are even slower (128
>>   Hz in FreeBSD, and possibly 100 Hz (?) jiffies in Linux).
>> - the statistics clock might be too synchronized with other interrupts.
>>   The above spinlock_exit and acpi_cpu_c1 times indicate that the
>>   statistics clock almost always fires on exit from another spinlock
>>   and/or inside ACPI, for waking up from idle for the latter.  Seeing
>>   lots of exits from spinlocks may indicated that spinlocks are being
>>   used too much.
>> But FreeBSD will report interrupt times and system for non-fast-interrupts
>> to an accuracy of about 1 microsecond, since it doesn't use the
>> statistics clock much for this.  OTOH, for fast interrupts it is typical
>> behaviour in FreeBSD and Linux to not see them at all from the statistics
>> clock interrupt, since they mask all interrupts so they mask the
>> statistics clock interrupt in particular.  In FreeBSD, lots of time
>> apparently spent in spinlock_exit is a typical result of this, or at
>> least similar things, since spinlock_enter masks all interrupts (except
>> in my version of course).  Linux doesn't have fast interrupts in the
>> same way that FreeBSD does, but at least in old versions almost all of
>> its interrupts masked other interrupts a lot.
> 
> I do some more test and build kernel with KTR.
> Now I don't think that inetrrupt overhead on FreeBSD weight: I try
> polling and don't see any difference.
> 
> I see many reported by netperf send errors. I found this
> http://docs.freebsd.org/cgi/mid.cgi?E1Aice9-0002by-00.
> 
> After insert into src/nettest_bsd.c usleep(1000) if ENOBUF I see 53%
> idle and ./loop 2000000000 "Elapsed 15188006 us" -- this near to linux
> (Elapsed 14107670 us).
> 
> 10% of difference may be by more weight network stack (only 32104
> ticks from 126136 in interrupt handler and task switching, and 94032
> -- UDP processing in network stack and passing datagram to driver).
> May be weight SOCKBUF_LOCK/SOCKBUF_UNLOCK and/or
> INP_INFO_RUNLOCK/INP_RUNLOCK.

Try to run with the same network buffers on FreeBSD and Linux.
I think, the default values in freebsd are much, much lower.
Also in the past ENOBUF was not handled properly in linux.

http://wiki.freebsd.org/AvoidingLinuxisms - Do not rely on Linux-specific socket behaviour. In particular, default socket buffer sizes are different (call setsockopt() with SO_SNDBUF and SO_RCVBUF), and while Linux's send() blocks when the socket buffer is full, FreeBSD's will fail and set ENOBUFS in errno.


--
Best Wishes,
Stefan Lambrev
ICQ# 24134177