CAM disk I/O starvation

[David Wolfskill]

On Tue, Apr 17, 2012 at 09:15:58PM +0200, Gary Jennejohn wrote:
> ...
> I still have the old problem kernel around, but it's probably not
> instrumented for any meaningful diagnoses.
> ...

Several months ago, I was running a set of meaurements (to determine how
performance for a certain task varied when I changed the hardware
configuration of the machine in question).

While it turned out that disk I/O was (surprisingly) not very
significant, I did find that extending the mechanism I had been using to
graph (aggregate) CPU utilization to graph the utilization of each core
was enlightening.

I don't know whether I can release the code or not -- I'll ask -- but
the basic idea is to look at the CPU state counters (they are an ordered
quintuple, for user, nice, system, interrupt, and idle) -- use the
sysctl OIDs idle kern.cp_time for the aggregate of all CPUs; use idle
kern.cp_times for an array of them, one quintuple per core.  I graphed
them using stacked barcharts; I used math/R to handle the graphing.

Since CPU (or any other) utilization only makes sense over an interval,
you also need to choose one; I used 10-second intervals by default.

In any case, even under 7.1, I noticed that one of the cores got the
vast bulk of the interrupt processing.  (I also saw some of the cores go
quite a bit more idle than others, which was fairly curious.)

Anyway: the point of the above rambling is that it isn't necessary
to actually "instrument" the kernel itself: the work I did was
deliberately designed to be able to run on an unmodified FreeBSD
system with no ports, packages, or other 3rd-party software installed
except for lang/perl.  I also tried comparing running under
/usr/bin/time vs. running under my Perl script (which invokes
/usr/bin/time to get the getrusage() info) several times, and found no
statistically significant difference in resource usage -- even when I
reduced the sampling interval down to 1/second.

(A sufficiently motivated & talented individual could probably replace
the Perl script with a shell script.  As it is, the Perl script
fork/execs a shell script to do the interval-sampling.)

Peace,
david
-- 
David H. Wolfskill				david at catwhisker.org
Depriving a girl or boy of an opportunity for education is evil.

See http://www.catwhisker.org/~david/publickey.gpg for my public key.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 196 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-hackers/attachments/20120417/a5112f10/attachment.pgp