Realtime process CPU starvation

[Dieter BSD]

Chris typed:
> I have a process running at realtime priority 0
> under FreeBSD 10.0. The main thread needs to run every 10 ms,

#include <standard_useless_response.h>
FreeBSD is not a true real time OS.  Please submit your patch
to make it one.

> How can a thread with the highest realtime priority not run for such a
> lengthy period?

Some device drivers do things like

for(ever)
  {
  DELAY(MAXINT);
  }

which does wonders for true-real-time jobs.  :-(
(and sucks even for non-real-time jobs)

I have also found that kernel printf(9) can spoil true-real-time jobs.
Hint: log(9) is your friend.

I run a lot of true-real-time jobs (needs something like < 4ms)
on a uniprocessor, and most of the time they run fine.  Even running
several of these jobs at once.  Carefully crafted userland program
running at rtprio 5.  It took a *lot* of work finding the problems and
a lot of kernel hacking (mostly device drivers) to get this far.  This
is with 8.2.  Tried 10.1 but it is severly broken (in non-real-time-ways)
and is unusable.  :-(  But even now I still get a true-real-time failure
occasionally (and data lost forever :-( ) and no clue why.  :-(

> ULE scheduler

Has problems.  I frequently run jobs that should be either cpu-bound
or disk-bound, but observe a significant percentage of cpu idle
time *and* the disk i/o rate is far lower than it should be.  No clue
what is going on there.  Fortunately these jobs are not true-real-time.
I guess I'll have to call these jobs "scheduler-bound".

> I was suspicious of the ZFS filesystem

How much grief would it be to try a different filesystem (e.g. FFS)?
(I run FFS with softdep.)

> During this 500 ms, two of the four CPUs were idle.

Hmmm, could the scheduler be trying to always run your true-real-time
job on the same cpu and it is waiting for that cpu to become free?
Is there a way to ask the scheduler to run your rt job on cpu0 and
zfs on cpu1?  That might help rule zfs in or out as a problem.
(Clearly I am not a scheduler wizard.)

Maybe see if there is a way to instrument the scheduler without having the
instrumentation creating its own rt or scheduling problem?