FreeBSD/ARM64 on Xen/ARM64

Personally, I think merely being able to boot FreeBSD and Linux on a
Raspberry PI 4B is boring.  What I want is to boot both FreeBSD and Linux
on a Raspberry PI 4B *at the same time*.  I've got it operational, though
it has only received light testing so far.

Since it is the way to present things:
https://gitlab.com/ehem/freebsd-src.git

The branches of note are "main", "submit", "phase1" and "phase2".  There
are some other branches distantly related to Xen.  Also note those
branches are different spots in a series, "submit" contains everything.

"main" is things which have presently been okayed in e-mail, but await
some testing on x86 by Roger Pau Monné.  A key item here is an interface
point for non-x86 interrupt implementations has been added.  With this no
more changes to common code should be needed.

"phase1" is a child of main.  This is 4 commits which are approved, but
3 may have committers and the 4th may become unnecessary.

"phase2" is a child of phase1.  This is roughly the approach Julien Grall
used for his proof of concept in 2015.  Rather a lot of work has been
done to clean things up and update to match the current FreeBSD.  All of
this needs approval, but it has been tested and it known to work (for me,
at least).

"submit" is a child of phase2.  This is a batch attempting to go after
some outstanding issues.  The situation for these is interesting, they
may be needed for context.


One outstanding issue is where/when to probe for Xen.  The presently
tested approach is to probe during the console probe.  The reason is
this is the first place where Xen's presence/absence needs to be known
(otherwise no console is available).

Roger Pau Monné dislikes the approach, but is okay with it.  As such I
would need suggestions for alternative probe places.  The constraints
are, this needs to be BEFORE the console probe, this needs to be AFTER
passed-in device-trees have been parsed.


When Julien Grall did his proof of concept he implemented it on top of
the kernel event interface.  This seems reasonable for 2015, but there
seems to be a chorus insisting upon reimplementing on top of INTRNG.

I've been taking some looks at INTRNG and I think I've finally run into
the documentation /I/ needed.  There is a major problem with INTRNG
though which needs fixing.

If reimplement on top of INTRNG, the straightforward implementation of
xen_arch_intr_release() would be to call intr_isrc_deregister().  If you
examine the source of intr_isrc_deregister() there is a major issue:

xen_arch_intr_release() ->
    intr_isrc_deregister() for non-IPI interrupts (common) ->
        isrc_release_counters() ->
            panic("%s: not implemented", __func__)

Actual hardware hot-plug interrupt controllers are pretty uncommon due to
their price.  For interrupt controllers implemented as software, hot-plug
is a cheap feature.  Xen's event channel is a software interrupt
implementation and interrupts being added or removed is routine.  As such
this is an absolute blocker.

I had been taking a look at fixing isrc_release_counters(), but that is
troublesome.  Any approach is going to involve substantial work in
several places.

A different approach came to mind while looking at this situation though.
isrc_release_counters() is the inverse of isrc_setup_counters().  If you
look, isrc_setup_counters() is actually pretty gross.  It is storing
equivalent values into isrc_index and isrc_count (they are readily
derived from each other).

When you look further, there is even more ugliness.  intrcnt_updatename()
simply copies data from ->isrc_event to intrnames.  Why is a redundant
copy of this data needed?

When you look at the other FreeBSD architectures near-identical ugliness
is present.  Always copying data to intrnames, then having duplicate
fields on the interrupt structures.


Well, "intrnames" and "intrcnt" were added in 2004.  As such due to being
around a long time there would be dozens, likely hundreds or even
thousands of uses if it was a crucial feature.

When I looked, what did I find?  In the FreeBSD kernel "intrnames" and
"intrcnt" are used in a grand total of 2 places.  First,
sys/kern/kern_intr.c uses them for the "hw.intrnames" and "hw.intrcnt"
sysctl()s.  Second, sys/kern/kern_clock.c uses them for the
watchdog_fire() function.

After 18 years there are a grand total of 2 uses.  Removing the uses
resulted in a net removal of more than 200 lines of source code.  This
has the characteristics of an albatross, not a valuable feature.  Likely
looked good in 2004, but now...


As such a blocking feature for use of INTRNG is D36610.  Is there a
reviewer in the house?  I'm guessing the initial reimplementation of
watchdog_fire() will need a different approach, but I'm wondering what
reviewers would prefer.

D36610 is the large batch near the tip of the "submit" branch.



Anything else for the portion between "phase1" and "phase2"?


-- 
(\___(\___(\______          --=> 8-) EHM <=--          ______/)___/)___/)
 \BS (    |         ehem+sigmsg@m5p.com  PGP 87145445         |    )   /
  \_CS\   |  _____  -O #include <stddisclaimer.h> O-   _____  |   /  _/
8A19\___\_|_/58D2 7E3D DDF4 7BA6 <-PGP-> 41D1 B375 37D0 8714\_|_/___/5445