Attempting to Get Xen FreeBSD Dom0 working

Mon Dec 15 00:30:30 UTC 2014

On Sun, Dec 14, 2014 at 11:34 PM, David P. Discher <dpd at dpdtech.com> wrote:
>
> First make sure you CPU supports both EPT and has a IOMMU.
>

Yes it does.

I should note theat I've already tested with Xen in this laptop using
Debian.

>
> It sounds like you have Xen to use the serial console.
>
> To /etc/ttys, you need to add:
>
>  xc0 "/usr/libexec/getty Pc"         vt100   on  secure
>
> Also, make sure all the ttyu[0-4] are off.  Recent versions of FreeBSD
> maybe putting onifconsole … and I can't remember all the combinations I
> used, but I had a lot of issues with Xen and FreeBSD fighting for the
> serial ports.
>

Those this make sense:
[miguelc at hpbsd]~% cat /etc/ttys|eg 'ttyuv|xc0'
xc0     "/usr/libexec/getty Pc"         vt100   on  secure
[miguelc at hpbsd]~% cat /etc/ttys | eg 'ttyu|xc0'
ttyu0   "/usr/libexec/getty 3wire"      vt100   onifconsole secure
ttyu1   "/usr/libexec/getty std.9600"   dialup  off secure
ttyu2   "/usr/libexec/getty std.9600"   dialup  off secure
ttyu3   "/usr/libexec/getty std.9600"   dialup  off secure
xc0     "/usr/libexec/getty Pc"         vt100   on  secure

(I've just added xc0 following the guide, not sure if ttyu0 onifconsole  is
correct or should just be off)

>
> Also, make sure /boot/loader.conf is configured for console=“vidconsole"
> on the freebsd side, and that the xen_cmdline should have “console=vga” and
> not =com1.  It’s not explicit, but the Xen kernel creates the “console”.
> The xc0 device (and Xen itself) seems to take care of piping the Dom0
> console over to whatever Xen is using as console.
>

Oh wait... com1 and vga ... I copy pasted from the guide and left it to
"com1".. this must be the issue, let me retry and I'll post some feedback.

>
>
> Other tips, if you are running ZFS, you will probably need to add
> "vm.max_wired=-1” to /etc/sysctl.conf (I’m actually not sure this is valid,
> but if you don’t you’ll run out of wired memory and all the “xl” commands
> will fail. Or limit the size of ARC. )
>

My arc_max is limitted to 2G atm but in any case I've set the vm.max_wired
to -1 and as you say we will see.

>
> You might need to turn off MSI interrupts on AHCI, "hint.ahci.0.msi=0” in
> /boot/loader.conf. However, try both ways (default I believe is =1).  I’m
> running Intel ICH10 and have to disable MSI.  Roger has ICH8 and doesn’t
> seem to have the issue.
>
> Do I have a way to see what ICH is it in freebsd... (I can ofc lookit up
online) its a I7 so I think its ICH10, but not sure.

> The latest issue is very poor network performance (with Intel 82574L) from
> the DomU (guests) over the bridge and to the network.  However, Guests ->
> Dom0 seem ok.
>
>
> -
> David P. Discher
> http://davidpdischer.com/
> AIM: DavidDPD | Y!M: daviddpdz
>
>
>
> On Dec 13, 2014, at 10:51 PM, Miguel Clara <miguelmclara at gmail.com> wrote:
>
> > I was just trying too boot  Freebsd Xen dom0 on a laptop but I just get
> a black screen after the boot process....
> >
> > any idea what it could be?
> >
> > the system boots fine wihtout loading "boot/xen", I'm not sure how to
> get more info with the black scren!
> >
> > thanks
> >
> >
> > Melhores Cumprimentos // Best Regards
> > -----------------------------------------------
> > Miguel Clara
> > IT - Sys Admin & Developer
> > E-mail:    miguelmclara at gmail.com
> > <linkedin.png>         www.linkedin.com/in/miguelmclara/
> >
> > On Tue, Dec 9, 2014 at 7:17 PM, David P. Discher <dpd at dpdtech.com>
> wrote:
> > ah, sorry missed that.   Will try that today.
> >
> > AHCI lasted over night with MSI off.   Something I noticed, is that when
> the AHCI bus was timing out, it looked like the Xen Kernel was re-scanning
> the PCI bus.  (Sorry, didn’t save these logs). I’ve love to dig into this
> further.
> >
> > Please let me know what/where to add some debugging code.
> >
> > -
> > David P. Discher
> > http://davidpdischer.com/
> > AIM: DavidDPD | Y!M: daviddpdz
> > Mobile: 408.368.3725
> >
> >
> >
> > On Dec 9, 2014, at 12:27 AM, Roger Pau Monné <roger.pau at citrix.com>
> wrote:
> >
> > > Hello,
> > >
> > > El 08/12/14 a les 23.45, David P. Discher ha escrit:
> > > <SNIP>
> > >>
> > >>      Sent SIGTERM to all processes
>            │
> > >>      Sent SIGKILL to all
> processes───────────────────────────────────────────────┘
> > >>      Requesting system reboot
> > >>      [ 1157.299205] Restarting system.
> > >>      root at borg:/zdata/debian #
> > >>      root at borg:/zdata/debian #
> > >>      root at borg:/zdata/debian # xl create -c debian.cfg
> > >>      root at borg:/zdata/debian # xl destroy debian
> > >>      xc: error: Could not bounce buffer for version hypercall (35 =
> Resource temporarily unavailabl): Internal error
> > >>      xc: error: Could not bounce buffer for version hypercall (35 =
> Resource temporarily unavailabl): Internal error
> > >>      xc: error: Could not bounce buffer for version hypercall (35 =
> Resource temporarily unavailabl): Internal error
> > >>      xc: error: Could not bounce buffer for version hypercall (35 =
> Resource temporarily unavailabl): Internal error
> > >>      xc: error: Could not bounce buffer for version hypercall (35 =
> Resource temporarily unavailabl): Internal error
> > >>      xc: error: Could not bounce buffer for version hypercall (35 =
> Resource temporarily unavailabl): Internal error
> > >>      libxl: error: libxl.c:658:libxl_list_domain: getting domain info
> list: Resource temporarily unavailable
> > >>      debian is an invalid domain identifier (rc=-5)
> > >>      root at borg:/zdata/debian #
> > >>
> > >> I’m running AHCI with MSI off in the FreeBSD kernel, and so far, so
> good on that front.  The great thing is now I got the Xen console working,
> so can get the debug output.   However the bounce buffer/hypercall issue i
> would think is far more important than MSI interrupts at the monument.
> > >
> > > Glad to know you got it working at the end! I've already pointed this
> > > out in my last email, but did you try to increase vm.max_wired even
> further?
> > >
> > > This usually happens when mlock in
> > > freebsd_privcmd_alloc_hypercall_buffer (xc_freebsd_osdep.c) fails to
> > > wire down the memory that would be used by the hypercalls.
> > >
> > > Roger.
> > >
> >
>
>