freebsd on opensolaris dom0

Fri Jun 5 05:01:04 UTC 2009

Adrian Chadd wrote:
> 2009/6/5 Bruno Damour <llama at ruomad.net>:
>   
>> Hello,
>> I have tried for some time to get to install freeBSD on an up-to-date (SXCE
>> b114) opensolaris xvm box  (xen 3.1.4) which already has had success in
>> running in a quite stable xay NetBSD and Gentoo Linux as PV domUs and
>> different Windows flavours (Server 2008, Win7 RC, Xp, Server 2003) as HVM
>> domUs.
>>     
>
> G'day!
>
> I've been meaning to trial FreeBSD under Opensolaris/Xen. Thanks for
> trying it out!
>
>   
>> However, freeBSD does not boot as an HVM host and xen-kernel have been
>> impossible to find.
>>     
>
> It should boot as a HVM host. Please post further details. I admit I
> haven't even looked at HVM support yet as I just use PVM exclusively
> here.
>
>   
>> So I'm glad eventually Adrian put some kernel/images, which I have tried to
>> test.
>> My goal is to be able to install and run a current freebsd.
>>
>> It has been a bit hacky, but it boots allright with adrians kernel and image
>> as xbd0. I've even succeeded to install a 7.2-RELEASE via sysinstall/ftp to
>> a zvol (under xbd2).
>>     
>
> Could you please post what you've done here?
>
>   
>> However, even if ftp install works, network traffic in general (ftp to dom0,
>> ssh, csup) panics the domu and in about 1 case of 4 even freezes the dom0.
>>     
>
> I'll upload a more recent filesystem image now for you to test. It
> should be more stable. I'd like to fix all all of the DomU panics as
> soon as possible. I'd also be curious about the DomU messing up the
> Dom0 - I've seen it happen before locally using Linux. I think the
> various Xen related groups out there would love to hear about any
> issues which upset dom0 stability.
>
>   
>> It might be related to my dom0 being an amd64 (intel core2 quad E6600)
>> whereas adrian's system  is i386 ?
>>     
>
> Later versions of the Xen hypervisor should support 32-bit PVM on
> 64-bit hypervisor/dom0.
>
>   
>> I cannot even try to build a new kernel because :
>> - there is no xen in 7.2-RELEASE which I can isntall
>> - updating via csup to current does not work (as mentioned)
>> - I cannot find a way to install a current snapshot as I cannot boot from
>> the downloaded iso images i have (no cdrom, Adrian's kernel only boots  from
>> root image)
>> - I have no bare metal freebsd host I can use to build a different kernel
>>
>> Any ideas ?
>>     
>
> I've built a more recent -current image here to do tests with. I'll
> upload it as a filesystem image for you to do testing with.
>
> I'd like whatever information you can provide about the domU panic. It
> should dump you into the kernel debugger. Please type "bt" at the
> (ddb) prompt and then provide me with a copy of the domU console
> output. Don't kill the domain either; I may ask for further ddb
> commands.
>
> My goal is to try and sort out the majority of stability issues in the
> single vcpu DomU case before 8.0-RELEASE. I don't have any plans (ie,
> I just can't dedicate any further unpaid time) to work on making 64
> bit support function or making the installation process easier.
>   
Thanks for your quick reply !

Here is a summary of the process to install on a zvol

    <xen config : dworkin.cfg
    [...]
    bootloader = "/usr/lib/xen/bin/pygrub"
    <kernel from adrian chadd>
    kernel = "/boot/kernel/kernel"
    extra = "vfs.root.mountfrom=ufs:xbd0"
    extra += ",boot_verbose=yes,kern.hz=256"
    [...]
    disk = [
        'file:/tank/xvm/FreeBSD/root-head-r192577.img,0xCA00,w',
        'phy:/dev/zvol/dsk/tank/dworkin.raw,0xCA20,w'
        ]
    <NB: important to use xbd2 not xbd1, it would be used as swap>

    # zfs create -V 15G tank/dworkin.raw
    # xm new dworkin.cfg
    <in xterm to get menu right>
    # xm start dworkin -c
    <boot>

    # sysinstall
    < ->Index->Partition
        xbd2, auto, write
        ->Exit Install>
    # halt
    # xm start dworkin -c
    </dev/xbd2s1 shows up>
    # sysinstall
    < ->Index->Label
        xbd1, auto, write (ignore errors)
        ->Exit Install>
    # halt
    <halt fails : # xm shutdown dworkin from Dom0)

    # xm start dworkin -c
    <all new slices show up>
    # mkdir /mnt/freebsd
    # newfs /dev/xbd2s1a
    # mount -t ufs /dev/xbd2s1a /mnt/freebsd/
    # mkdir /mnt/freebsd/var
    # newfs /dev/xbd2s1d
    # mount -t ufs /dev/xbd2s1d /mnt/freebsd/var/
    # mkdir /mnt/freebsd/tmp
    # newfs /dev/xbd2s1e
    # mount -t ufs /dev/xbd2s1e /mnt/freebsd/tmp
    # mkdir /mnt/freebsd/usr
    # newfs /dev/xbd2s1f
    # mount -t ufs /dev/xbd2s1f /mnt/freebsd/usr

    # swapon /dev/xbd2s1b

    # ifconfig xn0 inet 192.168.0.196 && route add default 192.168.0.1
    # echo domain ruomad.local >/etc/resolv.conf
    # echo nameserver 192.168.0.1 >>/etc/resolv.conf
    # cat /etc/resolv.conf
    # ping www.freebsd.org
    <network OK>

    # cd /mnt/freebsd
    # sysinstall
    <->Standard->
    sysinstall get all, install to /mnt/freebsd, tem /mnt/tmp, IPv6, >

    <reboot>
    cd /mnt/freebsd
    tweak /etc/ttys (add xc0), /etc resolv.conf, /etc/fstab
    <halt>
    change dworkin.cfg with :
    extra = "vfs.root.mountfrom=ufs:xbd2s1a"
    it boots !!! ;-) but I cannot cvs

And here is the panic.
This one is directly after booting to your kernel and image and cd to a 
free space on the freshly mounted zvol (as above).

    csup /root/stable-supfile
    Connected to 64.74.207.33
    Kernel page fault with the following non-sleepable locks held:
    exclusive sleep mutex xennetif_tx (network transmit lock) r = 0
    (0xc39440a0) locked @
    /home/adrian/work/freebsd/xen/svn/head/sys/dev/xen/netfront/netfront.c:1118
    KDB: stack backtrace:
    X_db_sym_numargs(c0360308,c3527ab8,c0111ac5,c0383ef6,45e,...) at
    X_db_sym_numargs+0x146
    kdb_backtrace(c0383ef6,45e,ffffffff,c0511ef4,c3527af0,...) at
    kdb_backtrace+0x29
    witness_display_spinlock(c036278d,c3527b04,4,1,0,...) at
    witness_display_spinlock+0x75
    witness_warn(5,0,c038c634,c3527b60,c,...) at witness_warn+0x1fd
    trap(c3527b8c) at trap+0x13e
    alltraps(c39440a0,0,c0383ef6,45e,6450ffff,...) at alltraps+0x1b
    xlvbd_add(c3944000,c3527cc8,c00c3814,c03d5d00,c3783738,...) at xlvbd_ad
    intr_event_execute_handlers(c37097ec,c3783700,c0358a72,4e9,c3783770,...
    _event_execute_handlers+0x125
    intr_event_add_handler(c3708bf0,c3527d38,c03587a5,336,c37097ec,...) at
    t_add_handler+0x41f
    fork_exit(c00afcd0,c3708bf0,c3527d38) at fork_exit+0xb8
    fork_trampoline() at fork_trampoline+0x8
    --- trap 0, eip = 0, esp = 0xc3527d70, ebp = 0 ---

    Fatal trap 12: page fault while in kernel mode
    cpuid = 0; apic id = 00
    fault virtual address   = 0x3
    fault code              = supervisor read, page not present
    instruction pointer     = 0x21:0xc0300ad9
    stack pointer           = 0x29:0xc3527bcc
    frame pointer           = 0x29:0xc3527bfc
    code segment            = base 0x0, limit 0xf9800, type 0x1b
                            = DPL 1, pres 1, def32 1, gran 1
    processor eflags        = interrupt enabled, resume, IOPL = 0
    current process         = 12 (irq135: xn)
    [thread pid 12 tid 100024 ]
    Stopped at      xlvbd_add+0x2039:       cmpl    $0,0(%edi)
    db>

And here is the only message I have got about Dom0 crash

    Jun  5 01:43:57 amber xnb: [ID 703724 kern.warning] WARNING:
    xnb_from_peer: failed to map buffer: -3

I will try your knew image this evening (after work ;-)
Thanks a lot

Bruno