Networking issues with FreeBSD HVM + SMP

Mon Aug 1 16:32:49 UTC 2011

On Sun, 2011-07-31 at 20:22 -0700, Ben C. wrote:
> Hello all,
> 
> Thanks for taking time to read this post.  It's a fairly long winded
> tale of confusion, but here goes:
> 
> * I originally installed FreeBSD 8.2-RELEASE fine working under 100%
> HVM.  Using the GENERIC kernel with xen's qemu
> * I trashed the box (kind of willingly, it's been a while since I've
> had the pleasure of FreeBSD and think of it as my first pancake)
> * Reinstalled.  Networking failed.  I had no idea why.  Several weeks
> past, I finally got networking back working after about 20-30 install
> attempts.
> * Setup the box properly.
> * Did some changes throughout the dom0/domu's - I had originally had
> the vcpu's pinned to physical cpu's and all this mojo.  I found that
> my load averages on the dom0 were significantly lower when the vcpu's
> were bound to 'any physical' -- but that's beside the point.  The
> point is, FreeBSD changed vcpu's from 1 to 2 - and networking stopped.
>  * I honestly thought it surely was something else.  I triple-checked
> and confirmed, when vcpu=1, networking worked as expected.  When vcpu
> was >1, networking failed with a vauge .. em0 (or re0, see below) ..
> watchdog timeout spewing all over.
> 
> If you take a look at my configuration below, I originally had
> model=e1000 in the vif, which I *thought* was what made it work
> originally.  I was apprently wrong.  vcpu pinning doesn't matter.
> Essentially, if it has more than 1 (v)cpu's, networking fails.
> 
> The dom0 is running NetBSD -CURRENT [current there is just like
> current here], on an Intel i7.  I could try the -current branch or
> using the xenhvm kernel/drivers.  I'm fairly sure the latter may help
> the situation, but honestly I'm just a bit bothered by the strangeness
> of this "bug".   Does anyone else have any suggestions on what I
> should try next?
> 
> Thanks for your time, Ben
> 
> 
> name ="a5freebsd"
> memory = 2048
> 
> kernel = "/usr/pkg/lib/xen/boot/hvmloader"
> builder='hvm'
> device_model = '/usr/pkg/libexec/qemu-dm'
> 
> disk = [
> 	'phy:/dev/mapper/rxenpool-a5freebsd_root,ioemu:hda,w',
> 	'file:/home/xen/iso/FreeBSD-8.2-RELEASE-amd64-disc1.iso,ioemu:hdc:cdrom,r'
> ]
> boot = 'cd'
> 
> vcpus=2
> #cpus=['2']
> 
> 
> #vif=[ 'type=ioemu,bridge=bridge0,model=e1000,mac=00:16:3e:4f:bb:78' ]
> vif=[ 'type=ioemu,bridge=bridge0,mac=00:16:3e:4f:bb:78' ]
> 
> vnc = 1
> vncdisplay = 2
> vnclisten = "1.2.3.4"
> vncpasswd = "removed"
> 
> on_reboot   = 'restart'
> on_crash    = 'restart'
> 
> usbdevice = 'tablet'

I haven't tried the e1000 emulation on the refernce build Xen images in
the cluster yet, but when I use the stock Generic kernel I seem to be
using re(4).  Not that this really matters, but its a reference point.

Also, I kind of gave up on the NetBSD dom0 a while ago so I could get
stuff working in the cluster.  Currently I'm utilizing a CentOS 5.6 Dom0
with Xen 3.4.3 from http://www.gitco.de/repo/

Here's the config I'm using for ref8-xen64.freebsd.org that seems to be
the most likely thing to work:

#============================================================================
# Python configuration setup for 'xm create'.
# This script sets the parameters used when a domain is created using 'xm create'.
# You use a separate script for each domain you want to create, or 
# you can set the parameters for the domain on the xm command line.
#============================================================================

#----------------------------------------------------------------------------
# Kernel image file.
kernel = "/usr/lib/xen/boot/hvmloader"

#----------------------------------------------------------------------------
# device model to use: only qemu-dm available for now
device_model = '/usr/lib64/xen/bin/qemu-dm'

builder='hvm'

# Initial memory allocation (in megabytes) for the new domain.
memory = 1024

# number of CPUS
vcpus = 2

# A name for your domain. All domains must have different names.
name = "ref8-xen64"
arch = "x86_64"

#Network interface. By default emules a realtek 8139. For a NetBSD guest you
# have to disable re(4) and let rtk attach to use it.
# ne2k_pci emulates a pci ne2000 clone; this his cpu-hungry in dom0
# pcnet emulates a AMD PCnet-PCI controller; but it corrupts packets with
# pcn(4) under NetBSD.
vif = [ 'mac=00:16:3e:00:00:02, bridge=xenbr0, type=ioemu' ]
#vif = [ 'mac=00:16:3e:00:00:02, bridge=xenbr0, type=vbd' ]

# Define the disk devices you want the domain to have access to, and
# what you want them accessible as.
# Each disk entry is of the form phy:UNAME,DEV,MODE
# where UNAME is the device, DEV is the device name the domain will see,
# and MODE is r for read-only, w for read-write.
# For hvm domains you can only use hda to hdd. You can set extra types
# (e.g. cdrom)

disk = [
        'file:/var/virt/FreeBSD-8.2-RELEASE-amd64-disc1.iso,hdc:cdrom,r',
        'file:/var/virt/ref8-xen64.bin,hda,w'
        ]

# floppy images; this doesn't seem to work currently. Use a iso image instead.
#fda = '/home/domains/boot1.fs'

# boot device: a = floppy, c= hard drive, d= cdrom (with the disk entry
# before)
#
# boot CDROM image
#boot='d' 
# boot from DISK file
boot='c'

# By default, 'xm create' will try to open an X window on the current display
# for the virtal framebuffer. You can have the virtal framebuffer in vnc
# instead, and connect using a vnc client (using localhost:$vncdisplay)
# If vncunused is set to 1 (this is the default value), vncdisplay
# will be set to the first unused port; so it's recommended to
vnc = 1
vncdisplay = 2
vncunused = 1
vncpasswd='<redacted>'

#Xen emulates a PS/2 mouse, but the pointer in the guest has difficulties
# tracking the absolute position. Xen can emulate a USB tablet in addition
# to the mouse which will report the absolute position of the pointer,
# and make the mouse much easier to use. 
# 
usb=1
usbdevice='tablet'
#usbdevice='mouse'

serial='pty'
on_reboot='restart'
#============================================================================