optimal kernel options for VMWARE guest system
Jeff Dickens
jeff at intranet.seamanpaper.com
Wed Oct 11 11:24:35 PDT 2006
Jeff Dickens wrote:
> John Nielsen wrote:
>> On Tuesday 03 October 2006 12:58, Jeff Dickens wrote:
>>
>>> I have some Freebsd systems that are running as VMware guests. I'd
>>> like
>>> to configure their kernels so as to minimize the overhead on the VMware
>>> host system. After reading and partially digesting the white paper on
>>> timekeeping in VMware virtual machines
>>> (http://www.vmware.com/pdf/vmware_timekeeping.pdf) it appears that I
>>> might want to make some changes.
>>>
>>> Has anyone addressed this issue?
>>>
>>
>> I haven't read the white paper (yet; thanks for the link), but I've
>> had good results with recent -STABLE VM's running under ESX server 3.
>> Some thoughts:
>>
>> As I do on most of my installs, I trimmed down GENERIC to include
>> just the drivers I use. In this case that was mpt for the disk and le
>> for the network (although I suspect forcing the VM to present e1000
>> hardware and then using the em driver would work as well if not better).
>>
>> The VMware tools package that comes with ESX server does a poor job
>> of getting itself to run, but it can be made to work without too much
>> difficulty. Don't use the port, run the included install script to
>> install the files, ignore the custom network driver and compile the
>> memory management module from source (included). If using X.org, use
>> the built-in vmware display driver, and copy the vmmouse driver .o
>> file from the VMware tools dist to the appropriate dir under
>> /usr/X11. Even though the included file is for X.org 6.8, it works
>> fine with 6.9/7.0 (X.org 7.1 should include the vmmouse driver.) Run
>> the VMware tools config script from a non-X terminal (and you can
>> ignore the warning about running it remotely if you're using SSH), so
>> it won't mess with your X display (it doesn't do anything not
>> accomplished above). Then run the rc.d script to start the VMware tools.
>>
>> I haven't noticed any timekeeping issues so far.
>>
>> JN
>> _______________________________________________
>>
> What is the advantage of using the "e1000 hardware", and is this
> documented somewhere? I got the vxn network driver working without
> issues; I just had to edit the .vxn file manually: I'm using the free
> VMware server V1 rather than the ESX server.
>
> ethernet0.virtualDev="vmxnet"
>
> I've got timekeeping running stably on these. I turn on time sync via
> vmware tools in the .vmx file:
>
> tools.syncTime = "TRUE"
>
> and in the guest file's rc.conf start ntpd with flags "-Aqgx &" so it
> just syncs once at boot and exits.
>
> I'm not using X on these. They're supposed to be clean & lean systems
> to run such things as djbdns and qmail. And they do work well.
> My main goal is to reduce the background load on the VMware host
> system so that it isn't spending more time than it has to simulating
> interrupt controllers for the guests. I'm wondering about the
> "disable ACPI" boot option. I suppose I first should figure out how
> to even roughly measure the effect of any changes I might make.
>
Well, I've done some pseudo-scientific measurement on this. I currently
have five freebsd virtual systems running, and one Centos 4 (linux
2.6), This command give some info on the background cpu usage:
(The host is a Centos 3 system, linux 2.4)
[root at otter root]# ps auxww | head -1
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
[root at otter root]# ps auxww | grep vmx
root 18031 12.7 1.5 175440 39916 ? S< Oct09 345:50
/usr/lib/vmware/bin/vmware-vmx -C /var/lib/vmware/Virtual
Machines/Goose/freebsd-6.1-i386.vmx -@ ""
root 18058 12.9 1.4 174772 36916 ? S< Oct09 351:01
/usr/lib/vmware/bin/vmware-vmx -C /var/lib/vmware/Virtual
Machines/Duck/freebsd-6.1-i386.vmx -@ ""
root 18072 16.2 5.5 246372 141776 ? S< Oct09 440:16
/usr/lib/vmware/bin/vmware-vmx -C /var/lib/vmware/Virtual
Machines/BlueJay/freebsd-6.1-i386.vmx -@ ""
root 18086 12.9 1.4 174688 38464 ? S< Oct09 351:47
/usr/lib/vmware/bin/vmware-vmx -C /var/lib/vmware/Virtual
Machines/Heron/freebsd-6.1-i386.vmx -@ ""
root 18100 9.4 4.1 385712 107348 ? S< Oct09 256:25
/usr/lib/vmware/bin/vmware-vmx -C /var/lib/vmware/Virtual
Machines/Newt/freebsd-6.1-i386.vmx -@ ""
root 18139 12.2 2.5 299388 65132 ? S< Oct09 330:35
/usr/lib/vmware/bin/vmware-vmx -C /var/lib/vmware/Virtual
Machines/Centos4/Centos4.vmx -@ ""
root 28930 0.0 0.0 3680 672 pts/3 S 14:08 0:00 grep vmx
[root at otter root]#
As one can see the one called "Newt" is consistently lower in the "%CPU"
column. Curiously enough, this *is* the one I built a custom kernel for.
The config file I used is posted below: Besides commenting out devices
I wasn't using & NFS, etc, I commented out the apic and pctimer
devices. Do you think I'm on the right track for reducing interrupt
frequency?
Also, if I were to want to move this kernel to other FreeBSD systems,
how much has to move, the whole /boot/kernel directory?
Finally I did have to re-run the vmware-config-tools.pl script after
rebuilding the kernel.
newt# cat VMWARE1
#
# GENERIC -- Generic kernel configuration file for FreeBSD/i386
#
# For more information on this file, please read the handbook section on
# Kernel Configuration Files:
#
#
http://www.FreeBSD.org/doc/en_US.ISO8859-1/books/handbook/kernelconfig-config.html
#
# The handbook is also available locally in /usr/share/doc/handbook
# if you've installed the doc distribution, otherwise always see the
# FreeBSD World Wide Web server (http://www.FreeBSD.org/) for the
# latest information.
#
# An exhaustive list of options and more detailed explanations of the
# device lines is also present in the ../../conf/NOTES and NOTES files.
# If you are in doubt as to the purpose or necessity of a line, check first
# in NOTES.
#
# $FreeBSD: src/sys/i386/conf/GENERIC,v 1.429.2.7.2.2 2006/05/01
00:15:12 scottl Exp $
machine i386
cpu I686_CPU
ident VMWARE1
# To statically compile in device wiring instead of /boot/device.hints
#hints "GENERIC.hints" # Default places to look for
devices.
makeoptions DEBUG=-g # Build kernel with gdb(1) debug
symbols
#options SCHED_ULE # ULE scheduler
options SCHED_4BSD # 4BSD scheduler
options PREEMPTION # Enable kernel thread preemption
options INET # InterNETworking
#options INET6 # IPv6 communications protocols
options FFS # Berkeley Fast Filesystem
options SOFTUPDATES # Enable FFS soft updates support
options UFS_ACL # Support for access control lists
options UFS_DIRHASH # Improve performance on big
directories
options MD_ROOT # MD is a potential root device
#options NFSCLIENT # Network Filesystem Client
#options NFSSERVER # Network Filesystem Server
#options NFS_ROOT # NFS usable as /, requires
NFSCLIENT
#options MSDOSFS # MSDOS Filesystem
options CD9660 # ISO 9660 Filesystem
options PROCFS # Process filesystem (requires
PSEUDOFS)
options PSEUDOFS # Pseudo-filesystem framework
options GEOM_GPT # GUID Partition Tables.
options COMPAT_43 # Compatible with BSD 4.3 [KEEP
THIS!]
options COMPAT_FREEBSD4 # Compatible with FreeBSD4
options COMPAT_FREEBSD5 # Compatible with FreeBSD5
options SCSI_DELAY=5000 # Delay (in ms) before probing SCSI
options KTRACE # ktrace(1) support
options SYSVSHM # SYSV-style shared memory
options SYSVMSG # SYSV-style message queues
options SYSVSEM # SYSV-style semaphores
options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time
extensions
options KBD_INSTALL_CDEV # install a CDEV entry in /dev
options AHC_REG_PRETTY_PRINT # Print register bitfields in debug
# output. Adds ~128k to driver.
options AHD_REG_PRETTY_PRINT # Print register bitfields in debug
# output. Adds ~215k to driver.
options ADAPTIVE_GIANT # Giant mutex is adaptive.
#device apic # I/O APIC
# Bus support.
#device eisa
device pci
# Floppy drives
#device fdc
# ATA and ATAPI devices
device ata
device atadisk # ATA disk drives
#device ataraid # ATA RAID drives
device atapicd # ATAPI CDROM drives
#device atapifd # ATAPI floppy drives
#device atapist # ATAPI tape drives
options ATA_STATIC_ID # Static device numbering
# SCSI Controllers
#device ahb # EISA AHA1742 family
#device ahc # AHA2940 and onboard AIC7xxx devices
#device ahd # AHA39320/29320 and onboard AIC79xx devices
#device amd # AMD 53C974 (Tekram DC-390(T))
#device isp # Qlogic family
##device ispfw # Firmware for QLogic HBAs- normally a
module
#device mpt # LSI-Logic MPT-Fusion
##device ncr # NCR/Symbios Logic
#device sym # NCR/Symbios Logic (newer chipsets +
those of `ncr')
#device trm # Tekram DC395U/UW/F DC315U adapters
#device adv # Advansys SCSI adapters
#device adw # Advansys wide SCSI adapters
#device aha # Adaptec 154x SCSI adapters
#device aic # Adaptec 15[012]x SCSI adapters,
AIC-6[23]60.
#device bt # Buslogic/Mylex MultiMaster SCSI adapters
#device ncv # NCR 53C500
#device nsp # Workbit Ninja SCSI-3
#device stg # TMC 18C30/18C50
# SCSI peripherals
#device scbus # SCSI bus (required for SCSI)
#device ch # SCSI media changers
#device da # Direct Access (disks)
#device sa # Sequential Access (tape etc)
#device cd # CD
#device pass # Passthrough device (direct SCSI access)
#device ses # SCSI Environmental Services (and SAF-TE)
# RAID controllers interfaced to the SCSI subsystem
#device amr # AMI MegaRAID
#device arcmsr # Areca SATA II RAID
#device asr # DPT SmartRAID V, VI and Adaptec SCSI RAID
#device ciss # Compaq Smart RAID 5*
#device dpt # DPT Smartcache III, IV - See NOTES for
options
#device hptmv # Highpoint RocketRAID 182x
#device rr232x # Highpoint RocketRAID 232x
#device iir # Intel Integrated RAID
#device ips # IBM (Adaptec) ServeRAID
#device mly # Mylex AcceleRAID/eXtremeRAID
#device twa # 3ware 9000 series PATA/SATA RAID
# RAID controllers
#device aac # Adaptec FSA RAID
#device aacp # SCSI passthrough for aac (requires CAM)
#device ida # Compaq Smart RAID
#device mfi # LSI MegaRAID SAS
#device mlx # Mylex DAC960 family
#device pst # Promise Supertrak SX6000
#device twe # 3ware ATA RAID
# atkbdc0 controls both the keyboard and the PS/2 mouse
device atkbdc # AT keyboard controller
device atkbd # AT keyboard
device psm # PS/2 mouse
device kbdmux # keyboard multiplexer
device vga # VGA video card driver
device splash # Splash screen and screen saver support
# syscons is the default console driver, resembling an SCO console
device sc
# Enable this for the pcvt (VT220 compatible) console driver
#device vt
#options XSERVER # support for X server on a vt console
#options FAT_CURSOR # start with block cursor
#device agp # support several AGP chipsets
# Power management support (see NOTES for more options)
#device apm
# Add suspend/resume support for the i8254.
#device pmtimer
# PCCARD (PCMCIA) support
# PCMCIA and cardbus bridge support
#device cbb # cardbus (yenta) bridge
#device pccard # PC Card (16-bit) bus
#device cardbus # CardBus (32-bit) bus
# Serial (COM) ports
#device sio # 8250, 16[45]50 based serial ports
# Parallel port
#device ppc
#device ppbus # Parallel port bus (required)
#device lpt # Printer
#device plip # TCP/IP over parallel
#device ppi # Parallel port interface device
##device vpo # Requires scbus and da
# If you've got a "dumb" serial or parallel PCI card that is
# supported by the puc(4) glue driver, uncomment the following
# line to enable it (connects to the sio and/or ppc drivers):
#device puc
# PCI Ethernet NICs.
#device de # DEC/Intel DC21x4x (``Tulip'')
#device em # Intel PRO/1000 adapter Gigabit
Ethernet Card
#device ixgb # Intel PRO/10GbE Ethernet Card
#device txp # 3Com 3cR990 (``Typhoon'')
device vx # 3Com 3c590, 3c595 (``Vortex'')
# PCI Ethernet NICs that use the common MII bus controller code.
# NOTE: Be sure to keep the 'device miibus' line in order to use these NICs!
device miibus # MII bus support
#device bce # Broadcom BCM5706/BCM5708 Gigabit Ethernet
#device bfe # Broadcom BCM440x 10/100 Ethernet
#device bge # Broadcom BCM570xx Gigabit Ethernet
#device dc # DEC/Intel 21143 and various workalikes
#device fxp # Intel EtherExpress PRO/100B (82557, 82558)
#device lge # Level 1 LXT1001 gigabit Ethernet
#device nge # NatSemi DP83820 gigabit Ethernet
#device nve # nVidia nForce MCP on-board Ethernet
Networking
#device pcn # AMD Am79C97x PCI 10/100(precedence
over 'lnc')
#device re # RealTek 8139C+/8169/8169S/8110S
#device rl # RealTek 8129/8139
#device sf # Adaptec AIC-6915 (``Starfire'')
#device sis # Silicon Integrated Systems SiS 900/SiS
7016
#device sk # SysKonnect SK-984x & SK-982x gigabit
Ethernet
#device ste # Sundance ST201 (D-Link DFE-550TX)
#device ti # Alteon Networks Tigon I/II gigabit
Ethernet
#device tl # Texas Instruments ThunderLAN
#device tx # SMC EtherPower II (83c170 ``EPIC'')
#device vge # VIA VT612x gigabit Ethernet
#device vr # VIA Rhine, Rhine II
#device wb # Winbond W89C840F
#device xl # 3Com 3c90x (``Boomerang'', ``Cyclone'')
# ISA Ethernet NICs. pccard NICs included.
#device cs # Crystal Semiconductor CS89x0 NIC
# 'device ed' requires 'device miibus'
#device ed # NE[12]000, SMC Ultra, 3c503, DS8390 cards
#device ex # Intel EtherExpress Pro/10 and Pro/10+
#device ep # Etherlink III based cards
#device fe # Fujitsu MB8696x based cards
#device ie # EtherExpress 8/16, 3C507, StarLAN 10 etc.
device lnc # NE2100, NE32-VL Lance Ethernet cards
#device sn # SMC's 9000 series of Ethernet chips
#device xe # Xircom pccard Ethernet
# Wireless NIC cards
#device wlan # 802.11 support
#device an # Aironet 4500/4800 802.11 wireless NICs.
#device awi # BayStack 660 and others
#device ral # Ralink Technology RT2500 wireless NICs.
#device wi # WaveLAN/Intersil/Symbol 802.11
wireless NICs.
##device wl # Older non 802.11 Wavelan
wireless NIC.
# Pseudo devices.
device loop # Network loopback
device random # Entropy device
device ether # Ethernet support
#device sl # Kernel SLIP
#device ppp # Kernel PPP
#device tun # Packet tunnel.
device pty # Pseudo-ttys (telnet etc)
device md # Memory "disks"
#device gif # IPv6 and IPv4 tunneling
#device faith # IPv6-to-IPv4 relaying (translation)
# The `bpf' device enables the Berkeley Packet Filter.
# Be aware of the administrative consequences of enabling this!
# Note that 'bpf' is required for DHCP.
device bpf # Berkeley packet filter
# USB support
#device uhci # UHCI PCI->USB interface
#device ohci # OHCI PCI->USB interface
#device ehci # EHCI PCI->USB interface (USB 2.0)
#device usb # USB Bus (required)
##device udbp # USB Double Bulk Pipe devices
#device ugen # Generic
#device uhid # "Human Interface Devices"
#device ukbd # Keyboard
#device ulpt # Printer
#device umass # Disks/Mass storage - Requires scbus and da
#device ums # Mouse
#device ural # Ralink Technology RT2500USB wireless NICs
#device urio # Diamond Rio 500 MP3 player
#device uscanner # Scanners
# USB Ethernet, requires miibus
#device aue # ADMtek USB Ethernet
#device axe # ASIX Electronics USB Ethernet
#device cdce # Generic USB over Ethernet
#device cue # CATC USB Ethernet
#device kue # Kawasaki LSI USB Ethernet
#device rue # RealTek RTL8150 USB Ethernet
# FireWire support
#device firewire # FireWire bus code
#device sbp # SCSI over FireWire (Requires scbus and da)
#device fwe # Ethernet over FireWire (non-standard!)
newt#
More information about the freebsd-questions
mailing list