domU page faults under heavy network load

Tue Feb 17 12:16:41 PST 2009

Hi,

in the kernel i disabled:

WITNESS
WITNESS_SKIPSPIN

in /etc/make.conf i enabled

DEBUG_FLAGS=-ggdb

To make sure that user/kernel are in sync i just rebuilt/installed both. I furthermore tracked down the problem, 
since it seems to occur only between the dom0 and its domU, but not between the dom0 and a "real" remote host.

FreeBSD 8.0-CURRENT #0 r188718: Tue Feb 17 18:29:43 CET 2009

# dd if=/dev/zero bs=1024k| cpipe -vt  | nc dom0 12345
thru: 5500.000ms at   23.3kB/s (  23.3kB/s avg)  128.0kB
thru: 3200.000ms at   40.0kB/s (  29.4kB/s avg)  256.0kB
...
thru:   0.000ms at    infGB/s ( 398.8kB/s avg)    5.4MB
thru:   0.000ms at    infGB/s ( 408.1kB/s avg)    5.5MB

Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address   = 0xd1d9a500
fault code              = supervisor read, page not present
instruction pointer     = 0x21:0xc02e59b0
stack pointer           = 0x29:0xc332aba8
frame pointer           = 0x29:0xc332abfc
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 1, pres 1, def32 1, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 12 (irq137: xn)
[thread pid 12 tid 100025 ]
Stopped at      xlvbd_add+0x2f70:       movl    0x900(%edx,%eax,4),%edx

Second System (AMD64, CentOS5.2, Xen 3.3.1):

....
thru:   0.000ms at    infGB/s ( 598.1kB/s avg)   82.9MB
thru:   0.000ms at    infGB/s ( 599.0kB/s avg)   83.0MB

Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address   = 0xc433a140
fault code              = supervisor read, page not present
instruction pointer     = 0x21:0xc02db38e
stack pointer           = 0x29:0xc332ab7c
frame pointer           = 0x29:0xc332ab84
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 1, pres 1, def32 1, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 12 (irq137: xn)
[thread pid 12 tid 100025 ]
Stopped at      gnttab_claim_grant_reference+0x2e:      movl    0(%eax,%edx,4),%eax

Best Kai

PS : Between the dom0 and some random system via Gigabit Ethernet i get only 1/4 of the real machine speed

# dd if=/dev/zero bs=1024k | cpipe -vt -b 65535 | nc other 12345
thru: 7003.500ms at    9.1MB/s (   9.1MB/s avg)   64.0MB
thru: 3301.650ms at   19.4MB/s (  12.4MB/s avg)  128.0MB
thru: 1600.800ms at   40.0MB/s (  16.1MB/s avg)  192.0MB
thru: 1800.900ms at   35.5MB/s (  18.7MB/s avg)  256.0MB
thru: 2601.300ms at   24.6MB/s (  19.6MB/s avg)  320.0MB
thru: 1400.700ms at   45.7MB/s (  21.7MB/s avg)  384.0MB
thru: 1200.600ms at   53.3MB/s (  23.7MB/s avg)  448.0MB
thru: 1600.800ms at   40.0MB/s (  25.0MB/s avg)  512.0MB
thru: 1500.750ms at   42.6MB/s (  26.2MB/s avg)  576.0MB
thru: 1900.682ms at   33.7MB/s (  26.8MB/s avg)  640.0MB
thru: 2000.035ms at   32.0MB/s (  27.2MB/s avg)  704.0MB
thru: 2300.000ms at   27.8MB/s (  27.2MB/s avg)  768.0MB
thru: 1700.000ms at   37.6MB/s (  27.8MB/s avg)  832.0MB

----- "Kip Macy" <kmacy at freebsd.org> schrieb:

> Is this a recent version of HEAD? I'm not able to reproduce.
> 
> <...>
> thru: 1947.346ms at   32.9MB/s (  27.8MB/s avg)    3.3GB
> thru: 1946.356ms at   32.9MB/s (  27.9MB/s avg)    3.4GB
> thru: 1946.910ms at   32.9MB/s (  27.9MB/s avg)    3.4GB
> <...>
> 
> Maybe some variation in the options?
> 
> Thanks,
> Kip
> 
> On Tue, Feb 10, 2009 at 11:40 AM, Kai Mosebach <xen at komadev.de> wrote:
> > 1024meg
> >
> >
> > On 2/10/09 8:30 PM, "Kip Macy" <kmacy at freebsd.org> wrote:
> >
> >> How big is the memory allocation to the VM?
> >>
> >> Thanks,
> >> Kip
> >>
> >>
> >> On Tue, Feb 10, 2009 at 7:59 AM, Kai Mosebach <xen at komadev.de>
> wrote:
> >>> Hi,
> >>>
> >>> running r188436M on Xen3.3.1 (Centos5, x86_64). The reason is not
> to clear
> >>> but seems related to heavy load on pipes + memory + network?
> >>> It is repeatable with :
> >>>
> >>> bsd8# dd if=/dev/zero | nc otherhost 12345
> >>>
> >>> Fatal trap 12: page fault while in kernel mode
> >>> cpuid = 0; apic id = 00
> >>> fault virtual address   = 0xd216dd00
> >>> fault code              = supervisor read, page not present
> >>> instruction pointer     = 0x21:0xc02e4d40
> >>> stack pointer           = 0x29:0xc332aba8
> >>> frame pointer           = 0x29:0xc332abfc
> >>> code segment            = base 0x0, limit 0xfffff, type 0x1b
> >>>                        = DPL 1, pres 1, def32 1, gran 1
> >>> processor eflags        = interrupt enabled, resume, IOPL = 0
> >>> current process         = 12 (irq137: xn)
> >>> [thread pid 12 tid 100025 ]
> >>> Stopped at      xlvbd_add+0x2f70:       movl   
> 0x900(%edx,%eax,4),%edx
> >>> db> bt
> >>> <console hangs>
> >>>
> >>>
> >>> On otherhost i get some stuff (between 19 and 62 MB), after nc
> closes down
> >>> eventually
> >>>
> >>> otherhost# nc -l 12345 | cpipe -vt -b 65535 >/dev/null
> >>> ...
> >>> thru: 108930.791ms at  184.3kB/s ( 184.3kB/s avg)   19.6MB
> >>>
> >>> best Kai
> >>> _______________________________________________
> >>> freebsd-xen at freebsd.org mailing list
> >>> http://lists.freebsd.org/mailman/listinfo/freebsd-xen
> >>> To unsubscribe, send any mail to
> "freebsd-xen-unsubscribe at freebsd.org"
> >>>
> >
> >
> >