i/o is very slow on FreeBSD dom0 with Xen-4.12 and Freebsd-12

Sat Oct 12 16:43:09 UTC 2019

Replying from my phone, sorry for the format.

El ds., 12 d’oct. 2019, 17:32, Brian Buhrow <buhrow at nfbcal.org> va escriure:

>         Hello.  Here are the logs with remapping enabled, plus the i, m
> and z
> xen console commands.
> Here's hoping you can provide a fix soon!  Or, perhaps, a clue on how to
> fix.
> -thanks
> -Brian
>
>
> <loader.conf>
> boot_serial="YES"               # -h: Use serial console
> comconsole_speed="115200"       # Set the current serial console speed
> comconsole_port="0x2f8" # Set the current serial console port
> #console="vidconsole"           # A comma separated list of console(s)
> console="comconsole"            # A comma separated list of console(s)
> geom_mirror_load="YES"          # RAID1 disk driver (see gmirror(8))
> ahci_load="YES"         # ahci driver
> ipmi_load="YES"         # Ripmi driver
> if_tap_load="YES"               # Load bridge driver.
> if_vlan_load="YES"              # Load vlan driver.
> # Turn on Xen (BB 10/08/2019)
> xen_kernel="/boot/xen"
> xen_cmdline="dom0_mem=4096m dom0_max_vcpus=4 dom0=pvh,verbose
> console=com2,vga iommu=verbose,debug sync_console=true com2=115200,8n1
> guest_loglvl=all loglvl=all"
>
> <xen log, with i, m and z commands inserted>
>
> (XEN) Xen version 4.12.1 (buhrow@) (FreeBSD clang version 6.0.1
> (tags/RELEASE_601/final 335540) (based on LLVM 6.0.1)) debug=n  Thu Oct  3
> 11:34:20 PDT 2019
> (XEN) Latest ChangeSet:
> (XEN) Console output is synchronous.
> (XEN) Bootloader: FreeBSD Loader
> (XEN) Command line: dom0_mem=4096m dom0_max_vcpus=4 dom0=pvh,verbose
> console=com2,vga iommu=verbose,debug sync_console=true com2=115200,8n1
> guest_loglvl=all loglvl=all
> (XEN) Xen image load base address: 0
> (XEN) Video information:
> (XEN)  VGA is text mode 80x25, font 8x16
> (XEN)  VBE/DDC methods: none; EDID transfer time: 0 seconds
> (XEN)  EDID info not retrieved because no DDC retrieval method detected
> (XEN) Disc information:
> (XEN)  Found 4 MBR signatures
> (XEN)  Found 4 EDD information structures
> (XEN) Xen-e820 RAM map:
> (XEN)  0000000000000000 - 0000000000098400 (usable)
> (XEN)  0000000000098400 - 00000000000a0000 (reserved)
> (XEN)  00000000000e0000 - 0000000000100000 (reserved)
> (XEN)  0000000000100000 - 00000000bf780000 (usable)
> (XEN)  00000000bf78e000 - 00000000bf790000 type 9
> (XEN)  00000000bf790000 - 00000000bf79e000 (ACPI data)
> (XEN)  00000000bf79e000 - 00000000bf7d0000 (ACPI NVS)
> (XEN)  00000000bf7d0000 - 00000000bf7e0000 (reserved)
> (XEN)  00000000bf7ec000 - 00000000c0000000 (reserved)
> (XEN)  00000000e0000000 - 00000000f0000000 (reserved)
> (XEN)  00000000fee00000 - 00000000fee01000 (reserved)
> (XEN)  00000000ffc00000 - 0000000100000000 (reserved)
> (XEN)  0000000100000000 - 0000001840000000 (usable)
> (XEN) New Xen image base address: 0xbf000000
> (XEN) ACPI: RSDP 000FACE0, 0024 (r2 ACPIAM)
> (XEN) ACPI: XSDT BF790100, 008C (r1 SMCI            20120803 MSFT       97)
> (XEN) ACPI: FACP BF790290, 00F4 (r3 080312 FACP1521 20120803 MSFT       97)
> (XEN) ACPI: DSDT BF7906A0, 6580 (r1  10600 10600000        0 INTL 20051117)
> (XEN) ACPI: FACS BF79E000, 0040
> (XEN) ACPI: APIC BF790390, 011E (r1 080312 APIC1521 20120803 MSFT       97)
> (XEN) ACPI: MCFG BF7904B0, 003C (r1 080312 OEMMCFG  20120803 MSFT       97)
> (XEN) ACPI: SLIT BF7904F0, 0030 (r1 080312 OEMSLIT  20120803 MSFT       97)
> (XEN) ACPI: OEMB BF79E040, 0086 (r1 080312 OEMB1521 20120803 MSFT       97)
> (XEN) ACPI: SRAT BF79A6A0, 01D0 (r1 080312 OEMSRAT         1 INTL        1)
> (XEN) ACPI: HPET BF79A870, 0038 (r1 080312 OEMHPET  20120803 MSFT       97)
> (XEN) ACPI: DMAR BF79E0D0, 0130 (r1    AMI  OEMDMAR        1 MSFT       97)
> (XEN) ACPI: SSDT BF7A1710, 0363 (r1 DpgPmm    CpuPm       12 INTL 20051117)
> (XEN) ACPI: EINJ BF79A8B0, 0130 (r1  AMIER AMI_EINJ 20120803 MSFT       97)
> (XEN) ACPI: BERT BF79AA40, 0030 (r1  AMIER AMI_BERT 20120803 MSFT       97)
> (XEN) ACPI: ERST BF79AA70, 01B0 (r1  AMIER AMI_ERST 20120803 MSFT       97)
> (XEN) ACPI: HEST BF79AC20, 00A8 (r1  AMIER ABC_HEST 20120803 MSFT       97)
> (XEN) System RAM: 98295MB (100654176kB)
> (XEN) SRAT: PXM 0 -> APIC 00 -> Node 0
> (XEN) SRAT: PXM 0 -> APIC 02 -> Node 0
> (XEN) SRAT: PXM 0 -> APIC 04 -> Node 0
> (XEN) SRAT: PXM 0 -> APIC 06 -> Node 0
> (XEN) SRAT: PXM 0 -> APIC 01 -> Node 0
> (XEN) SRAT: PXM 0 -> APIC 03 -> Node 0
> (XEN) SRAT: PXM 0 -> APIC 05 -> Node 0
> (XEN) SRAT: PXM 0 -> APIC 07 -> Node 0
> (XEN) SRAT: PXM 1 -> APIC 10 -> Node 1
> (XEN) SRAT: PXM 1 -> APIC 12 -> Node 1
> (XEN) SRAT: PXM 1 -> APIC 14 -> Node 1
> (XEN) SRAT: PXM 1 -> APIC 16 -> Node 1
> (XEN) SRAT: PXM 1 -> APIC 11 -> Node 1
> (XEN) SRAT: PXM 1 -> APIC 13 -> Node 1
> (XEN) SRAT: PXM 1 -> APIC 15 -> Node 1
> (XEN) SRAT: PXM 1 -> APIC 17 -> Node 1
> (XEN) SRAT: Node 0 PXM 0 0-a0000
> (XEN) SRAT: Node 0 PXM 0 100000-c0000000
> (XEN) SRAT: Node 0 PXM 0 100000000-c40000000
> (XEN) SRAT: Node 1 PXM 1 c40000000-1840000000
> (XEN) NUMA: Allocated memnodemap from 183e1b7000 - 183e1d0000
> (XEN) NUMA: Using 8 for the hash shift.
> (XEN) Domain heap initialised DMA width 32 bits
> (XEN) CPU Vendor: Intel, Family 6 (0x6), Model 26 (0x1a), Stepping 5 (raw
> 000106a5)
> (XEN) found SMP MP-table at 000ff780
> (XEN) DMI present.
> (XEN) Using APIC driver default
> (XEN) ACPI: PM-Timer IO Port: 0x808 (32 bits)
> (XEN) ACPI: SLEEP INFO: pm1x_cnt[1:804,1:0], pm1x_evt[1:800,1:0]
> (XEN) ACPI:             wakeup_vec[bf79e00c], vec_size[20]
> (XEN) ACPI: Local APIC address 0xfee00000
> (XEN) ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
> (XEN) ACPI: LAPIC (acpi_id[0x02] lapic_id[0x10] enabled)
> (XEN) ACPI: LAPIC (acpi_id[0x03] lapic_id[0x02] enabled)
> (XEN) ACPI: LAPIC (acpi_id[0x04] lapic_id[0x12] enabled)
> (XEN) ACPI: LAPIC (acpi_id[0x05] lapic_id[0x04] enabled)
> (XEN) ACPI: LAPIC (acpi_id[0x06] lapic_id[0x14] enabled)
> (XEN) ACPI: LAPIC (acpi_id[0x07] lapic_id[0x06] enabled)
> (XEN) ACPI: LAPIC (acpi_id[0x08] lapic_id[0x16] enabled)
> (XEN) ACPI: LAPIC (acpi_id[0x09] lapic_id[0x01] enabled)
> (XEN) ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x11] enabled)
> (XEN) ACPI: LAPIC (acpi_id[0x0b] lapic_id[0x03] enabled)
> (XEN) ACPI: LAPIC (acpi_id[0x0c] lapic_id[0x13] enabled)
> (XEN) ACPI: LAPIC (acpi_id[0x0d] lapic_id[0x05] enabled)
> (XEN) ACPI: LAPIC (acpi_id[0x0e] lapic_id[0x15] enabled)
> (XEN) ACPI: LAPIC (acpi_id[0x0f] lapic_id[0x07] enabled)
> (XEN) ACPI: LAPIC (acpi_id[0x10] lapic_id[0x17] enabled)
> (XEN) ACPI: LAPIC (acpi_id[0x11] lapic_id[0x90] disabled)
> (XEN) ACPI: LAPIC (acpi_id[0x12] lapic_id[0x91] disabled)
> (XEN) ACPI: LAPIC (acpi_id[0x13] lapic_id[0x92] disabled)
> (XEN) ACPI: LAPIC (acpi_id[0x14] lapic_id[0x93] disabled)
> (XEN) ACPI: LAPIC (acpi_id[0x15] lapic_id[0x94] disabled)
> (XEN) ACPI: LAPIC (acpi_id[0x16] lapic_id[0x95] disabled)
> (XEN) ACPI: LAPIC (acpi_id[0x17] lapic_id[0x96] disabled)
> (XEN) ACPI: LAPIC (acpi_id[0x18] lapic_id[0x97] disabled)
> (XEN) ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])
> (XEN) Overriding APIC driver with bigsmp
> (XEN) ACPI: IOAPIC (id[0x08] address[0xfec00000] gsi_base[0])
> (XEN) IOAPIC[0]: apic_id 8, version 32, address 0xfec00000, GSI 0-23
> (XEN) ACPI: IOAPIC (id[0x09] address[0xfec8a000] gsi_base[24])
> (XEN) IOAPIC[1]: apic_id 9, version 32, address 0xfec8a000, GSI 24-47
> (XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
> (XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
> (XEN) ACPI: IRQ0 used by override.
> (XEN) ACPI: IRQ2 used by override.
> (XEN) ACPI: IRQ9 used by override.
> (XEN) Enabling APIC mode:  Phys.  Using 2 I/O APICs
> (XEN) ACPI: HPET id: 0x8086a301 base: 0xfed00000
> (XEN) PCI: MCFG configuration 0: base e0000000 segment 0000 buses 00 - ff
> (XEN) PCI: MCFG area at e0000000 reserved in E820
> (XEN) PCI: Using MCFG for segment 0000 bus 00-ff
> (XEN) [VT-D]Host address width 40
> (XEN) [VT-D]found ACPI_DMAR_DRHD:
> (XEN) [VT-D]  dmaru->address = fbffe000
> (XEN) [VT-D]drhd->address = fbffe000 iommu->reg = ffff82c00021d000
> (XEN) [VT-D]cap = c90780106f0462 ecap = f020fe
> (XEN) [VT-D] IOAPIC: 0000:f0:1f.7
> (XEN) [VT-D] IOAPIC: 0000:00:13.0
> (XEN) [VT-D]  flags: INCLUDE_ALL
> (XEN) [VT-D]found ACPI_DMAR_RMRR:
> (XEN) [VT-D] endpoint: 0000:00:1d.0
> (XEN) [VT-D] endpoint: 0000:00:1d.1
> (XEN) [VT-D] endpoint: 0000:00:1d.2
> (XEN) [VT-D] endpoint: 0000:00:1d.7
> (XEN) [VT-D] endpoint: 0000:00:1a.0
> (XEN) [VT-D] endpoint: 0000:00:1a.1
> (XEN) [VT-D] endpoint: 0000:00:1a.2
> (XEN) [VT-D] endpoint: 0000:00:1a.7
> (XEN) [VT-D]found ACPI_DMAR_RMRR:
> (XEN) [VT-D] endpoint: 0000:00:1d.0
> (XEN) [VT-D] endpoint: 0000:00:1d.1
> (XEN) [VT-D] endpoint: 0000:00:1d.2
> (XEN) [VT-D] endpoint: 0000:00:1d.7
> (XEN) [VT-D] endpoint: 0000:00:1a.0
> (XEN) [VT-D] endpoint: 0000:00:1a.1
> (XEN) [VT-D] endpoint: 0000:00:1a.2
> (XEN) [VT-D] endpoint: 0000:00:1a.7
> (XEN) [VT-D]found ACPI_DMAR_ATSR:
> (XEN) [VT-D]  atsru->all_ports: 0
> (XEN) [VT-D] bridge: 0000:00:01.0 start=0 sec=1 sub=1
> (XEN) [VT-D] bridge: 0000:00:03.0 start=0 sec=2 sub=2
> (XEN) [VT-D] bridge: 0000:00:05.0 start=0 sec=3 sub=3
> (XEN) [VT-D] bridge: 0000:00:07.0 start=0 sec=4 sub=4
> (XEN) [VT-D] bridge: 0000:00:09.0 start=0 sec=5 sub=5
> (XEN) Failed to get Error Log Address Range.
> (XEN) HEST: Table parsing has been initialized
> (XEN) Using ACPI (MADT) for SMP configuration information
> (XEN) SMP: Allowing 24 CPUs (8 hotplug CPUs)
> (XEN) IRQ limits: 48 GSI, 3040 MSI/MSI-X
> (XEN) CPU0: Intel machine check reporting enabled
> (XEN) Speculative mitigation facilities:
> (XEN)   Hardware features:
> (XEN)   Compiled-in support: SHADOW_PAGING
> (XEN)   Xen settings: BTI-Thunk N/A, SPEC_CTRL: No, Other:
> (XEN)   L1TF: believed vulnerable, maxphysaddr L1D 44, CPUID 40, Safe
> address 10000000000
> (XEN)   Support for HVM VMs: RSB EAGER_FPU
> (XEN)   Support for PV VMs: RSB EAGER_FPU
> (XEN)   XPTI (64-bit PV only): Dom0 enabled, DomU enabled (without PCID)
> (XEN)   PV L1TF shadowing: Dom0 disabled, DomU enabled
> (XEN) Using scheduler: SMP Credit Scheduler rev2 (credit2)
> (XEN) Initializing Credit2 scheduler
> (XEN)  load_precision_shift: 18
> (XEN)  load_window_shift: 30
> (XEN)  underload_balance_tolerance: 0
> (XEN)  overload_balance_tolerance: -3
> (XEN)  runqueues arrangement: socket
> (XEN)  cap enforcement granularity: 10ms
> (XEN) load tracking window length 1073741824 ns
> (XEN) Adding cpu 0 to runqueue 0
> (XEN)  First cpu on runqueue, activating
> (XEN) Platform timer is 14.318MHz HPET
> (XEN) Detected 2266.768 MHz processor.
> (XEN) Initing memory sharing.
> (XEN) alt table ffff82d0804309b8 -> ffff82d0804318d0
> (XEN) Intel VT-d iommu 0 supported page sizes: 4kB.
> (XEN) Intel VT-d Snoop Control enabled.
> (XEN) Intel VT-d Dom0 DMA Passthrough not enabled.
> (XEN) Intel VT-d Queued Invalidation enabled.
> (XEN) Intel VT-d Interrupt Remapping enabled.
> (XEN) Intel VT-d Posted Interrupt not enabled.
> (XEN) Intel VT-d Shared EPT tables not enabled.
> (XEN) I/O virtualisation enabled
> (XEN)  - Dom0 mode: Relaxed
> (XEN) Interrupt remapping enabled
> (XEN) nr_sockets: 3
> (XEN) ENABLING IO-APIC IRQs
> (XEN)  -> Using new ACK method
>

Can you try to add ioapic_ack=old to the xen command line and report if
that makes a difference?

Roger.