FreeBSD on Ryzen

Wed Mar 22 22:41:17 UTC 2017

On 03/22/2017 22:30, Don Lewis wrote:
> I put together a Ryzen 1700X machine over the weekend and installed the
> 12.0-CURRENT r315413 snapshot on it a couple of days ago.  The RAM is
> DDR4 2400.

Very interesting, thank you for sharing!

> First impression is that it's pretty zippy.  Compared to my previous
> fastest machine:
>   CPU: AMD FX-8320E Eight-Core Processor (3210.84-MHz K8-class CPU)
> make -j8 buildworld using tmpfs is a bit more than 2x faster.  Since the
> Ryzen has SMT, it's eight cores look like 16 CPUs to FreeBSD, I get
> almost a 2.6x speedup with -j16 as compared to my old machine.
> 
> I do see that the reported total CPU time increases quite a bit at -j16
> (~19900u) as compared to -j8 (~13600u) so it is running into some
> hardware bottlenecks that are slowing down instruction execution.  It
> could be the resources shared by both SMT threads that share each core,
> or it could be cache or memory bandwidth related.  The Ryzen topology is
> a bit complicated. There are two groups of four cores, where each group
> of four cores shares half of the L3 cache, with a slowish interconnect
> bus between the groups.  This probably causes some NUMA-like issues.  I
> wonder if the ULE scheduler could be tweaked to handle this better.
> 
> % sysctl kern.sched.topology_spec
> kern.sched.topology_spec: <groups>
>  <group level="1" cache-level="0">
>   <cpu count="16" mask="ffff,0,0,0">0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15</cpu>
>   <children>
>    <group level="2" cache-level="3">
>     <cpu count="8" mask="ff,0,0,0">0, 1, 2, 3, 4, 5, 6, 7</cpu>
>     <children>
>      <group level="3" cache-level="2">
>       <cpu count="2" mask="3,0,0,0">0, 1</cpu>
>      </group>
>      <group level="3" cache-level="2">
>       <cpu count="2" mask="c,0,0,0">2, 3</cpu>
>      </group>
>      <group level="3" cache-level="2">
>       <cpu count="2" mask="30,0,0,0">4, 5</cpu>
>      </group>
>      <group level="3" cache-level="2">
>       <cpu count="2" mask="c0,0,0,0">6, 7</cpu>
>      </group>
>     </children>
>    </group>
>    <group level="2" cache-level="3">
>     <cpu count="8" mask="ff00,0,0,0">8, 9, 10, 11, 12, 13, 14, 15</cpu>
>     <children>
>      <group level="3" cache-level="2">
>       <cpu count="2" mask="300,0,0,0">8, 9</cpu>
>      </group>
>      <group level="3" cache-level="2">
>       <cpu count="2" mask="c00,0,0,0">10, 11</cpu>
>      </group>
>      <group level="3" cache-level="2">
>       <cpu count="2" mask="3000,0,0,0">12, 13</cpu>
>      </group>
>      <group level="3" cache-level="2">
>       <cpu count="2" mask="c000,0,0,0">14, 15</cpu>
>      </group>
>     </children>
>    </group>
>   </children>
>  </group>
> </groups>

The discovered topology looks like what I would expect.  Each complex has its L3
cache and that's reflected above.  L2 and L1 caches are per core (2 hardware
"threads") and that's correct too.
The only thing missing is the SMT flag per each SMT pair.  I think that it does
not have a dramatic effect on scheduling, but I can look into this shortcoming.

> I'll run some tests with SMT disabled when I get a chance.
> 
> The amdtemp driver doesn't attach to this CPU, so I can't monitor the
> temperature at runtime.
> 
> I've got ECC RAM, but unfortunately the motherboard I'm using doesn't
> enable ECC.

Are you absolutely sure of that?

> The chipset is the B350.
> 

Could you please share dmesg from a verbose boot?
At least the portion where CPUs and topology are described.
Thanks!

> Copyright (c) 1992-2017 The FreeBSD Project.
> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
> 	The Regents of the University of California. All rights reserved.
> FreeBSD is a registered trademark of The FreeBSD Foundation.
> FreeBSD 12.0-CURRENT #0 r315413: Thu Mar 16 17:23:31 UTC 2017
>     root at releng3.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64
> FreeBSD clang version 4.0.0 (tags/RELEASE_400/final 297347) (based on LLVM 4.0.0)
> WARNING: WITNESS option enabled, expect reduced performance.
> SRAT: No memory found for CPU 0
> VT(vga): resolution 640x480
> CPU: AMD Ryzen 7 1700X Eight-Core Processor          (3393.71-MHz K8-class CPU)
>   Origin="AuthenticAMD"  Id=0x800f11  Family=0x17  Model=0x1  Stepping=1
>   Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT>
>   Features2=0x7ed8320b<SSE3,PCLMULQDQ,MON,SSSE3,FMA,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
>   AMD Features=0x2e500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,RDTSCP,LM>
>   AMD Features2=0x35c233ff<LAHF,CMP,SVM,ExtAPIC,CR8,ABM,SSE4A,MAS,Prefetch,OSVW,SKINIT,WDT,TCE,Topology,PCXC,PNXC,DBE,PL2I,MWAITX>
>   Structured Extended Features=0x209c01a9<FSGSBASE,BMI1,AVX2,SMEP,BMI2,RDSEED,ADX,SMAP,CLFLUSHOPT,SHA>
>   XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES>
>   SVM: NP,NRIP,VClean,AFlush,DAssist,NAsids=32768
>   TSC: P-state invariant, performance statistics
> real memory  = 68719476736 (65536 MB)
> avail memory = 66671529984 (63582 MB)
> Event timer "LAPIC" quality 100
> ACPI APIC Table: <ALASKA A M I>
> FreeBSD/SMP: Multiprocessor System Detected: 16 CPUs
> FreeBSD/SMP: 1 package(s) x 16 core(s)
> random: unblocking device.
> ACPI BIOS Warning (bug): Optional FADT field Pm2ControlBlock has valid Length but zero Address: 0x0000000000000000/0x1 (20170303/tbfadt-796)
> ioapic0: Changing APIC ID to 17
> ioapic1: Changing APIC ID to 18
> ioapic0 <Version 2.1> irqs 0-23 on motherboard
> ioapic1 <Version 2.1> irqs 24-55 on motherboard
> SMP: AP CPU #12 Launched!
> SMP: AP CPU #1 Launched!
> SMP: AP CPU #4 Launched!
> SMP: AP CPU #7 Launched!
> SMP: AP CPU #6 Launched!
> SMP: AP CPU #3 Launched!
> SMP: AP CPU #11 Launched!
> SMP: AP CPU #5 Launched!
> SMP: AP CPU #15 Launched!
> SMP: AP CPU #9 Launched!
> SMP: AP CPU #10 Launched!
> SMP: AP CPU #8 Launched!
> SMP: AP CPU #2 Launched!
> SMP: AP CPU #14 Launched!
> SMP: AP CPU #13 Launched!
> Timecounter "TSC-low" frequency 1696854133 Hz quality 1000
> random: entropy device external interface
> netmap: loaded module
> module_register_init: MOD_LOAD (vesa, 0xffffffff80f49c70, 0) error 19
> random: registering fast source Intel Secure Key RNG
> random: fast provider: "Intel Secure Key RNG"
> kbd1 at kbdmux0
> nexus0
> vtvga0: <VT VGA driver> on motherboard
> cryptosoft0: <software crypto> on motherboard
> acpi0: <ALASKA A M I> on motherboard
> acpi0: Power Button (fixed)
> cpu0: <ACPI CPU> on acpi0
> cpu1: <ACPI CPU> on acpi0
> cpu2: <ACPI CPU> on acpi0
> cpu3: <ACPI CPU> on acpi0
> cpu4: <ACPI CPU> on acpi0
> cpu5: <ACPI CPU> on acpi0
> cpu6: <ACPI CPU> on acpi0
> cpu7: <ACPI CPU> on acpi0
> cpu8: <ACPI CPU> on acpi0
> cpu9: <ACPI CPU> on acpi0
> cpu10: <ACPI CPU> on acpi0
> cpu11: <ACPI CPU> on acpi0
> cpu12: <ACPI CPU> on acpi0
> cpu13: <ACPI CPU> on acpi0
> cpu14: <ACPI CPU> on acpi0
> cpu15: <ACPI CPU> on acpi0
> attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0
> Timecounter "i8254" frequency 1193182 Hz quality 0
> Event timer "i8254" frequency 1193182 Hz quality 100
> atrtc0: <AT realtime clock> port 0x70-0x71 on acpi0
> Event timer "RTC" frequency 32768 Hz quality 0
> hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff irq 0,8 on acpi0
> Timecounter "HPET" frequency 14318180 Hz quality 950
> Event timer "HPET" frequency 14318180 Hz quality 350
> Event timer "HPET1" frequency 14318180 Hz quality 350
> Event timer "HPET2" frequency 14318180 Hz quality 350
> Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
> acpi_timer0: <32-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
> pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
> pci0: <ACPI PCI bus> on pcib0
> pci0: <base peripheral, IOMMU> at device 0.2 (no driver attached)
> pcib1: <ACPI PCI-PCI bridge> at device 1.3 on pci0
> pci1: <ACPI PCI bus> on pcib1
> xhci0: <XHCI (generic) USB 3.0 controller> mem 0xfd6a0000-0xfd6a7fff irq 32 at device 0.0 on pci1
> xhci0: 32 bytes context size, 64-bit DMA
> usbus0 on xhci0
> usbus0: 5.0Gbps Super Speed USB v3.0
> ahci0: <AHCI SATA controller> mem 0xfd680000-0xfd69ffff irq 33 at device 0.1 on pci1
> ahci0: AHCI v1.31 with 8 6Gbps ports, Port Multiplier supported
> ahcich0: <AHCI channel> at channel 0 on ahci0
> ahcich1: <AHCI channel> at channel 1 on ahci0
> ahcich4: <AHCI channel> at channel 4 on ahci0
> ahcich5: <AHCI channel> at channel 5 on ahci0
> pcib2: <ACPI PCI-PCI bridge> irq 34 at device 0.2 on pci1
> pci2: <ACPI PCI bus> on pcib2
> pcib3: <ACPI PCI-PCI bridge> irq 32 at device 0.0 on pci2
> pci3: <ACPI PCI bus> on pcib3
> re0: <RealTek 8168/8111 B/C/CP/D/DP/E/F/G PCIe Gigabit Ethernet> port 0xf000-0xf0ff mem 0xfd500000-0xfd500fff,0xf2100000-0xf2103fff irq 32 at device 0.0 on pci3
> re0: Using 1 MSI-X message
> re0: Chip rev. 0x4c000000
> re0: MAC rev. 0x00000000
> miibus0: <MII bus> on re0
> rgephy0: <RTL8251/8153 1000BASE-T media interface> PHY 1 on miibus0
> rgephy0:  none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow
> re0: Using defaults for TSO: 65518/35/2048
> re0: Ethernet address: 1c:1b:0d:9a:99:cb
> re0: netmap queues/slots: TX 1/256, RX 1/256
> pcib4: <PCI-PCI bridge> irq 33 at device 1.0 on pci2
> pci4: <PCI bus> on pcib4
> pcib5: <PCI-PCI bridge> irq 32 at device 4.0 on pci2
> pci5: <PCI bus> on pcib5
> pcib6: <ACPI PCI-PCI bridge> at device 3.1 on pci0
> pci6: <ACPI PCI bus> on pcib6
> vgapci0: <VGA-compatible display> port 0xe000-0xe07f mem 0xfc000000-0xfcffffff,0xe0000000-0xefffffff,0xf0000000-0xf1ffffff irq 54 at device 0.0 on pci6
> vgapci0: Boot video device
> hdac0: <NVIDIA (0x0be3) HDA Controller> mem 0xfd080000-0xfd083fff irq 55 at device 0.1 on pci6
> pcib7: <ACPI PCI-PCI bridge> at device 7.1 on pci0
> pci7: <ACPI PCI bus> on pcib7
> pci7: <unknown> at device 0.0 (no driver attached)
> pci7: <encrypt/decrypt> at device 0.2 (no driver attached)
> xhci1: <XHCI (generic) USB 3.0 controller> mem 0xfd200000-0xfd2fffff irq 37 at device 0.3 on pci7
> xhci1: 64 bytes context size, 64-bit DMA
> usbus1 on xhci1
> usbus1: 5.0Gbps Super Speed USB v3.0
> pcib8: <ACPI PCI-PCI bridge> at device 8.1 on pci0
> pci8: <ACPI PCI bus> on pcib8
> pci8: <unknown> at device 0.0 (no driver attached)
> ahci1: <AHCI SATA controller> mem 0xfd708000-0xfd708fff irq 42 at device 0.2 on pci8
> ahci1: AHCI v1.31 with 2 6Gbps ports, Port Multiplier supported with FBS
> ahcich8: <AHCI channel> at channel 0 on ahci1
> ahcich9: <AHCI channel> at channel 1 on ahci1
> hdac1: <AMD (0x1457) HDA Controller> mem 0xfd700000-0xfd707fff irq 43 at device 0.3 on pci8
> isab0: <PCI-ISA bridge> at device 20.3 on pci0
> isa0: <ISA bus> on isab0
> acpi_button0: <Power Button> on acpi0
> atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
> atkbd0: <AT Keyboard> irq 1 on atkbdc0
> kbd0 at atkbd0
> atkbd0: [GIANT-LOCKED]
> uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
> ppc0: cannot reserve I/O port range
> hwpstate0: <Cool`n'Quiet 2.0> on cpu0
> ZFS filesystem version: 5
> ZFS storage pool version: features support (5000)
> Timecounters tick every 1.000 msec
> hdacc0: <NVIDIA GT21x HDA CODEC> at cad 0 on hdac0
> hdaa0: <NVIDIA GT21x Audio Function Group> at nid 1 on hdacc0
> pcm0: <NVIDIA GT21x (HDMI/DP 8ch)> at nid 5 on hdaa0
> hdacc1: <NVIDIA GT21x HDA CODEC> at cad 1 on hdac0
> hdaa1: <NVIDIA GT21x Audio Function Group> at nid 1 on hdacc1
> pcm1: <NVIDIA GT21x (HDMI/DP 8ch)> at nid 5 on hdaa1
> hdacc2: <NVIDIA GT21x HDA CODEC> at cad 2 on hdac0
> hdaa2: <NVIDIA GT21x Audio Function Group> at nid 1 on hdacc2
> pcm2: <NVIDIA GT21x (HDMI/DP 8ch)> at nid 5 on hdaa2
> hdacc3: <NVIDIA GT21x HDA CODEC> at cad 3 on hdac0
> hdaa3: <NVIDIA GT21x Audio Function Group> at nid 1 on hdacc3
> pcm3: <NVIDIA GT21x (HDMI/DP 8ch)> at nid 5 on hdaa3
> hdacc4: <Realtek ALC887 HDA CODEC> at cad 0 on hdac1
> hdaa4: <Realtek ALC887 Audio Function Group> at nid 1 on hdacc4
> pcm4: <Realtek ALC887 (Rear Analog)> at nid 20 and 24,26 on hdaa4
> pcm5: <Realtek ALC887 (Front Analog)> at nid 27 and 25 on hdaa4
> pcm6: <Realtek ALC887 (Onboard Digital)> at nid 17 on hdaa4
> ugen1.1: <0x1022 XHCI root HUB> at usbus1
> ugen0.1: <0x1022 XHCI root HUB> at usbus0
> uhub0: <0x1022 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus1
> uhub1: <0x1022 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
> ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
> ada0: <WDC WD10EACS-00ZJB0 01.01B01> ATA8-ACS SATA 2.x device
> ada0: Serial Number WD-WCASJ1778605
> ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
> ada0: Command Queueing enabled
> ada0: 953869MB (1953525168 512 byte sectors)
> WARNING: WITNESS option enabled, expect reduced performance.
> Trying to mount root from zfs:zroot/ROOT/default []...
> Root mount waiting for: usbus1 usbus0
> uhub0: 8 ports with 8 removable, self powered
> uhub1: 22 ports with 22 removable, self powered
> ugen1.2: <vendor 0x058f Mass Storage> at usbus1
> umass0 on uhub0
> umass0: <vendor 0x058f Mass Storage, class 0/0, rev 2.00/1.12, addr 1> on usbus1
> umass0:  SCSI over Bulk-Only; quirks = 0x4101
> umass0:6:0: Attached to scbus6
> da0 at umass-sim0 bus 0 scbus6 target 0 lun 0
> da0: < USB Disk 8.07> Removable Direct Access SPC-2 SCSI device
> da0: Serial Number 5E2CDC72
> da0: 40.000MB/s transfers
> da0: 3840MB (7864320 512 byte sectors)
> da0: quirks=0x2<NO_6_BYTE>
> GEOM: da0: the secondary GPT header is not in the last LBA.
> Root mount waiting for: usbus1
> ugen1.3: <vendor 0x04b3 product 0x3107> at usbus1
> re0: link state changed to DOWN
> ums0 on uhub0
> ums0: <vendor 0x04b3 product 0x3107, class 0/0, rev 1.10/5.00, addr 2> on usbus1
> ums0: 3 buttons and [XYZ] coordinates ID=0
> 

-- 
Andriy Gapon