ixgbe interrupt rate on 11-STABLE

[Babak Farrokhi]

Hi all,

I encountered a new behavior with ixgbe interrupt rates on 11-STABLE which 
was not present in 10-STABLE. 
We disable aim and manually set the interrupt rate. On 10-STABLE the interrupt 
rate will not exceed the configured value, but it is not the case in 11-STABLE.
I am wondering if this is expected behavior or not.

Here is what I see:

# sysctl dev.ix | grep interrupt_rate
dev.ix.1.queue7.interrupt_rate: 500000
dev.ix.1.queue6.interrupt_rate: 500000
dev.ix.1.queue5.interrupt_rate: 100000
dev.ix.1.queue4.interrupt_rate: 500000
dev.ix.1.queue3.interrupt_rate: 500000
dev.ix.1.queue2.interrupt_rate: 500000
dev.ix.1.queue1.interrupt_rate: 500000
dev.ix.1.queue0.interrupt_rate: 500000
dev.ix.0.queue7.interrupt_rate: 500000
dev.ix.0.queue6.interrupt_rate: 500000
dev.ix.0.queue5.interrupt_rate: 100000
dev.ix.0.queue4.interrupt_rate: 5813
dev.ix.0.queue3.interrupt_rate: 500000
dev.ix.0.queue2.interrupt_rate: 500000
dev.ix.0.queue1.interrupt_rate: 500000
dev.ix.0.queue0.interrupt_rate: 500000


Our setup:

# sysctl hw.ix
hw.ix.rxd: 4096
hw.ix.txd: 4096
hw.ix.num_queues: 8
hw.ix.enable_msix: 1
hw.ix.advertise_speed: 0
hw.ix.flow_control: 0
hw.ix.tx_process_limit: -1
hw.ix.rx_process_limit: -1
hw.ix.max_interrupt_rate: 16000
hw.ix.enable_aim: 0


# sysctl dev.ix | grep aim
dev.ix.1.enable_aim: 0
dev.ix.0.enable_aim: 0


# vmstat -ai | grep ix
irq265: ix0:q0                1252479695      10058
irq266: ix0:q1                1232959515       9901
irq267: ix0:q2                1227101328       9854
irq268: ix0:q3                1229457979       9873
irq269: ix0:q4                1237070699       9934
irq270: ix0:q5                1231407684       9889
irq271: ix0:q6                1234032662       9910
irq272: ix0:q7                1230403197       9881
irq273: ix0:link                     107          0
irq274: ix1:q0                2928503268      23517
irq275: ix1:q1                2526264197      20287
irq276: ix1:q2                2543700719      20427
irq277: ix1:q3                2552022026      20494
irq278: ix1:q4                2584441166      20754
irq279: ix1:q5                2563593862      20587
irq280: ix1:q6                2567670402      20619
irq281: ix1:q7                2581949876      20734
irq282: ix1:link                       2          0


Hardware:

# pciconf -lv | grep -A3 ix
ix0 at pci0:5:0:0:	class=0x020000 card=0x7b118086 chip=0x154d8086 rev=0x01 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Ethernet 10G 2P X520 Adapter'
    class      = network
--
ix1 at pci0:5:0:1:	class=0x020000 card=0x7b118086 chip=0x154d8086 rev=0x01 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Ethernet 10G 2P X520 Adapter'
    class      = network


# dmesg | grep ^ix
ix0: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 3.1.13-k> port 0x2020-0x203f mem 0x92300000-0x923fffff,0x92404000-0x92407fff irq 34 at device 0.0 numa-domain 0 on pci5
ix0: Using MSIX interrupts with 9 vectors
ix0: ixgbe_allocate_msix: number of queues (8) != number of RSS buckets (128); performance will be impacted.
ix0: Ethernet address: a0:36:9f:da:05:4c
ix0: PCI Express Bus: Speed 5.0GT/s Width x8
ix0: Error 2 setting up SR-IOV
ix0: netmap queues/slots: TX 8/4096, RX 8/4096
ix1: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 3.1.13-k> port 0x2000-0x201f mem 0x92200000-0x922fffff,0x92400000-0x92403fff irq 37 at device 0.1 numa-domain 0 on pci5
ix1: Using MSIX interrupts with 9 vectors
ix1: ixgbe_allocate_msix: number of queues (8) != number of RSS buckets (128); performance will be impacted.
ix1: Ethernet address: a0:36:9f:da:05:4e
ix1: PCI Express Bus: Speed 5.0GT/s Width x8
ix1: Error 2 setting up SR-IOV
ix1: netmap queues/slots: TX 8/4096, RX 8/4096


Any ideas?

Thanks,
Babak
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 931 bytes
Desc: OpenPGP digital signature
URL: <http://lists.freebsd.org/pipermail/freebsd-net/attachments/20170208/4c709275/attachment.sig>