Test on 10GBE Intel based network card

Tue Aug 4 08:14:49 UTC 2009

Ray,

>To me it looks like interrupt coalescing is not switched on for some reason.
>Are you passing any parameters to the driver in boot.conf.

This is my loader.conf

        kern.ipc.nmbclusters=65635
        kern.hz=1000
        net.bpf_jitter.enable=1
        # net.graph.threads=32
        # if_em_load="YES"

        # NETGRAPH TUNING
        net.graph.maxdata=1024

        kern.ipc.somaxconn=4096
        net.inet.tcp.recvspace=78840
        net.inet.tcp.sendspace=78840
        kern.ipc.shmmax=67108864
        kern.ipc.shmmni=200
        kern.ipc.shmseg=128
        kern.ipc.semmni=70

        net.local.stream.sendspace=82320
        net.local.stream.recvspace=82320
        net.inet.tcp.local_slowstart_flightsize=10
        net.inet.tcp.nolocaltimewait=1
        net.inet.tcp.hostcache.expire=3900

        kern.maxusers=512

        kern.ipc.nmbclusters=32768
        kern.ipc.maxsockets=81920
        kern.ipc.maxsockbuf=1048576

        net.inet.tcp.tcbhashsize=4096
        net.inet.tcp.hostcache.hashsize=1024

>Could you retest with vmstat switched on "vmstat 3" and send us the output.
>I expect we are going to see alot of interrupts.

Sending 535714 pps (64bytes-long)

INTRUDER-64# vmstat 3
 procs      memory      page                    disks     faults         cpu
 r b w     avm    fre   flt  re  pi  po    fr  sr da0 da1   in   sy   cs us sy id
 0 0 0  95420K  7203M    19   0   0   0    17   0   0   0  642   66 1078  0  2 98
 0 0 0  95420K  7203M     0   0   0   0     0   0   2   0   18   65  527  0  0 100
 0 0 0  95420K  7203M     0   0   0   0     0   0   0   0   18   67  527  0  0 100
 0 0 0  95420K  7203M     0   0   0   0     0   0   0   0   17   64  525  0  0 100
 0 0 0  95420K  7203M     0   0   0   0     0   0   0   0 31526   64 31402  0 87 13
 0 0 0  95420K  7203M     0   0   0   0     0   0   0   0 36767   64 33320  0 99  1
 0 0 0  95420K  7203M   423   0   0   0   406   0   0   0 36174  384 28107  0 99  1
 0 0 0  95420K  7203M     0   0   0   0     0   0   0   0 36706   64 27043  0 99  1
 0 0 0  95420K  7203M     0   0   0   0     0   0   0   0 34006   64 13117  0 91  9
 2 0 0  95420K  7203M     0   0   0   0     0   0   0   0   17   64  550  0  1 99
 0 0 0  95420K  7203M     0   0   0   0     3   0   3   0   19   68  507  0  0 100

dev.ix.0.%desc: Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 1.7.4
dev.ix.0.%driver: ix
dev.ix.0.%location: slot=0 function=0
dev.ix.0.%pnpinfo: vendor=0x8086 device=0x10c6 subvendor=0x8086 subdevice=0xa15f class=0x020000
dev.ix.0.%parent: pci3
dev.ix.0.stats: -1
dev.ix.0.debug: -1
dev.ix.0.flow_control: 0
dev.ix.0.enable_lro: 1

Adaptive Interrupt Mitigation is enabled:
dev.ix.0.enable_aim: 1

I did not changed  AIM settings since is quite complex to tune them up. I tried to reverse engineering the algorithm of AIM (see attached picture) but I can't obtain tangible improvements playing with these paramenters. My understanding is that I should reduce the low_latency, but it seems not to work.
dev.ix.0.low_latency: 128
dev.ix.0.ave_latency: 400
dev.ix.0.bulk_latency: 1200

Not sure about this
dev.ix.0.hdr_split: 0

Not sure about the meaning of
dev.ix.0.rx_processing_limit: 100

These are the settings I am using in the ixgbe driver:
#define DEFAULT_TXD     1024
#define PERFORM_TXD     2048
#define MAX_TXD         4096
#define MIN_TXD         64

#define DEFAULT_RXD     1024
#define PERFORM_RXD     2048
#define MAX_RXD         4096
#define MIN_RXD         64

#define IXGBE_TX_CLEANUP_THRESHOLD      (adapter->num_tx_desc / 1)
#define IXGBE_TX_OP_THRESHOLD           (adapter->num_tx_desc / 4)

I see that I had a good performance improvement setting IXGBE_TX_CLEANUP_THRESHOLD to the tx queue size.
This is clear to understand since this (greatly) minimizes context switching in send process reducing the number of time the txq function is called. (Of course with this settings send latency is increased, but this is not an issue).

I can't understand the meaning of these parameters and if they could help:
/*
 * This parameter controls the maximum no of times the driver will loop in
 * the isr. Minimum Value = 1
 */
#define MAX_LOOP        10

/*
 * This parameter controls the duration of transmit watchdog timer.
 */
#define IXGBE_TX_TIMEOUT                   5    /* set to 5 seconds */

Fabrizio

Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle persone indicate. La diffusione, copia o qualsiasi altra azione derivante dalla conoscenza di queste informazioni sono rigorosamente vietate. Qualora abbiate ricevuto questo documento per errore siete cortesemente pregati di darne immediata comunicazione al mittente e di provvedere alla sua distruzione, Grazie.

This e-mail and any attachments is confidential and may contain privileged information intended for the addressee(s) only. Dissemination, copying, printing or use by anybody else is unauthorised. If you are not the intended recipient, please delete this message and any attachments and advise the sender by return e-mail, Thanks.