SOLVED: Degraded TCP performace on Intel PRO/1000

[Marian Durkovic]

Hi all,


  seems we've found the problem. The performance degradation was happening
it the TX path, due to insufficient setting of TX packet buffer FIFO on the
chip.

  To achieve wirespeed performance, the TX FIFO must be large enough to
accomodate 2 jumbo packets (not just 1 as the driver was assuming).
There was also a typo in the driver, causing the PBA tuning on most
cards to be non-functional.

  Due to above limitation, the 82547 chipset (featuring only 40 KB of 
RX/TX FIFO) only supports wirespeed tranfers upto 8 KB TCP payload
(MTU 9000 bytes).

  Please be sure to use em driver version 1.7.41 or newer, either from
the CVS (branch RELENG_4_11) or from Intel's downloads.

  The PBA code needs to be modified as follows:

--- if_em.c     Wed Jan  5 00:55:03 2005
+++ if_em.c.new Fri May  6 08:44:58 2005
@@ -816,17 +816,18 @@
         *   Default allocation: PBA=48K for Rx, leaving 16K for Tx.
         * After the 82547 the buffer was reduced to 40K.
         *   Default allocation: PBA=30K for Rx, leaving 10K for Tx.
-        *   Note: default does not leave enough room for Jumbo Frame >10k.
+        *  BEWARE: For wirespeed performance, Tx buffer must be able
+        *          to accommodate 2 frames
         */
        if(adapter->hw.mac_type < em_82547) {
                /* Total FIFO is 64K */
-               if(adapter->rx_buffer_len > EM_RXBUFFER_8192)
+               if(adapter->hw.max_frame_size > 8192)
                        pba = E1000_PBA_40K; /* 40K for Rx, 24K for Tx */
                else
                        pba = E1000_PBA_48K; /* 48K for Rx, 16K for Tx */
        } else {
                /* Total FIFO is 40K */
-               if(adapter->hw.max_frame_size > EM_RXBUFFER_8192) {
+               if(adapter->hw.max_frame_size > 5120) {
                        pba = E1000_PBA_22K; /* 22K for Rx, 18K for Tx */
                } else {
                        pba = E1000_PBA_30K; /* 30K for Rx, 10K for Tx */


   With the above modifications, the em driver is able to run wirespeed
also with maximum TCP payload of 6 KB and 8KB - tested on 82547 and 82546
controllers.


	With kind regards,

		M.

On Thu, May 05, 2005 at 03:32:50PM +0200, Marian Durkovic wrote:
> Hi all,
> 
>   recently we've found serious performance degradation for TCP connections
> between FreeBSD 4.9 and 4.11 at larger packet sizes.
> 
>   We're using two identical machines with Intel PRO/1000 82547EI chipset,
> which were able to run wirespeed at all packet sizes when 4.9 was installed
> on both of them. The tests are performed using nttcp -T -n500000 
> 
>   Here are the results:
> 
> TCP payload            4.9->4.11           4.11->4.9
> ----------------------------------------------------
> 1448 Bytes             941.0007 Mbps       941.1021 Mbps
> 2048                   957.5791            957.5312
> 4096                   978.1912            978.2145
> 6144                   715.3935            715.7244
> 8192                   631.8491            988.7455
> 
>    It is obvious, that transfers with TCP payloads upto 4 kB are wirespeed,
> while larger packets have significantly degraded performance.
> 
>    Any ideas?
> 
> 
> 	Thanks & kind regards,
> 

--------------------------------------------------------------------------
----                                                                  ----
----   Marian Durkovic                       network  manager         ----
----                                                                  ----
----   Slovak Technical University           Tel: +421 2 524 51 301   ----
----   Computer Centre, Nam. Slobody 17      Fax: +421 2 524 94 351   ----
----   812 43 Bratislava, Slovak Republic    E-mail/sip: md at bts.sk    ----
----                                                                  ----
--------------------------------------------------------------------------