buf_ring in HEAD is racy

[Julien Charbon]

 Hi,

On 30/10/14 20:39, K. Macy wrote:
>> I also suspect there are further problems with buf_ring.  A full wrap
>> around of the atomically swapped value is possible. I.e. the code thinks
>> it just atomically updated a head/tail index when in fact a full wrap
>> around occurred leading to undefined land. A relatively simple way to
>> avoid this is to only mask on ring array access, and to let the
>> head/tail/prod/cons indices overflow the array.
> 
> Up until Rui Paulo complained to me of packet drops with buf_ring a
> couple of days ago I had thought that this patch had been committed.
> This patch (now 273866) fixes the problem for him. Without further
> scrutiny and testing I won't provide the UL guarantee for
> buf_ring_enqueue, but this is a clear improvement.

 I have tested r273866 fix against our traffic and it seems it fixes the
ENOBUFS issue for us too.  Below the Dtrace script used for checking who
was returning ENOBUFS:

$ cat enobufs.d
fbt::sendit:return
/arg1 == ENOBUFS/
{
    @senditENOBUFS[execname,pid] = count();
}

fbt::ixgbe_mq_start:return
/arg1 == ENOBUFS/
{
    @ixgbeStartENOBUFS[execname,pid] = count();
}

fbt::ip_output:return
/arg1 != 0/
{
    @ipOutputErr[execname,pid] = count();
}

END
{
    printf("\nsendit ENOBUFS:\n");
    printa(@senditENOBUFS);

    printf("\nixgbe_mq_start ENOBUFS:\n");
    printa(@ixgbeStartENOBUFS);

    printf("\nip_output Errors:\n");
    printa(@ipOutputErr);
}

 The result without the fix:

$ dtrace -s enobufs.d
dtrace: script 'enobufs.d' matched 4 probes
^C
CPU     ID                    FUNCTION:NAME
  0      2                             :END
sendit ENOBUFS:

  server      1779               47

ixgbe_mq_start ENOBUFS:

  server      1779               47

ip_output Errors:

  server      1779               47

 Thus 47 ENOBUFS errors, all returned by ixgbe_mq_start() which calls
drbr_enqueue().

 And result with the fix applied:

$ dtrace -s enobufs.d
dtrace: script 'enobufs.d' matched 4 probes
^C
CPU     ID                    FUNCTION:NAME
  3      2                             :END
sendit ENOBUFS:

ixgbe_mq_start ENOBUFS:

ip_output Errors:

 Thus 0 ENOBUFS errors.  I will let Marc review this fix further to
check his full wrap case.

 Thanks.

--
Julien

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 536 bytes
Desc: OpenPGP digital signature
URL: <http://lists.freebsd.org/pipermail/freebsd-net/attachments/20141031/87ece8c3/attachment.sig>