SNMP High Capacity Counters

[Hartmut Brandt]

Eugene Grosbein wrote:
> On Sat, Oct 18, 2008 at 07:18:52PM +0200, Hartmut Brandt wrote:
> 
>>> I've just found that ports/net-snmp (version 5.4) built
>>> WITH_MFD_REWRITES=yes supports IF-MIB, and in theory should show 64-bit
>>> ifHC* counters but it does not.
>>>
>>> It seems agent/mibgroup/if-mib/data_access/interface_sysctl.c that obtains
>>> interface statistics from the kernel.
>>> The function netsnmp_arch_interface_container_load() has the following 
>>> code:
>>>
>>>        /* get counters */
>>>        entry->stats.ibytes.low = ifp->ifm_data.ifi_ibytes;
>>>        entry->stats.ibytes.high = 0;
>>>        entry->stats.iucast.low = ifp->ifm_data.ifi_ipackets;
>>>        entry->stats.iucast.high = 0;
>>>        entry->stats.imcast.low = ifp->ifm_data.ifi_imcasts;
>>>        entry->stats.imcast.high = 0;
>>>
>>> So, it always produce 32-bit quantities. My question is:
>>> does FreeBSD/i386 kernel maintain 64-bit counters for interface statictics
>>> these days? If yes, since what version?
> 
>> It does not, because not all architectures have atomic 64-bit increments 
>> and adds. Implementing 64-bit counters on these architectures would 
>> require some kind of locking. This was discussed in the past.
> 
> Yes, I've read archives and saw a discussion dated 2002 or 2003.
> Some time has passed since than, generic CPU horsepower has changed.
> And I'm sure IPFW maintains 64-bit counters for FreeBSD/i386
> without a problem. Yes, IPFW is additional feature and needs to be enables
> explicitly. I run it since 2.2.8 in every kernel and had no problems with
> (lots of) its 64-bit counters. Would (optionally) having
> another pack of them hurt? I've read someone made a path for 64-bit
> statictics counters in 2002.

The problem is not the CPU horsepower. The problem is that you update 
these counters on each incoming packet. No problem for desktops, but if 
you want to route several 100kps this may hurt. I have no idea, how IPFW 
does it. Perhaps it just declares the variables as 64-bit and if the 
value is wrong because of a race condition, then it is just wrong. With 
SNMP I would not like to have wrong counters - people use these counters 
for accounting purposes.

> 
>> You might look at the IF-MIB implementation of bsnmp (it is in the base 
>> system). It uses periodic polling to detect wraps of the 32-bit 
>> counters. The poll interval is tuned to the fastest interface in the 
>> system (given that all interfaces reported the correct speed).
>>
>> Note, that the netsnmp implementation is plain wrong - if the daemon 
>> does not support the HC counters it should never pretend to do. This is 
>> explicitely stated somewhere in the RFCs.
> 
> It does really support them for Linux and Solaris.
> They seem to have solved this problem somehow.
> It would be very nice to have, say, kernel option to enable
> the support. Just like options IPFIREWALL.

I remember a mail from, I think, phk (cannot find it, though) where he 
described a method to handle, I think, geom statistics in a lockless 
way. The idea was to put a generation count at the begin and the end of 
the data area. The consumer would copy the entire area and compare the 
generation counts. If they are equal, the stats are ok, if they differ, 
you take another turn. You need of course the right memory barriers in 
the code so that neither the CPU nor the compiler reorders the 
increments and this will cost some cycles (no idea how many). I don't 
know what happened to this (never looked at the geom code) but it sounds 
usable for interface statistics too. Would be nice, though, to have this 
mechansim in the sysctl handler so that userspace needs not to care 
about this stuff.

harti