cvs commit: src/sys/net if_ethersubr.c

[Bruce Evans]

On Tue, 12 Dec 2006, Luigi Rizzo wrote:

> On Tue, Dec 12, 2006 at 02:13:34PM +0100, Bernd Walter wrote:
>> On Fri, Dec 08, 2006 at 10:36:46AM +0000, Luigi Rizzo wrote:
>>> luigi       2006-12-08 10:36:45 UTC
>>>
>>>   FreeBSD src repository
>>>
>>>   Modified files:
>>>     sys/net              if_ethersubr.c
>>>   Log:
>>>   Fix an oscure bug triggered by a recent change in kern_socket.c.
>>
>> This change produces an alignment panic on arm.
>> Reverting it gets my system back to live.

Something reduced my ttcp tx bandwidth for tiny packets on an i386 (Athlon
XP 2700 overclocked) from 350 kpps to 300 kpps (after I thought increased
it from 320-330 kpps to 350-360 kpps) by reducing copying).  I suspected
this commit, then misalignment.  I still suspect alignment...

> then i suppose the proper fix is to revert to m_copy() and
> work on if_simloop() so that 1. it handles a readonly chain, and
> 2. when doing so, it passes up a properly aligned packet...
>
> however note that there is already some code in net/if_loop.c::if_simloop(),
> just that it uses this:
>
> 	#if defined(__ia64__) || defined(__sparc64__)
>                /*
>                 * Some archs do not like unaligned data, so
>                 * we move data down in the first mbuf.
>                 */
>                if (mtod(m, vm_offset_t) & 3) {
>                        KASSERT(hlen >= 3, ("if_simloop: hlen too small"));
>                        bcopy(m->m_data,
>                            (char *)(mtod(m, vm_offset_t)
>                                - (mtod(m, vm_offset_t) & 3)),
>                            m->m_len);
>                        m->m_data -= (mtod(m,vm_offset_t) & 3);
>                }
> 	#endif
>
> to detect whether the architecture is alignment-sensitive.
> Is there any other identifier that we can use to check ?

__NO_STRICT_ALIGNMENT.  Many drivers already use this.  It's stiall an
abuse since __FOO is in the implementation namespace, and of course
__NO_STRICT_ALIGNMENT is documented mainly (only?) in its log message.

No arches "like" unaligned data; some just handle it without trapping
or other extreme slowness.  I think copying of data to fix its alignment
is best for most arches if the data is small but not if it is large.

About my ttcp tx bandwith problem:
- tx is almost entirely syscall-bound (< 2-4% interrupt overhead)
- the slowdown was mostly caused by adding about 500 bytes in unrelated
   code (in nfs_diskless.c) that was never executed.  Removing these bytes
   or moving nfs_diskless.o to the end of the object file list recovered
   30-40 kpps of the lost bandwidth.  It looks like there are lots of
   cache conflicts for the I-cache, but I thought that this problem mostly
   went away with i486's.
- the bandwidth depended mysteriously on related and unrelated previous
   changes, but no single change (including ones that added much more than
   500 bytes) affected it by as much as 360 -> 300.
- the bandwidth also depends on the ttcp binary and has a high variance
   in separate runs (but not in the same run).  I now suspect that I-cache
   conflicts cause this too.
- I found some silly data alignment that doesn't seem to affect ttcp.
   struct sockaddr is essentially char [16] so it asks for pessimal
   alignment (not misalignment unless it is type punned).  I think gcc
   normally gives struct sockaddr's on the stack 4-byte alignment.
   sun_noname is perfectly misaligned in all my kernels, because it
   happens to get allocated after some strings with an odd total size
   in the read-only text section.  It is a sockaddr, not a sockaddr_un
   despite its name.  struct sockaddr_un is essentially char [106],
   so it also asks for pessimal alignment, with more cost since it is
   quite large.  Grepping nm output showed this some other alignment
   problems.  E.g., futex functions in i386/support.s have random
   alignment and other bugs caused by not using ENTRY() for them.

Bruce