ether_ifdetach() races round 3(?)

[John Baldwin]

I had another moment of inspiration regarding the ether_ifdetach() races 
during my shower this morning.  Maybe this will gives us a viable solution 
this time.

The reason we are having races is that once we call foo_stop(), the driver 
state is out of sync with the ifnet state because IFF_UP is still set even 
though the driver effectively has marked the interface down.  The obvious 
solution to that is to have the ifnet code "officially" mark the driver down 
by clearing IFF_UP.  This would result in foo_ioctl() calling foo_stop() 
rather than foo_detach() calling it explicitly.  I think this is actually 
cleaner in that drivers already rely on the ifnet layer calling foo_init() 
and don't call foo_init() in foo_attach().  Moving foo_stop() out of 
foo_detach() and into the ifnet layer would thus be more consistent.  
foo_detach() would go from:

	if (device_is_attached(sc)) {
		FOO_LOCK(sc);
		foo_stop(sc);
		FOO_UNLOCK(sc);
		callout_drain(&sc->sc_stat_callout);
		ether_ifdetach(sc->sc_ifp);
	}

to just:

	if (device_is_attached(sc)) {
		ether_ifdetach(sc->sc_ifp);
		callout_drain(&sc->sc_stat_callout);
	}

The only question then is when should ether_ifdetach() mark the interface as 
down, and this actually ends up nice as it lets us fix all the other races 
with BPF and userland ioctls.  Basically, we should detach the ifnet from 
userland so no more userland requests can come in, then tear down kernel 
consumers such as BPF, and mark the interface down resulting in an ioctl() 
that clears IFF_UP and calls foo_stop().

-- 
John Baldwin <jhb at FreeBSD.org>  <><  http://www.FreeBSD.org/~jhb/
"Power Users Use the Power to Serve"  =  http://www.FreeBSD.org