splxxx level?

[M. Warner Losh]

In message: <4133682D.3000403 at freebsd.org>
            Scott Long <scottl at freebsd.org> writes:
: M. Warner Losh wrote:
: 
: > In message: <20040830.102606.130865377.imp at bsdimp.com>
: >             "M. Warner Losh" <imp at bsdimp.com> writes:
: > : In message: <41334C3B.4070101 at freebsd.org>
: > :             Scott Long <scottl at freebsd.org> writes:
: > : : Sam wrote:
: > : : 
: > : : > Hello -
: > : : > 
: > : : > I'm almost to testing on my AoE driver for 4.x and have
: > : : > a question about interrupt priority levels.
: > : : > 
: > : : > There are currently three entry points into the driver:
: > : : > 
: > : : > a) strategy routine
: > : : > b) network frame reception routine
: > : : > c) timer rexmit routine
: > : : > 
: > : : > Any of the three can diddle with the device structure
: > : : > and thusly I need to ensure they're not running simultaneously.
: > : : > For example, the network reception can cause a buf to be completed
: > : : > and the rexmit timer can cause a buf to be failed.
: > : : > 
: > : : > So, what kind of contexts are the callout, strategy, and
: > : : > network soft interrupt called in?  Which splxxx will give
: > : : > one of them exclusive access to whatever they need?
: > : : > 
: > : : > Just as a reality check -- I am thinking about this correct, right?
: > : : > 
: > : : > Cheers,
: > : : > 
: > : : > Sam
: > : : > 
: > : : 
: > : : With 4.x, only one CPU can be in the kernel at a time.  You won't have
: > : : to worry about multiple processes trying to get into strategy at the
: > : : same time and whatnot.  However, you can be preempted by your interrupt
: > : : handler or by a timeout or by a software interrupt like the netisr.  I
: > : : don't remember if your driver is for a specific piece of hardware or if
: > : : it's a generic layer that sits in between the network interface and the
: > : : block layer.  If it's for dedicated hardware then you'll need to define
: > : : a interrupt type in bus_setup_intr() and use that type for the spl
: > : : (i.e. INTR_TYPE_NET translates to splnet(), INTR_TYPE_BIO translates to
: > : : splbio(), etc).
: > : : 
: > : : The safe way to go is to protect all of your critical code sections with
: > : : the appropriate spl level regardless.  spls are very cheap and can be
: > : : set/reset from an interrupt context so there is little penalty in using
: > : : them liberally at first and then narrowing them down later.  Just make
: > : : sure that you don't leak an spl references, and don't hold an spl for so
: > : : long that it creates priority inversions.  Since the only interrupts and
: > : : timeouts that you'll likely be dealing with are  network related,
: > : : splnet() is probably the right one to use.
: > : 
: > : splimp() is what you want to use, not splnet().  Yes, this is
: > : confusing, but it appears to be what all the other network drivers
: > : use.  None of them are using splnet() that I could find.  splimp() is
: > : also used by the mbuf routines to protect mbuf operations.
: > : 
: > : splnet() is a list of the software interrupts that are network
: > : drivers.
: > : 
: > : splimp() is splnet() plus the hardware interrupts, so is more
: > : appropriate to block things called from the driver.  Especially one
: > : that's described as having timeouts.  If it is a network driver, you
: > : might consider using the timeout functionality in the net stack as
: > : opposed to the callout functions.  This makes it possible to have
: > : almost the entire driver w/o doing any spls (most of the network
: > : drivers in 4 don't do spl at all, except for entry points that are
: > : outside the scope of the network/interrupt entry points, eg
: > : device_suspend).
: > 
: > Ah, just saw 'AoE' in the reply.
: > 
: > This likely means that you are writing a network stack that's glued
: > into the system with the strategy routine.  Inside the strategy
: > routine, splbio() is likely what you need to hold while dealing with
: > the struct buf that's passed into that routine.  I'm not sure what
: > spl level you need to be at to call into the network code.  I'm
: > thinking it is splimp(), but I'm not 100% sure about that.  Stevens
: > will likely be a good resource for 4.x.
: > 
: > You may have to define a software interrupt to pass the packets to the
: > network code to make the spls work out correctly.
: > 
: > Warner
: 
: No, splbio() is not explicitely needed in this case.  You'll be
: dealing with a bioq that might need protection, but you have total
: control over that.  There is nothing else in the block layer that
: will interrupt you, not like the netisr in the network stack or the
: camisr in the CAM layer.

Why does da use splbio in its strategy routine then:

static void
dastrategy(struct buf *bp)
{
...

	/*
	 * Mask interrupts so that the pack cannot be invalidated until
	 * after we are in the queue.  Otherwise, we might not properly
	 * clean up one of the buffers.
	 */
	s = splbio();
	
...(error cases)
	/*
	 * Place it in the queue of disk activities for this disk
	 */
	bufqdisksort(&softc->buf_queue, bp);

	splx(s);
	
	/*
	 * Schedule ourselves for performing the work.
	 */
	xpt_schedule(periph, /* XXX priority */1);

	return;
... (error cases)
}

Is that due to the ISR routine from the cam layer?  Or is the
softc->buf_queue the 'bioq' that you were talking about?

Warner