svn commit: r291716 - in head: share/man/man4 sys/cam sys/cam/ata sys/cam/scsi sys/dev/md sys/geom sys/kern sys/pc98/include sys/sys usr.sbin usr.sbin/camdd
Kenneth D. Merry
ken at FreeBSD.ORG
Thu Dec 3 21:45:10 UTC 2015
On Thu, Dec 03, 2015 at 13:13:25 -0800, Bryan Drewery wrote:
> On 12/3/15 12:54 PM, Kenneth D. Merry wrote:
> > Author: ken
> > Date: Thu Dec 3 20:54:55 2015
> > New Revision: 291716
> > URL: https://svnweb.freebsd.org/changeset/base/291716
> >
> > Log:
> > Add asynchronous command support to the pass(4) driver, and the new
> > camdd(8) utility.
> >
> > CCBs may be queued to the driver via the new CAMIOQUEUE ioctl, and
> > completed CCBs may be retrieved via the CAMIOGET ioctl. User
> > processes can use poll(2) or kevent(2) to get notification when
> > I/O has completed.
> >
> > While the existing CAMIOCOMMAND blocking ioctl interface only
> > supports user virtual data pointers in a CCB (generally only
> > one per CCB), the new CAMIOQUEUE ioctl supports user virtual and
> > physical address pointers, as well as user virtual and physical
> > scatter/gather lists. This allows user applications to have more
> > flexibility in their data handling operations.
> >
> > Kernel memory for data transferred via the queued interface is
> > allocated from the zone allocator in MAXPHYS sized chunks, and user
> > data is copied in and out. This is likely faster than the
> > vmapbuf()/vunmapbuf() method used by the CAMIOCOMMAND ioctl in
> > configurations with many processors (there are more TLB shootdowns
> > caused by the mapping/unmapping operation) but may not be as fast
> > as running with unmapped I/O.
> >
> > The new memory handling model for user requests also allows
> > applications to send CCBs with request sizes that are larger than
> > MAXPHYS. The pass(4) driver now limits queued requests to the I/O
> > size listed by the SIM driver in the maxio field in the Path
> > Inquiry (XPT_PATH_INQ) CCB.
> >
> > There are some things things would be good to add:
> >
> > 1. Come up with a way to do unmapped I/O on multiple buffers.
> > Currently the unmapped I/O interface operates on a struct bio,
> > which includes only one address and length. It would be nice
> > to be able to send an unmapped scatter/gather list down to
> > busdma. This would allow eliminating the copy we currently do
> > for data.
> >
> > 2. Add an ioctl to list currently outstanding CCBs in the various
> > queues.
> >
> > 3. Add an ioctl to cancel a request, or use the XPT_ABORT CCB to do
> > that.
> >
> > 4. Test physical address support. Virtual pointers and scatter
> > gather lists have been tested, but I have not yet tested
> > physical addresses or scatter/gather lists.
> >
> > 5. Investigate multiple queue support. At the moment there is one
> > queue of commands per pass(4) device. If multiple processes
> > open the device, they will submit I/O into the same queue and
> > get events for the same completions. This is probably the right
> > model for most applications, but it is something that could be
> > changed later on.
> >
> > Also, add a new utility, camdd(8) that uses the asynchronous pass(4)
> > driver interface.
> >
> > This utility is intended to be a basic data transfer/copy utility,
> > a simple benchmark utility, and an example of how to use the
> > asynchronous pass(4) interface.
> >
> > It can copy data to and from pass(4) devices using any target queue
> > depth, starting offset and blocksize for the input and ouptut devices.
> > It currently only supports SCSI devices, but could be easily extended
> > to support ATA devices.
> >
> > It can also copy data to and from regular files, block devices, tape
> > devices, pipes, stdin, and stdout. It does not support queueing
> > multiple commands to any of those targets, since it uses the standard
> > read(2)/write(2)/writev(2)/readv(2) system calls.
> >
> > The I/O is done by two threads, one for the reader and one for the
> > writer. The reader thread sends completed read requests to the
> > writer thread in strictly sequential order, even if they complete
> > out of order. That could be modified later on for random I/O patterns
> > or slightly out of order I/O.
> >
> > camdd(8) uses kqueue(2)/kevent(2) to get I/O completion events from
> > the pass(4) driver and also to send request notifications internally.
> >
> > For pass(4) devcies, camdd(8) uses a single buffer (CAM_DATA_VADDR)
> > per CAM CCB on the reading side, and a scatter/gather list
> > (CAM_DATA_SG) on the writing side. In addition to testing both
> > interfaces, this makes any potential reblocking of I/O easier. No
> > data is copied between the reader and the writer, but rather the
> > reader's buffers are split into multiple I/O requests or combined
> > into a single I/O request depending on the input and output blocksize.
> >
> > For the file I/O path, camdd(8) also uses a single buffer (read(2),
> > write(2), pread(2) or pwrite(2)) on reads, and a scatter/gather list
> > (readv(2), writev(2), preadv(2), pwritev(2)) on writes.
> >
> > Things that would be nice to do for camdd(8) eventually:
> >
> > 1. Add support for I/O pattern generation. Patterns like all
> > zeros, all ones, LBA-based patterns, random patterns, etc. Right
> > Now you can always use /dev/zero, /dev/random, etc.
> >
> > 2. Add support for a "sink" mode, so we do only reads with no
> > writes. Right now, you can use /dev/null.
> >
> > 3. Add support for automatic queue depth probing, so that we can
> > figure out the right queue depth on the input and output side
> > for maximum throughput. At the moment it defaults to 6.
> >
> > 4. Add support for SATA device passthrough I/O.
> >
> > 5. Add support for random LBAs and/or lengths on the input and
> > output sides.
> >
> > 6. Track average per-I/O latency and busy time. The busy time
> > and latency could also feed in to the automatic queue depth
> > determination.
> >
> > sys/cam/scsi/scsi_pass.h:
> > Define two new ioctls, CAMIOQUEUE and CAMIOGET, that queue
> > and fetch asynchronous CAM CCBs respectively.
> >
> > Although these ioctls do not have a declared argument, they
> > both take a union ccb pointer. If we declare a size here,
> > the ioctl code in sys/kern/sys_generic.c will malloc and free
> > a buffer for either the CCB or the CCB pointer (depending on
> > how it is declared). Since we have to keep a copy of the
> > CCB (which is fairly large) anyway, having the ioctl malloc
> > and free a CCB for each call is wasteful.
> >
> > sys/cam/scsi/scsi_pass.c:
> > Add asynchronous CCB support.
> >
> > Add two new ioctls, CAMIOQUEUE and CAMIOGET.
> >
> > CAMIOQUEUE adds a CCB to the incoming queue. The CCB is
> > executed immediately (and moved to the active queue) if it
> > is an immediate CCB, but otherwise it will be executed
> > in passstart() when a CCB is available from the transport layer.
> >
> > When CCBs are completed (because they are immediate or
> > passdone() if they are queued), they are put on the done
> > queue.
> >
> > If we get the final close on the device before all pending
> > I/O is complete, all active I/O is moved to the abandoned
> > queue and we increment the peripheral reference count so
> > that the peripheral driver instance doesn't go away before
> > all pending I/O is done.
> >
> > The new passcreatezone() function is called on the first
> > call to the CAMIOQUEUE ioctl on a given device to allocate
> > the UMA zones for I/O requests and S/G list buffers. This
> > may be good to move off to a taskqueue at some point.
> > The new passmemsetup() function allocates memory and
> > scatter/gather lists to hold the user's data, and copies
> > in any data that needs to be written. For virtual pointers
> > (CAM_DATA_VADDR), the kernel buffer is malloced from the
> > new pass(4) driver malloc bucket. For virtual
> > scatter/gather lists (CAM_DATA_SG), buffers are allocated
> > from a new per-pass(9) UMA zone in MAXPHYS-sized chunks.
> > Physical pointers are passed in unchanged. We have support
> > for up to 16 scatter/gather segments (for the user and
> > kernel S/G lists) in the default struct pass_io_req, so
> > requests with longer S/G lists require an extra kernel malloc.
> >
> > The new passcopysglist() function copies a user scatter/gather
> > list to a kernel scatter/gather list. The number of elements
> > in each list may be different, but (obviously) the amount of data
> > stored has to be identical.
> >
> > The new passmemdone() function copies data out for the
> > CAM_DATA_VADDR and CAM_DATA_SG cases.
> >
> > The new passiocleanup() function restores data pointers in
> > user CCBs and frees memory.
> >
> > Add new functions to support kqueue(2)/kevent(2):
> >
> > passreadfilt() tells kevent whether or not the done
> > queue is empty.
> >
> > passkqfilter() adds a knote to our list.
> >
> > passreadfiltdetach() removes a knote from our list.
> >
> > Add a new function, passpoll(), for poll(2)/select(2)
> > to use.
> >
> > Add devstat(9) support for the queued CCB path.
> >
> > sys/cam/ata/ata_da.c:
> > Add support for the BIO_VLIST bio type.
> >
> > sys/cam/cam_ccb.h:
> > Add a new enumeration for the xflags field in the CCB header.
> > (This doesn't change the CCB header, just adds an enumeration to
> > use.)
> >
> > sys/cam/cam_xpt.c:
> > Add a new function, xpt_setup_ccb_flags(), that allows specifying
> > CCB flags.
> >
> > sys/cam/cam_xpt.h:
> > Add a prototype for xpt_setup_ccb_flags().
> >
> > sys/cam/scsi/scsi_da.c:
> > Add support for BIO_VLIST.
> >
> > sys/dev/md/md.c:
> > Add BIO_VLIST support to md(4).
> >
> > sys/geom/geom_disk.c:
> > Add BIO_VLIST support to the GEOM disk class. Re-factor the I/O size
> > limiting code in g_disk_start() a bit.
> >
> > sys/kern/subr_bus_dma.c:
> > Change _bus_dmamap_load_vlist() to take a starting offset and
> > length.
> >
> > Add a new function, _bus_dmamap_load_pages(), that will load a list
> > of physical pages starting at an offset.
> >
> > Update _bus_dmamap_load_bio() to allow loading BIO_VLIST bios.
> > Allow unmapped I/O to start at an offset.
> >
> > sys/kern/subr_uio.c:
> > Add two new functions, physcopyin_vlist() and physcopyout_vlist().
> >
> > sys/pc98/include/bus.h:
> > Guard kernel-only parts of the pc98 machine/bus.h header with
> > #ifdef _KERNEL.
> >
> > This allows userland programs to include <machine/bus.h> to get the
> > definition of bus_addr_t and bus_size_t.
> >
> > sys/sys/bio.h:
> > Add a new bio flag, BIO_VLIST.
> >
> > sys/sys/uio.h:
> > Add prototypes for physcopyin_vlist() and physcopyout_vlist().
> >
> > share/man/man4/pass.4:
> > Document the CAMIOQUEUE and CAMIOGET ioctls.
> >
> > usr.sbin/Makefile:
> > Add camdd.
> >
> > usr.sbin/camdd/Makefile:
> > Add a makefile for camdd(8).
> >
> > usr.sbin/camdd/camdd.8:
> > Man page for camdd(8).
> >
> > usr.sbin/camdd/camdd.c:
> > The new camdd(8) utility.
> >
> > Sponsored by: Spectra Logic
> > MFC after: 1 week
> >
> > Added:
> > head/usr.sbin/camdd/
> > head/usr.sbin/camdd/Makefile (contents, props changed)
>
> > +++ usr.sbin/camdd/Makefile
> > @@ -0,0 +1,11 @@
> > +# $FreeBSD$
> > +
> > +PROG= camdd
> > +SRCS= camdd.c
> > +SDIR= ${.CURDIR}/../../sys
> > +DPADD= ${LIBCAM} ${LIBMT} ${LIBSBUF} ${LIBBSDXML} ${LIBUTIL} ${LIBTHR}
> > +LDADD= -lcam -lmt -lsbuf -lbsdxml -lutil -lthr
>
>
> We use LIBADD in head now, not DPADD and LDADD. See r291718.
Oops, sorry about that! Thanks for fixing it.
Ken
--
Kenneth Merry
ken at FreeBSD.ORG
More information about the svn-src-head
mailing list