From nobody Wed May 22 21:03:55 2024
X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Vl3hC14pWz5L0jG
	for <freebsd-hackers@mlmmj.nyi.freebsd.org>; Wed, 22 May 2024 21:04:11 +0000 (UTC)
	(envelope-from wlosh@bsdimp.com)
Received: from mail-lf1-x132.google.com (mail-lf1-x132.google.com [IPv6:2a00:1450:4864:20::132])
	(using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)
	 key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256
	 client-signature RSA-PSS (2048 bits) client-digest SHA256)
	(Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4Vl3hB1cNNz4G27
	for <freebsd-hackers@freebsd.org>; Wed, 22 May 2024 21:04:10 +0000 (UTC)
	(envelope-from wlosh@bsdimp.com)
Authentication-Results: mx1.freebsd.org;
	dkim=pass header.d=bsdimp-com.20230601.gappssmtp.com header.s=20230601 header.b=n8bDBRsl;
	dmarc=none;
	spf=none (mx1.freebsd.org: domain of wlosh@bsdimp.com has no SPF policy when checking 2a00:1450:4864:20::132) smtp.mailfrom=wlosh@bsdimp.com
Received: by mail-lf1-x132.google.com with SMTP id 2adb3069b0e04-51f40b5e059so7687976e87.0
        for <freebsd-hackers@freebsd.org>; Wed, 22 May 2024 14:04:10 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=bsdimp-com.20230601.gappssmtp.com; s=20230601; t=1716411848; x=1717016648; darn=freebsd.org;
        h=cc:to:subject:message-id:date:from:in-reply-to:references
         :mime-version:from:to:cc:subject:date:message-id:reply-to;
        bh=yeZ2KNIOcHFBP6GDljBMIcbXCgRudN69IjqSGv73pJc=;
        b=n8bDBRslyHUUVA2j/I3PWx/ssf6jO8ucTkyqaFroWoJ4W14VUVmHXwvLkb17so5Hs9
         4JeUE1B9HrgnoXoI9pOE1/u9WDYtJNpTWDxPznaLivWkNGFoJNImdsOgLvEgMa/mmj4U
         nb0DVRMKmIHdNPrKSJrlHXtbdhajR0l+qs9Bpuzwws8ehnaHdkwvZipmCrh9+X6nReB6
         spPUHxUTwkDz3fwjEZPeCreSbJ1/VBEijVwYDc7817Vk5zxyFHHlnjH2UqP1VMPua8HZ
         Rbc3xcydCk0/DKsNgmkTqNEVNfFOQqvORdCXu9UQhIWo/sXt0EJOwQx5xmCLaLLWME84
         DCOg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20230601; t=1716411848; x=1717016648;
        h=cc:to:subject:message-id:date:from:in-reply-to:references
         :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id
         :reply-to;
        bh=yeZ2KNIOcHFBP6GDljBMIcbXCgRudN69IjqSGv73pJc=;
        b=s6mAVYIXDhVy5JL5TQsN0pTR39qNtmL9gRnhR88m0QCn2jUtGpYr6PQMQRuHyIW2fm
         0H1LQ7uCM6KU3GgOg7LOGOwg1Tmtsg9jDpuBNrDvjFQsa6rp52MKBXvc4y0i8f2uLcwW
         fAM4GC4nQ/MlAd2aTQex6IrjAjhev0CXEwNsDBe0xnGK5FGlD3ZYE7gYQN38UAcgfaPf
         cRkQs5tAuMhsgQIiP6INQoLsJ4C8erWCkLuKF6cjz7W+3UFlZAnB5RCKrOXzIz475UAQ
         qbMYWStdRL2RlbXOU9UXrq2FkPBkentaFSHF/ZNE/lAnCpBpjOGUNvsFeBp613qqNYkX
         GLaA==
X-Gm-Message-State: AOJu0YwlQQdYjbDaNxyfoBnXj9gPx5KJXtspa3TYTXHpVeeo8WZgaZ3I
	mPY5wT+JrWsH3XBPO7BkG6UrJhyrraNVisL4IvrscJbfwys0nMxL7eouuBTDJiL2HTM4FXXRWmF
	PZ4jT05GEWhVueYLTuUktmovLmx7qbxgsA9E9Dg==
X-Google-Smtp-Source: AGHT+IF8mDHcdKXGFpA0wbqrG1nKKCvEalBtjoRyJl+ETJlgXoTOuSI1mYAIjVQQeTeH/9cNjUnQ5EoVKdKC+a9+AXg=
X-Received: by 2002:ac2:59c6:0:b0:523:9747:4920 with SMTP id
 2adb3069b0e04-526be316495mr2213133e87.36.1716411847863; Wed, 22 May 2024
 14:04:07 -0700 (PDT)
List-Id: Technical discussions relating to FreeBSD <freebsd-hackers.freebsd.org>
List-Archive: https://lists.freebsd.org/archives/freebsd-hackers
List-Help: <mailto:freebsd-hackers+help@freebsd.org>
List-Post: <mailto:freebsd-hackers@freebsd.org>
List-Subscribe: <mailto:freebsd-hackers+subscribe@freebsd.org>
List-Unsubscribe: <mailto:freebsd-hackers+unsubscribe@freebsd.org>
Sender: owner-freebsd-hackers@FreeBSD.org
MIME-Version: 1.0
References: <087AB4C4-9CE0-4024-8E1B-4C636014C8CC@lysator.liu.se>
In-Reply-To: <087AB4C4-9CE0-4024-8E1B-4C636014C8CC@lysator.liu.se>
From: Warner Losh <imp@bsdimp.com>
Date: Wed, 22 May 2024 15:03:55 -0600
Message-ID: <CANCZdfoAtNO_KCo6i--FH3+d-uRK+9UC9JZEUBpfo2HHJguBZw@mail.gmail.com>
Subject: Re: Trying to understand CAM (and the cciss driver)
To: Peter Eriksson <pen@lysator.liu.se>
Cc: FreeBSD Hackers <freebsd-hackers@freebsd.org>
Content-Type: multipart/alternative; boundary="0000000000007b16e4061911469c"
X-Spamd-Bar: --
X-Spamd-Result: default: False [-3.00 / 15.00];
	NEURAL_HAM_MEDIUM(-1.00)[-1.000];
	NEURAL_HAM_LONG(-1.00)[-1.000];
	NEURAL_HAM_SHORT(-1.00)[-0.999];
	FORGED_SENDER(0.30)[imp@bsdimp.com,wlosh@bsdimp.com];
	R_DKIM_ALLOW(-0.20)[bsdimp-com.20230601.gappssmtp.com:s=20230601];
	MIME_GOOD(-0.10)[multipart/alternative,text/plain];
	ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US];
	RCVD_COUNT_ONE(0.00)[1];
	MIME_TRACE(0.00)[0:+,1:+,2:~];
	RCVD_TLS_LAST(0.00)[];
	ARC_NA(0.00)[];
	MISSING_XM_UA(0.00)[];
	R_SPF_NA(0.00)[no SPF record];
	DMARC_NA(0.00)[bsdimp.com];
	RCPT_COUNT_TWO(0.00)[2];
	FROM_NEQ_ENVFROM(0.00)[imp@bsdimp.com,wlosh@bsdimp.com];
	FROM_HAS_DN(0.00)[];
	RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::132:from];
	TO_MATCH_ENVRCPT_SOME(0.00)[];
	TO_DN_ALL(0.00)[];
	PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org];
	MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org];
	DKIM_TRACE(0.00)[bsdimp-com.20230601.gappssmtp.com:+]
X-Rspamd-Queue-Id: 4Vl3hB1cNNz4G27

--0000000000007b16e4061911469c
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Hey Peter,

My apologies for my tardy response. The CISS driver is a bit older, and
most of my experience is with newer drivers...

On Thu, May 9, 2024 at 11:59=E2=80=AFAM Peter Eriksson <pen@lysator.liu.se>=
 wrote:

> Hi,
>
> I=E2=80=99m trying to fix a bug in the cciss driver that has been there =
=E2=80=9Cforever=E2=80=9D
> when using it with an HP H241 SAS HBA card.
> The driver works fine when all (SAS, spinning rust) drives are behaving
> well, but when some of them are starting to go bad it often goes into spi=
n
> and either hangs the kernel or panics. I=E2=80=99ve been trying to add
> instrumentation to it in order to pin-point the problem and have been
> attempting some workarounds (like clearing cr_complete since without that
> hack sometime the driver get many many non-busy repeated requests with th=
e
> same =E2=80=9Ctag=E2=80=9D and then It panics with:
>
> > ciss1: WARNING: completing non-busy request
> > ciss1: WARNING: completing non-busy request
> > ciss1: WARNING: completing non-busy request
>

This means that we're not tracking the state of the requests correctly.


> > panic: cam_periph_done_panic: already done with ccb 0xfffff88abefc4000
>

And I think this is the panic when we detect that.


> > cpuid =3D 0
> > time =3D 1714852260
> > KDB: stack backtrace:
> > #0 0xffffffff80c541b5 at kdb_backtrace+0x65
> > #1 0xffffffff80c06b31 at vpanic+0x151
> > #2 0xffffffff80c069d3 at panic+0x43
> > #3 0xffffffff8039017c at cam_periph_done_panic+0x1c
> > #4 0xffffffff80397193 at xpt_done_process+0x313
> > #5 0xffffffff803990a5 at xpt_done_td+0xf5
> > #6 0xffffffff80bc333e at fork_exit+0x7e
> > #7 0xffffffff8108a44e at fork_trampoline+0xe
>
> but it=E2=80=99s still there=E2=80=A6
>
> I think I could use some input/suggestions from someone more knowledgeabl=
e
> with CAM and device drivers in general=E2=80=A6
>
> An example (this is with extra instrumentation added so not quite a stock
> kernel driver):
>
> ciss1: *** Drive failure imminent, Port=3D1E Box=3D1 Bay=3D17 reason=3D0x=
37
> [ts=3D4460, class=3D4, subclass=3D0, detail=3D1, tag=3D0x2]
> (da122:ciss1:32:78:0): WRITE(16). CDB: 8a 00 00 00 00 04 7f 17 51 98 00 0=
0
> 00 10 00 00
> (da122:ciss1:32:78:0): CAM status: CCB request completed with an error
> (da122:ciss1:32:78:0): Retrying command, 3 more tries remain
> (da122:ciss1:32:78:0): WRITE(16). CDB: 8a 00 00 00 00 04 7f 17 52 18 00 0=
0
> 00 10 00 00
> (da122:ciss1:32:78:0): CAM status: CCB request completed with an error
> (da122:ciss1:32:78:0): Retrying command, 3 more tries remain
> (da122:ciss1:32:78:0): READ(16). CDB: 88 00 00 00 00 04 7f 17 65 c8 00 00
> 02 00 00 00
> (da122:ciss1:32:78:0): CAM status: CCB request completed with an error
> (da122:ciss1:32:78:0): Retrying command, 3 more tries remain
> (da122:ciss1:32:78:0): READ(16). CDB: 88 00 00 00 00 04 7f 17 63 c8 00 00
> 02 00 00 00
> (da122:ciss1:32:78:0): CAM status: CCB request completed with an error
> (da122:ciss1:32:78:0): Retrying command, 3 more tries remain
> (da122:ciss1:32:78:0): READ(16). CDB: 88 00 00 00 00 04 7f 17 61 c8 00 00
> 02 00 00 00
> (da122:ciss1:32:78:0): CAM status: CCB request completed with an error
> (da122:ciss1:32:78:0): Retrying command, 3 more tries remain
> (da122:ciss1:32:78:0): READ(16). CDB: 88 00 00 00 00 04 7f 17 5f c8 00 00
> 02 00 00 00
> (da122:ciss1:32:78:0): CAM status: CCB request completed with an error
> (da122:ciss1:32:78:0): Retrying command, 3 more tries remain
> (da122:ciss1:32:78:0): READ(16). CDB: 88 00 00 00 00 04 7f 17 5f a8 00 00
> 00 20 00 00
> (da122:ciss1:32:78:0): CAM status: CCB request completed with an error
> (da122:ciss1:32:78:0): Retrying command, 3 more tries remain
> (da122:ciss1:32:78:0): READ(16). CDB: 88 00 00 00 00 04 7f 17 5f 38 00 00
> 00 70 00 00
> (da122:ciss1:32:78:0): CAM status: CCB request completed with an error
> (da122:ciss1:32:78:0): Retrying command, 3 more tries remain
> (da122:ciss1:32:78:0): READ(16). CDB: 88 00 00 00 00 04 7f 17 5d 38 00 00
> 02 00 00 00
> (da122:ciss1:32:78:0): CAM status: CCB request completed with an error
> (da122:ciss1:32:78:0): Retrying command, 3 more tries remain
> (da122:ciss1:32:78:0): READ(16). CDB: 88 00 00 00 00 04 7f 17 5b 38 00 00
> 02 00 00 00
> (da122:ciss1:32:78:0): CAM status: CCB request completed with an error
> (da122:ciss1:32:78:0): Retrying command, 3 more tries remain
> (da122:ciss1:32:78:0): READ(16). CDB: 88 00 00 00 00 04 7f 17 59 38 00 00
> 02 00 00 00
> (da122:ciss1:32:78:0): CAM status: CCB request completed with an error
> (da122:ciss1:32:78:0): Retrying command, 3 more tries remain
> (da122:ciss1:32:78:0): READ(16). CDB: 88 00 00 00 00 04 7f 17 58 48 00 00
> 00 f0 00 00
> (da122:ciss1:32:78:0): CAM status: CCB request completed with an error
> (da122:ciss1:32:78:0): Retrying command, 3 more tries remain
> (da122:ciss1:32:78:0): READ(16). CDB: 88 00 00 00 00 04 7f 17 56 48 00 00
> 02 00 00 00
> (da122:ciss1:32:78:0): CAM status: CCB request completed with an error
> (da122:ciss1:32:78:0): Retrying command, 3 more tries remain
> (da122:ciss1:32:78:0): WRITE(16). CDB: 8a 00 00 00 00 04 88 d8 48 d0 00 0=
0
> 00 10 00 00
> (da122:ciss1:32:78:0): CAM status: CCB request completed with an error
> (da122:ciss1:32:78:0): Retrying command, 3 more tries remain
> (da122:ciss1:32:78:0): WRITE(16). CDB: 8a 00 00 00 00 04 88 d8 46 e0 00 0=
0
> 01 f0 00 00
> (da122:ciss1:32:78:0): CAM status: CCB request completed with an error
> (da122:ciss1:32:78:0): Retrying command, 3 more tries remain
> (da122:ciss1:32:78:0): READ(16). CDB: 88 00 00 00 00 04 88 d8 8f d0 00 00
> 02 00 00 00
> (da122:ciss1:32:78:0): CAM status: CCB request completed with an error
> (da122:ciss1:32:78:0): Retrying command, 3 more tries remain
> (da122:ciss1:32:78:0): READ(16). CDB: 88 00 00 00 00 04 88 d8 8d d0 00 00
> 02 00 00 00
> (da122:ciss1:32:78:0): CAM status: CCB request completed with an error
> (da122:ciss1:32:78:0): Retrying command, 3 more tries remain
> ciss1: WARNING: completing non-busy request #1 [onq=3D0, length=3D262144,
> ccphys=3D1791048448, tag=3D72, flags=3D0x50 (----I-C), sg_tag=3D0x3,
> cr_complete=3D0xffffffff806059a0]
> ciss1: WARNING: clearing cr_complete due to ciss_cam_complete & nonbusy
> ciss1: WARNING: completing non-busy request #1 [onq=3D0, length=3D262144,
> ccphys=3D1791318368, tag=3D313, flags=3D0x50 (----I-C), sg_tag=3D0x3,
> cr_complete=3D0xffffffff806059a0]
> ciss1: WARNING: clearing cr_complete due to ciss_cam_complete & nonbusy
> ciss1: WARNING: completing non-busy request #1 [onq=3D0, length=3D8192,
> ccphys=3D1791226528, tag=3D231, flags=3D0x50 (----I-C), sg_tag=3D0x3,
> cr_complete=3D0xffffffff806059a0]
> ciss1: WARNING: clearing cr_complete due to ciss_cam_complete & nonbusy
> ciss1: WARNING: completing non-busy request #1 [onq=3D0, length=3D262144,
> ccphys=3D1791313888, tag=3D309, flags=3D0x50 (----I-C), sg_tag=3D0x5,
> cr_complete=3D0xffffffff806059a0]
> ciss1: WARNING: clearing cr_complete due to ciss_cam_complete & nonbusy
> ciss1: WARNING: completing non-busy request #1 [onq=3D0, length=3D8192,
> ccphys=3D1791206368, tag=3D213, flags=3D0x50 (----I-C), sg_tag=3D0x3,
> cr_complete=3D0xffffffff806059a0]
> ciss1: WARNING: clearing cr_complete due to ciss_cam_complete & nonbusy
> panic: camq_remove: Attempt to remove out-of-bounds index -1 from queue
> 0xfffff80107fa9838 of size 1
>

So these seem to line up with the commands that encountered an error. The
errors handling code of most CAM drivers is the hardest to write and
validate because misbehaving disks, while not rare, are generally not used
in validation.  All my good ones that were dying have since stopped working
completely, for example.

So in this case, you'll need to look at the state tracking code for the
ciss requests, and make sure that in the cases of errors the right
transitions happen. I've fixed bugs in mpr/mps that amounted to marking a
request as busy again when it errored out in some of the weirder error
scenarios (I don't recall the exact cases though). You might look at
mpr/mps for the evolution of this code there. IIRC, the same author did
both ciss and mps and many of the techniques were retained between the two.
You might get good hints for bugs to fix by looking at how things evolved
there.

Basically, you're going to have to find out if these complaints are due to
the state tracking being messed up, but the request really should be doing
the thing that it's doing when the complaint is lodged (in which case you
need to fix the point where the state becomes incorrect). Or if it's caught
a real bug and the state of the request is proper, but we're doing
something like completing something twice, or failing to allocate a new
request or one of the many other silly bugs this tracking code usually
prevents.

Warner


> cpuid =3D 7
> time =3D 1715266713
> KDB: stack backtrace:
> #0 0xffffffff80c542d5 at kdb_backtrace+0x65
> #1 0xffffffff80c06c51 at vpanic+0x151
> #2 0xffffffff80c06af3 at panic+0x43
> #3 0xffffffff80390676 at camq_remove+0xf6
> #4 0xffffffff803936c8 at xpt_run_devq+0x148
> #5 0xffffffff80392b0c at xpt_action_default+0x35c
> #6 0xffffffff803c3544 at dastart+0x314
> #7 0xffffffff80394f73 at xpt_run_allocq+0x83
> #8 0xffffffff803c5152 at dastrategy+0x82
> #9 0xffffffff80b438d4 at g_disk_start+0x314
> #10 0xffffffff80b46b55 at g_io_request+0x1d5
> #11 0xffffffff80b46b55 at g_io_request+0x1d5
> #12 0xffffffff821a34dc at vdev_geom_io_start+0x23c
> #13 0xffffffff82347fb4 at zio_vdev_io_start+0x204
> #14 0xffffffff8234204f at zio_nowait+0x12f
> #15 0xffffffff822a0ce7 at vdev_mirror_io_start+0x177
> #16 0xffffffff82347fb4 at zio_vdev_io_start+0x204
> #17 0xffffffff8234204f at zio_nowait+0x12f
> Uptime: 10h16m26s
>
>
> The drive it complains about in the first line is one that has SMART
> errors:
>
> >        connector 1E box  1 bay 17                 HP      MB010000JWAYK
>                                   7PH8LU3G     HPD5 S.M.A.R.T. predictive
> failure.
>
> It is _not_ da122 though=E2=80=A6 The drive with a SMART error (da85) is =
not in
> use so basically is just sitting there.
>
>
>
> The code where I=E2=80=99m trying to figure out what=E2=80=99s going on i=
s this part in
> sys/dev/ciss/ciss.c:
>
> static void
> ciss_complete(struct ciss_softc *sc, cr_qhead_t *qh)
> {
>     struct ciss_request *cr;
>
>     debug_called(2);
>
>     /*
>
>
>      * Loop taking requests off the completed queue and performing
>
>
>      * completion processing on them.
>
>
>      */
>     for (;;) {
>       int got_nonbusy =3D 0;
>
>         if ((cr =3D ciss_dequeue_complete(sc, qh)) =3D=3D NULL)
>             break;
>         ciss_unmap_request(cr);
>
>         if ((cr->cr_flags & CISS_REQ_BUSY) =3D=3D 0) {
>           cr->nonbusy_request_counter++;
>           ciss_printf(sc,
>                       "WARNING: completing non-busy request #%d [onq=3D%d=
,
> length=3D%u, ccphys=3D%u, tag=3D%d, flags=3D0x%x (%s), sg_tag=3D0x%x,
> cr_complete=3D%p]\n",
>                       cr->nonbusy_request_counter,
>                       cr->cr_onq, cr->cr_length, cr->cr_ccphys, cr->cr_ta=
g,
>                       cr->cr_flags, cr_flags2str(cr->cr_flags),
>                       cr->cr_sg_tag, cr->cr_complete
>                       );
>           got_nonbusy =3D 1;
>         }
>
>         cr->cr_flags &=3D ~CISS_REQ_BUSY;
>
>         /*
>
>
>          * If the request has a callback, invoke it.
>
>
>          */
>         if (cr->cr_complete !=3D NULL) {
>           cr->cr_complete(cr);
>
>         /* pen: Attempt to make sure ciss_cam_complete() doesn=E2=80=99t =
get
> called multiple times */
>
>           if (cr->cr_complete =3D=3D ciss_cam_complete && got_nonbusy) {
>             ciss_printf(sc, "WARNING: clearing cr_complete due to
> ciss_cam_complete & nonbusy\n");
>             cr->cr_complete =3D NULL;
>           }
>           continue;
>         }
>
>         /*
>
>
>          * If someone is sleeping on this request, wake them up.
>
>
>          */
>         if (cr->cr_flags & CISS_REQ_SLEEP) {
>           cr->cr_flags &=3D ~CISS_REQ_SLEEP;
>           wakeup(cr);
>           continue;
>         }
>
>         /*
>
>
>          * If someone is polling this request for completion, signal.
>
>
>          */
>         if (cr->cr_flags & CISS_REQ_POLL) {
>           cr->cr_flags &=3D ~CISS_REQ_POLL;
>           continue;
>         }
>
>         /*
>
>
>          * Give up and throw the request back on the free queue.  This
>
>
>          * should never happen; resources will probably be lost.
>
>
>          */
>         ciss_printf(sc, "WARNING: completed command with no submitter\n")=
;
>         ciss_enqueue_free(cr);
>     }
> }
>
>
> - Any suggestions? (Probably not but I thought I=E2=80=99d ask anyway :-)
>
> - Peter
>
>
>
>

--0000000000007b16e4061911469c
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div dir=3D"ltr">Hey Peter,<div><br></div><div>My apologie=
s for my tardy response. The CISS driver is a bit older, and most of my exp=
erience is with newer drivers...</div></div><br><div class=3D"gmail_quote">=
<div dir=3D"ltr" class=3D"gmail_attr">On Thu, May 9, 2024 at 11:59=E2=80=AF=
AM Peter Eriksson &lt;<a href=3D"mailto:pen@lysator.liu.se">pen@lysator.liu=
.se</a>&gt; wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"marg=
in:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1e=
x">Hi,<br>
<br>
I=E2=80=99m trying to fix a bug in the cciss driver that has been there =E2=
=80=9Cforever=E2=80=9D when using it with an HP H241 SAS HBA card. <br>
The driver works fine when all (SAS, spinning rust) drives are behaving wel=
l, but when some of them are starting to go bad it often goes into spin and=
 either hangs the kernel or panics. I=E2=80=99ve been trying to add instrum=
entation to it in order to pin-point the problem and have been attempting s=
ome workarounds (like clearing cr_complete since without that hack sometime=
 the driver get many many non-busy repeated requests with the same =E2=80=
=9Ctag=E2=80=9D and then It panics with: <br>
<br>
&gt; ciss1: WARNING: completing non-busy request<br>
&gt; ciss1: WARNING: completing non-busy request<br>
&gt; ciss1: WARNING: completing non-busy request<br></blockquote><div><br><=
/div><div>This means that we&#39;re not tracking the state of the requests =
correctly.</div><div>=C2=A0</div><blockquote class=3D"gmail_quote" style=3D=
"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-le=
ft:1ex">
&gt; panic: cam_periph_done_panic: already done with ccb 0xfffff88abefc4000=
<br></blockquote><div><br></div><div>And I think this is the panic when we =
detect that.</div><div>=C2=A0</div><blockquote class=3D"gmail_quote" style=
=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding=
-left:1ex">
&gt; cpuid =3D 0<br>
&gt; time =3D 1714852260<br>
&gt; KDB: stack backtrace:<br>
&gt; #0 0xffffffff80c541b5 at kdb_backtrace+0x65<br>
&gt; #1 0xffffffff80c06b31 at vpanic+0x151<br>
&gt; #2 0xffffffff80c069d3 at panic+0x43<br>
&gt; #3 0xffffffff8039017c at cam_periph_done_panic+0x1c<br>
&gt; #4 0xffffffff80397193 at xpt_done_process+0x313<br>
&gt; #5 0xffffffff803990a5 at xpt_done_td+0xf5<br>
&gt; #6 0xffffffff80bc333e at fork_exit+0x7e<br>
&gt; #7 0xffffffff8108a44e at fork_trampoline+0xe<br>
<br>
but it=E2=80=99s still there=E2=80=A6 <br>
<br>
I think I could use some input/suggestions from someone more knowledgeable =
with CAM and device drivers in general=E2=80=A6<br>
<br>
An example (this is with extra instrumentation added so not quite a stock k=
ernel driver):<br>
<br>
ciss1: *** Drive failure imminent, Port=3D1E Box=3D1 Bay=3D17 reason=3D0x37=
 [ts=3D4460, class=3D4, subclass=3D0, detail=3D1, tag=3D0x2]<br>
(da122:ciss1:32:78:0): WRITE(16). CDB: 8a 00 00 00 00 04 7f 17 51 98 00 00 =
00 10 00 00 <br>
(da122:ciss1:32:78:0): CAM status: CCB request completed with an error<br>
(da122:ciss1:32:78:0): Retrying command, 3 more tries remain<br>
(da122:ciss1:32:78:0): WRITE(16). CDB: 8a 00 00 00 00 04 7f 17 52 18 00 00 =
00 10 00 00 <br>
(da122:ciss1:32:78:0): CAM status: CCB request completed with an error<br>
(da122:ciss1:32:78:0): Retrying command, 3 more tries remain<br>
(da122:ciss1:32:78:0): READ(16). CDB: 88 00 00 00 00 04 7f 17 65 c8 00 00 0=
2 00 00 00 <br>
(da122:ciss1:32:78:0): CAM status: CCB request completed with an error<br>
(da122:ciss1:32:78:0): Retrying command, 3 more tries remain<br>
(da122:ciss1:32:78:0): READ(16). CDB: 88 00 00 00 00 04 7f 17 63 c8 00 00 0=
2 00 00 00 <br>
(da122:ciss1:32:78:0): CAM status: CCB request completed with an error<br>
(da122:ciss1:32:78:0): Retrying command, 3 more tries remain<br>
(da122:ciss1:32:78:0): READ(16). CDB: 88 00 00 00 00 04 7f 17 61 c8 00 00 0=
2 00 00 00 <br>
(da122:ciss1:32:78:0): CAM status: CCB request completed with an error<br>
(da122:ciss1:32:78:0): Retrying command, 3 more tries remain<br>
(da122:ciss1:32:78:0): READ(16). CDB: 88 00 00 00 00 04 7f 17 5f c8 00 00 0=
2 00 00 00 <br>
(da122:ciss1:32:78:0): CAM status: CCB request completed with an error<br>
(da122:ciss1:32:78:0): Retrying command, 3 more tries remain<br>
(da122:ciss1:32:78:0): READ(16). CDB: 88 00 00 00 00 04 7f 17 5f a8 00 00 0=
0 20 00 00 <br>
(da122:ciss1:32:78:0): CAM status: CCB request completed with an error<br>
(da122:ciss1:32:78:0): Retrying command, 3 more tries remain<br>
(da122:ciss1:32:78:0): READ(16). CDB: 88 00 00 00 00 04 7f 17 5f 38 00 00 0=
0 70 00 00 <br>
(da122:ciss1:32:78:0): CAM status: CCB request completed with an error<br>
(da122:ciss1:32:78:0): Retrying command, 3 more tries remain<br>
(da122:ciss1:32:78:0): READ(16). CDB: 88 00 00 00 00 04 7f 17 5d 38 00 00 0=
2 00 00 00 <br>
(da122:ciss1:32:78:0): CAM status: CCB request completed with an error<br>
(da122:ciss1:32:78:0): Retrying command, 3 more tries remain<br>
(da122:ciss1:32:78:0): READ(16). CDB: 88 00 00 00 00 04 7f 17 5b 38 00 00 0=
2 00 00 00 <br>
(da122:ciss1:32:78:0): CAM status: CCB request completed with an error<br>
(da122:ciss1:32:78:0): Retrying command, 3 more tries remain<br>
(da122:ciss1:32:78:0): READ(16). CDB: 88 00 00 00 00 04 7f 17 59 38 00 00 0=
2 00 00 00 <br>
(da122:ciss1:32:78:0): CAM status: CCB request completed with an error<br>
(da122:ciss1:32:78:0): Retrying command, 3 more tries remain<br>
(da122:ciss1:32:78:0): READ(16). CDB: 88 00 00 00 00 04 7f 17 58 48 00 00 0=
0 f0 00 00 <br>
(da122:ciss1:32:78:0): CAM status: CCB request completed with an error<br>
(da122:ciss1:32:78:0): Retrying command, 3 more tries remain<br>
(da122:ciss1:32:78:0): READ(16). CDB: 88 00 00 00 00 04 7f 17 56 48 00 00 0=
2 00 00 00 <br>
(da122:ciss1:32:78:0): CAM status: CCB request completed with an error<br>
(da122:ciss1:32:78:0): Retrying command, 3 more tries remain<br>
(da122:ciss1:32:78:0): WRITE(16). CDB: 8a 00 00 00 00 04 88 d8 48 d0 00 00 =
00 10 00 00 <br>
(da122:ciss1:32:78:0): CAM status: CCB request completed with an error<br>
(da122:ciss1:32:78:0): Retrying command, 3 more tries remain<br>
(da122:ciss1:32:78:0): WRITE(16). CDB: 8a 00 00 00 00 04 88 d8 46 e0 00 00 =
01 f0 00 00 <br>
(da122:ciss1:32:78:0): CAM status: CCB request completed with an error<br>
(da122:ciss1:32:78:0): Retrying command, 3 more tries remain<br>
(da122:ciss1:32:78:0): READ(16). CDB: 88 00 00 00 00 04 88 d8 8f d0 00 00 0=
2 00 00 00 <br>
(da122:ciss1:32:78:0): CAM status: CCB request completed with an error<br>
(da122:ciss1:32:78:0): Retrying command, 3 more tries remain<br>
(da122:ciss1:32:78:0): READ(16). CDB: 88 00 00 00 00 04 88 d8 8d d0 00 00 0=
2 00 00 00 <br>
(da122:ciss1:32:78:0): CAM status: CCB request completed with an error<br>
(da122:ciss1:32:78:0): Retrying command, 3 more tries remain<br>
ciss1: WARNING: completing non-busy request #1 [onq=3D0, length=3D262144, c=
cphys=3D1791048448, tag=3D72, flags=3D0x50 (----I-C), sg_tag=3D0x3, cr_comp=
lete=3D0xffffffff806059a0]<br>
ciss1: WARNING: clearing cr_complete due to ciss_cam_complete &amp; nonbusy=
<br>
ciss1: WARNING: completing non-busy request #1 [onq=3D0, length=3D262144, c=
cphys=3D1791318368, tag=3D313, flags=3D0x50 (----I-C), sg_tag=3D0x3, cr_com=
plete=3D0xffffffff806059a0]<br>
ciss1: WARNING: clearing cr_complete due to ciss_cam_complete &amp; nonbusy=
<br>
ciss1: WARNING: completing non-busy request #1 [onq=3D0, length=3D8192, ccp=
hys=3D1791226528, tag=3D231, flags=3D0x50 (----I-C), sg_tag=3D0x3, cr_compl=
ete=3D0xffffffff806059a0]<br>
ciss1: WARNING: clearing cr_complete due to ciss_cam_complete &amp; nonbusy=
<br>
ciss1: WARNING: completing non-busy request #1 [onq=3D0, length=3D262144, c=
cphys=3D1791313888, tag=3D309, flags=3D0x50 (----I-C), sg_tag=3D0x5, cr_com=
plete=3D0xffffffff806059a0]<br>
ciss1: WARNING: clearing cr_complete due to ciss_cam_complete &amp; nonbusy=
<br>
ciss1: WARNING: completing non-busy request #1 [onq=3D0, length=3D8192, ccp=
hys=3D1791206368, tag=3D213, flags=3D0x50 (----I-C), sg_tag=3D0x3, cr_compl=
ete=3D0xffffffff806059a0]<br>
ciss1: WARNING: clearing cr_complete due to ciss_cam_complete &amp; nonbusy=
<br>
panic: camq_remove: Attempt to remove out-of-bounds index -1 from queue 0xf=
ffff80107fa9838 of size 1<br></blockquote><div><br></div><div>So these seem=
 to line up with the commands that encountered an error. The errors handlin=
g code of most CAM drivers is the hardest to write and validate because mis=
behaving disks, while not rare, are generally not used in validation.=C2=A0=
 All my good ones that were dying have since stopped working completely, fo=
r example.</div><div><br></div><div>So in this case, you&#39;ll need to loo=
k at the state tracking code for the ciss requests, and make sure that in t=
he cases of errors the right transitions happen. I&#39;ve fixed bugs in mpr=
/mps that amounted to marking a request as busy again when it errored out i=
n some of the weirder error scenarios (I don&#39;t recall the exact cases t=
hough). You might look at mpr/mps for the evolution of this code there. IIR=
C, the same author did both ciss and mps and many of the techniques were re=
tained between the two. You might get good hints for bugs to fix by looking=
 at how things evolved there.</div><div><br></div><div>Basically, you&#39;r=
e going to have to find out if these complaints are due to the state tracki=
ng being messed up, but the request really should be doing the thing that i=
t&#39;s doing when the complaint is lodged (in which case you need to fix t=
he point where the state becomes incorrect). Or if it&#39;s caught a real b=
ug and the state of the request is proper, but we&#39;re doing something li=
ke completing something twice, or failing to allocate a new request or one =
of the many other silly bugs this tracking code usually prevents.</div><div=
><br></div><div>Warner</div><div>=C2=A0</div><blockquote class=3D"gmail_quo=
te" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204=
);padding-left:1ex">
cpuid =3D 7<br>
time =3D 1715266713<br>
KDB: stack backtrace:<br>
#0 0xffffffff80c542d5 at kdb_backtrace+0x65<br>
#1 0xffffffff80c06c51 at vpanic+0x151<br>
#2 0xffffffff80c06af3 at panic+0x43<br>
#3 0xffffffff80390676 at camq_remove+0xf6<br>
#4 0xffffffff803936c8 at xpt_run_devq+0x148<br>
#5 0xffffffff80392b0c at xpt_action_default+0x35c<br>
#6 0xffffffff803c3544 at dastart+0x314<br>
#7 0xffffffff80394f73 at xpt_run_allocq+0x83<br>
#8 0xffffffff803c5152 at dastrategy+0x82<br>
#9 0xffffffff80b438d4 at g_disk_start+0x314<br>
#10 0xffffffff80b46b55 at g_io_request+0x1d5<br>
#11 0xffffffff80b46b55 at g_io_request+0x1d5<br>
#12 0xffffffff821a34dc at vdev_geom_io_start+0x23c<br>
#13 0xffffffff82347fb4 at zio_vdev_io_start+0x204<br>
#14 0xffffffff8234204f at zio_nowait+0x12f<br>
#15 0xffffffff822a0ce7 at vdev_mirror_io_start+0x177<br>
#16 0xffffffff82347fb4 at zio_vdev_io_start+0x204<br>
#17 0xffffffff8234204f at zio_nowait+0x12f<br>
Uptime: 10h16m26s<br>
<br>
<br>
The drive it complains about in the first line is one that has SMART errors=
:<br>
<br>
&gt;=C2=A0 =C2=A0 =C2=A0 =C2=A0 connector 1E box=C2=A0 1 bay 17=C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0HP=C2=A0 =C2=A0 =C2=A0 =
MB010000JWAYK=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0=
 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 7PH8LU3G=C2=
=A0 =C2=A0 =C2=A0HPD5 S.M.A.R.T. predictive failure.<br>
<br>
It is _not_ da122 though=E2=80=A6 The drive with a SMART error (da85) is no=
t in use so basically is just sitting there.<br>
<br>
<br>
<br>
The code where I=E2=80=99m trying to figure out what=E2=80=99s going on is =
this part in sys/dev/ciss/ciss.c:<br>
<br>
static void<br>
ciss_complete(struct ciss_softc *sc, cr_qhead_t *qh)<br>
{<br>
=C2=A0 =C2=A0 struct ciss_request *cr;<br>
<br>
=C2=A0 =C2=A0 debug_called(2);<br>
<br>
=C2=A0 =C2=A0 /*=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 <br>
=C2=A0 =C2=A0 =C2=A0* Loop taking requests off the completed queue and perf=
orming=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0=
 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 <br>
=C2=A0 =C2=A0 =C2=A0* completion processing on them.=C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<br>
=C2=A0 =C2=A0 =C2=A0*/<br>
=C2=A0 =C2=A0 for (;;) {<br>
=C2=A0 =C2=A0 =C2=A0 int got_nonbusy =3D 0;<br>
<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 if ((cr =3D ciss_dequeue_complete(sc, qh)) =3D=
=3D NULL)<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 break;<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 ciss_unmap_request(cr);<br>
<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 if ((cr-&gt;cr_flags &amp; CISS_REQ_BUSY) =3D=
=3D 0) {<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 cr-&gt;nonbusy_request_counter++;<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 ciss_printf(sc,<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 &quot;WARNING: completing non-busy request #%d [onq=3D%d, length=3D%u, =
ccphys=3D%u, tag=3D%d, flags=3D0x%x (%s), sg_tag=3D0x%x, cr_complete=3D%p]\=
n&quot;,<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 cr-&gt;nonbusy_request_counter,<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 cr-&gt;cr_onq, cr-&gt;cr_length, cr-&gt;cr_ccphys, cr-&gt;cr_tag,<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 cr-&gt;cr_flags, cr_flags2str(cr-&gt;cr_flags),<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 cr-&gt;cr_sg_tag, cr-&gt;cr_complete<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 );<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 got_nonbusy =3D 1;<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 }<br>
<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 cr-&gt;cr_flags &amp;=3D ~CISS_REQ_BUSY;<br>
<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 /*=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 <br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0* If the request has a callback, invoke i=
t.=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 <br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0*/<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 if (cr-&gt;cr_complete !=3D NULL) {<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 cr-&gt;cr_complete(cr);<br>
<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 /* pen: Attempt to make sure ciss_cam_complete(=
) doesn=E2=80=99t get called multiple times */<br>
<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 if (cr-&gt;cr_complete =3D=3D ciss_cam_c=
omplete &amp;&amp; got_nonbusy) {<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 ciss_printf(sc, &quot;WARNING: cl=
earing cr_complete due to ciss_cam_complete &amp; nonbusy\n&quot;);<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 cr-&gt;cr_complete =3D NULL;<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 }<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 continue;<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 }=C2=A0 =C2=A0 =C2=A0 <br>
<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 /*=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 <br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0* If someone is sleeping on this request,=
 wake them up.=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 <br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0*/<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 if (cr-&gt;cr_flags &amp; CISS_REQ_SLEEP) {<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 cr-&gt;cr_flags &amp;=3D ~CISS_REQ_SLEEP=
;<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 wakeup(cr);<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 continue;<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 }<br>
<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 /*=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 <br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0* If someone is polling this request for =
completion, signal.=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0*/<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 if (cr-&gt;cr_flags &amp; CISS_REQ_POLL) {<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 cr-&gt;cr_flags &amp;=3D ~CISS_REQ_POLL;=
<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 continue;<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 }<br>
<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 /*=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 <br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0* Give up and throw the request back on t=
he free queue.=C2=A0 This=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 <br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0* should never happen; resources will pro=
bably be lost.=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 <br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0*/<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 ciss_printf(sc, &quot;WARNING: completed comman=
d with no submitter\n&quot;);<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 ciss_enqueue_free(cr);<br>
=C2=A0 =C2=A0 }<br>
}<br>
<br>
<br>
- Any suggestions? (Probably not but I thought I=E2=80=99d ask anyway :-)<b=
r>
<br>
- Peter<br>
<br>
<br>
<br>
</blockquote></div></div>

--0000000000007b16e4061911469c--