From nobody Wed Dec 29 21:57:27 2021 X-Original-To: scsi@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id C64021913F39 for ; Wed, 29 Dec 2021 21:57:35 +0000 (UTC) (envelope-from mavbsd@gmail.com) Received: from mail-qk1-x735.google.com (mail-qk1-x735.google.com [IPv6:2607:f8b0:4864:20::735]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4JPQHg52rNz3Ljk; Wed, 29 Dec 2021 21:57:35 +0000 (UTC) (envelope-from mavbsd@gmail.com) Received: by mail-qk1-x735.google.com with SMTP id 202so17334623qkg.13; Wed, 29 Dec 2021 13:57:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=sender:subject:to:cc:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-language:content-transfer-encoding; bh=RrAY2U4ccBPaEmffFkY7G9mAytSrgQaSAjZQ2BPO0II=; b=Cpp8rH9cL6Ple80On5OgnJcerDouBGldAhMtERvCLzsrSbguNoVnkGEKXnCMtEGbt/ HW5wyIYZI/8K7yhVlEZfjw+QQbqYo8ocGn2COL0rdgVthT1YGj/9qT5LgESa1TB33Ewr OS0Iu9BzLhGJq1tTfQ8/gd7/VdKl/ke1bpL8JNtQNFGhDNNXM7CKTuJ64RPoWt4zPMT5 UH1lzplxN7EVrIwMIgZwUOSDAGyDttofIDWZ3/Etn5ls5cTPMsJy+s2yDed5/NTXatgh IQHLlVbB/D84aJLrI2+7VX7DldK4M+NELJH6l288T+yyDn+6wIWJCvQwmzLu8TI252py 5Vmg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:sender:subject:to:cc:references:from:message-id :date:user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=RrAY2U4ccBPaEmffFkY7G9mAytSrgQaSAjZQ2BPO0II=; b=zWKHWmw3WCLiDL8MI2Nf/ArG2OWJKLXggSmtJlGnYMY5fLCV1QAI3se5oeewz7zcC5 zX7gEBli/OrJ+/DatMvhoRKgfiG9goBM66JccbbI3qXZIMWrhVvmbvo3vkq4eofjbcLw aYtMisr4aMum02gcFYf/4QPG/cZMufbNNR2vumTFqkaMRTAxbd0m0BgYdVC5khNV34p8 dpJh59peNZJ3uoOPqQc432pP24oeDZPrwAGotR9nE0fwCLqwjiJZTg/vAC40HEgRC5Mi 8H0YxajCJLURx2nOU3q23bg7DO/jDTb8TmXZg8HBCDfPRxnDgzsa0E4UeKb7ytu8w9WP PN+Q== X-Gm-Message-State: AOAM533SzL5Rv9w/hg7JsQKTi6G3w2N51Y5FKZ0SHca5NRD19IpH/JCV 65CxMNQis7H1W5TABlvVpN/bLcD0dA0= X-Google-Smtp-Source: ABdhPJwAKM06cBZPDsdnAcpTxyk+s4zSQt7+nSRZQ0O9FQA5ZZvfiVaeLzvZA4Zme4wa8Fft09IV0w== X-Received: by 2002:a37:a156:: with SMTP id k83mr18904598qke.401.1640815048996; Wed, 29 Dec 2021 13:57:28 -0800 (PST) Received: from spectre.mavhome.dp.ua (104-55-12-234.lightspeed.knvltn.sbcglobal.net. [104.55.12.234]) by smtp.gmail.com with ESMTPSA id v12sm19094004qtx.80.2021.12.29.13.57.28 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 29 Dec 2021 13:57:28 -0800 (PST) Subject: Re: iSCSI target: Handling in-flight requests during ctld shutdown To: John Baldwin , scsi@FreeBSD.org Cc: =?UTF-8?Q?Edward_Tomasz_Napiera=c5=82a?= References: From: Alexander Motin Message-ID: Date: Wed, 29 Dec 2021 16:57:27 -0500 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:78.0) Gecko/20100101 Thunderbird/78.13.0 List-Id: SCSI subsystem List-Archive: https://lists.freebsd.org/archives/freebsd-scsi List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-scsi@freebsd.org X-BeenThere: freebsd-scsi@freebsd.org MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 4JPQHg52rNz3Ljk X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] X-ThisMailContainsUnwantedMimeParts: N On 29.12.2021 16:39, John Baldwin wrote: > One of the tests Chelsio QA has been running against our iSCSI stack > with cxgbei offload enabled is to run a bunch of iozone's on an > initiator while running a script on the target that keeps stopping > ctld (for a minute or so), then starting it again and letting it run > for about 5 minutes until stopping it again. > > One of the errors found last night is that the target reported the > following error to the initiator: > > (da7:iscsi10:0:0:0): CAM status: SCSI Status Error > (da7:iscsi10:0:0:0): SCSI status: Check Condition > (da7:iscsi10:0:0:0): SCSI sense: HARDWARE FAILURE asc:44,0 (Internal > target failure) > (da7:iscsi10:0:0:0): Actual Retry Count: 44 > (da7:iscsi10:0:0:0): Error 5, Unretryable error > g_vfs_done():da7[WRITE(offset=9797632, length=32768)]error = 6 > UFS: forcibly unmounting /dev/da7 from /ISCSI8 > So my question I think is what is the expected behavior?  Is the > internal error > really expected to make it on the wire to be sent to the other side?  Since > the connection is shutting down should we just discard the reply altogether > rather than reporting an internal error?  If we discarded the reply then > the > initiator in this particular test would have retried the original > request once > ctld was restarted and continued running without an error. The HARDWARE ERROR is obviously not expected by the initiator. It should better not be leaked after we decided to kill the connection. Initiator may retry it and still work happily after reconnect, but cleaner would be to not rely on that. cfiscsi_session_terminate_tasks() aborts all running commands by CTL_TASK_I_T_NEXUS_RESET, that make them not return statuses to initiator, but I suppose this is the other side of the race now. -- Alexander Motin