From nobody Wed Mar 27 16:00:06 2024 X-Original-To: bugs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4V4WbC0FnKz5FkDc for ; Wed, 27 Mar 2024 16:00:07 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4V4WbB5Xc5z4FnG for ; Wed, 27 Mar 2024 16:00:06 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1711555206; a=rsa-sha256; cv=none; b=qUkjzVy+YygJBtSUhdsxYGI9HntweU1X38s2RJq9dMBFx6L7ahKW61s0KLjs7FazCx9HJm A6yq/aq90sbrpXQjKjlf2RgoxOEh4H4G9G18Hl2nJPHaBFMxr1eEHEza9ySz7fbweI60W3 uW/yHG5WWNzB7tBvp6TxRsbS2nimmISDIO1JO0sacZUyLLYP+uI3ndKh02efaw3I6hVcec i3RjQ1RiF/4U/L63vqhB1SV0T69WbcKk1hOBdCQVqB4Fa+/auvdCpiZdDxwZxEo6acTp9U r00KwUBz2aNerjpd/YN9eoOKHOD2BIX/7qdJa7wD6WZKUtkiCQtAC5unWTjFJA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1711555206; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=2aip8nADONOLffR7qglKLQj7Hv2sxCvNSV2E6RrKHCE=; b=t5n21bT7wuXcvrJZMJUd7FRkzduSESJawNGNeGsnuLQjcRv30tS1/zQAKUnGzMCUbAfby8 Bw9MOL9QmtYmoQJ/jWHNkrh/fTWMgRDxKlH0o3toRWIEainuWGW09pyMk0Y3kUPBoOv/2f 62/MShTPChiGPazKf2ukonZlyiMc/kcB/QI/V/j0Q4HaDT+2wPUhiFkuqTEkjeDqCkLzZI OVXoMajzvpBik5KEqpA5veBOIUbgp2237zcZnya94gF1PUp1ebAm98kpHiVYJvE9Xqajjv s7DSzT8y6L1zJFnnDSRh/WmzxO1zQX/6EsMh6ImULpqAFZfBLcYM6jSnIi9kmw== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4V4WbB51q8zqH2 for ; Wed, 27 Mar 2024 16:00:06 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 42RG06CJ067556 for ; Wed, 27 Mar 2024 16:00:06 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 42RG06NW067549 for bugs@FreeBSD.org; Wed, 27 Mar 2024 16:00:06 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: bugs@FreeBSD.org Subject: [Bug 277992] mpr and possible trim issues Date: Wed, 27 Mar 2024 16:00:06 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 14.0-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: mike@sentex.net X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: bugs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status bug_severity priority component assigned_to reporter Message-ID: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Bug reports List-Archive: https://lists.freebsd.org/archives/freebsd-bugs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-bugs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D277992 Bug ID: 277992 Summary: mpr and possible trim issues Product: Base System Version: 14.0-STABLE Hardware: amd64 OS: Any Status: New Severity: Affects Some People Priority: --- Component: kern Assignee: bugs@FreeBSD.org Reporter: mike@sentex.net The thread https://lists.freebsd.org/archives/freebsd-hardware/2024-March/000094.html = has most of the details.=20 In summary, a set of WD Blue SA510 SSDs with the latest firmware as of Mar = 2024 will eventually start throwing errors and detach from the controller when I copy and then destroy a zfs dataset with several million files. It sort of feels like a TRIM issue, but not sure. Putting the disks off the onboard S= ATA controller does not recreate the issue.=20 If I start with a low level trim (trim -f /dev/daX), create a raidz1 zfs po= ol with 4, one TB WD disks, import a dataset of about 280GB (compressed) that = has many (20+mill files), do a zfs send original pool | zfs recv copy-of-pool, = then zfs destroy copy-of-pool and repeat about 4 or 5 times, the drives in the p= ool will start throwing errors. If I do a hard trim of the disks, I can start from scratch and again get 4 = or 5 cycles before the errors. Hence, it sort of feels like a broken trim issue= ? I tried with auto trim on and off, a manual zfs trim between zfs sen= d| zfs recv tests to no avail. When the disks are on the mpr controller I will= get errors such as=20 (da6:mpr0:0:16:0): READ(10). CDB: 28 00 6d e0 ae 28 00 00 08 00 (da6:mpr0:0:16:0): CAM status: CCB request completed with an error (da6:mpr0:0:16:0): Retrying command, 3 more tries remain (da6:mpr0:0:16:0): WRITE(10). CDB: 2a 00 0c cb 3f 00 00 00 e8 00 (da6:mpr0:0:16:0): CAM status: CCB request completed with an error (da6:mpr0:0:16:0): Retrying command, 3 more tries remain (da6:mpr0:0:16:0): READ(10). CDB: 28 00 6d e0 ad 28 00 01 00 00 (da6:mpr0:0:16:0): CAM status: CCB request completed with an error (da6:mpr0:0:16:0): Retrying command, 3 more tries remain (da6:mpr0:0:16:0): READ(10). CDB: 28 00 6d e0 ac 28 00 00 f8 00 (da6:mpr0:0:16:0): CAM status: CCB request completed with an error (da6:mpr0:0:16:0): Retrying command, 3 more tries remain (da6:mpr0:0:16:0): WRITE(10). CDB: 2a 00 40 07 df 88 00 01 00 00 (da6:mpr0:0:16:0): CAM status: CCB request completed with an error (da6:mpr0:0:16:0): Retrying command, 3 more tries remain (da6:mpr0:0:16:0): WRITE(10). CDB: 2a 00 3f 48 72 08 00 01 00 00 (da6:mpr0:0:16:0): CAM status: SCSI Status Error (da6:mpr0:0:16:0): SCSI status: Check Condition (da6:mpr0:0:16:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on, reset,=20 or bus device reset occurred) (da6:mpr0:0:16:0): Retrying command (per sense data) mpr0: Controller reported scsi ioc terminated tgt 15 SMID 2036 loginfo=20 31110f00 mpr0: Controller reported scsi ioc terminated tgt 15 SMID 637 loginfo=20 31110f00 (da5:mpr0:0:15:0): WRITE(10). CDB: 2a 00 41 98 42 00 00 01 00 00 mpr0: Controller reported scsi ioc terminated tgt 15 SMID 1242 loginfo=20 31110f00 mpr0: Controller reported scsi ioc terminated tgt 15 SMID 979 loginfo=20 31110f00 mpr0: Controller reported scsi ioc terminated tgt 15 SMID 1243 loginfo=20 31110f00 mpr0: Controller reported scsi ioc terminated tgt 15 SMID 2091 loginfo=20 31110f00 mpr0: Controller reported scsi ioc terminated tgt 15 SMID 1612 loginfo=20 31110f00 mpr0: Controller reported scsi ioc terminated tgt 15 SMID 2093 loginfo=20 31110f00 mpr0: Controller reported scsi ioc terminated tgt 15 SMID 152 loginfo=20 31110f00 mpr0: Controller reported scsi ioc terminated tgt 15 SMID 2132 loginfo=20 31110f00 (da5:mpr0:0:15:0): CAM status: CCB request completed with an error (da5:mpr0:0:15:0): Retrying command, 3 more tries remain (da5:mpr0:0:15:0): WRITE(10). CDB: 2a 00 43 17 dc 88 00 01 00 00 (da5:mpr0:0:15:0): CAM status: CCB request completed with an error (da5:mpr0:0:15:0): Retrying command, 3 more tries remain (da5:mpr0:0:15:0): WRITE(10). CDB: 2a 00 41 98 43 00 00 00 50 00 (da5:mpr0:0:15:0): CAM status: CCB request completed with an error (da5:mpr0:0:15:0): Retrying command, 3 more tries remain (da5:mpr0:0:15:0): WRITE(10). CDB: 2a 00 0c d4 f6 80 00 00 68 00 (da5:mpr0:0:15:0): CAM status: CCB request completed with an error (da5:mpr0:0:15:0): Retrying command, 3 more tries remain (da5:mpr0:0:15:0): WRITE(10). CDB: 2a 00 0c d4 f5 80 00 01 00 00 (da5:mpr0:0:15:0): CAM status: CCB request completed with an error (da5:mpr0:0:15:0): Retrying command, 3 more tries remain (da5:mpr0:0:15:0): READ(10). CDB: 28 00 05 dc 12 28 00 00 f8 00 (da5:mpr0:0:15:0): CAM status: CCB request completed with an error (da5:mpr0:0:15:0): Retrying command, 3 more tries remain (da5:mpr0:0:15:0): READ(10). CDB: 28 00 05 dc 0f b0 00 00 88 00 (da5:mpr0:0:15:0): CAM status: CCB request completed with an error (da5:mpr0:0:15:0): Retrying command, 3 more tries remain (da5:mpr0:0:15:0): WRITE(10). CDB: 2a 00 02 96 7e 80 00 00 10 00 (da5:mpr0:0:15:0): CAM status: CCB request completed with an error (da5:mpr0:0:15:0): Retrying command, 3 more tries remain (da5:mpr0:0:15:0): READ(10). CDB: 28 00 6f 5b 8d 68 00 01 00 00 (da5:mpr0:0:15:0): CAM status: CCB request completed with an error (da5:mpr0:0:15:0): Retrying command, 3 more tries remain (da5:mpr0:0:15:0): WRITE(10). CDB: 2a 00 41 98 42 00 00 01 00 00 (da5:mpr0:0:15:0): CAM status: SCSI Status Error (da5:mpr0:0:15:0): SCSI status: Check Condition (da5:mpr0:0:15:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on, reset,=20 or bus device reset occurred) (da5:mpr0:0:15:0): Retrying command (per sense data) The same tests with Samsung disks work without issue or at least I was not = able to recreate the error.=20 # mprutil show adapter mpr0 Adapter: Board Name: INSPUR 3008IT Board Assembly: INSPUR Chip Name: LSISAS3008 Chip Revision: ALL BIOS Revision: 18.00.00.00 Firmware Revision: 16.00.12.00 Integrated RAID: no SATA NCQ: ENABLED PCIe Width/Speed: x8 (8.0 GB/sec) IOC Speed: Full Temperature: 56 C I originally ran into this problem with the same series of LSI adapter, but= it was not in IT mode and instead was using the mrsas driver.=20=20 When on the ATA controller the disks are DSM_TRIM. When on MPR, they are ATA_TRIM. --=20 You are receiving this mail because: You are the assignee for the bug.=