From nobody Wed Jul 12 11:48:01 2023 X-Original-To: bugs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4R1GFt42Tdz4lTLS for ; Wed, 12 Jul 2023 11:48:02 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4R1GFs2vw3z44Bt for ; Wed, 12 Jul 2023 11:48:01 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1689162481; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=26Nmsi2z63mmwH0ZWdiKapH5iWlJt8ecnwW/tjjdJcs=; b=IqIuoDBM6bjbOpmbNaKEmPfRl3bFB5rNXw0/11ZUhO3vM77VLC0UL50e8BOZWOd8Koh3Xc 1/C1B7vXhzrA/6SG36Qgvboj0EFxiHhQge4kfx+kE6Ar4J2EPREikyyhulRhr9Qrkk9c5R 3hehwLy0zhz9LdaMJdTrZv/SlrxMwWtfhMLHdHhIgol+TbDUJOq458No2GXHdY7WOz7MmA 823vihhruLPJ8/GQ2uwAVB2H6BoAmA3PTK9fACkZbwFZ6BF9j6H4Czz3NRMwzqxedczFw3 xcFNXniaExDePPnt1YY4aHD0PjxoVMeVphkK2DB/YyNC9WCL0hx0l9o8L00BwQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1689162481; a=rsa-sha256; cv=none; b=cKr+3k+RhGQip/DTc6pk3So/9M7OU3GzgH6bs9OZLDok3Nrtx2VBJR0p3WtVO2EFKANYVJ HAOj9z/fd9EOiUDXz82jfX+PshUj/uRqtknp/HWYpFs8eDgI5V5wHRRW+FTwkV/kJD1eU3 vLgIBV0zuws101JQEK26/n7eFWcsq/AUTd4la7jOFJJItjmPu+XI7TM2nf/dKLvUe8wJZo VPprpPwHTlhX7P4dXSbOqj/1tlmm/Oh3mrKnyLaW1+9NT6cTcUdIh1DRRl5/glnfNUtueD xlN9Mjke2v4Ydyxxqy41GYtTFsDljxDrOAEldxyTdi6zAS+GXfXVAwnN48Zi+Q== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4R1GFs1vp5z171n for ; Wed, 12 Jul 2023 11:48:01 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 36CBm1Nt050699 for ; Wed, 12 Jul 2023 11:48:01 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 36CBm1mt050698 for bugs@FreeBSD.org; Wed, 12 Jul 2023 11:48:01 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: bugs@FreeBSD.org Subject: [Bug 272469] Broadcom mpi3mr driver: MSIX allocation fail on DELL PowerEdge R7625 system Date: Wed, 12 Jul 2023 11:48:01 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 13.2-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: chandrakanth.patil@broadcom.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: bugs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status bug_severity priority component assigned_to reporter attachments.created Message-ID: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Bug reports List-Archive: https://lists.freebsd.org/archives/freebsd-bugs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-bugs@freebsd.org MIME-Version: 1.0 X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D272469 Bug ID: 272469 Summary: Broadcom mpi3mr driver: MSIX allocation fail on DELL PowerEdge R7625 system Product: Base System Version: 13.2-RELEASE Hardware: amd64 OS: Any Status: New Severity: Affects Many People Priority: --- Component: kern Assignee: bugs@FreeBSD.org Reporter: chandrakanth.patil@broadcom.com Created attachment 243352 --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=3D243352&action= =3Dedit msix_table_dump mpi3mr avenger driver: system details:=20 1. Dell PowerEdge R7625 with 196 physical cores and 256 logical cores mpi3mr driver will allocate the single msix for handshaking with the driver during the initial load phase using pci_alloc_msix() API. After allocating the sin= gle msix, the driver is sending the get IOC_FACTS commands to firmware through which the driver will fetch all the controller properties. The issue is the driver is not getting the interrupt for IOC_FACTS completion leads to timeo= ut which in turn leads to driver load failure. but the driver can see that the command is completed by the firmware if it polls the reply queue. After creating the single msix in the driver, the vmstat -i in the OS should show the interrupt but it is not showing so the interrupt binding is failin= g. ideally in this case the pci_alloc_msix() API should throw some error during allocation but it is not throwing any error. Note:=20 1. This issue is happening only on this specific server where the numb= er of=20 CPUs are > 128 (total CPUs are 256). 2. But when we reduce the number of cores to 24 in the BIOS then the driver=20 is working without any issues.=20 We have dumped the MSIX table before and after the allocation of a single m= six and after the command times out. Please find it in the attachment. I wanted to understand if is there any OS limitation w.r.t MSIX allocation = on larger cores system. Please find attached driver logs and MSIX table dump. --=20 You are receiving this mail because: You are the assignee for the bug.=