From nobody Fri Apr 12 14:28:40 2024 X-Original-To: freebsd-arm@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4VGJpP0Mkgz5HTTD for ; Fri, 12 Apr 2024 14:28:45 +0000 (UTC) (envelope-from hp@tmm.cx) Received: from klara.tmm.cx (klara.tmm.cx [176.9.32.87]) by mx1.freebsd.org (Postfix) with ESMTP id 4VGJpN1rHtz4cxP for ; Fri, 12 Apr 2024 14:28:44 +0000 (UTC) (envelope-from hp@tmm.cx) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=tmm.cx header.s=mail header.b=lkSo7xWc; dmarc=pass (policy=reject) header.from=tmm.cx; spf=pass (mx1.freebsd.org: domain of hp@tmm.cx designates 176.9.32.87 as permitted sender) smtp.mailfrom=hp@tmm.cx Received: from [192.168.1.103] (D57D887A.static.ziggozakelijk.nl [213.125.136.122]) by klara.tmm.cx (Postfix) with ESMTPSA id 95BD19B31EB4 for ; Fri, 12 Apr 2024 14:28:41 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 klara.tmm.cx 95BD19B31EB4 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tmm.cx; s=mail; t=1712932123; bh=HIB6xQmnIxFhdWew7mdq/1QEI2wQb5hDocgV/emeOn8=; h=Subject:From:To:Date:From; b=lkSo7xWcesEJl7oQMCtvGGLTL48FCmrbs3MlVW3q76zsf8fyj8uSzEOnP4bJTpXK6 WFDJADAysDCmylGxTX8NPgsgiP+8lw2lekPQp2XZhgsXjYXaGkhxEBgCSjUKQKPVkM cgK7jaJ/Sb+FIppFLwnnNR6jE002aB4aM+AnH7Kc= Message-ID: <704b2199b4450557a9f615bd5373446a1a1a8cbb.camel@tmm.cx> Subject: DMA and PCIe on the Raspberry Pi Compute Module 4 From: HP van Braam To: freebsd-arm@FreeBSD.org Date: Fri, 12 Apr 2024 16:28:40 +0200 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable User-Agent: Evolution 3.52.0 (3.52.0-1.fc40) List-Id: Porting FreeBSD to ARM processors List-Archive: https://lists.freebsd.org/archives/freebsd-arm List-Help: List-Post: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Spam-Status: No, score=0.9 required=2.4 tests=NO_FM_NAME_IP_HOSTN,SPF_FAIL autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on klara.tmm.cx X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.78 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.98)[-0.982]; DMARC_POLICY_ALLOW(-0.50)[tmm.cx,reject]; R_SPF_ALLOW(-0.20)[+a:klara.tmm.cx]; R_DKIM_ALLOW(-0.20)[tmm.cx:s=mail]; MIME_GOOD(-0.10)[text/plain]; RCVD_NO_TLS_LAST(0.10)[]; ONCE_RECEIVED(0.10)[]; MIME_TRACE(0.00)[0:+]; TO_DN_NONE(0.00)[]; FROM_HAS_DN(0.00)[]; RCVD_COUNT_ONE(0.00)[1]; ARC_NA(0.00)[]; ASN(0.00)[asn:24940, ipnet:176.9.0.0/16, country:DE]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; PREVIOUSLY_DELIVERED(0.00)[freebsd-arm@freebsd.org]; MID_RHS_MATCH_FROM(0.00)[]; MLMMJ_DEST(0.00)[freebsd-arm@FreeBSD.org]; DKIM_TRACE(0.00)[tmm.cx:+] X-Rspamd-Queue-Id: 4VGJpN1rHtz4cxP Hello, I've been working on getting the RPI compute module 4 PCIe bus to work with external devices, and I've made some progress. The following patch enables l1ss and limits the pcie generation to 2, which in turn seems to make most things work (note that this needs some clean-up, I'm not trying to get this sub-par patch included :), without this patch booting with almost any PCIe card inserted will result in an SError during pcib initializaation. : diff --git a/sys/arm/broadcom/bcm2835/bcm2838_pci.c b/sys/arm/broadcom/bcm2835/bcm2838_pci.c index 2dfd6744127a..e2ecfb861697 100644 --- a/sys/arm/broadcom/bcm2835/bcm2838_pci.c +++ b/sys/arm/broadcom/bcm2835/bcm2838_pci.c @@ -74,6 +74,7 @@ #define REG_CPU_WINDOW_LOW 0x4070 #define REG_CPU_WINDOW_START_HIGH 0x4080 #define REG_CPU_WINDOW_END_HIGH 0x4084 +#define REG_PCIE_CAP 0x00ac =20 #define REG_MSI_ADDR_LOW 0x4044 #define REG_MSI_ADDR_HIGH 0x4048 @@ -730,6 +731,26 @@ bcm_pcib_attach(device_t dev) if (error !=3D 0) return (error); =20 + DELAY(100); + + uint32_t tmp =3D bcm_pcib_read_reg(sc, 0x4204); + tmp |=3D 0x2; + tmp |=3D 0x00200000; + bcm_pcib_set_reg(sc, 0x4204, tmp); + + DELAY(100); + + // Set PCIe generation to 2, any higher and the controller fails + uint16_t lnkctl2 =3D bcm_pcib_read_reg(sc,REG_PCIE_CAP + PCIER_LINK_CTL2); + uint32_t lnkcap =3D bcm_pcib_read_reg(sc, REG_PCIE_CAP + PCIER_LINK_CAP); + + lnkcap =3D (lnkcap & ~0x0000000f) | 2; + bcm_pcib_set_reg(sc, REG_PCIE_CAP + PCIER_LINK_CAP, lnkcap); + lnkctl2 =3D (lnkctl2 & ~0xf) | 2; + bcm_pcib_set_reg(sc, REG_PCIE_CAP + PCIER_LINK_CTL2, lnkctl2); + + DELAY(100); + /* Done. */ device_add_child(dev, "pci", -1); return (bus_generic_attach(dev)); After this patch PCIe devices seem to work, including a USB3 add-in card, various sata controllers, and an AMD GPU. However I have a problem that I can't seem to make work, when attaching a PCIe to PCI bridge to the controller, and then attaching a PCI SCSI card to that, things work well in initiator mode, I am able to run benchmarks on attached SCSI devices for hours without fail, and performance is exactly what I'd expect. However when using ahc(4) in target mode it appears I'm getting random garbage over the SCSI bus. I have tested this setup as well with a qemu aarch64 virtual machine and the same card passed through via vfio, and on an x86_64 machine in the same setup and after fixing a problem in ahc(4) see f1e4c09, these configurations both work. So I don't think there's a problem with either ctl(4) or ahc(4) at this point. I believe the problem is related to the DMA setup of the broadcom pci controller, but I'm really not sure where to even begin debugging this. The reason why I think that is that I can sometimes see strings of ascii text in the garbage over the SCSI bus, for instance text such as "libc.so.7" which kind of suggests I got some part of an inode perhaps? I have also tried to fill all of userspace accessible memory with the "@" symbol to see if perhaps I would get a bunch of "@" symbols over the scsi bus, but this never happened. Based on reading the Linux driver, and various forums on rpi4 and pcie I have also tried limiting the amount of ram on the cm4 to 1, 2, and 3 GB but all to no avail. I'm kind of at the end of my knowledge here and I'd love some help. Thank you! - HP