From nobody Wed Apr 24 22:03:11 2024 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4VPtKN4Z9tz5JCFH for ; Wed, 24 Apr 2024 22:03:20 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-oi1-x233.google.com (mail-oi1-x233.google.com [IPv6:2607:f8b0:4864:20::233]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4VPtKM3mc8z4bQy for ; Wed, 24 Apr 2024 22:03:19 +0000 (UTC) (envelope-from markjdb@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=Uu1OoQg+; dmarc=fail reason="SPF not aligned (relaxed), DKIM not aligned (relaxed)" header.from=freebsd.org (policy=none); spf=pass (mx1.freebsd.org: domain of markjdb@gmail.com designates 2607:f8b0:4864:20::233 as permitted sender) smtp.mailfrom=markjdb@gmail.com Received: by mail-oi1-x233.google.com with SMTP id 5614622812f47-3c5ee4ce695so175644b6e.0 for ; Wed, 24 Apr 2024 15:03:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1713996194; x=1714600994; darn=freebsd.org; h=content-disposition:mime-version:message-id:subject:to:from:date :sender:from:to:cc:subject:date:message-id:reply-to; bh=vVdyD6o82Kb7F+1TulbIyQWboaHZ85N8WbdplVMGkHE=; b=Uu1OoQg+4e6vBbAwOgSwLMxGamH9BmyrTVV+nwXkM6ksGtEwRnyQAK2z4i2WMpdzyg YIs/19hXrx59FbZW7V8l8qgwnOEgUKYVP/lVJrfkU19ATQh/SyG85lw0bTAcKLrPAz95 Pz0uvH4ede7HPPUO5IqPFi9YbauZqMYNv56m4NmAzJK92cnSRiGfWAeAZj6QudPnRpvF 6QQy0GHFR5Rl1uW+LaqkueR9XQjAiSiE4M7QSYvYRjZaDMgNwEcKGlrkXgRllyGNXPQi nwJ94m4D4peMGjO2jU/UPGcEDPWkWtnjMOARrif1Fl5H3NODFm6sj/1RP/bVbwlHpS7A qvUg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713996194; x=1714600994; h=content-disposition:mime-version:message-id:subject:to:from:date :sender:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=vVdyD6o82Kb7F+1TulbIyQWboaHZ85N8WbdplVMGkHE=; b=Bj7u8O/lRFsEm1zMby0MUSRB2C3FyxNyFl5JmPgJGIZUkgGxuZaHmIGJaTicHF5BCl M5QxhMQmq+iCMzpz9JBQN50s9j/lSykGcLKS+xrGVf84EPgalxbmtNtMtBPJIFEFifR/ kaMu4P4AX+Or2nh50MWwRk1iNwPbTIp3Wt9QgcQ9uTbGjspwPeGb+hYfGYk6NjPiT+Fx avT9llk7xALbYyc/oLNSv17idoXb5nF3LVVzZ18HTk77w/Dqnj9moGU9rV8mKTTFGfCx bv7LGHtII5HjRP87yGJbuIn8A9As+VyFD0WDh5u8vg31N3Yhad6xPnZmoP5vgx1dptsI t3iQ== X-Gm-Message-State: AOJu0YyKn6YxI0cAPauqjKsnTpYat+jvxwZqAh8uIkiZGEd37Rz6ljC3 5ZP8bHwgv78T8QVNIvVtBCCpLvBL+w5XPijGw15v6hMQVVqyB8nIFa9wyA== X-Google-Smtp-Source: AGHT+IEWpDZTeHlcXXJoMyPgG9ck+FvBraZ73DehiGDt212768UcIoc3EXGUXIULegTaVFih575kVA== X-Received: by 2002:a05:6808:1707:b0:3c8:4aff:d7ae with SMTP id bc7-20020a056808170700b003c84affd7aemr1688629oib.51.1713996194393; Wed, 24 Apr 2024 15:03:14 -0700 (PDT) Received: from nuc (192-0-220-237.cpe.teksavvy.com. [192.0.220.237]) by smtp.gmail.com with ESMTPSA id j9-20020a05620a000900b0079092bc2a87sm614026qki.4.2024.04.24.15.03.13 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 24 Apr 2024 15:03:13 -0700 (PDT) Date: Wed, 24 Apr 2024 18:03:11 -0400 From: Mark Johnston To: freebsd-virtualization@freebsd.org Subject: PCI BAR registration overlap Message-ID: List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.59 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.993]; MID_RHS_NOT_FQDN(0.50)[]; FORGED_SENDER(0.30)[markj@freebsd.org,markjdb@gmail.com]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; MIME_GOOD(-0.10)[text/plain]; DMARC_POLICY_SOFTFAIL(0.10)[freebsd.org : SPF not aligned (relaxed), DKIM not aligned (relaxed),none]; RCVD_TLS_LAST(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; MIME_TRACE(0.00)[0:+]; ARC_NA(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MISSING_XM_UA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; TO_DN_NONE(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_NEQ_ENVFROM(0.00)[markj@freebsd.org,markjdb@gmail.com]; DKIM_TRACE(0.00)[gmail.com:+]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; TO_DOM_EQ_FROM_DOM(0.00)[]; MLMMJ_DEST(0.00)[freebsd-virtualization@freebsd.org]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::233:from] X-Rspamd-Queue-Id: 4VPtKM3mc8z4bQy I've been working on porting bhyvectl and vmrun.sh to arm64, as the initial arm64 bhyve port has landed in main. In the process I discovered a problem with bhyve's handling of BARs that I'd appreciate some help with. Suppose I configure a VM with a virtio-blk and virtio-net device. Both register BAR 0 as an I/O BAR with sizes of 128 and 64, respectively, and bhyve puts them at 0xdf000000 and 0xdf000080. If the network device is PCI device 2 and the block device is device 3, which happens automatically when using vmrun.sh, then for some reason u-boot suffles them around: it unmaps the network device BAR at 0xdf000080, then maps it at 0xdf000000, and then it unmaps the block device BAR at 0xdf000000, and puts it at 0xdf000080. I don't know yet why it does this; if the PCI device numbers are swapped then the BARs aren't moved around. This behaviour interacts with an oddity of bhyve's register_mem_init(): when u-boot maps the network device BAR at 0xdf000000, register_mem_init() does a lookup, finds that a MMIO region is there, and silently succeeds without inserting an entry into the MMIO RB tree. Later, u-boot moves the block device BAR, so nothing is mapped at 0xdf000000, and then when FreeBSD reprograms BARs it triggers an assertion failure because unregister_mem() isn't allowed to fail. What's the right solution? At a glance, QEMU appears to permit overlapping mappings and has a priority-based scheme for deciding which mapping a given access corresponds to. I verified that u-boot doesn't actually access the mappings in the window that the overlap, so it seems pretty safe to allow overlapping mappings and handle access conflicts with an assertion failure or so. Is there some context I'm missing? Should I look further into what u-boot is doing instead? Thanks in advance for any pointers.