From nobody Sat Apr 13 10:40:15 2024 X-Original-To: xen@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4VGqhL5dvdz5HCZ9 for ; Sat, 13 Apr 2024 10:40:18 +0000 (UTC) (envelope-from roger.pau@cloud.com) Received: from mail-wm1-x32c.google.com (mail-wm1-x32c.google.com [IPv6:2a00:1450:4864:20::32c]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4VGqhL3hZqz4njq for ; Sat, 13 Apr 2024 10:40:18 +0000 (UTC) (envelope-from roger.pau@cloud.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-wm1-x32c.google.com with SMTP id 5b1f17b1804b1-41550858cabso11598085e9.2 for ; Sat, 13 Apr 2024 03:40:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=citrix.com; s=google; t=1713004817; x=1713609617; darn=freebsd.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=X7pZm8cWV9cPjYooBvPnuFBbCgm90N82Cq1O/oCNJ4A=; b=BvaFs9zma1qKhuU0a18DTQ2qepnThneaekQjM6kZWqV41BY945LO5jidkmugVFAoXA ig0insBTmPN+/TdfCzqFk3uJlYik2OEGdVuQHU54COpJZi7j8kkhviAT7TzwuC5chmiy EomKcmyB3x/m32qnHd9bampqRTst3woqVaVXk= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713004817; x=1713609617; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=X7pZm8cWV9cPjYooBvPnuFBbCgm90N82Cq1O/oCNJ4A=; b=SGJG7T3lQTB4MkixgCphKZW9VMzIdSVdVn6JTH1BFZwbMgq2ay2lrUFdjG+kA/d7Ta kbI3RR+VJP1bY7HcilKmmgRm7IcCblIBGzb2Vn0B2Kt7FHuXLicUaVNLx/abtwPW5fNY EgKFQaUcZk/K0fR8Z2R211cHuHmS2G8mwIRGverXJ9LuiETmI/5ir8z1IbUE5DTJF66j 4CIQNm1yGQZVS0SohDODV23XklnFIeJBjhTWO4WXxB49mtocGcrQAVOi+qLUKHk7RM9E guimFU/Af373u4+MLHiVORcl3AW9J5B6YzuBonSoG2I3cG74k556vcQPKPEeef4hwdnD OulA== X-Gm-Message-State: AOJu0YzM4rF8qvqnvE2dcrWdGLi139GU31dL99Xj1O4tuer9jiKQEECt NcGjiQeYjILBtl7DwXsQN49mVY8ZC/N8Iy87D00JbyomAq8G8J5qXQ7SOayMnyf/RxBCt9GsIq9 4 X-Google-Smtp-Source: AGHT+IE0cw5enu6bGSJPbhbxIVCkNg/q8JCLvw/OQpfjzvxuQxhT3SvBXq/EQ7jDbYoTXWkh92YGnA== X-Received: by 2002:a05:600c:1e0f:b0:418:19d4:fe5c with SMTP id ay15-20020a05600c1e0f00b0041819d4fe5cmr1298315wmb.14.1713004816901; Sat, 13 Apr 2024 03:40:16 -0700 (PDT) Received: from localhost ([85.31.135.62]) by smtp.gmail.com with ESMTPSA id w7-20020a5d6807000000b0034354a99d43sm6295269wru.43.2024.04.13.03.40.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 13 Apr 2024 03:40:16 -0700 (PDT) Date: Sat, 13 Apr 2024 12:40:15 +0200 From: Roger Pau =?utf-8?B?TW9ubsOp?= To: Brian Buhrow Cc: xen@freebsd.org Subject: Re: Xen-4.16.0 + FreeBSD-13.1 dom0 fails on large ADM64 system Message-ID: References: <202404130557.43D5vWP5014615@nfbcal.org> List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-xen List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-xen@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <202404130557.43D5vWP5014615@nfbcal.org> X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Queue-Id: 4VGqhL3hZqz4njq Hello, Thanks for the report. On Fri, Apr 12, 2024 at 10:57:32PM -0700, Brian Buhrow wrote: > Hello. I'm trying to load xen on a large AMD64 server, with 600G of RAM and 56 CPUs. I'm > using a xen-4.16 image with FreeBSD-13.1, a duplicate image, in fact, with a couple of other > machines I have running in production and which have been running without issues for a couple > of years. I think I'm running into some defined limits in the sense that I think the machine > has more of something than xen is configured for. Unfortunately, pouring through the source > doesn't clue me into exactly what's rong, except I think it has something to do with mmio > allocations. Can someone take a look at the boot message below and give a clue as to what > might be wrong? Do I need to recompile xen with some new limits, turn something on or off in > BIOS or change some parameter on the xen command line? Would it be possible for you to test with the latest version of Xen in the ports tree (currently 4.18.2.20240411). It would also be helpful if you could boot with a debug Xen kernel instead of the release one, for that you must set the following in /boot/loader.conf: xen_kernel="/boot/xen-debug" > Or, is this something that requires a > newer version of Xen? Any thoughts would be greatly appreciated. > > The error is: Failed to identity map [ffffffffc7ffb, ffffffffc7ffb] for d0: -22 > > that's EINVAL, from sys/errno.h, suggesting something is out of range. As a first guess I think there's a PCI device with a BAR positioned at 0xffffffffc7ffb, and that's outside of the physical range supported by Xen EPT. That physical address uses 52bits, while the EPT implementation in Xen only supports up to 48bit wide physical addresses. Is there any option in the BIOS/firmware that you can set to attempt to prevent the BIOS from positioning BARs past the 48bit boundary? The error you are getting is kind of expected, however the page-fault (and Xen crash) that you hit afterwards is definitely not expected. AFAICT the page-fault is likely fixed by Xen commit: 465217b0f872 vPCI: account for hidden devices Which is present in 4.18. Please test with that version and report back. Thanks, Roger.