From nobody Wed Mar 27 16:29:47 2024 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4V4XFb4X43z5FmdM; Wed, 27 Mar 2024 16:29:55 +0000 (UTC) (envelope-from ticso@cicely7.cicely.de) Received: from raven.bwct.de (raven.bwct.de [IPv6:2a02:21e0:16e0:fe::101:1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "raven.bwct.de", Issuer "raven.bwct.de" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4V4XFZ6zxfz4JQq; Wed, 27 Mar 2024 16:29:54 +0000 (UTC) (envelope-from ticso@cicely7.cicely.de) Authentication-Results: mx1.freebsd.org; none Received: from mail.cicely.de (mail.cicely.de [IPv6:2a02:21e0:16e0:20fe:0:0:101:c]) by raven.bwct.de (8.15.2/8.15.2) with ESMTPS id 42RGTlPU029767 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=FAIL); Wed, 27 Mar 2024 17:29:50 +0100 (CET) (envelope-from ticso@cicely7.cicely.de) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=cicely.de; s=default; t=1711556990; bh=fByhYszKjckUNt9Bvb3w4qY6RMUwUbrFGSljt9RardQ=; h=Date:From:To:Cc:Subject:References:In-Reply-To; b=J9wf+S1XY6BhfFvkpCN/RRhXlOJ0HJ+Zc5xhI5kxvJR3RwvDtD3iWMi8XjukOLEl4 oPbscVLVUD4HrYdF4NYYjhTYEBsto0Tsvv5idpTtMy2yNssHH3u9GhejH30f+DUNml wHEcJxM1+9H0pcTaRqA49ihOjXRSdTQ8/KoxjcVw= Received: from cicely7.cicely.de (c7-old.cicely.de [IPv6:2a02:21e0:16e0:20fe:0:0:101:d]) by mail.cicely.de (8.16.1/8.16.1) with ESMTPS id 42RGTlcB057531 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Wed, 27 Mar 2024 17:29:47 +0100 (CET) (envelope-from ticso@cicely7.cicely.de) Received: from cicely7.cicely.de (localhost [127.0.0.1]) by cicely7.cicely.de (8.17.1/8.17.1) with ESMTP id 42RGTl9d000426; Wed, 27 Mar 2024 17:29:47 +0100 (CET) (envelope-from ticso@cicely7.cicely.de) Received: (from ticso@localhost) by cicely7.cicely.de (8.17.1/8.17.1/Submit) id 42RGTlQd000425; Wed, 27 Mar 2024 17:29:47 +0100 (CET) (envelope-from ticso) Date: Wed, 27 Mar 2024 17:29:47 +0100 From: Bernd Walter To: Konstantin Belousov Cc: ticso@cicely.de, freebsd-current@freebsd.org, freebsd-hackers@freebsd.org, Bernd Walter Subject: Re: Only 1TB RAM out of 1.5TB on amd64 Message-ID: References: List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Operating-System: FreeBSD cicely7.cicely.de 13.2-RELEASE-p3 amd64 X-Rspamd-Server: rspamd.fizon.de X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:44700, ipnet:2a02:21e0::/32, country:DE] X-Rspamd-Queue-Id: 4V4XFZ6zxfz4JQq On Wed, Mar 27, 2024 at 05:55:20PM +0200, Konstantin Belousov wrote: > On Wed, Mar 27, 2024 at 03:01:11AM +0100, Bernd Walter wrote: > > Same Problem with current: > > reeBSD 15.0-CURRENT #0 main-n268793-220ee18f1964: Thu Mar 14 02:58:39 UTC 2024 > > > > real memory = 1649265344512 (1572862 MB) > > avail memory = 1057118396416 (1008146 MB) > Real memory is really the max physical address of the valid byte. > If you have holes in the phys map, real memory should reflect these > holes. Avail memory is the total memory as reported by EFI map. Damn - thank you very much. That was the answer. I was under the impression that the machine has 1.5T installed, but in fact it has only 1T installed. I was expecting holes because of NUMA, but my assumption was more that the holes would led to memory being outside of the direct map space. > > Verify all that by looking at sysctl machdep.efi_map output. > > > > On Fri, Mar 22, 2024 at 07:54:06PM +0100, Bernd Walter wrote: > > > .. > > > real memory = 1649265344512 (1572862 MB) > > > avail memory = 1057138888704 (1008166 MB) > > > .. > > > > > > I suspect address space issues, but don't know for sure. > > > Changing NUMA nodes per socket in the BIOS had an influence to make it > > > worse, but not better. > > > > > > > > > Copyright (c) 1992-2021 The FreeBSD Project. > > > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 > > > The Regents of the University of California. All rights reserved. > > > FreeBSD is a registered trademark of The FreeBSD Foundation. > > > FreeBSD 13.3-RELEASE releng/13.3-n257428-80d2b634ddf0 GENERIC amd64 > > > FreeBSD clang version 17.0.6 (https://github.com/llvm/llvm-project.git llvmorg-17.0.6-0-g6009708b4367) > > > VT(efifb): resolution 1024x768 > > > CPU: AMD EPYC 7352 24-Core Processor (2300.07-MHz K8-class CPU) > > > Origin="AuthenticAMD" Id=0x830f10 Family=0x17 Model=0x31 Stepping=0 > > > Features=0x178bfbff > > > Features2=0x7ed8320b > > > AMD Features=0x2e500800 > > > AMD Features2=0x75c237ff > > > Structured Extended Features=0x219c91a9 > > > Structured Extended Features2=0x400004 > > > XSAVE Features=0xf > > > AMD Extended Feature Extensions ID EBX=0x18cf757 > > > SVM: NP,NRIP,VClean,AFlush,DAssist,NAsids=32768 > > > TSC: P-state invariant, performance statistics > > > real memory = 1649265344512 (1572862 MB) > > > avail memory = 1057138888704 (1008166 MB) > > > Event timer "LAPIC" quality 600 > > > ACPI APIC Table: > > > FreeBSD/SMP: Multiprocessor System Detected: 96 CPUs > > > FreeBSD/SMP: 2 package(s) x 8 cache groups x 3 core(s) x 2 hardware threads > > > random: registering fast source Intel Secure Key RNG > > > random: fast provider: "Intel Secure Key RNG" > > > random: unblocking device. > > > ioapic0 irqs 0-23 > > > ioapic1 irqs 24-55 > > > ioapic2 irqs 56-87 > > > ioapic3 irqs 88-119 > > > ioapic4 irqs 120-151 > > > ioapic5 irqs 152-183 > > > ioapic6 irqs 184-215 > > > ioapic7 irqs 216-247 > > > ioapic8 irqs 248-279 > > > Launching APs: 32 30 34 13 14 16 23 21 19 24 28 9 86 10 87 7 26 37 39 41 45 46 43 66 71 69 54 57 59 80 82 78 49 53 51 60 65 62 84 88 91 95 74 73 76 92 33 40 11 2 27 22 20 5 44 18 15 36 6 8 25 89 79 85 93 56 48 70 31 81 12 50 38 77 55 63 75 58 1 90 29 94 52 72 67 83 35 42 61 64 68 47 4 17 3 > > > random: entropy device external interface > > > kbd1 at kbdmux0 > > > .. > > > > > > kern.maxphys: 1048576 > > > hw.physmem: 1099322220544 > > > hw.usermem: 1091323355136 > > > hw.realmem: 1649265344512 > > > vm.phys_locality: > > > 0: 10 32 > > > 1: 32 10 > > > > > > vm.phys_segs: > > > SEGMENT 0: > > > > > > start: 0x10000 > > > end: 0xa0000 > > > domain: 0 > > > free list: 0xffffffff81ec7ea0 > > > > > > SEGMENT 1: > > > > > > start: 0x100000 > > > end: 0x1000000 > > > domain: 0 > > > free list: 0xffffffff81ec7ea0 > > > > > > SEGMENT 2: > > > > > > start: 0x1000000 > > > end: 0x74000000 > > > domain: 0 > > > free list: 0xffffffff81ec7c30 > > > > > > SEGMENT 3: > > > > > > start: 0x74043000 > > > end: 0x75db0000 > > > domain: 0 > > > free list: 0xffffffff81ec7c30 > > > > > > SEGMENT 4: > > > > > > start: 0x76000000 > > > end: 0x963b8000 > > > domain: 0 > > > free list: 0xffffffff81ec7c30 > > > > > > SEGMENT 5: > > > > > > start: 0x963c2000 > > > end: 0x963f9000 > > > domain: 0 > > > free list: 0xffffffff81ec7c30 > > > > > > SEGMENT 6: > > > > > > start: 0x96500000 > > > end: 0xa57f7000 > > > domain: 0 > > > free list: 0xffffffff81ec7c30 > > > > > > SEGMENT 7: > > > > > > start: 0xa8e96000 > > > end: 0xac000000 > > > domain: 0 > > > free list: 0xffffffff81ec7c30 > > > > > > SEGMENT 8: > > > > > > start: 0x10000e000 > > > end: 0x7d07800000 > > > domain: 0 > > > free list: 0xffffffff81ec79c0 > > > > > > SEGMENT 9: > > > > > > start: 0x10000019000 > > > end: 0x1797de00000 > > > domain: 1 > > > free list: 0xffffffff81ec8110 > > > > > > SEGMENT 10: > > > > > > start: 0x17ffdc00000 > > > end: 0x17ffdde7000 > > > domain: 1 > > > free list: 0xffffffff81ec8110 > > > > > > > > > > > > -- > > > B.Walter https://www.bwct.de > > > Modbus/TCP Ethernet I/O Baugruppen, ARM basierte FreeBSD Rechner uvm. > > > > > > > -- > > B.Walter https://www.bwct.de > > Modbus/TCP Ethernet I/O Baugruppen, ARM basierte FreeBSD Rechner uvm. -- B.Walter https://www.bwct.de Modbus/TCP Ethernet I/O Baugruppen, ARM basierte FreeBSD Rechner uvm.