From nobody Sat Sep 24 13:38:16 2022 X-Original-To: dev-commits-src-main@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4MZVTN6XFJz4ch3M; Sat, 24 Sep 2022 13:38:16 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4MZVTN5ngxz42Dt; Sat, 24 Sep 2022 13:38:16 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1664026696; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=0PxB3U3tb5Y0dvev5D6Fdn4Jgnkq1tO5pXOVkYR/7/k=; b=auvaV0lEmPWJ277i35IOnSwvps8ID+hhMdALpqjeCKANxnjjC4gI1a1v9qdjCmF6yOkWoU 544dlXlK4rsFl0Dgkabd4kDWoWTdhL1f+SCwf3TpYC6jEWSesgVXQLwZWeQswIIY9xxxNG fBpowQh5DGEkzgeM5tqolYLTqO9R+fbKhi9RjRU4npYDHquGbZlEJRzsPgAS2/pS6SkFDM 4jcqJg5tM73dMxS1abZxRT8CTytfD3hn7vWcEaqk9LNTjXSzA655fMsKCcVsfRe8qrHrJI 2TXd8RMOOjAsdMMbJoiE+leGu6qVHeMTwidxY1WohiXPUbhgZkhjniYN8TRACw== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4MZVTN4Tc4z15VM; Sat, 24 Sep 2022 13:38:16 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 28ODcGW8023600; Sat, 24 Sep 2022 13:38:16 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 28ODcGko023598; Sat, 24 Sep 2022 13:38:16 GMT (envelope-from git) Date: Sat, 24 Sep 2022 13:38:16 GMT Message-Id: <202209241338.28ODcGko023598@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Mark Johnston Subject: git: 0b29f5efcc7e - main - amd64: Make it possible to grow the KERNBASE region of KVA List-Id: Commit messages for the main branch of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-main List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-main@freebsd.org X-BeenThere: dev-commits-src-main@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: markj X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: 0b29f5efcc7ee8271ad2f6b6447898b489d618ec Auto-Submitted: auto-generated ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1664026696; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=0PxB3U3tb5Y0dvev5D6Fdn4Jgnkq1tO5pXOVkYR/7/k=; b=tHTxD2bj9S6hst8mGYSjL6ANSJ35u9FAg2/SicXMU8CoR46abeTG9RvBh9OaCVJkR9WpbW 6AftegQg1ZQRdwRtHTprVpfZ9cTzMzNsi7zd+s+eGhfMlzvFpPSjiK281BdG1zlfvdubMS b2AaWoahbvRxz3vKT58cKeS1bmvPmcOv9NATnDGRpJzxPWpt0Ou8IyFoHa054sSm4GpAVG CZ8RGzRikwIt6VEQbKD3D7odxepLd8oof71ukGYXiprY9NQ3VzsCnQmHdwTctDSBf5XSYi Lh76v1J3IUhrBz631mbEvVcGydiSKt36ecokl8AP9zuvddtEqhEigLnw3Ez63g== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1664026696; a=rsa-sha256; cv=none; b=vKACNFmCM6OwvWhjPtHxAubtxKqyBeNjzT3X/OW982qkHsNO3FjIbTQzDtJhiChPYeOOJl FjbnMpKiUOhk4PEGs2Vlzxu+7XahZhOXSY0yyckUPqGfnmBTPwzH0XfiaWgJIUxRpTiAfI oJOEj2ULvYlpVHuRDOuGbovWcWTeUN+MV+nHnZOLzvaugdYIWWIvIc5D/weXt5iUwiUTAx hyFkv5LmrDTfCcdg1yl1TJBJeiYNNqkTQRowm6LamxgDqYsgPnaSgNmC+mGt47PwJITWag TMEAJWT6w6kWQB9zCFlvuOKAhPl6XrjUhzETQaIBnOZ+qT5fZj+lPb3mIdcJWw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N The branch main has been updated by markj: URL: https://cgit.FreeBSD.org/src/commit/?id=0b29f5efcc7ee8271ad2f6b6447898b489d618ec commit 0b29f5efcc7ee8271ad2f6b6447898b489d618ec Author: Mark Johnston AuthorDate: 2022-09-24 13:19:21 +0000 Commit: Mark Johnston CommitDate: 2022-09-24 13:27:50 +0000 amd64: Make it possible to grow the KERNBASE region of KVA pmap_growkernel() may be called when mapping a region above KERNBASE, typically for a kernel module. If we have enough PTPs left over from bootstrap, pmap_growkernel() does nothing. However, it's possible to run out, and in this case pmap_growkernel() will try to grow the kernel map all the way from kernel_vm_end to somewhere past KERNBASE, which can easily run the system out of memory. This happens with large kernel modules such as the nvidia GPU driver. There is also a WIP dtrace provider which needs to map KVA in the region above KERNBASE (to provide trampolines which allow a copy of traced kernel instruction to be executed), and its allocations could potentially trigger this scenario. This change modifies pmap_growkernel() to manage the two regions separately, allowing them to grow independently. The end of the KERNBASE region is tracked by modifying "nkpt". PR: 265019 Reviewed by: alc, imp, kib MFC after: 2 weeks Differential Revision: https://reviews.freebsd.org/D36673 --- sys/amd64/amd64/pmap.c | 68 +++++++++++++++++++++++++++++++++----------------- 1 file changed, 45 insertions(+), 23 deletions(-) diff --git a/sys/amd64/amd64/pmap.c b/sys/amd64/amd64/pmap.c index d7aeb8dcbd98..9a33298944cc 100644 --- a/sys/amd64/amd64/pmap.c +++ b/sys/amd64/amd64/pmap.c @@ -4997,13 +4997,21 @@ pmap_growkernel(vm_offset_t addr) vm_page_t nkpg; pd_entry_t *pde, newpdir; pdp_entry_t *pdpe; + vm_offset_t end; mtx_assert(&kernel_map->system_mtx, MA_OWNED); /* - * Return if "addr" is within the range of kernel page table pages - * that were preallocated during pmap bootstrap. Moreover, leave - * "kernel_vm_end" and the kernel page table as they were. + * The kernel map covers two distinct regions of KVA: that used + * for dynamic kernel memory allocations, and the uppermost 2GB + * of the virtual address space. The latter is used to map the + * kernel and loadable kernel modules. This scheme enables the + * use of a special code generation model for kernel code which + * takes advantage of compact addressing modes in machine code. + * + * Both regions grow upwards; to avoid wasting memory, the gap + * in between is unmapped. If "addr" is above "KERNBASE", the + * kernel's region is grown, otherwise the kmem region is grown. * * The correctness of this action is based on the following * argument: vm_map_insert() allocates contiguous ranges of the @@ -5015,22 +5023,32 @@ pmap_growkernel(vm_offset_t addr) * any new kernel page table pages between "kernel_vm_end" and * "KERNBASE". */ - if (KERNBASE < addr && addr <= KERNBASE + nkpt * NBPDR) - return; + if (KERNBASE < addr) { + end = KERNBASE + nkpt * NBPDR; + if (end == 0) + return; + } else { + end = kernel_vm_end; + } addr = roundup2(addr, NBPDR); if (addr - 1 >= vm_map_max(kernel_map)) addr = vm_map_max(kernel_map); - if (kernel_vm_end < addr) - kasan_shadow_map(kernel_vm_end, addr - kernel_vm_end); - if (kernel_vm_end < addr) - kmsan_shadow_map(kernel_vm_end, addr - kernel_vm_end); - while (kernel_vm_end < addr) { - pdpe = pmap_pdpe(kernel_pmap, kernel_vm_end); + if (addr <= end) { + /* + * The grown region is already mapped, so there is + * nothing to do. + */ + return; + } + + kasan_shadow_map(end, addr - end); + kmsan_shadow_map(end, addr - end); + while (end < addr) { + pdpe = pmap_pdpe(kernel_pmap, end); if ((*pdpe & X86_PG_V) == 0) { - /* We need a new PDP entry */ nkpg = pmap_alloc_pt_page(kernel_pmap, - kernel_vm_end >> PDPSHIFT, VM_ALLOC_WIRED | + pmap_pdpe_pindex(end), VM_ALLOC_WIRED | VM_ALLOC_INTERRUPT | VM_ALLOC_ZERO); if (nkpg == NULL) panic("pmap_growkernel: no memory to grow kernel"); @@ -5039,31 +5057,35 @@ pmap_growkernel(vm_offset_t addr) X86_PG_A | X86_PG_M); continue; /* try again */ } - pde = pmap_pdpe_to_pde(pdpe, kernel_vm_end); + pde = pmap_pdpe_to_pde(pdpe, end); if ((*pde & X86_PG_V) != 0) { - kernel_vm_end = (kernel_vm_end + NBPDR) & ~PDRMASK; - if (kernel_vm_end - 1 >= vm_map_max(kernel_map)) { - kernel_vm_end = vm_map_max(kernel_map); + end = (end + NBPDR) & ~PDRMASK; + if (end - 1 >= vm_map_max(kernel_map)) { + end = vm_map_max(kernel_map); break; } continue; } - nkpg = pmap_alloc_pt_page(kernel_pmap, - pmap_pde_pindex(kernel_vm_end), VM_ALLOC_WIRED | - VM_ALLOC_INTERRUPT | VM_ALLOC_ZERO); + nkpg = pmap_alloc_pt_page(kernel_pmap, pmap_pde_pindex(end), + VM_ALLOC_WIRED | VM_ALLOC_INTERRUPT | VM_ALLOC_ZERO); if (nkpg == NULL) panic("pmap_growkernel: no memory to grow kernel"); paddr = VM_PAGE_TO_PHYS(nkpg); newpdir = paddr | X86_PG_V | X86_PG_RW | X86_PG_A | X86_PG_M; pde_store(pde, newpdir); - kernel_vm_end = (kernel_vm_end + NBPDR) & ~PDRMASK; - if (kernel_vm_end - 1 >= vm_map_max(kernel_map)) { - kernel_vm_end = vm_map_max(kernel_map); + end = (end + NBPDR) & ~PDRMASK; + if (end - 1 >= vm_map_max(kernel_map)) { + end = vm_map_max(kernel_map); break; } } + + if (end <= KERNBASE) + kernel_vm_end = end; + else + nkpt = howmany(end - KERNBASE, NBPDR); } /***************************************************