9.2 + ZFS + i386 = panic: pmap_enter: attempted pmap_enter on 4MB page

[Konstantin Belousov]

On Tue, Oct 15, 2013 at 11:17:53AM -0400, J David wrote:
> What are the necessary loader.conf / kernel config invocations to make
> ZFS stable on 9.2-RELEASE i386 node with 3GB RAM?
> 
> This machine was rock solid under 8.4, but since upgrading to 9.2 it
> has been a disaster.  It crashes every few hours with "panic:
> pmap_enter: attempted pmap_enter on 4MB page."
> 
> Here are a couple of stack traces:
> 
> panic: pmap_enter: attempted pmap_enter on 4MB page
> cpuid = 0
> KDB: stack backtrace:
> #0 0xc0afc092 at kdb_backtrace+0x52
> #1 0xc0ac249c at panic+0x1bc
> #2 0xc0f1825d at pmap_enter+0x63d
> #3 0xc0d34ee5 at vm_fault_hold+0x1c45
> #4 0xc0d33262 at vm_fault+0x82
> #5 0xc0f1f5d6 at trap_pfault+0x186
> #6 0xc0f1ed4b at trap+0x51b
> #7 0xc0f0887c at calltrap+0x6
> #8 0xc16770b6 at zio_execute+0x116
> #9 0xc15d9460 at taskq_run_safe+0x10
> #10 0xc0b08f26 at taskqueue_run_locked+0xe6
> #11 0xc0b097f7 at taskqueue_thread_loop+0xb7
> #12 0xc0a90a73 at fork_exit+0xa3
> #13 0xc0f08924 at fork_trampoline+0x8
> 
> panic: pmap_enter: attempted pmap_enter on 4MB page
> cpuid = 1
> KDB: stack backtrace:
> #0 0xc0afc092 at kdb_backtrace+0x52
> #1 0xc0ac249c at panic+0x1bc
> #2 0xc0f1825d at pmap_enter+0x63d
> #3 0xc0d34ee5 at vm_fault_hold+0x1c45
> #4 0xc0d33262 at vm_fault+0x82
> #5 0xc0f1f5d6 at trap_pfault+0x186
> #6 0xc0f1ed4b at trap+0x51b
> #7 0xc0f0887c at calltrap+0x6
> #8 0xc1679bdf at zio_dva_allocate+0x9f
> #9 0xc16770b6 at zio_execute+0x116
> #10 0xc15d9460 at taskq_run_safe+0x10
> #11 0xc0b08f26 at taskqueue_run_locked+0xe6
> #12 0xc0b097f7 at taskqueue_thread_loop+0xb7
> #13 0xc0a90a73 at fork_exit+0xa3
> #14 0xc0f08924 at fork_trampoline+0x8
> 
> For the last crash, top was running and this is what was on the screen
> when it died:
> 
> CPU:  0.6% user,  0.0% nice,  2.2% system,  0.2% interrupt, 97.1% idle
> Mem: 442M Active, 91M Inact, 79M Wired, 3764K Cache, 4960K Buf, 2380M Free
> ARC: 40M Total, 8520K MFU, 30M MRU, 304K Anon, 494K Header, 1062K Other
> Swap: 4096M Total, 4096M Free
> Write failed: Broken pipe
> 
> It doesn't seem to matter what KVA_PAGES, vm.kmem_size,
> vfs.zfs.arc_max or vfs.zfs.vdev.cache.size is set to, and the ZFS
> tuning guides in the wiki (albeit appearing to be dating to the 7.x
> era) provides guidelines for tuning down to 768M.  So 3GB should be
> enough for a machine that is 99% idle.  (Particularly given that it
> dies with >2GB free.)
> 
> This seems to be specific to i386, this problem hasn't cropped up on
> any amd64 nodes.  No compression, deduplication, snapshots or anything
> like that is in use.

It seems that i386 pmap_enter() cannot deal with the superpages, while
amd64 can.  On the other hand, I am not quite sure that this is your
problem, because you get traps on accessing KVA, and I suspect that
ZFS uses wired memory.  Obtain the kernel dump and do full backtrace
for the panic.

Anyway, I believe that i386 pmap_enter() should do a demotion when needed.
Below is the prototyped change for this.

diff --git a/sys/i386/i386/pmap.c b/sys/i386/i386/pmap.c
index 64bf1a3..91453da 100644
--- a/sys/i386/i386/pmap.c
+++ b/sys/i386/i386/pmap.c
@@ -3473,17 +3473,21 @@ pmap_enter(pmap_t pmap, vm_offset_t va, vm_prot_t access, vm_page_t m,
 	PMAP_LOCK(pmap);
 	sched_pin();
 
-	/*
-	 * In the case that a page table page is not
-	 * resident, we are creating it here.
-	 */
-	if (va < VM_MAXUSER_ADDRESS) {
+	pde = pmap_pde(pmap, va);
+	if ((*pde & PG_PS) != 0) {
+		/* PG_V is asserted by pmap_demote_pde */
+		pmap_demote_pde(pmap, pde, va);
+		if (va < VM_MAXUSER_ADDRESS) {
+			mpte = PHYS_TO_VM_PAGE(*pde & PG_FRAME);
+			mpte->wire_count++;
+		}
+	} else if (va < VM_MAXUSER_ADDRESS) {
+		/*
+		 * In the case that a page table page is not resident,
+		 * we are creating it here.
+		 */
 		mpte = pmap_allocpte(pmap, va, M_WAITOK);
 	}
-
-	pde = pmap_pde(pmap, va);
-	if ((*pde & PG_PS) != 0)
-		panic("pmap_enter: attempted pmap_enter on 4MB page");
 	pte = pmap_pte_quick(pmap, va);
 
 	/*
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 834 bytes
Desc: not available
URL: <http://lists.freebsd.org/pipermail/freebsd-fs/attachments/20131015/60fa7f1a/attachment.sig>