SAS3008 On FreeBSD 12 wGeli - OS Wont Boot

[Wallace Barrow]

Hi Scott,

I finally had some time to write back on this issue and it seems to get more odd while I troubleshoot it.

The server I was talking about when starting this post had to be put into production without Geli along with the 3 other Dell R640s just like it. This cluster of 4 servers, that are exactly the same, and had no issues installing FreeBSD 12 without Geli.

The 2nd cluster of servers consists of two Dell R640's as well. The only difference between this group from the four above is the Intel Silver CPU type, everything else is the same.

When trying to install FreeBSD 12 on these two servers without geli I ran into the max_chains error constantly. I got the error in the following ways:

When booting from the ISO I would get to the FreeBSD boot loader screen and let it continue, hoping I would get to the install screen but it would throw the max_chains error before that. Rebooting the server a few more times trying to install I randomly by chance got to the FreeBSD 12 installer and installed without issue, then on reboot, booting into my new install I got the max_chain error.

I fixed the issue by: When booting from the ISO and on the boot menu I set the boot value for max_chains to 4096 and got to the FreeBSD installer. Before rebooting after the installer finished I opened the shell and put that value in loader.conf. The server comes up cleanly every time now, 5+ reboots so far.

To	set the	maximum	number of DMA chains allocated for all adapters, set
     this tunable in loader.conf(5):

	   hw.mpr.max_chains=NNNN

The default max_chains value is 2048.

I had some issues with getting SOL working via SSH to get the console output, but, I was able to screen capture some errors when we hit the bug.

http://incin.me/bsd/chains1.png
http://incin.me/bsd/chains2.png
http://incin.me/bsd/chains3.png
http://incin.me/bsd/chains4.png

I hope the screen shots help. I also ran out of time to play with these servers and they are needed in our environment.

Thank you!

-- 
  Wallace Barrow
  incin at incin.me

On Fri, Apr 19, 2019, at 1:21 PM, Wallace Barrow wrote:
> Scott, 
> 
> You are correct. Installing onto a disk controlled by the SAS3008 works 
> fine when Geli is not enabled.
> 
> -- 
>   Wallace Barrow
>   incin at incin.me
> 
> On Fri, Apr 19, 2019, at 1:06 PM, Scott Long wrote:
> > From the loader, set
> > 
> > hw.mpr.debug_level=0xffff
> > 
> > It’ll dump a lot of text to the console when the kernel boots, but if 
> > you could send me all if it, I’d appreciate it.  For my understanding, 
> > installing onto a disk controlled by the SAS3008 works fine for you as 
> > long as you don’t enable GELI?
> > 
> > Thanks,
>