Dell NVMe issues

[Slawa Olhovchenkov]

On Tue, Oct 06, 2015 at 10:35:57AM -0500, Sean Kelly wrote:

> 
> 
> > On Oct 6, 2015, at 10:29 AM, Slawa Olhovchenkov <slw at zxy.spb.ru> wrote:
> > 
> > On Tue, Oct 06, 2015 at 10:18:11AM -0500, Sean Kelly wrote:
> > 
> >> Back in May, I posted about issues I was having with a Dell PE R630 with 4x800GB NVMe SSDs. I would get kernel panics due to the inability to assign all the interrupts because of https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=199321 <https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=199321> <https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=199321 <https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=199321>>. Jim Harris helped fix this issue so I bought several more of these servers, Including ones with 4x1.6TB drives...
> >> 
> >> while the new servers with 4x800GB drives still work, the ones with 4x1.6TB drives do not. When I do a
> >> 	zpool create tank mirror nvd0 nvd1 mirror nvd2 nvd3
> >> the command never returns and the kernel logs:
> >> 	nvme0: resetting controller
> >> 	nvme0: controller ready did not become 0 within 2000 ms
> >> 
> >> I've tried several different things trying to understand where the actual problem is.
> >> WORKS: dd if=/dev/nvd0 of=/dev/null bs=1m
> >> WORKS: dd if=/dev/zero of=/dev/nvd0 bs=1m
> >> WORKS: newfs /dev/nvd0
> >> FAILS: zpool create tank mirror nvd[01]
> >> FAILS: gpart add -t freebsd-zfs nvd[01] && zpool create tank mirror nvd[01]p1
> >> FAILS: gpart add -t freebsd-zfs -s 1400g nvd[01[ && zpool create tank nvd[01]p1
> >> WORKS: gpart add -t freebsd-zfs -s 800g nvd[01] && zpool create tank nvd[01]p1
> >> 
> >> NOTE: The above commands are more about getting the point across, not validity. I wiped the disk clean between gpart attempts and used GPT.
> > 
> > Just for purity of the experiment: do you try zpool on raw disk, w/o
> > GPT? I.e. zpool create tank mirror nvd0 nvd1
> > 
> 
> Yes, that was actually what I tried first. I headed down the path of
> GPT because it allowed me a way to restrict how much disk zpool
> touched. zpool on the bare NVMe disks also triggers the issue.

Can you snoop disk i/o operation by dtrace at time zpool create?