solaris assert: avl_is_empty(&dn -> dn_dbufs) panic

[Jason Unovitch]

For reference I opened https://bugs.FreeBSD.org/202607 before I came
across this discussion.

> Hi,
> 
> I'll need a little time to fully reload the context for these changes. However, reintroducing a blocking loop is not the right fix - it was a hack in the original code. :-) My hunch is that removing the assert is safe, but it would be nice to have a core dump to better understand why the list isn't empty.
> 
> --
> Justin
> 

Justin,
I have the contents of my /var/crash available.  I also have a beadm
boot environment of a known bad (r287028) as well as the known good
(r286204) that I am currently running on.  I should be able to replicate
this as needed and provide some assistance.

<snip>
> > On Aug 21, 2015, at 12:38 PM, Xin Li <delphij at delphij.net> wrote:
> > 
> > Hi,
> > 
> > A quick glance at the changes suggests that Justin's changeset may be
> > related.  The reasoning is here:
> > 
> > https://reviews.csiden.org/r/131/
> > 
> > Related Illumos ticket:
> > 
> > https://www.illumos.org/issues/5056
> > 
> > In dnode_evict_dbufs(), remove multiple passes over dn->dn_dbufs.
> > This is possible now that objset eviction is asynchronously
> > completed in a different context once dbuf eviction completes.
> > 
> > In the case of objset eviction, any dbufs held by children will
> > be evicted via dbuf_rele_and_unlock() once their refcounts go
> > to zero.  Even when objset eviction is not active, the ordering
> > of the avl tree guarantees that children will be released before
> > parents, allowing the parent's refcounts to naturally drop to
> > zero before they are inspected in this single loop.
> > 
> > ====
> > 
> > So, upon return from dnode_evict_dbufs(), there could be some
> > DB_EVICTING buffers on the AVL pending release and thus breaks the
> > invariant.
> > 
> > Should we restore the loop where we yield briefly with the lock
> > released, then reacquire and recheck?
> > 
> > Cheers,

Jason