process shared mutexes?

[Konstantin Belousov]

On Mon, Nov 21, 2016 at 05:14:54PM +0100, Volker Lendecke wrote:
> On Mon, Nov 21, 2016 at 05:58:23PM +0200, Konstantin Belousov wrote:
> > On Mon, Nov 21, 2016 at 04:25:42PM +0100, Volker Lendecke wrote:
> > > On Mon, Nov 21, 2016 at 05:10:40PM +0200, Konstantin Belousov wrote:
> > > > Please see the libthr(3) man page, in particular, read the RUN-TIME
> > > > SETTINGS section, the description of the kern.ipc.umtx_vnode_persistent
> > > > sysctl.
> > > > 
> > > > Does setting the sysctl to 1 allow your program to run ?
> > > 
> > > Yes, that does make it work. The description says that the umtx vnode
> > > is dropped on the last munmap. If I #if 0 the middle mmap, it works,
> > > although there is no mmap around anymore. So the description is not
> > > 100% accurate I'd say.
> > No, not umtx vnode is dropped, but the shared umutexes attached to the
> > file' page.
> > 
> > What you observed is the consequence of an implementation detail.  It is
> > impossible to execute umtx cleanup while dereferencing vm object, due to
> > locking issues.  An asynchronous task is scheduled to perform the cleanup.
> > But when the task is run, it is quite possible that your process is already
> > executed second mmap() and pthread_mutex_lock(), creating another reference
> > on the umtx data.
> 
> Hmm. If I do a poll(NULL, 0, 60000) between the munmap and mmap
> without the intervening mmap, it still works. It's really the
> mmap(NULL,0xb0...) that kills it.
Yes, because at that time, the cleanup task already completed.  So the
mutex is auto-reinited with default attributes, but as shared.  When
the cleanup is pending, the mutex off-page data is marked for pending
removal, and lookup of that data in pthread_mutex_lock() returns an
error.

This is unfortunate consequence of the initially limiting ABI which
we are trying to preserve still.
> 
> > Look at the Single Unix Specification, particularly to the following
> > paragraph in the mmap() description:
> > 
> > The state of synchronization objects such as mutexes, semaphores,
> > barriers, and conditional variables placed in shared memory mapped
> > with MAP_SHARED becomes undefined when the last region in any process
> > containing the synchronization object is unmapped.
> 
> Thanks! Hidden deep in mmap(2)... No hint in any of the pthread calls.
> 
> So -- all of the above discussion becomes irrelevant if I change tdb
> such that it keeps the mutex area mmappe'd at least once? Then no GC
> will kick in regardless of the sysctl? This would be possible, because
> we use mutexes on so-called CLEAR_IF_FIRST databases only. When the last
> process closes the db, it will be wiped on the next open.

If the file is mmaped, then yes, the mutex must be not destroyed.  If it
is, then there is a bug in the current implementation.