close(2) while accept(2) is blocked

[Jilles Tjoelker]

On Thu, Mar 28, 2013 at 06:54:31PM +0200, Andriy Gapon wrote:

> So, this started as a simple question, but the answer was quite
> unexpected to me.

> Let's say we have an opened and listen-ed socket and let's assume that
> we know that one thread is blocked in accept(2) and another thread is
> calling close(2). What is going to happen?

> Turns out that practically nothing.  For kernel the close call would
> be almost a nop.
> My understanding is this:
> - when socket is created, its reference count is 1
> - when accept(2) is called, fget in kernel increments the reference
>   count (kept in an associated struct file)
> - when close(2) is called, the reference count is decremented

> The reference count is still greater than zero, so fdrop does not call
> fo_close. That means that in the case of a socket soclose is not
> called.

> I am sure that the reference counting in this case is absolutely
> correct with respect to managing kernel side structures.

I agree this is expected and correct from the kernel point of view.

> But I am not that it is correct with respect to hiding the explicit
> close(2) call from other threads that may be waiting on the socket. In
> other words, I am not sure if fo_close is supposed to signify that
> there are no uses of a file, or that userland close-d the file.  Or
> perhaps these should be two different methods.

It would be possible to keep track of the file descriptor number but I
think it is not worth the large amount of extra code.

Keeping track of file descriptor number would be necessary to interrupt
waits after a close or dup2 on the file descriptor that was passed to
the blocking call, even if the object remains open on a different file
descriptor number or in a different process.

Also, most people would use the new functionality incorrectly anyway. A
close() on a file descriptor another thread is using is risky since it
is in most cases impossible to prove that the other thread is in fact
blocked on the file descriptor and not preempted right before making the
system call. In the latter case, the other thread might accept a
connection from a different socket created later. A dup2() of /dev/null
onto the file descriptor would be safer.

> Additional note is that shutdown(2) doesn't wake up the thread in
> accept(2) either.  At least that's true for unix domain sockets.
> Not sure if this is a bug.

I think it is a bug. It works properly for IPv4 TCP sockets.

The resulting error is [ECONNABORTED] for a blocking socket which likely
leads to infinite looping if the thread does not know about the
shutdown(2) (because that error normally means the accept should be
retried later). For a non-blocking socket the error is [EWOULDBLOCK]
which also leads to infinite looping and is certainly wrong because
select/poll do report the socket as readable. Both of these are in
kern_accept() in sys/kern/uipc_syscalls.c.

POSIX does not say which error code we should return here but these two
are almost certainly wrong (it is usable for waking up threads stuck in
accept() if those threads check a variable after every accept() failure
and do not rely on the exact value of errno). Linux returns [EINVAL] for
both blocking and non-blocking sockets, probably from the POSIX error
condition "The socket is not accepting connections." In our man page
that error condition is formulated "listen(2) has not been called on the
socket descriptor." which is clearly not the case.

Also, I think a non-blocking accept() should immediately fail with the
head->so_error if it is set, rather than returning [EWOULDBLOCK] until
another connection arrives. Likewise, filt_solisten() in
sys/kern/uipc_socket.c only returns true if there is a connection, not
if there was an error or shutdown() has been called. On the other hand,
sopoll_generic() looks correct.

Error reporting on non-blocking accept() might usefully be postponed
until there is a connection or the socket has been shut down, to reduce
context switches.

> But the summary seems to be is that currently it is not possible to
> break a thread out of accept(2) (at least without resorting to
> signals).

Pthread cancellation works better than raw signals for this use case. In
a proper implementation such as in FreeBSD 9.0 or newer and used
properly, it allows avoiding the resource leak that may happen when
calling longjmp() or pthread_exit() in a signal handler just after
accept() has created a new socket.

-- 
Jilles Tjoelker