_ftello() modification requires additional capsicum rights, breaking tcpdump and dhclient

[Mateusz Guzik]

On Fri, Sep 12, 2014 at 09:29:56PM -0700, Peter Wemm wrote:
> On Thursday, September 11, 2014 12:38:02 PM Patrick Kelsey wrote:
> > On Wed, Sep 10, 2014 at 3:00 AM, Andrey Chernov <ache at freebsd.org> wrote:
> > > On 09.09.2014 21:53, Patrick Kelsey wrote:
> > > > I don't think it is worth the trouble, as given the larger pattern of
> > > > libc routines requiring multiple capsicum rights, it seems one will in
> > > > general have to have libc implementation knowledge when using it in
> > > > concert with capsicum.  For example, consider the limitfd() routine in
> > > > kdump.c, which provides rights for the TIOCGETA ioctl to be used on
> > > > stdout so the eventual call to isatty() via printf() will work as
> > > 
> > > intended.
> > > 
> > > > I think the above kdump example is a good one for the subtle issues that
> > > > can arise when using capsicum with libc.  That call to isatty() is via a
> > > > widely-used internal libc routine __smakebuf().  __smakebuf() also calls
> > > > __swhatbuf(), which in turn calls _fstat(), all to make sure that output
> > > > to a tty is line buffered by default.  It would appear that programs
> > > > that restrict rights on stdout without allowing CAP_IOCTL and CAP_FSTAT
> > > > could be disabling the normally default line buffering when stdout is a
> > > > tty.  kdump goes the distance, but dhclient does not (restricting stdout
> > > > to CAP_WRITE only).
> > > > 
> > > > In any event, the patch attached to my first message is seeming like the
> > > > way to go.
> > > 
> > > Well, then commit it (if capsicum team agrees).
> > 
> > Will do - thanks for the feedback.
> > 
> > -Patrick
> 
> Is there any possibility that this is related to the problem we've recently 
> hit in the freebsd.org cluster with this month's refresh?
> 
> After running for a while:
> Sep 10 02:39:44 ns2 unbound: [65258:0] notice: init module 0: validator
> Sep 10 02:39:44 ns2 unbound: [65258:0] notice: init module 1: iterator
> Sep 10 11:44:29 ns2 unbound: [65258:3] fatal error: event_dispatch returned 
> error -1, errno is Capabilities insufficient
> 
> Sep 10 16:21:16 ns2 unbound: [28212:0] warning: did not exit gracefully last 
> time (65258)
> Sep 10 16:21:16 ns2 unbound: [28213:0] notice: init module 0: validator
> Sep 10 16:21:16 ns2 unbound: [28213:0] notice: init module 1: iterator
> Sep 11 10:23:49 ns2 unbound: [28213:5] fatal error: event_dispatch returned 
> error -1, errno is Capabilities insufficient
> 
> Sep 11 13:48:46 ns2 unbound: [79419:0] warning: did not exit gracefully last 
> time (28213)
> Sep 11 13:48:46 ns2 unbound: [79420:0] notice: init module 0: validator
> Sep 11 13:48:46 ns2 unbound: [79420:0] notice: init module 1: iterator
> Sep 11 18:42:56 ns2 unbound: [79420:6] fatal error: event_dispatch returned 
> error -1, errno is Capabilities insufficient
> 
> I believe this jail was started from the boot process. If I restart the jail 
> by hand from a ssh session the problem goes away.
> 
> This is unbound from ports and I don't have any more details than this.  This 
> is new this month.
> 

Is this thingy multithreaded?

Currently there is a race in the kernel where fd is visible before
relevant capabilities are installed. This can result in an error like
this one for weird processes.

I got a patch for this:
http://lists.freebsd.org/pipermail/freebsd-arch/2014-August/015788.html

but it got stalled on 'memory barrier' discussion. I'll try to ping
people to move it forward.

IIRC there was a report of unbound failing this way, apparently fixed
with aforementioned patch.

-- 
Mateusz Guzik <mjguzik gmail.com>