setfib and RSTs

[Nikolay Denev]

On Tue, Dec 30, 2014 at 12:51 AM, Bjoern A. Zeeb <
bzeeb-lists at lists.zabbadoz.net> wrote:

>
> > On 29 Dec 2014, at 21:03 , Nikolay Denev <ndenev at gmail.com> wrote:
> >
> > No, no PR yet, but I will file one. I wanted to collect some more data
> > first.
> >
> > So, I've did some dtrace digging :
> >
> > [20:54][root at nas:~]#cat reset.d
> > #!/usr/sbin/dtrace -s
> >
> > fbt:kernel:tcp_dropwithreset:entry
> > {
> >    printf("reason %d fib %d src_port %d dst_port %d", args[4], args[2] ?
> > args[2]->t_inpcb->inp_inc.inc_fibnum : -1, ntohs(args[1]->th_sport),
> > ntohs(args[1]->th_dport));
> > /* stack(); */
> > }
> > …
>
> > The port numbers here match RST packets that I'm seeing with tcpdump in
> > another window.
> > reason 3 is BANDLIM_RST_CLOSEDPORT (from icmp_var.h)
> > Looking at tcp_input.c I see that there are cases where the INPCB does
> not
> > exists, and from what I see this is how the FIB gets determined.
> > Also here I see that tcp_dropwithreset() is called with tcpcb set to
> NULL,
> > so probably this is why the FIB is not found.
> >
> > Why this is happening, I have no idea yet.
>
> Could you also check for the mbuf *m and the fibnum from the pkthdr there?
>
> It might be even more interesting to see this for tcp_respond and the
> following ip_output as well, in case you want to keep state in the d
> script;  otherwise just tcp_dropwithreset and/or tcp_respond should be fine.
>
> Usually I would expect for the tcp_dropwithreset case that inp will be
> NULL in tcp_respond, the mbuf *m and th will be valid and thus the FIB
> number from the incoming mbuf would be re-used as the mbuf will be re-used,
> but for that the mbuf needs to be properly tagged on receive.
>
> /bz
>
> —
> Bjoern A. Zeeb                                  Charles Haddon Spurgeon:
> "Friendship is one of the sweetest joys of life.  Many might have failed
>  beneath the bitterness of their trial  had they not found a friend."
>
>
If I got this right i see FIB 0 here:

fbt:kernel:tcp_dropwithreset:entry
{
    this->mbuf = (struct mbuf *)args[0];
    printf("reason %d fib %d src_port %d dst_port %d", args[4],
this->mbuf->M_dat.MH.MH_pkthdr.fibnum, ntohs(args[1]->th_sport),
ntohs(args[1]->th_dport));
}


I've did some tracing of the the transmission-daemon, and I saw that after
it calls connect() on the sockets (which are non-blocking), the connect
returns with EINPROGRESS (as expected),
but in most cases after about 20-60us close() is called. (not sure why it's
doing this, but this is what I saw)
So, by that time the SYN-ACK from the remote end is received the socket is
already gone and RST is sent using default FIB, which does seem to be the
correct behaviour.
I've now set the fib for the tun interface with "ifconfig tun2 fib 1", and
I see the RSTs sent back over the tunnel.

--Nikolay