Inconsistent TCP state in tcp_do_segment()

[Jacques Fourie]

Hi,

I've been using the kernel socket API and noticed that every once in a
while the data received at the remote end of a TCP connection doesn't match
the data sent locally using sosend(). I tracked it down to the piece of
code in tcp_do_segment() starting at line 2665: (svn rev 245286)

               if (acked > so->so_snd.sb_cc) {
                        tp->snd_wnd -= so->so_snd.sb_cc;
                        sbdrop_locked(&so->so_snd, (int)so->so_snd.sb_cc);
                        ourfinisacked = 1;
                } else {
                        sbdrop_locked(&so->so_snd, acked);
                        tp->snd_wnd -= acked;
                        ourfinisacked = 0;
                }
                sowwakeup_locked(so);
                /* Detect una wraparound. */
                if (!IN_RECOVERY(tp->t_flags) &&
                    SEQ_GT(tp->snd_una, tp->snd_recover) &&
                    SEQ_LEQ(th->th_ack, tp->snd_recover))
                        tp->snd_recover = th->th_ack - 1;
                /* XXXLAS: Can this be moved up into cc_post_recovery? */
                if (IN_RECOVERY(tp->t_flags) &&
                    SEQ_GEQ(th->th_ack, tp->snd_recover)) {
                        EXIT_RECOVERY(tp->t_flags);
                }
                tp->snd_una = th->th_ack;
                if (tp->t_flags & TF_SACK_PERMIT) {
                        if (SEQ_GT(tp->snd_una, tp->snd_recover))
                                tp->snd_recover = tp->snd_una;
                }
                if (SEQ_LT(tp->snd_nxt, tp->snd_una))
                        tp->snd_nxt = tp->snd_una;

The issue is that sowwakeup_locked() is called before all the tcpcb fields
are updated to account for the current ACK processing as can be seen from
the tp->snd_una = th->th_ack in line 2686. My code that uses the kernel
socket API calls sosend() in the upcall path resulting from
sowwakeup_locked() which in turn can lead to tcp_output() being called with
inconsistent TCP state. If I move the call to sowwakeup_locked() to after
the 'if (SEQ_LT(tp->snd_nxt, tp->snd_una))' line in the code snippet above
the data corruption issue seems to be fixed.