Re: Request for Testing: TCP RACK

No.  The goal is to run on every return to userspace for every thread.

Drew

On Mon, Mar 18, 2024, at 3:41 PM, Konstantin Belousov wrote:
> On Mon, Mar 18, 2024 at 03:13:11PM -0400, Drew Gallatin wrote:
> > I got the idea from
> > https://people.mpi-sws.org/~druschel/publications/soft-timers-tocs.pdf
> > The gist is that the TCP pacing stuff needs to run frequently, and
> > rather than run it out of a clock interrupt, its more efficient to run
> > it out of a system call context at just the point where we return to
> > userspace and the cache is trashed anyway. The current implementation
> > is fine for our workload, but probably not idea for a generic system.
> > Especially one where something is banging on system calls.
> >
> > Ast's could be the right tool for this, but I'm super unfamiliar with
> > them, and I can't find any docs on them.
> >
> > Would ast_register(0, ASTR_UNCOND, 0, func) be roughly equivalent to
> > what's happening here?
> This call would need some AST number added, and then it registers the
> ast to run on next return to userspace, for the current thread.
> 
> Is it enough?
> >
> > Drew
> 
> > 
> > On Mon, Mar 18, 2024, at 2:33 PM, Konstantin Belousov wrote:
> > > On Mon, Mar 18, 2024 at 07:26:10AM -0500, Mike Karels wrote:
> > > > On 18 Mar 2024, at 7:04, tuexen@freebsd.org wrote:
> > > > 
> > > > >> On 18. Mar 2024, at 12:42, Nuno Teixeira <eduardo@freebsd.org> wrote:
> > > > >>
> > > > >> Hello all!
> > > > >>
> > > > >> It works just fine!
> > > > >> System performance is OK.
> > > > >> Using patch on main-n268841-b0aaf8beb126(-dirty).
> > > > >>
> > > > >> ---
> > > > >> net.inet.tcp.functions_available:
> > > > >> Stack                           D Alias                            PCB count
> > > > >> freebsd                           freebsd                          0
> > > > >> rack                            * rack                             38
> > > > >> ---
> > > > >>
> > > > >> It would be so nice that we can have a sysctl tunnable for this patch
> > > > >> so we could do more tests without recompiling kernel.
> > > > > Thanks for testing!
> > > > >
> > > > > @gallatin: can you come up with a patch that is acceptable for Netflix
> > > > > and allows to mitigate the performance regression.
> > > > 
> > > > Ideally, tcphpts could enable this automatically when it starts to be
> > > > used (enough?), but a sysctl could select auto/on/off.
> > > There is already a well-known mechanism to request execution of the
> > > specific function on return to userspace, namely AST.  The difference
> > > with the current hack is that the execution is requested for one callback
> > > in the context of the specific thread.
> > > 
> > > Still, it might be worth a try to use it; what is the reason to hit a thread
> > > that does not do networking, with TCP processing?
> > > 
> > > > 
> > > > Mike
> > > > 
> > > > > Best regards
> > > > > Michael
> > > > >>
> > > > >> Thanks all!
> > > > >> Really happy here :)
> > > > >>
> > > > >> Cheers,
> > > > >>
> > > > >> Nuno Teixeira <eduardo@freebsd.org> escreveu (domingo, 17/03/2024 à(s) 20:26):
> > > > >>>
> > > > >>> Hello,
> > > > >>>
> > > > >>>> I don't have the full context, but it seems like the complaint is a performance regression in bonnie++ and perhaps other things when tcp_hpts is loaded, even when it is not used.  Is that correct?
> > > > >>>>
> > > > >>>> If so, I suspect its because we drive the tcp_hpts_softclock() routine from userret(), in order to avoid tons of timer interrupts and context switches.  To test this theory,  you could apply a patch like:
> > > > >>>
> > > > >>> It's affecting overall system performance, bonnie was just a way to
> > > > >>> get some numbers to compare.
> > > > >>>
> > > > >>> Tomorrow I will test patch.
> > > > >>>
> > > > >>> Thanks!
> > > > >>>
> > > > >>> --
> > > > >>> Nuno Teixeira
> > > > >>> FreeBSD Committer (ports)
> > > > >>
> > > > >>
> > > > >>
> > > > >> -- 
> > > > >> Nuno Teixeira
> > > > >> FreeBSD Committer (ports)
> > > > 
> > > 
>