From nobody Mon Mar 14 15:38:16 2022 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 1D7151A14CED for ; Mon, 14 Mar 2022 15:38:19 +0000 (UTC) (envelope-from joh.hendriks@gmail.com) Received: from mail-ed1-x52c.google.com (mail-ed1-x52c.google.com [IPv6:2a00:1450:4864:20::52c]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4KHLKQ3klnz3sws; Mon, 14 Mar 2022 15:38:18 +0000 (UTC) (envelope-from joh.hendriks@gmail.com) Received: by mail-ed1-x52c.google.com with SMTP id y22so20527015eds.2; Mon, 14 Mar 2022 08:38:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:date:mime-version:user-agent:subject:content-language:to :cc:references:from:in-reply-to; bh=YgRp7v+Zbugl89j19+iekQzEilAz8Q2+SHCs/6oGl00=; b=hl+HBYkfEJOcdTlcRtkSE0QKNqVkFgxXTQG+0d2cyn01EU3heTKphIYDi5WbjvKRvb 8Pm+B7VjUnYGsySisfgG6iQVctLwP+aS0SR88nOU5IBSAR3i5MdfM5KOHF8lVxaZx6RF Oce1hveKD7qtl/CGvf2qLxdKhd8kfThPeXny1bUfdkzDXRDI0NeKLHN3sAa7xyB+nRD8 rCrRHaz0eKqV310J2Dx4mluFv8Azan5ObTq2LqFlJTFwFVQL/XliHzkObkyna82WoMR2 pmzEYLAtNJfOB1j0AEYWgJSzMgGOEntbTdqOBI1a8GxsP071yD0aRCPgFQR6jwmOjaaZ HWKw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:date:mime-version:user-agent:subject :content-language:to:cc:references:from:in-reply-to; bh=YgRp7v+Zbugl89j19+iekQzEilAz8Q2+SHCs/6oGl00=; b=wvu17BXobwQlvQNy1yKSd2fdFjH9HMajiyYz/CQdCOYytn14CJxLVnyMmZ27FQOVS+ GJuWudYWYNQfZOVs/BTEDwZnBPHhLZbe29uph5lMGLd9qNKlG/X3Q57GbflXG/YGHIzy qCCnXEY25svuNcjbHSAfnBfUK7vtY5nMSHkOq9EHs5XCXYbG9TL54ln8X1UTUWCRKZze 5XoR6lgDO4SD6FrN10EOPBpmsVZr7n3+EOFfO3/JUizcxCmhQGm5qFoVbs1Gy9lsW541 3Q16IF9wyvA1wTIRebM7SU/85H8f9Fq8TfKEYSqDoXe+jx4KqJvOyAx1er79ay1jDhcj ghCg== X-Gm-Message-State: AOAM533ifm/3cBKhPyp1cvEU/PjeCPdEeafSIsZZhOW40sTWIHKN47/7 mFTIAqeliShH2Jd33wgyO5v/qW7qBhpYxA== X-Google-Smtp-Source: ABdhPJwlwOAhTpmHD1RJHTxSn3KnFc0ofvb+7P6++YoxuGSA9sBxMwAe5g7TvPO1NFnqrho8XFcHdQ== X-Received: by 2002:a05:6402:280b:b0:416:7a43:b30e with SMTP id h11-20020a056402280b00b004167a43b30emr21413050ede.371.1647272297216; Mon, 14 Mar 2022 08:38:17 -0700 (PDT) Received: from [192.168.1.25] (85-147-130-226.cable.dynamic.v4.ziggo.nl. [85.147.130.226]) by smtp.gmail.com with ESMTPSA id n27-20020a1709062bdb00b006da975173bfsm7088041ejg.170.2022.03.14.08.38.16 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 14 Mar 2022 08:38:16 -0700 (PDT) Content-Type: multipart/alternative; boundary="------------S0ZHVqMT6lBFx5c85G4rrsWY" Message-ID: Date: Mon, 14 Mar 2022 16:38:16 +0100 List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0) Gecko/20100101 Thunderbird/91.7.0 Subject: Re: epair and vnet jail loose connection. Content-Language: en-US To: Kristof Provost , Michael Gmelin Cc: "Bjoern A. Zeeb" , "Patrick M. Hausen" , freeBSD-net References: <797A280E-5DF2-4276-BB72-E4E1053A19FA@lists.zabbadoz.net> <6086BA6D-3D54-4851-B636-3B32FACB35E9@freebsd.org> <3B5E2D6F-5444-4448-B7C3-704E294368C3@lists.zabbadoz.net> <20220314144451.35f803a9.grembo@freebsd.org> From: Johan Hendriks In-Reply-To: X-Rspamd-Queue-Id: 4KHLKQ3klnz3sws X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b=hl+HBYkf; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of johhendriks@gmail.com designates 2a00:1450:4864:20::52c as permitted sender) smtp.mailfrom=johhendriks@gmail.com X-Spamd-Result: default: False [-2.20 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36:c]; FREEMAIL_FROM(0.00)[gmail.com]; RCPT_COUNT_FIVE(0.00)[5]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; MID_RHS_MATCH_FROM(0.00)[]; TAGGED_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.999]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; FROM_HAS_DN(0.00)[]; NEURAL_SPAM_SHORT(0.79)[0.790]; NEURAL_HAM_LONG(-0.99)[-0.995]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::52c:from]; MLMMJ_DEST(0.00)[freebsd-net]; RCVD_TLS_ALL(0.00)[] X-ThisMailContainsUnwantedMimeParts: N This is a multi-part message in MIME format. --------------S0ZHVqMT6lBFx5c85G4rrsWY Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit On 14/03/2022 16:09, Kristof Provost wrote: > > On 14 Mar 2022, at 7:44, Michael Gmelin wrote: > > On Sun, 13 Mar 2022 17:53:44 +0000 > "Bjoern A. Zeeb" wrote: > > On 13 Mar 2022, at 17:45, Michael Gmelin wrote: > > On 13. Mar 2022, at 18:16, Bjoern A. Zeeb > wrote: > > On 13 Mar 2022, at 16:33, Michael Gmelin wrote: > > It's important to point out that this only happens > with > kern.ncpu>1. With kern.ncpu==1 nothing gets stuck. > > This perfectly fits into the picture, since, as > pointed out by > Johan, > the first commit that is affected[0] is about > multicore support. > > Ignore my ignorance, what is the default of > net.isr.maxthreads and > net.isr.bindthreads (in stable/13) these days? > > My tests were on CURRENT and I’m afk, but according to > cgit[0][1], > max is 1 and bind is 0. > > Would it make sense to repeat the test with max=-1? > > I’d say yes, I’d also bind, but that’s just me. > > I would almost assume Kristof running with -1 by default (but > he can > chime in on that). > > I tried various configuration permutations, all with ncpu=2: > > - 14.0-CURRENT #0 main-n253697-f1d450ddee6 > - 13.1-BETA1 #0 releng/13.1-n249974-ad329796bdb > - net.isr.maxthreads: -1 (which results in 2 threads), 1, 2 > - net.isr.bindthreads: -1, 0, 1, 2 > - net.isr.dispatch: direct, deferred > > All resulting in the same behavior (hang after a few seconds). > They all > work ok when running on a single core instance (threads=1 in this > case). > > I also ran the same test on 13.0-RELEASE-p7 for > comparison (unsurprisingly, it's ok). > > I placed the script to reproduce the issue on freefall for your > convenience, so running it is as simple as: > > fetch https://people.freebsd.org/~grembo/hang_epair.sh > # inspect content > sh hang_epair.sh > > or, if you feel lucky > > fetch -o - https://people.freebsd.org/~grembo/hang_epair.sh | sh > > > With that script I can also reproduce the problem. > > I’ve experimented with this hack: > > |diff --git a/sys/net/if_epair.c b/sys/net/if_epair.c index > c39434b31b9f..1e6bb07ccc4e 100644 --- a/sys/net/if_epair.c +++ > b/sys/net/if_epair.c @@ -415,7 +415,10 @@ epair_ioctl(struct ifnet > *ifp, u_long cmd, caddr_t data) case SIOCSIFMEDIA: case SIOCGIFMEDIA: > + printf("KP: %s() SIOCGIFMEDIA\n", __func__); sc = ifp->if_softc; + > taskqueue_enqueue(epair_tasks.tq[0], &sc->queues[0].tx_task); + error > = ifmedia_ioctl(ifp, ifr, &sc->media, cmd); break; | > > That kicks the receive code whenever I |ifconfig epair0a|, and I see a > little more traffic every time I do so. > That suggests pretty strongly that there’s an issue with how we > dispatch work to the handler thread. So presumably there’s a race > between epair_menq() and epair_tx_start_deferred(). > > epair_menq() tries to only enqueue the receive work if there’s nothing > in the buf_ring, on the grounds that if there is the previous packet > scheduled the work. Clearly there’s an issue there. > > I’ll try to dig into that in the next few days. > > Kristof > I am willing to try patches, so if you have them i can try them on 14-HEAD, 13-STABLEand 13.1-PRERELEASE. --------------S0ZHVqMT6lBFx5c85G4rrsWY Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit


On 14/03/2022 16:09, Kristof Provost wrote:

On 14 Mar 2022, at 7:44, Michael Gmelin wrote:

On Sun, 13 Mar 2022 17:53:44 +0000
"Bjoern A. Zeeb" <bzeeb-lists@lists.zabbadoz.net> wrote:

On 13 Mar 2022, at 17:45, Michael Gmelin wrote:

On 13. Mar 2022, at 18:16, Bjoern A. Zeeb
<bzeeb-lists@lists.zabbadoz.net> wrote:

On 13 Mar 2022, at 16:33, Michael Gmelin wrote:

It's important to point out that this only happens with
kern.ncpu>1. With kern.ncpu==1 nothing gets stuck.

This perfectly fits into the picture, since, as pointed out by
Johan,
the first commit that is affected[0] is about multicore support.

Ignore my ignorance, what is the default of net.isr.maxthreads and
net.isr.bindthreads (in stable/13) these days?

My tests were on CURRENT and I’m afk, but according to cgit[0][1],
max is 1 and bind is 0.

Would it make sense to repeat the test with max=-1?

I’d say yes, I’d also bind, but that’s just me.

I would almost assume Kristof running with -1 by default (but he can
chime in on that).

I tried various configuration permutations, all with ncpu=2:

- 14.0-CURRENT #0 main-n253697-f1d450ddee6
- 13.1-BETA1 #0 releng/13.1-n249974-ad329796bdb
- net.isr.maxthreads: -1 (which results in 2 threads), 1, 2
- net.isr.bindthreads: -1, 0, 1, 2
- net.isr.dispatch: direct, deferred

All resulting in the same behavior (hang after a few seconds). They all
work ok when running on a single core instance (threads=1 in this case).

I also ran the same test on 13.0-RELEASE-p7 for
comparison (unsurprisingly, it's ok).

I placed the script to reproduce the issue on freefall for your
convenience, so running it is as simple as:

fetch https://people.freebsd.org/~grembo/hang_epair.sh
# inspect content
sh hang_epair.sh

or, if you feel lucky

fetch -o - https://people.freebsd.org/~grembo/hang_epair.sh | sh


With that script I can also reproduce the problem.

I’ve experimented with this hack:

diff --git a/sys/net/if_epair.c b/sys/net/if_epair.c
index c39434b31b9f..1e6bb07ccc4e 100644
--- a/sys/net/if_epair.c
+++ b/sys/net/if_epair.c
@@ -415,7 +415,10 @@ epair_ioctl(struct ifnet *ifp, u_long cmd, caddr_t data)

        case SIOCSIFMEDIA:
        case SIOCGIFMEDIA:
+               printf("KP: %s() SIOCGIFMEDIA\n", __func__);
                sc = ifp->if_softc;
+               taskqueue_enqueue(epair_tasks.tq[0], &sc->queues[0].tx_task);
+
                error = ifmedia_ioctl(ifp, ifr, &sc->media, cmd);
                break;

That kicks the receive code whenever I ifconfig epair0a, and I see a little more traffic every time I do so.
That suggests pretty strongly that there’s an issue with how we dispatch work to the handler thread. So presumably there’s a race between epair_menq() and epair_tx_start_deferred().

epair_menq() tries to only enqueue the receive work if there’s nothing in the buf_ring, on the grounds that if there is the previous packet scheduled the work. Clearly there’s an issue there.

I’ll try to dig into that in the next few days.

Kristof

I am willing to try patches, so if you have them i can try them on 14-HEAD, 13-STABLEand 13.1-PRERELEASE.


--------------S0ZHVqMT6lBFx5c85G4rrsWY--