performance issue within VNET jail

[Michael Grimm]

Hi —

[  I am including freebsd-pf at FreeBSD.org now and removing freebsd-jail at FreeBSD.org             ]
[  Thread starts at https://lists.freebsd.org/pipermail/freebsd-net/2017-December/049470.html  ]

Eugene Grosbein <eugen at grosbein.net> wrote:
> Michael Grimm wrote:
>> Kristof Provost <kristof at sigsegv.be> wrote:

>>> I run a very similar setup (although on CURRENT), and see no performance issues from my jails.
>> 
>> In utter despair I did upgrade one server to CURRENT (#327076) today, but that hasn't been successful :-(
>> 
>> Ok, right now I do know:
>> 
>> (#) there is *no* performance loss (TCP) when:
>> 
>> 	(-) fetching files from outside through PF/extIF to host
>> 	(-) fetching files from partner server host via IPSEC tunnel bound to extIF (ESP) to host
>> 	(-) fetching files from partner server host via IPSEC tunnel bound to extIF (ESP) to jail via bridge
>> 	(-) fetching files from partner server jail via bridge and then via IPSEC tunnel bound to extIF (ESP) to host
>> 	(-) fetching files from partner server jail via bridge and then via IPSEC tunnel bound to extIF (ESP) and then via bridge to jail
>> 
>> (#) there is a *dramatic* performance loss (TCP) when:
>> 
>> 	(-) fetching files from outside through PF/extIF via bridge to jail
>> 
>> (#) I did try to tweak the following settings *without* success:
>> 
>> 	(-) sysctl net.inet.tcp.tso=0 
>> 	(-) sysctl net.link.bridge.pfil_onlyip=0
>> 	(-) sysctl net.link.bridge.pfil_bridge=0
>> 	(-) sysctl net.link.bridge.pfil_member=0 
>> 	(-) reducing mtu to 1400 (1490 before) on all interfaces extIF, bridge, epairXs
>> 	(-) deactivating "scrub in all" and "scrub out on $extIF all random-id" in /etc/pf.conf
>> 	(-) setting "set require-order yes" and "set require-order no" in /etc/pf.conf [1]
>> 
>> [1] I do see more a lot of out-of-order packages within a jail "netstat -s -p tcp" after those slow downloads, but not after downloads via IPSEC tunnel from partner host.
>> 
>> That leads me to the conclusions:
>> 
>> 	(#) the bridge is not to blame
>> 	(#) it's either the PF/NATing or something else, right?
>> 
>> Thanks for your suggestions so far, but I am lost here. Any ideas?
> 
> It seems to me some kind of bug in the PF.
> I personally never tried it, I use ipfw and it works just fine.

Before testing IPFW (which I have never used before) I'd like to ask the experts in freebsd-pf at FreeBSD.org about possible tests/tweaks regarding PF.

Thanks to all involved so far and regards,
Michael