performance issue within VNET jail

[Michael Grimm]

Kristof Provost <kristof at sigsegv.be> wrote:

> I run a very similar setup (although on CURRENT), and see no performance issues from my jails.

In utter despair I did upgrade one server to CURRENT (#327076) today, but that hasn't been successful :-(

Ok, right now I do know:

(#) there is *no* performance loss (TCP) when:

	(-) fetching files from outside through PF/extIF to host
	(-) fetching files from partner server host via IPSEC tunnel bound to extIF (ESP) to host
	(-) fetching files from partner server host via IPSEC tunnel bound to extIF (ESP) to jail via bridge
	(-) fetching files from partner server jail via bridge and then via IPSEC tunnel bound to extIF (ESP) to host
	(-) fetching files from partner server jail via bridge and then via IPSEC tunnel bound to extIF (ESP) and then via bridge to jail

(#) there is a *dramatic* performance loss (TCP) when:

	(-) fetching files from outside through PF/extIF via bridge to jail

(#) I did try to tweak the following settings *without* success:

	(-) sysctl net.inet.tcp.tso=0 
	(-) sysctl net.link.bridge.pfil_onlyip=0
	(-) sysctl net.link.bridge.pfil_bridge=0
	(-) sysctl net.link.bridge.pfil_member=0 
	(-) reducing mtu to 1400 (1490 before) on all interfaces extIF, bridge, epairXs
	(-) deactivating "scrub in all" and "scrub out on $extIF all random-id" in /etc/pf.conf
	(-) setting "set require-order yes" and "set require-order no" in /etc/pf.conf [1]

[1] I do see more a lot of out-of-order packages within a jail "netstat -s -p tcp" after those slow downloads, but not after downloads via IPSEC tunnel from partner host.

That leads me to the conclusions:

	(#) the bridge is not to blame
	(#) it's either the PF/NATing or something else, right?

Thanks for your suggestions so far, but I am lost here. Any ideas?

Regards,
Michael