amd64/157785: amd64 + jail + ipfw + natd = very slow outbound traffic from jail (5KB/s)

[Ben Baron]

>Number:         157785
>Category:       amd64
>Synopsis:       amd64 + jail + ipfw + natd = very slow outbound traffic from jail (5KB/s)
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    freebsd-amd64
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Sun Jun 12 04:50:00 UTC 2011
>Closed-Date:
>Last-Modified:
>Originator:     Ben Baron
>Release:        8.2-RELEASE
>Organization:
Be Five Enterprises, Inc.
>Environment:
FreeBSD localhost 8.2-RELEASE FreeBSD 8.2-RELEASE #0: Thu Feb 17 02:41:51 UTC 2011     root at mason.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC  amd64
>Description:
I have a newly set up FreeBSD 8.2 server with one public network interface setup with a jail for housing Apache. I was using ipfw + natd as my solution to firewall and nat traffic between the machine and jail (actually had multiple jails for different processes). The setup was working fine, however transfers out from my Apache jail were limited to about 5 KB/sec (~40kbit/sec) (I didn't test other jails but I assume they were the same speed, also didn't test jail to jail). Traffic into the jail from the internet was full speed. Traffic out from the host was also full speed. Only traffic out from the jail was affected.

I researched the issue online and determined that this is a known issue with ipfw + natd and jails when using amd64 processors (in my case an Intel Core i7). I read conflicting statements on the FreeBSD forums about whether this also affects i386 (it looks like it doesn't, but one person did comment that they had the same issue on an i386 machine IIRC).
>How-To-Repeat:

1. Do a fresh install of FreeBSD 8.2

2. Create an lo1 interface with the IP 10.0.0.2/24

3. Use ezjail to create a jail with 10.0.0.2 IP address

4. Install Apache in jail

5. Setup ipfw with the following settings (edited from my original settings to remove some sensitive information):

# Interface: em1 
# List of Ports:
#   Host SSH: 22
#   JAIL WEB SERVER: 80, 443 

IPF="ipfw -q add"
ipfw -q -f flush

# Replace with your interface
PIF="em1"

# Host IP
inet="XXX.XXX.XXX.XXX"

# Jail IPs 
apache_jail="10.0.0.2"

# Loopback
$IPF 100 allow all from any to any via lo0 
$IPF 110 deny all from any to 127.0.0.0/8
$IPF 120 deny all from 127.0.0.0/8 to any 
$IPF 130 deny tcp from any to any frag

# Allow SSH to host
$IPF 300 allow tcp from any to me 22 established

# Allow out NATed traffic from Apache Jail. This is DNATed traffic.
$IPF 500 divert natd ip from $apache_jail to not me out via $PIF
$IPF 570 skipto 10000 ip from any to any diverted

# Allow remaining all outgoing traffic
$IPF 600 allow all from me to any out 

# Allow NATed traffic to Jail. This is SNATed traffic. natd (NAT daemon) will take care of these.
$IPF 700 divert natd ip from not me to any in via $PIF
$IPF 710 skipto 10000 ip from any to any diverted

# Deny ICMP
$IPF 800 allow icmp from any to any 

# Log an deny the traffic
$IPF 900 deny log logamount 10000 ip from any to any

# This is where all diverted NAT traffic are skipped to after being processed.
$IPF 10000 allow ip from any to any


6. Setup natd with the following settings:

# Apache Jail
redirect_port tcp 10.0.0.2:80 80
redirect_port tcp 10.0.0.2:443 443 

>Fix:
The only fix I was able to find was to switch to pf. After switching to pf for my firewall and nat solution, I'm getting full speed traffic both ways from the jail.

>Release-Note:
>Audit-Trail:
>Unformatted: