Issue with ipfw nat and 'Write failed: Permission denied' over SSH

[Luke Marsden]

Hi freebsd-ipfw,

I hope you can help me with an urgent issue relating to pushing to Git
over SSH from inside a FreeBSD 8.2 jail with IPFW NAT to the outside
world.

>From inside the jail, the push manages to send a bunch of data over the
connection, before erroring out with:

Running: git push --force git at github.com:XXX
Counting objects: 646, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (612/612), done.
Write failed: Permission denied 5.59 MiB | 187 KiB/s # <---
fatal: The remote end hung up unexpectedly
fatal: sha1 file '<stdout>' write error: Broken pipe
error: failed to push some refs to 'git at github.com:XXX'

Note that the 'Write failed: Permission denied' is printed during
pushing of the refs over the SSH connection - you can see it overwriting
the data transfer rate which Git prints in interactive mode.

Outside the jail, the same push works fine (of the same repo, straight
from the jail's filesystem):

Counting objects: 69854, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (22060/22060), done.
Writing objects: 100% (69854/69854), 294.04 MiB | 354 KiB/s, done.
Total 69854 (delta 47117), reused 69846 (delta 47113)
To git at github.com:XXX
 + 5d6c172...f143f83 replication-stable -> replication-stable (forced
update)

This is a reproducible failure and furthermore the "Write failed:
Permission denied" feels like it's coming straight out of the local
kernel, rather than being a transient issue with GitHub or the like.

The complete IPFW ruleset for the host is:

$ sudo ipfw list
60000 allow tcp from me to me dst-port 81 uid root keep-state
60001 deny tcp from any to me dst-port 81
60002 allow tcp from me to me dst-port 3307 uid root keep-state
60003 deny tcp from any to me dst-port 3307
60004 allow tcp from me to me dst-port 2121 uid root keep-state
60005 deny tcp from any to me dst-port 2121
60006 allow tcp from me to me dst-port 26 uid root keep-state
60007 deny tcp from any to me dst-port 26
60008 allow tcp from me to me dst-port 6969 keep-state
60010 deny tcp from any to me dst-port 6969
60011 allow tcp from me to any dst-port 25 uid mailnull keep-state
60012 allow tcp from me to any dst-port 25 uid root keep-state
60013 allow tcp from any to me dst-port 25 keep-state
60014 unreach filter-prohib log tcp from any to any dst-port 25
60015 allow tcp from me to any dst-port 587 uid mailnull keep-state
60016 allow tcp from me to any dst-port 587 uid root keep-state
60017 allow tcp from any to me dst-port 587 keep-state
60018 unreach filter-prohib log tcp from any to any dst-port 587
60020 nat 200 ip from 169.172.0.0/16 to any out xmit em0
60021 nat 200 ip from any to any in recv em0
60022 allow ip from any to any
65535 deny ip from any to any

The jail is configured with NAT on lo1; outside the jail (yes, we are
using a stupid net range which is actually not private):

lo1: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=3<RXCSUM,TXCSUM>
        inet 169.172.255.254 netmask 0xffff0000 
        inet 169.172.0.1 netmask 0xffff0000 
        inet 169.172.0.2 netmask 0xffff0000 
        inet 169.172.0.3 netmask 0xffff0000 
        inet 169.172.0.4 netmask 0xffff0000 
        [... other "local" IPs for other jails...]

>From inside the jail this looks like:

lo1: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=3<RXCSUM,TXCSUM>
        inet 169.172.0.4 netmask 0xffff0000 

Can anyone shed any light on why this configuration seems to work most
of the time but when pushing large refs over SSH to GitHub it fails with
the obscure 'Write failed: Permission denied'?  Is there any way to dig
into what caused this error, or a debugging mode I can enable for ipfw?

Thanks!
Luke Marsden

-- 
CEO, Hybrid Logic
+447791750420  |  +1-415-449-1165  |  www.hybrid-cluster.com