pf reply-to tcp connections stall

[Jeremy Chadwick]

On Tue, May 20, 2008 at 06:30:58PM +0300, Cristian Bradiceanu wrote:
> I am trying to set up split routing on two Internet links, each with
> one IP address:
> 
> em0 = wan1, $em0_gw gateway
> em1 = lan, NATed on em0 and em2
> em2 = wan2, default gateway
> 
> pass in on em0 reply-to (em0 $em0_gw) inet proto tcp from any to em0 flags S/SA keep state
> pass in on em0 reply-to (em0 $em0_gw) inet proto udp from any to em0 keep state
> pass in on em0 reply-to (em0 $em0_gw) inet proto icmp from any to em0 keep state
> 
> wan2 connections are working correct, no pf rules for policy routing
> 
> wan1 tcp connections to IP of em0 (e.g. ssh) stall when a large amount
> of data is sent (e.g. running dmesg or cat file). States are created
> correctly. When ssh stalls there are some icmp packets out on lo0 with
> source and destination ip address of em0, which I believe is not
> correct (set skip on lo0 does not help). Also tried with tcp ...
> modulate state but same result.

modulate state is known to be broken:

http://wiki.freebsd.org/JeremyChadwick/Commonly_reported_issues

Regarding the "when large amounts of data is sent, the connection
breaks" issue:

I've reproduced this a few times on our systems (using the exact same
method you do: dmesg, cat'ing large files, or scp'ing -- anything using
large TCP packets), and it's always been caused by improper pf(4) rules
where state was broken.  In every case, the "state mismatch" counter
shown in pfctl -s info would increase.

-- 
| Jeremy Chadwick                                jdc at parodius.com |
| Parodius Networking                       http://www.parodius.com/ |
| UNIX Systems Administrator                  Mountain View, CA, USA |
| Making life hard for others since 1977.              PGP: 4BD6C0CB |