FreeBSD Router Problem

Thu Feb 26 23:54:27 PST 2009

Hi,

On Fri, Feb 27, 2009 at 8:41 AM, Shawn Everett <shawn at tandac.com> wrote:

> >  Any error messages in dmesg output ?
> >  Significant changes in "netstat -m" output before and after ?
> >  The same for "pfctl -s all" output...
>
> The box has been up for about 12 hours now.  As a point of discussion here
> is the output from netstat and pfctl in case anything obvious jumps out.
>
> 385/905/1290 mbufs in use (current/cache/total)
> 384/484/868/25600 mbuf clusters in use (current/cache/total/max)
> 256/384 mbuf+clusters out of packet secondary zone in use (current/cache)
> 0/44/44/12800 4k (page size) jumbo clusters in use
> (current/cache/total/max)
> 0/0/0/6400 9k jumbo clusters in use (current/cache/total/max)
> 0/0/0/3200 16k jumbo clusters in use (current/cache/total/max)
> 864K/1370K/2234K bytes allocated to network (current/cache/total)
> 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
> 0/0/0 requests for jumbo clusters denied (4k/9k/16k)
> 0/5/6656 sfbufs in use (current/peak/max)
> 0 requests for sfbufs denied
> 0 requests for sfbufs delayed
> 0 requests for I/O initiated by sendfile
> 0 calls to protocol drain routines
>

This looks OK to me...

>
>
> # pfctl -s all
> No ALTQ support in kernel
> ALTQ related functions disabled
> TRANSLATION RULES:
> nat on ste0 inet from 172.16.3.0/24 to any -> (ste0) round-robin
> nat on ste1 inet from 172.16.3.0/24 to any -> (ste1) round-robin
>
> FILTER RULES:
> pass out on em0 inet from any to 172.16.3.0/24 flags S/SA keep state
> pass in quick on em0 inet from 172.16.3.0/24 to 172.16.3.253 flags S/SA
> keep state
> pass in on em0 route-to { (ste0 204.244.159.254), (ste1 204.244.159.254) }
> round-robin inet proto tcp from 172.16.3.0/24 to any flags S/SA modulate
> state
> pass in on em0 route-to { (ste0 204.244.159.254), (ste1 204.244.159.254) }
> round-robin inet proto udp from 172.16.3.0/24 to any keep state
> pass in on em0 route-to { (ste0 204.244.159.254), (ste1 204.244.159.254) }
> round-robin inet proto icmp from 172.16.3.0/24 to any keep state
> pass out on ste0 proto tcp all flags S/SA modulate state
> pass out on ste0 proto udp all keep state
> pass out on ste0 proto icmp all keep state
> pass out on ste1 proto tcp all flags S/SA modulate state
> pass out on ste1 proto udp all keep state
> pass out on ste1 proto icmp all keep state
> pass out on ste0 route-to (ste1 204.244.159.254) inet from 204.244.159.55
> to any flags S/SA keep state
> pass out on ste1 route-to (ste0 204.244.159.254) inet from 204.244.159.68
> to any flags S/SA keep state
>

Quite an evolved route forwarding setup ;)...

>
> STATES:
> all udp 172.16.3.255:137 <- 172.16.3.17:137       NO_TRAFFIC:SINGLE
> all udp 172.16.3.17:137 -> 204.244.159.68:57827 -> 172.16.3.255:137
> SINGLE:NO_TRAFFIC
> all tcp 10.170.54.1:81 <- 172.16.3.71:3064       CLOSED:SYN_SENT
> all tcp 172.16.3.71:3064 -> 204.244.159.55:56563 -> 10.170.54.1:81
> SYN_SENT:CLOSED
> all tcp 10.170.54.1:81 <- 172.16.3.30:2021       CLOSED:SYN_SENT
> all tcp 172.16.3.30:2021 -> 204.244.159.68:54557 -> 10.170.54.1:81
> SYN_SENT:CLOSED
> all tcp 10.170.54.1:81 <- 172.16.3.72:1414       CLOSED:SYN_SENT
> all tcp 172.16.3.72:1414 -> 204.244.159.55:52567 -> 10.170.54.1:81
> SYN_SENT:CLOSED
> all tcp 10.170.54.1:81 <- 172.16.3.31:2865       CLOSED:SYN_SENT
> all tcp 172.16.3.31:2865 -> 204.244.159.68:59429 -> 10.170.54.1:81
> SYN_SENT:CLOSED
> all tcp 10.170.54.1:81 <- 172.16.3.72:1415       CLOSED:SYN_SENT
> all tcp 172.16.3.72:1415 -> 204.244.159.55:61425 -> 10.170.54.1:81
> SYN_SENT:CLOSED
> all tcp 10.170.54.1:81 <- 172.16.3.49:1914       CLOSED:SYN_SENT
> all tcp 172.16.3.49:1914 -> 204.244.159.68:58532 -> 10.170.54.1:81
> SYN_SENT:CLOSED
> all udp 172.16.3.255:138 <- 172.16.3.39:138       NO_TRAFFIC:SINGLE
> all udp 172.16.3.39:138 -> 204.244.159.68:62224 -> 172.16.3.255:138
> SINGLE:NO_TRAFFIC
> all tcp 64.56.145.72:110 <- 172.16.3.48:1494       FIN_WAIT_2:FIN_WAIT_2
> all tcp 172.16.3.48:1494 -> 204.244.159.55:62928 -> 64.56.145.72:110
> FIN_WAIT_2:FIN_WAIT_2
> all udp 172.16.3.255:137 <- 172.16.3.49:137       NO_TRAFFIC:SINGLE
> all udp 172.16.3.49:137 -> 204.244.159.55:61053 -> 172.16.3.255:137
> SINGLE:NO_TRAFFIC
> all tcp 10.170.54.1:81 <- 172.16.3.37:1508       CLOSED:SYN_SENT
> all tcp 172.16.3.37:1508 -> 204.244.159.68:54656 -> 10.170.54.1:81
> SYN_SENT:CLOSED
> all tcp 10.170.54.1:81 <- 172.16.3.74:3126       CLOSED:SYN_SENT
> all tcp 172.16.3.74:3126 -> 204.244.159.55:61282 -> 10.170.54.1:81
> SYN_SENT:CLOSED
> all tcp 10.170.54.1:81 <- 172.16.3.18:2446       CLOSED:SYN_SENT
> all tcp 172.16.3.18:2446 -> 204.244.159.68:58385 -> 10.170.54.1:81
> SYN_SENT:CLOSED
> all tcp 10.170.54.1:81 <- 172.16.3.73:2057       CLOSED:SYN_SENT
> all tcp 172.16.3.73:2057 -> 204.244.159.55:61692 -> 10.170.54.1:81
> SYN_SENT:CLOSED
> all udp 198.208.22.27:53 <- 172.16.3.74:58071       SINGLE:MULTIPLE
> all udp 172.16.3.74:58071 -> 204.244.159.68:54669 -> 198.208.22.27:53
> MULTIPLE:SINGLE
> all udp 198.208.22.27:53 <- 172.16.3.74:57503       SINGLE:MULTIPLE
> all udp 172.16.3.74:57503 -> 204.244.159.55:64923 -> 198.208.22.27:53
> MULTIPLE:SINGLE
> all udp 198.208.22.27:53 <- 172.16.3.74:51153       SINGLE:MULTIPLE
> all udp 172.16.3.74:51153 -> 204.244.159.68:61637 -> 198.208.22.27:53
> MULTIPLE:SINGLE
> all udp 172.16.3.255:137 <- 172.16.3.74:137       NO_TRAFFIC:SINGLE
> all udp 172.16.3.74:137 -> 204.244.159.55:53474 -> 172.16.3.255:137
> SINGLE:NO_TRAFFIC
> all tcp 10.170.54.1:81 <- 172.16.3.71:3065       CLOSED:SYN_SENT
> all tcp 172.16.3.71:3065 -> 204.244.159.68:63354 -> 10.170.54.1:81
> SYN_SENT:CLOSED
> all tcp 10.170.54.1:81 <- 172.16.3.29:4434       CLOSED:SYN_SENT
> all tcp 172.16.3.29:4434 -> 204.244.159.55:62977 -> 10.170.54.1:81
> SYN_SENT:CLOSED
> all udp 172.16.3.255:137 <- 172.16.3.30:137       NO_TRAFFIC:SINGLE
> all udp 172.16.3.30:137 -> 204.244.159.68:61298 -> 172.16.3.255:137
> SINGLE:NO_TRAFFIC
> all tcp 63.241.234.60:443 <- 172.16.3.37:1509
> ESTABLISHED:ESTABLISHED
> all tcp 172.16.3.37:1509 -> 204.244.159.68:61873 -> 63.241.234.60:443
> ESTABLISHED:ESTABLISHED
> all udp 198.208.22.27:53 <- 172.16.3.72:59314       SINGLE:MULTIPLE
> all udp 172.16.3.72:59314 -> 204.244.159.55:62186 -> 198.208.22.27:53
> MULTIPLE:SINGLE
> all udp 198.208.22.27:53 <- 172.16.3.72:55934       SINGLE:MULTIPLE
> all udp 172.16.3.72:55934 -> 204.244.159.68:51479 -> 198.208.22.27:53
> MULTIPLE:SINGLE
> all udp 198.208.22.27:53 <- 172.16.3.72:52983       SINGLE:MULTIPLE
> all udp 172.16.3.72:52983 -> 204.244.159.55:55523 -> 198.208.22.27:53
> MULTIPLE:SINGLE
> all udp 172.16.3.255:137 <- 172.16.3.72:137       NO_TRAFFIC:SINGLE
> all udp 172.16.3.72:137 -> 204.244.159.68:58218 -> 172.16.3.255:137
> SINGLE:NO_TRAFFIC
> all tcp 10.170.54.1:81 <- 172.16.3.31:2868       CLOSED:SYN_SENT
> all tcp 172.16.3.31:2868 -> 204.244.159.55:60911 -> 10.170.54.1:81
> SYN_SENT:CLOSED
> all udp 172.16.3.255:137 <- 172.16.3.77:137       NO_TRAFFIC:SINGLE
> all udp 172.16.3.77:137 -> 204.244.159.55:59287 -> 172.16.3.255:137
> SINGLE:NO_TRAFFIC
> all tcp 10.170.54.1:81 <- 172.16.3.72:1416       CLOSED:SYN_SENT
> all tcp 172.16.3.72:1416 -> 204.244.159.68:59828 -> 10.170.54.1:81
> SYN_SENT:CLOSED
> all tcp 10.170.54.1:81 <- 172.16.3.49:1915       CLOSED:SYN_SENT
> all tcp 172.16.3.49:1915 -> 204.244.159.55:64580 -> 10.170.54.1:81
> SYN_SENT:CLOSED
> all tcp 10.170.54.1:81 <- 172.16.3.29:4435       CLOSED:SYN_SENT
> all tcp 172.16.3.29:4435 -> 204.244.159.68:60089 -> 10.170.54.1:81
> SYN_SENT:CLOSED
> all udp 172.16.3.255:137 <- 172.16.3.8:137       NO_TRAFFIC:SINGLE
> all udp 172.16.3.8:137 -> 204.244.159.68:60176 -> 172.16.3.255:137
> SINGLE:NO_TRAFFIC
> all tcp 10.170.54.1:81 <- 172.16.3.51:3433       CLOSED:SYN_SENT
> all tcp 172.16.3.51:3433 -> 204.244.159.55:63158 -> 10.170.54.1:81
> SYN_SENT:CLOSED
> all tcp 10.170.54.1:81 <- 172.16.3.37:1510       CLOSED:SYN_SENT
> all tcp 172.16.3.37:1510 -> 204.244.159.68:63197 -> 10.170.54.1:81
> SYN_SENT:CLOSED
> all tcp 10.170.54.1:81 <- 172.16.3.74:3127       CLOSED:SYN_SENT
> all tcp 172.16.3.74:3127 -> 204.244.159.55:61760 -> 10.170.54.1:81
> SYN_SENT:CLOSED
> all tcp 10.170.54.1:81 <- 172.16.3.18:2447       CLOSED:SYN_SENT
> all tcp 172.16.3.18:2447 -> 204.244.159.68:61951 -> 10.170.54.1:81
> SYN_SENT:CLOSED
> all tcp 10.170.54.1:81 <- 172.16.3.73:2058       CLOSED:SYN_SENT
> all tcp 172.16.3.73:2058 -> 204.244.159.55:53396 -> 10.170.54.1:81
> SYN_SENT:CLOSED
> all udp 198.208.22.27:53 <- 172.16.3.74:62024       SINGLE:MULTIPLE
> all udp 172.16.3.74:62024 -> 204.244.159.55:63136 -> 198.208.22.27:53
> MULTIPLE:SINGLE
> all tcp 72.14.162.41:80 <- 172.16.3.74:3128       TIME_WAIT:TIME_WAIT
> all tcp 172.16.3.74:3128 -> 204.244.159.68:58088 -> 72.14.162.41:80
> TIME_WAIT:TIME_WAIT
> all tcp 72.14.162.41:80 <- 172.16.3.74:3129       FIN_WAIT_2:FIN_WAIT_2
> all tcp 172.16.3.74:3129 -> 204.244.159.55:62718 -> 72.14.162.41:80
> FIN_WAIT_2:FIN_WAIT_2
> all udp 172.16.3.255:138 <- 172.16.3.71:138       NO_TRAFFIC:SINGLE
> all udp 172.16.3.71:138 -> 204.244.159.68:52993 -> 172.16.3.255:138
> SINGLE:NO_TRAFFIC
> all tcp 10.170.54.1:81 <- 172.16.3.71:3066       CLOSED:SYN_SENT
> all tcp 172.16.3.71:3066 -> 204.244.159.68:50898 -> 10.170.54.1:81
> SYN_SENT:CLOSED
>

Ooops, all TCP states are closed/dead ? No new TCP flows registered ?

You better check with "pfctl -v -s state" what's going on there, what's the
aging and expiring times...

Regards,
Adrian.