ipfw fwd sends to port but also through gateway
Adonis Peralta
donileo at gmail.com
Wed Jun 8 00:12:39 UTC 2016
For Test #1 you are correct that to solely test if the fwd was terminating the search the rule should have been
add 13 deny log logamount 5 tcp from 192.168.1.0/24{1-5,7-254} to any dst-port 80 in via igb0
I went ahead and tested that and YES the search does properly terminate.
The rule that I had added "00013 deny log logamount 5 tcp from any 80 to any via igb0” however was meant to show the weird behavior that I was seeing.
As I had mentioned before, nothing was running on 8080, 80, 8083 for these tests. I purposely stopped squid, nginx, and anything but the base system utilities. I definitely get that its super weird to see my freebsd box responding back to the client by spoofing the ip address on its own, but thats what happening for me. I think somehow what is happening is that since the packet comes in to the freebsd box even though the dst address of the packet is not the same as the system, the system simply responds back as if the packet was intended for it.. Is there any sysctl variable or anything that would cause my box to respond back as a proxy on its own? I don’t believe the behavior I'm stating here should be happening. It doesn't happen without the fwd rule and as soon as I put the fwd rule the bad behavior happens. Like I previously stated in my last email, even If I completely block internet (outside) access to my freebsd box to make sure none of the packets make it to the actual website and therefore prevent any site from responding back via my freebsd machine, the behavior above still happens.
Here is the result of the sockstat command to show what listening ports are currently running.
$ sockstat -46 -l
unbound unbound 5461 3 udp6 ::1:53 *:*
unbound unbound 5461 4 tcp6 ::1:53 *:*
unbound unbound 5461 5 udp4 127.0.0.1:53 *:*
unbound unbound 5461 6 tcp4 127.0.0.1:53 *:*
root sshd 922 3 tcp6 *:22 *:*
root sshd 922 4 tcp4 *:22 *:*
root smbd 865 27 tcp6 *:445 *:*
root smbd 865 28 tcp6 *:139 *:*
root smbd 865 29 tcp4 *:445 *:*
root smbd 865 30 tcp4 *:139 *:*
root nmbd 862 9 udp4 *:137 *:*
root nmbd 862 10 udp4 *:138 *:*
root nmbd 862 11 udp4 192.168.1.6:137 *:*
root nmbd 862 12 udp4 192.168.1.255:137 *:*
root nmbd 862 13 udp4 192.168.1.6:138 *:*
root nmbd 862 14 udp4 192.168.1.255:138 *:*
root ntpd 830 20 udp6 *:123 *:*
root ntpd 830 21 udp4 *:123 *:*
root ntpd 830 22 udp4 192.168.1.6:123 *:*
root ntpd 830 23 udp6 ::1:123 *:*
root ntpd 830 24 udp6 fe80::1%lo0:123 *:*
root ntpd 830 25 udp4 127.0.0.1:123 *:*
root syslogd 742 6 udp6 *:514 *:*
root syslogd 742 7 udp4 *:514 *:*
Here is my sysctl.conf
net.inet.ip.fw.one_pass=0
net.inet.ip.forwarding=1
net.inet.ip.fastforwarding=1
net.inet.tcp.tso=0
net.inet.tcp.msl=7500
net.inet.tcp.delayed_ack=0
net.inet.ip.fw.verbose_limit=5
In my rc.conf gateway_enable = “YES” is set.
Thanks,
Adonis
>
>
>> On May 30, 2016, at 2:13 AM, Julian Elischer <julian at freebsd.org <mailto:julian at freebsd.org>> wrote:
>>
>> On 28/05/2016 1:39 PM, Adonis Peralta wrote:
>>> Ok so some more information on what I am seeing, Now even a little more weird. My system is a relative stock FreeBSD 10.3 Release with latest patches installed. Pretty much stock with very little amount of third party libraries running.
>>>
>>> Test #1 — Lets add an additional rule after the fwd to see if the fwd is not terminating the search. Also, for this test I stopped Squid completely as a service and verified its not running. Nothing on port 80 is running either.
>>> Make a request to cnn.com <http://cnn.com/> via client.
>>>
>>> 00010 allow ip from any to any via lo0
>>>
>>> 00012 fwd 127.0.0.1,8083 tcp from 192.168.1.0/24{1-5,7-254} to any dst-port 80 in via igb0 #original fwd rule
>>>
>>> 00013 deny log logamount 5 tcp from any 80 to any via igb0 #new additional rule to test if fwd is terminating search
>> don't you mean
>> add 13 deny log logamount 5 tcp from 192.168.1.0/24{1-5,7-254} to any dst-port 80 in via igb0
>> to see if the original packet gets pas the fwd?
>>>
>>> 00015 allow ip from 192.168.1.0/24 to any in via igb0
>>> 00016 deny ip from any to any not antispoof in
>>> 00030 check-state
>>> 00040 allow tcp from any to any out via igb0 setup keep-state
>>> 00041 allow ip from any to any out via igb0 keep-state
>>> 00050 deny ip from 192.168.0.0/16 to any in via igb0
>>> 00051 deny ip from 172.16.0.0/12 to any in via igb0
>>> 00052 deny ip from 10.0.0.0/8 to any in via igb0
>>> 00053 deny ip from 127.0.0.0/8 to any in via igb0
>>> 00054 deny ip from 0.0.0.0/8 to any in via igb0
>>> 00055 deny ip from 169.254.0.0/16 to any in via igb0
>>> 00056 deny ip from 192.0.2.0/24 to any in via igb0
>>> 00057 deny ip from 204.152.64.0/23 to any in via igb0
>>> 00058 deny ip from 224.0.0.0/3 to any in via igb0
>>> 00070 deny tcp from any to any dst-port 113 in via igb0
>>> 00080 deny tcp from any to any dst-port 137 in via igb0
>>> 00081 deny tcp from any to any dst-port 138 in via igb0
>>> 00082 deny tcp from any to any dst-port 139 in via igb0
>>> 00083 deny tcp from any to any dst-port 81 in via igb0
>>> 00100 deny log logamount 5 tcp from any to any established in via igb0
>>> 00499 deny log logamount 5 ip from any to any
>>> 00500 allow ip from any to any
>>> 00599 deny log logamount 5 ip from any to any
>>> 65535 deny ip from any to any
>>>
>>> When I add rule 00013 to ipfw and restart it.. I get something very very interesting in the logs:
>>>
>>> May 28 00:51:29 bruce kernel: ipfw: 13 Deny TCP 157.166.226.26:80 192.168.1.3:62805 out via igb0 #####157.166.226.26 port 80 is the dns -resolved address of a website I made a request to via my client. But this is going OUT through my local lan link… What???
>> this is normal.
>> the fwd rule activaed IP spoofing on the proxy.. the proxy is responding to the original client as if it were the original destination.
>> of course if hte proxy is not running hten something else must hav ea socket open in listen mode.
>>
>>> May 28 00:51:29 bruce kernel: ipfw: 13 Deny TCP 157.166.226.25:80 192.168.1.3:62806 out via igb0
>>> May 28 00:51:30 bruce kernel: ipfw: 13 Deny TCP 157.166.226.26:80 192.168.1.3:62805 out via igb0
>>> May 28 00:51:30 bruce kernel: ipfw: 13 Deny TCP 157.166.226.25:80 192.168.1.3:62806 out via igb0
>>> May 28 00:51:31 bruce kernel: ipfw: 13 Deny TCP 157.166.226.26:80 192.168.1.3:62805 out via igb0
>>> May 28 00:51:31 bruce kernel: ipfw: limit 5 reached on entry 13
>>>
>>> As you can see above somehow someway my FreeBSD machine (bruce) is sending back responses back to the client by by using the requested website ip as its own ip now. That is super weird. The ip of my freebsd machine is 192.168.1.6 not what you see above.
>>
>>> I don’t believe any service is responsible for the above as (squid is not running, neither is any program binded to 8080 or 80) but yet somehow my bsd machine is for some it reason attempting to respond on its own back to the client using the destination ip address the client made a request to as its SRC address now.
>>>
>> Something must have a socket 8083 open that is responding.
>> unless the bug is that it is not being selective, and the 8083 is getting lost, and ANY socket that is listening is getting the packet.
>>
>>
>>> Now as crazy as that sounds, I went ahead and tried to take a look more closely at my tcpdump to see what I can find there. Here is some relevant data:
>>>
>>> 192.168.1.3.62805 > 157.166.226.26.http: Flags [S], cksum 0x7776 (correct), seq 31167568, win 65535, options [mss 1460,nop,wscale 5,nop,nop,TS val 321249059 ecr 0,sackOK,eol], length 0
>>>
>>> 00:51:29.741052 IP (tos 0x0, ttl 63, id 34976, offset 0, flags [DF], proto TCP (6), length 64)
>>> 192.168.1.3.62806 > 157.166.226.25.http: Flags [S], cksum 0x2a5b (correct), seq 1085448749, win 65535, options [mss 1460,nop,wscale 5,nop,nop,TS val 321249162 ecr 0,sackOK,eol], length 0
>>>
>>> Ok… Not much to see there because the 00013 is rule is actually blocking the outbound traffic.. Lets take away the 00013 rule and see what tcpdump gives us:
>>>
>> no your rule is blocking the return packets from the proxy to the client I think
>>
>>> 01:08:23.047688 IP (tos 0x0, ttl 63, id 40357, offset 0, flags [DF], proto TCP (6), length 64)
>>> 192.168.1.3.62857 > 157.166.226.25.http: Flags [S], cksum 0xe105 (correct), seq 691019997, win 65535, options [mss 1460,nop,wscale 5,nop,nop,TS val 321765751 ecr 0,sackOK,eol], length 0
>>>
>>> 01:08:23.047739 IP (tos 0x0, ttl 63, id 14370, offset 0, flags [DF], proto TCP (6), length 40)
>>> 157.166.226.25.http > 192.168.1.3.62857: Flags [R.], cksum 0x4186 (incorrect -> 0x2e7d), seq 0, ack 691019998, win 0, length 0
>>>
>>> 01:08:23.211028 IP (tos 0x0, ttl 63, id 61609, offset 0, flags [DF], proto TCP (6), length 64)
>>> 192.168.1.3.62858 > 157.166.226.26.http: Flags [S], cksum 0xda95 (correct), seq 2457714015, win 65535, options [mss 1460,nop,wscale 5,nop,nop,TS val 321765909 ecr 0,sackOK,eol], length 0
>>>
>>> 01:08:23.211081 IP (tos 0x0, ttl 63, id 14383, offset 0, flags [DF], proto TCP (6), length 40)
>>> 157.166.226.26.http > 192.168.1.3.62858: Flags [R.], cksum 0x4187 (incorrect -> 0x28ab), seq 0, ack 2457714016, win 0, length 0
>>>
>>> Ok so pay attention to the response times.. 01:08:23.047688 a request comes from the client.. and at 01:08:23.047739 a response goes back to the client??? Really? A network request to a website took less than .000051 seconds to go across and come back?? No way.. This response is for sure coming back from my machine.
>> yes it is.. that is what local fwd does. it's for transparent proxy. it wouldn't be transparent otherwise.
>>
>>>
>>> Considering how weird this is (Never seen this before) I went ahead and then essentially killed the internet connection on my FreeBSD machine (basically I blocked all requests at the router level to outside the local lan) and again made a request to cnn.com <http://cnn.com/>. What did I find? The same results as above.. My bsd machine attempts to sends replies back using the requested DST IP as the SRC address. TcpDump confirms the same. This definitely tells me that the above packets are sent by my BSD machine as theres no way any remote machine outside my local lan got any requests to even respond...
>>>
>>> So fine..lets continue testing further.. Lets remove the fwd rule 00012 and see what we get..
>>>
>>> Test #2 - Remove the fwd rule — No squid or service running on port 8080 or 80. Restore the internet connection on the BSD machine.
>>> Make a request to cnn.com <http://cnn.com/> via client.
>>>
>>> Our pertinent ipfw rules become:
>>>
>>> 00010 allow ip from any to any via lo0
>>>
>>> 00013 deny log logamount 5 tcp from any 80 to any via igb0 #This Rule stays fwd rule gets removed
>>>
>>> 00015 allow ip from 192.168.1.0/24 to any in via igb0
>>> 00016 deny ip from any to any not antispoof in
>>> 00030 check-state
>>> 00040 allow tcp from any to any out via igb0 setup keep-state
>>> 00041 allow ip from any to any out via igb0 keep-state
>>> 00050 deny ip from 192.168.0.0/16 to any in via igb0
>>> 00051 deny ip from 172.16.0.0/12 to any in via igb0
>>> 00052 deny ip from 10.0.0.0/8 to any in via igb0
>>> 00053 deny ip from 127.0.0.0/8 to any in via igb0
>>> 00054 deny ip from 0.0.0.0/8 to any in via igb0
>>> 00055 deny ip from 169.254.0.0/16 to any in via igb0
>>> 00056 deny ip from 192.0.2.0/24 to any in via igb0
>>> 00057 deny ip from 204.152.64.0/23 to any in via igb0
>>> 00058 deny ip from 224.0.0.0/3 to any in via igb0
>>> 00070 deny tcp from any to any dst-port 113 in via igb0
>>> 00080 deny tcp from any to any dst-port 137 in via igb0
>>> 00081 deny tcp from any to any dst-port 138 in via igb0
>>> 00082 deny tcp from any to any dst-port 139 in via igb0
>>> 00083 deny tcp from any to any dst-port 81 in via igb0
>>> 00100 deny log logamount 5 tcp from any to any established in via igb0
>>> 00499 deny log logamount 5 ip from any to any
>>> 00500 allow ip from any to any
>>> 00599 deny log logamount 5 ip from any to any
>>> 65535 deny ip from any to any
>>>
>>> What are our results? Rule 00013 doesn’t get matched as in Test #1 and we see no logs pertaining 13 Deny TCP as in Test#1. But what does tcpdump give us?
>>>
>>> 01:19:25.047593 IP (tos 0x0, ttl 63, id 27280, offset 0, flags [DF], proto TCP (6), length 64)
>>> 192.168.1.3.62898 > 157.166.226.26.http: Flags [S], cksum 0xc08f (correct), seq 3330522041, win 65535, options [mss 1460,nop,wscale 5,nop,nop,TS val 322152845 ecr 0,sackOK,eol], length 0
>>>
>>> 01:19:25.047612 IP (tos 0x0, ttl 62, id 27280, offset 0, flags [DF], proto TCP (6), length 64)
>>> 192.168.1.3.62898 > 157.166.226.26.http: Flags [S], cksum 0xc08f (correct), seq 3330522041, win 65535, options [mss 1460,nop,wscale 5,nop,nop,TS val 322152845 ecr 0,sackOK,eol], length 0
>>>
>>> 01:19:25.047731 IP (tos 0x0, ttl 61, id 27280, offset 0, flags [DF], proto TCP (6), length 64)
>>> 192.168.1.3.62898 > 157.166.226.26.http: Flags [S], cksum 0xc08f (correct), seq 3330522041, win 65535, options [mss 1460,nop,wscale 5,nop,nop,TS val 322152845 ecr 0,sackOK,eol], length 0
>>>
>>> 01:19:25.047739 IP (tos 0x0, ttl 60, id 27280, offset 0, flags [DF], proto TCP (6), length 64)
>>> 192.168.1.3.62898 > 157.166.226.26.http: Flags [S], cksum 0xc08f (correct), seq 3330522041, win 65535, options [mss 1460,nop,wscale 5,nop,nop,TS val 322152845 ecr 0,sackOK,eol], length 0
>>>
>>> 01:19:25.047853 IP (tos 0x0, ttl 59, id 27280, offset 0, flags [DF], proto TCP (6), length 64)
>>> 192.168.1.3.62898 > 157.166.226.26.http: Flags [S], cksum 0xc08f (correct), seq 3330522041, win 65535, options [mss 1460,nop,wscale 5,nop,nop,TS val 322152845 ecr 0,sackOK,eol], length 0
>>>
>>>
>>> So ok, the traffic now goes back to normal. This definitely tells us that whatever this weird behavior I am seeing seems to be tied to the fwd rule. It also tells us that this is not related to squid as for both tests squid was not running as was no service on port 8080 or 80.
>>>
>>>
>>> As far as Squid, whenever Squid is running what will happen is that I get the same traffic as in Test #1 but now squid picks up some of those packets as requests to itself and will masquerade and send requests down by using the server ip (as SRC) and it does respond back to the client correctly. However because of the additional traffic in Test #1 (what bsd sends back from 157.166.226.26) that shouldn’t be there the TCP flow between the client and squid gets messed up to the point of many TCP retransmissions and overall a sluggish or very very bad connection squid via my client.
>>>
>>> Hope this info helps and let me know if you any more details from my test.
>>>
>>> Thanks,
>>> Adonis
>>>
>>>> On May 26, 2016, at 10:28 AM, Julian Elischer <julian at freebsd.org <mailto:julian at freebsd.org>> wrote:
>>>>
>>>> On 26/05/2016 2:03 AM, Adonis Peralta wrote:
>>>>> Hi all,
>>>>>
>>>>> I am noticing something weird in regards to ipfw forwarding when I am attempting to set up squid web proxying.
>>>>>
>>>>> Here is the info:
>>>>>
>>>>> ipfw rule: ipfw -q add fwd 127.0.0.1,8080 tcp from 192.168.1.0/24{1-5,7-254} to any dst-port 80 in via igb0 //I exclude the servers ip 192.168.1.6 here to prevent a loop
>>>>> Squid Proxy: running on localhost (127.0.0.1) port 8080.
>>>>> Freebsd box ip: 192.168.1.6
>>>>> Router box: 192.168.1.1
>>>>>
>>>>> Essentially when any ip (not my freebsd ip) makes a request to port 80 my router will route that ip using policy based routing to my freebsd box. Then the ipfw fwd rule above sends that traffic over to my squid proxy port. This is working fine and the fwd rule above does definitely match.
>>>>> However the issue Im seeing is that ipfw fwd not only sends the packet out to the squid proxy but ALSO sends it out to the original destination causing all sorts of issues for my client because it messes up the tcp flow/handshaking.
>>>>>
>>>>> To be more clear what I see is when client 192.168.1.3 makes a request on port 80… my freebsd box receives it.. then forwards it to squid but also sends it out to the original destination so for every packet coming to port 80 i see two going out..
>>>>>
>>>>> To debug this problem a bit further I stopped squid, and setup "nc -l 8080" to catch incoming requests via the fwd.
>>>>>
>>>>> Doing a tcpdump I see:
>>>>>
>>>>> 192.168.1.3.57653 > s3-us-west-1.amazonaws.com.http: Flags [S], cksum 0x9385 (correct), seq 1939422713, win 65535, options [mss 1460,nop,wscale 5,nop,nop,TS val 1149232947 ecr 0,sackOK,eol], length 0
>>>>> 13:14:16.209753 IP (tos 0x0, ttl 64, id 10951, offset 0, flags [DF], proto TCP (6), length 60)
>>>>> s3-us-west-1.amazonaws.com.http > 192.168.1.3.57653: Flags [S.], cksum 0xe4da (incorrect -> 0x8343), seq 3934654233, ack 1939422714, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 1794161828 ecr 1149232947], length 0
>>>>>
>>>>> Netcat catches the HTTP Get request (i can see it in netcats console).. but the above tcpdump definitely tells me that the request was also sent to to aws itself this is implied by the fact that aws responded back to original ip (192.168.1.3).
>>>>>
>>>>> When I have squid running I see the same thing in the above tcpdump but also communication between my freebsd box ip 192.168.1.6 and the requested http site.
>>>>>
>>>>> Why is this happening? Is this a bug?
>>>> definitely sounds like a bug.. The fwd rule is supposed to terminate the search.. Can you confirm that a matching rule following the fwd does not see the packet continuing on? I used it for many years and it acted as expected.
>>>> Is there any rule you can add that catches the outgoing extra packet and blocks it (as a work-around)
>>>> what does squid's outgoing packet look like? is it masquerading the client or is it using its own address?
>>>>
>>>>
>>>>>
>>>>> -Adonis
>>>>> _______________________________________________
>>>>> freebsd-ipfw at freebsd.org <mailto:freebsd-ipfw at freebsd.org> mailing list
>>>>> https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw <https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw>
>>>>> To unsubscribe, send any mail to " <mailto:freebsd-ipfw-unsubscribe at freebsd.org>freebsd-ipfw-unsubscribe at freebsd.org <mailto:freebsd-ipfw-unsubscribe at freebsd.org>"
>>>
>>
>>
>
More information about the freebsd-ipfw
mailing list