FreeBSD sometimes uses the router for packets on the local network
Anton Farber
dr_sweety_1337 at hotmail.com
Mon Apr 6 18:15:46 UTC 2015
Good evening,
I've opened a thread on the FreeBSD networking forum (https://forums.freebsd.org/threads/jail-fails-to-connect-to-main-host.50833/) as sometime ago my FreeBSD server (initially running 10.1, now CURRENT) started to behave strangely after an upgrade from 10.0 to 10.1. I first noticed that a jail (192.168.1.5) wasn't able to contact the base system (192.168.1.1). Running a tcpdump revealed the following: the jail is using em0 instead of lo0 for communicating with the base system:
------------------------
* tcpdump on em0
Source Destination Protocol Length Info
192.168.1.5 192.168.1.1 TCP 74 28848→22 [SYN]
* tcpdump on lo0
Source Destination Protocol Length Info
192.168.1.1 192.168.1.5 TCP 64 22→28848 [SYN, ACK]
------------------------
I don't think that this is the way it's supposed to work. Next thing was, that random hosts on the network were unable to contact the server. The server on the other hand thought it needed to reach those hosts (in this example 192.168.1.61) via the router:
------------------------
# ping -c 1 192.168.1.61
PING 192.168.1.61 (192.168.1.61): 56 data bytes
36 bytes from router.local.lan (192.168.1.254): Redirect Host(New addr: 192.168.1.61)
Vr HL TOS Len ID Flg off TTL Pro cks Src Dst
4 5 00 0054 a734 0 0000 40 01 4fe6 192.168.1.1 192.168.1.61
64 bytes from 192.168.1.61: icmp_seq=0 ttl=64 time=1.009 ms
--- 192.168.1.61 ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 1.009/1.009/1.009/0.000 ms#
------------------------
Other hosts on the same network were able to reach 192.168.1.61 directly, so it can't be neither 192.168.1.61 nor the router (I do not have any static routes for 192.168.1.61/32 on my router) that are causing the problems. Status on the server:
------------------------
# arp -a
server.local.Lan (192.168.1.1) at 00:1c:c0:6f:c2:60 on em0 permanent [ethernet]
laptop.local.Lan (192.168.1.32) at 5c:26:0a:2a:37:10 on em0 expires in 1197 seconds [ethernet]
jail.local.Lan (192.168.1.5) at 00:1c:c0:6f:c2:60 on em0 permanent [ethernet]
WinXP.local.Lan (192.168.1.4) at 08:00:27:80:b8:10 on em0 expires in 1200 seconds [ethernet]
? (192.168.1.61) at 00:04:20:05:31:38 on em0 expires in 1150 seconds [ethernet]
? (192.168.1.255) at (incomplete) on em0 expired [ethernet]
Router.local.Lan (192.168.1.254) at 00:0d:b9:00:11:68 on em0 expires in 1150 seconds [ethernet]
Phone.local.Lan (192.168.1.21) at 00:0e:08:bc:ed:94 on em0 expires in 799 seconds [ethernet]
# netstat -rn
Routing tables
Internet:
Destination Gateway Flags Netif Expire
default 192.168.1.254 UGS em0
127.0.0.1 link#2 UH lo0
192.168.1.0/24 link#1 U em0
192.168.1.1 link#1 UHS lo0
192.168.1.5 link#1 UHS lo0
192.168.1.5/32 link#1 U em0
Internet6:
Destination Gateway Flags Netif Expire
::/96 ::1 UGRS lo0
::1 link#2 UH lo0
::ffff:0.0.0.0/96 ::1 UGRS lo0
fe80::/10 ::1 UGRS lo0
fe80::%lo0/64 link#2 U lo0
fe80::1%lo0 link#2 UHS lo0
fe80::%tun0/64 link#4 U tun0
fe80::21c:c0ff:fe6f:c260%tun0 link#4 UHS lo0
ff02::/16 ::1 UGRS lo0
------------------------
So the server knew the (correct) MAC address of .61 but still tried to reach it via the router. I've run a tcpdump on em0, this is where things get interesting:
------------------------
No. Time Source Destination Protocol Length Info
186 1.028186 192.168.1.1 192.168.1.61 ICMP 98 Echo (ping) request id=0x2220, seq=0/0, ttl=64 (reply in 190)
Ethernet II, Src: IntelCor_6f:c2:60 (00:1c:c0:6f:c2:60), Dst: PcEngine_00:11:68 (00:0d:b9:00:11:68)
No. Time Source Destination Protocol Length Info
189 1.029008 192.168.1.254 192.168.1.1 ICMP 70 Redirect (Redirect for host)
Ethernet II, Src: PcEngine_00:11:68 (00:0d:b9:00:11:68), Dst: IntelCor_6f:c2:60 (00:1c:c0:6f:c2:60)
No. Time Source Destination Protocol Length Info
190 1.029392 192.168.1.61 192.168.1.1 ICMP 98 Echo (ping) reply id=0x2220, seq=0/0, ttl=64 (request in 186)
Ethernet II, Src: SlimDevi_05:31:38 (00:04:20:05:31:38), Dst: IntelCor_6f:c2:60 (00:1c:c0:6f:c2:60)
------------------------
The server issues an echo request for .61 but is using the router's MAC address (:68)! Obviously the router answers with a redirect..
I've shut down all daemons and kernel modules if possible until just the absolute bare minimum (sshd, init and some other base services) was running. I've upgraded to CURRENT. This didn't change anything. What helped in the end was issuing a "route flush ; route add default 192.168.1.254": the server was immediately able to contact .61 directly.
The problem seemingly appears randomly, sometimes it takes weeks for it to reappear.
Sorry for the long post, I've tried to keep it as compact as possible. Does anyone have any idea what might be causing those problems?
Regards, Anton
More information about the freebsd-net
mailing list