Default route destination changing without warning follow-up

[Dominic Blais]

About hardware, the my server is a HP Proliant ML115. It has a Broadcom BCM5722 as network controller. In my network configuration, I'm using a single NIC (bge0). This interface has an IP address in the default gateway subnet. I also have a vlan interface on the same NIC for pppoe traffic.




--



-----Message d'origine-----
De : Dominic Blais 
Envoyé : 4 octobre 2012 09:41
À : 'Alexander V. Chernikov'
Cc : freebsd-net at freebsd.org
Objet : RE: Default route destination changing without warning follow-up

Hi,

The server that actually has the problem is:

- FreeBSD 9.0-RELEASE-p3
- Uses IPFW & dummynet for traffic shaping
- Uses PF for firewalling and NAT
- Uses MPD 5.6 for PPPoE server
- Uses freeradius and PostgreSQL for authentication.

I wrote a script some weeks ago that does the following:

=====
#!/bin/sh

RIGHT_GATEWAY='1.1.1.1'
NOC_EMAIL='blah at test.com'

i=5
while [ $i -gt 0 ]
do
    if [ `/usr/bin/netstat -rn |grep default |grep $RIGHT_GATEWAY |wc -l` -ne 1 ]
    then
        /sbin/route del -net default
        /sbin/route add -net default $RIGHT_GATEWAY
        tail -n 100 /tmp/route_monitor.log | mail -s "Default route bug happened" $NOC_EMAIL
    fi
    i=`expr $i - 1`
    sleep 10
done
=====

I've changed the variables for privacy purposes. This script is executed each minutes in crontab. So if the gateway changes, it is fixed within 10 seconds (a mean of 5 seconds).

I have seen multiple days where it doesn't happen at all... Today was a record!! 43 times between 22:43 and 23:56. It means, 43 times the script detected the gateway wasn't the good one and had to change it. It looks like it happens when network usage is higher than usual.. there's obviously a link with usage...The server isn't using all its resources. I usually see 25% CPU usage at higher loads...  All loads value are almost always under 0.4 and are mostly around 0.35 for past 15 minutes.

I'm running a route monitor in a screen and I'm sending the output to a textfile. As you see in the script, I'm sending me the last 100 lines of it and I only see my script changing the gateway... nothing before is changing it. The other people experiencing that bug seems to have the same behaviour.




-----Message d'origine-----
De : Alexander V. Chernikov [mailto:melifaro at FreeBSD.org] Envoyé : 30 septembre 2012 17:07 À : Dominic Blais Cc : freebsd-net at freebsd.org Objet : Re: Default route destination changing without warning follow-up

On 01.10.2012 00:59, Dominic Blais wrote:
> It's all about IPv4 in my case.

It will be great to supply some more details (e.g. like FreeBSD version, interfaces configuration, netstat -rn output).

How often does this happen ?
(e.g. while true; do echo -n `date` ; route -n get default | grep gate; sleep 1; done can help)

If this is reproducible, what actions precedes this change?
Maybe some ARP traffic on that interface, or interface creation/deletion, or.. ?

Is route monitor completely silent when the change happens?

>
>
> --
>
>
>
> -----Message d'origine-----
> De : Alexander V. Chernikov [mailto:melifaro at FreeBSD.org] Envoyé : 30 
> septembre 2012 16:39 À : Dominic Blais Cc : freebsd-net at freebsd.org 
> Objet : Re: Default route destination changing without warning 
> follow-up
>
> On 01.10.2012 00:33, Dominic Blais wrote:
>> Yes, I'm very sure of it! A "route monitor" show no changes when it happens. I've heard of another person with the same bug that has no traces in "route monitor" either... We have in common that we're using IPFW and PF.
> So, are we talking about IPv4 or IPv6?
>>
>>
>>
>> --
>>
>>
>>
>> -----Message d'origine-----
>> De : Alexander V. Chernikov [mailto:melifaro at FreeBSD.org] Envoyé : 30 
>> septembre 2012 16:31 À : Dominic Blais Cc : freebsd-net at freebsd.org 
>> Objet : Re: Default route destination changing without warning 
>> follow-up
>>
>> On 01.10.2012 00:00, Dominic Blais wrote:
>>> Hi,
>> Hello!
>>>
>>> I was just wondering if there was anything new about the bug of default route changing without warning...  Is there any test I can do to help fixing it?
>> Can you be a bit more precise and specify FreeBSD version and address family?
>>
>> Are you sure that it is not changed by some other userland process (e.g.
>> did you do some `route monitor` checks) ?
>>>
>>> --
>>> [cid:image001.gif at 01CD9F24.B366CBF0]
>>>
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> freebsd-net at freebsd.org mailing list 
>>> http://lists.freebsd.org/mailman/listinfo/freebsd-net
>>> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"
>>
>>
>
>