bind 8 slow when resolving new domains!

[dap99 at i-55.com]

I am having a big problem with slow internal DNS (bind 8 on FreeBSD 4.9).
If we do a query against a local domain (our DNS server is authoratative)
then the response is fast. If we do a query against anything in bind's
cache the resp. is fast. If we do a query for a new non-local domain then
the resp is SLOW or times-out. FYI, we are behind a NetScreen firewall at
a colo. The colo promises it is not them. Also, we are using their two DNS
servers as forwarders.

The colo promises it's not them, but frankly I can't see how it's us.

# tcpdump -n host ns2 and \( icmp or udp \)
10:07:37.832611 192.168.42.78.53 > isp-dns1.53:  4240+ [1au] A?
www.altavista.com. (46)
10:07:51.013213 192.168.42.78.53 > isp-dns2.53:  4240+ [1au] A?
www.altavista.com. (46)
10:07:51.074160 isp-dns2.53 > 192.168.42.78.53:  4240 2/9/10
CNAME[|domain] (DF)
10:07:51.074476 192.168.42.78.53 > isp-dns1.53:  17509+ [1au] A?
avatw.search.yahoo2.akadns.net. (59)
10:07:51.131568 isp-dns1.53 > 192.168.42.78.53:  17509 1/9/10 (393) (DF)

That's a query for www.altavista.com. That took around 13 seconds. I'm
surprised it didn't time-out!

Here is my options {} (more to follow after this):

options {
        directory "/etc/namedb";

        listen-on { 192.168.42.78; };

        forward only; // added while troubleshooting
        forward first; // added while troubleshooting
        forwarders {
                isp-dns1;
                isp-dns2;
        };

        allow-transfer {
                127.0.0.1;
                192.168.42.0/24;
        };

        fetch-glue no;

        // we have a firewall between us and the Internet, so let's
        // go ahead and define our query source port
        query-source address 192.168.42.78 port 53;

        named-xfer "/usr/libexec/named-xfer";
};

Okay, so what happens if I try to disable my forwarders?

I now have:

...
//      forward only;
//      forward first;
//        forwarders {
//                isp-dns1;
//                isp-dns2;
//        };
...

So let's try a random domain name:

ns2# nslookup www.looser.com
Server:  ns2
Address:  192.168.42.78

*** ns2 can't find www.looser.com: Non-existent host/domain
ns2# nslookup www.looser.com
Server:  ns2
Address:  192.168.42.78

Name:    www.looser.com
Address:  217.8.158.117

# tcpdump -n host ns2 and \( icmp or udp \)
tcpdump: listening on rl0
10:13:50.515557 192.168.42.78.53 > 192.33.4.12.53:  21568 [1au] A?
www.looser.com. (43)
10:13:50.562594 192.33.4.12.53 > 192.168.42.78.53:  21568- 0/13/14 (475)
10:13:50.563816 192.168.42.78.53 > 192.33.14.30.53:  39445 [1au] A?
www.looser.com. (43)
10:13:50.619570 192.33.14.30.53 > 192.168.42.78.53:  39445 FormErr- [0q]
0/0/0 (12) (DF)
10:13:50.619641 192.168.42.78.53 > 192.33.14.30.53:  39445 A?
www.looser.com. (32)
10:13:58.018699 192.168.42.78.53 > 192.55.83.30.53:  39445 [1au] A?
www.looser.com. (43)
10:13:58.249039 192.55.83.30.53 > 192.168.42.78.53:  39445 FormErr- [0q]
0/0/0 (12) (DF)
10:13:58.249153 192.168.42.78.53 > 192.55.83.30.53:  39445 A?
www.looser.com. (32)
10:14:06.018825 192.168.42.78.53 > 192.41.162.30.53:  39445 [1au] A?
www.looser.com. (43)
10:14:06.051960 192.41.162.30.53 > 192.168.42.78.53:  39445 FormErr- [0q]
0/0/0 (12) (DF)
10:14:06.052112 192.168.42.78.53 > 192.41.162.30.53:  39445 A?
www.looser.com. (32)
10:14:09.431353 192.168.42.78.53 > 192.33.14.30.53:  7462 A?
www.looser.com. (32)
10:14:09.489141 192.33.14.30.53 > 192.168.42.78.53:  7462- 0/2/2 (109) (DF)
10:14:09.489528 192.168.42.78.53 > 64.247.9.98.53:  56483 [1au] A?
www.looser.com. (43)
10:14:09.544852 64.247.9.98.53 > 192.168.42.78.53:  56483*- 1/2/1 A
217.8.158.117 (104) (DF)
10:14:14.018941 192.168.42.78.53 > 192.43.172.30.53:  39445 [1au] A?
www.looser.com. (43)
10:14:14.160251 192.43.172.30.53 > 192.168.42.78.53:  39445 FormErr- [0q]
0/0/0 (12) (DF)
10:14:14.160333 192.168.42.78.53 > 192.43.172.30.53:  39445 A?
www.looser.com. (32)
10:14:22.019082 192.168.42.78.53 > 192.54.112.30.53:  39445 [1au] A?
www.looser.com. (43)
10:14:22.147459 192.54.112.30.53 > 192.168.42.78.53:  39445 FormErr- [0q]
0/0/0 (12) (DF)
10:14:22.147543 192.168.42.78.53 > 192.54.112.30.53:  39445 A?
www.looser.com. (32)
10:14:30.019186 192.168.42.78.53 > 192.42.93.30.53:  39445 [1au] A?
www.looser.com. (43)
10:14:30.071152 192.42.93.30.53 > 192.168.42.78.53:  39445 FormErr- [0q]
0/0/0 (12) (DF)
10:14:30.071232 192.168.42.78.53 > 192.42.93.30.53:  39445 A?
www.looser.com. (32)
10:14:38.019329 192.168.42.78.53 > 192.31.80.30.53:  39445 [1au] A?
www.looser.com. (43)
10:14:38.052275 192.31.80.30.53 > 192.168.42.78.53:  39445 FormErr- [0q]
0/0/0 (12) (DF)
10:14:38.052367 192.168.42.78.53 > 192.31.80.30.53:  39445 A?
www.looser.com. (32)
10:14:46.019458 192.168.42.78.53 > 192.52.178.30.53:  39445 [1au] A?
www.looser.com. (43)
10:14:46.155902 192.52.178.30.53 > 192.168.42.78.53:  39445 FormErr- [0q]
0/0/0 (12) (DF)
10:14:46.156056 192.168.42.78.53 > 192.52.178.30.53:  39445 A?
www.looser.com. (32)
10:14:54.019582 192.168.42.78.53 > 192.12.94.30.53:  39445 [1au] A?
www.looser.com. (43)
10:14:54.061415 192.12.94.30.53 > 192.168.42.78.53:  39445 FormErr- [0q]
0/0/0 (12) (DF)
10:14:54.061511 192.168.42.78.53 > 192.12.94.30.53:  39445 A?
www.looser.com. (32)

Any ideas!?