Help scripting dns lookup using awk

[Edgar Pettijohn]

On Sat, Sep 16, 2017 at 10:24:16AM -0400, Ernie Luzar wrote:
> Jonathan McKeown wrote:
> > Ernie, I've been following your questions over the last month or so. I 
> > think I can guess your problem domain, and I suspect if you told the 
> > list what you're trying to achieve you'd get much better suggested 
> > solutions.
> > 
> > As it is I think you have one approach in mind, and all your questions 
> > relate to implementing parts of your idea.
> > 
> > My humble apologies if I'm wrong; but please consider explaining what 
> > the overall problem is. (Someone mentioned x-y problems - I think you 
> > have one here.) 
> 
> Yes all my different posts over the last month are related to a solution 
> I am trying to development. It all started with what looked like a very 
> simple request from top management. "Stop employees from using social 
> media from company PCs while at work"  The one and only Freebsd system 
> is the front door to the Company LAN and wifi. All LAN devices are 
> WINDOW machines either cabled or wifi including hand held smart phones. 
> So needed a single point solution that would effect the whole digital shop.
> 
> You ask what about smart phones accessing their wireless service. In the 
> USA a wireless signal jammer is not legal if the people being effected 
> are un-aware of its existence. On being hired all employees sign a legal 
> contract containing security requirements and are made aware that a cell 
> phone wireless signal jammer is employed covering the Company estate and 
> that Company land lines phone service is the only allowed way for phone 
> contact with the public for personal and Company business.
> 
> As the result of questions posted here, I learned about online providers 
> of "host" lists. These lists contain '127.0.0.1 domain-name" records of 
> known malware sites. These "host" lists can be used on WINDOWS and Unix 
> flavored operating systems by populating those machines host file. This 
> was not a single point solution.
> 
> Along comes using dns as a single point solution. The 3 main players 
> being bind, unbound, and Dnsmasq which all have the function to be 
> populated with domain-names to be blocked at the local host level as not 
> found. I chose unbound, but am having problems with /etc/resolv.conf and 
> resolvconf not working as documented. Also could not get the built-in 
> local-unbound to work with any local changes. Posted questions here 
> which went un-answered. local-unbound and resolvconf are new and don't 
> have a user base yet to draw answers from. So pretty much a dead end. I 
> finally installed the port version of unbound and got it working.
> 
> Using the public host files and unbound became a single point solution 
> to provide protection LAN wide that is un-seen by the user base. No more 
> installing browser plug-ins that tries to do the same blocking function. 
> The dns solution provides protection to the LAN users from LAN machines 
> that may become infected. There is no absolute solution just more layers 
> of protection.
> 
> These public available "host" files contain a lot of un-necessary junk 
> that needed to be cleaned away. I wrote a .sh script to do this, but it 
> was very slow. Got help from this list to convert it to awk. Using the 
> same sample input file .sh took 7+ minutes, the awk version took 4 
> seconds. No brainier about which version I plan to use.
> 
> As the last step in massaging the raw "host" file content what'd to do a 
> dns lookup to verify those host domain-names were really good. Feeding 
> unbound bogus domain-names is not going to hurt anything, but just 
> wanted to be thorough. Again I started with a .sh script using the host 
> system command which is very slow, I got help here from the list to 
> convert it to awk and it was only a few seconds faster over all. I 
> changed the .sh script to use the drill system command and it ran in 
> half the time the host command version took.
> 
> In reply to the subject of this post, I got the following;
> 
> " Almost certainly the reason it's slow is that you are doing sequential
>    synchronous lookups. Switching  to another language isn't going help
>    much. To speed it up you either need to switch to a language with a
>    DNS library that supports asynchronous lookups or fire-off parallel
>    child processes. The latter is easier."
> 
> So I posted my last reply asking;
> 
> How would I go about coding a sh script to fire-off parallel child 
> processes?
> 
> The only "other language" installed on my front door host is perl 
> because its part of the apache pkg. I don't want to install another 
> language just because it has a fast pre-canned dns lookup.
> 
> So if anyone knows of a perl dns lookup solution I sure would be 
> interested in hearing about it.
> 
> While waiting for a reply to that last question I have done more 
> testing. Using the drill command version of the .sh script against a 
> "host" file containing 409 records which is the smallest file I have, 
> found that 174 host names return mddomain or serverror. So it's 
> oblivious that all 12 host files need dns verification. Thats 900,000+ 
> records.
> 
> If I run that .sh script against the same host file I start receiving 
> this console message;
> 
> Error: error sending query: Could not send or receive, because of 
> network error

Looking at the source for drill. I believe this may be a generic error for
the sending, or receiving of the dns packet. Perhaps a snippet of the script
you are running showing how you are using drill may shed some light on it.

> 
> The results indicate all the hosts were looked up. My isp provides 1gb 
> upload and 3gb download speeds so limited speed is not the cause of the 
> network error.
> 
> Does anyone have any ideas about what is going here?
> 
> 
> 
> 
> 
> 
> 
> _______________________________________________
> freebsd-questions at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-questions
> To unsubscribe, send any mail to "freebsd-questions-unsubscribe at freebsd.org"