Help scripting dns lookup using awk
Ernie Luzar
luzar722 at gmail.com
Sat Sep 16 14:24:19 UTC 2017
Jonathan McKeown wrote:
> Ernie, I've been following your questions over the last month or so. I
> think I can guess your problem domain, and I suspect if you told the
> list what you're trying to achieve you'd get much better suggested
> solutions.
>
> As it is I think you have one approach in mind, and all your questions
> relate to implementing parts of your idea.
>
> My humble apologies if I'm wrong; but please consider explaining what
> the overall problem is. (Someone mentioned x-y problems - I think you
> have one here.)
Yes all my different posts over the last month are related to a solution
I am trying to development. It all started with what looked like a very
simple request from top management. "Stop employees from using social
media from company PCs while at work" The one and only Freebsd system
is the front door to the Company LAN and wifi. All LAN devices are
WINDOW machines either cabled or wifi including hand held smart phones.
So needed a single point solution that would effect the whole digital shop.
You ask what about smart phones accessing their wireless service. In the
USA a wireless signal jammer is not legal if the people being effected
are un-aware of its existence. On being hired all employees sign a legal
contract containing security requirements and are made aware that a cell
phone wireless signal jammer is employed covering the Company estate and
that Company land lines phone service is the only allowed way for phone
contact with the public for personal and Company business.
As the result of questions posted here, I learned about online providers
of "host" lists. These lists contain '127.0.0.1 domain-name" records of
known malware sites. These "host" lists can be used on WINDOWS and Unix
flavored operating systems by populating those machines host file. This
was not a single point solution.
Along comes using dns as a single point solution. The 3 main players
being bind, unbound, and Dnsmasq which all have the function to be
populated with domain-names to be blocked at the local host level as not
found. I chose unbound, but am having problems with /etc/resolv.conf and
resolvconf not working as documented. Also could not get the built-in
local-unbound to work with any local changes. Posted questions here
which went un-answered. local-unbound and resolvconf are new and don't
have a user base yet to draw answers from. So pretty much a dead end. I
finally installed the port version of unbound and got it working.
Using the public host files and unbound became a single point solution
to provide protection LAN wide that is un-seen by the user base. No more
installing browser plug-ins that tries to do the same blocking function.
The dns solution provides protection to the LAN users from LAN machines
that may become infected. There is no absolute solution just more layers
of protection.
These public available "host" files contain a lot of un-necessary junk
that needed to be cleaned away. I wrote a .sh script to do this, but it
was very slow. Got help from this list to convert it to awk. Using the
same sample input file .sh took 7+ minutes, the awk version took 4
seconds. No brainier about which version I plan to use.
As the last step in massaging the raw "host" file content what'd to do a
dns lookup to verify those host domain-names were really good. Feeding
unbound bogus domain-names is not going to hurt anything, but just
wanted to be thorough. Again I started with a .sh script using the host
system command which is very slow, I got help here from the list to
convert it to awk and it was only a few seconds faster over all. I
changed the .sh script to use the drill system command and it ran in
half the time the host command version took.
In reply to the subject of this post, I got the following;
" Almost certainly the reason it's slow is that you are doing sequential
synchronous lookups. Switching to another language isn't going help
much. To speed it up you either need to switch to a language with a
DNS library that supports asynchronous lookups or fire-off parallel
child processes. The latter is easier."
So I posted my last reply asking;
How would I go about coding a sh script to fire-off parallel child
processes?
The only "other language" installed on my front door host is perl
because its part of the apache pkg. I don't want to install another
language just because it has a fast pre-canned dns lookup.
So if anyone knows of a perl dns lookup solution I sure would be
interested in hearing about it.
While waiting for a reply to that last question I have done more
testing. Using the drill command version of the .sh script against a
"host" file containing 409 records which is the smallest file I have,
found that 174 host names return mddomain or serverror. So it's
oblivious that all 12 host files need dns verification. Thats 900,000+
records.
If I run that .sh script against the same host file I start receiving
this console message;
Error: error sending query: Could not send or receive, because of
network error
The results indicate all the hosts were looked up. My isp provides 1gb
upload and 3gb download speeds so limited speed is not the cause of the
network error.
Does anyone have any ideas about what is going here?
More information about the freebsd-questions
mailing list