awk help

[Ernie Luzar]

Andreas Perstinger wrote:
> (Sorry for messing up parts of the quoting in my former mail.)
> 
> On 2017-04-18 19:40, Andreas Perstinger wrote:
>> I think awk is the better tool for your task but you could still
>> simplify your shell script a little bit:
> 
> After hitting the send button I realized that there is a simpler 
> solution using a classical Unix pipe:
> 
> #!/bin/sh
> 
> added_date="`date +%Y%m%d`"
> 
> hits_rpt="hits_rpt"
> hits_new="hits.yes"
> hits_no="hits.no"
> truncate -s 0 $hits_rpt $hits_new $hits_no
> 
> ippool -l -d -m probing_ips > $hits_rpt 2> /dev/null
> 
> tail -n +4 $hits_rpt |          # start at 4th line
> paste - - |                     # join two consecutive lines
> sed -e 's:^ *::' -e 's:/32::' | # remove spaces at the beginning
>                                 # and "/32" suffix from IP address
> cut -w -f 2,4 |                 # extract IP and Hits from combined line
>                                 # (at 2nd and 4th field)
> while read ip hits              # read IP and Hits from each line
> do                              # and do your work
>     if [ "$hits" -gt 0 ]; then
>         echo "$added_date  ${ip};" >> $hits_new
>     fi
>     echo "$hits  ${ip};" >> $hits_no
> done
> exit 0
> 
> If the "ippool" uses tabs in the output just add "tr '\t' ' '" between 
> the "paste" and "sed" step.
> 
> Bye, Andreas

I really like this coding method. Had to change the tail command to 
bypass 3 lines and the cut commend was selecting the incorrect fields.

The following is the script I tested against live data.

#! /bin/sh

   date
   added_date="`date +%Y%m%d`"

#  hits_rpt="/root/hits.rpt"
   hits_rpt="/etc/ipf_pool_hits"
   hits_yes="/etc/ipf_pool_hits3_yes"
   hits_no="/etc/ipf_pool_hits3_no"
   truncate -s 0 $hits_yes $hits_no $hits_rpt

   ippool -l -d -m probing_ips > $hits_rpt 2> /dev/null

   tail -n +3 $hits_rpt |        # start at 4th line
   paste - - |                   # join two consecutive lines

   #sed -e 's:^ *::' -e 's:/32::' | # remove spaces at the beginning
   #                                # and "/32" suffix from IP address
   cut -w -f 3,5 |               # extract IP and Hits from combined line
                                 # (at 3rd and 5th fields)
   while read ip hits            # read IP and Hits from each line
   do                            # and do your work
     if [ "$hits" -gt 0 ]; then
         ip=${ip%%/*}            # strip off /32
         echo "$added_date  ${ip};" >> $hits_yes
     else
         echo "$hits  ${ip};" >> $hits_no
     fi
   done
   date
   exit 0

I ran this script using sed command and without it and in both cases it 
ran in one second.

I think that the paste command is where all the run elapse time saving 
is occurring. In my original script that took 5 minutes to run I had 
manually coded logic to combine the 2 lines together.

 From this experience I learned a creative use of the truncate & tail 
commands. But the real eye opener was the usage of the paste command. I 
didn't even know that command existed.

I used the date command to time the runs, problem is it only goes to a 
second. What is really needed here is a timer in hundredths of a second. 
Is there such a command that does that?

Thank you Andreas for taking the time to teach me these creative usages 
of these standard commands.