How to use awk to match all non-private range IP addresses in log file?

Question

I want to match ( and count their number of occurrence ) all the non-private range IP addresses from a mail log and I'm failing miserably.

From what I know:

• it should match 4x3 digits pattern, since it's an IP address, excluding:
  • 10\.
  • 192\.168\.
  • 172\.1[6789]\.
  • 172\.2[0-9]\.
  • 172\.3[01]\.
  • 127\.0\.0\.1

[1] example: a postfix mail log line looks like this: CONNECT from [217.113.54.21]:56590 to [192.168.1.100]:25

whereas the first match is a valid, external IP and that is what I need.

Anyone with the knowledge around?

Answer 1

This should be enough to get you going:

$ cat tst.awk
{
    while ( match($0,/\[([[:digit:]]{1,3}\.){3}[[:digit:]]{1,3}\]/) ) {
        ip = substr($0,RSTART+1,RLENGTH-2)
        $0 = substr($0,RSTART+RLENGTH)

        if ( ip ~ /^(\
10\.|\
192\.168\.|\
172\.1[6789]\.|\
172\.2[0-9]\.|\
172\.3[01]\.|\
127\.0\.0\.1)/ ) {
            invalid[ip]++
        }
        else {
            valid[ip]++
        }
    }
}

END {
    print "Valid IPs found:"
    for (ip in valid) {
        print "\t" ip, valid[ip]
    }

    print "\nInvalid IPs found:"
    for (ip in invalid) {
        print "\t" ip, invalid[ip]
    }
}

.

$ cat file
CONNECT from [217.113.54.21]:56590 to [192.168.1.100]:25
CONNECT from [217.113.54.21]:56590 to [192.168.2.99]:25

$ awk -f tst.awk file
Valid IPs found:
        217.113.54.21 2

Invalid IPs found:
        192.168.1.100 1
        192.168.2.99 1

Without some more sample input and expected output in your question, there's not much more we could do to help you at this point.

Answer 2

If I understand you correctly.

awk 'match($3, /\[([0-9.]+)\]:[0-9]+/, arr){}; arr[1] !~ /^(192.168|10|172.16|127).[.0-9]+/ {print arr[1]}' file

Answer 3

Private IP ranges are

1. 10.0.0.0 — 10.255.255.255 ~ Class A
2. 172.16.0.0 — 172.31.255.255 ~ Class B
3. 192.168.0.0 — 192.168.255.255 ~ Class C
4. 127.0.0.0/8 ~ localhost

This single liner should help

cat File | awk '!/^(192.168|10|127).[.0-9]+/' | awk '!/^(172).[.16-31]+/' > PublicIP.txt ; wc -l PublicIP.txt

This will create a file with all PublicIPs and show you how many total non-private IP are there