'Remove Duplicates numbers in Shell Script [closed]

I have a file called Sparrow.txt and its content is follows

ENV-NAME|ACTIVE-SITE|MCM|BDD|ELOW|OPT|MSP
======|===========|========|========|=======|=========|========
EAST| PROD | 2311| 2312 2312 | 2313 2313 | 2314 2314 | 2315 2315
EAST| STAGE | 2311 | 2312 2312 | 2313 2313 | 2314 2314 | 2315 2315

I want to remove the duplicates in the file and following should be the output

ENV-NAME|ACTIVE-SITE|MCM|BDD|ELOW|OPT|MSP
======|===========|========|========|=======|=========|========
EAST| PROD | 2311| 2312 | 2313 | 2314 | 2315 
EAST| STAGE | 2311 | 2312 | 2313 | 2314 | 2315

I have tried to sort and uniq but that didn't work

bashshell


Solution 1:[1]

If you can use perl :

perl -pe 's|\b([0-9]+) \1\b|\1|g' Sparrow.txt

Solution 2:[2]

$ cat tst.awk
BEGIN { FS=OFS="|" }
{
    for (i=1; i<=NF; i++) {
        if ( split($i,t," ") > 1 ) {
            $i = " " t[1] " "
        }
    }
    print
}


$ awk -f tst.awk file
ENV-NAME|ACTIVE-SITE|MCM|BDD|ELOW|OPT|MSP
======|===========|========|========|=======|=========|========
EAST| PROD | 2311| 2312 | 2313 | 2314 | 2315
EAST| STAGE | 2311 | 2312 | 2313 | 2314 | 2315

Solution 3:[3]

If you've got GNU Sed, try:

sed 's/\<\([[:digit:]]\+\)[[:space:]]\+\1\>/\1/g' Sparrow.txt

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source
Solution 1 Philippe
Solution 2 Ed Morton
Solution 3 pjh

Related Questions