'Writing a bash script to merge multiple .csv files into one and keeping the same header [closed]

I am trying to write a script to merge multiple .csv files(could be an empty file as well) into one .csv file. One way I found was using cat.

cat *.csv > outputFile.csv

This is working just fine. While merging the file I have to make sure that I am not copying any empty rows in the resultant file and for that I using sed command. The problem is, as all the csv file comes with the headers, those also get placed into the csv file. Can some please help me how to do that? I tried sort -u but not solving the problem. This is how my script file looks :

#!/bin/sh
cat *.csv | uniq > resultantFile.csv
sed -i '/,,/d' ./resultantFile.csv

Empty rows were getting copied as comma "," in them, so the last line is taking care of that.

bashcsvawksed


Solution 1:[1]

I would use awk:

awk '
    FNR == 1 && NR != 1 {next} # skip all headers except the first one
    /^,*$/ {next}              # skip all empty CSV rows
    !seen[$0]++                # print uniq records
' *.csv

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source
Solution 1

Related Questions