'Read n lines at a time using Bash

I read the help read page, but still don't quite make sense. Don't know which option to use.

How can I read N lines at a time using Bash?

bash


Solution 1:[1]

With Bash?4 you can use mapfile like so:

while mapfile -t -n 10 ary && ((${#ary[@]})); do
    printf '%s\n' "${ary[@]}"
    printf -- '--- SNIP ---\n'
done < file

That's to read 10 lines at a time.

Solution 2:[2]

While the selected answer works, there is really no need for the separate file handle. Just using the read command on the original handle will function fine.

Here are two examples, one with a string, one with a file:

# Create a dummy file
echo -e "1\n2\n3\n4" > testfile.txt

# Loop through and read two lines at a time
while read -r ONE; do
    read -r TWO
    echo "ONE: $ONE TWO: $TWO"
done < testfile.txt

# Create a dummy variable
STR=$(echo -e "1\n2\n3\n4")

# Loop through and read two lines at a time
while read -r ONE; do
    read -r TWO
    echo "ONE: $ONE TWO: $TWO"
done <<< "$STR"

Running the above as a script would output (the same output for both loops):

ONE: 1 TWO: 2
ONE: 3 TWO: 4
ONE: 1 TWO: 2
ONE: 3 TWO: 4

Solution 3:[3]

Simplest method - pretty self-explanatory. It is similar to the method provided by @Fmstrat, except the second read statement is before the do.

while read first_line; read second_line
do
    echo "$first_line" "$second_line"
done

You can use this by piping multiline input to it:

seq 1 10 | while read first_line; read second_line 
do
    echo "$first_line" "$second_line"
done

output:

1 2
3 4
5 6
7 8
9 10

Solution 4:[4]

That is much more simple! :)

cat input-file.txt | xargs -L 10 ./do_something.sh

or

cat input-file.txt | xargs -L 10 echo

Solution 5:[5]

I don't think there is a way to do it natively in bash, but one can create a convenient function for doing so:

#
# Reads N lines from input, keeping further lines in the input.
#
# Arguments:
#   $1: number N of lines to read.
#
# Return code:
#   0 if at least one line was read.
#   1 if input is empty.
#
function readlines () {
    local N="$1"
    local line
    local rc="1"

    # Read at most N lines
    for i in $(seq 1 $N)
    do
        # Try reading a single line
        read line
        if [ $? -eq 0 ]
        then
            # Output line
            echo $line
            rc="0"
        else
            break
        fi
    done

    # Return 1 if no lines where read
    return $rc
}

With this one can easily loop over N-line chunks of the data by doing something like

while chunk=$(readlines 10)
do
    echo "$chunk" | ... # Whatever processing
done

In this loop $chunk will contain 10 input lines at each iteration, except for the last one, which will contain the last lines of input, which might be less than 10 but always more than 0.

Solution 6:[6]

Depending on what you're trying to do, you can just store the previous lines.

LINE_COUNT=0
PREVLINE1=""
PREVLINE2=""
while read LINE
  do LINE_COUNT=$(($LINE_COUNT+1));
    if [[ $LINE_COUNT == 3 ]]; then
       LINE_COUNT=0
       # do whatever you want to do with the 3 lines
    done
    PREVLINE2="$PREVLINE1"
    PREVLINE1="$LINE"
  done
done < $FILE_IN

Solution 7:[7]

I came up with something very similar to @albarji's answer, but more concise.

read_n() { for i in $(seq $1); do read || return; echo $REPLY; done; }

while lines="$(read_n 5)"; do
    echo "========= 5 lines below ============"
    echo "$lines"
done < input-file.txt

The read_n function will read $1 lines from stdin (use redirection to make it read from a file, just like the built-in read command). Because the exit code from read is maintained, you can use read_n in a loop as the above example demonstrates.

Solution 8:[8]

Just use a for loop:

for i in $(seq 1 $N) ; do read line ; lines+=$line$'\n' ; done

In bash version 4, you can also use the mapfile command.

Solution 9:[9]

I know you asked about bash, but I am amazed that this works with zsh

#!/usr/bin/env zsh    
cat 3-lines.txt | read -d\4 my_var my_other_var my_third_var

Unfortunately, this doesn't work with bash, at least the versions I tried.

The "magic" here is the -d\4 (this doesn't work in bash), that sets the line delimiter to be the EOT character, which will be found at the end of your cat. or any command that produces output.

If you want to read an array of N items, bash has readarray and mapfile that can read files with N lines and save every line in one position of the array.

EDIT

After some tries, I just found out that this works with bash:

$ read -d# a b
Hello
World
#
$ echo $a $b
Hello World
$

However, I could not make { cat /tmp/file ; echo '#'; } | read -d# a b to work :(

Solution 10:[10]

The echo simulates a file with two lines input, use head -2 before paste if needed:

IFS=\; read A B < <(echo -en "X1 X2\nY1 Y2\n" | paste -s -d\;)

If you want to read lines in a loop and create pairs and lines have only single word in them use:

while read NAME VALUE; do 
    echo "$NAME=$VALUE"; 
done < <(echo -en "M\n1\nN\n2\nO\n3\n" | xargs -L2 echo)

Solution 11:[11]

Here's an alternative way of doing it:

//This will open the file and users can start adding variables.
cat > file
//After finished ctrl + D will close it
cat file|while read line;
do
  //do some stuff here
done

Solution 12:[12]

Awk is a funny way for this case:

~$ cat test.txt 
tom
[email protected]
jack
[email protected]
marry
[email protected]
gogo
[email protected]
~$ cat test.txt | awk 'BEGIN{c=1}{a=c;if(a==2){print b" "$0;c=1} if(a==1){b=$0;c=2}}'
tom [email protected]
jack [email protected]
marry [email protected]
gogo [email protected]

Solution 13:[13]

to read n+2 Lines from a file:-

2

4

6

8

.

.

so on

you can try this way:-

cat fileName | awk '!((NR - 0) % 2)'

Solution 14:[14]

Also you can group lines with awk:

$ seq -s ' ' 23 > file

$ cat file
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23

$ awk '(NR % 6 == 1) {print; for(i=1; i<6 && getline ; i++) { print }; printf "\n"}' RS=' ' ORS=' ' file
1 2 3 4 5 6 
7 8 9 10 11 12 
13 14 15 16 17 18 
19 20 21 22 23

Solution 15:[15]

Another option is to use the curly brace command grouping in bash.

{ read line1; read line2; } < test-file.txt

One thing to keep in mind tho is that if you have set -u then this will fail if the file being read has less lines than the number of variables you're attempting to fill. One solution is simply to add || true to the end of the above line.

Solution 16:[16]

After having looked at all the answers, I think the following is the simplest, ie more scripters would understand it better than any other solution, but only for small number of items: 

while read -r var1 && read -r var2; do 
    echo "$var1" "$var2"
done < yourfile.txt

The multi-command approach is also excellent, but it is lesser known syntax, although still intuitive: 

while read -r var1; read -r var2; do 
    echo "$var1" "$var2"
done < yourfile.txt

It has the advantage that you don't need line continuations for larger number of items: 

while 
    read -r var1
    read -r var2
    ...
    read -r varN
do 
    echo "$var1" "$var2"
done < yourfile.txt

The xargs answer posted is also nice in theory, but in practice processing the combined lines is not so obvious. For example one solution I came up with using this technique is:

while read -r var1 var2; do 
    echo "$var1" "$var2"
done <<< $(cat yourfile.txt | xargs -L 2 )

but again this uses the lesser known <<< operator. However this approach has the advantage that if your script was initially 

while read -r var1; do 
    echo "$var1"
done <<< yourfile.txt

then extending it for multiple lines is somewhat natural:

while read -r var1 var2; do 
    echo "$var1" "$var2"
done <<< $(cat endpoints.txt | xargs -L 2 )

The straightforward solution 

while read -r var1; do
    read -r var2
    echo "$var1" "$var2"
done < yourfile.txt

is the only other one that I would consider among the many given, for its simplicity, but syntactically it is not as expressive; compared to the && version or multi-command version it does not feel as right.

Related Questions