csv_reader read N lines at a time

Question

I have to read a CSV file N lines at a time.

csv_reader = csv.reader(csv_file, delimiter=',')
    line_count = 0
    for row in csv_reader:
        print row

I know I can loop N times at a time, build a list of list and process that way.

But is there a simpler way of using csv_reader so that I read n lines at a time.

Answer 1

Hi I don't think that you'll be able to do that without a loop with csv package.

You should use pandas (pip install --user pandas) instead:

import pandas

df = pandas.read_csv('myfile.csv')

start = 0
step = 2 # Your 'N'

for i in range(0, len(df), step):
    print(df[i:i+step])
    start = i

Answer 2

Pandas has a chunksize option to their read_csv() method and I would probably explore that option.

If I was going to do it myself by hand, I would probably do something like:

import csv

def process_batch(rows):
    print(rows)

def get_batch(reader, batch_size):
    return [row for _ in range(batch_size) if (row:=next(reader, None))]

with open("data.csv", "r") as file_in:
    reader = csv.reader(file_in)
    while batch := get_batch(reader, 5):
        process_batch(batch)