'Issue with Python Not Writing Lines

So I'm writing a script to take large csv files and divide them into chunks. These files each have lines formatted accordingly:

01/07/2003,1545,12.47,12.48,12.43,12.44,137423

Where the first field is the date. The next field to the right is a time value. These data points are at minute granularity. My goal is to fill files with 8 days worth of data, so I want to write all the lines from a file for 8 days worth into a new file.

Right now, I'm only seeing the program write one line per "chunk," rather than all the lines. Code shown below and screenshots included showing how the chunk directories are made and the file as well as its contents.

For reference, day 8 shown and 1559 means it stored the last line right before the mod operator became true. So I'm thinking that everything is getting overwritten somehow since only the last values are being stored.

Example of file and contents after program completion.

Directory structure w/ chunks

import os 
import time


CWD = os.getcwd()
WRITEDIR = CWD+"/Divided Data/"
if not os.path.exists(WRITEDIR):
    os.makedirs(WRITEDIR)

FILEDIR = CWD+"/SP500"
os.chdir(FILEDIR)

valid_files = []
filelist = open("filelist.txt", 'r')


for file in filelist:
    cur_file = open(file.rstrip()+".csv", 'r')
    cur_file.readline() #skip first line
    prev_day = ""

    count = 0
    chunk_count = 1

    for line in cur_file:

        day = line[3:5]

        WDIR = WRITEDIR + "Chunk"
        cur_dir = os.getcwd()


        path = WDIR + " "+ str(chunk_count)
        if not os.path.exists(path):
            os.makedirs(path)

        if(day != prev_day):
      #      print(day)
            prev_day = day
            count += 1

            #Create new directory
            if(count % 8 == 0):
                chunk_count += 1

                PATH = WDIR + " " + str(chunk_count)
                if not os.path.exists(PATH):
                    os.makedirs(PATH)


                print("Chunk count: " + str(chunk_count))
                print("Global count: " + str(count))


        temp_path = WDIR +" "+str(chunk_count)
        os.chdir(temp_path)

        fname = file.rstrip()+str(chunk_count)+".csv"
        with open(fname, 'w') as f:
            try:
                f.write(line + '\n')
            except: 
                print("Could not write to file. \n")

        os.chdir(cur_dir)

        if(chunk_count >= 406):
            continue        

cur_file.close()






#    count += 1


Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source