'separate files from the csv file in python
I am trying to separate a huge csv file into no of text files. But I didn't this early by reading all the content in the file and then separate it. Now the thing is that " the csv file must be read into separate chunks (example. at a time ,only 1000 lines must be read ) and each chunk must be separated into no of text files.
def ot(q,c_size,splitcount):
global Active
print (threading.current_thread().getName()+ " starts id - {}".format(threading.get_ident()))
df1=pd.DataFrame(columns=['1 Hz','2 Hz', '3Hz', '4Hz' ,'5Hz','6Hz','7Hz'])
df1.to_csv(config['csv_file']['name'])
while (Active or not q.empty()):
try:
t2receive=q.get()
t2receive.to_csv(config['csv_file']['name'],mode='a',index=False,header=False)
except Queue.empty():
break
time.sleep(1)
This code generate csv file appending values from the queue. Now when the queue becomes empty, the csv file must be read by portion by portion(portion can be customized by the user 'chunk size c_size) and thus every portion must be spillted into no of text files .How can i process each chunks and appends the values to the same text files.
Here is the code which reads the csv file as a whole and separate according to the user input.
a=df.to_numpy(dtype='int16')
a=a.flatten()
aa=[a[np.arange(i,len(a),n)] for i in range(n)]
for i,r in enumerate(aa): np.savetxt(f'file{i+1}.txt',r,fmt='%s',newline='\n')
Consider this as csv file, 1 Hz,2 Hz,3Hz,4Hz,5Hz,6Hz,7Hz 0.1,0.2,0.3,0.4,0.5,0.6,0.7 0.8,0.9,1.0,1.1,1.2,1.3,1.4
if n =1, one text must be generated
[0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 1.1 1.2 1.3 1.4]
if n=2, two text file must be generated
file1 [0.1 0.3 0.5 0.7 0.9 1.1 1.3 ]
file2 [0.2 0.4 0.6 0.8 1.0 1.2 1.4]
if n=3 , three textfile must be generated
file1[0.1 0.4 0.7 1.0 1.3]
file2[0.2 0.5 0.8 1.1 1.4]
file3[0.3 0.6 0.9 1.2]
In the similar way , the file must be separated not as a whole but by the each chunks. For each chunks obtained the file must be separated and values must be stored to the appropriate files for all chunks of the file.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
