'Importing 'prefixed-line' csv into pandas

I have a csv formatted like below which I'd like to import into pandas. Basically, it is a bunch of measurements and data from multiple samples.

dataformat1,sample1,sample_source,start_time
dataformat2,parameter1,spec_a_1,spec_b_1,measurement1
dataformat2,parameter2,spec_a_2,spec_b_2,measurement2
dataformat2,parameter3,spec_a_3,spec_b_3,measurement3
dataformat3,result_code_for_sample1,completion_time_for_sample1
...
dataformat1,sampleN,sample_source,start_time
dataformat2,parameter1,spec_a_1,spec_b_1,measurement1
dataformat2,parameter2,spec_a_2,spec_b_2,measurement2
dataformat2,parameter3,spec_a_3,spec_b_3,measurement3
dataformat3,result_code_for_sampleN,completion_time_for_sampleN

Essentially, the first field of the csv is describing the data format of the rest of that line. In the pandas dataframe I would like to import all these into a single dataframe and fill them to relevant section. I am currently planning to do this by:

  1. prepending the line number into the csv
  2. reading each dataformat# into a separate dataframe
  3. combining, sorting, and ffill/bfill/unpivot(?) shenanigans to fill in all the data

I assume this will work, but I am wondering if there's a cleaner way to do this either within pandas or using some other library. It is a somewhat common data logging paradigm in the work I do.



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source