'Writing a long array into a CSV column

Currently I wish to create a CSV dataset for use later on (as memory issues prevent other things), with three columns: id, category, signal.

ID and category are both integers while signal is a float32 array with length ~10001 and shape ~(10001, 401) for most cases.

I can create a dataframe for each of the samples easily as follows:

pd.DataFrame([{'id': i, 'category': y, 'signal': signal}])

However when calling to_csv('../dataset.csv', mode='a', header=False, index=False) it produces the output:

0,7,[[ 1.24496163e-03+0.0000000e+00j  1.17749616e-04+0.0000000e+00j
   8.92068347e-05+0.0000000e+00j ... -3.55575146e-04+0.0000000e+00j
  -3.53000971e-04+0.0000000e+00j -1.35809695e-03+0.0000000e+00j]
 [-1.20925368e-03+2.0470628e-05j -5.52154524e-05-3.7754296e-05j
  -7.05739149e-05-7.2288349e-05j ...  2.85753951e-04+4.1393949e-05j
   1.60366966e-04-2.7969104e-04j  1.15900207e-03+2.3369875e-04j]
 [ 1.19788689e-03+3.3121873e-05j -6.23480264e-06+4.2733776e-05j
   4.23972924e-05+8.2590450e-05j ... -1.44452570e-04+8.9506881e-05j
  -1.42675699e-05+1.8458923e-04j -9.61950631e-04-1.2690650e-04j]
 ...
 [-1.11952168e-05+1.8826026e-06j  3.46628576e-06-2.0878474e-06j
  -1.54314409e-06-2.8386687e-06j ... -1.41124383e-06-2.3261330e-06j
  -5.40845519e-07-2.5624531e-06j  1.46134671e-05-1.2955890e-06j]
 [ 8.70186886e-06-2.8798408e-06j -3.47046830e-06+4.1321928e-06j
   1.99560964e-06-2.9683741e-07j ... -3.09421989e-06+8.9018937e-07j
  -1.19136712e-06+2.9354976e-06j -1.56483602e-05+6.8048553e-07j]
 [-6.49670437e-06+0.0000000e+00j  4.49650315e-06+0.0000000e+00j
  -2.71528188e-06+0.0000000e+00j ...  6.58240697e-06+0.0000000e+00j
   3.21866310e-06+0.0000000e+00j  1.57587510e-05+0.0000000e+00j]]

With the long array on multiple lines and most of the array "cut out". Is there potentially a solution to this?



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source