'ParquetSharp's UTC adjustment

I am trying to use the ParquetSharp library (https://github.com/G-Research/ParquetSharp) to write some Parquet files from an SQL server database. As the original data is not adjusted to UTC, I want the output parquet file to also be "datetime64[ns]" instead of "datetime64[ns, UTC]".

I have tried changing the DateTime with argument LogicalType.Timestamp(isAdjustedToUtc: false) but this causes an exception as the original data is of type DateTime.


columns[i] = new Column(typeof(DateTime?), cdt.Schema[i].Name,LogicalType.Timestamp(isAdjustedToUtc: false, timeUnit: TimeUnit.Nanos)); 

I try to use LogicalWriterOverride as suggested in the documentation:

using (var dtWriter = groupWriter.NextColumn().LogicalWriterOverride<DateTime?>())
{
   dtWriter.WriteBatch(dtArray);
}

But a NotSupportedException gets thrown:

System.NotSupportedException: 'unsupported logical system type System.Nullable`1[System.DateTime] with logical type Timestamp(isAdjustedToUTC=false, timeUnit=nanoseconds, is_from_converted_type=false, force_set_converted_type=false)'

Edit: I managed to find somewhat of a solution, that does not conflict with the original data. In LogicalType.Timestamp(isAdjustedToUtc: false, timeUnit: TimeUnit.Nanos) I changed to TimeUnit.Micros as the DateTime type does not support Nanos precision, so then I override NextColumn().LogicalWriterOverride<DateTime?> and the UTC is removed in the generated parquet file, while the precision is in microseconds.



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source