Kedro SunPy - Writing Custom Data Set to S3

Question

I'm currently attempting to define a custom dataset to read/write .fits files to/from S3 as SunPy Maps.

The closest thing to this already in the data catalog is the pillow.ImageDataSet pillow.ImageDataSet, which supports passing a file object when loading: https://pillow.readthedocs.io/en/stable/reference/Image.html.

I'm unsure if Maps are flexible enough with inputs to justify a similar approach. My attempts so far at modifying the pillow.ImageDataSet _load method to include

smap = Map(fs_file)
return smap

results in the following error:

DataSetError: Failed while loading data from data set SunPyMapDataSet(filepath=sunspots/data/01_raw/map_sample.fits, protocol=s3, save_args={'overwrite': True}).
Invalid input: <File-like object S3FileSystem, sunspots/data/01_raw/map_sample.fits>

How might I get things working here?

Answer 1

I am unfamiliar with this SunPy library. I think your approach is correct so far.

the fs_file is a handler and you need a correct way to open this file. I think you are getting this error probably because Map(fs_file) isn't the correct way to load a file.

You should probably look for functions that load Map object from a file.