What's an appropriate way to background process file IO operations? [closed]

Question

I am presently using a FileSystemWatcher to spot when files arrive in a folder. However, locked files are a problem and so I wanted to try a different approach but still using a FileSystemWatcher, unless of course using sleep timers and actively looking works better in practice, but would rather avoid that.

The idea I wanted to do was to capture a file name which has arrived and place it onto a list of files, or maybe create a task for it for a ThreadPool, or something. The idea being that I could ensure the processing of the files could be done decoupled from the file watcher.

The processing is basically. File arrives, read it, append to it, and save a new file out. At the same time move the file that just arrived out the way.

What's a practice method for handling this sort of thing in .net using C#?

The files are not that big and are processed using CsvHelper to extract CSV data into a List<dtoobject> then use that to create a new list of DTO objects which have an extra column, then write the new list out as a new CSV.

Most of the incoming CSV files are less than 500KB, with many being less than 100KB. The output files are not that much larger.

The software will be running on someone's laptop and processing a small amount of files at a time.

I have been exploring options, but starting to get confused, and not sure which approach would be best.

This would be running in a console app initially, but eventually I might put it in a service.

Answer 1

You're summing it pretty well.

If you want all the files processed, I would recommend a couple of things:

• do not solely rely on the File Watcher; you want to recover if it fails due to locked files you want to retry.
• do not solely rely on on-the-fly processing; your service might be down.

What you can do is the following:

• make sure you keep track of the processed files.
• add code which you can run periodically to process al the non processed files in the folder. Call this code when application starts, and on a specific interval.
• use the file-watcher as trigger to process a single file. If it fails, you can implement a retry strategy or fall back to the previous mentioned mechanism.

I recommend you to start trying to implement these. Ask a new question if you run in anything specific.