How to continuously watch and read file asynchronously in Rust using tokio

Question

I would like to extract data continuously from some systemd log files. I have a regular expression which extracts what I'm interested in. What I'm not sure about is how to continously digest that file. I'm using tokio to keep things asynchronous.

I'm wondering if I should regularly (e.g. once per second) open that file, read into a buffer until the previously last line is matched, if I can keep that file open (I don't think this is a good idea from my Python experience, but I'm not sure if that would be ok in rust) or if there is another more elegant way to achieve that.

Thanks a lot in advance!

Answer 1

Don't read into a buffer until the previously last line is matched. Instead, keep track of your position in the file then seek to that position. You can make sure the seeked text matches (and that the seek succeeds) to be sure the file hasn't been rotated or truncated.

As for when to read, you can poll or you can use an API that notifies you of changes. I haven't used it, but I found the notify crate with a quick search. In addition to the crate documentation, the page here looks useful.

Answer 2

Based on the comment from piojo I found this a viable solution for me:

    let mut file = File::open(path).await.unwrap();
    let mut interval = time::interval(Duration::from_millis(1000));
    let mut contents = vec![];
    let mut position = 0;
    
    loop {
        contents.truncate(0);
        file.seek(SeekFrom::Start(position as u64)).await;
        position += file.read_to_end(&mut contents).await.unwrap();
        
        /// do_process(contents)
        
        interval.tick().await;
    }