Why is data written to a file opened with O_APPEND flag, always written at the end, even with `lseek`?

Question

I have been given a programming assignment:

Write a program that opens an existing file for writing with the O_APPEND flag, and then seeks to the beginning of the file before writing some data. Where does the data appear in the file? Why?

What I have come up with is:

main() {
    int fd = open("test.txt", O_WRONLY | O_APPEND);
    lseek(fd, 0, SEEK_SET);
    write(fd, "abc", 3);
    close(fd);
}

Upon trying the above, I have found that the data is always written at the end of the file. Why is that? Is it because I have indicated O_APPEND?

Answer 1

In effect, O_APPEND only affects the behavior of write, but not that of read. Whatever how the current position of a file is changed by lseek, write will always append-only.

When you open a file with O_RDWR | O_APPEND, read will still start from the beginning of the file.

In the manual of open (man 2 open),

O_APPEND The file is opened in append mode. Before each write(2), the file offset is positioned at the end of the file.

In the manual of write (man 2 write),

If the O_APPEND flag of the file status flags is set, the file offset shall be set to the end of the file prior to each write.

In the Linux kernel fs/ext4 syscall write -> vfs_write -> ext4_file_write_iter, the ext4_file_write_iter will call ext4_write_checks

then call generic_write_checks

you will find the place where setting the pos = file.size

/* FIXME: this is for backwards compatibility with 2.4 */
if (iocb->ki_flags & IOCB_APPEND)
    iocb->ki_pos = i_size_read(inode);
pos = iocb->ki_pos;

The following demo can verify it.

cat open_append.cc
#include <fcntl.h>
#include <sys/stat.h>
#include <unistd.h>
#include <errno.h>
#include <string.h>

#include <string>
#include <iostream>

int main(int argc, char *argv[]) {
  std::string path = "./test.txt";
  std::string content = "hello_world";
  std::string read_buf(content.size(), 0x0);
  struct stat st_buf;
  ssize_t bytes_read = -1;
  ssize_t bytes_write = -1;
  int ret = -1;
  off_t cur_off = -1;
  int fd = ::open(path.c_str(), O_CREAT | O_RDWR | O_TRUNC, 0644);
  if (fd < 0) {
    std::cerr << "open err path " << path
              << " errno " << errno << std::endl;
    return -1;
  }
  std::cout << "open ok path " << path
            << " fd " << fd << std::endl;

  // Step 1 write some data into an empty file
  bytes_write = ::write(fd, content.data(), content.size());
  if (bytes_write < 0) {
    std::cerr << "write err fd " << fd
              << " errno " << errno << std::endl;
    goto out;
  }
  std::cout << "write ok fd " << fd
            << " data " << content
            << " nbytes " << bytes_write << std::endl;
  ::close(fd);

  // Step 2 open the file again with O_APPEND
  fd = -1;
  fd = ::open(path.c_str(), O_CREAT | O_RDWR | O_APPEND, 0644);
  if (fd < 0) {
    std::cerr << "open again err path " << path
              << " errno " << errno << std::endl;
    return -1;
  }
  std::cout << "open again ok path " << path
            << " fd " << fd << std::endl;

  // Step 3 the current position of the file NOT affected by O_APPEND
  cur_off = ::lseek(fd, 0, SEEK_CUR);
  if (cur_off < 0) {
    std::cerr << "lseek err SEEK_CUR fd " << fd
              << " errno " << errno << std::endl;
    goto out;
  }
  // cur_off expected to be 0
  std::cout << "lseek ok SEEK_CUR fd " << fd
            << " cur_off " << cur_off << std::endl;

  // Step 4  the read will start from the beginning of the file
  bytes_read = read(fd, (char*)read_buf.data(), content.size());
  if (bytes_read < 0) {
    std::cerr << "read err fd " << fd
              << " errno " << errno << std::endl;
    goto out;
  }
  std::cout << "read ok fd " << fd
            << " data " << read_buf
            << " nbytes " << bytes_read << std::endl;

  // Step 5 change the position to the half of the file size
  cur_off = ::lseek(fd, content.size() / 2, SEEK_SET);
  if (cur_off < 0) {
    std::cerr << "lseek err SEEK_SET fd " << fd
              << " errno " << errno << std::endl;
    goto out;
  }
  // cur_off expected to be content.size() / 2
  std::cout << "lseek ok SEEK_SET fd " << fd
            << " cur_off " << cur_off << std::endl;

  // Step 6 write will append data from the end of the file
  // the current position is ignored
  bytes_write = ::write(fd, content.data(), content.size());
  if (bytes_write < 0) {
    std::cerr << "append write err fd " << fd
              << " errno " << errno << std::endl;
    goto out;
  }
  std::cout << "append write ok fd " << fd
            << " append data " << content
            << " append nbytes " << bytes_write << std::endl;

  // Step 7 the file size is double content.size()
  memset((void*)&st_buf, 0x0, sizeof(struct stat));
  ret = lstat(path.c_str(), &st_buf);
  if (ret < 0) {
    std::cerr << "lstat err path " << path
              << " errno " << errno << std::endl;
    goto out;
  }
  std::cout << "lstat ok path " << path
            << " st_size " << st_buf.st_size << std::endl;
  ret = 0;

out:
  if (fd >= 0) {
    close(fd);
  }
  return ret;
}

Output result

open ok path ./test.txt fd 3
write ok fd 3 data hello_world nbytes 11
open again ok path ./test.txt fd 3
lseek ok SEEK_CUR fd 3 cur_off 0
read ok fd 3 data hello_world nbytes 11
lseek ok SEEK_SET fd 3 cur_off 5
append write ok fd 3 append data hello_world append nbytes 11
lstat ok path ./test.txt st_size 22

Answer 2

O_APPEND flag forces file pointer to point at the end of file only. so if you do lseek from start of the file, it takes the updated file pointer position as a start of file i.e. end position of old file.

Answer 3

Some luck might lead you to a phrase in the book Advanced Programming in the UNIX? Environment, specifically in Section 3.10 page 77 (3rd edition):

If a file is opened with the O_APPEND flag, a corresponding flag is set in the file status flags of the file table entry. Each time a write is performed for a file with this append flag set, the current file offset in the file table entry is first set to the current file size from the i-node table entry. This forces every write to be appended to the current end of file.

which describes why.