How do I run the sed command with input and output as the same file?

Question

I'm trying to do use the sed command in a shell script where I want to remove lines that read STARTremoveThisComment and lines that read removeThisCommentEND.

I'm able to do it when I copy it to a new file using

sed 's/STARTremoveThisComment//' > test

But how do I do this by using the same file as input and output?

Answer 1

You can use the flag -i for in-place editing and the -e for specifying normal script expression:

sed -i -e 's/pattern_to_search/text_to_replace/' file.txt

To delete lines that match a certain pattern you can use the simpler syntax. Notice the d flag:

sed -i '/pattern_to_search/d' file.txt

Answer 2

You really should not use sed for that. This question seems to come up ridiculously often, and it seems very strange that it does since the general solution is so trivial. It seems bizarre that people want to know how to do it in sed, and in python, and in ruby, etc. If you want to have a filter operate on an input and overwrite it, use the following simple script:

#!/bin/sh -e
in=${1?No input file specified}
mv $in ${bak=.$in.bak}
shift
"$@" < $bak > $in

Put that in your path in an executable file name inline, and then the problem is solved in general. For example:

inline input-file sed -e s/foo/bar/g

Now, if you want to add logic to keep multiple backups, or if you have some options to change the backup naming scheme, or whatever, you fix it in one place. What's the command line option to get 1-up counters on the backup file when processing a file in-place with perl? What about with ruby? Is the option different for gnu-sed? How does awk handle it? The whole friggin' point of unix is that tools do one thing only. Handling logic for backup files is a second thing, and needs to be factored out. If you are implementing a tool, do not add logic to create backup files. Tell your users to use a 2nd tool for that. Integration is bad. Modularity is good. That is the unix way.

Notice that this script has several problems. The permissions/mode of the input file may be changed, for example. I'm sure there are innumerable other issues. However, by putting the backup logic in a wrapper script, you localize all of these issues and don't have to worry that sed overwrites the files and changes mode, while python keeps the file in place and does not change the inode (I made up those two cases, the point being that not all tools will use the same logic, while the wrapper script will.)

Answer 3

As far as I know it is not possible to use the same file for input and output. Though one solution is make a shell script which will save it to another file, delete the old input and rename the output to the input file name.

sed -e s/try/this/g input.file > output.file;mv output.file input.file

Answer 4

I suggest using sponge

sponge reads standard input and writes it out to the specified file. Unlike a shell redirect, sponge soaks up all its input before writing the output file. This allows constructing pipelines that read from and write to the same file.

cat test | sed 's/STARTremoveThisComment//' | sponge test