How to use the Linux sed command to edit many files in place (and make a backup copy)

Warning: The following Unix sed commands are very powerful, so you can modify a lot of files successfully — or really screw things up — all in one command. :)

Yesterday I ran into a situation where I had to edit over 250,000 files, and with that I also thought, “I need to remember how to use the Unix/Linux sed command.” I knew what editing commands I wanted to run — a series of simple find/replace commands — but my bigger problem was how to edit that many files in place.

A quick look at the sed man page showed that I needed to use the -i argument to edit the files in place:

-i[SUFFIX], --in-place[=SUFFIX]
   edit files in place (makes backup if extension supplied)

Since I did want to make a backup of each file, I included a filename extension, so my sed command looked a little like this example:

sed -i.bak -e's/2011/2012/' $filename

Note that I used the filename extension .bak, and not just bak. (You have to include the decimal if you want it, and I wanted my files to be named something like foo.html.bak, not foo.htmlbak.)

My full sed “edit multiple files in place” example

I just showed that example so you could see something simple. In reality what I did was to run a shell script that looked like this:

#!/bin/sh

# create 'html_files.txt' like this:
# find . -type f -name "*.html" > html_files.txt

for file in `cat html_files.txt`
do
  sed -i.bak -f my_commands.sed $file
done

Here’s what this shell script does:

1. It reads from a file name html_files.txt.
2. For every filename in that file, this script executes the sed command shown.
3. Like the earlier example, this sed command makes a backup copy of each file it works on, adding the .bak extension to the filename.
4. The sed command executes the editing commands that are contained in the file my_commands.sed.

For what I needed, the sed commands contained in the file my_commands.sed looked like this:

s|<TITLE>|<title>|
s|foo|bar|

These are just two sed swap/replace commands that I needed to run on every file. As I said, the actual editing was easy; the hard part was doing this on over 250,000 files.

Note about sed and macOS

On macOS, if you use the -i argument, you must follow it with an extension. However, the extension can be blank, so you can do this to create files with a .bak extension:

-i.bak

or this to edit files in place, but without making a backup:

-i''

This text comes from the macOS sed man page:

-i extension
   Edit files in-place, saving backups with the specified extension.  If a zero-length extension is given,
   no backup will be saved.  It is not recommended to give a zero-length extension when in-place editing
   files, as you risk corruption or partial content in situations where disk space is exhausted, etc.

Another sed example

Here’s another example of how to use this approach. I won’t write much about it, as I hope the comments explain the script. I will just say that I ran this on a Mac OS X system, and if you want to rename the files in place without making a backup, on OS X you need to include a blank string after the -i option, as shown:

#!/bin/sh
#
# Use this shell script to rename all occurrences of 'ExtJSLogin' in all files
# that are found by the `find` command shown in Step 1 below.
#
# To use this script:
#
# 1) create `files.txt` like this:
#
#    find . -type f -exec grep -l 'ExtJSLogin' {} \; | grep -v 'change-app-name.sh' > files.txt
#
# 2) change the NEW_APP_NAME in this script to whatever you want to name your
#    application.
#
# 3) run this script to change the application name in all the files
#

OLD_APP_NAME='ExtJSLogin'
NEW_APP_NAME='Focus'
FILELIST=files.txt

for file in `cat $FILELIST`
do
  # need the empty '' on mac osx systems
  sed -i '' "s/$OLD_APP_NAME/$NEW_APP_NAME/g" $file
done

Summary: How to use sed to edit many files in place

I hope these examples of how to use the Unix/Linux sed command to edit files in place has been helpful. For more information, take a look at the sed man page, or leave a note in the Comments section below.