How to use the Linux sed command to edit many files in place (and make a backup copy)

Warning: The following Unix sed commands are very powerful, so you can modify a lot of files successfully — or really screw things up — all in one command. :)

Yesterday I ran into a situation where I had to edit over 250,000 files, and with that I also thought, “I need to remember how to use the Unix/Linux sed command.” I knew what editing commands I wanted to run — a series of simple find/replace commands — but my bigger problem was how to edit that many files in place.

A quick look at the sed man page showed that I needed to use the -i argument to edit the files in place:

-i[SUFFIX], --in-place[=SUFFIX]
   edit files in place (makes backup if extension supplied)

Since I did want to make a backup of each file, I included a filename extension, so my sed command looked a little like this example:

sed -i.bak -e's/2011/2012/' $filename

Note that I used the filename extension .bak, and not just bak. (You have to include the decimal if you want it, and I wanted my files to be named something like foo.html.bak, not foo.htmlbak.)

Back to top

My full sed “edit multiple files in place” example

I just showed that example so you could see something simple. In reality what I did was to run a shell script that looked like this:

#!/bin/sh

# create 'html_files.txt' like this:
# find . -type f -name "*.html" > html_files.txt

for file in `cat html_files.txt`
do
  sed -i.bak -f my_commands.sed $file
done

Here’s what this shell script does:

  1. It reads from a file name html_files.txt.
  2. For every filename in that file, this script executes the sed command shown.
  3. Like the earlier example, this sed command makes a backup copy of each file it works on, adding the .bak extension to the filename.
  4. The sed command executes the editing commands that are contained in the file my_commands.sed.

For what I needed, the sed commands contained in the file my_commands.sed looked like this:

s|<TITLE>|<title>|
s|foo|bar|

These are just two sed swap/replace commands that I needed to run on every file. As I said, the actual editing was easy; the hard part was doing this on over 250,000 files.

Back to top

Note about sed and macOS

On macOS, if you use the -i argument, you must follow it with an extension. However, the extension can be blank, so you can do this to create files with a .bak extension:

-i.bak

or this to edit files in place, but without making a backup:

-i''

This text comes from the macOS sed man page:

-i extension
   Edit files in-place, saving backups with the specified extension.  If a zero-length extension is given,
   no backup will be saved.  It is not recommended to give a zero-length extension when in-place editing
   files, as you risk corruption or partial content in situations where disk space is exhausted, etc.
Back to top

Another sed example

Here’s another example of how to use this approach. I won’t write much about it, as I hope the comments explain the script. I will just say that I ran this on a Mac OS X system, and if you want to rename the files in place without making a backup, on OS X you need to include a blank string after the -i option, as shown:

#!/bin/sh
#
# Use this shell script to rename all occurrences of 'ExtJSLogin' in all files
# that are found by the `find` command shown in Step 1 below.
#
# To use this script:
#
# 1) create `files.txt` like this:
#
#    find . -type f -exec grep -l 'ExtJSLogin' {} \; | grep -v 'change-app-name.sh' > files.txt
#
# 2) change the NEW_APP_NAME in this script to whatever you want to name your
#    application.
#
# 3) run this script to change the application name in all the files
#

OLD_APP_NAME='ExtJSLogin'
NEW_APP_NAME='Focus'
FILELIST=files.txt

for file in `cat $FILELIST`
do
  # need the empty '' on mac osx systems
  sed -i '' "s/$OLD_APP_NAME/$NEW_APP_NAME/g" $file
done
Back to top

Summary: How to use sed to edit many files in place

I hope these examples of how to use the Unix/Linux sed command to edit files in place has been helpful. For more information, take a look at the sed man page, or leave a note in the Comments section below.

Back to top