Unix/Linux: Changing many files with the same multiline string pattern

Unix/Linux tip: How to change/edit the content on thousands of files using awk and a Unix shell script.

As a note to self, today I had to delete the same lines (or line patterns) from over 10,000 files on this Unix/Linux server. Basically I was looking in each file for a specific pattern, and if that pattern was found, a ceertain number of lines should be deleted, starting at that point.

WARNING: Mass-changing thousands of files like this is dangerous. Before doing anything like this, always make a backup of your files.

I decided the best solution was to create an Awk script like this:

#!/bin/awk -f

# set a flag when the trigger line is found
    trigger_found = 1
    line_count = 0

# increment line_count and do not print if trigger_found and line_count <= 10
trigger_found && line_count <= 10 {

# reset trigger_found and line_count if trigger_found and line_count > 10
trigger_found && line_count > 10 {
    trigger_found = 0
    line_count = 0

# print all other lines

Then I put a list of all the files I wanted to modify in a file named file_list, and ran this Unix/Linux shell script:


for file in `cat file_list`
    echo "$file ..."
    awk -f delete_unwanted_lines.awk $file > TMP
    mv TMP $file