linux

Linux backups: Using find, xargs, and tar to create a huge archive

I did something wrong in a previous blog entry that led me to use the pax command to create a large backup/archive. There’s nothing wrong with using pax — other than the fact that it’s not available for Cygwin — and I really needed to created a huge archive.

What wasn’t working

In my earlier blog post I stated that something like this did not work for me when trying to create a large backup using find, xargs, and tar:

find . -type f -name "*.java" | xargs tar cvf myfile.tar

What was happening was that as xargs was managing the input to the tar command, tar kept re-writing the archive. That is, each time xargs passed a new block of input files to tar, tar perceived it as a new command, and went on to re-create the file named myfile.tar. So, instead of the huge myfile.tar that I expected, I ended up with only a few files in the archive.

A big collection of Unix/Linux 'find' command examples

Linux/Unix FAQ: Can you share some Linux find command examples?

Sure. The Unix/Linux find command is very powerful. It can search the entire filesystem to find files and directories according to the search criteria you specify. Besides using the find command to locate files, you can also execute other Linux commands (grep, mv, rm, etc.) on the files and directories you find, which makes find extremely powerful.

How to run a PHP script from a Unix command line

PHP FAQ: How can I run a PHP script from the Linux/Unix command line?

Solution: To run a PHP script from a Unix command line you can use this syntax:

php -f myscript.php
Sorting Unix 'ls' command output by filesize alvin September 28, 2019 - 8:53am

I just noticed that some of the MySQL files on this website had grown very large, so I wanted to be able to list all of the files in the MySQL data directory and sort them by filesize, with the largest files shown at the end of the listing. This ls command did the trick, resulting in the output shown in the image:

ls -Slhr

The -S option is the key, telling the ls command to sort the file listing by size. The -h option tells ls to make the output human readable, and -r tells it to reverse the output, so in this case the largest files are shown at the end of the output.

Unix/Linux: Find all files that contain multiple strings/patterns alvin September 5, 2019 - 8:36am

When using Unix or Linux, if you ever need to find all files that contain multiple strings/patterns, — such as finding all Scala files that contain 'try', 'catch', and 'finally' — this find/awk command seems to do the trick:

find . -type f -name *scala -exec awk 'BEGIN {RS=""; FS="\n"} /try/ && /catch/ && /finally/ {print FILENAME}' {} \;

As shown in the image below, all of the matching filenames are printed out. As Monk says, you’ll thank me later. :)

(I should mention that I got part of the solution from this gnu.org page.)

Update: My File Find utility

For a potentially better solution, see my File Find utility, which lets you search for multiple regex patterns in files.

Linux: Recursive file searching with grep -r (like grep + find)

Linux grep FAQ: How can I perform a recursive search with the grep command in Linux?

Solution: find + grep

For years I always used variations of the following Linux find and grep commands to recursively search subdirectories for files that match a grep pattern:

find . -type f -exec grep -l 'alvin' {} \;

This command can be read as, “Search all files in all subdirectories of the current directory for the string ‘alvin’, and print the filenames that contain this pattern.” It’s an extremely powerful approach for recursively searching files in all subdirectories that match the pattern I specify.

The Unix/Linux `htop` command alvin July 22, 2019 - 3:07pm

One new thing I learned in the last week is the `htop` command. It’s a nicely-improved version of the standard top command. This image provides some hints as to its features. If you have a Mac, you can install it with Homebrew.

How to use the Linux ‘scp’ command without a password to make remote backups

Summary: How to create a public and private key pair to use ssh and scp without using a password, which lets you automate a remote server backup process.

Over the last two years I've ended up creating a large collection of websites and web applications on a variety of Linux servers that are hosted with different companies like GoDaddy and A2 Hosting. I recently embarked on a mission to automate the backup processes for all these sites, and as a result of this effort, I thought I'd share what I've learned here.

Linux crontab examples (every X minutes or hours)

Table of Contents1 - Linux crontab: How to run a command every minute2 - Descriptions of the crontab date/time fields3 - Run a crontab command every hour4 - Run a crontab entry every day5 - Run a crontab entry every 5 minutes6 - Unix and Linux “crontab every” summary7 - Unix and Linux crontab reference information

Linux crontab FAQ: How do I schedule Unix or Linux crontab jobs to run at intervals, like “Every five minutes,” “Every ten minutes,” “Every half hour,” and so on?

Solution: I’ve posted other Unix/Linux crontab tutorials here before (How to edit your Linux crontab file, Example Linux crontab file format), but I’ve never included a tutorial that covers the “every” options, so here are some examples to demonstrate this crontab syntax.

Linux/Unix: How to copy a directory and save the date/time file information

If you need to copy a directory on Unix/Linux and want to preserve the date/time information while copying the directory and files, use the -p option to save the date/time information, and the -r option to copy the directory properly. For instance, I just used this cp command to copy a directory named OldDir to a new directory named NewDir, while retaining all of the date/time file information: