A shell script to download a URL (and test website speed)

I’ve been having a performance problem with a GoDaddy website lately (see my GoDaddy 4GH performance problems page, and in an effort to get a better handle on both (a) GoDaddy website downtime and (b) GoDaddy 4GH performance, I wrote a Unix shell script to download a sample web page from my website using a curl command.

Solution: A website speed test shell script

To that end, I created the following shell script, and then ran it from my Mac every two minutes:

#!/bin/sh

#---------------------------------------------------------
# created by alvin alexander, http://alvinalexander.com
#---------------------------------------------------------
# released under the creative commons share-alike license:
# http://creativecommons.org/licenses/by-sa/2.5/
# link to this web page without the 'nofollow' tag if you
# copy this work.
#---------------------------------------------------------

# the output file
FILE=/Users/Al/Projects/TestWebsite/download.out

# the url to retrieve
URL=http://www.example.com/foo/bar

# write header information to the log file
start_date=`date`
echo "START-------------------------------------------------" >> $FILE
echo "" >> $FILE

# retrieve the web page using curl. time the process with the time command
time (curl --connect-timeout 100 $URL) >> $FILE 2>

# write additional footer information to the log file
echo "" >> $FILE
end_date=`date`
echo "STARTTIME: $start_date" >> $FILE
echo "END TIME:  $end_date" >> $FILE
echo "" >> $FILE

This script downloads the given web page one time. (See below for how it is run every two minutes.) The Unix time command that wraps around the curl command gives me output information like this so I can see how long the download took:

real  0m0.230s
user  0m0.020s
sys   0m0.090s

In this case the only thing I'm interested in is the 'real' output, as this refers to what I call "wall clock time". It's the same time you'd measure if you were looking at a clock on the wall. (And that clock had super-low resolution like this ... maybe I should call it "stopwatch time".) The time command's user and sys times are important in other contexts, but not here.

Running this command every two minutes

I've written before about how to run a Mac Unix command on a schedule, so I won't repeat that here. Here's a link to that Mac crontab launchctl command tutorial.

The important thing to note is that I used the Mac launchctl facility to run the shell script above every two minutes. After I set that up, I let my iMac run all night, and then checked the results this morning.

Checking the website performance results

I did a quick check on the results using this Unix command:

grep 'real' download.out | cut -c8-12 | sort -nr | more

This command can be read as:

• Use grep to get all lines from the file download.out containing the string 'real'.
• Only display columns 8-12 of that output (like '0.230').
• Sort the results numerically, in reverse (which makes the order descending).
• Pipe this output into the more command, so I can page through it.

By looking at columns 8-12 of the output I was just getting the numbers like '0.230', and skipping the leading minutes field and the ending letter 's'. I did this so I could sort the numbers using the Unix sort command. (I checked first to make sure all the minute field numbers were zero.)

As for the actual numbers, they varied anywhere from 0.4 seconds to 59 seconds, with about 20% of the requests taking 15 seconds or longer. As I’ve written before, the GoDaddy website hosting service is cheap, but you also get what you pay for.