How to execute external commands and use their STDOUT in Scala

This is an excerpt from the 1st Edition of the Scala Cookbook. This is Recipe 12.12, “How to execute external commands and use their STDOUT in Scala.”

Problem

You want to run an external command and then use the standard output (STDOUT) from that process in your Scala program.

Solution

Use the !! method to execute the command and get the standard output from the resulting process as a String.

Just like the ! command in the previous recipe, you can use !! after a String to execute a command, but !! returns the STDOUT from the command rather than the exit code of the command. This returns a multiline string, which you can process in your application:

scala> import sys.process._
import sys.process._

scala> val result = "ls -al" !!
result: String =
"total 64
drwxr-xr-x  10 Al  staff   340 May 18 18:00 .
drwxr-xr-x   3 Al  staff   102 Apr  4 17:58 ..
-rw-r--r--   1 Al  staff   118 May 17 08:34 Foo.sh
-rw-r--r--   1 Al  staff  2727 May 17 08:34 Foo.sh.jar
"

scala> println(result)
total 64
drwxr-xr-x  10 Al  staff   340 May 18 18:00 .
drwxr-xr-x   3 Al  staff   102 Apr  4 17:58 ..
-rw-r--r--   1 Al  staff   118 May 17 08:34 Foo.sh
-rw-r--r--   1 Al  staff  2727 May 17 08:34 Foo.sh.jar

If you prefer, you can do the same thing with a Process or Seq instead of a String:

val result = Process("ls -al").!!
val result = Seq("ls -al").!!

As shown in the previous recipe, using a Seq is a good way to execute a system command that requires arguments:

val output = Seq("ls", "-al").!!
val output = Seq("ls", "-a", "-l").!!
val output = Seq("ls", "-a", "-l", "/tmp").!!

The first element in the Seq is the name of the command to be run, and subsequent elements are arguments to the command. The following code segment shows how to run a complex Unix find command:

val dir = "/Users/Al/tmp"
val searchTerm = "dawn"
val results = Seq("find", dir, "-type", "f", "-exec", "grep", "-il", searchTerm, "{}", ";").!!
println(results)

This code is the equivalent of running the following find command at the Unix prompt:

find /Users/Al/tmp -type f -exec grep -il dawn {} \;

If you’re not familiar with Unix commands, this command can be read as, “Search all files under the /Users/Al/tmp directory for the string dawn, ignoring case, and print the names of all files where a match is found.”

Discussion

Use the ! method to get the exit code from a process, or !! to get the standard output from a process.

Be aware that attempting to get the standard output from a command exposes you to exceptions that can occur. As a simple example, if you write the following statement to get the exit code of a command using the ! operator, even though a little extra STDERR information is printed in the REPL, out is just assigned a nonzero exit code:

scala> val out = "ls -l fred" !
ls: fred: No such file or directory
out: Int = 1

But if you attempt to get the standard output from the same command using the !! method, an exception is thrown, and out is not assigned:

scala> val out = "ls -l fred" !!
ls: fred: No such file or directory
    java.lang.RuntimeException: Nonzero exit value: 1
    many more lines of output ...

Unexpected newline characters

When running an external command, you may expect a one-line string to be returned, but you can get a newline character as well:

scala> val dir = "pwd" !!
dir: String =
"/Users/Al/Temp
"

When this happens, just trim the result:

scala> val dir = "pwd".!!.trim
dir: java.lang.String = /Users/Al/Temp

Using the lines_! method

You may want to check to see whether an executable program is available on your system.

For instance, suppose you wanted to know whether the hadoop2 executable is available on a Unix-based system. A simple way to handle this situation is to use the Unix which command with the ! method, where a nonzero exit code indicates that the command isn’t available:

scala> val executable = "which hadoop2".!
executable: Int = 1

If the value is nonzero, you know that the executable is not available on the current system. More accurately, it may be on the system, but it’s not on the PATH (or much less likely, the which command is not available).

Another way to handle this situation is to use the lines_! method. This can be used to return a Some or None, depending on whether or not the hadoop2 command is found by which. The syntax for the lines_! method is shown in this example:

val executable = "which hadoop2".lines_!.headOption

In the Scala REPL, you can see that if the executable isn’t available on the current system, this expression returns None:

scala> val executable = "which hadoop2".lines_!.headOption
executable: Option[String] = None

Conversely, if the command is found, the expression returns a Some:

scala> val executable = "which ls".lines_!.headOption
executable: Option[String] = Some(/bin/ls)

Note the call to the headOption method at the end of this pipeline. You call this method because the lines_! method returns a Stream, but you want the Option immediately.

See Table 12-1 for a description of the lines_! method.