How to build a pipeline of external commands in Scala

This is an excerpt from the 1st Edition of the Scala Cookbook (partially modified for the internet). This is Recipe 12.14, “How to build a pipeline of external commands in Scala.”

Problem

You want to execute a series of external commands in a Scala application, redirecting the output from one command to the input of another command, i.e., you want to pipe the commands together.

Solution

Use the #| method to pipe the output from one command into the input stream of another command. When doing this, use ! at the end of the pipeline if you want the exit code of the pipeline, or !! if you want the output from the pipeline.

The !! approach is shown in the following example where the output from the ps command is piped as the input to the wc command:

import sys.process._

val numProcs = ("ps auxw" #| "wc -l").!!.trim
println(s"#procs = $numProcs")

Because the output from the ps command is piped into a line count command (wc -l), that code prints the number of processes running on a Unix system. The following command creates a list of all Java processes running on the current system:

val javaProcs = ("ps auxw" #| "grep java").!!.trim

There are other ways to write these commands, but because I usually end up trimming the result I get back from commands, I find this syntax to be the most readable approach.

Discussion

If you come from a Unix background, the #| command is easy to remember because it’s just like the Unix pipe symbol, but preceded by a # character. In fact, with the exception of the ### operator (which is used instead of the Unix ; symbol), the entire library is consistent with the equivalent Unix commands.

Note that attempting to pipe commands together inside a String and then execute them with ! won’t work:

// won't work
val result = ("ls -al | grep Foo").!!

This doesn’t work because the piping capability comes from a shell (Bourne shell, Bash, etc.), and when you run a command like this, you don’t have a shell.

To execute a series of commands in a shell, such as the Bourne shell, use a Seq with multiple parameters, like this:

val r = Seq("/bin/sh", "-c", "ls | grep .scala").!!

This approach runs the ls | grep .scala command inside a Bourne shell instance. A quick run in the REPL demonstrates this:

scala> val r = Seq("/bin/sh", "-c", "ls | grep .scala").!!
r: String =
"Bar.scala
Baz.scala
Foo.scala
"

However, note that when using !!, you’ll get the following exception if there are no .scala files in the directory:

java.lang.RuntimeException: Nonzero exit value: 1

I’ve found it best to wrap commands executed with !! in a try/catch expression.

See Also

My tutorial, “How to Execute a System Command Pipeline in Java,” discusses the need for a shell when piping commands.