Java exec - execute system processes with Java ProcessBuilder and Process (part 3)

<< Back to "Java exec with ProcessBuilder and Process, part 2"

Our ThreadedStreamHandler class

Based on an old-but-good article at JavaWorld, I'm developing the following Java ThreadedStreamHandler class. (It's still a work-in-progress, but as mentioned, commands that don't have to be run using sudo seem to work just fine right now.)

As you have seen from the way this code is used, this class is used to read the system command's standard error (stderr) and standard output (stdout) streams in standard Java Thread objects. Let me share the code here, and then I'll discuss how it works.

package com.devdaily.system;

import java.io.*;

/**
 * ThreadedStreamHandler.java
 * @version 0.1
 *
 * This class is intended to be used with the SystemCommandExecutor
 * class to let users execute system commands from Java applications.
 * 
 * This class is based on work that was shared in a JavaWorld article
 * named "When System.exec() won't". That article is available at this
 * url:
 * 
 * http://www.javaworld.com/javaworld/jw-12-2000/jw-1229-traps.html
 * 
 * Copyright 2010 alvin j. alexander, devdaily.com.
 * 
 * This program is free software: you can redistribute it and/or modify
 * it under the terms of the GNU Lesser Public License as published by
 * the Free Software Foundation, either version 3 of the License, or
 * (at your option) any later version.

 * This program is distributed in the hope that it will be useful,
 * but WITHOUT ANY WARRANTY; without even the implied warranty of
 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 * GNU Lesser Public License for more details.

 * You should have received a copy of the GNU Lesser Public License
 * along with this program.  If not, see <http://www.gnu.org/licenses/>.
 * 
 * Please ee the following page for the LGPL license:
 * http://www.gnu.org/licenses/lgpl.txt
 * 
 */
class ThreadedStreamHandler extends Thread
{
  InputStream inputStream;
  String adminPassword;
  OutputStream outputStream;
  PrintWriter printWriter;
  StringBuilder outputBuffer = new StringBuilder();
  private boolean sudoIsRequested = false;
  
  /**
   * A simple constructor for when the sudo command is not necessary.
   * This constructor will just run the command you provide, without
   * running sudo before the command, and without expecting a password.
   * 
   * @param inputStream
   * @param streamType
   */
  ThreadedStreamHandler(InputStream inputStream)
  {
    this.inputStream = inputStream;
  }

  /**
   * Use this constructor when you want to invoke the 'sudo' command.
   * The outputStream must not be null. If it is, you'll regret it. :)
   * 
   * TODO this currently hangs if the admin password given for the sudo command is wrong.
   * 
   * @param inputStream
   * @param streamType
   * @param outputStream
   * @param adminPassword
   */
  ThreadedStreamHandler(InputStream inputStream, OutputStream outputStream, String adminPassword)
  {
    this.inputStream = inputStream;
    this.outputStream = outputStream;
    this.printWriter = new PrintWriter(outputStream);
    this.adminPassword = adminPassword;
    this.sudoIsRequested = true;
  }
  
  public void run()
  {
    // on mac os x 10.5.x, when i run a 'sudo' command, i need to write
    // the admin password out immediately; that's why this code is
    // here.
    if (sudoIsRequested)
    {
      //doSleep(500);
      printWriter.println(adminPassword);
      printWriter.flush();
    }

    BufferedReader bufferedReader = null;
    try
    {
      bufferedReader = new BufferedReader(new InputStreamReader(inputStream));
      String line = null;
      while ((line = bufferedReader.readLine()) != null)
      {
        outputBuffer.append(line + "\n");
      }
    }
    catch (IOException ioe)
    {
      // TODO handle this better; users won't want the code doing this
      ioe.printStackTrace();
    }
    catch (Throwable t)
    {
      // TODO handle this better; users won't want the code doing this
      t.printStackTrace();
    }
    finally
    {
      try
      {
        bufferedReader.close();
      }
      catch (IOException e)
      {
        // ignore this one
      }
    }
  }
  
  private void doSleep(long millis)
  {
    try
    {
      Thread.sleep(millis);
    }
    catch (InterruptedException e)
    {
      // ignore
    }
  }
  
  public StringBuilder getOutputBuffer()
  {
    return outputBuffer;
  }

}

As you can see, that's a lot of wrapper and error-handling code wrapped around some more basic code that looks like this:

BufferedReader bufferedReader = null;
bufferedReader = new BufferedReader(new InputStreamReader(inputStream));
String line = null;
while ((line = bufferedReader.readLine()) != null)
{
  outputBuffer.append(line + "\n");
}

Putting everything together, this code, running in a Java thread, lets your program read the stdout and stderr streams from the command you're running. As mentioned in that JavaWorld program, this is very important, because if you don't read these streams in Java Threads, your system call may block. This is eloquently stated in the Process class Javadoc this way:

Because some native platforms only provide limited buffer size for standard input and output streams, failure to promptly write the input stream or read the output stream of the subprocess may cause the subprocess to block, and even deadlock.

I've found this to be especially true when running a sudo command with this code on Mac OS X; if I don't immediately write my admin password to the process, the code will hang.

This "java exec" article and class updates

As I mentioned above, this code is a work in progress, and the API will continue to change for the foreseeable future. If you just want to run a simple Unix ls or ps command, or even a command pipeline, this code seems to work fine on my Mac OS X 10.5 systems.

But if you want to run something like a sudo command, I still need to fix a few problems, and to that end, I've even removed a special constructor from the SystemCommandExecutor class that should be used when you want to run a sudo command.

I hope to resolve the sudo "wrong password" problem shortly, but other than that, I'm not aware of any other bugs (just the TODO items shown).

If you have any comments, opinions, or suggestions on how this API should evolve, feel free to make a comment below, or send me an email using our contact form.

Download our Java exec classes

You are welcome to download these classes and use them in your own programs. Here are direct links to the three classes discussed above:

I hope this article and source code help you understand the process of executing system commands from a Java application.

 

<< Back to "Java exec with ProcessBuilder and Process, part 2"

Permalink

Thanks for all the hard work ...

I did notice that the SystemCommandExecutor file for download was missing the getStandardOutputFromCommand and getStandardErrorFromCommand mrthods were missing . Thats easily fixed .

However the inputStreamHandler.getOutputBuffer() and errorStreamHandler.getOutputBuffer() also appear to be missing .

Where should we be getting these methods ... ?

Permalink

Hi,

Really good article, but I´ve tried to run the code provided, but I´ve got the following errors:

The method executeCommand() is undefined for the type SystemCommandExecutor

The method getStandardErrorFromCommand() is undefined for the type SystemCommandExecutor

The method getStandardOutputFromCommand() is undefined for the type SystemCommandExecutor

Can you help me

Thanks in Advance

Pedro

Permalink

This code looks really useful! I have a suggestion - in the SystemCommandExecutor() constructor, you have a question about which Exception to throw. I'd suggest IllegalArgumentException.

Thanks for your work!

Permalink

The subject of the article is very interesting to me, but the example code I downloaded did not work: The class SystemCommandExecutor does not contain a methode named executeCommand, but in the class ProcessBuilderExample the method executeCommand is called. Could you supply a complete version of class SystemCommandExecutor?

More methods are missing in class SystemCommandExecutor :
getStandardOutputFromCommand()
getStandardInputFromCommand()

Kind regards, Ulrich.

Permalink

I downloaded 3 attached java files but the class SystemCommandExecutor doesn't have 2 methods getStandardOutputFromCommand and getStandardErrorFromCommand.
How can I have them? Add two methods you posted in the part2?

Yes, thank you, I did look at the Apache exec project. I noted that in the first page of this tutorial:

Java exec with ProcessBuilder and Process, Part 1

As I mention there, if you just want a tool to solve this problem, by all means go ahead and use their code.

(Sorry that I haven't linked these pages and comments together better. There is a way to do that with another Drupal module, but I haven't experimented with that module yet.)

 

 

Okay, again, sorry for the long delay here, but I believe I now have the corrected code out here.

I'm currently working on another project, but I hope to get back to this project and make improvements to this code in May and June. In the meantime, if anyone has any suggestions, such as the IllegalArgumentException suggestion from Andy Cohen above, please let me know.

Permalink

Hi Alvin,

Thanks for the very useful article.
There is one thing I haven't been able to figure out when playing with executing Unix commands: how do I get a command to use a stream that was produced by another command?

I need two versions of this, I'll illustrate with an example.
1. grep 'e' file.txt | sort
execute grep, direct the stream into sort (and no, I am not looking for a way to execute the intere pipe in one process. One command at a time.) and then print the outcome of sort.
2. grep 'e' file.txt | sed '1d' | sort
execute grep, catch the stream, on the fly removing the first line, direct the stream to sort and then print the outcome of sort.

Hope I make any sense at all... Have you any idea how to address this?

Nik

If I understand your question right, this is a fun problem that I ran into in my original "Java system command" tutorial. The short answer is that you need to run your Unix pipe command in a Unix shell, so ... if you change a line of code that looks like this:

Process p = Runtime.getRuntime().exec("ps aux | wc -l");

to something like this:

String[] cmd = { "/bin/sh", "-c", "ps aux | wc -l" };
Process p = Runtime.getRuntime().exec(cmd);

that second version will work, because it runs inside the Bourne shell.

The thing I didn't know until digging into this just now is that when you run exec, you aren't actually running your commands in a shell, you're really just running them (with no shell environment). So to use a feature like a pipe (pipeline) -- which is a shell feature -- you have to invoke a shell, and then run your commands inside that shell.

That's what I'm doing in the two lines of code above, invoking a shell, and then running the "ps auxx | wc -l" command pipeline in that shell.

I haven't adapted that example for this tutorial (I'm on the way out the door atm), but I hope that helps.

Permalink

Hi Alvin,

Sorry for the massive delay... :$ and thank you for your reply/time/solution.

I have asked a lot of people, and they all come up with the solution you showed above. For now, this is how I use it.
I do manipulations on files. The user inputs a full path with filename that will be manipulated. I first get the working directory, which is the folder the file lives in:

File workingDir = new File(file.getParent());

Then I put together the command I'd like to execute, for example:

String[] cmd = {"/bin/sh", "-c", "grep 'e' " + file + " | sed '1d' | sort > sorted.txt"};

I choose to capture the output directly from the shell command using redirection because I have no further manipulations to do, just return the location and name of the output file ("return /path/to/sorted.txt";).
This command gets executed like this:

Process p = Runtime.getRuntime().exec(cmd, null, workingDir);

This works like a charm.

However. What I was looking for is somewhat different: execute 'grep -e file.txt', catch the stream. As soon as Java gets hold of the first line of output of the grep command, manipulate it and send it to the next shell command. This way, a stream of lines coming from 'grep' will pass through some Java manipulation and then it will be passed to for example 'sort'.
So in my above example I had this command: grep 'e' file.txt | sed '1d' | sort I'd like to replace the 'sed' command with something Java.

Since I am having a lot of trouble making people understand this concept, it wouldn't surprise me if it weren't possible.

For now, I don't have the time to change my program, your solutions works fine for now. But perhaps it challenges you to find a solution, in that case, I'd love to hear it. Otherwise, don't bother ;-)

Anyways, your blog really did help me to understand how to execute a command from Java so many thanks!

Nik

Permalink

Hi.
If you are working in this proyect yet, how about stopping or killing the process if it hangs on ?

suppose you try a process that takes more than 1 minute, could I kill it if event exceed a time limit (using a thread to control or something like)??

Permalink

Hi Alvin,

Thank you for this very useful solution. I spent much time yesterday evening forcing Runtime.exec() to execute some piped commands until I found your article.

I have a little concern. Every execution of command's pipe like this on Linux, JVM 1.6:
ps -ef | grep 5645 | grep -v 'grep' throws:

java.io.InterruptedIOException
at java.io.PipedInputStream.read(PipedInputStream.java:324)
at java.lang.ProcessPipedInputStream.read(UNIXProcess.java:403)
at java.io.PipedInputStream.read(PipedInputStream.java:373)
at java.lang.ProcessInputStream.read(UNIXProcess.java:484)
at sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:452)
at sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:494)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:222)
at java.io.InputStreamReader.read(InputStreamReader.java:177)
at java.io.BufferedReader.fill(BufferedReader.java:148)
at java.io.BufferedReader.readLine(BufferedReader.java:311)
at java.io.BufferedReader.readLine(BufferedReader.java:374)
at com.kbl.unixsystems.nagiosmonitoring.agent.util.ThreadedStreamHandler.run(ThreadedStreamHandler.java:98)

The numeric result of the command was: 0

Stdout is undeterministic, sometimes it gets content sometimes not. I have solved a problem to modify command to ps -e | grep 4554 | grep -v 'grep', where output is smaller. However now I have an exception sometimes but mostly the stdout arrives to Java.

Do you have idea how to solve this problem?

Thanks in advance.
Regards

Permalink

Hi,

I'm trying to execute tcpdump with various parameters (options, filter) from a Java program. More precisely, the command I write in a shell terminal looks like:

sudo tcpdump -i ath0 -p -n -w /path/to/file%S.out -G 10 -z /path/to/script.sh [filter_elements] &

I need this command to run in background (&). In a shell , this command allows to periodically (10 seconds) generate a dump file named "file[seconds].out" that is given as the first argument to the script "script.sh", itself executed each time a rotation/period ends. And this works!

But when I try to execute it from my Java code, I face the following issues:

String cmd = "sudo tcpdump -i ath0 -p -n -w /path/to/file%S.out -G 10 -z /path/to/script.sh [filter_elements] &"

1) Process p = Runtime.getRuntime().exec(cmd);

=> returns "tcpdump: syntax error" on stdErr. I think this can be explained because '&' is considered as a tcpdump argument when shell in not invoked... Your opinion?

2) String[] shellCmd = {"/bin/sh", "-c", cmd};
Process p = Runtime.getRuntime().exec(shellCmd);

=> the process is running in background... Well, but the call to my script (given with the -z option) does not work. Does anyone have an idea of what happens? It works when ran it in a shell...

3) I also tried this:
String cmd = "sudo tcpdump -i ath0 -p -n -w /path/to/file%S.out -G 10 -z /path/to/script.sh [filter_elements]" (without the '&' at the end)
String[] shellCmd = {"/bin/sh", "-c", cmd, "&"};
Process p = Runtime.getRuntime().exec(shellCmd);

=> the process is not ran in background... so that it blocks the rest of my program as my "executeCommand(String command)" function ends with "return (p.waitFor() == 0)". Is there a case where "&" as an element of the String array is meaningfull?

Thanks a lot for your answers. It would really help me to go further on interesting things!

Ju

Permalink

Great article.

I am curerntly working with the ProcessBuilder to run a batch process for file conversion (50K records per batch) on a Windows XP machine.

Each batch process:
copies a subset of files
(image data files; ordered & renamed)
converts copied files creating new output
(using a PDF conversion utility)
deletes the copied files

(repeating until the end of the batch)

When unit testing my code and noticed when I failed to used the
waitfor() function, the batch process ending up trying to delete
my copied files before the conversion process completed. This confirms the "external" conversion process must be waited on before the delete can be carried out. The JVM just "submits" the process to the OS "shell" and continues. If subsequent processing is dependent on the submitted process you are at the mercy of the OS. I coded a waitfor() since it is a simple command line conversion utility (and it creates logs and error files during the
conversion process). Too much can happen in the OS enviroment that from the JVM you have no control. If the conversion process were to take a "long time", and I did not have any logs or error files to monitor the conversion process, I would opt to create another thread to manage/monitor the waitfor() (using sleep most likely or another timer). This way I could gracefully manage the submitted process. If it took "too long", I can end gracefully.

Cheers - Ken

Permalink

Hi Alvin,

First up, thanks for the great article, it has helped me a lot so far.

However, I am trying to build an application for my mac, and I would like to run some sudo commands from within my java code.

I know you say that the code will hang if the password is wrong, but if it is possible I would like to know how to make the code execute a sudo command..

I tried adding a constructor that sets the adminPassword, but I didn't have any results.. I got no errors when running a sudo command, but the command was never executed.

Any help is appreciated

Thanks
Chris

Okay, if everyone promises not to ask for support :) I've put the version of this code that lets you call sudo at this URL. Again, please look at the caveats in this article before trying to use that code.

Also, there are differences between these classes and the classes shared in this article, so please be careful to keep them separate, or keep a copy of each, as they aren't subsequent versions, but more like parallel versions of the same source.

The reason for my delay in answering many questions here is that this code needs a rewrite (the API essentially needs to be improved), but I don't have the time to focus on it right now. As mentioned, it will run sudo if you supply it the right password, but give it a bad password and it will hang, and I'm sure there are other bugs as well.

Er ... other than those warnings, good luck! :)

Add new comment

The content of this field is kept private and will not be shown publicly.

Anonymous format

  • Allowed HTML tags: <em> <strong> <cite> <code> <ul type> <ol start type> <li> <pre>
  • Lines and paragraphs break automatically.
By submitting this form, you accept the Mollom privacy policy.