Jump to content


Check out our Community Blogs

Register and join over 40,000 other developers!


Recent Status Updates

View All Updates

Photo
- - - - -

Execute a Console Program and Capture Its Output

console program capture console output createprocess execute other program shell program console

  • Please log in to reply
7 replies to this topic

#1 Luthfi

Luthfi

    CC Leader

  • Expert Member
  • PipPipPipPipPipPipPip
  • 1320 posts
  • Programming Language:PHP, Delphi/Object Pascal, Pascal, Transact-SQL
  • Learning:C, Java, PHP

Posted 16 October 2012 - 08:52 PM

Overview

There are times where what we want to do in our own code is already perfectly done by another program. If this another program is a console program then you are in luck. Because you can easily execute that program within your own code and then capture its output. Using the captured output, you can analyze whether the wanted operation has been done successfully. You are luckier if that another program is small and efficient in doing its job.

This tutorial will show you how to do such thing through a Delphi program. However I believe you can also use this program using FreePascal/Lazarus with no (or very small) adjustment.


Some Theory

Any console program will have three standard channels for communication with their "executor". The "executor" can be the shell (e.g. DOS prompt) or another program (like we are about to do). They are:

  • StdIn, or standard input. This is where data (usually text) goes into the console program.
  • StdOut, or standard output. This is where the console program writes its output data. Again the data usually in the form of text.
  • StdErr, or standard error. This is where the console program send its error or diagnostic messages.

Wikipedia has good article about these standard channels. Check it out here.

It is the job of the executor of a console program to assign proper channels for these three, in order to be able to execute/use the program properly. When the shell is acting as the executor, it usually assign keyboard stream for StdIn, and screen (in text mode) for StdOut and StdErr. Another program can assign different channels for them to suit its mission.


Windows Communication Pipe

In Windows environment, we usually use communication pipes to redirect stream of data of StdIn, StdOut, and StdErr. Check this msdn page to learn more about pipes in Windows (not those plumbing materials, lol).

There are two types of pipes In Windows. Named pipes and unnamed/anonymous pipes. The main difference of these two is that a single instance named pipe can be "created" and accessed from several processes. This made communication between several processes through pipe easier, since there is this name constant that guarantees access to the same pipe over and over again. Of course this difference is not the only one. But discussing them is not within the scope of this writing.

For our task, anonymous pipe would be enough.

To create anonymous pipe we use Windows API CreatePipe. Look for its official information in this msdn page.

BOOL WINAPI CreatePipe(
  _Out_     PHANDLE hReadPipe,
  _Out_     PHANDLE hWritePipe,
  _In_opt_  LPSECURITY_ATTRIBUTES lpPipeAttributes,
  _In_      DWORD nSize
);


From the provided information, we know that we need to have two variables with type of Handle or THandle (one to hold the pipe's read handle, and the other to hold the write handle), a record containing security attributes for the pipe, and a buffer size recommendation. We can simply pass 0 for buffer size recommendation. This will have the system assign default buffer size for the created size.

After its creation, the pipe's handle can be used like a file. If we want to read from the pipe, we need to use Windows API ReadFile by passing the pipe's read handle. Logically, to write to the pipe we use Windows API WriteFile by using the pipe's write handle.


Pipes to Standard Channels Assignment

Initially you might wonder when do we assign anonymous pipes we created to a console program. Of course we need to assign them when we start executing the console program. That means we are interested with Windows API CreateProcess.

BOOL WINAPI CreateProcess(
  _In_opt_     LPCTSTR lpApplicationName,
  _Inout_opt_  LPTSTR lpCommandLine,
  _In_opt_     LPSECURITY_ATTRIBUTES lpProcessAttributes,
  _In_opt_     LPSECURITY_ATTRIBUTES lpThreadAttributes,
  _In_         BOOL bInheritHandles,
  _In_         DWORD dwCreationFlags,
  _In_opt_     LPVOID lpEnvironment,
  _In_opt_     LPCTSTR lpCurrentDirectory,
  _In_         LPSTARTUPINFO lpStartupInfo,
  _Out_        LPPROCESS_INFORMATION lpProcessInformation
);


The lpStartupInfo looks interesting. Its explanation says:

lpStartupInfo [in]

A pointer to a STARTUPINFO or STARTUPINFOEX structure.

To set extended attributes, use a STARTUPINFOEX structure and specify EXTENDED_STARTUPINFO_PRESENT in the dwCreationFlags parameter.

Handles in STARTUPINFO or STARTUPINFOEX must be closed with CloseHandle when they are no longer needed.

Important The caller is responsible for ensuring that the standard handle fields in STARTUPINFO contain valid handle values. These fields are copied unchanged to the child process without validation, even when the dwFlags member specifies STARTF_USESTDHANDLES. Incorrect values can cause the child process to misbehave or crash. Use the Application Verifier runtime verification tool to detect invalid handles.


The last paragraph is very interesting. If you dig more information about the record structure, you will get:

typedef struct _STARTUPINFO {
  DWORD  cb;
  LPTSTR lpReserved;
  LPTSTR lpDesktop;
  LPTSTR lpTitle;
  DWORD  dwX;
  DWORD  dwY;
  DWORD  dwXSize;
  DWORD  dwYSize;
  DWORD  dwXCountChars;
  DWORD  dwYCountChars;
  DWORD  dwFillAttribute;
  DWORD  dwFlags;
  WORD   wShowWindow;
  WORD   cbReserved2;
  LPBYTE lpReserved2;
  HANDLE hStdInput;
  HANDLE hStdOutput;
  HANDLE hStdError;
} STARTUPINFO, *LPSTARTUPINFO;


If you inspect carefully the last three members of the structure, you will find them very familiar. Yes, their purpose is to pass the standard communication channels from the process creator to the created process. For us, we need to assign proper pipes' handles to them. But remember the direction. StdIn is channel to send/write data to a console program. Therefore for StdIn of the console process we have to assign read handle of a pipe. StdOut and StdErr on the other hand is used to read output ("normal" output in the case of StdOut, and error messages in the case of StdErr) from a console program. Therefore for these two we need to assign write handle(s) of communication pipe(s) to the console process.


Coding

Let's start writing codes! Now we know that we need the following steps:

  • Create anonymous pipe(s)
  • Assign the pipe(s) to the console process
  • Create the process (execute the console program)
  • Wait until the console process terminated
  • Read from standard output pipe

To make our code a little bit easier to read and maintain, we create new record type for holding an anonymous pipe information. Like below.

type
  TAnoPipe=record
    Input : THandle; // Handle to send data to the pipe
    Output: THandle; // Handle to read data from the pipe
  end;

Please pay special attention of the Input and Output purposes. They were named from the perspective of the pipe. Input is the handle you need to send data to the pipe. Use it when writing data using WriteFile. When you need to read data from the pipe, read from the pipe's Output. So pass handle stored in Output when you are using ReadFile.

Let's implement the collected information into a function named ExecAndCapture that returns integer value. This function will take two parameters, ACmdLine, and AOutput. Both with type of string. ACmdLine is where you pass the console program that you want to be executed. AOutput is where the function pass the output of the executed console program. The integer value returned by ExecAndCapture indicates how many bytes returned by the executed console program (in the AOutput).


Creating the Anonymous Pipes

We need at least two anonymous pipes. One for standard input, and another one for standard output. So we add two TAnoPipe variables, like this.

function ExecAndCapture(const ACmdLine: string; var AOutput: string): Integer;
var
  vStdInPipe : TAnoPipe;  // pipe for standard input
  vStdOutPipe: TAnoPipe;  // pipe for standard output

Now for the pipes' creation, and cleaning, codes. Note the use of try..finally block in order to make sure the cleaning codes will be reached no matter what happened after successful creation.

begin
  ...
  ...
  with vSecurityAttributes do
  begin
    nlength := SizeOf(TSecurityAttributes);
    binherithandle := True;
    lpsecuritydescriptor := nil;
  end;

  // Create anonymous pipe for standard input
  if not CreatePipe(vStdInPipe.Output, vStdInPipe.Input, @vSecurityAttributes, 0) then
    raise Exception.Create('Failed to create pipe for standard input. System error message: ' + SysErrorMessage(GetLastError));

  try
    // Create anonymous pipe for standard output (and also for standard error)
    if not CreatePipe(vStdOutPipe.Output, vStdOutPipe.Input, @vSecurityAttributes, 0) then
      raise Exception.Create('Failed to create pipe for standard output. System error message: ' + SysErrorMessage(GetLastError));

    try
      ...
      ...
    finally
      CloseHandle(vStdOutPipe.Input);
      CloseHandle(vStdOutPipe.Output);
    end;

  finally
    CloseHandle(vStdInPipe.Input);
    CloseHandle(vStdInPipe.Output);
  end;
  ...
  ...
end;


Assign the Pipe(s) to the Console Process

We assign the proper handles of the pipes through startup information that later will be used when creating the console process. So, declare a TStartUpInfo variable, and initialize like below.

  ...
  ...

        // initialize the startup info to match our purpose
        FillChar(vStartupInfo, Sizeof(TStartUpInfo), #0);
        vStartupInfo.cb         := SizeOf(TStartUpInfo);
        vStartupInfo.wShowWindow:= SW_HIDE;  // we don't want to show the process
        // assign our pipe for the process' standard input
        [B]vStartupInfo.hStdInput  := vStdInPipe.Output;[/B]
        // assign our pipe for the process' standard output
        [B]vStartupInfo.hStdOutput := vStdOutPipe.Input;[/B]
        vStartupInfo.dwFlags    := STARTF_USESTDHANDLES or STARTF_USESHOWWINDOW;

  ...
  ...


Create the Console Process (Execute the Console Program)

Now that we have our standard channels redirections ready, it's time to execute the console program. But first I have to remind you that before capturing the output, we have to wait for the console program termination to make sure that it completed its job. This is why we have to save the created process information since it contains a handle that we can "wait" for. Using one of the "waiting" Windows API routine, we can wait until the created process terminated.

So, we need a variable with the type of TProcessInformation. And here is the codes to create the process.

var
  ...
  vProcessInfo: TProcessInformation;
  ...
begin
  ...
  ...
        if not CreateProcess(nil
                             , PChar(ACmdLine)
                             , @vSecurityAttributes
                             , @vSecurityAttributes
                             , True
                             , NORMAL_PRIORITY_CLASS
                             , nil
                             , nil
                             , vStartupInfo
                             , vProcessInfo) then
          raise Exception.Create('Failed creating the console process. System error msg: ' + SysErrorMessage(GetLastError));
  ...
  ...
end;

Click to more about CreateProcess in MSDN.


Wait Until the Console Process Terminated

Now let's give the console program enough time to complete its task. We do this by waiting until it got terminated. This can be accomplished by "waiting" for the process handle in the process information returned by previous CreateProcess. The code would be something like this.

          // wait until the console program terminated
          while WaitForSingleObject(vProcessInfo.hProcess, 50)=WAIT_TIMEOUT do
            Sleep(0);

Click to read more about WaitForSingleObject on MSDN.


Read From Standard Output Pipe

Since the console program had terminated properly, we know that it had completed its task. Time to get the result by reading its output. Let's do ReadFile using the "output end" of our pipe.

          // clear the output storage  
          AOutput := '';
          // Read text returned by the console program in its StdOut channel
          repeat
            ReadFile(vStdOutPipe.Output, vBuffer^, cBufferSize, vReadBytes, nil);
            if vReadBytes > 0 then
            begin
              AOutput := AOutput + StrPas(vBuffer);
              Inc(Result, vReadBytes);
            end;
          until (vReadBytes < cBufferSize);

Note that we read the pipe's output in a loop. We keep reading while the pipe still spitting data as big as the buffer size. Because that condition indicates very high possibility of data left in the pipe. We exit the loop after the pipe gave data less that the buffer size.


Full Source Codes of ExecAndCapture

And here is the full source code of ExecAndCapture function. Now that you get the whole picture, I hope you can get better understanding.

function ExecAndCapture(const ACmdLine: string; var AOutput: string): Integer;
const
  cBufferSize = 2048;
var
  vBuffer: Pointer;
  vStartupInfo: TStartUpInfo;
  vSecurityAttributes: TSecurityAttributes;
  vReadBytes: DWord;
  vProcessInfo: TProcessInformation;
  vStdInPipe : TAnoPipe;
  vStdOutPipe: TAnoPipe;
begin
  Result := 0;

  with vSecurityAttributes do
  begin
    nlength := SizeOf(TSecurityAttributes);
    binherithandle := True;
    lpsecuritydescriptor := nil;
  end;

  // Create anonymous pipe for standard input
  if not CreatePipe(vStdInPipe.Output, vStdInPipe.Input, @vSecurityAttributes, 0) then
    raise Exception.Create('Failed to create pipe for standard input. System error message: ' + SysErrorMessage(GetLastError));

  try
    // Create anonymous pipe for standard output (and also for standard error)
    if not CreatePipe(vStdOutPipe.Output, vStdOutPipe.Input, @vSecurityAttributes, 0) then
      raise Exception.Create('Failed to create pipe for standard output. System error message: ' + SysErrorMessage(GetLastError));

    try
      GetMem(vBuffer, cBufferSize);
      try
        // initialize the startup info to match our purpose
        FillChar(vStartupInfo, Sizeof(TStartUpInfo), #0);
        vStartupInfo.cb         := SizeOf(TStartUpInfo);
        vStartupInfo.wShowWindow:= SW_HIDE;  // we don't want to show the process
        // assign our pipe for the process' standard input
        vStartupInfo.hStdInput  := vStdInPipe.Output;
        // assign our pipe for the process' standard output
        vStartupInfo.hStdOutput := vStdOutPipe.Input;
        vStartupInfo.dwFlags    := STARTF_USESTDHANDLES or STARTF_USESHOWWINDOW;

        if not CreateProcess(nil
                             , PChar(ACmdLine)
                             , @vSecurityAttributes
                             , @vSecurityAttributes
                             , True
                             , NORMAL_PRIORITY_CLASS
                             , nil
                             , nil
                             , vStartupInfo
                             , vProcessInfo) then
          raise Exception.Create('Failed creating the console process. System error msg: ' + SysErrorMessage(GetLastError));

        try
          // wait until the console program terminated
          while WaitForSingleObject(vProcessInfo.hProcess, 50)=WAIT_TIMEOUT do
            Sleep(0);

          // clear the output storage  
          AOutput := '';
          // Read text returned by the console program in its StdOut channel
          repeat
            ReadFile(vStdOutPipe.Output, vBuffer^, cBufferSize, vReadBytes, nil);
            if vReadBytes > 0 then
            begin
              AOutput := AOutput + StrPas(vBuffer);
              Inc(Result, vReadBytes);
            end;
          until (vReadBytes < cBufferSize);
        finally
          CloseHandle(vProcessInfo.hProcess);
          CloseHandle(vProcessInfo.hThread);
        end;
      finally
        FreeMem(vBuffer);
      end;
    finally
      CloseHandle(vStdOutPipe.Input);
      CloseHandle(vStdOutPipe.Output);
    end;
  finally
    CloseHandle(vStdInPipe.Input);
    CloseHandle(vStdInPipe.Output);
  end;
end;


That's it! Feel free to use and perhaps improve the ExecAndCapture routine. I have created a simple demo project to show how it works. The download link is at the end of this article.

Image below showed when the demo project used ExecAndCapture to execute dos command "IPConfig /all".

RunTime001_ExecuteIPConfig-All.png


Attached File  RunConsoleProgAndCapture.zip   254.41KB   3137 downloads
  • 1

#2 MichaelLeahy

MichaelLeahy

    CC Lurker

  • Just Joined
  • Pip
  • 1 posts

Posted 28 June 2013 - 08:39 AM

I am porting code to Delphi XE4 to take advantage of one codebase for Win and Mac. Does this approach to spawning and conversing with a console app work the same way on Macintosh? If not, is a similar approach using Delphi for Mac documented somewhere?

 

Thanks!

 

 

Mike


  • 0

#3 Luthfi

Luthfi

    CC Leader

  • Expert Member
  • PipPipPipPipPipPipPip
  • 1320 posts
  • Programming Language:PHP, Delphi/Object Pascal, Pascal, Transact-SQL
  • Learning:C, Java, PHP

Posted 30 June 2013 - 06:56 AM

Hi Michael,

Unfortunately codes used in this tutorial intimately use Windows API. Therefore they will not work in Mac environment. However the StdIn and StdOut features are kind of "standard" in general purpose operating system, such as iOS. Therefore similar API must be available in iOS. You just need to abstract the features inside a class and in implementation you can use compiler directives to detect what environment the codes are compiled in and only use matching codes. Then you will have multiplatform library for this.
  • 0

#4 AlexandreAlexandre

AlexandreAlexandre

    CC Lurker

  • Just Joined
  • Pip
  • 1 posts

Posted 26 December 2013 - 07:17 AM

Hi, 

With this example i can't pass a command or parameters when the application is running .. for example if i have a application where i have to execute and it will terminate only when i will pass the parameter q .. the application will be blocked 

 

ipconfig.exe is a application that will work without problem beaucause it will terminate automaticly after run 

 

example when it will blocked:

execute telnet.exe host ( here i have to pass some parameter and when finish i have to pass exit)

how can i do that  with delphi

thx


  • 0

#5 BlackRabbit

BlackRabbit

    CodeCall Legend

  • Expert Member
  • PipPipPipPipPipPipPipPip
  • 3871 posts
  • Location:Argentina
  • Programming Language:C, C++, C#, PHP, JavaScript, Transact-SQL, Bash, Others
  • Learning:Java, Others

Posted 26 December 2013 - 07:05 PM

Your problem has nothing to do with the tutorial, passing parameters means to give them before execution, and that's it.

 

Your real question is how to execute a program in either attended and unattended way. Which means, should my program wait for the executed program to finish, yes or not


  • 0

#6 Luthfi

Luthfi

    CC Leader

  • Expert Member
  • PipPipPipPipPipPipPip
  • 1320 posts
  • Programming Language:PHP, Delphi/Object Pascal, Pascal, Transact-SQL
  • Learning:C, Java, PHP

Posted 28 December 2013 - 05:42 AM

Hi, 
With this example i can't pass a command or parameters when the application is running .. for example if i have a application where i have to execute and it will terminate only when i will pass the parameter q .. the application will be blocked 
 
ipconfig.exe is a application that will work without problem beaucause it will terminate automaticly after run 
 
example when it will blocked:
execute telnet.exe host ( here i have to pass some parameter and when finish i have to pass exit)
how can i do that  with delphi
thx

Yes, you are correct. The tutorial does not address how to manipulate active session. However the answer to your problem is already explained (albeit just a little). You can use the StdIn pipe to send command to your console program, and read from StdOut pipe to get its response. Also keep an eye to content of StdErr pipe to detect error.

Although it might sound complicated, but actually it is not. Although you might want to use separate thread for the manipulation to avoid blocking of the main application.

What you want to watch out for is that some console program write directly to the "screen" using coordinates, making the output text hard to understand.

EDIT:

Check out this tutorial: http://forum.codecal...ipe-into-class/ for pointers of how to easily manipulate windows anonymous pipes like used in this tutorial. With the class explained there, it's easy to create, read, and write windows anonymous pipes.

Edited by Luthfi, 28 December 2013 - 05:48 AM.

  • 0

#7 RamiroCruzo

RamiroCruzo

    CC Lurker

  • Just Joined
  • Pip
  • 2 posts
  • Programming Language:Delphi/Object Pascal
  • Learning:C, C++, (Visual) Basic

Posted 19 June 2016 - 11:23 PM

Well, here's a better function, atleast a lot less scary than the one you posted Sir,

procedure ExecAndWait(sExe, sCommandLine, sWorkDir: string;
  wait, Freeze: Boolean; Log: TStringList = nil);
  function GetModuleName: string;
  var
    szFileName: array [0 .. MAX_PATH] of char;
  begin
    FillChar(szFileName, sizeof(szFileName), #0);
    GetModuleFileName(hInstance, szFileName, MAX_PATH);
    Result := szFileName;
  end;

const
  ReadBuffer = 65535;
var
  StartupInfo: TStartUpInfo;
  ProcessInfo: TProcessInformation;
  dwExitCode: DWORD;
  WorkDir: PChar;
  Security: TSecurityAttributes;
  tmpName: string;
  tmp: THandle;
  X: UInt64;
begin
  if sWorkDir <> '' then
    WorkDir := PChar(sWorkDir)
  else
    WorkDir := nil;
  FillChar(Security, sizeof(Security), #0);
  with Security do
  begin
    nlength := sizeof(TSecurityAttributes);
    binherithandle := True;
    lpsecuritydescriptor := nil;
  end;
  if Log <> nil then
  begin
    tmpName := ChangeFileExt(ExtractFileName(GetModuleName), '.log');
    tmp := CreateFile(PChar(tmpName), GENERIC_WRITE, File_Share_Write,
      @Security, Create_Always, FILE_ATTRIBUTE_NORMAL, 0);
  end;
  FillChar(StartupInfo, sizeof(StartupInfo), #0);
  StartupInfo.cb := sizeof(StartupInfo);
  if Log <> nil then
    StartupInfo.hStdOutput := tmp;
  StartupInfo.wShowWindow := SW_HIDE;
  StartupInfo.dwFlags := StartF_UseStdHandles + STARTF_USESHOWWINDOW;
  X := 0;
  if CreateProcess(nil, PChar('"' + sExe + '" ' + sCommandLine + ''), @Security,
    @Security, True, 0, nil, WorkDir, StartupInfo, ProcessInfo) and wait then
  begin
    while WaitForSingleObject(ProcessInfo.hProcess, 0) = WAIT_TIMEOUT do
    begin
      if Freeze = false then
        Sleep(1);
    end;
    WaitForSingleObject(ProcessInfo.hProcess, INFINITE);
    GetExitCodeProcess(ProcessInfo.hProcess, dwExitCode);
    if Log <> nil then
      CloseHandle(tmp);
    CloseHandle(ProcessInfo.hProcess);
    CloseHandle(ProcessInfo.hThread);
    if Log <> nil then
    begin
      if FileExists(tmpName) then
        Log.LoadFromFile(tmpName);
      DeleteFile(PChar(tmpName));
    end;
  end;
end;

  • 1

#8 IvanPolyacov

IvanPolyacov

    CC Lurker

  • Just Joined
  • Pip
  • 1 posts

Posted 22 December 2016 - 03:06 AM

Full Source Codes of ExecAndCapture

 
Thanks for the function - I've tried it and found a problem with it: it hangs if there is zero-size output, because ReadFile() never returns. So I've changed it a bit to avoid this:

...
     // Read text returned by the console program in its StdOut channel
     repeat
       if not PeekNamedPipe(vStdOutPipe.Output,vBuffer,cBufferSize,@vReadBytes,nil,nil) then break;
       if vReadBytes=0 then break;
       windows.ReadFile(vStdOutPipe.Output, vBuffer^, cBufferSize, vReadBytes, nil);
       if vReadBytes > 0 then
...

  • 0





Also tagged with one or more of these keywords: console program, capture console output, createprocess, execute other program, shell program, console