[SOLVED] Read stdout and stderr streams of process by lines

[eMko]

Hello,

is there a way how to read TProcess.Output (which is of type TInputPipeStream) line by line, please?

I would like to make a simple UI over a legacy prolog program (C executable which embeds swi-prolog). This communicates through standard input/output/error - reads a line from stdin and writes all resolutions to stdout, one by line. I can't use my beloved .Net (= puts the food on my table) for this task* and Java is not a convenient option for this either**. Python suffers from a Global Interpreter Lock and I need real concurrent-running threads. Doing the GUI in C (Gtk+) is total pain in the back and the C++ (Gtk-mm, WxWidgets) are no better. Qt is OK, but in version 5 it takes the 40MB of libraries with itself on Windows and I am not speaking about the hell of setting the C++ environment on Windows other than Visual Studio.

I looked on this tutorial http://wiki.lazarus.freepascal.org/Executing_External_Programs#Reading_large_output . I am pleased that the API is easy and intuitive, but every code on that page (and even in lazarus-ccr linked from there) is reading the chunks of memory rather than lines. So is there some way how to read whole line from stdout/stderr, please, or I should handle that myself? And if no, how do you break the string/pchar (whatever) into lines? Just UTF8Pos(LineEnding, ...) ?

Thank you very much!

* a) Mono is not preinstalled on targetting linux machines (at university). b) The newest .Net 4.5 does not work on Win XP. But is a replace-update for 4.0 on Win 7 and is preinstalled on Win8 and the beast has different bugs than 4.0, which makes maintenance a nightmare and I pull hair almost every day in my work.

** Mainly because it does not like to return the no-longer used heap memory to the operating system, which is inconvenient when you have a longer running task with peak memory requirements, which this could be. I don't care about its verbosity - I paid for Intellij IDEA Ultimate which helps you handle this (e.g. generates some code for you, hides the ActionListeners inner classes in Swing and displays that as closure etc.) - neither the lack of language features - IDEA has a support for Clojure, which is a lisp on JVM. So I usually use the JVM for university/home projects, but not like to do on this one.

[ludob]

Just use a TStringlist and load it from the stream

Code: [Select]

  sl:=TStringList.create;
  sl.LoadFromStream(TProcess.Output);
You'll find your individual lines as

Code: [Select]

sl[i]

[Leledumbo]

Here lies the Pascal heaven answer to your needs. An example:

Code: [Select]

uses
  StreamIO;
...
var
  F: TextFile;
  S: String;
...
  AssignStream(F,AProcess.Output);
  Reset(F);
  while not EOF(F) do begin
    ReadLn(F,S); // read a single line
    // do something with S
  end;
...


[eMko]

Thank you!

[Tony Stone]

Sorry to re-open a very old discussion. I am hoping one of the experienced people here can answer  my question on this example.  I am actually using this example in a program... slightly modified howver I don't understand how F is not becoming a massive variable holding all the data coming from the aprocess.output?  I am currently using it to read a program that is piping hundreds of millions of lines per day and i am reading and parsing the lines with no issue.  But I am just confused on why F doesn't grow out of control?  What am I missing here?

Thanks 

Here lies the Pascal heaven answer to your needs. An example:

Code: [Select]

uses
  StreamIO;
...
var
  F: TextFile;
  S: String;
...
  AssignStream(F,AProcess.Output);
  Reset(F);
  while not EOF(F) do begin
    ReadLn(F,S); // read a single line
    // do something with S
  end;
...


[af0815]

I am currently using it to read a program that is piping hundreds of millions of lines per day and i am reading and parsing the lines with no issue.  But I am just confused on why F doesn't grow out of control?  What am I missing here?

Here lies the Pascal heaven answer to your needs. An example:

Code: [Select]

var
  F: TextFile;


You are missing that F only holds a descriptor of a file/device. So it is only the information where the information is coming from. This is holding no colleted data. The readed data is stored in a string (or other place of your need).

[Tony Stone]

Ahhh... Ok I think I get it.  I suppose I will have to read some documentation again in regards to reading files, pipes etc.  It is sometimes hard to look for documentation when you don't really know proper terminology.  I will get a better understanding of all this.  Thanks again.   

I am currently using it to read a program that is piping hundreds of millions of lines per day and i am reading and parsing the lines with no issue.  But I am just confused on why F doesn't grow out of control?  What am I missing here?

Here lies the Pascal heaven answer to your needs. An example:

Code: [Select]

var
  F: TextFile;


You are missing that F only holds a descriptor of a file/device. So it is only the information where the information is coming from. This is holding no colleted data. The readed data is stored in a string (or other place of your need).

[marcov]

I looked on this tutorial http://wiki.lazarus.freepascal.org/Executing_External_Programs#Reading_large_output . I am pleased that the API is easy and intuitive, but every code on that page (and even in lazarus-ccr linked from there) is reading the chunks of memory rather than lines.

Well yeah. Isn't that logical? If you have to say up front how many bytes to read, you don't know when the line is going to end?

So is there some way how to read whole line from stdout/stderr, please, or I should handle that myself?

The base asynchronous stream concept is there. You need something to take a stream, read the bytes, divide into line, buffer incomplete ones,  and then have some way to tell if there are lines to process or so, and iterate over them. Something like that isn't there yet, but it could be fairly generic.

For inspiration, maybe look at Francois Piette's (of ICS fame) terminal routines or so.

[engkin]

@marcov, you are answering the old, 2013, question. The new one is in post #4

@Tony Stone, kindly don't revive an old thread. Simply create a new one with a link.