I'm currently using subprocess / msvcrt to asynchronously receive data
from an external program.
I would like to make this platform independent, so switching to
wx.Process + wx.Execute seems like a good idea.
However I'm struggling with a problem:
While the external application is running, it periodically prints its
status to stdout. Unfortunately it terminates each line with a
carriage return, but NOT a linefeed. This means that subsequent status
messages overwrite previous ones when viewed in a terminal.
Unfortunately (on linux at least) wxProcess only seems to output data
to the InputStream after a linefeed is received. So I don't receive
these status messages until a linefeed is printed (just as the
subprocess exits, in this case)
Is there any way to work around this limitation, and read from the
stream before a linefeed is received? Would it help to override the
OnSysRead/Write InputStream methods? Is this even possible?
On Aug 15, 12:54 pm, Pete <sneakypet...@gmail.com> wrote:
I'm currently using subprocess / msvcrt to asynchronously receive data
from an external program.
I would like to make this platform independent, so switching to
wx.Process + wx.Execute seems like a good idea.
However I'm struggling with a problem:
While the external application is running, it periodically prints its
status to stdout. Unfortunately it terminates each line with a
carriage return, but NOT a linefeed. This means that subsequent status
messages overwrite previous ones when viewed in a terminal.
Unfortunately (on linux at least) wxProcess only seems to output data
to the InputStream after a linefeed is received. So I don't receive
these status messages until a linefeed is printed (just as the
subprocess exits, in this case)
Is there any way to work around this limitation, and read from the
stream before a linefeed is received? Would it help to override the
OnSysRead/Write InputStream methods? Is this even possible?
Any help appreciated..
Thanks.
Pete
Maybe I'm missing something here, but what's wrong with using
subprocess? It's cross-platform too...
As far as I'm aware, subprocess doesn't work asynchronously - you
can't read anything from stdout until the process terminates.
I needed to add some os-specific magic to read asynchronously (get
file handle with msvcrt.get_osfhandle, peek into the pipe with
win32.PeekNamedPipe, and read using win32file.ReadFile).
wx.Process seems to handle all of this in an os-independent manner...
I just can't get it to receive any data until a line-feed is output by
the subprocess.
···
2009/8/17 Mike Driscoll <kyosohma@gmail.com>:
On Aug 15, 12:54 pm, Pete <sneakypet...@gmail.com> wrote:
I would like to make this platform independent, so switching to
wx.Process + wx.Execute seems like a good idea.
Maybe I'm missing something here, but what's wrong with using
subprocess? It's cross-platform too...
On Aug 17, 1:17 pm, Pete <sneakypet...@gmail.com> wrote:
2009/8/17 Mike Driscoll <kyoso...@gmail.com>:
> On Aug 15, 12:54 pm, Pete <sneakypet...@gmail.com> wrote:
>> I would like to make this platform independent, so switching to
>> wx.Process + wx.Execute seems like a good idea.
> Maybe I'm missing something here, but what's wrong with using
> subprocess? It's cross-platform too...
As far as I'm aware, subprocess doesn't work asynchronously - you
can't read anything from stdout until the process terminates.
I needed to add some os-specific magic to read asynchronously (get
file handle with msvcrt.get_osfhandle, peek into the pipe with
win32.PeekNamedPipe, and read using win32file.ReadFile).
wx.Process seems to handle all of this in an os-independent manner...
I just can't get it to receive any data until a line-feed is output by
the subprocess.
I admit my experience with subprocess isn't extensive, but according
to what I've read, you should be able to interact with a process and
send it new commands. This link talks about doing just that:
This is the code I use in peppy to manage subprocesses, which works
for me on all 3 platforms. It's based on wx.Process/wx.Execute and
does handle asynchronous reads.
I added a small test case kicked off by the OnStartCRnoLF callback
(using the "Start Sample Python Loop: CR, no LF" button in the demo)
that attempts to reproduce your issue using a small python program as
the subprocess:
import time
for x in range(10):
print 'loop #%d\r' % x,
time.sleep(1)
It does seem to print each line in the output window even though there
isn't a carriage return.
It's probably not a great simulation of your exact problem, but maybe
you replace the call to ProcessManager().run() in OnStartCRnoLF with
your program and see if it works?
On Mon, Aug 17, 2009 at 11:17 AM, Pete<sneakypete81@gmail.com> wrote:
wx.Process seems to handle all of this in an os-independent manner...
I just can't get it to receive any data until a line-feed is output by
the subprocess.
I'm currently using subprocess / msvcrt to asynchronously receive data
from an external program.
I would like to make this platform independent, so switching to
wx.Process + wx.Execute seems like a good idea.
However I'm struggling with a problem:
While the external application is running, it periodically prints its
status to stdout. Unfortunately it terminates each line with a
carriage return, but NOT a linefeed. This means that subsequent status
messages overwrite previous ones when viewed in a terminal.
Unfortunately (on linux at least) wxProcess only seems to output data
to the InputStream after a linefeed is received. So I don't receive
these status messages until a linefeed is printed (just as the
subprocess exits, in this case)
Is there any way to work around this limitation, and read from the
stream before a linefeed is received?
I don't think that wx.Process or wx.Execute exposes any access to doing it from the parent side, but the child could put its stdout into unbuffered mode. (If your child process happens to be a Python app then run Python with the -u flag.)
Would it help to override the
OnSysRead/Write InputStream methods? Is this even possible?
No it would not help since it is the OS buffering data on the file handle. That is where the change needs to happen. And No, it is not possible since wx.Process is creating those stream instances for you without giving you the chance to tell it to use a different class.
Any help appreciated..
AFAIK the subprocess module should let you do what you need.
On Aug 15, 12:54 pm, Pete <sneakypet...@gmail.com> wrote:
I would like to make this platform independent, so switching to
wx.Process + wx.Execute seems like a good idea.
Maybe I'm missing something here, but what's wrong with using
subprocess? It's cross-platform too...
As far as I'm aware, subprocess doesn't work asynchronously - you
can't read anything from stdout until the process terminates.
Pass stdout=PIPE to subprocess.Popen, then use the stdout attribute of the process object. You may still have issues with the buffer not being flushed until it sees a \n, but at least with subprocess you'll have access to the pipe (and its file handle) to be able to change that setting.
Thanks all for the tips - Rob your code was invaluable!
It turned out that I was stupidly not looking at stderr, and that's
where the subprocess was outputting its progress messages..
So the wxProcess method was actually working fine, and the lack of
line feed was a red herring. Sorry for the confusion, and thanks
again.
Pete
···
2009/8/17 Rob McMullen <rob.mcmullen@gmail.com>:
This is the code I use in peppy to manage subprocesses, which works
for me on all 3 platforms. It's based on wx.Process/wx.Execute and
does handle asynchronous reads.
I've found my mistake here too - I was using stdout.read() with no
length specified, which was quite rightly blocking until the process
terminated.
stdout.read(1) or stdout.readline() works as you describe.
Thanks heaps for your help
Pete
···
2009/8/17 Robin Dunn <robin@alldunn.com>:
Pete wrote:
As far as I'm aware, subprocess doesn't work asynchronously - you
can't read anything from stdout until the process terminates.
Pass stdout=PIPE to subprocess.Popen, then use the stdout attribute of
the process object. You may still have issues with the buffer not being
flushed until it sees a \n, but at least with subprocess you'll have
access to the pipe (and its file handle) to be able to change that setting.