python Popen stderr=PIPE 6000+ lines causes hang

wxPython Users
1

I am running a command line application via Popen in OSX which produces many
lines of command line readout, passing this via stderr to my wx.TextCtrl
window. I am using threads and classes and everything has been working fine,
until I run something which produces thousands of lines, at which point my
application runs good until it hangs at line 6498 and i'm not sure why, any
ideas please?

self.process1 = Popen(shlex.split(command), shell=False, stderr=PIPE)
while True:
    line = self.process1.stderr.readline().decode('utf-8')
    wx.CallAfter(self.frame.running_log1.AppendText, line)
    self.process1.stderr.flush()
    if "Some text" in line:
        break

···

--
View this message in context: http://wxpython-users.1045709.n5.nabble.com/python-Popen-stderr-PIPE-6000-lines-causes-hang-tp5721433.html
Sent from the wxPython-users mailing list archive at Nabble.com.

2

Sounds like a buffer overrun.

···

On 28/06/14 08:32, speedyrazor wrote:

I am running a command line application via Popen in OSX which produces many
lines of command line readout, passing this via stderr to my wx.TextCtrl
window. I am using threads and classes and everything has been working fine,
until I run something which produces thousands of lines, at which point my
application runs good until it hangs at line 6498 and i'm not sure why, any
ideas please?

self.process1 = Popen(shlex.split(command), shell=False, stderr=PIPE)
while True:
     line = self.process1.stderr.readline().decode('utf-8')
     wx.CallAfter(self.frame.running_log1.AppendText, line)
     self.process1.stderr.flush()
     if "Some text" in line:
         break

--
View this message in context: http://wxpython-users.1045709.n5.nabble.com/python-Popen-stderr-PIPE-6000-lines-causes-hang-tp5721433.html
Sent from the wxPython-users mailing list archive at Nabble.com.

3

I’d call flush() before calling readline.

···

On Saturday, June 28, 2014 12:32:52 AM UTC-7, kevjwells wrote:

I am running a command line application via Popen in OSX which produces many

lines of command line readout, passing this via stderr to my wx.TextCtrl

window. I am using threads and classes and everything has been working fine,

until I run something which produces thousands of lines, at which point my

application runs good until it hangs at line 6498 and i’m not sure why, any

ideas please?

self.process1 = Popen(shlex.split(command), shell=False, stderr=PIPE)

while True:

line = self.process1.stderr.readline().decode('utf-8')

wx.CallAfter(self.frame.running_log1.AppendText, line)

self.process1.stderr.flush()

if "Some text" in line:

    break

–

View this message in context: http://wxpython-users.1045709.n5.nabble.com/python-Popen-stderr-PIPE-6000-lines-causes-hang-tp5721433.html

Sent from the wxPython-users mailing list archive at Nabble.com.

4

also, try adding to Popen the bufsize arg:
https://docs.python.org/2/library/subprocess.html

"

bufsize, if given, has the same meaning as the corresponding argument to the built-in open() function: 0 means unbuffered, 1 means line buffered, any other positive value means use a buffer of (approximately) that size. A negative bufsize means to use the system default, which usually means fully buffered. The default value for bufsize is 0 (unbuffered).

Note

If you experience performance issues, it is recommended that you try to enable buffering by setting bufsize to either -1 or a large enough positive value (such as 4096).

"

On second thought, I’m not sure you calling flush on that will have any meaning, as this end of the PIPE is the receiving end, while I believe the process generating the output controls flushing.

So maybe try commenting out the flush, and play with bufsize.

···

On Saturday, June 28, 2014 12:32:52 AM UTC-7, kevjwells wrote:

I am running a command line application via Popen in OSX which produces many

lines of command line readout, passing this via stderr to my wx.TextCtrl

window. I am using threads and classes and everything has been working fine,

until I run something which produces thousands of lines, at which point my

application runs good until it hangs at line 6498 and i’m not sure why, any

ideas please?

self.process1 = Popen(shlex.split(command), shell=False, stderr=PIPE)

while True:

line = self.process1.stderr.readline().decode('utf-8')

wx.CallAfter(self.frame.running_log1.AppendText, line)

self.process1.stderr.flush()

if "Some text" in line:

    break

–

View this message in context: http://wxpython-users.1045709.n5.nabble.com/python-Popen-stderr-PIPE-6000-lines-causes-hang-tp5721433.html

Sent from the wxPython-users mailing list archive at Nabble.com.

5

speedyrazor wrote:

self.process1 = Popen(shlex.split(command), shell=False, stderr=PIPE)

The process might be stuck on a blocking write to stdout. You may
need to create a stdout pipe as well, and read from that in a separate
thread.

regards, Anders