Subprocess
Popen.communicate(input=None)
-
option :
input :[string] sent to the child process. default None
-
workflow:
Send data to stdin. Read data from stdout and stderr until end-of-file is reached. Wait for process to terminate.
-
return :
[tuple] (stdoutdata, stderrdata)
Note :
that if you want to send data to the process’s stdin, you need to create the Popen object with stdin=PIPE. Similarly, to get anything other than None in the result tuple, you need to give stdout=PIPE and/or stderr=PIPE too.
Note
The data read is buffered in memory, so do not use this method if the data size is large or unlimited.
##EXAMPLE 1 ###test.py
#! /usr/bin python
def test():
print "Something to print"
while(True):
input =raw_input("output what you input(y/n)")
if r=='n':
print "Exiting"
break
else :
print input
if __name__ == "__main__":
test()
###subproc.py
#! /usr/bin python
def subproc():
p=subprocess.Popen(["python","test.py"],stdin=PIPE,stdout=PIPE)
print p.communicate()[0]
if __name__ == "__main__":
subproc()
###Ouput
Traceback (most recent call last):
File "./test.py", line 13, in <module>
test()
File "./test.py", line 5, in test
input =raw_input("output what you input, input 'n'exit\n")
EOFError: EOF when reading a line
Something to print
output what you input, input 'n'exit
###Analysis
look at the workflow: .communicate() writes input , reads all output, and waits for the subprocess to exit.
there is no input in this case so it just closes subprocess’ stdin to indicate to the subprocess that there is no more input
p.stdout.read() hangs forever because it tries to read all output from the child at the same time as the child waits for input (raw_input()) that causes a deadlock.
###Correct
To avoid the deadlock you need to read/write asynchronously (e.g., by using threads or select) or to know exactly when and how much to read/write
from subprocess import PIPE, Popen
p = Popen(["python", "-u", "test.py"], stdin=PIPE, stdout=PIPE, bufsize=1)
print p.stdout.readline(), # read the first line "Something to print"
for i in range(10): # repeat several times to show that it works
print >>p.stdin, i # write input, stdin=PIPE
# the above print is equal to follow two sentence
# p.stdin.write(str(i)+"\n")
# p.stdin.flush()
print p.stdout.readline(), # read output,stdout=PIPE
print p.communicate("n\n")[0], # signal the child to exit,
# read the rest of the output,
# wait for the child to exit
###Example2
#!/usr/bin/python
from subprocess import Popen, PIPE
import threading
p = Popen('ls', stdout=PIPE)
class ReaderThread(threading.Thread):
def __init__(self, stream):
threading.Thread.__init__(self)
self.stream = stream
def run(self):
while True:
line = self.stream.readline()
if len(line) == 0:
break
print line,
reader = ReaderThread(p.stdout)
reader.start()
# Wait until subprocess is done
p.wait()
# Wait until we've processed all output
reader.join()
print "Done!"
###Example3
The task I try to accomplish is to stream a ruby file and print out the output. (NOTE: I don’t want to print out everything at once)
####main.py
from subprocess import Popen, PIPE, STDOUT
import pty
import os
file_path = '/Users/luciano/Desktop/ruby_sleep.rb'
command = ' '.join(["ruby", file_path])
master, slave = pty.openpty()
proc = Popen(command, bufsize=0, shell=True, stdout=slave, stderr=slave, close_fds=True)
stdout = os.fdopen(master, 'r', 0)
while proc.poll() is None:
data = stdout.readline()
if data != "":
print(data)
else:
break
print("This is never reached!")
ruby_sleep.rb
puts "hello"
sleep 2
puts "goodbye!"
Problem
Streaming the file works fine. The hello/goodbye output is printed with the 2 seconds delay. Exactly as the script should work. The problem is that readline() hangs in the end and never quits. I never reach the last print.
I know there is a lot of questions like this here a stackoverflow but non of them made me solve the problem. I’m not that into the whole subprocess thing so please give me a more hands-on/concrete answer. pty is Linux only as said in the docs:
solution
Because pseudo-terminal handling is highly platform dependent, there is code to do it only for Linux. (The Linux code is supposed to work on other platforms, but hasn’t been tested yet.)
It is unclear how well it works on other OSes.You could try pexpect:
import sys
import pexpect
pexpect.run("ruby ruby_sleep.rb", logfile=sys.stdout)
Or stdbuf to enable line-buffering in non-interactive mode:
from subprocess import Popen, PIPE, STDOUT
proc = Popen(['stdbuf', '-oL', 'ruby', 'ruby_sleep.rb'],
bufsize=1, stdout=PIPE, stderr=STDOUT, close_fds=True)
for line in iter(proc.stdout.readline, b''):
print line,
proc.stdout.close()
proc.wait()
Or using pty based on @Antti Haapala's answer:
import os
import pty
import select
from subprocess import Popen, STDOUT
master_fd, slave_fd = pty.openpty() # provide tty to enable
# line-buffering on ruby's side
proc = Popen(['ruby', 'ruby_sleep.rb'],
bufsize=1, stdout=slave_fd, stderr=STDOUT, close_fds=True)
timeout = .04 # seconds
while 1:
ready, _, _ = select.select([master_fd], [], [], timeout)
if ready:
data = os.read(master_fd, 512)
if not data:
break
print("got " + repr(data))
elif proc.poll() is not None: # select timeout
break # proc exited
os.close(slave_fd) # can't do it sooner: it leads to errno.EIO error
os.close(master_fd)
proc.wait()
print("This is reached!")
All three code examples print 'hello' immediately (as soon as the first EOL is seen).
###Example 4
The process I’m using right now is this:
Attach a pty to the subprocess's stdout
Loop until the subprocess exits by checking subprocess.poll
When there is data available in the stdout write that data immediately to the current stdout.
Finish!
After the child process has completed, the parent process hangs if I do not specify a timeout when using select.select. I would really prefer not to set a timeout. It just seems a bit dirty. However, all the other ways I’ve tried to get around the issue don’t seem to work. Pexpect seems to get around it by using os.execv and pty.fork instead of subprocess.Popen and pty.openpty a solution I do not prefer. Am I doing something wrong with how I check for the life of the subprocess? Is my approach incorrect?
The code I’m using is below. I’m using this on a Mac OS X 10.6.8, but I need it to work on Ubuntu 12.04 as well.
This is the subprocess runner runner.py:
import subprocess
import select
import pty
import os
import sys
def main():
master, slave = pty.openpty()
process = subprocess.Popen(['python', 'outputter.py'],
stdin=subprocess.PIPE,
stdout=slave, stderr=slave, close_fds=True)
while process.poll() is None:
# Just FYI timeout is the last argument to select.select
rlist, wlist, xlist = select.select([master], [], [])
for f in rlist:
output = os.read(f, 1000) # This is used because it doesn't block
sys.stdout.write(output)
sys.stdout.flush()
print "**ALL COMPLETED**"
if __name__ == '__main__':
main()
This is the subprocess code outputter.py. The strange random parts are just to simulate a program outputting data at random intervals. You can remove it if you wish. It shouldn’t matter:
import time
import sys
import random
def main():
lines = ['hello', 'there', 'what', 'are', 'you', 'doing']
for line in lines:
sys.stdout.write(line + random.choice(['', '\n']))
sys.stdout.flush()
time.sleep(random.choice([1,2,3,4,5])/20.0)
sys.stdout.write("\ndone\n")
sys.stdout.flush()
if __name__ == '__main__':
main()
###Example 5
Non-blocking read on a subprocess.PIPE in python
#!/usr/bin/python
# Runner with stdout/stderr catcher
from sys import argv
from subprocess import Popen, PIPE
import os, io
from threading import Thread
import Queue
def __main__():
if (len(argv) > 1) and (argv[-1] == "-sub-"):
import time, sys
print "Application runned!"
time.sleep(2)
print "Slept 2 second"
time.sleep(1)
print "Slept 1 additional second",
time.sleep(2)
sys.stderr.write("Stderr output after 5 seconds")
print "Eol on stdin"
sys.stderr.write("Eol on stderr\n")
time.sleep(1)
print "Wow, we have end of work!",
else:
os.environ["PYTHONUNBUFFERED"]="1"
try:
p = Popen( argv + ["-sub-"],
bufsize=0, # line-buffered
stdin=PIPE, stdout=PIPE, stderr=PIPE )
except WindowsError, W:
if W.winerror==193:
p = Popen( argv + ["-sub-"],
shell=True, # Try to run via shell
bufsize=0, # line-buffered
stdin=PIPE, stdout=PIPE, stderr=PIPE )
else:
raise
inp = Queue.Queue()
sout = io.open(p.stdout.fileno(), 'rb', closefd=False)
serr = io.open(p.stderr.fileno(), 'rb', closefd=False)
def Pump(stream, category):
queue = Queue.Queue()
def rdr():
while True:
buf = stream.read1(8192)
if len(buf)>0:
queue.put( buf )
else:
queue.put( None )
return
def clct():
active = True
while active:
r = queue.get()
try:
while True:
r1 = queue.get(timeout=0.005)
if r1 is None:
active = False
break
else:
r += r1
except Queue.Empty:
pass
inp.put( (category, r) )
for tgt in [rdr, clct]:
th = Thread(target=tgt)
th.setDaemon(True)
th.start()
Pump(sout, 'stdout')
Pump(serr, 'stderr')
while p.poll() is None:
# App still working
try:
chan,line = inp.get(timeout = 1.0)
if chan=='stdout':
print "STDOUT>>", line, "<?<"
elif chan=='stderr':
print " ERROR==", line, "=?="
except Queue.Empty:
pass
print "Finish"
if __name__ == '__main__':
__main__()
fcntl, select, asyncproc won’t help in this case. no good, uses select, unavailable in windows to file descriptors. Reliable way to read a stream without blocking on both Windows and Linux is to use Queue.get_nowait():
import sys
from subprocess import PIPE, Popen
from threading import Thread
try:
from Queue import Queue, Empty
except ImportError:
from queue import Queue, Empty # python 3.x
ON_POSIX = 'posix' in sys.builtin_module_names
def enqueue_output(out, queue):
for line in iter(out.readline, b''):
queue.put(line)
out.close()
p = Popen(['myprogram.exe'], stdout=PIPE, bufsize=1, close_fds=ON_POSIX)
q = Queue()
t = Thread(target=enqueue_output, args=(p.stdout, q))
t.daemon = True # thread dies with the program
t.start()
# ... do other things here
# read line without blocking
try: line = q.get_nowait() # or q.get(timeout=.1)
except Empty:
print('no output yet')
else: # got line
# ... do something with line
Try the asyncproc module.which work on linux,not windows For example:
import os
from asyncproc import Process
myProc = Process("myprogram.app")
while True:
# check to see if process has ended
poll = myProc.wait(os.WNOHANG)
if poll != None:
break
# print any new output
out = myProc.read()
if out != "":
print out
###Example 6
import subprocess
some_string = 'input_data'
sort_out = file('outfile.txt','w')
sort_in = subprocess.Popen('sort', stdin=subprocess.PIPE, stdout=sort_out).stdin
subprocess.Popen(['awk', '-f', 'script.awk'], stdout=sort_in, \
stdin=subprocess.PIPE).communicate(some_string)
#!/usr/bin/env python
from subprocess import Popen, PIPE
a = Popen(["a"], stdin=PIPE, stdout=PIPE)
with open("outfile.txt", "wb") as outfile:
b = Popen(["b"], stdin=a.stdout, stdout=outfile)
a.stdout.close() # notify `a` if `b` exits (doesn't accept input anymore)
a.stdin.write(b"input data")
a.stdin.close() # no more input
b.wait()
a.wait() # should return on EOF in stdin or after an attempt to write to stdout
[1] (http://stackoverflow.com/questions/16768290/working-of-popen-communicate?answertab=active#tab-top) [2] (http://stackoverflow.com/questions/12419198/python-subprocess-readlines-hangs/12471855#12471855) [3] (http://stackoverflow.com/questions/12419198/python-subprocess-readlines-hangs/12471855#12471855) [4] (http://stackoverflow.com/questions/375427/non-blocking-read-on-a-subprocess-pipe-in-python) [5] (http://stackoverflow.com/questions/295459/how-do-i-use-subprocess-popen-to-connect-multiple-processes-by-pipesp 封装包 [5] (http://code.activestate.com/recipes/440554-module-to-allow-asynchronous-subprocess-use-on-win/)