I was looking for a way to run an external process from python script and print its stdout messages during the execution.
The code below works, but prints no stdout output during runtime. When it exits I am getting the following error:
sys.stdout.write(nextline) TypeError:must be str,not bytes
p = subprocess.Popen(["demo.exe"],stdout = subprocess.PIPE, stderr= subprocess.PIPE)
# Poll process for new output until finished
while True:
nextline = p.stdout.readline()
if nextline == '' and p.poll() != None:
break
sys.stdout.write(nextline)
sys.stdout.flush()
output = p.communicate()[0]
exitCode = p.returncode
I am using python 3.3.2
This question is related to
python
Python 3 handles strings a bit different. Originally there was just one type for
strings: str. When unicode gained traction in the '90s the new unicode type
was added to handle Unicode without breaking pre-existing code1. This is
effectively the same as str but with multibyte support.
In Python 3 there are two different types:
bytes type. This is just a sequence of bytes, Python doesn't know
anything about how to interpret this as characters.str type. This is also a sequence of bytes, but Python knows how to
interpret those bytes as characters.unicode type was dropped. str now supports unicode.In Python 2 implicitly assuming an encoding could cause a lot of problems; you
could end up using the wrong encoding, or the data may not have an encoding at
all (e.g. it’s a PNG image).
Explicitly telling Python which encoding to use (or explicitly telling it to
guess) is often a lot better and much more in line with the "Python philosophy"
of "explicit is better than implicit".
This change is incompatible with Python 2 as many return values have changed,
leading to subtle problems like this one; it's probably the main reason why
Python 3 adoption has been so slow. Since Python doesn't have static typing2
it's impossible to change this automatically with a script (such as the bundled
2to3).
str to bytes with bytes('h€llo', 'utf-8'); this should
produce b'H\xe2\x82\xacllo'. Note how one character was converted to three
bytes.bytes to str with b'H\xe2\x82\xacllo'.decode('utf-8').Of course, UTF-8 may not be the correct character set in your case, so be sure to use the correct one.
In your specific piece of code, nextline is of type bytes, not str,
reading stdout and stdin from subprocess changed in Python 3 from str to
bytes. This is because Python can't be sure which encoding this uses. It
probably uses the same as sys.stdin.encoding (the encoding of your system),
but it can't be sure.
You need to replace:
sys.stdout.write(nextline)
with:
sys.stdout.write(nextline.decode('utf-8'))
or maybe:
sys.stdout.write(nextline.decode(sys.stdout.encoding))
You will also need to modify if nextline == '' to if nextline == b'' since:
>>> '' == b''
False
Also see the Python 3 ChangeLog, PEP 358, and PEP 3112.
1 There are some neat tricks you can do with ASCII that you can't do with multibyte character sets; the most famous example is the "xor with space to switch case" (e.g. chr(ord('a') ^ ord(' ')) == 'A') and "set 6th bit to make a control character" (e.g. ord('\t') + ord('@') == ord('I')). ASCII was designed in a time when manipulating individual bits was an operation with a non-negligible performance impact.
2 Yes, you can use function annotations, but it's a comparatively new feature and little used.
While the accepted answer will work fine if the bytes you have from your subprocess are encoded using sys.stdout.encoding (or a compatible encoding, like reading from a tool that outputs ASCII and your stdout uses UTF-8), the correct way to write arbitrary bytes to stdout is:
sys.stdout.buffer.write(some_bytes_object)
This will just output the bytes as-is, without trying to treat them as text-in-some-encoding.
Source: Stackoverflow.com