Friday, June 27, 2008

Python: subprocesses after all

Back in September 2007 I was sitting with dipesh until late in the night on a very interesting problem:
We wanted to implement a "tee"-like class in python which would work as the same command under unix: get input via one stream, double the input and give it out into a file and into a stream (like sys.stdout). This shouldn't be a big problem but when you try to give this class to subprocess.Popen as one of the streams you will find out that Popen doesn't use the write function but instead uses the fileno() function of the fileobject. But there would be a workaround for that:
We use subprocess.PIPE for the wanted input stream, then we check regularly for any stuff within the pipe - that this will not work on windows one could expect.
The problem is that all available read processes will block execution if not enough bytes can be read.

The solution is called pywin32 and is a python package for windows. Mark Hammond from starship has made them and they include a function called PeekNamedPipe which does exactly what we need: peek into the pipe without blocking it. I should leave the rest as a homework to the reader, but for those who don't want to invest any more research, I'll give a first solution from where you can start (look at the class Popen).

4 comments:

Anonymous said...

And thanks for sharing those research. Some more keywords I remember to have used to grep for that case;
python, popen2, popen3, popen4, os.waitpid, poll, child process, read, readline, write, flush, wait, thread, non-blocking, windows, win32, like at linux, posix, WTF?

yay, 10 months! :-)

Unknown said...

I wanted to do something similar, I think.

I wanted to run programs and have their output redirected to the logging module. So I just called select() on stdout and stderr and when there was data to be read I'd read it and log it.

IIRC, select.select is implemented for Windows as well. Does that not do what you want?

SaroEngels said...

hm, not sure about select.select. It is definitely not as easy as I would expect it from python. But maybe I have to dig still deeper into it.

Anonymous said...

I like to earn rappelz rupees, because it is very interesting. I think if you meet the rappelz gold, you will like it too. And when you play the game well you can get some cheap rappelz rupees as the rewards from the game. As long as you have a lot of rappelz money in the game you will be strong, and you can go to buy rupees with the other players.