I'm working on a project where I will be running potentially malicious code. It's basic organization is that there is a master and a slave process. The slave process runs the potentially malicious code, and has seccomp enabled.
import prctl
prctl.set_seccomp(True)
This is how seccomp is turned on. I can communicate fine FROM the slave TO the master, but not the other way around. When I don't turn on seccomp, I can use:
import sys
lines = sys.stdin.read()
Or something along those lines. I found this quite odd, I should have access to read and write given the default parameters of seccomp, especially for stdin/out. I have even tried opening stdin before I turn on seccomp. For example.
stdinFile = sys.stdin
prctl.set_seccomp(True)
lines = stdinFile.read()
But still to no avail. I have also tried readlines() which doesn't work. A friend suggested that I try Unix Domain Sockets, opening it before seccomp goes on, and then just using the write() call. This didn't work either. If anyone has any suggestions on how to combat this problem, please post them! I have seen some code in C for something like
seccomp_add_rule(stuff)
But I have been unsuccessful at using this in Python with the cffi module.
sys.stdin is not a file handle, you need to open it and get a file handle before calling set_seccomp. You could use os.fdopen for this. The file descriptor for stdin / stdout is available as sys.stdin.fileno().
Related
I'm new in this world of python. Recently I have been asked to make an interface between XFoil (an aerodynamics program) and python. After researching a little bit, I found the subprocess module. As the documentation says it's used to "Spawn new processes, connect to their input/output/error pipes, and obtain their return codes."
The problem is that I need some output archives that XFoil creates while its running. If I close the program, the archives are accesible, but if I try to open or read them while the subprocess is still opened it gives me the following error (Although I can see the archive in the folder):
OSError: save not found.
Here the code:
import subprocess
import numpy as np
import os
process = subprocess.Popen(['<xfoil_path>'], stdin=subprocess.PIPE, universal_newlines=True, creationflags = subprocess.CREATE_NEW_PROCESS_GROUP)
airfoil_path = '<path to airfoil>'
process.stdin.write(f'\nload\n{airfoil_path}')
process.stdin.write('\n\n\noper\nalfa\n2\ncpwr\nsave\n')
process.stdin.tell()
print(os.listdir())
c = np.loadtxt('save', skiprows=1)
print(c)
process.stdin.write('\n\n\noper\nalfa\n3\ncpwr\nsave2\n')
stdin.tell is used to get this output archives, but they are not accesible.
Someone knows why this could be happening?
Why do you imagine process.stdin.tell() should "get this output archives"? It retrieves the file pointer's position.
I'm imagining that the actual problem here is that the subprocess doesn't write the files immediately. Maybe just time.sleep(1) before you try to open them, or figure out a way for it to tell you when it's done writing (some OSes let you tell whether another process has a file open for writing, but I have no idea whether this is possible on Windows, let alone reliable).
Sleeping for an arbitrary time is obviously not very robust; you can't predict how long it takes for the subprocess to write out the files. But if that solves your immediate problem, at least you have some idea of what caused it and maybe where to look next.
As an aside, maybe look into the input= keyword parameter for subprocess.run(). If having the subprocess run in parallel is not crucial, that might be more pleasant as well as more robust.
(Converted into an answer from a comment thread.)
I am trying to use subprocess.Popen to control an ssh process and interact with it via pipes, like so:
p=subprocess.Popen(['ssh','-tt','LOGIN#HOSTNAME'], stdin=subprocess.PIPE,
stdout=subprocess.PIPE, stderr=subprocess.PIPE,
universal_newlines=True)
while True:
(out_stdout,out_stderr)=p.communicate(timeout=10)
if out_stderr:
print(out_stderr)
if not out_stdout:
raise EOFError
print(out_stdout)
This works fine without the '-tt' option to ssh. However the program I need to interact with on the remote side of the ssh breaks if there is no pseudo tty allocated, so I am forced to use it.
What this seems to do is that the p.communicate reads then block indefinitely (or until timeout), even if input is available.
I have rewritten this using lower level calls to io.read, select.select etc to avoid going through Popen.communicate. Select will actually return the file descriptor as ready, but a subsequent io.read to that file descriptor will also block. If I disable 'universal newlines' and set 'bufsize=0' in the Popen call it then works fine, but then I am forced to do binary/unicode conversion and line ending processing myself.
It's worth saying though, that disabling universal_newlines in the p.communicate version also blocks indefinitely, so its not just that.
Any advice on how I can get line buffered input working properly here without having to reimplement everything?
There are library alternatives to subprocess and the SSH binary that are better suited for such tasks.
parallel-ssh:
from pssh.pssh2_client import ParalellSSHClient
client = ParallelSSHClient(['HOSTNAME'], user='LOGIN')
output = client.run_command('echo', use_pty=True)
for host, host_output in output.items():
for line in host_output.stdout:
print(line)
Replace echo with the command you need to run, or leave as-is if no command is required. The libraries require that something is passed in as command even if the remote side executes something automatically.
See also documentation for the single host SSHClient of the same project.
Per documentation, line parsing and encoding are handled by the library which is also cross-platform.
There are others like paramiko and ssh2-python that are lower level and need more code for the equivalent above - see their respective home pages for examples.
I have a web server in Python (2.7) that uses Popen to delegate some work to a child process:
url_arg = "http://localhost/index.html?someparam=somevalue"
call = ('phantomjs', 'some/phantom/script.js', url_arg)
imageB64data = tempfile.TemporaryFile()
errordata = tempfile.TemporaryFile()
p = Popen(call, stdout=imageB64data, stderr=errordata, stdin=PIPE)
p.communicate(input="")
I am seeing intermittent issues where after some number of these Popens have occurred (roughly 64), the process runs out of file descriptors and is unable to function -- it becomes completely unresponsive and all threads seem to block forever if they attempt to open any files or sockets.
(Possibly relevant: the phantomjs child process loads a URL calls back into the server that spawned it.)
Based on this Python bug report, I believe I need to set close_fds=True on all Popen calls from inside my server process in order to mitigate the leaking of file descriptors. However, I am unfamiliar with the machinery around exec-ing subprocesses and inheritance of file descriptors so much of the Popen documentation and the notes in the aforementioned bug report are unclear to me.
It sounds like it would actually close all open file descriptors (which includes active request sockets, log file handles, etc.) in my process before executing the subprocess. This sounds like it would be strictly better than leaking the sockets, but would still result in errors.
However, in practice, when I use close_fds=True during a web request, it seems to work fine and thus far I have been unable to construct a scenario where it actually closes any other request sockets, database requests, etc.
The docs state:
If close_fds is true, all file descriptors except 0, 1 and 2 will be closed before the child process is executed.
So my question is: is it "safe" and "correct" to pass close_fds=True to Popen in a multithreaded Python web server? Or should I expect this to have side effects if other requests are doing file/socket IO at the same time?
I tried the following test with the subprocess32 backport of Python 3.2/3.3's subprocess:
import tempfile
import subprocess32 as subprocess
fp = open('test.txt', 'w')
fp.write("some stuff")
echoed = tempfile.TemporaryFile()
p = subprocess.Popen(("echo", "this", "stuff"), stdout=echoed, close_fds=True)
p.wait()
echoed.seek(0)
fp.write("whatevs")
fp.write(echoed.read())
fp.close()
and I got the expected result of some stuffwhatevsecho this stuff in test.txt.
So it appears that the meaning of close in close_fds does not mean that open files (sockets, etc.) in the parent process will be unusable after executing a child process.
Also worth noting: subprocess32 defaults close_fds=True on POSIX systems, AFAICT. This implies to me that it is not as dangerous as it sounds.
I suspect that close_fds solves the problem of file descriptors leaking to subprocesses. Imagine opening a file, and then running some task using subprocess. Without close_fds, the file descriptor is copied to the subprocess, so even if the parent process closes the file, the file remains open due to the subprocess. Now, let's say we want to delete the directory with the file in another thread using shutil.rmtree. On a regular filesystem, this should not be an issue. The directory is just removed as expected. However, when the file resides on NFS, the following happens: First, Python will try to delete the file. Since the file is still in use, it gets renamed to .nfsXXX instead, where XXX is a long hexadecimal number. Next, Python will try to delete the directory, but that has become impossible because the .nfsXXX file still resides in it.
This question already has answers here:
read subprocess stdout line by line
(10 answers)
Closed 21 days ago.
How can I receive input from the terminal in Python?
I am using Python to interface with another program which generates output from user input.
I am using subprocess.Popen() to input to the program, but I can't set stdout to subprocess.PIPE because the program does not seem to flush ever, so everything gets stuck in the buffer.
The program's standard output seems to be to print to terminal, and I see output when I do not redirect stdout. However, I need Python to read and interpret the output which is now in the terminal.
Sorry if this is a stupid question, but I can't seem to get this to work.
Buffering in child processes is a common problem. Here are four possible approaches.
First, and easiest, you could read one byte at a time from your pipe. This is what I would call a "dirty hack" and it carries a performance penalty, but it's easy and it guarantees that your read() calls will only block until the first byte comes in, rather than wait for a buffer to fill up that's never going to fill up. However, this does not force the other process to flush its write buffer, so if that is the issue this approach will not help you anyway.
Second, and I think next-easiest, consider using the Twisted framework which has a facility for using a virtual terminal, or pty ("pseudo-teletype" I think) to talk to your child process. However, this can affect the design of your application (possibly for the better, but this may not be in the cards for you regardless). http://twistedmatrix.com/documents/current/core/howto/process.html
If neither of the above options works for you, you're reduced to solving gritty I/O concurrency issues yourself.
Third, try setting your pipes (all of them, before fork()) to non-blocking mode using fcntl() with O_NONBLOCK. Then you can use select() to test for read/write readiness before trying the read/write; but you still have to catch IOError and test for EAGAIN because it can happen even in this case. This may, depending on the behavior of the child process, allow you to wait until the data really shows up before trying to read it in.
The last resort is to implement the PTY logic yourself. If you've seen references to stuff like termio options, ioctl() calls, etc. then that's what you're up against. I have not done this before, because it's complicated and I have never really needed to. If this is your destiny, good luck.
Have you tried setting the bufsize in your Popen object to 0? I'm not sure if you can force the buffer to be unbuffered from the receiving size, but I'd try it.
http://docs.python.org/library/subprocess.html#using-the-subprocess-module
There's a similar question to mine on [this thread][1].
I want to send a command to my subprocess, interpret the response, then send another command. It would seem a shame to have to start a new subprocess to accomplish this, particularly if subprocess2 must perform many of the same tasks as subprocess1 (e.g. ssh, open mysql).
I tried the following:
subprocess1.stdin.write([my commands])
subprocess1.stdin.flush()
subprocess1.stout.read()
But without a definite parameter for bytes to read(), the program gets stuck executing that instruction, and I can't supply an argument for read() because I can't guess how many bytes are available in the stream.
I'm running WinXP, Py2.7.1
EDIT
Credit goes to #regularfry for giving me the best solution for my real intention (read the comments in his response, as they pertain to accomplishing my goal through an SSH tunnel). (His/her answer has been voted up.) For the benefit of any viewer who hereafter comes for an answer to the title question, however, I've accepted #Mike Penningtion's answer.
Your choices are:
Use a line-oriented protocol (and use readline() rather than read()), and ensure that every possible line sent is a valid message;
Use read(1) and a parser to tell you when you've read a full message; or
Pickle message objects into the stream from the subprocess, then unpickle them in the parent. This handles the message length problem for you.
#JellicleCat, I'm following up on the comments. I believe wexpect is a part of sage... AFAIK, it is not packaged separately, but you can download wexpect here.
Honestly, if you're going to drive programmatic ssh sessions, use paramiko. It is supported as an independent installation, has good packaging, and should install natively on windows.
EDIT
Sample paramiko script to cd to a directory, execute an ls and exit... capturing all results...
import sys
sys.stderr = open('/dev/null') # Silence silly warnings from paramiko
import paramiko as pm
sys.stderr = sys.__stderr__
import os
class AllowAllKeys(pm.MissingHostKeyPolicy):
def missing_host_key(self, client, hostname, key):
return
HOST = '127.0.0.1'
USER = ''
PASSWORD = ''
client = pm.SSHClient()
client.load_system_host_keys()
client.load_host_keys(os.path.expanduser('~/.ssh/known_hosts'))
client.set_missing_host_key_policy(AllowAllKeys())
client.connect(HOST, username=USER, password=PASSWORD)
channel = client.invoke_shell()
stdin = channel.makefile('wb')
stdout = channel.makefile('rb')
stdin.write('''
cd tmp
ls
exit
''')
print stdout.read()
stdout.close()
stdin.close()
client.close()
This approach will work (I've done this) but will take some time and it uses Unix-specific calls. You'll have to abandon the subprocess module and roll your own equivalent based on fork/exec and os.pipe().
Use the fcntl.fcntl function to place the stdin/stdout file descriptors (read and write) for your child process into non-blocking mode (O_NONBLOCK option constant) after creating them with os.pipe().
Use the select.select function to poll or wait for availability on your file descriptors. To avoid deadlocks you will need to use select() to ensure that writes will not block, just like reads. Even still, you must account for OSError exceptions when you read and write, and retry when you get EAGAIN errors. (Even when using select before read/write, EAGAIN can occur in non-blocking mode; this is a common kernel bug that has proven difficult to fix.)
If you are willing to implement on the Twisted framework, they have supposedly solved this problem for you; all you have to do is write a Process subclass. But I haven't tried that myself yet.