We Keep Coding

python subprocess popen execute as different user

I am trying to execute a command in python 3.6 as a different user with popen from subprocess but it will still execute as the user who called the script (i plan to call it as root). I am using threads and therefore it is important that i don't violate the user rights when 2 threads execute in parallel.
proc = subprocess.Popen(['echo $USER; touch myFile.txt'],
shell=True,
env={'FOO':'bar', 'USER':'www-data'},
stdout=subprocess.PIPE)
The example above will still create the myFile.txt with my user_id 1000
I tried different approaches :
tried with as described in Run child processes as different user from a long running Python process by copying the os.environment and changed the user, etc (note this is for python 2)
tried with as described in https://docs.python.org/3.6/library/subprocess.html#popen-constructor by using start_new_session=True
My Last option is to prefix the command with sudo -u username command but i don't think this is the elegant way.

The standard way [POSIX only] would be to use preexec_fn to set gid and uid as described in more detail in this answer
Something like this should do the trick -- for completeness I've also modified your initial snippet to include the other environment variables you'd likely want to set, but just setting preexec_fn should be sufficient for the simple command you are running:
import os, pwd, subprocess
def demote(user_uid, user_gid):
def result():
os.setgid(user_gid)
os.setuid(user_uid)
return result
def exec_cmd(username):
# get user info from username
pw_record = pwd.getpwnam(username)
homedir = pw_record.pw_dir
user_uid = pw_record.pw_uid
user_gid = pw_record.pw_gid
env = os.environ.copy()
env.update({'HOME': homedir, 'LOGNAME': username, 'PWD': os.getcwd(), 'FOO': 'bar', 'USER': username})
# execute the command
proc = subprocess.Popen(['echo $USER; touch myFile.txt'],
shell=True,
env=env,
preexec_fn=demote(user_uid, user_gid),
stdout=subprocess.PIPE)
proc.wait()
exec_cmd('www-data')
Note that you'll also need to be sure the current working directory is accessible (e.g. for writing) by the demoted user since not overriding it explicitly

Perfect Wrapper (in Python)

I run a configuration management tool which calls /usr/bin/dpkg, but does not show the stdout/stderr.
Something goes wrong and I want to debug the root of the problem.
I want to see all calls to dpkg and stdout/stderr.
I moved the original /usr/bin/dpkg to /usr/bin/dpkg-orig and wrote a wrapper:
#!/usr/bin/env python
import os
import sys
import datetime
import subx
import psutil
cmd=list(sys.argv)
cmd[0]='dpkg-orig'
def parents(pid=None):
if pid==1:
return '\n'
if pid is None:
pid = os.getpid()
process = psutil.Process(pid)
lines = [parents(process.ppid())]
lines.append('Parent: %s' % ' '.join(process.cmdline()))
return '\n'.join(lines)
result = subx.call(cmd, assert_zero_exit_status=False)
with open('/var/tmp/dpkg-calls.log', 'ab') as fd:
fd.write('----------- %s\n' % (datetime.datetime.now()))
fd.write('%s\n' % parents())
fd.write('stdout:\n%s\n\n' % result.stdout)
sys.stdout.write(result.stdout)
fd.write('stderr:\n%s\n' % result.stderr)
fd.write('ret: %s\n' % result.ret)
sys.stderr.write(result.stderr)
sys.exit(result.ret)
Now I run the configuration management tool again and searched for non zero "ret:" lines.
The output:
Parent: /usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold -o DPkg::Options::=--force-confdef install openssl-foo-bar-aptguettler.cert
Parent: python /usr/bin/dpkg --force-confold --force-confdef --status-fd 67 --no-triggers --unpack --auto-deconfigure /var/cache/apt/archives/openssl-foo-bar-aptguettler.cert_1-2_all.deb
stdout:
stderr:
dpkg: error: unable to read filedescriptor flags for <package status and progress file descriptor>: Bad file descriptor
ret: 2
This happens because my wrapper is not perfect yet.
The tool which calls dpkg wants to read the file descriptor but this does not work with my wrapper.
My goal:
Capture all calls to dpkg and write it to a logfile (works)
Write out the parent processes (works)
The parent process of dpkg should not notice a difference and not fail like above (does not work yet).
Any idea how to achieve this?

I wrote a simple python script which solves this:
https://github.com/guettli/wrap_and_log_calls
Wrapper to log all calls to a linux command
particular use case: My configuration management tool calls
/usr/bin/dpkg. An error occurs, but unfortunately my configuration
management tool does not show me the whole stdout/stderr. I have no
clue what's wrong.
General use case: Wrap a linux command like /usr/bin/dpkg and write
out all calls to this.

How can I use an executable jar file with a mainClass in python? [duplicate]

I have been looking for an answer for how to execute a java jar file through python and after looking at:
Execute .jar from Python
How can I get my python (version 2.5) script to run a jar file inside a folder instead of from command line?
How to run Python egg files directly without installing them?
I tried to do the following (both my jar and python file are in the same directory):
import os
if __name__ == "__main__":
os.system("java -jar Blender.jar")
and
import subprocess
subprocess.call(['(path)Blender.jar'])
Neither have worked. So, I was thinking that I should use Jython instead, but I think there must a be an easier way to execute jar files through python.
Do you have any idea what I may do wrong? Or, is there any other site that I study more about my problem?

I would use subprocess this way:
import subprocess
subprocess.call(['java', '-jar', 'Blender.jar'])
But, if you have a properly configured /proc/sys/fs/binfmt_misc/jar you should be able to run the jar directly, as you wrote.
So, which is exactly the error you are getting?
Please post somewhere all the output you are getting from the failed execution.

This always works for me:
from subprocess import *
def jarWrapper(*args):
process = Popen(['java', '-jar']+list(args), stdout=PIPE, stderr=PIPE)
ret = []
while process.poll() is None:
line = process.stdout.readline()
if line != '' and line.endswith('\n'):
ret.append(line[:-1])
stdout, stderr = process.communicate()
ret += stdout.split('\n')
if stderr != '':
ret += stderr.split('\n')
ret.remove('')
return ret
args = ['myJarFile.jar', 'arg1', 'arg2', 'argN'] # Any number of args to be passed to the jar file
result = jarWrapper(*args)
print result

I used the following way to execute tika jar to extract the content of a word document. It worked and I got the output also. The command I'm trying to run is "java -jar tika-app-1.24.1.jar -t 42250_EN_Upload.docx"
from subprocess import PIPE, Popen
process = Popen(['java', '-jar', 'tika-app-1.24.1.jar', '-t', '42250_EN_Upload.docx'], stdout=PIPE, stderr=PIPE)
result = process.communicate()
print(result[0].decode('utf-8'))
Here I got result as tuple, hence "result[0]". Also the string was in binary format (b-string). To convert it into normal string we need to decode with 'utf-8'.

With args: concrete example using Closure Compiler (https://developers.google.com/closure/) from python
import os
import re
src = test.js
os.execlp("java", 'blablabla', "-jar", './closure_compiler.jar', '--js', src, '--js_output_file', '{}'.format(re.sub('.js$', '.comp.js', src)))
(also see here When using os.execlp, why `python` needs `python` as argv[0])

How about using os.system() like:
os.system('java -jar blabla...')
os.system(command)
Execute the command (a string) in a subshell. This is implemented by calling the Standard C function system(), and has the same limitations. Changes to sys.stdin, etc. are not reflected in the environment of the executed command.

How to check whether a shell command returned nothing or something

I am writing a script to extract something from a specified path. I am returning those values into a variable. How can i check whether the shell command has returned something or nothing.
My Code:
def any_HE():
global config, logger, status, file_size
config = ConfigParser.RawConfigParser()
config.read('config2.cfg')
for section in sorted(config.sections(), key=str.lower):
components = dict() #start with empty dictionary for each section
#Retrieving the username and password from config for each section
if not config.has_option(section, 'server.user_name'):
continue
env.user = config.get(section, 'server.user_name')
env.password = config.get(section, 'server.password')
host = config.get(section, 'server.ip')
print "Trying to connect to {} server.....".format(section)
with settings(hide('warnings', 'running', 'stdout', 'stderr'),warn_only=True, host_string=host):
try:
files = run('ls -ltr /opt/nds')
if files!=0:
print '{}--Something'.format(section)
else:
print '{} --Nothing'.format(section)
except Exception as e:
print e
I tried checking 1 or 0 and True or false but nothing seems to be working. In some servers, the path '/opt/nds/' does not exist. So in that case, nothing will be there on files. I wanted to differentiate between something returned to files and nothing returned to files.

First, you're hiding stdout.
If you get rid of that you'll get a string with the outcome of the command on the remote host. You can then split it by os.linesep (assuming same platform), but you should also take care of other things like SSH banners and colours from the retrieved outcome.

As perror commented already, the python subprocess module offers the right tools.
https://docs.python.org/2/library/subprocess.html
For your specific problem you can use the check_output function.
The documentation gives the following example:
import subprocess
subprocess.check_output(["echo", "Hello World!"])
gives "Hello World"

plumbum is a great library for running shell commands from a python script. E.g.:
from plumbum.local import ls
from plumbum import ProcessExecutionError
cmd = ls['-ltr']['/opt/nds'] # construct the command
try:
files = cmd().splitlines() # run the command
if ...:
print ...:
except ProcessExecutionError:
# command exited with a non-zero status code
...
On top of this basic usage (and unlike the subprocess module), it also supports things like output redirection and command pipelining, and more, with easy, intuitive syntax (by overloading python operators, such as '|' for piping).

In order to get more control of the process you run, you need to use the subprocess module.
Here is an example of code:
import subprocess
task = subprocess.Popen(['ls', '-ltr', '/opt/nds'], stdout=subprocess.PIPE)
print task.communicate()

python copy file in local network (linux -> linux) and output

I'm trying to write a script to copy files in my RaspberryPi, from my Desktop PC.
Here is my code: (a part)
print "start the copy"
path_pi = '//192.168.2.2:22/home/pi/Stock/'
file_pc = path_file + "/" + file
print "the file to copy is: ", file_pc
shutil.copy2(file_pc, path_pi + file_pi)
Actually I have this error: (in french)
IOError: [Errno 2] Aucun fichier ou dossier de ce type: '//192.168.2.2:22/home/pi/Stock/exemple.txt'
So, how could I proceed? Must I connect the 2 machines before trying to copy?
I have tryed with:
path_pi = r'//192.168.2.2:22/home/pi/Stock'
But the problem is the same. (And file_pc is a variable)
Thanks
Edit:
Ok, I found this:
command = 'scp', file_pc, file_pi
p = subprocess.Popen(command, stdout=subprocess.PIPE)
But no way to have the output... (work with Shell=False)

shutil.copy2() works with local files. 192.168.2.2:22 suggests that you want to copy files over ssh. You could mount the remote directory (RaspberryPi) onto a local directory on your desktop machine (sshfs) so that shutil.copy2() would work.
If you want to see the output of a command then don't set stdout=PIPE (note: if you set stdout=PIPE then you should read from p.stdout otherwise the process may block forever):
from subprocess import check_call
check_call(['scp', file_pc, file_pi])
scp will print to whatever places your parent Python script prints.
To get the output as a string:
from subprocess import check_output
output = check_output(['scp', file_pc, file_pi])
Though It looks like scp doesn't print anything by default if the output is redirected.
You could use pexpect to make scp think that it runs in a terminal:
import pipes
import re
import pexpect # $ pip install pexpect
def progress(locals):
# extract percents
print(int(re.search(br'(\d+)%[^%]*$', locals['child'].after).group(1)))
command = "scp %s %s" % tuple(map(pipes.quote, [file_pc, file_pi]))
status = pexpect.run(command, events={r'\d+%': progress}, withexitstatus=1)[1]
print("Exit status %d" % status)

Do you have SSH enabled? Something like this could help you:
import os
os.system("scp FILE USER#SERVER:PATH")