I want to call a subprocess to backup mysql database.
A command line which ran fine in the terminal (and created a file named mydatabase.sql) is:
mysqldump -uroot -ppassword --add-drop-database --database mydatabase > mydatabase.sql
Now the code to be ran by python to call a subprocess:
args = shlex.split('mysqldump -uroot -ppassword --add-drop-database --database mydatabase > mydatabase.sql')
subprocess.check_call(args)
The exeption is raised (no file created):
Traceback (most recent call last):
File "<pyshell#29>", line 1, in <module>
subprocess.check_call(args)
File "/usr/lib/python3.2/subprocess.py", line 485, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['mysqldump', >'-uroot', '-ppassword', '--add-drop-database', '--database', >'mydatabase', '>', 'mydatabase.sql']' returned non-zero exit status 2
I have tried different ways, but they still don't work:
args = shlex.split('/opt/lampp/bin/mysqldump -uroot -ppassword --add-drop-database --database mydatabase > mydatabase.sql')
subprocess.check_call(args)
or
args = shlex.split('/opt/lampp/bin/mysqldump -uroot -ppassword --add-drop-database --database mydatabase > mydatabase.sql')
subprocess.Popen(args)
I also tried with shell=True or or shell=False. In both cases, they still don't work.
I have read the docs, google for the answer for my problem, but I haven't got a clue how to show my problem. stackoverflow is probably my last hope.
the problem here is the way you're redirecting the output.
if you pass the command as a list of arguments, then ">" will always be interpreted as a literal >, no matter if you use shell=True or shell=False
if you pass the command as a single string, then it hould work, but only if you have shell=True.
the best way do do what you want would be to redirect the output to a file directly from python:
args = shlex.split('/opt/lampp/bin/mysqldump -uroot -ppassword --add-drop-database --database mydatabase')
output = open("mydatabase.sql", "w")
subprocess.Popen(args, stdout=output)
The problem may be the shell redirection. If you run with shell=True don't use shlex.split. In other words, try:
args = '/opt/lampp/bin/mysqldump -uroot -ppassword --add-drop-database --database mydatabase > mydatabase.sql'
subprocess.Popen(args,shell=True)
Of course, the safer solution would be to remove the shell redirection, use shlex.split on the arguments (without shell=True), and use subprocess.PIPE to capture the output (which you can then send to a file or do whatever you want with in your program)
For example:
args = '/opt/lampp/bin/mysqldump -uroot -ppassword --add-drop-database --database mydatabase'
p=psubprocess.Popen(shlex.split(args),shell=False,stdout=subprocess.PIPE)
p.wait()
returnvalue=p.returncode
data=p.stdout.read()
with open('mydatabase.sql','w') as f:
f.write(data)
...
If you don't want to do anything with the data in your program, you can do the redirection a little more easily as described by mata.
Related
I'm trying to import a json file with the command : mongoimport -c collection -d db < file.json. When I call this command from the shell, every documents are imported. However, when I try to call it from a python script with the module subprocess like this : subprocess.call('mongoimport -c collection -d db < file.json', shell=True), I have the following error: Failed: error processing document #37: unexpected EOF. Can someone explain me why does it work with the shell but not when it is called from the script please?
Thanks in advance.
I thing is because subprocess.Popen constructor accepts list of args, not string, like this:
p = subprocess.Popen(['mongoimport', '--db', 'AutoPrivilege', '-c','cars', 'stockvo.json', '--jsonArray', '--upsert','--drop'])...
You can use os command:
for dump_file in dumped_files:
collection_name = dump_file.split('/')[-1].split('.')[0]
# Restoring collection
os.system(f'mongoimport --host {host} -d {database} --port {port} --collection {collection_name} --file {backup_file} --jsonArray --upsert')
I am invoking shell script using os.execvp() in python. my shell script has some echo statements whcih I want to redirect in file.
Here is what I am trying:
cmd = "/opt/rpm/rpm_upgrade.sh >& /opt/rpm/upgrader.log"
cmdline = ["/bin/sh", cmd]
os.execvp(cmdline[0], cmdline)
Below is the error I am getting:
Error: /bin/sh: /opt/rpm/rpm_upgrade.sh >& /opt/rpm/upgrader.log: No such file or directory
Can any one help?
This is happening because you are passing this entire string as if it were the program name to execute:
"/opt/rpm/rpm_upgrade.sh >& /opt/rpm/upgrader.log"
The easy way to fix this is:
cmdline = ["/bin/sh", "/opt/rpm/rpm_upgrade.sh",
">&", "/opt/rpm/upgrader.log"]
os.execvp(cmdline[0], cmdline)
Now sh will receive three arguments rather than one.
Or you can switch to the more full-featured subprocess module, which lets you redirect output in Python:
import subprocess
with open("/opt/rpm/upgrader.log", "wb") as outfile:
subprocess.check_call(["/opt/rpm/rpm_upgrade.sh"], shell=True,
stdout=outfile, stderr=subprocess.STDOUT)
I have to connect to a sybase database and run a simple select query using python script
On my server isql command can run only from sybase bin directory, so i have to cd to that directory before firing the query.
---------------------------Edited-----------------------------
Uptill now i'm able to do this:-
#!/usr/bin/python
import subprocess
path = "path/to/sybase/bin"
os.chdir(path)
arguments = ['./isql',"-S server_name", "-U user", "-P password", "-D database","""<<EOF
SELECT * FROM sometable
go
EOF"""]
ps = subprocess.Popen(arguments)
out = ps.communicate()
print out
The errors are just out of my understanding capability :(
Traceback (most recent call last):
File "./test_db.py", line 8, in ?
ps = subprocess.Popen(arguments)
File "/usr/lib64/python2.4/subprocess.py", line 542, in __init__
errread, errwrite)
File "/usr/lib64/python2.4/subprocess.py", line 975, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
I'm able to do this outside my python script using isql command on my unix terminal
how can i use isql in python subprocess module?
There is a particular Popen argument for that: cwd, as mentioned here. Provide your command using an array and then the cwd parameter with where the command must be executed:
subprocess.Popen(['ls', '-l'], cwd="/path/to/folder")
Popen only takes one args argument, for the command to run. You could try calling a shell with both the cd and isql commands as arguments, but changing the working directory from python is probably simpler
For the former approach:
subprocess.Popen('/bin/sh -c "cd /path/to/... && isql -arg1..'...)
for the latter:
os.chdir('/path/to...')
subprocess.Popen('isql -arg1..'...)
Try:
import os
import subprocess
os.chdir('/path/to/sybase/bin')
if os.path.exists('isql') or os.path.exists(os.path.join('/path/to/sybase/bin', 'isql')):
ps = subprocess.Popen('isql -S %s -U %s -P %s -D %s <<EOF SELECT * FROM sometable EOF' % (server,user,passwd,database), stdout=subprocess.PIPE, shell=True)
out, err = ps.communicate()
else:
print "isql does not exists in this folder"
I am not super experienced with subprocess but this is how I generally use it on the odd occasion. Hopefully someone else can give a better answer/explanation.
Edit: removed the square brackets to remove confusion.
i know it's been long but just wanted to close this question
from subprocess import Popen, PIPE
from textwrap import dedent
isql = Popen(['./isql', '-I', '/app/sybase/...',
'-S', mdbserver,
'-U', muserid,
'-P', password, ...,
'-w', '99999'], stdin=PIPE, stdout=PIPE, cwd=sybase_path)
output = isql.communicate(dedent("""\
SET NOCOUNT ON
{}
go
""".format(User_Query)))[0]
I'm setting up a program to connect my computer to our schools proxy and currently have something like this:
import subprocess
import sys
username = 'fergus.barker'
password = '*************'
proxy = 'proxy.det.nsw.edu.au:8080'
options = '%s:%s#%s' % (username, password, proxy)
subprocess.Popen('export http_proxy=' + options)
But upon running I get:
Traceback (most recent call last):
File "school_proxy_settings.py", line 19, in <module>
subprocess.Popen('export http_proxy=' + options)
File "/usr/lib/python2.6/subprocess.py", line 621, in __init__
errread, errwrite)
File "/usr/lib/python2.6/subprocess.py", line 1126, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
Why is this happening please guys?
The problem is that export is not an actual command or file. It is a built-in command to shells like bash and sh, so when you attempt a subprocess.Popen you will get an exception because it can not find the export command. By default Popen does an os.execvp() to spawn a new process, which would not allow you to use shell intrinsics.
You can do something like this, though you have to change your call to Popen.
http://docs.python.org/library/subprocess.html
You can specify shell=True to make it use shell commands.
class subprocess.Popen(args, bufsize=0, executable=None, stdin=None, stdout=None, stderr=None, preexec_fn=None, close_fds=False, shell=False, cwd=None, env=None, universal_newlines=False, startupinfo=None, creationflags=0)
On Unix, with shell=True: If args is a string, it specifies the command string to execute through the shell. This means that the string must be formatted exactly as it would be when typed at the shell prompt. This includes, for example, quoting or backslash escaping filenames with spaces in them. If args is a sequence, the first item specifies the command string, and any additional items will be treated as additional arguments to the shell itself. That is to say, Popen does the equivalent of:
Popen(['/bin/sh', '-c', args[0], args[1], ...])
export is not a separate binary on your system, it is actually just a command within the shell itself. For example, try using which rm on your system. You'll probably see something like:
[21:16:28] ~ $ which rm
/bin/rm
Now try using which export. You'll get something like:
[21:16:37] ~ $ which export
/usr/bin/which: no export in (/usr/lib/qt-3.3/bin:/usr/kerberos/sbin:
/usr/kerberos/bin:/usr/lib/ccache:/usr/local/bin:/usr/bin:/bin:
/usr/local/sbin:/usr/sbin:/sbin:/home/carter/bin)
So you can't actually invoke an export process/subprocess by default. You may want to look at os.putenv() and os.environ() instead.
What I'd like to achieve is the launch of the following shell command:
mysql -h hostAddress -u userName -p userPassword
databaseName < fileName
From within a python 2.4 script with something not unlike:
cmd = ["mysql", "-h", ip, "-u", mysqlUser, dbName, "<", file]
subprocess.call(cmd)
This pukes due to the use of the redirect symbol (I believe) - mysql doesn't receive the input file.
I've also tried:
subprocess.call(cmd, stdin=subprocess.PIPE)
no go there ether
Can someone specify the syntax to make a shell call such that I can feed in a file redirection ?
Thanks in advance.
You have to feed the file into mysql stdin by yourself. This should do it.
import subprocess
...
filename = ...
cmd = ["mysql", "-h", ip, "-u", mysqlUser, dbName]
f = open(filename)
subprocess.call(cmd, stdin=f)
The symbol < has this meaning (i. e. reading a file to stdin) only in shell. In Python you should use either of the following:
1) Read file contents in your process and push it to stdin of the child process:
fd = open(filename, 'rb')
try:
subprocess.call(cmd, stdin=fd)
finally:
fd.close()
2) Read file contents via shell (as you mentioned), but redirect stdin of your process accordingly:
# In file myprocess.py
subprocess.call(cmd, stdin=subprocess.PIPE)
# In shell command line
$ python myprocess.py < filename
As Andrey correctly noticed, the < redirection operator is interpreted by shell. Hence another possible solution:
import os
os.system("mysql -h " + ip + " -u " + mysqlUser + " " + dbName)
It works because os.system passes its argument to the shell.
Note that I assumed that all used variables come from a trusted source, otherwise you need to validate them in order to prevent arbitrary code execution. Also those variables should not contain whitespace (default IFS value) or shell special characters.