I'm writing a small method to replace some text in a file.
The only argument I need is the new text, as it is always the same file and text to be replaced.
I'm having a problem using the os.system() call, when I try to use the argument of the method
If I use a string like below, everything runs ok:
stringId = "GRRRRRRRRR"
cmd="sed '1,$s/MANAGER_ID=[0-9]*/MANAGER_ID=" + stringId + "/g' path/file.old > path/file.new"
os.system(cmd)
Now, if i try to give a string as a parameter like below, the command is not executed.
I do a print to see if the command is correct, and it is. I can even execute it with success if I copy / paste to my shell
import os
def updateExportConfigId(id):
stringId = "%s" % id
cmd= "sed '1,$s/MANAGER_ID=[0-9]*/MANAGER_ID=" + stringId + "/g' path/file.old > path/file.new"
print "command is " + cmd
os.system(cmd)
Does anyone knows what is wrong?
Thanks
Obligatory: don't use os.system - use the subprocess module:
import subprocess
def updateExportConfigId(m_id, source='path/file.old',
destination='path/file.new'):
if isinstance(m_id, unicode):
m_id = m_id.encode('utf-8')
cmd= [
"sed",
",$s/MANAGER_ID=[0-9]*/MANAGER_ID=%s/g" % m_id,
source,
]
subprocess.call(cmd, stdout=open(destination, 'w'))
with this code you can pass the manager id, it can have spaces, quote chars, etc. The file names can also be passed to the function, and can also contain spaces and some other special chars. That's because your shell is not unnecessarly invoked, so one less process is started on your OS, and you don't have to worry on escaping special shell characters.
Another option: Don't launch sed. Use python's re module.
import re
def updateExportConfigID(m_id, source, destination):
if isinstance(m_id, unicode):
m_id = m_id.encode('utf-8')
for line in source:
new_line = re.sub(r'MANAGER_ID=\d*',
r'MANAGER_ID=' + re.escape(m_id),
line)
destination.write(new_line)
and call it like this:
updateExportConfigID('GRRRR', open('path/file.old'), open('path/file.new', 'w'))
No new processes needed.
To help you debug it, try adding:
print repr(cmd)
It might be that some special characters slipped into the command that normal print is hiding when you copy and paste it.
Maybe some indentation problem?
The following works correctly:
import os
def updateExportConfigId(id):
stringId = "%s" % id
cmd= "sed '1,$s/MANAGER_ID=[0-9]*/MANAGER_ID=" + stringId + "/g' test.dat > test.new"
print "command is " + cmd
os.system(cmd)
updateExportConfigId("adsf")
Also do not use reserved words (id) as variables.
What is wrong is that there is some difference. Yeah, I know that's not helpful, but you need to figure out the difference.
Try running this:
import os
def updateExportConfigId(id):
stringId = "%s" % id
cmd1 = "sed '1,$s/MANAGER_ID=[0-9]*/MANAGER_ID=" + stringId + "/g' path/file.old > path/file.new"
stringId = "GRRRRRRRRR"
cmd2 = "sed '1,$s/MANAGER_ID=[0-9]*/MANAGER_ID=" + stringId + "/g' path/file.old > path/file.new"
print "cmd1:" , cmd1
print "cmd2:" , cmd2
print cmd1 == cmd2
updateExportConfigId("GRRRRRRRRR")
The code should print:
sed '1,$s/MANAGER_ID=[0-9]*/MANAGER_ID=GRRRRRRRRR/g' path/file.old > path/file.new
sed '1,$s/MANAGER_ID=[0-9]*/MANAGER_ID=GRRRRRRRRR/g' path/file.old > path/file.new
True
Thereby showing that they are exactly the same. If the last line is "False" then they are not the same, and you should be able to see the difference.
So from previous answers we now know that id is a Unicode string, which makes cmd1 a Unicode string, which os.system() is converting to a byte string for execution in the default encoding.
a) I suggest using subprocess rather than os.system()
b) I suggest not using the name of a built-in function as a variable (id).
c) I suggest explicitly encoding the string to a byte string before executing:
if isinstance(cmd,unicode):
cmd = cmd.encode("UTF-8")
d) For Lennart Regebro's suggestion add:
assert type(cmd1) == type(cmd2)
after
print cmd1 == cmd2
Maybe it helps to use only raw strings.
Finally, I found a way to run the os.system(cmd)!
Simple trick, to "clean" the cmd string:
os.system(str(cmd))
Now, I'm able to build the cmd with all arguments I need and at the end I just "clean" it with str() call before run it with os.system() call.
Thanks a lot for your answers!
swon
Related
i'm writing a python script to execute shell command, and i'm taking arguments and i want to pass the value of that to the command
#!/usr/bin/env python
import commands
import subprocess
import sys
command = commands.getoutput('fs_cli -x "sofia profile external restart"')
this code works fine
when i try to take the argument and pass to command it fails
command = commands.getoutput('fs_cli -x "sofia profile" + sys.argv[1]
+ " restart"')
supp folks
You should write:
command = commands.getoutput('fs_cli -x "sofia profile ' + sys.argv[1] + ' restart"')
Take a look to argparse and subprocess.
One of the way to do this is to convert your command that you want to execute into string and then execute it as eval()
example :
eval(expression/command in string)
I'm trying to execute 10 python scripts from python code and open each of them in a new shell window.
My code :
for i in range(10):
name_of_file = "myscript"+str(i)+".py"
cmd = "python " + name_of_file
os.system("gnome-terminal -e 'bash -c " + cmd + "'")
But each script file are not executing, I get only the live interpreter of python in the new terminal...
Thank you guys
I would suggest using the subprocess module (https://docs.python.org/2/library/subprocess.html).
In this way, you'll write something like the following:
import subprocess
cmd = ['gnome-terminal', '-x', 'bash', '-c']
for i in range(10):
name_of_file = "myscript"+str(i)+".py"
your_proc = subprocess.Popen(cmd + ['python %s' % (name_of_file)])
# or if you want to use the "modern" way of formatting string you can write
# your_proc = subprocess.Popen(cmd + ['python {}'.format(name_of_file)])
...
and you have more control over the processes you start.
If you want to keep using os.system(), build your command string first, then pass it to the function. In your case would be:
cmd = 'gnome-terminal -x bash -c "python {}"'.format(name_of_file)
os.system(cmd)
something along these lines.
Thanks to #anishsane for some suggestions!
I think that it is to do with the string quoting of the argument to os.system. Try this:
os.system("""gnome-terminal -e 'bash -c "{}"'""".format(cmd))
I'm attempting to execute a command over SSH, but bash on the other end doesn't think it's escaped properly.
Here, self._client is a paramiko.SSHClient object; args is a list of arguments, the command to execute.
def run(self, args, stdin=None, capture_stdout=False):
"""Runs a command.
On success, returns the output, if requested, or None.
On failure, raises CommandError, with stderr and, if captured, stdout,
as well as the exit code.
"""
command = ' '.join(_shell_escape(arg) for arg in args)
print('About to run command:\n {}'.format(command))
print('About to run command:\n {!r}'.format(command))
channel = self._client.get_transport().open_session()
channel.exec_command(command)
_shell_escape:
_SHELL_SAFE = _re.compile(r'^[-A-Za-z0-9_./]+$')
def _shell_escape(s):
if _SHELL_SAFE.match(s):
return s
return '\'{}\''.format(s.replace('\'', '\'\\\'\''))
I'm attempt to run some Python through this. On stderr, I get back:
bash: -c: line 5: unexpected EOF while looking for matching `''
bash: -c: line 6: syntax error: unexpected end of file
The output from the two print statements:
About to run command:
python -c 'import os, sys
path = sys.argv[1]
if sys.version_info.major == 2:
path = path.decode('\''utf-8'\'')
entries = os.listdir(path)
out = b'\'''\''.join(e.encode('\''utf-8'\'') + b'\'''\'' for e in entries)
sys.stdout.write(out)
' .
About to run command:
"python -c 'import os, sys\npath = sys.argv[1]\nif sys.version_info.major == 2:\n path = path.decode('\\''utf-8'\\'')\nentries = os.listdir(path)\nout = b'\\'''\\''.join(e.encode('\\''utf-8'\\'') + b'\\''\x00'\\'' for e in entries)\nsys.stdout.write(out)\n' ."
If I copy and paste the output of command, and paste it into bash, it executes, so it really does appear to be properly escaped. My current understanding is that SSH, on the other end, will take command, and run [my_shell, '-c', command].
Why is bash erroring on that command?
The input contains an embedded nul character, which bash appears to treat as the end of the string. (I'm not sure there's any way it couldn't!). This is visible in my question, where I output command:
About to run command:
"python -c 'import os, sys [SNIP…] + b'\\''\x00'\\'' for [SNIP…]"
That's a repr output, but notice the single slash before the x in \x00: that's an actual \x00 that made it through. My original code has this Python embedded as a snippet, which I didn't include (I didn't believe it was relevant):
_LS_CODE = """\
import os, sys
path = sys.argv[1]
if sys.version_info.major == 2:
path = path.decode('utf-8')
entries = os.listdir(path)
out = b''.join(e.encode('utf-8') + b'\x00' for e in entries)
sys.stdout.write(out)
"""
Here, Python's """ is still processing \ as an escape character. I need to double up, or look into raw strings (r""")
You need to escape newlines as well. A better option is to put the program text in a here document.
Make the output of "About to run command:" to look like
python -c << EOF
import os, sys
path = sys.argv[1]
if sys.version_info.major == 2:
path = path.decode('\''utf-8'\'')
entries = os.listdir(path)
out = b'\'''\''.join(e.encode('\''utf-8'\'') + b'\'''\'' for e in entries)
sys.stdout.write(out)
.
EOF
Maybe you wouldn't need to escape anything at all.
I'm building up several command strings to pass to os.system. I want to group the common stuff into a string then add the specific stuff as needed. For example:
CMD = "python app.py %s -o %s > /dev/null"
option1 = "-i 192.169.0.1"
option2 = "results-file"
cmd = CMD, (option1, option2) #doesn't work
os.system(cmd)
I know cmd is a tuple. How do I get cmd to be the command string I want?
You use the % operator.
cmd = CMD % (option1, option2)
See also: Python documentation on string formatting
You could do it this way using the string format() method which processes Format String Syntax:
CMD = "python app.py {} -o {} > /dev/null"
option1 = "-i 192.169.0.1"
option2 = "results-file"
os.system(CMD.format(option1, option2))
cmd = CMD % (option1, option2)
This is explained here.
What I'd like to achieve is the launch of the following shell command:
mysql -h hostAddress -u userName -p userPassword
databaseName < fileName
From within a python 2.4 script with something not unlike:
cmd = ["mysql", "-h", ip, "-u", mysqlUser, dbName, "<", file]
subprocess.call(cmd)
This pukes due to the use of the redirect symbol (I believe) - mysql doesn't receive the input file.
I've also tried:
subprocess.call(cmd, stdin=subprocess.PIPE)
no go there ether
Can someone specify the syntax to make a shell call such that I can feed in a file redirection ?
Thanks in advance.
You have to feed the file into mysql stdin by yourself. This should do it.
import subprocess
...
filename = ...
cmd = ["mysql", "-h", ip, "-u", mysqlUser, dbName]
f = open(filename)
subprocess.call(cmd, stdin=f)
The symbol < has this meaning (i. e. reading a file to stdin) only in shell. In Python you should use either of the following:
1) Read file contents in your process and push it to stdin of the child process:
fd = open(filename, 'rb')
try:
subprocess.call(cmd, stdin=fd)
finally:
fd.close()
2) Read file contents via shell (as you mentioned), but redirect stdin of your process accordingly:
# In file myprocess.py
subprocess.call(cmd, stdin=subprocess.PIPE)
# In shell command line
$ python myprocess.py < filename
As Andrey correctly noticed, the < redirection operator is interpreted by shell. Hence another possible solution:
import os
os.system("mysql -h " + ip + " -u " + mysqlUser + " " + dbName)
It works because os.system passes its argument to the shell.
Note that I assumed that all used variables come from a trusted source, otherwise you need to validate them in order to prevent arbitrary code execution. Also those variables should not contain whitespace (default IFS value) or shell special characters.