Somebody came up with the brilliant idea of putting spaces in a filename. I need to do scp from python using that filename, which is problematic because the shell parses the command, and scp has also some quircks regarding spaces. This is my test code:
import subprocess
import shlex
def split_it(command):
return shlex.split(command)
#return command.split(" ")
def upload_file(localfile, host, mypath):
command = split_it('scp {} {}:"{}"'.format(localfile, host, mypath))
print(command)
res = subprocess.run(command, stdout=subprocess.PIPE)
return res.stdout.decode()
upload_file("localfile.txt", "hostname", "/some/directory/a file with spaces.txt")
Which gives:
['scp', 'localfile.txt', 'hostname:/some/directory/a file with spaces.txt']
scp: ambiguous target
Using the naive version with command.split(" "):
['scp', 'localfile.txt', 'hostname:"/some/directory/a', 'file', 'with', 'spaces.txt"']
spaces.txt": No such file or directory
The right, working scp command would be:
['scp', 'localfile.txt', 'hostname:"/some/directory/a file with spaces.txt"']
Is there a ready solution for this?
If not, what would be the robust way of doing:
split_it('scp localfile.txt hostname:"/some/directory/a file with spaces.txt"')
# returns ['scp', 'localfile.txt', 'hostname:"/some/directory/a file with spaces.txt"']
command = split_it('scp {} {}:"{}"'.format(localfile, host, mypath))
Instead of building a command string, only to split_it again, directly build a list of arguments.
In order to add one layer of quoting to the remote file path, use shlex.quote (or pipes.quote if using older Python versions).
command = ['scp', localfile, '{}:{}'.format(host, shlex.quote(mypath))]
Sources/related posts:
How to escape spaces in path during scp copy in linux?
Python scp copy file with spaces in filename
https://docs.python.org/3/library/subprocess.html#popen-constructor
Related
I am new to python programming, I am trying to build a script that will take Casandra metadata backup.
My script is working fine when there is authentication configured in yaml file but it failed when we turned on authentication.
This is the part where I am calling CQLSH.
with open(save_path + '/' + filename, 'w') as f:
query_process = subprocess.Popen(['echo', query], stdout=subprocess.PIPE)
cqlsh = subprocess.Popen(('/bin/cqlsh', host),
stdin=query_process.stdout, stdout=f)
query_process.stdout.close()
return (save_path + filename)
It will be really helpful for me if anyone can help.
Depending on your configuration and deployment there are a couple of options.
You might just choose to pass them as command line options to your popen command.
Another alternative is to have them in a cqlshrc file, which is either read from the standard location (~/.cassandra/cqlshrc), or an alternative path passed as another command line option.
I want to redirect o/p of shell commands to file using variable "path" but it is not working
import os, socket, shutil, subprocess
host = os.popen("hostname -s").read().strip()
path = "/root/" + host
if os.path.exists(path):
print(path, "Already exists")
else:
os.mkdir("Directory", path , "Created")
os.system("uname -a" > path/'uname') # I want to redirect o/p of shell commands to file using varibale "path" but it is not working
os.system("df -hP"> path/'df')
I think the problem is the bare > and / symbols in the os.system command...
Here is a python2.7 example with os.system that does what you want
import os
path="./test_dir"
command_str="uname -a > {}/uname".format(path)
os.system(command_str)
Here's a very minimal example using subprocess.run. Also, search StackOverflow for "python shell redirect", and you'll get this result right away:
Calling an external command in Python
import subprocess
def run(filename, command):
with open(filename, 'wb') as stdout_file:
process = subprocess.run(command, stdout=subprocess.PIPE, shell=True)
stdout_file.write(process.stdout)
return process.returncode
run('test_out.txt', 'ls')
I have been looking for an answer for how to execute a java jar file through python and after looking at:
Execute .jar from Python
How can I get my python (version 2.5) script to run a jar file inside a folder instead of from command line?
How to run Python egg files directly without installing them?
I tried to do the following (both my jar and python file are in the same directory):
import os
if __name__ == "__main__":
os.system("java -jar Blender.jar")
and
import subprocess
subprocess.call(['(path)Blender.jar'])
Neither have worked. So, I was thinking that I should use Jython instead, but I think there must a be an easier way to execute jar files through python.
Do you have any idea what I may do wrong? Or, is there any other site that I study more about my problem?
I would use subprocess this way:
import subprocess
subprocess.call(['java', '-jar', 'Blender.jar'])
But, if you have a properly configured /proc/sys/fs/binfmt_misc/jar you should be able to run the jar directly, as you wrote.
So, which is exactly the error you are getting?
Please post somewhere all the output you are getting from the failed execution.
This always works for me:
from subprocess import *
def jarWrapper(*args):
process = Popen(['java', '-jar']+list(args), stdout=PIPE, stderr=PIPE)
ret = []
while process.poll() is None:
line = process.stdout.readline()
if line != '' and line.endswith('\n'):
ret.append(line[:-1])
stdout, stderr = process.communicate()
ret += stdout.split('\n')
if stderr != '':
ret += stderr.split('\n')
ret.remove('')
return ret
args = ['myJarFile.jar', 'arg1', 'arg2', 'argN'] # Any number of args to be passed to the jar file
result = jarWrapper(*args)
print result
I used the following way to execute tika jar to extract the content of a word document. It worked and I got the output also. The command I'm trying to run is "java -jar tika-app-1.24.1.jar -t 42250_EN_Upload.docx"
from subprocess import PIPE, Popen
process = Popen(['java', '-jar', 'tika-app-1.24.1.jar', '-t', '42250_EN_Upload.docx'], stdout=PIPE, stderr=PIPE)
result = process.communicate()
print(result[0].decode('utf-8'))
Here I got result as tuple, hence "result[0]". Also the string was in binary format (b-string). To convert it into normal string we need to decode with 'utf-8'.
With args: concrete example using Closure Compiler (https://developers.google.com/closure/) from python
import os
import re
src = test.js
os.execlp("java", 'blablabla', "-jar", './closure_compiler.jar', '--js', src, '--js_output_file', '{}'.format(re.sub('.js$', '.comp.js', src)))
(also see here When using os.execlp, why `python` needs `python` as argv[0])
How about using os.system() like:
os.system('java -jar blabla...')
os.system(command)
Execute the command (a string) in a subshell. This is implemented by calling the Standard C function system(), and has the same limitations. Changes to sys.stdin, etc. are not reflected in the environment of the executed command.
Context:
I am modifying a small python script, so that the output of a lldb command that I run from Xcode debugger, will be output to a file and open up in sublime text.
import lldb
import os
import subprocess
def print_to_file(debugger, command, result, dict):
#Change the output file to a path/name of your choice
f=open("/Users/venkat13/pcomponents.txt","w")
debugger.SetOutputFileHandle(f,True);
debugger.HandleCommand(command)
path = "/Users/venkat13/pcomponents.txt"
sublimePath = '/Applications/Sublime\ Text.app'
subprocess.Popen("%s %s" % (sublimePath, path))
def __lldb_init_module (debugger, dict):
debugger.HandleCommand('command script add -f po.print_to_file print_to_file ')
Problem :
This above script is generating the file, but it does not open in sublime text. Where am I going wrong ?
That's because Popen will not directly pass the entire string to the shell unless you specify shell=True (It just seems to me that you expecting the command to work the same way you would typically type on a shell).
Instead use:
subprocess.Popen([sublimePath, path])
Alternatively, (NOT RECOMMENDED):
subprocess.Popen("%s %s" % (sublimePath, path), shell=True)
What I'd like to achieve is the launch of the following shell command:
mysql -h hostAddress -u userName -p userPassword
databaseName < fileName
From within a python 2.4 script with something not unlike:
cmd = ["mysql", "-h", ip, "-u", mysqlUser, dbName, "<", file]
subprocess.call(cmd)
This pukes due to the use of the redirect symbol (I believe) - mysql doesn't receive the input file.
I've also tried:
subprocess.call(cmd, stdin=subprocess.PIPE)
no go there ether
Can someone specify the syntax to make a shell call such that I can feed in a file redirection ?
Thanks in advance.
You have to feed the file into mysql stdin by yourself. This should do it.
import subprocess
...
filename = ...
cmd = ["mysql", "-h", ip, "-u", mysqlUser, dbName]
f = open(filename)
subprocess.call(cmd, stdin=f)
The symbol < has this meaning (i. e. reading a file to stdin) only in shell. In Python you should use either of the following:
1) Read file contents in your process and push it to stdin of the child process:
fd = open(filename, 'rb')
try:
subprocess.call(cmd, stdin=fd)
finally:
fd.close()
2) Read file contents via shell (as you mentioned), but redirect stdin of your process accordingly:
# In file myprocess.py
subprocess.call(cmd, stdin=subprocess.PIPE)
# In shell command line
$ python myprocess.py < filename
As Andrey correctly noticed, the < redirection operator is interpreted by shell. Hence another possible solution:
import os
os.system("mysql -h " + ip + " -u " + mysqlUser + " " + dbName)
It works because os.system passes its argument to the shell.
Note that I assumed that all used variables come from a trusted source, otherwise you need to validate them in order to prevent arbitrary code execution. Also those variables should not contain whitespace (default IFS value) or shell special characters.