Using wildcards in filename in scp in python - python

I want to execute a simple scp command in a python script, copying files following a certain name pattern.
I'm executing the following command:
filename = '\*last_processed_date\*.txt'
command = ''' scp test#100.41.14.27:/home/test/test2/test3/%s %s '''\
% (filename,self.unprocessed_file_dir)
os.system(command)
I understand that I have to escape the wildcard '*', which I'm doing..but still I get:
scp: /home/test/test2/test3/*last_processed_date*.txt: No such file or directory
I'm wondering what I'm doing wrong..
EDIT:
It was a careless mistake from my side. I should have done:
command = ''' scp 'test#100.41.14.27:/home/test/test2/test3/%s' %s '''
instead of:
command = ''' scp test#100.41.14.27:/home/test/test2/test3/%s %s '''\
% (filename,self.unprocessed_file_dir)

This works on my system:
host = 'test#100.41.14.27'
filename = '*last_processed_date*.txt'
rpath = '/home/test/test2/test3'
lpath = self.unprocessed_file_dir
command = 'scp %s:%s/%s %s' % (host, rpath, filename, lpath)
os.system(command)
If it gives you an error, try this on from the terminal first:
ssh test#100.41.14.27 ls /home/test/test2/test3/*last_processed_date*.txt

Related

Error with python string formatting

I am working on a script that takes a text file containing IP addresses (one per line) and then passes each IP into a non-Python program command.
The result is an error:
TypeError: not all arguments converted during string formatting
import subprocess
list='c:\cmc_list.txt'
with open(list,'r') as cmc_list:
for i in cmc_list:
racadm_command = "racadm -r %s -u root -p calvin getslotname" % i
output = subprocess.Popen(racadm_command % i, stdout = subprocess.PIPE,
shell=True).communicate()[0]
print(racadm_command, output)
The string passed to the Popen command has already been formatted, so it has no % left to consume the i. Take away the "% i" and I think you'll be fine.
I believe you meant to do the formatting only once
import subprocess
cmc_list='c:\cmc_list.txt'
racadm_command_template = "racadm -r {} -u root -p calvin getslotname"
with open(cmc_list,'r') as f:
for ip in f:
cmd = racadm_command_template.format(ip)
output = subprocess.Popen(cmd, stdout = subprocess.PIPE, shell=True).communicate()
print(cmd, output[0])
I also suggest using getpass for your password prompt, or import from an environment variable. Additionally, don't print out the password and please change it from the default

pass python var to bash

I'm making a script to take pictures and write them to a folder created/named with the "data&time"
I made this part to create the directory and take the pictures
pathtoscript = "/home/pi/python-scripts"
current_time = time.localtime()[0:6]
dirfmt = "%4d-%02d-%02d-%02d-%02d-%02d"
dirpath = os.path.join(pathtoscript , dirfmt)
dirname = dirpath % current_time[0:6] #dirname created with date and time
os.mkdir(dirname) #mkdir
pictureName = dirname + "/image%02d.jpg" #path+name of pictures
camera.capture_sequence([pictureName % i for i in range(9)])
Then I would like to pass the dirname to a bash script (picturesToServer) which uploads the pictures to a server.
How can I do it?
cmd = '/home/pi/python-scripts/picturesToServer >/dev/null 2>&1 &'
call ([cmd], shell=True)
Maybe I could stay in the python script scp the pictures to the server? I have a ssh-agent with the paraphrase set (ssh-add mykey).
Place the variable in the environment (it'll be available as a regular bash variable in the bash script, e.g. as VAR_NAME in the example below) by replacing your call with:
import subprocess
p = subprocess.Popen(cmd, shell=True, env={"VAR_NAME": dirname})
Or pass it as a positional argument (it'll be available in $1 in the script) by replacing your cmd with:
cmd = '/home/pi/python-scripts/picturesToServer >/dev/null 2>&1 "{0}" &'.format(dirname)
As a side note, consider not using shell = True when you call a subprocess. Using shell = True is a bad idea for a lot of reasons that are documented in the Python docs

python copy file in local network (linux -> linux) and output

I'm trying to write a script to copy files in my RaspberryPi, from my Desktop PC.
Here is my code: (a part)
print "start the copy"
path_pi = '//192.168.2.2:22/home/pi/Stock/'
file_pc = path_file + "/" + file
print "the file to copy is: ", file_pc
shutil.copy2(file_pc, path_pi + file_pi)
Actually I have this error: (in french)
IOError: [Errno 2] Aucun fichier ou dossier de ce type: '//192.168.2.2:22/home/pi/Stock/exemple.txt'
So, how could I proceed? Must I connect the 2 machines before trying to copy?
I have tryed with:
path_pi = r'//192.168.2.2:22/home/pi/Stock'
But the problem is the same. (And file_pc is a variable)
Thanks
Edit:
Ok, I found this:
command = 'scp', file_pc, file_pi
p = subprocess.Popen(command, stdout=subprocess.PIPE)
But no way to have the output... (work with Shell=False)
shutil.copy2() works with local files. 192.168.2.2:22 suggests that you want to copy files over ssh. You could mount the remote directory (RaspberryPi) onto a local directory on your desktop machine (sshfs) so that shutil.copy2() would work.
If you want to see the output of a command then don't set stdout=PIPE (note: if you set stdout=PIPE then you should read from p.stdout otherwise the process may block forever):
from subprocess import check_call
check_call(['scp', file_pc, file_pi])
scp will print to whatever places your parent Python script prints.
To get the output as a string:
from subprocess import check_output
output = check_output(['scp', file_pc, file_pi])
Though It looks like scp doesn't print anything by default if the output is redirected.
You could use pexpect to make scp think that it runs in a terminal:
import pipes
import re
import pexpect # $ pip install pexpect
def progress(locals):
# extract percents
print(int(re.search(br'(\d+)%[^%]*$', locals['child'].after).group(1)))
command = "scp %s %s" % tuple(map(pipes.quote, [file_pc, file_pi]))
status = pexpect.run(command, events={r'\d+%': progress}, withexitstatus=1)[1]
print("Exit status %d" % status)
Do you have SSH enabled? Something like this could help you:
import os
os.system("scp FILE USER#SERVER:PATH")

mp3info giving an error with variable filename

I have been using mp3info to calculate my file length using the following syntax:
mp3info -p "%S" /path/to/file
whenever I use the code with the filename I get the correct output:
mp3info -p "%S" far_from_love.mp3
However, on storing the filename in a string variable and then using the variable I get an error:
SyntaxError: invalid syntax
Could anyone tell me how to correctly use the command with a variable filename?
this is the python code which uses mp3info
listing=os.listdir("C:\\Python27")
for f in listing:
if fnmatch.fnmatch(f,'*.mp3'):
ext=f[:-4] #extract name of file without extension
WAVE_OUTPUT_FILENAME="%s.wav"%ext
print WAVE_OUTPUT_FILENAME#save output filename as wav extension
print f
x=os.popen('mp3info -p "%S" f).read()
print x
x=os.popen('mp3info -p "%S" f).read()
is missing a closing quote:
x=os.popen('mp3info -p "%S" ' + f).read()
You may also want to use the safer subprocess module:
import subprocess
x = subprocess.check_output(['mp3info', '-p', '%S', f])

Run shell command with input redirections from python 2.4?

What I'd like to achieve is the launch of the following shell command:
mysql -h hostAddress -u userName -p userPassword
databaseName < fileName
From within a python 2.4 script with something not unlike:
cmd = ["mysql", "-h", ip, "-u", mysqlUser, dbName, "<", file]
subprocess.call(cmd)
This pukes due to the use of the redirect symbol (I believe) - mysql doesn't receive the input file.
I've also tried:
subprocess.call(cmd, stdin=subprocess.PIPE)
no go there ether
Can someone specify the syntax to make a shell call such that I can feed in a file redirection ?
Thanks in advance.
You have to feed the file into mysql stdin by yourself. This should do it.
import subprocess
...
filename = ...
cmd = ["mysql", "-h", ip, "-u", mysqlUser, dbName]
f = open(filename)
subprocess.call(cmd, stdin=f)
The symbol < has this meaning (i. e. reading a file to stdin) only in shell. In Python you should use either of the following:
1) Read file contents in your process and push it to stdin of the child process:
fd = open(filename, 'rb')
try:
subprocess.call(cmd, stdin=fd)
finally:
fd.close()
2) Read file contents via shell (as you mentioned), but redirect stdin of your process accordingly:
# In file myprocess.py
subprocess.call(cmd, stdin=subprocess.PIPE)
# In shell command line
$ python myprocess.py < filename
As Andrey correctly noticed, the < redirection operator is interpreted by shell. Hence another possible solution:
import os
os.system("mysql -h " + ip + " -u " + mysqlUser + " " + dbName)
It works because os.system passes its argument to the shell.
Note that I assumed that all used variables come from a trusted source, otherwise you need to validate them in order to prevent arbitrary code execution. Also those variables should not contain whitespace (default IFS value) or shell special characters.

Categories