I'm making a script to take pictures and write them to a folder created/named with the "data&time"
I made this part to create the directory and take the pictures
pathtoscript = "/home/pi/python-scripts"
current_time = time.localtime()[0:6]
dirfmt = "%4d-%02d-%02d-%02d-%02d-%02d"
dirpath = os.path.join(pathtoscript , dirfmt)
dirname = dirpath % current_time[0:6] #dirname created with date and time
os.mkdir(dirname) #mkdir
pictureName = dirname + "/image%02d.jpg" #path+name of pictures
camera.capture_sequence([pictureName % i for i in range(9)])
Then I would like to pass the dirname to a bash script (picturesToServer) which uploads the pictures to a server.
How can I do it?
cmd = '/home/pi/python-scripts/picturesToServer >/dev/null 2>&1 &'
call ([cmd], shell=True)
Maybe I could stay in the python script scp the pictures to the server? I have a ssh-agent with the paraphrase set (ssh-add mykey).
Place the variable in the environment (it'll be available as a regular bash variable in the bash script, e.g. as VAR_NAME in the example below) by replacing your call with:
import subprocess
p = subprocess.Popen(cmd, shell=True, env={"VAR_NAME": dirname})
Or pass it as a positional argument (it'll be available in $1 in the script) by replacing your cmd with:
cmd = '/home/pi/python-scripts/picturesToServer >/dev/null 2>&1 "{0}" &'.format(dirname)
As a side note, consider not using shell = True when you call a subprocess. Using shell = True is a bad idea for a lot of reasons that are documented in the Python docs
Related
I have been on this problem for quite a while now. I have this command line that I want run trough python:
Users\name.lastname\Desktop\TESTER\Latitude 5431\Latitude-5431-46KCM_Win10_1.0_A01.exe /s /e=C:Users\name.lastname\Desktop\TESTER\Latitude 5431
this should run the .exe and then extract the files to the specified folder. I tried this with os.system and it worked but when I run it with
import subprocess
x = '"' + "\\Users\\name.lastname\\Desktop\\TESTER\\Latitude 5431\\Latitude-5431-46KCM_Win10_1.0_A01.exe" + '" ' + "/s /e=C:Users\\name.lastname\\Desktop\\TESTER\\Latitude 5431"
p1 = subprocess.run(x, shell=True)
it only shows me 'tips' like these but no error message and the .exe is not executed.
Pass command line arguments directly to vendor installer.
Turn the return code to success if required
Latitude-5431-46KCM_Win10_1.0_A01.exe /factoryinstall /passthrough D:\Sample.xml C:\log\FI.log
Change from the default log location to C:\my path with spaces\log.txt
Latitude-5431-46KCM_Win10_1.0_A01.exe /l="C:\my path with spaces\log.txt"
Force update to continue, even on "soft" qualification errors
Latitude-5431-46KCM_Win10_1.0_A01.exe /s /f
Try running without shell=True as it makes things more complicated than it helps:
import subprocess
prog = r"C:\Users\name.lastname\Desktop\TESTER\Latitude 5431\Latitude-5431-46KCM_Win10_1.0_A01.exe"
args = ["/s", r"/e=C:\Users\name.lastname\Desktop\TESTER\Latitude 5431"]
subprocess.run([prog]+args)
I want to execute the commands in a python script :
Open cmd as administrator
cd C:\elastic_stack\logstash-7.6.2
.\bin\logstash -f C:/Users/Asus/Desktop/flask_project_part2/project/logstash_file.conf
This what I'm trying to do but it doesn't execute the last config file :
import os, subprocess
from subprocess import *
os.chdir("C:\\Users")
cmd = subprocess.Popen(["runas", "/noprofile", "/user:Administrator", "|", "cd", "C:/elastic_stack/logstash-7.6.2"], shell=True)
cmd.subprocess.run(["./bin/logstash", "-f", "C:/Users/Asus/Desktop/flask_project_part2/project/logstash_file.conf"], shell=True)
You have a combination of forward and backward slashes, but I am assuming that you have installed ELK on a windows machine:
Unfortunately I do not have access to a windows machine, so I didn't get a chance to test the code. But mainly it should be like this:
import os
import subprocess
# desired path
target_dir = os.path.join("C:",os.sep,"elastic_stack",os.sep,"logstash-7.6.2")
# small check
if os.path.isdir(target_dir):
os.chdir(target_dir)
else:
print(" pathname does not refer to an existing directory")
# current working directory
print(os.getcwd())
# start logstash directly os.system will return the return code of the command if it's 0 means OK
os.system(".\\bin\\logstash -f C:\\Users\\Asus\\Desktop\\flask_project_part2\\project\\logstash_file.conf")
# if you need the output after you started logstash it will work ONLY in Python3
process = subprocess.run([".\\bin\\logstash", "-f", "C:\\Users\\Asus\\Desktop\\flask_project_part2\\project\\logstash_file.conf"], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
output = process.stdout
errors = process.stderr
PS:
You can os.sep when constructing absolute or relative paths because in this way the separator is system agnostic.
shell argument default is set to False and it means there is no system shell started up, and if shell=True means system shell will first spin up.
I write lots of small scripts to manipulate files on a Bash-based server. I would like to have a mechanism by which to log which commands created which files in a given directory. However, I don't just want to capture every input command, all the time.
Approach 1: a wrapper script that uses a Bash builtin (a la history or fc -ln -1) to grab the last command and write it to a log file. I have not been able to figure out any way to do this, as the shell builtin commands do not appear to be recognized outside of the interactive shell.
Approach 2: a wrapper script that pulls from ~/.bash_history to get the last command. This, however, requires setting up the Bash shell to flush every command to history immediately (as per this comment) and seems also to require that the history be allowed to grow inexorably. If this is the only way, so be it, but it would be great to avoid having to edit the ~/.bashrc file on every system where this might be implemented.
Approach 3: use script. My problem with this is that it requires multiple commands to start and stop the logging, and because it launches its own shell it is not callable from within another script (or at least, doing so complicates things significantly).
I am trying to figure out an implementation that's of the form log_this.script other_script other_arg1 other_arg2 > file, where everything after the first argument is logged. The emphasis here is on efficiency and minimizing syntax overhead.
EDIT: iLoveTux and I both came up with similar solutions. For those interested, my own implementation follows. It is somewhat more constrained in its functionality than the accepted answer, but it also auto-updates any existing logfile entries with changes (though not deletions).
Sample usage:
$ cmdlog.py "python3 test_script.py > test_file.txt"
creates a log file in the parent directory of the output file with the following:
2015-10-12#10:47:09 test_file.txt "python3 test_script.py > test_file.txt"
Additional file changes are added to the log;
$ cmdlog.py "python3 test_script.py > test_file_2.txt"
the log now contains
2015-10-12#10:47:09 test_file.txt "python3 test_script.py > test_file.txt"
2015-10-12#10:47:44 test_file_2.txt "python3 test_script.py > test_file_2.txt"
Running on the original file name again changes the file order in the log, based on modification time of the files:
$ cmdlog.py "python3 test_script.py > test_file.txt"
produces
2015-10-12#10:47:44 test_file_2.txt "python3 test_script.py > test_file_2.txt"
2015-10-12#10:48:01 test_file.txt "python3 test_script.py > test_file.txt"
Full script:
#!/usr/bin/env python3
'''
A wrapper script that will write the command-line
args associated with any files generated to a log
file in the directory where the files were made.
'''
import sys
import os
from os import listdir
from os.path import isfile, join
import subprocess
import time
from datetime import datetime
def listFiles(mypath):
"""
Return relative paths of all files in mypath
"""
return [join(mypath, f) for f in listdir(mypath) if
isfile(join(mypath, f))]
def read_log(log_file):
"""
Reads a file history log and returns a dictionary
of {filename: command} entries.
Expects tab-separated lines of [time, filename, command]
"""
entries = {}
with open(log_file) as log:
for l in log:
l = l.strip()
mod, name, cmd = l.split("\t")
# cmd = cmd.lstrip("\"").rstrip("\"")
entries[name] = [cmd, mod]
return entries
def time_sort(t, fmt):
"""
Turn a strftime-formatted string into a tuple
of time info
"""
parsed = datetime.strptime(t, fmt)
return parsed
ARGS = sys.argv[1]
ARG_LIST = ARGS.split()
# Guess where logfile should be put
if (">" or ">>") in ARG_LIST:
# Get position after redirect in arg list
redirect_index = max(ARG_LIST.index(e) for e in ARG_LIST if e in ">>")
output = ARG_LIST[redirect_index + 1]
output = os.path.abspath(output)
out_dir = os.path.dirname(output)
elif ("cp" or "mv") in ARG_LIST:
output = ARG_LIST[-1]
out_dir = os.path.dirname(output)
else:
out_dir = os.getcwd()
# Set logfile location within the inferred output directory
LOGFILE = out_dir + "/cmdlog_history.log"
# Get file list state prior to running
all_files = listFiles(out_dir)
pre_stats = [os.path.getmtime(f) for f in all_files]
# Run the desired external commands
subprocess.call(ARGS, shell=True)
# Get done time of external commands
TIME_FMT = "%Y-%m-%d#%H:%M:%S"
log_time = time.strftime(TIME_FMT)
# Get existing entries from logfile, if present
if LOGFILE in all_files:
logged = read_log(LOGFILE)
else:
logged = {}
# Get file list state after run is complete
post_stats = [os.path.getmtime(f) for f in all_files]
post_files = listFiles(out_dir)
# Find files whose states have changed since the external command
changed = [e[0] for e in zip(all_files, pre_stats, post_stats) if e[1] != e[2]]
new = [e for e in post_files if e not in all_files]
all_modded = list(set(changed + new))
if not all_modded: # exit early, no need to log
sys.exit(0)
# Replace files that have changed, add those that are new
for f in all_modded:
name = os.path.basename(f)
logged[name] = [ARGS, log_time]
# Write changed files to logfile
with open(LOGFILE, 'w') as log:
for name, info in sorted(logged.items(), key=lambda x: time_sort(x[1][1], TIME_FMT)):
cmd, mod_time = info
if not cmd.startswith("\""):
cmd = "\"{}\"".format(cmd)
log.write("\t".join([mod_time, name, cmd]) + "\n")
sys.exit(0)
You can use the tee command, which stores its standard input to a file and outputs it on standard output. Pipe the command line into tee, and pipe tee's output into a new invocation of your shell:
echo '<command line to be logged and executed>' | \
tee --append /path/to/your/logfile | \
$SHELL
i.e., for your example of other_script other_arg1 other_arg2 > file,
echo 'other_script other_arg1 other_arg2 > file' | \
tee --append /tmp/mylog.log | \
$SHELL
If your command line needs single quotes, they need to be escaped properly.
OK, so you don't mention Python in your question, but it is tagged Python, so I figured I would see what I could do. I came up with this script:
import sys
from os.path import expanduser, join
from subprocess import Popen, PIPE
def issue_command(command):
process = Popen(command, stdout=PIPE, stderr=PIPE, shell=True)
return process.communicate()
home = expanduser("~")
log_file = join(home, "command_log")
command = sys.argv[1:]
with open(log_file, "a") as fout:
fout.write("{}\n".format(" ".join(command)))
out, err = issue_command(command)
which you can call like (if you name it log_this and make it executable):
$ log_this echo hello world
and it will put "echo hello world" in a file ~/command_log, note though that if you want to use pipes or redirection you have to quote your command (this may be a real downfall for your use case or it may not be, but I haven't figured out how to do this just yet without the quotes) like this:
$ log_this "echo hello world | grep h >> /tmp/hello_world"
but since it's not perfect, I thought I would add a little something extra.
The following script allows you to specify a different file to log your commands to as well as record the execution time of the command:
#!/usr/bin/env python
from subprocess import Popen, PIPE
import argparse
from os.path import expanduser, join
from time import time
def issue_command(command):
process = Popen(command, stdout=PIPE, stderr=PIPE, shell=True)
return process.communicate()
home = expanduser("~")
default_file = join(home, "command_log")
parser = argparse.ArgumentParser()
parser.add_argument("-f", "--file", type=argparse.FileType("a"), default=default_file)
parser.add_argument("-p", "--profile", action="store_true")
parser.add_argument("command", nargs=argparse.REMAINDER)
args = parser.parse_args()
if args.profile:
start = time()
out, err = issue_command(args.command)
runtime = time() - start
entry = "{}\t{}\n".format(" ".join(args.command), runtime)
args.file.write(entry)
else:
out, err = issue_command(args.command)
entry = "{}\n".format(" ".join(args.command))
args.file.write(entry)
args.file.close()
You would use this the same way as the other script, but if you wanted to specify a different file to log to just pass -f <FILENAME> before your actual command and your log will go there, and if you wanted to record the execution time just provide the -p (for profile) before your actual command like so:
$ log_this -p -f ~/new_log "echo hello world | grep h >> /tmp/hello_world"
I will try to make this better, but if you can think of anything else this could do for you, I am making a github project for this where you can submit bug reports and feature requests.
I have a problem with a backup script which is supposed to call a bash starting/stopping script, in which a "daemon" (via GNU screen) is managed. For the moment my python backup script is called via cron. Within the launch.sh script there is a determination of the given parameter. If "stop" is given the script echos "Stopping..." and runs the GNU screen command to shut down the session. The same goes for "start". If the script is called via subprocess.call(...,Shell=True) in Python the string is shown but the screen session remains untouched. If it gets called directly in bash everything works fine.
#!/usr/bin/env python
'''
Created on 27.07.2013
BackUp Script v0.2
#author: Nerade
'''
import time
import os
from datetime import date
from subprocess import check_output
import subprocess
script_dir = '/home/minecraft/automated_backup'
#folders = ['/home/minecraft/staff']
folders = ['/home/minecraft/bspack2','/home/minecraft/staff']
# log = 0
backup_date = date.today()
backup_dir = '/home/minecraft/automated_backup/' + backup_date.isoformat()
def main():
global log
init_log()
init_dirs()
for folder in folders:
token = folder.split("/")
stopCmd = folder + '/launch.sh stop'
log.write("Stopping server %s...\n" % (token[3]))
subprocess.call(stopCmd,shell=True)
#print stopCmd
while screen_present(token[3]):
time.sleep(0.5)
log.write("Server %s successfully stopped!\n" % (token[3]))
specificPath = backup_dir + '/' + token[3]
os.makedirs(specificPath)
os.system("cp /home/minecraft/%s/server.log %s/server.log" % (token[3],specificPath))
backup(folder,specificPath + '/' + backup_date.isoformat() + '.tar.gz')
dumpDatabase(backup_dir)
for folder in folders:
token = folder.split("/")
startCmd = folder + '/launch.sh start'
log.write("Starting server %s...\n" % (token[3]))
subprocess.call(startCmd,shell=True)
time.sleep(1)
log.write(screen_present(token[3]))
#print startCmd
def dumpDatabase(target):
global log
log.write("Dumping Database...\n")
cmd = "mysqldump -uroot -p<password> -A --quick --result-file=%s/%s.sql" % (backup_dir,backup_date.isoformat())
os.system(cmd)
#print cmd
def backup(source,target):
global log
log.write("Starting backup of folder %s to %s\n" % (source,target))
cmd = 'tar cfvz %s --exclude-from=%s/backup.conf %s' % (target,source,source)
os.system(cmd)
#print cmd
def screen_present(name):
var = check_output(["screen -ls; true"],shell=True)
if "."+name+"\t(" in var:
return True
else:
return False
def init_log():
global log
log = open("%s/backup.log" % script_dir,'a')
log.write(
"Starting script at %s\n" % time.strftime("%m/%d/%Y %H:%M:%S")
)
def init_dirs():
global backup_dir,log
log.write("Checking and creating directories...\n")
if not os.path.isdir(backup_dir):
os.makedirs(backup_dir)
if __name__ == '__main__':
main()
And the launch.sh:
#!/bin/sh
if [ $# -eq 0 ] || [ "$1" = "start" ]; then
echo "Starting Server bspack2"
screen -S bspack2 -m -d java -Xmx5G -Xms4G -jar mcpc-plus-legacy-1.4.7-R1.1.jar nogui
fi
if [ "$1" = "stop" ]; then
screen -S bspack2 -X stuff 'stop\015'
echo "Stopping Server bspack2"
fi
What's my problem here?
I'm sure by now you've solved this problem, but looking through your question I'd bet the answer is remarkably simple -- mcpc-plus-legacy-1.4.7-R1.1.jar isn't found by java, which fails, and subsequently screen terminates.
In launch.sh, screen will execute in the same directory as the calling script. In this case, your python script, when run by cron, will have an active directory of the running user's home directory (so root crontabs will run in /root/, for instance, and a user crontab in /home/username/).
Simple solution is just to the following:
cd /home/minecraft/bspack2
as the second line in your launch.sh script, just after #!/bash/sh.
In the future, I'd recommend when interacting with screen to leverage the -L parameter. This turns on autologging. By default, in the current directory a file "screenlog.0" will be generated when screen terminates, showing you a log history of activity during the screen session. This will allow you to debug screen problems with ease, and help encourage keeping track of "current directory" while working with shell scripts, to make finding the screen log output simple.
I'm running a script from crontab that will just ssh and run a command and store the results in a file.
The function that seems to be failing is subprocess.popen.
Here is the python function:
def _executeSSHCommand(sshcommand,user,node):
'''
Simple function to execute an ssh command on a remote node.
'''
sshunixcmd = '/usr/bin/ssh %s#%s \'%s\'' % (user,node,sshcommand)
process = subprocess.Popen([sshunixcmd],
shell=True,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
process.wait()
result = process.stdout.readlines()
return result
When it's run from the command line, it executes correctly, from cron it seems to fail with the error message below.
Here are the crontab entries:
02 * * * * /home/matt/scripts/check-diskspace.py >> /home/matt/logs/disklog.log
Here are the errors:
Sep 23 17:02:01 timmy CRON[13387]: (matt) CMD (/home/matt/scripts/check-diskspace.py >> /home/matt/logs/disklog.log)
Sep 23 17:02:01 timmy CRON[13386]: (CRON) error (grandchild #13387 failed with exit status 2)
I'm going blind trying to find exactly where I have gone so wrong. Any ideas?
The cron PATH is very limited. You should either set absolute path to your ssh /usr/bin/ssh or set the PATH as a first line in your crontab.
You probably need to pass ssh the -i argument to tell ssh to use a specific key file. The problem is that your environment is not set up to tell ssh which key to use.
The fact that you're using python here is a bit of a red herring.
For everything ssh-related in python, you might consider using paramiko. Using it, the following code should do what you want.
import paramiko
client = paramiko.SSHClient()
client.load_system_host_keys()
client.connect(node, username=user)
stdout = client.exec_command(ssh_command)[0]
return stdout.readlines()
When running python scripts from cron, the environment PATH can be a hangup, as user1652558 points out.
To expand on this answer with example code to add custom PATH values to the environment for a subprocess call:
import os
import subprocess
#whatever user PATH values you need
my_path = "/some/custom/path1:/some/custom/path2"
#append the custom values to the current PATH settings
my_env = os.environ.copy()
my_env["PATH"] = my_path + ":" + my_env["PATH"]
#subprocess call
resp = subprocess.check_output([cmd], env=my_env, shell=True)