Have written the below script to delete files in a folder not matching the dates in the "keep" period. Eg. Delete all except files partly matching this name.
The command works from the shell but fails with the subprocess call.
/bin/rm /home/backups/!(*"20170920"*|*"20170919"*|*"20170918"*|*"20170917"*|*"20170916"*|*"20170915"*|*"20170914"*)
#!/usr/bin/env python
from datetime import datetime
from datetime import timedelta
import subprocess
### Editable Variables
keepdays=7
location="/home/backups"
count=0
date_string=''
for count in range(0,keepdays):
if(date_string!=""):
date_string+="|"
keepdate = (datetime.now() - timedelta(days=count)).strftime("%Y%m%d")
date_string+="*\""+keepdate+"\"*"
full_cmd="/bin/rm "+location+"/!("+date_string+")"
subprocess.call([full_cmd], shell=True)
This is what the script returns:
#./test.py
/bin/rm /home/backups/!(*"20170920"*|*"20170919"*|*"20170918"*|*"20170917"*|*"20170916"*|*"20170915"*|*"20170914"*)
/bin/sh: 1: Syntax error: "(" unexpected
Python version is Python 2.7.12
Just as #hjpotter said, subprocess will use /bin/sh as default shell, which doesn't support the kind of globbing you want to do. See official documentation. You can change that using the executable parameter to subprocess.call() with a more appropriate shell (/bin/bash or /bin/zsh for example): subprocess.call([full_cmd], executable="/bin/bash", shell=True)
BUT you can be a lot better served by Python itself, you don't need to call a subprocess to delete a file:
#!/usr/bin/env python
from datetime import datetime
from datetime import timedelta
import re
import os
import os.path
### Editable Variables
keepdays=7
location="/home/backups"
now = datetime.now()
keeppatterns = set((now - timedelta(days=count)).strftime("%Y%m%d") for count in range(0, keepdays))
for filename in os.listdir(location):
dates = set(re.findall(r"\d{8}", filename))
if not dates or dates.isdisjoint(keeppatterns):
abs_path = os.path.join(location, filename)
print("I am about to remove", abs_path)
# uncomment the line below when you are sure it won't delete any valuable file
#os.path.delete(abs_path)
Related
I have multiple python scripts in various complexity I am running from another application as part of data load steps.
Some are in production for more than 2 years some new.
I have observed lately a script starts and does nothing for 10+ minutes.
Example : The application has the start timestamp as 2022-05-17 15:48:42
The first entry in the log file for this script is 2022-05-17 16:01:04
Script logs the start time as a first step.
The list of libraries imported for the example run above is a short one since this is one of the areas I looked at
import snowflake.connector
import json
import os
import datetime
import threading
from pathlib import Path
Any ideas what could be causing this behavior?
How can I further debug this?
Update : Here is the minimal example of the script
import snowflake.connector
import json
import os
import datetime
import threading
from pathlib import Path
def to_log(msg, include_timestamp=True):
"""Write the log entry to log file"""
# add the current date to the log file name in order to create a file for everyday
log_file = Path(
work_dir
/ (
f"log/kafka_loader_"
+ "{:%Y%m%d}".format(datetime.datetime.now())
+ ".log"
)
)
if include_timestamp:
msg = "{:%Y-%m-%d %H:%M:%S}".format(datetime.datetime.now()) + f" {msg}"
print(msg)
with open(log_file, "a") as l_f:
l_f.write(msg + "\n")
def proces():
...
if __name__ == "__main__":
to_log("Starting a new session")
process()
to_log("Session completed")
The application used to invoke the script is called WhereScape.
In the UI I see the started start timestamp above.
I'm calling this function and using %s*silent to read files that have names with the following format: name.number.silent.
I get the name from start_model.split('/')[-1].split('.')[0] so don't worry about it.
This is obviously not working because these commands are actually never passed to the shell. If I were to use glob, how can I modify my code to do what I'm doing below?
from subprocess import call
def fragment_score(rosetta_path, silent_input_and_score_output, start_model):
call([rosetta_path,
'-mode score',
'-in::file::silent', '%s/%s*silent' % (silent_input_and_score_output, start_model.split('/')[-1].split('.')[0]),
'-scorefile', '%s/scores1' % silent_input_and_score_output,
'-n_matches', '50'])
Use the Python glob module to generate a list of glob results, and splice it into your argument list at the same position where you would otherwise have a shell replacing a glob expression with the list of associated matches:
from subprocess import call
from glob import glob
def fragment_score(rosetta_path, silent_input_and_score_output, start_model):
glob_exp = '%s/%s*silent' % (silent_input_and_score_output, start_model.split('/')[-1].split('.')[0])
glob_results = glob(glob_exp)
call([rosetta_path,
'-mode score',
'-in::file::silent'
] + glob_results + [
'-scorefile', '%s/scores1' % silent_input_and_score_output,
'-n_matches', '50'])
In current Python 3.x, there's syntax that makes this a bit more natural:
call([rosetta_path,
'-mode score',
'-in::file::silent',
*glob_results,
'-scorefile', '%s/scores1' % silent_input_and_score_output,
'-n_matches', '50'])
I have the following python structure
directory structure
In login_logout_test.py I use the following imports and everything works fine when I run this script (all the modules are imported properly)
import sys, os
parentPath = os.path.abspath("..")
if parentPath not in sys.path:
sys.path.insert(0, parentPath)
import common_test_functions as cts
from functions import login_logout_functions as llf
But when this script (login_logout_test.py) is called by CommandRunner.py this error occurs:
No module named 'common_test_functions'
Actually I have contrived a solution to my own problem:
import sys, os
from selenium import webdriver
# adding path to common_test_functions to the sys.path
# so this module could be invoked
parentPath = os.path.abspath("..\\..")
if parentPath not in sys.path:
sys.path.append(parentPath)
from functions import login_logout_functions as llf
from tests import common_test_functions as cts
Also there is a file, that holds necessary for the script parameters. Here is the code to have this file path in both cases (running this script by itself or calling it by another script):
parameters_directory_path = os.path.dirname(os.path.realpath(__file__))
parameters_file_name = "login_logout_test_parameters.tsv"
parameters_file_path = os.path.join(parameters_file_path, parameters_file_name)
If someone has a better one, please post it.
Thank you in advance.
Stefan
File setup:
...\Project_Folder
...\Project_Folder\Project.py
...\Project_folder\Script\TestScript.py
I'm attempting to have Project.py import modules from the folder Script based on user input.
Python Version: 3.4.2
Ideally, the script would look something like
q = str(input("Input: "))
from Script import q
However, python does not recognize q as a variable when using import.
I've tried using importlib, however I cannot figure out how to import from the Script folder mentioned above.
import importlib
q = str(input("Input: "))
module = importlib.import_module(q, package=None)
I'm not certain where I would implement the file path.
Repeat of my answer originally posted at How to import a module given the full path?
as this is a Python 3.4 specific question:
This area of Python 3.4 seems to be extremely tortuous to understand, mainly because the documentation doesn't give good examples! This was my attempt using non-deprecated modules. It will import a module given the path to the .py file. I'm using it to load "plugins" at runtime.
def import_module_from_file(full_path_to_module):
"""
Import a module given the full path/filename of the .py file
Python 3.4
"""
module = None
try:
# Get module name and path from full path
module_dir, module_file = os.path.split(full_path_to_module)
module_name, module_ext = os.path.splitext(module_file)
# Get module "spec" from filename
spec = importlib.util.spec_from_file_location(module_name,full_path_to_module)
module = spec.loader.load_module()
except Exception as ec:
# Simple error printing
# Insert "sophisticated" stuff here
print(ec)
finally:
return module
# load module dynamically
path = "<enter your path here>"
module = import_module_from_file(path)
# Now use the module
# e.g. module.myFunction()
I did this by defining the entire import line as a string, formatting the string with q and then using the exec command:
imp = 'from Script import %s' %q
exec imp
I am new to python and I'm trying to create a program that creates a directory with todays date, create a sandbox into that directory and run the make file in the sandbox. I am having trouble getting the variables to be picked up in the os.path lines. The code is posted below:
#!/usr/bin/python
import mks_function
from mks_function import mks_create_sandbox
import sys, os, time, datetime
import os.path
today = datetime.date.today() # get today's date as a datetime type
todaystr = today.isoformat() # get string representation: YYYY-MM-DD
# from a datetime type.
if not os.path.exists('/home/build/test/sandboxes/'+todaystr):
os.mkdir(todaystr)
else:
pass
if not os.path.exists('/home/build/test/sandboxes/'+todaystr+'/new_sandbox/project.pj'):
mks_create_sandbox()
else:
pass
if os.path.exists('/home/build/test/sandboxes/'+todaystr+'/new_sandbox/Makefile'):
os.system("make >make_results.txt 2>&1")
Any help would be appreciated,
Thanks
a couple of notes:
#!/usr/bin/env python
# import mks_function .. you won't need this ...
from mks_function import mks_create_sandbox
import os, datetime
# import time, sys .. these aren't used in this snippet
# import os.path .. just refer to os.path, since os is already imported
# get today's date as a datetime type
todaystr = datetime.date.today().isoformat()
# .. use os.path.join()
if not os.path.exists(os.path.join('/home/build/test/sandboxes/', todaystr)):
os.mkdir(os.path.join('/home/build/test/sandboxes/', todaystr))
# .. 'else: pass' is unnecessary
if not os.path.exists(os.path.join(
'/home/build/test/sandboxes/', todaystr, '/new_sandbox/project.pj')):
# i'm not seen, that the sandbox is created in the right directory here
# maybe you should change the working directory via ..
# os.chdir(os.path.join('/home/build/test/sandboxes/', todaystr))
mks_create_sandbox()
if os.path.exists(os.path.join(
'/home/build/test/sandboxes/', todaystr, '/new_sandbox/Makefile')):
# .. change to the right directory
os.chdir(os.path.join(
'/home/build/test/sandboxes/', todaystr, '/new_sandbox/'))
os.system("make > make_results.txt 2>&1")
Please try adding chdir code before you call make
if os.path.exists('/home/build/test/sandboxes/'+todaystr+'/new_sandbox/Makefile'):
os.chdir('/home/build/test/sandboxes/'+todaystr+'/new_sandbox/')
os.system("make >make_results.txt 2>&1")
I think you want to change a few things:
def makeSandbox():
sbdir = os.path.join('/home/build/test/sandboxes/',todaystr)
if not os.path.exists(sbdir):
os.mkdir(sbdir) # <- fully qualified path
else:
pass
And I don't really see what variables need to be picked up, seems fine to me.
Not sure what the module mks_function does. But I see one issue with your code.
For example,
if not os.path.exists('/home/build/test/sandboxes/'+todaystr):
os.mkdir(todaystr)
In the above chunk you check if the directory "/home/build/test/sandboxes/+'todaystr'"
exists and a create a directory by name "value contained in todaystr" (say 2009-12-21). This creates directory by name '2009-12-21' in the current working directory, rather than under : /home/build/test/sandboxes
which is what you intended I guess. So change to the above directory before the call to mkdir. Also it is good to check the return status of mkdir to verify if the directory creation succeeded.
path module might help in this case:
#!/usr/bin/env python
from mks_function import mks_create_sandbox
import os, datetime
from path import path
sandboxes = path('/home/build/test/sandboxes/')
today = sandboxes / datetime.date.today().isoformat()
today.mkdir() # create directory if it doesn't exist
project = today / "new_sandbox/project.pj"
project.parent.mkdir() # create sandbox directory if it doesn't exist
if not project.isfile():
mks_create_sandbox()
makefile = project.parent / "Makefile"
if makefile.isfile():
os.chdir(makefile.parent)
os.system("make >make_results.txt 2>&1")