We Keep Coding

os.system argument from config file

I made this activity and it works. I need to have config file with USB/VID/PID.
def resetactivity():
os.system(r'"devcon.exe restart "*USB\VID_04E8&PID_3321*"')
I try to do this with config parser. I made config.txt:
[My Section]
usbdev = r'"devcon.exe restart "*USB\VID_04E8&PID_3321*"'
I read my config file in Python:
config = configparser.ConfigParser()
config.read('config.txt')
usbdev = config.get('My Section', 'usbdev')
And when I am trying to use this in os.system command like this:
def resetactivity():
os.system(usbdev)
I get this result:
The filename, directory name, or volume label syntax is incorrect.
'PID_3321*"'' is not recognized as an internal or external command,
operable program or batch file.

Try this code
import configparser
import os
def resetactivity():
config = configparser.ConfigParser()
config.read('config.txt')
usbdev = config.get('My Section', 'usbdev')
print(usbdev)
os.system(usbdev)
if __name__ == "__main__":
resetactivity()
With config.txt formatted as
[My Section]
usbdev = devcon.exe restart "USB\VID_04E8&PID_3321"

How do I pass variables defined from argparse in config as a module to other scripts?

The main problem I have is accessing variables defined in a configuration file as a module in other scripts when those variables are defined by argparse inputs. If I encapsulate this code within a function, I will not have access to it in other scripts. Similarly, if I don't use a function, the configuration file will expect inputs whenever it is called as a module. I think global variables might be the answer, but I don't know how to set them up in this use case?
import shutil
import inspect
import configparser
import sys
import argparse
def cmd_inputs():
parser = argparse.ArgumentParser()
parser.add_argument('-dir','--geofloodhomedir',help="File path to directory that will hold the cfg file \
and the inputs and outputs folders. Default is the GeoNet3 directory.",
type=str)
parser.add_argument('-p','--project',
help="Folder within GIS, Hydraulics, and NWM directories \
of Inputs and Outputs. Default is 'my_project'.",
type=str)
parser.add_argument('-n','--DEM_name',
help="Name of Input DEM (without extension) and the prefix used for all \
project outputs. Default is 'dem'",
type=str)
parser.add_argument('--no_chunk',help="Dont batch process DEM's > 1.5 GB for Network Extraction. Default is to chunk DEMs larger than 1.5 GB.",
action='store_true')
global home_dir
args = parser.parse_args()
if args.geofloodhomedir:
home_dir = args.geofloodhomedir
if os.path.abspath(home_dir) == os.path.dirname(os.path.abspath(__file__)):
home_dir = os.getcwd()
print("hello")
print('GeoNet and GeoFlood Home Directory: ')
print(home_dir)
else:
ab_path = os.path.abspath(__file__)
home_dir = os.path.dirname(ab_path)
print('Using default GeoNet and GeoFlood home directory:')
print(home_dir)
if args.project:
project_name=args.project
print(f"Project Name: {project_name}")
else:
project_name='my_project'
print(f"Default Project Name: {project_name}")
if args.DEM_name:
dem_name = args.DEM_name
print(f'DEM Name: {dem_name}')
else:
dem_name='dem'
print(f'Default DEM: {dem_name}')
if not args.no_chunk:
print("Chunking DEMs > 1.5 GBs in Network Extraction script")
chunk = 1
else:
print("Not chunking input DEM or its products in Network Extraction script")
chunk = 0
config = configparser.ConfigParser()
config['Section']={'geofloodhomedir':home_dir,
'projectname':project_name,
'dem_name':dem_name,
'Chunk_DEM':chunk}
with open(os.path.join(home_dir,'GeoFlood.cfg'),'w') as configfile:
config.write(configfile)
if __name__=='__main__':
cmd_inputs()
print ("Configuration Complete")```

Python subprocess script failing

Have written the below script to delete files in a folder not matching the dates in the "keep" period. Eg. Delete all except files partly matching this name.
The command works from the shell but fails with the subprocess call.
/bin/rm /home/backups/!(*"20170920"*|*"20170919"*|*"20170918"*|*"20170917"*|*"20170916"*|*"20170915"*|*"20170914"*)
#!/usr/bin/env python
from datetime import datetime
from datetime import timedelta
import subprocess
### Editable Variables
keepdays=7
location="/home/backups"
count=0
date_string=''
for count in range(0,keepdays):
if(date_string!=""):
date_string+="|"
keepdate = (datetime.now() - timedelta(days=count)).strftime("%Y%m%d")
date_string+="*\""+keepdate+"\"*"
full_cmd="/bin/rm "+location+"/!("+date_string+")"
subprocess.call([full_cmd], shell=True)
This is what the script returns:
#./test.py
/bin/rm /home/backups/!(*"20170920"*|*"20170919"*|*"20170918"*|*"20170917"*|*"20170916"*|*"20170915"*|*"20170914"*)
/bin/sh: 1: Syntax error: "(" unexpected
Python version is Python 2.7.12

Just as #hjpotter said, subprocess will use /bin/sh as default shell, which doesn't support the kind of globbing you want to do. See official documentation. You can change that using the executable parameter to subprocess.call() with a more appropriate shell (/bin/bash or /bin/zsh for example): subprocess.call([full_cmd], executable="/bin/bash", shell=True)
BUT you can be a lot better served by Python itself, you don't need to call a subprocess to delete a file:
#!/usr/bin/env python
from datetime import datetime
from datetime import timedelta
import re
import os
import os.path
### Editable Variables
keepdays=7
location="/home/backups"
now = datetime.now()
keeppatterns = set((now - timedelta(days=count)).strftime("%Y%m%d") for count in range(0, keepdays))
for filename in os.listdir(location):
dates = set(re.findall(r"\d{8}", filename))
if not dates or dates.isdisjoint(keeppatterns):
abs_path = os.path.join(location, filename)
print("I am about to remove", abs_path)
# uncomment the line below when you are sure it won't delete any valuable file
#os.path.delete(abs_path)

Updating of a config file written in python

I'm writing a simulation software which should support reading parameters from a config file or from the command line. It is very important to be able to track what was the configuration of a simulation, I'm committing the config file to a local git repository at the start of simulation.
Now if I have parameters on the command line they have higher priority than the ones in the config file. But I also want to commit them. I guess I could save the python objects of a configured simulation, just before it is started. But it would be more elegant, if I could just update the config file with the command line parameters before committing it.
The reason I write the config file in python is that I have to define some python objects in it. I have something like
import SomeSimulationClass
SIMULATOR = SomeSimulationClass
in my config file and the SIMULATOR can then be swapped easily.
If I want to use something like configparser I can't have objects I believe.
Is there any easy way to update a python config file? All variable names in it are already defined, I just want to change the values. The only thing I can think of is parsing the file, comparing strings between the file and the command line parameters ...

You can write whatever you want into a file, and then later, Configparser can read
from it using values from your variables. Here is an example on how I used Configparser to read environment from config file.
import os
from ConfigParser import SafeConfigParser
conf_filename = os.getenv("CONFIG_FILE")
src_dir = os.getenv("CONFIG_DIR")
conf_file = os.path.join(src_dir,conf_filename)
parser = SafeConfigParser()
parser.read(conf_file)
section = env
server = parser.get(section, 'host')
db_port = parser.get(section, 'db_port')
ws_port = parser.get(section, 'ws_port')
and the config file itself:
[PROD]
host=xxx-yyy-15
db_port=1521
ws_port=8280
ora_server=xxx-xxx-xxx.com
sid=XXXXX
userid=xxxx
passwd=xxxx
[STAGE]
host=xxx-yyy-04
db_port=1521
ws_port=8280
ora_server=yyy-yyy-yyy.com
sid=YYYYYY
userid=yyyy
passwd=yyyy

I found a way to do what I want. Some small modifications were necessary to my python config module to allow it to be rewritten with the following script, but it works for my purposes:
with open('merged_config.py', 'w') as merged_config, \
open(base_config_module.__file__, 'r') as base_config:
for line in base_config:
if 'import' in line:
# copy imports from bas config
merged_config.write(line)
for item in dir(base_config_module):
if item.startswith("__"):
# ignore __variables like '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__' ...
continue
if item == 'SimulationSteps':
# ingoring my imports
continue
item_val = getattr(base_config_module, item)
# I had to overwrite the __repr__() method of Enums which I used. Everyting else worked fine.
merged_config.write('%s = %s\n' % (item, repr(item_val)))

Python calling a file from another directory

I've been trying to find the answer to this but everything i look at is reading a file from another directory or running a file from another directory. I'm sure what i want to do isn't that hard but i am new at this and don't know how to describe what it's called.
I have a python script run.py that is in the /src directory. I run that script form the /src directory. run.py calls two files (configuration.py and server.py). These two files are in a folder called lib (src/lib).All folders have an empty __init__.py file.
When i take these files out of lib and put them just in src i can run the script when the script looks like it does below.
import os
import inspect
import sys
import configuration
import server
# Initialize Parameters
f_path = os.path.abspath(inspect.getfile(inspect.currentframe()))
absolute_path = os.path.dirname(f_path)
if __name__ == "__main__":
from optparse import OptionParser, OptionGroup
parser = OptionParser()
parser.usage = "usage: %prog [options] "
parser.description = "Primary application entry point."
parser.add_option("-v", "--verbose", dest="verbose", action="store_true",
default=False, help="Run verbose.")
group = OptionGroup(parser, "Node operations")
group.add_option("--do-queue-job", dest="do_queue_job", action="store_true",
help="Run the next job in the quasar-server queue.")
parser.add_option_group(group)
(options, args) = parser.parse_args()
# Clear argv to prevent issues with web.py automatically using arguments to
# bind ip addresses.
sys.argv = []
configuration = configuration.Configuration("/home/mscarpa/PhpstormProjects/quasar-node/quasar-node/quasar-node/src/config.yml")
if (options.do_queue_job):
# Get server instance
server_connection = server.QuasarConnection(configuration)
#return server_connection
# Get next job from server
next_job = server_connection.get_next_job()
#return next_job
The two parts of the code i know i have to change if i move the two files to /src/lib are the following:
configuration = configuration.Configuration("/home/mscarpa/PhpstormProjects/quasar-node/quasar-node/quasar-node/src/config.yml")
server_connection = server.QuasarConnection(configuration)
i am thinking that i would just have to put.lib before them like so, but every time i try it it says lib is not defined.
configuration = lib.configuration.Configuration("/home/mscarpa/PhpstormProjects/quasar-node/quasar-node/quasar-node/src/config.yml")
server_connection = lib.server.QuasarConnection(configuration)
This is probably a noob question, but does anyone know how to target these files if the are in the src/lib directory as opposed to just the src directory

You just need to change your import statement to reflect the module's new location:
import lib.configuration as configuration
import lib.server as server
And the rest of your script doesn't really need to change.

I got it. I think your answer may have worked in certain cases but i think my problem being new at this is figuring out what to search.
it was a sys.arg thing so i had to include that path to that lib folder before i imported the files.
sys.path.insert(0, '/home/mscarpa/PhpstormProjects/quasar-node/quasar-node/quasar-node/src/lib')
import configuration
import server