I have a collection of scripts written in Python. Each of them can be executed independently. However, most of the time they should be executed one after the other, so there is a MainScript.py which calls them in the appropriate order. Each script has some configurable variables (let's call them Root_Dir, Data_Dir and LinWinFlag). If this collection of scripts is moved to a different computer, or different data needs to be processed, these variable values need to be changed. As there are many scripts this duplication is annoying and error-prone. I would like to group all configuration variables into a single file.
I tried making Config.py which would contain them as per this thread, but import Config produces ImportError: No module named Config because they are not part of a package.
Then I tried relying on variable inheritance: define them once in MainScript.py which calls all the others. This works, but I realized that each script would not be able to run on its own. To solve this, I tried adding useGlobal=True in MainScript.py and in other files:
if (useGlobal is None or useGlobal==False):
# define all variables
But this fails when scripts are run standalone: NameError: name 'useGlobal' is not defined. The workaround is to define useGlobal and set it to False when running the scripts independently of MainScript.py. It there a more elegant solution?
The idea is that python wants to access files - including the Config.py - primarily as part of a module.
The nice thing is that Python makes building modules (i.e. python packages) really easy - initializing it can be done by creating a
__init__.py
file in each directory you want as a module, a submodule, a subsubmodule, and so on.
So your import should go through if you have created this file.
If you have further questions, look at the excellent python documentation.
The best way to do this is to use a configuration file placed in your home directory (~/.config/yourscript/config.json).
You can then load the file on start and provide default values if the file does not exist :
Example (config.py) :
import json
default_config = {
"name": "volnt",
"mail": "oh#hi.com"
}
def load_settings():
settings = default_config
try:
with open("~/.config/yourscript/config.json", "r") as config_file:
loaded_config = json.loads(config_file.read())
for key in loaded_config:
settings[key] = loaded_config[key]
except IOError: # file does not exist
pass
return settings
For a configuration file it's a good idea to use json and not python, because it makes it easy to edit for people using your scripts.
As suggested by cleros, ConfigParser module seems to be the closest thing to what I wanted (one-line statement in each file which would set up multiple variables).
Related
I have a project which executes large number of python files in a folder. All of them have a line PATH="defaultstring". I want to make it more dynamic i.e. replace "defaultstring" in all the python files to some "otherstring" provided on run time. If no string is provided on runtime then "defaultstring" will be default value.
I am building the project using gradle. One of the thing I can do is execute some python script say "main.py" before those group of files are executed. "main.py" can iterate to all the files, open them, and replace PATH="defaultstring" to "otherstring".
Is it possible to do this without changing all those files in folder every time I run ? This is because if I change "defaultstring" to "otherstring" in first run and in second run suppose I don't give any runtime input then by default PATH="otherstring" will be executed but default value I want to keep is "defaultstring". I can change "otherstring" to "defaultstring" in this case but I want to do with some method which does not iterate through all files and change those lines.
Can we do this using fixture injection (which I don't have much idea about so any other technique will also be helpful).
Assuming that your files have some naming pattern that is unique and can be recognised you can do the following:
Find all your files using glob or something similar
Import the files dynamically
Change the variable using the global keyword
If you create a function that does these steps and execute it as the first action of your code in the main function, you should be able to change the variable PATH globally at runtime without interacting with the file stored on your hard drive (and consequently not changing the default value).
Edit: While the above approach will hopefully work, it is nevertheless bad practice. It would be better if you have a single file that contains such variables that all other files import. Then you only have to change a single variable and the intention is much clearer than in the version that you originally intended to do.
Edit2: You can structure your project to contain the following files with respective functions or variables. This setup allows you to change the variable PATH at runtime upon starting the script if all file import a common file containing path. The other option of changing a variable in each file is more cumbersome and I would only add the option if really necessary.
globals.py
PATH = 'original/path'
def modify_path(new_path):
global PATH
PATH = new_path
some_file1.py
import globals
def get_path():
return globals.PATH
some_file2.py
import globals
def get_path():
return globals.PATH
main.py
import globals
if necessary:
globals.modify_path('new/path')
Over the years I have written a few hundred functions in matlab for space engineering purposes which I use on a daily basis. They are all nicely put in one folder ordered in subfolders, and in matlab I just have an addpath() command for the root folder in the startup.m file, and then I can use any of the functions right away after starting up matlab.
I am now trying to do the same in python.
As far as I understand in python I shouldn't have 1 file for every function like in matlab, but rather bundle all the functions together in 1 py file. Is this correct? I am trying to avoid this, since I have a strong preference for short scripts rather than 1 huge one, due to it being way more intuitive for me that way.
And then, once I have all my python scripts, can I place them anywhere in order to use them? Because I read that python works differently than matlab in this aspect, and scripts need to be in the working directory in order to import them. However, I want to load the scripts and be able to use them regardless of my active working directory. So I am guessing I have to do something with paths. I have found that I can append to pythonpath using sys.path.insert or append, however this feels like a workaround to me, or is it the way to go?
So considering I have put all my rewritten matlab fuctions in a single python file (lets call it agfunctions.py) saved in a directory (lets call it PythonFunctions). The core of my startup.py would then be something like (I have added the startup file to PYTHONSTARTUP path):
# startup.py
import os, sys
import numpy as np
import spiceypy as spice
sys.path.append('C:\Users\AG5\Documents\PythonFunctions')
import agfunctions as ag
Does any of this make sense? Is this the way to go, or is there a better way in python?
Well, the python package is probably the best way to solve your problem. You can read more here. Python packages have no need to be built and not always are created for sharing, so do not worry.
Assume you had this file structure:
Documents/
startup.py
PythonFunctions/
FirstFunc.py
SecondFunc.py
Then you can add file __init__.py in your PythonFunctions directory with next content:
__all__ = ['FirstFunc', 'SecondFunc']
Init file must be updated if you change filenames, so maybe it isn't best solution for you. Now, the directory looks like:
Documents/
startup.py
PythonFunctions/
__init__.py
FirstFunc.py
SecondFunc.py
And it's all - PythonFunctions now is a package. You can import all files by one import statement and use them:
from PythonFunctions import *
result_one = FirstFunc.function_name(some_data)
result_two = SecondFunc.function_name(some_other_data)
And if your startup.py is somewhere else, you can update path before importing as next:
sys.path.append('C:\Users\AG5\Documents')
More detailed explanations can be found here
There is no need to write all your matlab functions within one file. You can also keep (kind of) your desired folder structure of your library. However, you should write all functions in the deepest subfolder into one *.py file.
Suppose your MATLAB library is in the folder space_engineering and is set up like this:
space_engineering\
subfolder1\
functiongroup1\
printfoo.m
printbar.m
subfolder2\
...
subfolder3\
...
...
After doing addpath(genpath('\your_path\space_engineering')) all functions in subfolders subfolder* and functiongroup* are available in the global namespace like this
>> printfoo % function printfoo does only do fprintf('foo')
foo
I understand this is an behaviour your want to preserve in your migrated python library. And there is a way to do so. Your new python library would be structured like this:
space_engineering\
__init__.py
subfolder1\
__init__.py
functiongroup1.py
functiongroup2.py
subfolder2\
__init__.py
...
subfolder3\
__init__.py
...
...
As you can see the deepest subfolder layers functiongroup* of the MATLAB structure is replaced now by functiongroup*.py files, so called modules. The only compromise you have to allow for is that functions printfoo() and printbar() are now defined in this .py modules instead of having individual .py files.
# content of functiongroup1.py
def printfoo():
print(foo)
def printbar():
print(bar)
To allow for doing the same function calling as in MATLAB you have to make the function names printfoo and printbar available in the global namespace by adjusting the __init__.py files of each subfolder
# content of space_enginieering\subfolder1\__init__.py
from .functiongroup1 import *
from .functiongroup2 import *
as well as the __init__.py of the main folder
# content of space_engineering\__init__.py
from .subfolder1 import *
from .subfolder2 import *
from .subfolder3 import *
The from .functiongroup1 import * statement loads all names from module functiongroup1 into the namespace of subfolder1. Successively from .subfolder1 import * will forward them to the global namespace.
Like this you can do in an python console (or in any script) e.g.:
>>> sys.path.append('\your_path\space_engineering')
>>> from space_engineering import *
>>> printfoo()
foo
This way you can use your new python library in the same way as you former used the MATLAB library.
HOWEVER: The usage of from xyz import * statement is not recommend in python (see here why, it is similar to why not using eval in MATLAB) and some Pythonistas may complain recommending this. But for your special case, where you insist on creating a python library with MATLAB like comfort, it is a proper solution.
I am trying to work with Behave on Python.
I was wondering if there would be a way to put my .py files somewhere else instead of being forced to put them all inside the "steps" folder. My current structure would look like this
tests/
features/
steps/ #all code inside here, for now
What I would like to accomplish is something like
tests/
features/ #with all the .feature files
login/ #with all the .py files for logging in inside a service
models/ #with all the .py files that represents a given object
and so on
The only BDD framework that I used before Behave was Cucumber with Java, which allowed to insert the step definitions wherever I wanted to (and the rest was handled by Cucumber itself).
I am asking this because I would like to have a lot of classes in my project in order to organize my code in a better way.
This may be a bit late but you can do the following:
Have the structure like this:
tests/
features/
steps/
login
main_menu
all_steps.py
In the subfolders in steps you can create your _steps.py file with the implementation and then in the all_steps.py(or how you want to name it) you just need to import them:
from tests.steps.login.<feature>_step import *
from tests.steps.main_menu.<feature>_step import *
etc
And when you run this it should find the step files. Alternatively you can have the files anywhere in the project as long as you have 1 Steps folder and a file in the file were you import all steps
First, from the behave Documentation (Release 1.2.7.dev0):
behave works with three types of files:
feature files written by your Business Analyst / Sponsor / whoever with your behaviour scenarios in it, and
a “steps” directory with Python step implementations for the scenarios.
optionally some environmental controls (code to run before and after steps, scenarios, features or the whole
shooting match).
So a steps/ directory is required.
To accomplish a workaround similar to what you have in mind, I tried creating a subdirectory in the /steps directory: /steps/deeper/ and inserted my Python file there: /steps/deeper/testing.py. After running behave, I received the "NotImplementedError", meaning the step definitions in /deeper/testing.py were not found.
It appears that behave doesn't search recursively through subdirectories of the steps/ directory for any additional Python files.
As for what you're trying to do, I think it's decent organizational idea, but since it's not doable, you could do this: instead of having directories for the Python files in your tests/ directory, why not have a good naming convention for your Python file and separate the associated functions into their own Python files? That is:
tests/
features/
steps/
login_prompt.py # contains all the functions for logging in inside a service
login_ssh.py # contains all the functions for SSH login
models_default.py # contains all the functions for the default object
models_custom.py # contains all the functions for a custom object
and so on...
Of course, at this point, it really doesn't matter if you separate them into different Python files, since behave searches through all the Python files in steps/ when called, but for organization's sake, it accomplishes the same effect.
You can do it with additional method, smth like this:
def import_steps_from_subdirs(dir_path):
for directory in walk(dir_path):
current_directory = directory[0] + '/'
all_modules = [module_info[1] for module_info in iter_modules(path=[current_directory])]
current_directory = current_directory.replace(Resources.BASE_DIR + '/', '')
for module in all_modules:
import_module(current_directory.replace('/', '.') + module)
Then call this method in before_all layer
I've been working on a project that creates its own .py files that store handlers for the method, I've been trying to figure out how to store the Python files in folder and open them. Here is the code I'm using to create the files if they don't already exist, then importing the file:
if os.path.isfile("Btn"+ str(self.ButtonSet[self.IntBtnID].IntPID) +".py") == False:
TestPy = open("Btn"+ str(self.ButtonSet[self.IntBtnID].IntPID) +".py","w+")
try:
TestPy.write(StrHandler)
except Exception as Error:
print(Error)
TestPy.close()
self.ButtonSet[self.IntBtnID].ImpHandler = __import__("Btn" + str(self.IntBtnID))
self.IntBtnID += 1
when I change this line:
self.ButtonSet[self.IntBtnID].ImpHandler = __import__("Btn" + str(self.IntBtnID))
to this:
self.ButtonSet[self.IntBtnID].ImpHandler = __import__("Buttons\\Btn" + str(self.IntBtnID))
the fill can't be found and ends up throwing an error because it can't find the file in the folder.
Do know why it doesn't work I just don't know how to get around the issue:/
My question is how do I open the .py when its stored in a folder?
There are a couple of unidiomatic things in your code that may be the cause of your issue. First of all, it is generally better to use the functions in os.path to manipulate paths to files. From your backslash usage, it appears you're working on Windows, but the os.path module ensures consistent behaviour across all platforms.
Also there is importlib.import_module, which is usually recommended over __import__. Furthermore, in case you want to load the generated module more than once during the lifetime of your program, you have to do that explicitly using imp.reload.
One last tip: I'd factor out the module path to avoid having to change it in more than one place.
You can't reference a path directory when you are importing files. Instead, you want to add the directory to your path and then import the name of the module.
import sys
sys.path.append( "Buttons" )
__import__("Btn"+str(self.IntBtnId))
See this so question for more information.
The first argument to the __import__() function is the name of the module, not a path to it. Therefore I think you need to use:
self.ButtonSet[self.IntBtnID].ImpHandler = __import__("Buttons.Btn" + str(self.IntBtnID))
You may also need to put an empty __init__.py file in the Buttons folder to indicate it's a package of modules.
I am working on a game engine which includes a simple GUI development tool. The GUI tool allows a user to define various entities and components, which can then be saved in a configuration file. When the game engine runtime loads the configuration file, it can determine how to create the various entities and components for use in the game.
For a configuration file saving mechanism, I am using PyYAML. The issue that I am having stems from the fact that the serialization process occurs in a module which is in a different directory than the module which loads and parses the file through PyYAML.
Simplified Serializer
import yaml
def save_config(context, file_name):
config_file = file(file_name, 'w')
# do some various processing on the context dict object
yaml.dump(context, config_file)
config_file.close()
This takes the context object, which is a dict that represents various game objects, and writes it to a config file. This works without issue.
Simplified Deserializer in engine
import yaml
def load(file_name):
config_file = open(file_name, 'r')
context = yaml.load(config_file)
return context
This is where the problem occurs. On yaml.load(config_file), I will receive an error, because it fails to find a various name on a certain module. I understand why this is happening. For example, when I serialize the config file, it will list an AssetComponent (a component type in the engine) as being at engine.common.AssetComponent. However, from the deserializer's perspective, the AssetComponent should just be at common.AssetComponent (because the deserialization code itself exists within the engine package), so it fails to find it under engine.
Is there a way to manually handle paths when serializing or deserializing with PyYAML? I would like to make sure they both happen from the same "perspective."
Edit:
The following shows what a problematic config file might look like, followed by what the manually corrected config would look like
Problematic
!!python/object/apply:collections.defaultdict
args: [!!python/name:__builtin__.dict '']
dictitems:
assets:
- !!python/object:common.Component
component: !!python/object:engine.common.AssetComponent {file_name: ../content/sticksheet.png,
surface: null}
text: ../content/sticksheet.png
type_name: AssetComponent
Corrected
!!python/object/apply:collections.defaultdict
args: [!!python/name:__builtin__.dict '']
dictitems:
assets:
- !!python/object:tools.common.Component
component: !!python/object:common.AssetComponent {file_name: ../content/sticksheet.png,
surface: null}
text: ../content/sticksheet.png
type_name: AssetComponent
Your problem lies in a mismatch between your package structure and your __main__ routines. The module containing your __main__ will be inside a package, but it has no way of knowing that. So, therefore, you will use imports relative to the location of the file containing __main__ and not relative to the top level structure of your package.
See Relative imports for the billionth time for a longer (and probably better) explanation.
So, how can you fix it?
Inside the file containing __main__ you do:
from tools.common import Component
# instead of from common import Component
c = Component()
print yaml.dump(c)
Another thing you must ensure is that python will know how to load your modules. If you have installed your package this will be done automatically, but during development this is usually not the case. So during development you will also want to make your development modules findable.
The easiest way (but not very clean) is to use sys.path.append('the directory containing tools and engine'). Another way (cleaner) is to set the PYTHONPATH environment variable to include your top level directory containing tools and engine.
You can explicitly declare the Python type of an object in a YAML document:
!!python/object:module_foo.ClassFoo {
attr_foo: "spam",
…,
}