I have a file structure like the following:
config.py
main.py
some_other_file.py
where config.py contains easily accessible parameters but not much code otherwise. These should be accessible to all other code files. Normally import config would do, but in this case the python script is called externally from another program, and therefore the root calling directory is not the same as the one the files are located at (so just an import results in an exception since it does not find the files).
Right now, the solution I have is to include into my main.py file (the one that is directly called by the third program) the following:
code_path = "Path\\To\\My\\Project\\"
import sys
sys.path.insert(0, code_path)
import config
import some_other_file
...
However, this means having to modify main.py every time the code is moved around. I could live with that, but I would certainly like having one single, simple file with all necessary configuration, not needing to dig through the others (especially since the code may later be passed onto others who just want it to work as effortlessly as possible).
I tried having the sys.path.insert inside the config file, and having that be the file directly called by the external script (and all other files called from there). However, I run into the problem that when the other files do import config, it results in an import loop since they are also being imported from config.py. Typically, I believe this is solved by making the imports in the config.py file only once through something like if __name__ == "__main__": (see below). This does not work in my case, and the script never goes into the if statement, possibly because it is being called as a sub-routine by a third program and it is not the main program itself. As a result, I have no way of enforcing a portion of the code in config.py to only be executed once.
This is what I meant above for config.py (which does not work for my case):
... # Some parameter definitions
if __name__ == "__main__":
code_path = "Path\\To\\My\\Project\\"
import sys
sys.path.insert(0, code_path)
import main # Everything is then executed in main.py (where config.py is also cross-imported)
Is there any way to enforce the portion of code inside the if above to only be executed once even if cross-imported, but without relying on __name__ == "__main__"? Or any other way to handle this at all, while keeping all parameters and configuration data within one single, simple file.
By the way, I am using IronPython for this (not exactly a choice), but since I am sticking to hopefully very simple stuff, I believe it is common to all python versions.
tl;dr: I want a config.py file with a bunch of parameters, including the directory where the program is located, to be accessible to multiple .py files. I want to avoid needing the project directory written in the "main" code file, since that should be a parameter and therefore only in config.py. The whole thing is passed to and executed by a third external program, so the directory where these files are located is not the same as where they are called from (therefore the project directory has to be included to system path at some point to import the different files).
A possible design alternative that is fairly common would be to rely on environment variables configured with a single file. Your multi-program system would then be started with some run script and your python application would then need to use something along the lines of os.env[…] to get/set/check the needed variables. Your directory would then look something along the lines of:
.
.
.
.env (environment variables - doesn't have to be called .env)
main.py
run.sh (starts system of programs - doesn't have to be called run.sh)
.
.
.
For the run script, you could then "activate" the environment variables and, after, start the relevant programs. If using bash as your terminal:
source .env # "activate" your environment variables
# Then the command to start whatever you need to; for example
#
# python main.py
# or
# ./myprogram
Related
What I want to achieve: I have a function (object/module/etc) that allows me to run a model. This model requires a few parameters, so I want to have them in a config file rather than pass all through the code. So what I want is to have a config file created automatically when my module is imported. Is it possible in python? Can I have some pointers on where to start please?
All the code in a python file is run when its imported. If you have
def f():
print("ran")
print("imported")
Then when you import it it will print imported
This is also why you sometimes see if __name__=="__main__":
In some files. The code in that block is only run if the file is run as main not imported.
However creating files in a predetermined location may be bad UX, so if you want other people to use your library id think of a better solution.
Generally you can just put code at top level of a module and it will run.
I have a project where I want to structure the code in layers. The different parts of the program do very different things, and I wish to have a clean upper layer which binds all of the code in sub-directories together.
However, I struggle with importing modules correctly.
Say I have the structure
Project
└──manage.py
└──part a
├──script_a.py
├──__init__.py
└──modules_a
├──module_a1.py
├──module_a2.py
├──module_a3.py
└──__init__.py
└──part b
├──script_b.py
├──__init__.py
└──modules_b
├──module_b1.py
├──module_b2.py
├──module_b3.py
└──__init__.py
If I am writing code in script_a.py that depends on something from module_a1.py I use
from modules_a import module_a1
This works, but VS Code is never happy about the importing, always marking the imports with error. Therefore, I am wondering if there is something that I have logically misunderstood, especially since the script_a.py is not in the root folder?
If you are within a package and you want to access a sub package you have to put a . in front of the sub package. So change your import statement from
from modules_a import module_a1
to
from .modules_a import module_a1
Then the error disappears.
I decided to solve it by adding a testing file in the root folder and only running the script from the testing file, which will have similar functionality to the manage.py that will be my execution script later.
I have simple question.
I have a python module "dev1.py" that needs a file "dev1_blob"
If I have everything in one directory.
my_app loads the dev1 like
from dev1 import func1
it works fine.
I want to move dev1.py to a directory "./dev_files" with init.py in it.I can load the dev1.py as
from dev_files.dev1 import func1
However when func1 runs to access the "device_blob" -- it barfs as:
resource not found ..
This is so basic that I believe I am missing something.
I can't figure out why great minds of python want everything to refer to __file__ (cwd) and force me to modify dev1.py based on where it's being run from. i.e. in dev1.py refer to the file as: 'dev_files/device_blob'
I can make it work this way, but it's purely absurd way of writing code.
Is there a simple way to access a file next to the module files or in the tree below?
Relative pathing is one of Python's larger flaws.
For this use case, you might be able to call open('../dev_files/device_blob') to go back a dir first.
My general solution is to have a "project.py" file containing an absolute path to the project directory. Then I call open(os.path.join(PROJECT_DIR, 'dev_files', 'device_blob')).
Using Linux, while the program itself and some other config files are located in one of the user's home directory (and run from there), because of hardware restrictions I am forced to run the main Python code prefixed by sudo. Now I want to call a function that already exists in one other Python file inside that user directory.
Simply using import functionfile does not seem to work, because (I assume) the program looks for that file in the root directory.
For "normal" tasks such as handling a config file that exists in that user directory, I can recreate the complete path by getting the user that was logged in before launching the sudo command, something like:
import os
sudo_username = os.getenv("SUDO_USER")
home_dir = "/home/" + sudo_username
However, I cannot imagine a method to import functionfile from other user directory, especially if I want to avoid absolute reference to usernames.
Is this technically possible ?
It is possible, if you import sys and add the directory to sys.path (or alternatively if you modify the PYTHONPATH), and the file is readable by root.
But it may not be a good idea. It may leave a root-owned .pyc file in that directory that the user cannot overwrite, and it is a huge security hole for a random user to be able to change code that root will be running.
I am trying to be able to dynamically run python code, with variables being able to be passed through to the code. I was able to do this on my computer before I added my project to the google app engine environment (because I can access all the files, but now, with google app engine, I can not do that.
I am struggling to find a solution to this problem. It does not need to be too terribly fancy, just send variables in and get html out, as well as scripts being able to be added client side (the crucial part) to whatever database method that is used.
Edit:
well basically what I mean by dynamically is so that I can import (or thats what I did in IDLE when I tested the prototype, the solution will probably not be called importing) a python script with the name of the library being stored in a variable, as well as an unknown number of variables that would be added. I got this to work on Idle, but now I need to get it to work in the google app engine environment, and people need to be able to upload scripts as well (which is the main problem that cascades into many more problems)
Edit:
When I say that I managed to get this to work on my local machine, I mean I was able to manually drop scripts into the same directory as my main script. The script would later import and execute the scripts when necessary. I was able to get this to work with the following code:
#calling function
mod = __import__('actions.'+folder+'.'+FILE)
VAR = getattr(getattr(mod, folder), FILE)
response = VAR.Main()
print response
This code worked on both my laptop and in the google app engine environment, But When I try to add more scripts to the directory is when things get problematic. On my laptop I could just move the file over one way or another because I had full access to the file directory. On Google App engine I do not have the ability to just upload a file to the same directory or subdirectory of the rest of my python scripts. So basically the problem comes up when trying to design a way to allow more code to come into the system (in my case, adding more 'plugins').
The answer is the exec statement (also known as the exec() function) or the eval() function. See http://docs.python.org/reference/simple_stmts.html#the-exec-statement and http://docs.python.org/library/functions.html?highlight=eval#eval. These can execute arbitrary Python code from a string. exec() runs a script and you get the side effects; eval() takes an expression and returns its value. Typically you pass input in as variables in the local namespace.
Ok, So what I eventually did was use the datastore to upload everything such as the name, description, uploader and code of the plugin (for now the code is just entered into a textarea box). I then, instead of importing a file located in a folder under the same directory of my code like I had before when running everything off of my desktop, Imported the plaintext code into a module using this little bit of magic:
#Initiating Variables for use by importing functions
module_name = 'mymod'
filename = 'action_file'
source = PossibleMatches[0][1] #the source code from the best matched option
# define module_name somewhere
import types
module = types.ModuleType(module_name)
# source should the code to execute
# filename should be a pseudo-filename that the code's from
# (it doesn't actually have to exist; it's used for error messages)
code_object = compile(source, filename, 'exec')
#execute the code in the context of the module
exec code_object in module.__dict__
#Executing the 'Main' Function from the code
return module.Main()