Importing an arbitrary amount of python files - python

I have different python files containing Neural Networks. Each python file has associated weights.h5 file.
Now I'd like to make a python evaluation file, which loads all networks / python files and their weights, creates one instance of each and compares their performance.
So far I tried to import as package but then I'm unable to access the modules by an index. How could I import all of the models and put one instance of them in a list, such that I can access them by an index?
An example
from evaluation.v03 import DQNSolver as DQN1
from evaluation.v04 import DQNSolver as DQN2
from evaluation.v05 import DQNSolver as DQN3
...
this works, but I have to hard code each import. Additionally I was not able to create instances by an index or access them by an index to make comparisons between all of the them.

Use __import__() function instead of import statement. Like this:
modules = []
for i in range(10):
modules.append( __import__('evaluation.v{:>02}'.format(i)) )
Then you can access them like modules[x].DQNSolver

Making use of import_module(), which is recommended over using __import__() directly:
from importlib import import_module
solvers = [getattr(import_module(f'evaluation.v{i:02d}'), 'DQNSolver') for i in range(5)]
solver = solvers[1]()
# solver -> <evaluation.v01.DQNSolver object at 0x7f0b7b5e5e10>

Related

Partial Imports: Import everything up to a error

Is there a way to perform a python import which is not atomic?
For instance, I have a file as follows:
# Filename: a.py
myvariable = 1
mylist = [1, 2, 3]
raise ImportError
donotimportthis = 5
I then have a separate file which does the following:
import a
a.myvariable == 1 # This is okay as it imported it
a.donotimportthis # <-- raise an exception as this is not imported.
I have a file which contains some python code, this follows the format of:
...variables...
import X
I do not have X installed nor do I want it however I do want the variables.
Note: This file is autogenerated not by me but by a tool whose version is frozen.
Two choices, in descending order of preference:
Change the autogeneration process. Instead of invoking proprietary_autogen_process, invoke custom_autogen_wrapper. This wrapper in turn first invokes the proprietary third-party tool, and then modifies the produced module source code by searching for the code that imports module X, and deletes everything after it.
This is relatively straightforward. You just need to take some care to not introduce false positives or false negatives by performing too loose (or too strict) matching of the import code. Ideally you’d use an AST rewriter but that’s probably overkill; a regular expression search for import X might work, although it will yield wrong results if this text appears inside a comment, string literal or inside a method which isn’t executed.
Generate a stub module X in a location where it will be found by the autogenerated module when importing the latter. I don’t recommend this because it’s tedious: You probably can’t just generate an empty module, since the autogenerated module will want to use X. You need to generate meaningful method stubs.
You can do specific imports with
from a import myvariable
EDIT: The above won't work if anything that is flat in the file raises an error. If you have no way to edit the imported file then I don't know if there is a (resonable) solution to this. Sorry didn't realise.
(an unreasonable solution would be to read in the file as text, slice it, and then run eval on it).
Or, as mentioned in the comments, put the stuff you don't want under
if __name__=="__main__":
<here>
Then it will only be invoked if you run the file directly.
What you can do is removing the donotimportthis variable at the end of the module, as follows: del donotimportthis. I hope it helps

Cannot import a function from a module

Basically I have 3 modules that all communicate with eachother and import eachother's functions. I'm trying to import a function from my shigui.py module that creates a gui for the program. Now I have a function that gets the values of user entries in the gui and I want to pass them to the other module. I'm trying to pass the function below:
def valueget():
keywords = kw.get()
delay = dlay.get()
category = catg.get()
All imports go fine, up until I try to import this function with
from shigui import valueget to another module that would use the values. In fact, I can't import any function to any module from this file. Also I should add that they are in the same directory. I'm appreciative of any help on this matter.
Well, I am not entirely sure of what imports what, but here is what I can tell you. Python can sometimes allow for circular dependencies. However, it depends on what the layout of your dependencies is. First and foremost, I would say see if there is any way you can avoid this happening (restructuring your code, etc.). If it is unavoidable then there is one thing you can try. When Python imports modules, it does so in order of code execution. This means that if you have a definition before an import, you can sometimes access the definition in the first module by importing that first module in the second module. Let me give an example. Consider you have two modules, A and B.
A:
def someFunc():
# use B's functionality from before B's import of A
pass
import B
B:
def otherFunc():
# use A's functionality from before A's import of B
pass
import A
In a situation like that, Python will allow this. However, everything after the imports is not always fair game so be careful. You can read up on Python's module system more if you want to know why this works.
Helpful, but not complete link: https://docs.python.org/3/tutorial/modules.html

Best way to import several classes

I have defined several classes in a single python file. My wish is to create a library with these. I would ideally like to import the library in such a way that I can use the classes without a prefix (like mylibrary.myclass() as opposed to just myclass() ), if that's what you can call them, I am not entirely sure as I am a beginner.
What is the proper way to achieve this, or the otherwise best result? Define all classes in __init __? Define them all in a single file as I currently have like AllMyClasses.py? Or should I have a separate file for every class in the library directory like FirstClass.py, SecondClass.py etc.
I realize this is a question that should be easy enough to google, but since I am still quite new to python and programming in general I haven't quite figured out what the correct keywords are for a problem in this context(such as my uncertainty about "prefix")
More information can be found in the tutorial on modules (single files) or packages (when in a directory with an __init__.py file) on the python site.
The suggested way (according to the style guide) is to spell out each class import specifically.
from my_module import MyClass1, MyClass2
object1 = MyClass1()
object2 = MyClass2()
While you can also shorten the module name:
import my_module as mo
object = mo.MyClass1()
Using from my_module import * is recommended to be avoided as it can be confusing (even if it is the recommended way for some things, like tkinter)
If it's for your personal use, you can just put all your classes Class1, Class2, ... in a myFile.py and to use them call import myFile (without the .py extension)
import myFile
myVar1 = myFile.Class1()
myVar2 = myFile.Class2()
from within another script. If you want to be able to use the classes without the file name prefix, import the file like this:
from myFile import *
Note that the file you want to import should be in a directory where Python can find it (the same where the script is running or a directory in PYTHONPATH).
The _init_ is needed if you want to create a Python module for distribution. Here are the instructions: Distributing Python Modules
EDIT after checking the Python's style guide PEP 8 on imports:
Wildcard imports (from import) should be avoided, as they make it unclear which names are present in the namespace, confusing both readers and many automated tools
So in this example you should have used
from myFile import Class1, Class2

What's the correct way for importing the whole module as well as a couple of its functions in Python?

I know that from module import * will import all the functions in current namespace but it is a bad practice. I want to use two functions directly and use module.function when I have to use any other function from the module. What I am doing currently is:
import module
from module import func1, func2
# DO REST OF MY STUFF
Is it a good practice? Does the order of first two statements matter?
Is there a better way using which I can use these two functions directly and use rest of the functions as usual with the module's name prepended to them?
Using just import module results in very long statements with a lot of repetition if I use the same function from the given module five times in a single statement. That's what I want to avoid.
The order doesn't matter and it's not a pythonic way. When you import the module there is no need to import some of its functions separately again. If you are not sure how many of the functions you might need to use just import the module and access to the functions on demand with a simple reference.
# The only import you need
import module
# Use module.funcX when you need any of its functions
After all, if you want to use some of your functions (much) more than the others, as the cost of attribute access is greater than importing the functions separately, you better to import them as you've done.
And still, the order doesn't matter. You can do:
import module
from module import func1, func2
For more info read the documentation https://www.python.org/dev/peps/pep-0008/#imports
It is not good to do (may be opinion based):
import module
from module import func1, func2 # `func1` and `func2` are already part of module
Because you already hold a reference to module.
If I were you, I would import it in the form of import module. Since your issue is that module.func1() becomes too long. I may import the module and use as for creating a alias for the name. For example:
import module as mo
# ^ for illustration purpose. Even the name of
# your actual module wont be `module`.
# Alias should also be self-explanatory
# For example:
import database_manager as db_manager
Now I may access the functions as:
mo.func1()
mo.func2()
Edit: Based on the edit in actual question
If your are calling same function in the same line, there is possibility that your are already doing some thing wrong. It will be great if you can share what your that function does.
For example: Want to the rertun value of those functions to be passed as argument to another function? as:
test_func(mo.func1(x), mo.func1(y). mo.func1(z))
could be done as:
params_list = [x, y, z]
func_list = [mo.func1(param) for param in params_list]
test_func(*func_list)

Passing class functions to PySpark RDD

I have a class named some_class() in a Python file here:
/some-folder/app/bin/file.py
I am importing it to my code here:
/some-folder2/app/code/file2.py
By
import sys
sys.path.append('/some-folder/app/bin')
from file import some_class
clss = some_class()
I want to use this class's function named some_function in map of spark
sc.parallelize(some_data_iterator).map(lambda x: clss.some_function(x))
This is giving me an error :
No module named file
While class.some_function when I am calling it outside map function of pyspark i.e. normally but not in pySpark's RDD. I think this has something to do with pyspark. I have no idea where am I going wrong in this.
I tried broadcasting this class and still didn't work.
All Python dependencies have to be either present on the search path of the worker nodes or distributed manually using SparkContext.addPyFile method so something like this should do the trick:
sc.addPyFile("/some-folder/app/bin/file.py")
It will copy the file to all the workers and place in the working directory.
On a side note please don't use file as module name, even if it is only an example. Shadowing built-in functions in Python is not a very good idea.

Categories