We Keep Coding

Python import from own module

I have a module 'hydro' with the structure:
hydro/
__init__.py
read.py
write.py
hydro_main.py
This gets used as a submodule for several other modules, which have scripts with similar names:
scenarios/
__init__.py
read.py
write.py
scenarios_main.py
hydro/
__init__.py
read.py
write.py
hydro_main.py
In order to keep the script names straight, I want to specify the module name on import. So in the header of hydro_main.py, I'd have:
import hydro.read
and in scenarios_main.py, I'd have:
import hydro.read as read_hydro
import scenarios.read as read_scenarios
The problem is that when I attempt to run hydro_main.py from the package root, I get the following error:
ModuleNotFoundError: No module named 'hydro'
How can I set the package name for 'hydro' such that it will allow me to refer to the package name on import? I thought adding __init__.py was supposed to initialize the package, but __package__ still comes back as None.

You can import just the entire module as one instance.
import hydro
from hydro import read as read_hydro, hydro_main as main
hydro.hydro_main()
main() # same as above
hydro.read()
read_hydro() #same as above

It is a sub module so you have to use parentModule.subModule.* . Your first line will change to import scenarios.hydro.read as read_hydro
scenarios/hydro/hydro_main.py
print("I am in hydro_main")
scenarios/hydro/read.py
print("I am in hydro read")
scenarios/hydro/write.py
print("I am in hydro write")
scenarios/read.py
print("I am in scenarios read")
scenarios/write.py
print("I am in scenarios write")
scenarios/scenarios_main.py
import scenarios.hydro.read as read_hydro
import scenarios.read as read_scenarios
I am in hydro read
I am in scenarios read

How to conditionally import and run a method from another Python package?

I have a project that contains several test suites. I want to be able to specify which suite I'd like to run in the command line:
suite=multiplication python3 .
Here's my current file structure:
__main__.py
suites/
__init__.py
addition.py
subtraction.py
multiplication.py
division.py
suites/__init__.py
__all__ = ['addition', 'subtraction', 'multiplication', 'division']
subtraction.py
def testSuite():
# Bunch of tests
__main__.py
import os
import suites
# Get suite name from 'suite=xxx' in command line
suiteName = os.getenv('suite')
# Based on suiteName, load the correct file
suite = suites[suiteName]
# Call the suite loaded from the file
suite()
This errors out:
suite = suites[suiteName]
TypeError: 'module' object is not subscriptable
What's the best way to conditionally import and run a script from another package?

Use importlib.import_module:
from importlib import import_module
suite = import_module('suites.' + suiteName)
suite.testSuite()

Dynamically import packages with multiple files in Python

I am trying to dynamically import modules in python.
I need something like a plugin import system.
I use the following code for import of a module, and it works fine, as long as the entire code of the module is in the same file.
caller.py code:
PluginFolder = "./plugins"
MainModule = "__init__"
possibleplugins = os.listdir(PluginFolder)
for possible_plugin in possibleplugins:
location = os.path.abspath(os.path.join(PluginFolder, possible_plugin))
if not os.path.isdir(location) or not MainModule + ".py" in os.listdir(location):
continue
info = imp.find_module(MainModule, [location])
plugin = {"name": possible_plugin, "info": info}
After that, I use the load_module method to load the module:
module = imp.load_module(MainModule, *plugin["info"])
The structure of the directories is as follows:
Project/
--plugins/
----plugin_1/
-------__init__.py
-------X.py
caller.py
Everything works great when all the methods are in the same file (__init__.py).
When the method I use in __init__.py calls another method from another file (in the same package) than an error saying "No module named X"
the __init__.py code fails at the line:
import X
I also tried
import plugin_1.X
and other variations.
Just to make sure you guys understand- importing and using the module in a normal way (not dynamically) works fine.
What am I missing?
Is there another way to do this? Maybe use the __import__ method, or something else?

I usually use importlib. It's load_module method does the job. You can put the importing code into plugins/__init__.py to do the following:
import importlib
import os
import logging
skip = set(("__init__.py",))
plugins = []
cwd = os.getcwd()
os.chdir(os.path.dirname(__file__))
for mod in glob.glob("*.py"):
if mod in skip:
continue
try:
mod = mod.replace(".py", "")
plugin = importlib.import_module("." + mod, __name__)
plugin.append(plugin)
except Exception as e:
logging.warn("Failed to load %s: %s. Skipping", mod, e)
os.chdir(cwd)

Importing modules from different directories

I have a problem importing a module:
It is under this directory ./dao and the code that calls it is here ./core. Schematically represented as:
rnaspace/
__init__.py
core/
__init__.py
logger.py
dao/
__init__.py
storage_configuration_reader.py
This is the error message:
Traceback (most recent call last): File "logger.py", line 21, in <module>
from rnaspace.dao.storage_configuration_reader import storage_configuration_reader ImportError: No module named rnaspace.dao.storage_configuration_reader
This file it is there /rnaspace/dao/storage_configuration_reader.py and in the same folder the __init__.py file as follows:
""" Package dao
Gathers files that access to the plateform data
"""
If I understood well this question, it should work. I think that the problem is that one is not the subdirectory of the other (or that the path is not exaclly that one), there is a way to go around it? Or need I to apply the solution to this question?
EDIT
The __init__.py file of the rnaspace folder:
import rnaspace.dao.storage_configuration_reader as scr
def update_conf(conf_path, predictors_conf_dir):
scr.update_conf(conf_path, predictors_conf_dir)

from rnaspace.dao.storage_configuration_reader import storage_configuration_reader
That is wrong because there is no "storage_configuration_reader" directory in "dao" directory
This is how it should be:
from rnaspace.dao import storage_configuration_reader
EDIT:
or this way:
import rnaspace.dao.storage_configuration_reader

I finally found the solution in another question, it is using the module imp.
I just needed to add the name of the module, and the absolute path where it was:
imp.load_source("storage_configuration_reader","./rnaspace/dao/storage_configuration_reader.py")

Hiding implementation files in a package

I have a module called spellnum. It can be used as a command-line utility (it has the if __name__ == '__main__': block) or it can be imported like a standard Python module.
The module defines a class named Speller which looks like this:
class Speller(object):
def __init__(self, lang="en"):
module = __import__("spelling_" + lang)
# use module's contents...
As you can see, the class constructor loads other modules at runtime. Those modules (spelling_en.py, spelling_es.py, etc.) are located in the same directory as the spellnum.py itself.
Besides spellnum.py, there are other files with utility functions and classes. I'd like to hide those files since I don't want to expose them to the user and since it's a bad idea to pollute the Python's lib directory with random files. The only way to achieve this that I know of is to create a package.
I've come up with this layout for the project (inspired by this great tutorial):
spellnum/ # project root
spellnum/ # package root
__init__.py
spellnum.py
spelling_en.py
spelling_es.py
squash.py
# ... some other private files
test/
test_spellnum.py
example.py
The file __init__.py contains a single line:
from spellnum import Speller
Given this new layout, the code for dynamic module loading had to be changed:
class Speller(object):
def __init__(self, lang="en"):
spelling_mod = "spelling_" + lang
package = __import__("spellnum", fromlist=[spelling_mod])
module = getattr(package, spelling_mod)
# use module as usual
So, with this project layout a can do the following:
Successfully import spellnum inside example.py and use it like a simple module:
# an excerpt from the example.py file
import spellnum
speller = spellnum.Speller(es)
# ...
import spellnum in the tests and run those tests from the project root like this:
$ PYTHONPATH="`pwd`:$PYTHONPATH" python test/test_spellnum.py
The problem
I cannot execute spellnum.py directly with the new layout. When I try to, it shows the following error:
Traceback (most recent call last):
...
File "spellnum/spellnum.py", line 23, in __init__
module = getattr(package, spelling_mod)
AttributeError: 'module' object has no attribute 'spelling_en'
The question
What's the best way to organize all of the files required by my module to work so that users are able to use the module both from command line and from their Python code?
Thanks!

How about keeping spellnum.py?
spellnum.py
spelling/
__init__.py
en.py
es.py

Your problem is, that the package is called the same as the python-file you want to execute, thus importing
from spellnum import spellnum_en
will try to import from the file instead of the package. You could fiddle around with relative imports, but I don't know how to make them work with __import__, so I'd suggest the following:
def __init__(self, lang="en"):
mod = "spellnum_" + lang
module = None
if __name__ == '__main__':
module = __import__(mod)
else:
package = getattr(__import__("spellnum", fromlist=[mod]), mod)