I need some help with working with a folder structure in python. I was given an structure like this:
/main-folder
/assets
somefiles.txt
/integrations
/module-folder
__init__.py
ingestion.py
__init__.py
models.py
Inside ingestion.py I have:
import os
from models import MyModel
PROJECT_DIR = os.path.dirname(os.path.dirname(os.path.dirname(__file__)))
some_function()
some_processing()
if __name__ == "__main__":
some_function()
Both __init__.py mentioned above are empty.
So I need to process some info and use the models module to store them. when trying to execute intestion.py directly from its dir it says: No module named 'models'. So I'm guessing I have to execute the whole thing as a package. I have no idea how should I import a module located above the package and can't touch the structure.
Any help woud be appreciated.
What you have to do is to add the module's directory to the PYTHONPATH environment variable. If you don't want to do this however, You can modify the sys.path list in your program where the Python interpreter searches for the modules to import, the python documentation says:
When a module named spam is imported, the interpreter first searches for a built-in module with that name. If not found, it then searches for a file named spam.py in a list of directories given by the variable sys.path. sys.path is initialized from these locations:
the directory containing the input script (or the current directory).
PYTHONPATH (a list of directory names, with the same syntax as the shell variable PATH).
the installation-dependent default.
After initialization, Python programs can modify sys.path. The directory containing the script being run is placed at the beginning of the search path, ahead of the standard library path. This means that scripts in that directory will be loaded instead of modules of the same name in the library directory. This is an error unless the replacement is intended.
Knowing this, you can do the following in your program:
import sys
# Add the main-folder folder path to the sys.path list
sys.path.append('/path/to/main-folder/')
# Now you can import your module
from main-folder import models
# Or just
import main-folder
Related
Assume I have the following files,
pkg/
pkg/__init__.py
pkg/main.py # import string
pkg/string.py # print("Package's string module imported")
Now, if I run main.py, it says "Package's string module imported".
This makes sense and it works as per this statement in this link:
"it will first look in the package's directory"
Assume I modified the file structure slightly (added a core directory):
pkg/
pkg/__init__.py
plg/core/__init__.py
pkg/core/main.py # import string
pkg/string.py # print("Package's string module imported")
Now, if I run python core/main.py, it loads the built-in string module.
In the second case too, if it has to comply with the statement "it will first look in the package's directory" shouldn't it load the local string.py because pkg is the "package directory"?
My sense of the term "package directory" is specifically the root folder of a collection of folders with __init__.py. So in this case, pkg is the "package directory". It is applicable to main.py and also files in sub- directories like core/main.py because it is part of this "package".
Is this technically correct?
PS: What follows after # in the code snippet is the actual content of the file (with no leading spaces).
Packages are directories with a __init__.py file, yes, and are loaded as a module when found on the module search path. So pkg is only a package that you can import and treat as a package if the parent directory is on the module search path.
But by running the pkg/core/main.py file as a script, Python added the pkg/core directory to the module search path, not the parent directory of pkg. You do have a __init__.py file on your module search path now, but that's not what defines a package. You merely have a __main__ module, there is no package relationship to anything else, and you can't rely on implicit relative imports.
You have three options:
Do not run files inside packages as scripts. Put a script file outside of your package, and have that import your package as needed. You could put it next to the pkg directory, or make sure the pkg directory is first installed into a directory already on the module search path, or by having your script calculate the right path to add to sys.path.
Use the -m command line switch to run a module as if it is a script. If you use python -m pkg.core Python will look for a __main__.py file and run that as a script. The -m switch will add the current working directory to your module search path, so you can use that command when you are in the right working directory and everything will work. Or have your package installed in a directory already on the module search path.
Have your script add the right directory to the module search path (based on os.path.absolute(__file__) to get a path to the current file). Take into account that your script is always named __main__, and importing pkg.core.main would add a second, independent module object; you'd have two separate namespaces.
I also strongly advice against using implicit relative imports. You can easily mask top-level modules and packages by adding a nested package or module with the same name. pkg/time.py would be found before the standard-library time module if you tried to use import time inside the pkg package. Instead, use the Python 3 model of explicit relative module references; add from __future__ import absolute_import to all your files, and then use from . import <name> to be explicit as to where your module is being imported from.
Say I have a folder named foo. Inside that folder is __init__.py, a folder called test, and another Python file called t1.py. Inside folder test is a Python file called bar.py, and in that file I am trying to do something like:
from foo import t1
And it gives me this error:
ModuleNotFoundError: No module named 'gmuwork'
Do I need to add something to environment variables or sys.path?
Absolute import
If you want to use
from foo import t1
Then yes, foo must be contained in sys.path.
From the docs:
When importing the package, Python searches through the directories on
sys.path looking for the package subdirectory.
In that case take a look at questions such as adding a file path to sys.path in python.
Relative import
Alternatively inside of bar.py you should be able to use
from ..foo import t1
as an intra-package reference.
Lastly: either way, you should put another empty __init__.py file inside of test to let Python know that folder is a subpackage.
I have a custom module that I am trying to read from a folder under a hierarchy:
> project-source
/tests
/provider
my_provider.py
settings_mock.py
__init__.py
I am trying to call, from my_provider.py
import tests.settings_mock as settings
Example from command line:
project-source> python tests/provider/my_provider.py
Error:
... ImportError: No module named settings_mock
I keep getting No module named settings_mock as error. I have already exported project_source path to PYTHONPATH. I have made tests into a package by creating a __init__.py file in its root, but no change in the error then.
I can print the settings_mock.py attributes when cd'ing project source
>>> import tests.settings_mock as settings
>>> print settings.storage_provider
correct storage provider value
Is anyone able to point out my mistake here? Thanks!
You only have one small mistake. To use subfolders, you need __init__.py, not init.py as you stated in the question. The difference is that __init__ is a builtin function of python, whereas init is not. Having this file in each subfolder tells the pyhon interpreter that the folder is a "package" that needs to be initialized.
UPDATED: It should be noted that python usually runs from the current directory that the script is located. If your executable main script is my_provider.py, then it's not going to know what to import, since the main script is located in a lower directory than the object it is trying to import. Think of it as a hierarchy. Scripts can only import things that are beneath them. Try separating out the executable from everything else in that file, if there are things that settings_mock needs to import.
I am trying to run some python 3.4 example on visual studio 2013. when I try to import some module from a parent folder and run it from inside visual studio 2013, I always has the error of ImportError: No module named 'foo'
However, when I run it from the console using the python command python boo.py, it executes well.
As an example, this is my project structure
myproject/
foo.py
__init__.py
koo/
boo.py
__init__.py
foo.py content is
def do1():
print('Inside foo module')
boo.py content is
import sys
sys.path.append("..")
import foo
foo.do1()
I guess, this issue is not about Visual Studio and not about why it doesn't work in VS. The real question is why it works in the terminal. Probably it is because the terminal runs under different environmental settings, where the python interpreter can find the parent directory and thus the foo.py:
When a module named spam is imported, the interpreter first searches
for a built-in module with that name. If not found, it then searches
for a file named spam.py in a list of directories given by the
variable sys.path. sys.path is initialized from these locations:
the directory containing the input script (or the current directory).
PYTHONPATH (a list of directory names, with the same syntax as the
shell variable PATH).
the installation-dependent default.
After initialization, Python programs can modify sys.path. The
directory containing the script being run is placed at the beginning
of the search path, ahead of the standard library path. This means
that scripts in that directory will be loaded instead of modules of
the same name in the library directory. This is an error unless the
replacement is intended. See section Standard Modules for more
information.
So, add the parent dir to pythonpath, and it will work.
Or modify sys.path adding the parent dir to it.
Try this it worked for me:
import sys
sys.path.append("Folder PATH"). "Folder PATH" given as...C:\\Working_directory\\VSProject
import <MODULE_NAME>
I am creating module (with submodules). Lets call it lib. I am trying to make it work as following:
I am able to run it (there is lib.__main__). It uses lib.utils inside.
When executed part of its job is to load other file/module passed by user. Currently it does it by importlib.import_module( name ).
This loaded module also needs to use lib.utils.
I am having following choice:
In loaded module use import utils instead of import lib.utils. I find it somehow misleading and would like to aviod this.
Run module in any external way, even using file with only import lib.__main__ inside.
Only other thing I have tought of was doing sys.path.append(os.getcwd()). Not only it seems very dirty, but also makes log.utils module to load twice.
Is there anything that would allow me to execute lib.__main__, but require using import lib.utils in loaded module?
From the docs
If the script name refers to a directory or zipfile, the script name is added to the start of sys.path and the main.py file in that location is executed as the main module.
In your case, if you run python lib mymodule, lib is added to sys.path and __main__.py is executed. lib is not a package, its simply a directory in sys.path. __main__.py is not in a package and so package-relative imports don't work.
Since lib is in sys.path, its top level .py files can be imported directly and any subdirectories with __init__.py are importable packages. So, both __main__.py and mymodule could do import utils and get the same thing.
Now it gets confusing. Because you are sitting in lib's parent directory and because there is a lib.__init__.py, lib.utils is also valid. Its only that way because of your current directory (or maybe you added the directory to PYTHONPATH or something). So, you've got two different modules as far as python is concerned because you got there on two different paths.
The solution is to delete lib/__init__.py. lib shouldn't be package. Then you have the question of what to do with the modules like lib/utils.py. Normally, one would create a package directory and move the scripts there so that you get namespace encapsulation. Supposing you call that directory mypackages, then __main__.py and mymodule.py could both import mypackages.utils.