I wrote a package, which lives in /home/michael/python/mcdb-mail-parser/
The file structure in there is:
.
├── __init__.py
├── mcdb_mail_parser
│ ├── __init__.py
│ ├── MCDBAttachment.py
│ ├── MCDBEmail.py
│ ├── Options.py
├── mcdb-mail-parser.conf.sample
├── mcdb-mail-parser.py
├── README.md
mcdb-mail-parser.py imports from the mcdb_mail_parser subdirectory.
if I run the scripts from the source directory (/home/michael/python/src/mcdb_mail_parser) it works fine because the mcdb_mail_parser directory is immediately available in the current directory. However, I need to run it from the home directory of another user (via a cronjob, or from another script via subprocess), python complains it cannot find the module:
I tried to execute it with python3 -m /home/michael/python/src/mcdb_mail_parser, but it complains:
michael#d8:~$ python3 -m /home/michael/python/mcdb-mail-parser/
/usr/bin/python3: No module named /home/michael/python/mcdb-mail-parser/
I am not sure where to go from here. I think that it's a path issue. I could add /home/michael/python/src/mcdb_mail_parser to the system path, or perhaps python path, but that seems like the wrong solution. I certainly don't want to hard code paths into any scripts either.
How do I tell python: "Run the mcdb-mail-parser.py script from /home/michael/python/src/mcdb_mail_parser directory?
Closing notes
The accepted answer was useful, and so was the link they provided. Here's what I eventually did:
1. I moved the contents of mcdb_mail_parser from the subdirectory into the same directory as README.md, thus removing one level of complexity.
2. I added the import statements to __init__.py as suggested.
3. Python complained it couldn't find __main__.py, so I renamed mcdb-mail-parser.py to __main__.py
List the modules in the __init__.py that's in the sub directory and then have the Import in mcdb-mail-parser.py reference that directory
Very similar to this previous StackOverflow Post ->
Loading all modules in a folder in Python
Related
I have such structure of project:
lib/
...
scripts/
...
I have many Python scripts in the scripts/ directory. All of them contains relative imports: from lib import ...
So, how can I easy run scripts from the root of project /, without changing scripts (without write chdir in each script)?
Maybe can I use some __init__ file to change work dir? Or maybe can I use special command to run python scripts with root folder? Any other ways?
Your question is not clear to me and its to much for a comment... Your structure is like that?
root/
├── lib/
│ ├── __init__.py
│ ├── lib_foo.py
│ ├── lib_bar.py
├── scripts/
│ ├── script_util.py
│ └── script_yeah.py
└── main.py
And your program starts always at main.py? Or do you have python files with main also in the scripts folder ?
Never use chdir except if you have a very good reason. Add init files as detailed in the other answer and run your script from the parent directory (say root) as
$ python -m scripts.yourscript # note no .py
If you must run the scripts from their own directory and not use the suggestion from Mr_and_Mrs_D, then the simplest way to handle your use case is to manipulate the search path for the default module finder:
import sys
sys.path.append('..')
from lib import foo
sys.path.pop()
print(foo.bar())
It was a decision in Python to prevent explicitly loading a module from a relative folder above the executing process root. You can do it by creating a new 'finder' for importlib that returns a ModuleSpec, but the amount of code needed is somewhat excessive.
I have a directory structure with 2 basic python files inside seperate directories:
├── package
│ ├── subpackage1
│ │ └── module1.py
└── subpackage2
└── module2.py
module1.py:
def module1():
print('hello world')
module2.py:
from ..subpackage1.module1 import module1
module1()
When running python3 module2.py I get the error: ImportError: attempted relative import with no known parent package
However when I run it with the imports changed to use sys.path.append() it runs successfully
import sys
sys.path.append('../subpackage1/')
from module1 import module1
module1()
Can anyone help me understand why this is and how to correct my code so that I can do this with relative imports?
To be considered a package, a Python directory has to include an __init__.py file. Since your module2.py file is not below a directory that contains an __init__.py file, it isn't considered to be part of a package. Relative imports only work inside packages.
UPDATE:
I only gave part of the answer you needed. Sorry about that. This business of running a file inside a package as a script is a bit of a can of worms. It's discussed pretty well in this SO question:
Relative imports in Python 3
The main take-away is that you're better off (and you're doing what Guido wants you to) if you don't do this at all, but rather move directly executable code outside of any module. You can usually do this by adding an extra file next to your package root dir that just imports the module you want to run.
Here's how to do that with your setup:
.
├── package
│ ├── __init__.py
│ ├── subpackage1
│ │ └── module1.py
│ └── subpackage2
│ └── module2.py
└── test.py
test.py:
import package.subpackage2.module2
You then run test.py directly. Because the directory containing the executed script is included in sys.path, this will work regardless of what the working directory is when you run the script.
You can also do basically this same thing without changing any code (you don't need test.py) by running the "script" as a module.
python3 -m package.subpackage2.module2
If you have to make what you're trying to do work, I think I'd take this approach:
import os, sys
sys.path.append(os.path.join(os.path.dirname(__file__), '..'))
from subpackage1.module1 import module1
module1()
So you compute in a relative way where the root of the enclosing package is in the filesystem, you add that to the Python path, and then you use an absolute import rather than a relative import.
There are other solutions that involve extra tools and/or installation steps. I can't think why you could possibly prefer those solutions to the last solution I show.
By default, Python just considers a directory with code in it to be a directory with code in it, not a package/subpackage. In order to make it into a package, you'll need to add an __init__.py file to each one, as well as an __init__.py file to within the main package directory.
Even adding the __init__.py files won't be enough, but you should. You should also create a setup.py file next to your package directory. Your file tree would look like this:
├── setup.py
└── package
├── __init__.py
└── subpackage1
│ ├── __init__.py
│ └── module1.py
└── subpackage2
├── __init__.py
└── module2.py
This setup.py file could start off like this:
from setuptools import setup
setup(
name='package',
packages=['package'],
)
These configurations are enough to get you started. Then, on the root of your directory (parent folder to package and setup.py), you will execute next command in you terminal pip install -e . to install your package, named package, in development mode. Then you'll be able to navigate to package/subpackage2/ and execute python module2.py having your expected result. You could even execute python package/subpackage2/module2.py and it works.
The thing is, modules and packages don't work the same way they work in another programming languages. Without the creation of setup.py if you were to create a program in your root directory, named main.py for example, then you could import modules from inside package folder tree. But if you're looking to execute package\subpackage2\module2.py.
If you want relative imports without changing your directory structure and without adding a lot of boilerplate you could use my import library: ultraimport
It gives the programmer more control over their imports and lets you do file system based relative or absolute imports.
Your module2.py could then look like this:
import ultraimport
module1 = ultraimport('__dir__/../subpackage1/module1.py')
This will always work, no matter how you run your code or if you have any init files and independent of sys.path.
I have a question in how to properly create a path in Python (Python 3.x).
I developed a small scraping app in Python with the following directory structure.
root
├── Dockerfile
├── README.md
├── tox.ini
├── src
│ └── myapp
│ ├── __init__.py
│ ├── do_something.py
│ └── do_something_else.py
└── tests
├── __init__.py
├── test_do_something.py
└── test_do_something_else.py
When I want to run my code, I can go to the src directory and do with
python do_something.py
But, because do_something.py has an import statement from do_something_else.py, it fails like:
Traceback (most recent call last):
File "src/myapp/do_something.py", line 1, in <module>
from src.myapp.do_something_else import do_it
ModuleNotFoundError: No module named 'src'
So, I eventually decided to use the following command to specify the python path:
PYTHONPATH=../../ python do_something.py
to make sure that the path is seen.
But, what are the better ways to feed the path so that my app can run?
I want to know this because when I run pytest via tox, the directory that I would run the command tox would be at the root so that tox.ini is seen by tox package. If I do that, then I most likely run into a similar problem due to the Python path not properly set.
Questions I want to ask specifically are:
where should I run my main code when creating my own project like this? root as like python src/myapp/do_something.py? Or, go to the src/myapp directory and run like python do_something.py?
once, the directory where I should execute my program is determined, what is the correct way to import modules from other py file? Is it ok to use from src.myapp.do_something_else import do_it (this means I must add path from src directory)? Or, different way to import?
What are ways I can have my Python recognize the path? I am aware there are several ways to make the pass accessible as below:
a. write export PYTHONPATH=<path_of_my_choice>:$PYTHONPATH to make the
path accessible temporarily, or write that line in my .bashrc to make it permanent (but it's hard to reproduce when I want to automate creating Python environment via ansible or other automation tools)
b. write import sys; sys.path.append(<root>) to have the root as an accessible path
c. use pytest-pythonpath package (but this is not really a generic answer)
Thank you so much for your inputs!
my environment
OS: MacOS and Amazon Linux 2
Python Version: 3.7
Dependency in Python: pytest, tox
I would suggest to use setup.py to make this a python package. Then you can install it in development mode python setup.py develop. This way it will be available in your python environment w/o needing to specify the PYTHONPATH.
For testing, you can simply install the package python setup.py install.
Hope that helps.
Two simple steps should make it happen. Python experts can comment if this is a good way to do it (especially going by the concluding caution raised towards the end of this post).
I would have done it like below.
First I would have put a "__init__.py" in root so that hierarchy looks like below. This way python will treat the folder as a package.
root
├── Dockerfile
├── README.md
├── tox.ini
├── __init__.py
├── src
│ └── myapp
│ ├── __init__.py
│ ├── do_something.py
│ └── do_something_else.py
└── tests
├── __init__.py
├── test_do_something.py
└── test_do_something_else.py
Then in "do_something.py", I would have added these lines at the top. In the second line please put the full path to the "root" directory.
import sys
sys.path += ['/home/SomeUserName/SomeFolderPath/root']
from src.myapp.do_something_else import do_it
Please note that the second line will essentially modify the sys.path by adding the root folder path (I guess until the interpreter quits). If this is not what you can afford then I am sorry.
I am having problems with importing modules/packages with Python. I noticed this problem when I ran it in my terminal (CMD),
rather than my IDE (I use PyCharm). In PyCharm, I use a virtual enviroment setting with Python 3.7 and everything works as a charm
and as intended.
For reference this is how the imports were done in test_suite.py:
...
from tests.scenarios.test_scenario_01 import TestScenario # They work perfectly fine
from tests.scenarios.test_scenario_02 import TestScenario2 # written like this in PyCharm venv Python 3.7, but why?
...
This is a simplified version of my directory (without the unneccesary files):
QA System/
├── locators/
│ ├── locators.py
│ ├── __init__.py
├── pages/
│ ├── pages.py
│ └── __init__.py
└── tests/
├── reports
├── test_scenarios
├── test_scenario_01.py
├── test_scenario_02.py
├── __init__.py
|── test_suite.py
|── __init__.py
However when running the file test_suite.py manually through my CMD (because I want to integrate it with Jenkins
eventually), I get this error (py -3 test_suite.py):
ModuleNotFoundError: No module named 'tests'
Note: I am using the newest Python 3.7
From what I know about Python imports, for a directory to be treated like a python module, there needs to be a '__init__.py' file
included in the same directory.
After a bit of research I found out that it is possible to do a different type of imports in Python 3 and tried it out (putting a .
before the name of the imports). Like this:
from .scenarios.test_scenario_01 import TestScenario
from .scenarios.test_scenario_02 import TestScenario2
But still, it didn't run successfully and this was the error I've gotten:
ModuleNotFoundError: No module named '__main__.scenarios'; '__main__' is not a package
Could you please help me out on this one?
TLDR: Imports work in a Python3.7 venv, but not outside it
I fixed the issue by setting up a PYTHONPATH in system environment variables with the path to the project. As a value, I put the dir to the project. Thanks for the tips.
In case of directory It is mandate to use init.py. In modules it does not require to include init.py. Please, check what is your main directory and which is your modules..Hope this helps....
I have a python 3 package with the following structure:
.
├── package
│ └── bin
└── main_module
│ └── lib
│ ├── __init__.py
│ ├── module1.py
│ ├── module2.py
│ └── module3.py
│ └── test
│ ├── test1.py
│ ├── test2.py
│ └── test3.py
│ └── setup.py
Usually, one runs $ python3 setup.py install and all is good. However, I want to use this package on a cluster server, where I don't have write permissions for /usr/lib/. The following solutions came to my mind.
Somehow install the package locally in my user folder.
Modify the package such that it runs without installation.
Ask the IT guys to install the package for me.
I want to avoid 3., so my question is whether 1. is possible and if not, how I have to modify the code (particularly the imports) in order to be able to use the package without installation. I have been reading about relative imports in python all morning and I am now even more confused than before. I added __init__.py to package and bin and from what I read I assumed it has to be from package.lib import module1, but I always get ImportError: No module named lib.
In order for Python to be able to find your modules you need to add the path of your package to sys.path list. As a general way you can use following snippet:
from sys import path as syspath
from os import path as ospath
syspath.append(ospath.join(ospath.expanduser("~"), 'package_path_from_home'))
os.path.expanduser("~") will give you the path of the home directory and you can join it with the path of your package using os.path.join and then append the final path to sys.path.
If package is in home directory you can just add the following at the leading of your python code that is supposed to use this package:
syspath.append(ospath.join(ospath.expanduser("~"), 'package'))
Also make sure that you have an __init__.py in all your modules.
I had the same problem. I used the first approach
install the package locally in my user folder by running
python setup.py install --user
This will install your module in ~/.local/lib/python3/
Just add the path of your 'package' to environment variable PYTHONPATH. This will get rid of the error you are getting.
OR
programmatically add path of the package to sys.path.append()
you can add this to the "main file" of the package
import sys, os
sys.path.append(os.path.dirname(__file__) + "/..")
you can find the "main file" by looking for this pattern
if __name__ == "__main__":
some_function()