I'm trying to build a Python program with the "structure" shown at the end.
The problem is that actions should be able to be executed as scripts as well (I've already included a main in them). When I try to execute DummyAction.py imports keep complaining they can't find misc.
How can I use DummyAction.py as a script and still use the functions in utils.py?
DummyAction.py contains a class called DummyActionClass, and the same for DummyTrigger.py. In utils.py there are several functions that both actions and triggers use and MMD.py contains the main.
/MMD
├── __init__.py
├── MMD.py
├── /actions
│ ├── __init__.py
│ ├── DummyAction.py
├── /misc
│ ├── __init__.py
│ ├── utils.py
└── /triggers
├── DummyTrigger.py
└── __init__.py
The import in DummyAction.py and DummyTrigger.py is:
from misc import utils
And the error is:
File "DDM/actions/DummyAction.py", line 11, in <module>
from misc import utils
ImportError: No module named misc
Seen the updated question, I think the problem is that you should do the import including the root of your dependecies tree: MMD.
So they should all look like:
from MMD.misc import utils
And also you need to call python with the -m option:
python -m MMD.actions.DummyAction
Edit: You said that MMD.py contains the main but it can't be your executable, and that's because is a module (is inside a directory with an __init__.py file). MMD is like your library so you need the executable to be outside and use such library.
You can find [here] some guidelines on how to organize your project.
If you can change you project structure I'll suggest to do it like this:
MMD/
├── runner.py
└── mmd
├── __init__.py
├── main.py
├── /actions
│ ├── __init__.py
│ ├── DummyAction.py
├── /misc
│ ├── __init__.py
│ ├── utils.py
└── /triggers
├── DummyTrigger.py
└── __init__.py
Then in any file inside the mmd directory every import should start with mmd, for example:
from mmd.misc import utils
from mmd.actions import DummyActions
And you put your main code that now is inside MMD.py inside a Main class in main.py, with something like:
# main.py
from mmd.misc import utils
class Main:
def start_session(self):
utils.function()
# etc ...
And then in runner.py you do something like:
# runner.py
from mmd.main import Main
cli = Main()
cli.start_session()
This way inside the MMD directory calling python runner.py you would execute your code, and you can also make executable runner.py so a simply ./runner.py will run your code.
And run your module with:
python -m mmd.actions.DummyAction
I'd do it like this, becasue this way is open to future implementation (and is almost like in the line guides).
If instead you can't, then you can give it a try at removing __init__.py from the MMD directory.
I assume this is a directory structure you're talking about, in which case python doesn't know where to look for utils.py - it tries the local directory, a few places in the path then gives up. It's simple enough to modify the path:
import sys
sys.path.append("/MMD/misc")
import utils
and you should be away.
I've found a "workarround"
try:
#When executing from the main
from misc import utils
except:
#When executing as a standalone script
from MMD.misc import utils
that allows to:
Call the module as a script (while using other modules): python -m MMD.actions.DummyAction
Call the main program and use the module: python MMD/MMD.py
Although I'm not sure if it's strictly correct to use a try - except block with an import, so feel free to add comments or other solutions.
The complete solution would be:
MMD.main
from misc import utils
from actions import DummyAction
class MMD():
def __init__(self):
a = DummyAction.DummyActionClass()
utils.foo()
if __name__ == '__main__':
d = MMD()
Then in actions/DummyAction.py:
try:
#When executing from the main
from misc import utils
except:
#When executing as a standalone script
from MMD.misc import utils
class DummyActionClass():
def __init__(self):
utils.foo()
if __name__ == '__main__':
a = DummyActionClass()
And finally in misc/utils.py:
def foo():
print "Foo was called"
Related
I have the following project structure in Python (the ... means I have n crawler_.py files).
project
├── crawlers
│ ├── __init__.py
│ ├── crawler_1.py
│ ├── crawler_2.py
│ ...
│ ├── crawler_n.py
│ └── useful_functions.py
├── main.py
└── __init__.py
I need to import all crawlers from crawler into main, so I use this.
# main.py
from crawlers import crawler_1
from crawlers import crawler_2
...
from crawlers import crawler_n
But I also need useful_functions.py inside all crawler_.py files, so I use this in each one.
# crawler_.py
import useful_functions
But when I ran main.py I got ModuleNotFoundError: No module named 'useful_functions' when it tried to import crawler_1.
So I tried the following
# crawler_.py
from crawlers import useful_functions
And it works when I run main.py. The problem is that I might want run only one of the crawler_.py directly. Using this last import statement, I get ModuleNotFoundError: No module named 'crawlers'. Not sure how to address this problem, if there's something inside the code I should adjust or if the structure that I'm using is fundamentally wrong (I'm perfectly okay with adjusting the project structure).
You can use this inside the crawler_n.py
if __name__ == '__main__':
import useful_functions
else:
import crawlers.useful_functions as useful_functions
__name__ == '__main__' checks if the module is called or is imported and thus makes the imports accordingly.
I have this code structure in python3:
- datalake
__init__.py
utils
__init__.py
utils.py
lambdas
__init__.py
my-lambdas.py
- tests
__init__.py
demo.py
All init__.py files are empty.
My problem is how I can import datalake module from tests/demo.py?
I tried from datalake.utils import utils in demo.py but when I run python tests/demo.py from command line, I get this error ModuleNotFoundError: No module named 'datalake'.
If I use this code:
from ..datalake.utils import utils
I will get error ValueError: attempted relative import beyond top-level package.
I also tried to import the module utils from my-lambda.py file which also failed. The code in my-lambda.py is from datalake.utils import utils but I get ModuleNotFoundError: No module named 'datalake' error when run python datalake/lambda/my-lambda.py from command line.
How can I import the module?
When you run a command like python tests/demo.py, the folder you are in does not get added to the PYTHONPATH, the script folder does. So a top-level import like import datalake will fail. To get around this you can run your tests as a module:
Python 2:
python -m tests/demo
Python 3:
python -m tests.demo
and any datalake imports in demo.py will work.
It sounds like what you really want to do is have a folder with tests separate to your main application and run them. For this I recommend py.test, for your case you can read Tests Outside Application Code for how to do it. TL;DR is run your tests from your top level project folder with python -m py.test and it will work.
First of all, my-lambdas.py is not importable with the import statement as hyphens are not valid in Python identifiers. Try to follow PEP-8's naming conventions, such as mylambdas.py.
Otherwise the package structure looks good, and it should be importable as long as you are at the level above datalake/, e.g., if you were in the directory myproject/ below:
myproject
├── datalake
│ ├── __init__.py
│ ├── utils
│ │ ├── __init__.py
│ │ └── utils.py
│ └── lambdas
│ ├── __init__.py
│ └── mylambdas.py
└── tests
├── __init__.py
└── demo.py
Then this should work:
~/myproject$ python -c 'from datalake import utils'
Otherwise, setting the environment variable PYTHONPATH to the path above datalake/ or modifying sys.path are both ways of changing where Python can import from. See the official tutorial on modules for more information.
Also some general advice: I've found it useful to stick with simple modules rather than packages (directories) until there is a need to expand. Then you can change foo.py into a foo/ directory with an __init__.py file and import foo will work as before, although you may need to add some imports to the __init__.py to maintain API compatibility. This would leave you with a simpler structure:
myproject
├── datalake
│ ├── __init__.py
│ ├── utils.py
│ └── lambdas.py
└── tests
├── __init__.py
└── demo.py
You can add the module directory into your sys.path:
import sys
sys.path.append("your/own/modules/folder") # like sys.path.append("../tests")
but this is a one-shot method, which is just valid at this time, the added path is not permanent, it will be eliminated after the code completed execution.
One of the ways to import the file directly instead of using from, like import util
you can try run :
python -m datalake.lambda.my-lambda
follow: https://docs.python.org/3.7/using/cmdline.html#cmdoption-m
I'm trying to import a file but I can only get it to work from one context at a time.
This my project structure:
.
├── module/
│ ├── __init__.py
│ ├── script.py
│ ├── utilities1.py
│ └── utilities2.py
└── test.py
script.py is usually called externally directly it imports utilities1.py
utilities1.py imports utilities2.py
test.py Is a file that contains tests and includes both utilities1.py and utilities2.py
My question is how to do the import statement in utilities1.py
When I call it from script.py it needs to be
import utilities2
But when I call it from test.py that results in an error requiring it to be
import module.utilities2
Is there a way I can get the import statement right in both contexts?
Or do I need to change something structurally in my project?
Thank you :)
If what you want is being able to use import utilities1 from test.py you could modify the search path of modules. sys.path is the list of paths where the interpreter will look for modules to import. Do print(sys.path) and you'll see. You can also modify it while running your script.
For example, keeping with the file structure you described
# script.py
import utilities1
import utilities2
utilities1.show_myself()
utilities2.show_myself()
# utilities1.py
def show_myself():
print("I'm utilities1")
def test_myself():
print("Testing who I am... The answer: utilities1")
# utilities2.py
def show_myself():
print("I'm utilities2")
def test_myself():
print("Testing who I am... The answer: utilities2")
# test.py
import sys
sys.path.insert(1, "module")
import utilities1
import utilities2
utilities1.test_myself()
utilities2.test_myself()
In test.py I have inserted module in sys.path which is the relative path where the running script will to be looking into for modules utilities1 or utilities2. That's why it is able to acess directly yo those two modules.
If that's not what you were trying to do, please explain further.
Update: I have changed my file directory
I have a directory structure as follows and I would like to import a module in a parent directory.
**project**/
__init__.py
main.py
**APP_NAME**/
**parser**/
__init__.py
parser.py
**test**/
__init__.py
parser_test.py
parser.py
class Parser(object):
pass
main.py (Works fine)
from APP_NAME.parser.parser import Parser
parser_test.py (Throws error)
from ..APP_NAME.parser.parser import Parser
Throws the following error at parser_test.py
Parent module '' not loaded, cannot perform relative import
I know I can fix it using sys.path.append(), but I want to import it like a package the way I did it in main.py.
Any help is appreciated. Thanks.
I had to check back at one of my projects for a reference.
To test files in the tests folder you must first create setup.py, so that you can install you project for python to use it.
If on linux use the command, sudo python setup.py install to install the package. When changes have been made to the project, you must install again for the changes to take place.
These folder will be created in your root project directory after installing.
build, dist, and project.egg-info.
You may need to clean the build directory before re-installing to update.
python setup.py clean
python setup.py build
python setup.py install
Project Structure
project
├── setup.py
├── tests
│ └── parser_test.py
│
└── project
├── __init__.py
├── __init__.pyc
├── main.py
└── parser
├── __init__.py
├── __init__.pyc
├── parser.py
└── parser.pyc
project/setup.py
from setuptools import setup
# Make sure the project name will not conflict with other libraries
# For example do not name the project, 'os', 'sys', ect.
setup(
name='project',
description='My project description',
author='your_online_name',
license='MIT', # Check out software licenses
packages=['project', 'tests']
)
project/tests/parser_test.py
from project.parser import Parser
parser = Parser()
project/project/__init__.py
from . import parser
project/project/parser/__init__.py
from .parser import Parser
project/project/parser/parser.py
class Parser(object):
pass
You shouldn't be using absolute import within your package. In-package imports should be done with relative imports this way:
parser_test.py
from ..parser.parser import Parser
With relative imports in Python, the first point refers to the file's directory and each extra point refers to the parent directory.
In this case, you would be pointing to the project/parser/parser.py file which from test_parser.py standpoint's is ../parser.py
If you are using Python 2, you should add the following line at the top of all the files in your parser package
from __future__ import absolute_import
This will avoid that you use absolute imports inside you package files by mistake.
Still assuming you are working with Python 2, you should also import unicode_literals for native unicode support and print_function to replace the print command by the print() function.
However, I would rather have my tests in the top folder of the package, which, assuming the package is called project and not parser, would give the following directory structure:
project/ # top project directory
├── main.py
└── project # top package directory
├── __init__.py # this file is required even if it is empty
├── parser
│ ├── __init__.py
│ └── parser.py
└── tests
└── test_parser.py
Also, the project/project/parser/__init__.py could contain the following:
from .parser import Parser
So that your main.py file could import the Parser class like this:
from project.parser import Parser
instead of the more tedious:
from project.parser.parser import Parser
Your test_parser.py file, however, will still have to import the Parser class like this:
from ..parser.parser import Parser
because the classes exposed in an __init__.py file are not available to relative imports.
Finally, if you are starting a new independent project, you should do it in Python 3 (that's a PEP recommendation), where all the above rules apply, except the from __future__ imports which are unnecessary.
Sources: https://axialcorps.wordpress.com/2013/08/29/5-simple-rules-for-building-great-python-packages/
TL;DR
Here's an example repository that is set up as described in the first diagram (below): https://github.com/Poddster/package_problems
If you could please make it look like the second diagram in terms of project organisation and can still run the following commands, then you've answered the question:
$ git clone https://github.com/Poddster/package_problems.git
$ cd package_problems
<do your magic here>
$ nosetests
$ ./my_tool/my_tool.py
$ ./my_tool/t.py
$ ./my_tool/d.py
(or for the above commands, $ cd ./my_tool/ && ./my_tool.py is also acceptable)
Alternatively: Give me a different project structure that allows me to group together related files ('package'), run all of the files individually, import the files into other files in the same package, and import the packages/files into other package's files.
Current situation
I have a bunch of python files. Most of them are useful when callable from the command line i.e. they all use argparse and if __name__ == "__main__" to do useful things.
Currently I have this directory structure, and everything is working fine:
.
├── config.txt
├── docs/
│ ├── ...
├── my_tool.py
├── a.py
├── b.py
├── c.py
├── d.py
├── e.py
├── README.md
├── tests
│ ├── __init__.py
│ ├── a.py
│ ├── b.py
│ ├── c.py
│ ├── d.py
│ └── e.py
└── resources
├── ...
Some of the scripts import things from other scripts to do their work. But no script is merely a library, they are all invokable. e.g. I could invoke ./my_tool.py, ./a.by, ./b.py, ./c.py etc and they would do useful things for the user.
"my_tool.py" is the main script that leverages all of the other scripts.
What I want to happen
However I want to change the way the project is organised. The project itself represents an entire program useable by the user, and will be distributed as such, but I know that parts of it will be useful in different projects later so I want to try and encapsulate the current files into a package. In the immediate future I will also add other packages to this same project.
To facilitate this I've decided to re-organise the project to something like the following:
.
├── config.txt
├── docs/
│ ├── ...
├── my_tool
│ ├── __init__.py
│ ├── my_tool.py
│ ├── a.py
│ ├── b.py
│ ├── c.py
│ ├── d.py
│ ├── e.py
│ └── tests
│ ├── __init__.py
│ ├── a.py
│ ├── b.py
│ ├── c.py
│ ├── d.py
│ └── e.py
├── package2
│ ├── __init__.py
│ ├── my_second_package.py
| ├── ...
├── README.md
└── resources
├── ...
However, I can't figure out an project organisation that satisfies the following criteria:
All of the scripts are invokable on the command line (either as my_tool\a.py or cd my_tool && a.py)
The tests actually run :)
Files in package2 can do import my_tool
The main problem is with the import statements used by the packages and the tests.
Currently, all of the packages, including the tests, simply do import <module> and it's resolved correctly. But when jiggering things around it doesn't work.
Note that supporting py2.7 is a requirement so all of the files have from __future__ import absolute_import, ... at the top.
What I've tried, and the disastrous results
1
If I move the files around as shown above, but leave all of the import statements as they currently are:
$ ./my_tool/*.py works and they all run properly
$ nosetests run from the top directory doesn't work. The tests fail to import the packages scripts.
pycharm highlights import statements in red when editing those files :(
2
If I then change the test scripts to do:
from my_tool import x
$ ./my_tool/*.py still works and they all run properly
$ nosetests run from the top directory doesn't work. Then tests can import the correct scripts, but the imports in the scripts themselves fail when the test scripts import them.
pycharm highlights import statements in red in the main scripts still :(
3
If I keep the same structure and change everything to be from my_tool import then:
$ ./my_tool/*.py results in ImportErrors
$ nosetests runs everything ok.
pycharm doesn't complain about anything
e.g. of 1.:
Traceback (most recent call last):
File "./my_tool/a.py", line 34, in <module>
from my_tool import b
ImportError: cannot import name b
4
I also tried from . import x but that just ends up with ValueError: Attempted relative import in non-package for the direct running of scripts.
Looking at some other SO answers:
I can't just use python -m pkg.tests.core_test as
a) I don't have main.py. I guess I could have one?
b) I want to be able to run all of the scripts, not just main?
I've tried:
if __name__ == '__main__' and __package__ is None:
from os import sys, path
sys.path.append(path.dirname(path.dirname(path.abspath(__file__))))
but it didn't help.
I also tried:
__package__ = "my_tool"
from . import b
But received:
SystemError: Parent module 'loading_tool' not loaded, cannot perform relative import
adding import my_tool before from . import b just ends up back with ImportError: cannot import name b
Fix?
What's the correct set of magical incantations and directory layout to make all of this work?
Once you move to your desired configuration, the absolute imports you are using to load the modules that are specific to my_tool no longer work.
You need three modifications after you create the my_tool subdirectory and move the files into it:
Create my_tool/__init__.py. (You seem to already do this but I wanted to mention it for completeness.)
In the files directly under in my_tool: change the import statements to load the modules from the current package. So in my_tool.py change:
import c
import d
import k
import s
to:
from . import c
from . import d
from . import k
from . import s
You need to make a similar change to all your other files. (You mention having tried setting __package__ and then doing a relative import but setting __package__ is not needed.)
In the files located in my_tool/tests: change the import statements that import the code you want to test to relative imports that load from one package up in the hierarchy. So in test_my_tool.py change:
import my_tool
to:
from .. import my_tool
Similarly for all the other test files.
With the modifications above, I can run modules directly:
$ python -m my_tool.my_tool
C!
D!
F!
V!
K!
T!
S!
my_tool!
my_tool main!
|main tool!||detected||tar edit!||installed||keys||LOL||ssl connect||parse ASN.1||config|
$ python -m my_tool.k
F!
V!
K!
K main!
|keys||LOL||ssl connect||parse ASN.1|
and I can run tests:
$ nosetests
........
----------------------------------------------------------------------
Ran 8 tests in 0.006s
OK
Note that I can run the above both with Python 2.7 and Python 3.
Rather than make the various modules under my_tool be directly executable, I suggest using a proper setup.py file to declare entry points and let setup.py create these entry points when the package is installed. Since you intend to distribute this code, you should use a setup.py to formally package it anyway.
Modify the modules that can be invoked from the command line so that, taking my_tool/my_tool.py as example, instead of this:
if __name__ == "__main__":
print("my_tool main!")
print(do_something())
You have:
def main():
print("my_tool main!")
print(do_something())
if __name__ == "__main__":
main()
Create a setup.py file that contains the proper entry_points. For instance:
from setuptools import setup, find_packages
setup(
name="my_tool",
version="0.1.0",
packages=find_packages(),
entry_points={
'console_scripts': [
'my_tool = my_tool.my_tool:main'
],
},
author="",
author_email="",
description="Does stuff.",
license="MIT",
keywords=[],
url="",
classifiers=[
],
)
The file above instructs setup.py to create a script named my_tool that will invoke the main method in the module my_tool.my_tool. On my system, once the package is installed, there is a script located at /usr/local/bin/my_tool that invokes the main method in my_tool.my_tool. It produces the same output as running python -m my_tool.my_tool, which I've shown above.
Point 1
I believe it's working, so I don't comment on it.
Point 2
I always used tests at the same level as my_tool, not below it, but they should work if you do this at the top of each tests files (before importing my_tool or any other py file in the same directory)
import os
import sys
sys.path.insert(0, os.path.abspath(__file__).rsplit(os.sep, 2)[0])
Point 3
In my_second_package.py do this at the top (before importing my_tool)
import os
import sys
sys.path.insert(0,
os.path.abspath(__file__).rsplit(os.sep, 2)[0] + os.sep
+ 'my_tool')
Best regards,
JM
To run it from both command line and act like library while allowing nosetest to operate in a standard manner, I believe you will have to do a double up approach on Imports.
For example, the Python files will require:
try:
import f
except ImportError:
import tools.f as f
I went through and made a PR off the github you linked with all test cases working.
https://github.com/Poddster/package_problems/pull/1
Edit: Forgot the imports in __init__.py to be properly usable in other packages, added. Now should be able to do:
import tools
tools.c.do_something()