I'm quite new to working with more complex modules, but I think is about time to implement them in my work flow. Also, I don't come from a software engineering background so some of my languages might be inaccurate, please bear with me. My module folder structure is like this:
+-- module_name
| +-- ml.py
| +-- exp_an.py
| +-- plotting.py
+-- etl
| +-- machine_learning.py
| +-- data_manipulation.py
The reason I have two folders is because the scripts directly on module_name are our personal library for use on most projects and the etl has code that is specific to this project.
In the beginning I had both folders on the same directory but I was having trouble importing from module_name to etl.
The thing is, in machine_learning.py I want to call a function from ml.py. I tried doing:
import sys
sys.append('../')
import module_name as mn
But this seems to bring some recursivity issues, because when I try to call mn.ml I get a mn has no attribute called ml error.
So, my question is, what is the right way to do this? Let's say I want to call a function called transform() that is inside ml.py in my machine_learning.py script. Is there a way to do this? Is there a better way to do this? Thanks
In order for your directories to be interpreted as modules, you need to add __init__.py in each directory. Your directory structure will look like this.
+-- module_name
| +-- __init__.py
| +-- ml.py
| +-- exp_an.py
| +-- plotting.py
+-- etl
| +-- __init__.py
| +-- machine_learning.py
| +-- data_manipulation.py
Then you'd use relative imports to get the modules. Example ->
# Inside machine_learning.py you are importing ml.py
import ..ml as ml
ml.transform()
Here is an example of a larger project. You can see how relative imports are used and the directories have their __init__.py.
https://github.com/TileThePlane/docusign-python-client
It sounds like you want to add the parent directory of module_name and etl (whatever that directory is) to your PYTHONPATH.
After doing that, you should be able to do all the imports you asked for.
To explain the mn has no attribute called ml error, that happened because you did not import ml. You just tried to use it as an attribute of mn, and you can't do that.
Related
A have several DAGs of similar structure and I wanted to use advice described in Airflow docs as Modules Management:
This is an example structure that you might have in your dags folder:
<DIRECTORY ON PYTHONPATH>
| .airflowignore -- only needed in ``dags`` folder, see below
| -- my_company
| __init__.py
| common_package
| | __init__.py
| | common_module.py
| | subpackage
| | __init__.py
| | subpackaged_util_module.py
|
| my_custom_dags
| __init__.py
| my_dag1.py
| my_dag2.py
| base_dag.py
In the case above, these are the ways you could import the python
files:
from my_company.common_package.common_module import SomeClass
from my_company.common_package.subpackage.subpackaged_util_module import AnotherClass
from my_company.my_custom_dags.base_dag import BaseDag
That works fine in Airflow.
However I used to validate my DAGs locally by running (also as advised by a piece of documentation - DAG Loader Test):
python my_company/my_custom_dags/my_dag1.py
When using the imports, it complains:
Traceback (most recent call last):
File "/[...]/my_company/my_custom_dags/my_dag1.py", line 1, in <module>
from my_company.common_package.common_module import SomeClass
ModuleNotFoundError: No module named 'my_company'
How should I run it so that it understands the context and reckognizes the package?
It works when run this way:
PYTHONPATH=. python my_company/my_custom_dags/my_dag1.py
It seems that when entry point is my_dag1.py that's inside my_company/my_custom_dags Python considers it as its "working directory" and only looks for modules within that path. When I add . to PYTHONPATH it can also look at entire directory structure and reckognize module my_company and below.
(I'm not expert in Python, so above explanation might be somewhat innacurrate, but this is my understanding. I'm also not sure if that's indeed the cleanest way to make it work)
I am working on a program and access a set of custom modules. I organized the modules in a subfolder as own package with an __init__.py. Moreover, on the main level of the directory I have created a virtual environment that holds my dependencies. The folder structure is as follows:
project
+-- main_program.py
+-- venv
| +-- cool_package.py
+---mypackage
| +-- module1.py
| +-- module2.py
| +-- __init__.py
The issue is, that module2.py depends on a package I installed in venv. Running module2.py from main_program.py gives an error "cool_package.py" not found.
How do I organize stuff like that so that I can accesses cool_package.py from main_program.py with all the other needed packaged. And make cool_package.py accessible for the custom package with module2.py as well?
I may have misunderstood what you mean by your virtual env but based on your folder and file layout I think you need to add an __init.py__ file to your venv folder to make it a package and then you should be able to import venv.cool_package.
thanks for all answers - it eventually worked by properly activating the environment before running the script. Something must have gone wrong the first time - all works now and the folder structure is correct.
Best
Moritz
I use sphinx-click extension for my click application, and i want to use sphinx-apidoc to generate .rst files for only submodules.
I've been working with pythons Sphinx documentation library for about a year now, and I have a usecase... I cannot quite figure out.. maybe I'm just blind right now.
Anyways, I have a cli tool structured something like
my_tool/
+-- __init__.py
+-- cli_entry.py
+-- utils
| +-- __init__.py
| +-- foo.py
| +-- bar.py
Where cli_entry is a click application, and it imports my_tool.utils.foo and my_tools.utils.bar
Since this is using the Click library. I've decided to use the sphinx_click extension to document any command that is in cli_entry.py (which documents all the commands great).
But heres the issue, I want to use sphinx-apidoc to generate the .rst files for everything that is in the ./my_tool/utils/ modules.
When I use the command as sphinx-apidoc -o ../docs/utils my_tool/utils the output files that I get include
docs/
+-- utils
| +-- module.rst
| +-- utils.rst
Which looks great at first, but upon opening utils.rst the file looks something like
utils package
=============
Submodules
----------
utils.foo module
----------------------
.. automodule:: utils.foo
:members:
:undoc-members:
:show-inheritance:
utils.bar module
----------------------
.. automodule:: utils.bar
:members:
:undoc-members:
:show-inheritance:
Then when I build the documentation using make html (from sphinx generated makefile) I get an error saying Failed to import 'utils.foo': no module named utils.foo thats because the import should exist as my_tool.utils.foo
How can I use sphinx-apidoc to generate only submodules and include the correct import paths?? Maybe its something I'm missing in the conf.py.. maybe it's an option I'm missing from sphinx-apidoc?
EDIT: I should mention that I could use the exclude_pattern argument... but I would prefer to not have to specify each cli file in my root. ex. In the case that I have cli_entry.py, cli_commandgroup1.py... cli_commandgroupN.py. I want this solution to be dynamic enough to just support only submodules.
EDIT: I tried using sphinx-apidoc -o ../docs/utils my_tool/ and this creates the following output
docs/
+-- utils
| +-- module.rst
| +-- my_tool.cli_entry.rst
| +-- my_tool.utils.rst
Now in the my_tool.utils.rst file, the imports are correct, and the docs can be generated.
my_tool.utils.foo module
----------------------
.. automodule:: my_tool.utils.foo
:members:
:undoc-members:
:show-inheritance:
The issue with specifying my_tool/ is that my_tool.cli_entry.rst gets created, when this .rst file would already be created using the click-sphinx extension.
I have been working on a python project and I am new to it.
I have made a small library for my project in which I have several different modules doing different tasks.
For example: I have 5 modules namely add, subtract, multiply, divide and root.
I call all these .pyc files into my main.py file and my code runs properly if all of them are in the same folder.
Now, I want to store my main.py at: D:\project\main.py
and these 5 .pyc files at : D:\project\Lib\ (In the Lib folder)
I found a solution as to mention the path of the folder Lib into the code but I can not do so as I need to submit the code somewhere and if they try to run this on their PC, it might not import these files.
What would be the possible solution to this?
Try creating a package.
Use a directory structure like this:
.
+-- main.py
+-- lib
+-- __init__.py
+-- add.pyc
+-- substract.pyc
+-- ...
Then, in your main.py file, you can import them like this:
from lib import add
More on packages on Python docs
Inside D:\project\Lib create an __init__.py file. and put all your modules in D:\project\Lib now lib works as a python package.you dir structure should now look like this:
D:\project\Lib
|
+--- __init__.py
+--- add.py
+--- sub.py
+--- multiply.py
Now from any file inside (say for ex main.py) D:\project call any module you want like this.
from Lib.add import something.
final dir structure will roughly look like this.
D:\project
|
+-- main.py
+-- Lib
|
+--- __init__.py
+--- add.py
+--- sub.py
+--- multiply.py
I have the following setup of a library in python:
library
| - __init__.py
| - lib1.py
| - ...
| - tools
| - __init__.py
| - testlib1.py
so in other words, the directory library contain python modules and the directory tools, which is a subdirectory of library contains e.g. one file testlib1.pyto test the library lib1.py.
testlib1.py therefore need to import lib1.py from the directory above to do some tests etc., just by calling python testlib1.py from somewhere on the computer, assuming the file is in the search path.
In addition, I only want ONE PYTHONPATH to be specified.
But we all know the following idea for testlib1.py does not work because the relative import does not work:
from .. import lib1
...
do something with lib1
I accept two kind of answers:
An answer which describes how still to be possible to call testlib1.py directly as the executing python script.
An answer that explains a better conceptual setup of of the modules etc, with the premise that everything has to be in the directory project and the tools has to be in a different directory than the actual libraries.
If something is not clear I suggest to ask and I will update the question.
Try adding a __init__.py to the tools directory. The relative import should work.
You can't. If you plan to use relative imports then you can't run the module by itself.
Either drop the relative imports or drop the idea of running testlib1.py directly.
Also I believe a test file should never use relative imports. Test should check whether the library works, and thus the code should be as similar as possible to the code that the users would actually use. And usually users do not add files to the library to use relative imports but use absolute imports.
By the way, I think your file organization is too much "java-like": mixing source code and tests. If you want to do tests then why not have something like:
project/
|
+-- src/
| |
| +--library/
| | |
| | +- lib1.py
| | |
| | #...
| +--library2/ #etc.
|
+-- tests/
|
+--testlibrary/
| |
| +- testlib1.py
#etc
To run the tests just use a tool like nosetests which automatically looks for this kind of folder structure and provide tons of options to change the search/test settings.
Actually, I found a solution, which does
works with the current implementation
does not require any changes in PYTHONPATH
without the need to hardcode the top-lebel directory
In testlib1.py the following code will do (has been tested):
import os
import sys
dirparts = os.path.dirname(os.path.abspath(__file__)).split('/')
sys.path.append('/'.join(dirparts[:-1]))
import mylib1
Not exactly sure this is a very clean or straightforward solution, but it allows to import any module from the directory one level up, which is what I need (as the test code or extra code or whetever is always located one level below the actual module files).