This question already has answers here:
How to read a (static) file from inside a Python package?
(6 answers)
Closed 2 years ago.
I am writing a python module and need to install a directory along with the module. This is what my file tree looks like:
├── module
│ └── temp
│ | └── __init.py__
| | file2.yaml
| | file.yaml
│ └── module.py
| __init__.py
├── setup.py
As you can see, module.py is the main module with all my functions, but it needs to access what is in the temp directory. When I use setup tools to install my module locally, using pip, it installs the module.py perfectly, but it won't install the temp directory.
Here is my setup.py:
setup(name='module',
packages=find_packages(),
version=VERSION,
description=DESCRIPTION,
long_description=LONG_DESCRIPTION,
license=LICENSE,
author=AUTHORS,
install_requires=[
],
include_package_data=False
)
My theory is that I need to pass something through find packages().
Side note:
I am running setup.py as python setup.py bdist_wheel. To clarify, everything is working fine, my directory is just not being installed with the package.
When I go to where the package is stored, __init__.py and module.py are the only things that are installed in my module package directory.
How can I make sure that the temp directory is installed as well?
Any help is appreciated.
EDIT:
My first file tree was wrong. The temp directory is inside the directory with the module.
When python install packages, the module directory will be copied to the directory like /usr/local/lib/python3.6/dist-packages/module. If you add the extra tmp directory, it may conflict with other module.
How about considering moving the tmp directory to the module directory. like this:
├── module
│ └── temp
│ | └── __init.py__
| | file2.yaml
| | file.yaml
│ └── module.py
| __init__.py
├── setup.py
Related
I have a project for developing a Python package where the structure of the project is similar to the following:
myproject/
├── README.md
├── examples/
│ ├── ex1.py
│ └── ex2.py
├── pyproject.toml
├── src/
│ └── mypackage/
│ ├── __init__.py
│ ├── adder.py
│ └── divider.py
└── tests/
├── test_adder.py
├── test_divider.py
└── test_examples.py
The project is for developing a Python package named mypackage which is located in the src directory. The package is uploaded to PyPI where users can pip install it. Tests for the package are run with pytest and are located in the tests directory. Examples of using the package are in the examples directory. The examples are just scripts as shown below for ex1.py
"""
Example 1
"""
from mypackage import adder
x = 2.5
y = 8
a = adder(x, y)
print('a is', a)
The purpose of test_examples.py is to test the example files, its contents are shown below:
from examples import ex1
from examples import ex2
def test_ex1():
ex1
def test_ex2():
ex2
When I run pytest in the myproject directory I get the error shown here:
$ cd myproject
$ pytest
platform darwin -- Python 3.10.6, pytest-7.1.2, pluggy-1.0.0
rootdir: /Users/gavinw/Desktop/test-examples
collected 2 items / 1 error
================================================================== ERRORS ==================================================================
_________________________________________________ ERROR collecting tests/test_examples.py __________________________________________________
ImportError while importing test module '/Users/gavinw/Desktop/test-examples/tests/test_examples.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/opt/miniconda3/envs/ztest/lib/python3.10/importlib/__init__.py:126: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
tests/test_examples.py:1: in <module>
from examples import ex1
E ModuleNotFoundError: No module named 'examples'
========================================================= short test summary info ==========================================================
ERROR tests/test_examples.py
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
============================================================= 1 error in 0.05s =============================================================
It looks like pytest is not able to run the files in the examples directory because the location does not allow them to be imported. Any suggestions on how I can test the example files? Should I even use pytest for testing examples or is there a different testing tool for this?
You should ensure that the examples directory contains __init__.py so it can be imported correctly.
If that is not enough, you can use PYTHONPATH:
PYTHONPATH="/path/to/your/code/project/:/path/to/your/code/project/examples" pytest
PYTHONPATH might be tricky, see https://stackoverflow.com/a/4580120/3800552 and https://stackoverflow.com/a/39682723/3800552 for some usage examples.
There are two approaches that fix the problem. The first approach is to run pytest using the following command:
python -m pytest
The second approach is to add the project directory in the pyproject.toml file using the pythonpath setting as shown below. Then just use the pytest command to run the tests.
[tool.pytest.ini_options]
pythonpath = ["."]
I have the following folder structure:
PROJECT_DIR
| --helpers
| |--utils.py
| --stuff
| |--script.py
I need to run script.py as a script, and from it, I need to use a function from helpers/utils.py.
I tried relative importing from ..helpers.utils import func, but it says
ImportError: attempted relative import with no known parent package
so I added an empty init.py file to each folder, including PROJECT_DIR.
Then I read that while running as a script, the python compiler runs the script as if it was the main module, so it doesn't see any other modules outside so relative import cannot be used.
But what should I do if I need to use that function? It's a fairly simple use case, I can't get my head around why it's so hard to import a function from a file outside the current directory. Tho I'm not really interested in the whys, I'd just like to know a solution how people do this.
root_project
└── proj
├── __init__.py
├── helpers
│ ├── __init__.py
│ └── utils.py
└── stuff
├── __init__.py
└── script.py
With this structure just cd to root_project and use this command:
python -m proj.stuff.script
I have a python 3 package with the following structure:
.
├── package
│ └── bin
└── main_module
│ └── lib
│ ├── __init__.py
│ ├── module1.py
│ ├── module2.py
│ └── module3.py
│ └── test
│ ├── test1.py
│ ├── test2.py
│ └── test3.py
│ └── setup.py
Usually, one runs $ python3 setup.py install and all is good. However, I want to use this package on a cluster server, where I don't have write permissions for /usr/lib/. The following solutions came to my mind.
Somehow install the package locally in my user folder.
Modify the package such that it runs without installation.
Ask the IT guys to install the package for me.
I want to avoid 3., so my question is whether 1. is possible and if not, how I have to modify the code (particularly the imports) in order to be able to use the package without installation. I have been reading about relative imports in python all morning and I am now even more confused than before. I added __init__.py to package and bin and from what I read I assumed it has to be from package.lib import module1, but I always get ImportError: No module named lib.
In order for Python to be able to find your modules you need to add the path of your package to sys.path list. As a general way you can use following snippet:
from sys import path as syspath
from os import path as ospath
syspath.append(ospath.join(ospath.expanduser("~"), 'package_path_from_home'))
os.path.expanduser("~") will give you the path of the home directory and you can join it with the path of your package using os.path.join and then append the final path to sys.path.
If package is in home directory you can just add the following at the leading of your python code that is supposed to use this package:
syspath.append(ospath.join(ospath.expanduser("~"), 'package'))
Also make sure that you have an __init__.py in all your modules.
I had the same problem. I used the first approach
install the package locally in my user folder by running
python setup.py install --user
This will install your module in ~/.local/lib/python3/
Just add the path of your 'package' to environment variable PYTHONPATH. This will get rid of the error you are getting.
OR
programmatically add path of the package to sys.path.append()
you can add this to the "main file" of the package
import sys, os
sys.path.append(os.path.dirname(__file__) + "/..")
you can find the "main file" by looking for this pattern
if __name__ == "__main__":
some_function()
File structure:
.
|-- rdir
| |-- __init__.py
| |-- core
| | |-- __init__.py
| | |-- rdir_core.py
| | |-- rdir_node.py
| |-- generateHTML
| | |-- __init__.py
| |-- rdir.py
|-- setup.py
Setup.py:
from setuptools import setup, find_packages
setup(
name="rdir",
version="0.40",
description="....",
author="lhfcws",
author_email="lhfcws#gmail.com",
url="...",
license="MIT",
packages=["rdir"],
scripts=["rdir/rdir.py"],
install_requires=['colorama', 'pyquery'],
)
Commands:
sudo python setup.py install #local install
sudo python setup.py sdist upload #pypi upload
Try from rdir import rdir in other path like home directory, only meets with:
ImportError: No module named core.rdir_core
Of course it works well if I import rdir in the project directory.
And I looked into site-packages/rdir.egg-info/, I found that all the .py files were moved into a flat structure:
EGG-INFO
├── PKG-INFO
├── SOURCES.txt
├── dependency_links.txt
├── not-zip-safe
├── requires.txt
├── scripts
│ ├── __init__.py
│ ├── generate_page.py
│ ├── rdir.py
│ ├── rdir_core.py
│ └── rdir_node.py
└── top_level.txt
I also tried if I just import rdir_core in rdir.py, it compiles correctly. So I guessed there's sth wrong with my setup.py, and I read some demos, some setup.py of famous python projects on github and some official manuals. I changed my setup.py according to those reference, but all failed. I have no idea so I have to ask for help.
Is it something wrong with my setup.py? Or is there anything I've missed?
Or please show me a good example of a setup.py of multi file structure projects. Thank you!
BTW, if these above cannot offer you enough information, please look at rdir on Github
It's the problem of the packages keyword in your setup.py. You should list sub packages as well as the top level package.
packages=['rdir', 'rdir.core', 'rdir.generateHTML'],
Or, using find_packages, which you already imported
packages=find_packages(),
I didn't try the sdist stuff, maybe it's just collecting all py files as scripts.
P.S. You can use python setup.py build to test the result folder structure.
Inspired by Ray, I've solved by myself.
It is the problem of scripts in the setup.py.
The project was a flat structure, so it works well before 0.30. However in my computer, the scripts options will install the script to /usr/local/bin and generate a flat structure in EGG.
If I dont remove the old scripts in /usr/local/bin, python interpreter will check those scripts under /usr/local/bin first and cause error.
So the solution is:
remove scripts in setup.py
set packages=find_package()
sudo rm /usr/local/bin/rdir*
Thanks to you all.
This question already has an answer here:
Is "from ... import ..." sometimes required and plain "import ..." not always working? Why?
(1 answer)
Closed 2 years ago.
I have the following structure:
project
├── sum_package
│ └── sum_module.py
├── testing.py
I try to access sum_module.py from testing.py in two different ways:
First:
from sum_package import sum_module
sum_module.great_summation(1,2)
Second:
import sum_package
sum_package.sum_module.great_summation(1,2)
First works, second doesn't:
AttributeError: module 'sum_package' has no attribute 'sum_module'
Can someone explain why?
EDIT: tried adding __init__.py into either/both the package folder / the project folder, does not help.
EDIT 2: so it seems like the problem is $PYTHONPATH. I've tried adding my src (project) directory as PYTHONPATH, like this:
export PYTHONPATH = "$PWD"
while in src, but I still get the same error. What else am I missing?:)
project
├── __init__.py
|
├── sum_package
├── __init__.py
|
│ └── sum_module.py
├── testing.py
The __init__.py files are required to make python treat directories containing the file as packages. Make your folder structure like this. In the simplest case, __init__.py can just be an empty file.
Then try
testing.py
from sum_package import sum_module
sum_module.great_summation(1,2)