I feel a bit dumb asking this since there are a lot of similar questions, but I've honestly searched around a lot and could not find a solution for this. Here goes:
I have a Python package (on TestPyPi, here is the source code, note that it uses and needs python3.8) with the following structure:
paillier/
setup.py
test/
paillier/
__init__.py
keygen.py
util/
__init__.py
math_shortcuts.py
My use case is: in keygen.py, I want to use util/math_shortcuts.py.
So, in keygen.py, I have the following import:
from paillier.util.math_shortcuts import generate_coprime, lcm, get_mu.
However, when I try to use my package (by doing from paillier.keygen import generate_keys), I am greeted with the Error ModuleNotFoundError: No module named 'paillier.util'
This ModuleNotFoundError is always present when I install it using pip from TestPyPi, but it doesn't happen when I build the package locally: when I run pip install -e . in the paillier/ directory (where setup.py lives), I can run from paillier.keygen import generate_keys, even when my working directory is somewhere else.
I've tried to do from .util.math_shortcuts ..., or from util.math_shortcuts ..., or from paillier.paillier.util.math_shortcuts ..., but all to no avail.
In short, when doing pip install --index-url <TestPyPi> rens-paillier my files can't seem to find the submodules.
However, when doing pip install -e . in the outer paillier/ dir, it seems to work.
I ran into the same problem. (my -e installation worked, normal installation didn't) My solution is, to actually name the subpackages in the setup.py.
packages=['paillier', 'paillier.util']
While this works, I am not sure why :D
I found that removing the packages parameter from the setup function call in the setup.py file resolved a similar issue I had with my module not resolving submodules.
looking around for the same question (credits to this post), the easiest is probably to use
setuptools.find_packages()
some optional arguments:
where="src" for package structures where the files are in src/
exclude=["*-old"] to exclude some packages (in my case, the my_package-old that I want to keep until my refactoring is done)
Related
I saw this nice explanation video (link) of packaging using pip and I got two questions:
The first one is:
I write a code which I want to share with my colleagues, but I do not aim to share it via pypi. Thus, I want to share it internally, so everyone can install it within his/ her environment.
I actually needn't to create a wheel file with python setup.py bdist_wheel, right? I create the setup.py file and I can install it with the command pip install -e . (for editable use), and everyone else can do it so as well, after cloning the repository. Is this right?
My second question is more technical:
I create the setup.py file:
from setuptools import setup
setup(
name = 'helloandbyemate',
version = '0.0.1',
description="Say hello in slang",
py_modules=['hellomate'],
package_dir={"": "src"}
)
To test it, I write a file hellomate.py which contains a function printing hello, mate!. I put this function in src/. In the setup.py file I put only this module in the list py_modules. In src/ is another module called byemate.py. When I install the whole module, it installs the module byemate.py as well, although I only put hellomate in the list of py_modules. Has anyone an explanation for this behaviour?
I actually needn't to create a wheel file ... everyone else can do it so as well, after cloning the repository. Is this right?
This is correct. However, the installation from source is slower, so you may want to publish wheels to an index anyway if you would like faster installs.
When I install the whole module, it installs the module byemate.py as well, although I only put hellomate in the list of py_modules. Has anyone an explanation for this behaviour?
Yes, this is an artifact of the "editable" installation mode. It works by putting the src directory onto the sys.path, via a line in the path configuration file .../lib/pythonX.Y/site-packages/easy-install.pth. This means that the entire source directory is exposed and everything in there is available to import, whether it is packaged up into a release file or not.
The benefit is that source code is "editable" without reinstallation (adding/removing/modifying files in src will be reflected in the package immediately)
The drawback is that the editable installation is not exactly the same as a "real" installation, where only the files specified in the setup.py will be copied into site-packages directly
If you don't want other files such as byemate.py to be available to import, use a regular install pip install . without the -e option. However, local changes to hellomate.py won't be reflected until the installation step is repeated.
Strict editable installs
It is possible to get a mode of installation where byemate.py is not exposed at all, but live modifications to hellomate.py are still possible. This is the "strict" editable mode of setuptools. However, it is not possible using setup.py, you have to use a modern build system declaration in pyproject.toml:
[build-system]
requires = ["setuptools"]
build-backend = "setuptools.build_meta"
[project]
name = "helloandbyemate"
version = "0.0.1"
description = "Say hello in slang"
[tool.setuptools]
py-modules = ["hellomate"]
include-package-data = false
[tool.setuptools.package-dir]
"" = "src"
Now you can perform a strict install with:
pip install -e . --config-settings editable_mode=strict
I'm trying to import https://github.com/chrisconlan/algorithmic-trading-with-python in my code. I've never imported anything from GitHub before and have looked at various other questions that have been asked on Stack Overflow regarding this problem but it just doesn't work. When I try to run the 'portfolio.py' code for example I keep getting a ModuleNotFound error for 'pypm'. What exactly is the correct way to import such a module or the whole GitHub directory?
I'm working with Visual Studio Code on Windows.
You will need to pip install the module. In your case the command you would need to run is python -m pip install -U git+https://github.com/chrisconlan/algorithmic-trading-with-python. Once you have done that you need to find the name of the module. You can do this with pip list. Find the name of the module you just installed.
Then you just stick import <module name> at the top of your code with the rest of your imports.
What i used to do in this is to clone the repository on the folder where are installed the python's packages. This is useful when you do not want to use the pip cmd tool, keeping the pip's cache memory under control.
Is there a way to modify the location of the .egg-info directory that is generated upon:
pip install --editable .
I'm asking because I store my source code on (locally synchronized) cloud storage, and I want to install the package in editable mode on independent computers. So, ideally, the package directory would not be polluted with anything related to a given installation of the package.
I have tried using the --src option but this did not work; I don't understand what this option is meant to do.
You can achieve this by adding the egg_base option to setup.cfg:
[egg_info]
egg_base = relative/path/to/egg_info_folder
I have used this successfully in pip 19.3.1.
In my environment, the actual files that this altered are:
/anaconda/envs/my_env/lib/python3.6/site-packages/easy-install.pth
/anaconda/envs/my_env/lib/python3.6/site-packages/package_name.egg-link
Note, pip install raises an error if the egg_base base is not a relative path. But directly altering the files appears to work:
/anaconda/envs/my_env/lib/python3.6/site-packages/easy-install.pth:
/path/to/repository/folder
/anaconda/envs/my_env/lib/python3.6/site-packages/package_name.egg-link:
/path/to/egg_info/folder
/path/to/repository/folder/
Not sure if still relevant, but here is a setup.py based solution: https://jbhannah.net/articles/python-docker-disappearing-egg-info
I am installing the Django==1.9.1 project of the former team (I am newcomer) to my server it runs on Centos7.
The project structure is:
agros>
address.json
agros>
agros.ini
agros.log
apps>
insurance>
mixins.py
user>
models.py
I did as I usually do: installed all the required things and run:
pip install -r requrements.txt
Then I run the following code:
python manage.py makemigrations
The problematic package is imported in this way
from apps.insurance.mixins import .......
But I am receiving the following error
ImportError: No module name insurance.mixins
insurance.mixins is my library. I checked and it is where it should be. What can I do to fix this!
pip install usually installs from PyPi. insurance looks like a custom module. This wouldn't be in PyPi. So find where insurance module is and copy it to the proper location. If you don't know where to copy the module run ,
import sys
sys.path
and check site-packages is located. And then move your insurance module over there. You can also move your insurance package to any of the paths returned by sys.path as it checks all of them. You can also add new paths to this list.
Hopefully that should fix the problem.
I forked the nice module multiscorer and I am trying to turn it into a package that I could install in different environments.
My fork can be found here. The steps I took are
Create a new environment (using conda) and activate it
python setup.py install from the root directory of the fork
In a new terminal, activate the environment and move to some arbitrary location. Start ipython and try from multiscorer import MultiScorer
I get the following error ImportError: cannot import name 'MultiScorer'. Note however, that import multiscorer works just fine. What do I have to change in the code to enable an installation using python setup.py install?
Another attempt: I tried to replace packages=['multiscorer'] with py_modules=['multiscorer.multiscorer']. Didn't help...
Your setup.py is alright. The problem is the package structure. Right now the correct way to import the Multiscorer class is this: from multiscorer.multiscorer import Multiscorer. The first multiscorer is for the folder (package) of the same name and the second multiscorer is for the multiscorer.py module inside the package.
The docs recommend putting all your code inside the __init__.py for such small packages.
If your codebase later grows too large for one file, you can start introducing other modules and use the __init__.py for exposing classes/functions on the package level.
Hope this helps.
It turns out I tried to import the wrong thing. The following: from multiscorer.multiscorer import MultiScorer works.
I am now wondering is this the pythonic way.