I'm trying to build a CLI in python by using the package click. The Python version I'm using is 3.6
This is the main of my application:
import os
import click
cmd_folder = os.path.join(os.path.dirname(__file__), 'commands')
class IAMCLI(click.MultiCommand):
def list_commands(self, ctx):
rv = []
for filename in os.listdir(cmd_folder):
if filename.endswith('.py') and \
filename.startswith('cmd_'):
rv.append(filename[4:-3])
rv.sort()
return rv
def get_command(self, ctx, cmd_name):
ns = {}
fn = os.path.join(cmd_folder, 'cmd_{}.py'.format(cmd_name))
with open(fn) as f:
code = compile(f.read(), fn, 'exec')
eval(code, ns, ns)
return ns['cli']
#click.command(cls=IAMCLI)
#click.option('--env', default='dev', type=click.Choice(['dev', 'staging', 'production']),
help='AWS Environment')
#click.pass_context
def cli():
"""AWS IAM roles and policies management CLI."""
pass
if __name__ == '__main__':
cli()
and this is the tree:
├── cli
│ ├── __init__.py
│ ├── aws
│ │ ├── __init__.py
│ │ ├── policy.py
│ │ └── role.py
│ ├── cli.py
│ └── commands
│ ├── __init__.py
│ └── cmd_dump.py
the cmd_dump.py looks like this:
import click
from cli.aws.role import fetch_roles
#click.command('dump', short_help='Dump IAM resources')
#click.pass_context
def cli():
pass
the problem is that when I try to run python cli/cli.py --help this is what I get:
File "cli/commands/cmd_dump.py", line 3, in <module>
from cli.aws.role import fetch_roles
ModuleNotFoundError: No module named 'cli.aws'; 'cli' is not a package
Any idea about that?
I will try to give another answer based on my approach when starting development of a new python project. Do you plan to distribute your project, or maybe just share it with someone? If you do, what do you think - will this someone be happy with needing to remember the command
$ python path/to/project/codebase/cli/cli.py --help
to use your tool? Wouldn't it be easier for him to remember the command
$ cli --help
instead?
I suggest you to start with packaging of your project right away - write a minimal setup script:
from setuptools import setup, find_packages
setup(
name='mypkg',
version='0.1',
packages=find_packages(),
install_requires=['click'],
entry_points={
'console_scripts': ['cli=cli.cli:cli'],
},
)
You can always enhance your setup script when new requirements emerge. Place the setup script in the root directory of your codebase:
├── setup.py
├── cli
│ ├── __init__.py
│ ├── aws
...
Now run python setup.py develop or even better, pip install --editable=.1 from the codebase root directory (where the setup.pyscript is). You have installed your project in the development mode and now can invoke
$ cli --help
with all the imports being resolved correctly (which would resolve your issue). But you gain much more besides that - you gain a way to package your project to be ready to be distributed to your target users and a clean command line interface which your users will invoke the same way as you just did.
Now continue with the project development. If you change the code of the cli command, it will be applied on the fly so you don't need to reinstall the project each time you change anything.
Once you are ready with the project development and want to deliver it to your users, issue:
$ python setup.py bdist_wheel
This will package your project into a installable wheel file (you will need to install wheel package to be able to invoke the command: pip install wheel --user). Usually it will reside in the dist subdirectory of the codebase root dir. Give this file to the user. To install the file, he will issue
$ pip install Downloads/mypkg-0.1-py3-none.whl --user
and can start tp use your tool right away:
$ cli --help
This is a very much simplified description and there is a lot of stuff to learn, but there is also a ton of helpful materials to guide you through the process.
If you want to learn more about the topic: as a quickstart reference, I would recommend the excellent PyPA packaging guide. For packaging click commands, their own docs are more than sufficient.
I would encourage you to use pip for distribution and packaging development where applicable as it is a standard tool for that.
Do not run scripts inside packages! Packages are made to be imported in code but not to run scripts inside them. The rest is non related with import errors.
For example:
├── cli # package
│ ├── __init__.py
│ ├── aws
│ │ ├── __init__.py
│ │ ├── policy.py
│ │ └── role.py
│ ├── cli.py
│ │ └── commands
│ │ ├── __init__.py
│ │ └── cmd_dump.py
├── run_this_module.py
Module to execute run_this_module.py:
import cli
"""Do your code here"""
I have done this a hundred times. It is tempting to name a package the same thing at multiple levels, but resist! I tend to have __main__.py in my packages these days, which helps solve the problem that you are having.
I suspect you are having a namespace issue. Try renaming your package or the internal file called cli.py. Be sure to refactor any imports that are attempting to use these.
Also, change your function name to something besides cli. Same issue.
├── cli <- rename this to 'my_app' or something else
│ ├── __init__.py
│ ├── aws
│ │ ├── __init__.py
│ │ ├── policy.py
│ │ └── role.py
│ ├── cli.py <- or maybe it is easier to rename this
│ └── commands
│ ├── __init__.py
│ └── cmd_dump.py
Related
I am having some trouble adding packages with my particular setup:
.
├── pkg_a
│ ├── pkg_a
│ │ ├── __init__.py
│ │ └── module_a.py
│ └── run_a.py
├── pkg_b
│ ├── pkg_b
│ │ ├── __init__.py
│ │ └── module_b.py
│ └── run_b.py
└── setup.py
My goal is to be able to import package modules without repeating package name twice.
For example, in run_a.py I'd like to be able to call from pkg_a import module_a instead of calling from pkg_a.pkg_a import module_a
I tried to follow Section 2.1 of doc here. By creating setup.py as follow:
from setuptools import setup
setup(
name="test",
packages=['pkg_a', 'pkg_b'],
package_dir={'pkg_a':'pkg_a/pkg_a', 'pkg_b':'pkg_b/pkg_b'}
)
But this does not achieve the desired effect as mentioned above as I tried to call python setup.py develop and then python -c 'from pkg_a import module_a'.
Is this particular setup achievable? And what am I messing up here? Thanks all!
package_dir modifications do not work with editable (aka develop) installations. The only acceptable package_dir modification for editable installations is the one that covers the so-called src-layout:
package_dir={'': 'src'},
I have a project
testci/
├── __init__.py
├── README.md
├── requirements.txt
├── src
│ ├── __init__.py
│ └── mylib.py
└── test
├── __init__.py
└── pow_test.py
When I run python3.6 test/pow_test.py I see an error:
File "test/pow_test.py", line 3, in
import testci.src.mylib as mylib
ModuleNotFoundError: No module named 'testci'
pow_test.py
from testci.src.mylib import get_abs
def test_abs():
assert get_abs(-10) == 10
How can I fix this error?
System details: Ububntu 16.04 LTS, Python 3.6.10
try this
from .src import mylib
from mylib import get_abs
if it won't work then import one by one. But don't import the root folder since the file you are importing to is on the same folder you are trying to import then it will always raise an error
Run Python with the -m argument within the base testsci package to execute as a submodule.
I made a similar mock folder structure:
├───abc_blah
│ │ abc_blah.py
│ │ __init__.py
│
└───def
│ def.py
│ __init__.py
abc_blah.py
print('abc')
def.py
import abc_blah.abc_blah
Execute like such:
python -m def.def
Correctly prints out 'abc' as expected here.
simply add __package__ = "testci" and also it is a good practice to add a try and except block
Your final code should look something like
try:
from testci.src.mylib import get_abs
except ModuleNotFoundError:
from ..testci.src.mylib import get_abs
for running it, type python -m test.pow_test
I think your issue is how the package is installed. The import looks fine to me. As it says CI I'm guessing you're having the package installed remotely with only the test folder somehow.
Try adding a setup.py file where you define that both the test as well as the src packages are part of your testci package.
there are many ways to organize a project, keep things consider in mind, structure should be simple and more scaleable, can differentiate the things in codebase easily.
one of the few good possible ways to structure a project is below
project/
├── app.py
├── dockerfile
├── pipfile
├── Readme.md
├── requiements.txt
├── src_code
│ ├── code
│ │ ├── __init__.py
│ │ └── mylib.py
│ └── test
│ ├── __init__.py
│ └── test_func.py
└── travisfile
here app.py is main file which is responsible to run your entire project
I'm trying to build my first python package public available but I'm having some trouble with installing it on another machine, not sure what is wrong. My project is here.
After all the CI steps on the master branch, Travis publishes the latest version to the pypi. After that, we can install the package in any place:
pip install spin-clustering
But when I try to import it on my regular python it says that the module does not exist.
$ python -c "import spin"
Traceback (most recent call last):
File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'spin'
My package was originally called "spin" but the name was already taken on pypi, I changed it to "spin-clustering", but as scikit-learn is imported with "sklearn" I thought that would be possible to import my package as "spin". Not sure what I'm missing here.
This is my package structure:
├── LICENSE
├── Makefile
├── Pipfile
├── README.md
├── examples
│ ├── circle-example.ipynb
│ └── random-cluster-example.ipynb
├── setup.cfg
├── setup.py
└── spin
├── __init__.py
├── distances
│ ├── __init__.py
│ ├── distances.py
│ └── tests
│ └── __init__.py
├── neighborhood_spin.py
├── side_to_side_spin.py
├── tests
│ ├── __init__.py
│ ├── test_spin.py
│ └── test_utils.py
└── utils.py
And my setup.py
from setuptools import setup, find_packages
setup(name="spin-clustering",
maintainer="otaviocv",
maintainer_email="otaviocv.deluqui#gmail.com",
description="SPIN clustering method package.",
license="MIT",
version="0.0.3",
python_requires=">=3.6",
install_requires=[
'numpy>=1.16.4',
'matplotlib>=3.1.0'
]
)
In your setup.py, you also need to specify what packages will be installed. The simplest way is using the provided find_packages function, which will scan your folders and try to figure out what the packages are (in some slightly unusual cases, your project organization will make this not work right). Your code imports find_packages, but is not using it.
Since you have none listed, nothing is actually installed (except the requirements, if missing).
I have a below structure
in migrations/env.py file I am trying to import from database import *
but it shows no module name database
I tried from ..database imprt * and adding file in pythonpath also but no luck :(
Your directory structure looks a bit suspicious to me. The alembic.ini shouldn't normally be part of the package (and setuptools won't by default pick it up when packaging). I think this would better be placed into the project-root.
Something like this would be more standard:
├── alembic.ini
├── migrations
│ ├── env.py
│ ├── script.py.mako
│ └── versions
│ └── ...
├── package_name
│ └── database
│ ├── __init__.py
│ └── ...
│ └── models
│ └── __init__.py
│ └── ...
├── README.md
└── setup.py
└── ...
Now, this alone would not make database available from env.py. For this to work you have to somehow make your package discoverable. Usually this would be done by installing package_name into some virtualenv. In that environment you could then use from package_name.database import * in your env.py.
Migrations needs to know where to import from, they either belong to the same package:
A:
migrations
database
init.py
And then in migrations:
from A.database.whatever import whatever else
Or you install them as packages separatedly inside your virtualenv:
And then each of them is dependent on the other, but because they are installed they can be invoked:
database/setup.py
migrations/setup.py
Then both are installed and migrations/env.py can call the installed package database
I'm trying to set sub-packages for a python project. Please refer to the structure below. The main setup.py will call "setup.py" in each sub packages.
my_project
├── my_sub_package1
│ ├── foo2.py
│ ├── foo.py
│ └── setup.py
├── my_sub_package2
│ ├── bar2.py
│ ├── bar.py
│ └── setup.py
└── setup.py [main]
With this structure, in other projects, if the user only needs a sub_package, the user can choose to install "my_sub_package1" only, instead of installing the whole package (which can become bulky over time as number of packages increases).
Does anyone know if this is the correct way of doing it? Thanks!