I am not very familiar with python, I only done automation with so I am a new with packages and everything.
I am creating an API with Flask, Gunicorn and Poetry.
I noticed that there is a version number inside the pyproject.toml and I would like to create a route /version which returns the version of my app.
My app structure look like this atm:
├── README.md
├── __init__.py
├── poetry.lock
├── pyproject.toml
├── tests
│ └── __init__.py
└── wsgi.py
Where wsgi.py is my main file which run the app.
I saw peoples using importlib but I didn't find how to make it work as it is used with:
__version__ = importlib.metadata.version("__package__")
But I have no clue what this package mean.
You should not use __package__, which is the name of the "import package" (or maybe import module, depending on where this line of code is located), and this is not what importlib.metadata.version() expects. This function expects the name of the distribution package (the thing that you pip-install), which is the one you write in pyproject.toml as name = "???".
You can extract version from pyproject.toml using toml package to read toml file and then display it in a webpage.
Related
I have a question in how to properly create a path in Python (Python 3.x).
I developed a small scraping app in Python with the following directory structure.
root
├── Dockerfile
├── README.md
├── tox.ini
├── src
│ └── myapp
│ ├── __init__.py
│ ├── do_something.py
│ └── do_something_else.py
└── tests
├── __init__.py
├── test_do_something.py
└── test_do_something_else.py
When I want to run my code, I can go to the src directory and do with
python do_something.py
But, because do_something.py has an import statement from do_something_else.py, it fails like:
Traceback (most recent call last):
File "src/myapp/do_something.py", line 1, in <module>
from src.myapp.do_something_else import do_it
ModuleNotFoundError: No module named 'src'
So, I eventually decided to use the following command to specify the python path:
PYTHONPATH=../../ python do_something.py
to make sure that the path is seen.
But, what are the better ways to feed the path so that my app can run?
I want to know this because when I run pytest via tox, the directory that I would run the command tox would be at the root so that tox.ini is seen by tox package. If I do that, then I most likely run into a similar problem due to the Python path not properly set.
Questions I want to ask specifically are:
where should I run my main code when creating my own project like this? root as like python src/myapp/do_something.py? Or, go to the src/myapp directory and run like python do_something.py?
once, the directory where I should execute my program is determined, what is the correct way to import modules from other py file? Is it ok to use from src.myapp.do_something_else import do_it (this means I must add path from src directory)? Or, different way to import?
What are ways I can have my Python recognize the path? I am aware there are several ways to make the pass accessible as below:
a. write export PYTHONPATH=<path_of_my_choice>:$PYTHONPATH to make the
path accessible temporarily, or write that line in my .bashrc to make it permanent (but it's hard to reproduce when I want to automate creating Python environment via ansible or other automation tools)
b. write import sys; sys.path.append(<root>) to have the root as an accessible path
c. use pytest-pythonpath package (but this is not really a generic answer)
Thank you so much for your inputs!
my environment
OS: MacOS and Amazon Linux 2
Python Version: 3.7
Dependency in Python: pytest, tox
I would suggest to use setup.py to make this a python package. Then you can install it in development mode python setup.py develop. This way it will be available in your python environment w/o needing to specify the PYTHONPATH.
For testing, you can simply install the package python setup.py install.
Hope that helps.
Two simple steps should make it happen. Python experts can comment if this is a good way to do it (especially going by the concluding caution raised towards the end of this post).
I would have done it like below.
First I would have put a "__init__.py" in root so that hierarchy looks like below. This way python will treat the folder as a package.
root
├── Dockerfile
├── README.md
├── tox.ini
├── __init__.py
├── src
│ └── myapp
│ ├── __init__.py
│ ├── do_something.py
│ └── do_something_else.py
└── tests
├── __init__.py
├── test_do_something.py
└── test_do_something_else.py
Then in "do_something.py", I would have added these lines at the top. In the second line please put the full path to the "root" directory.
import sys
sys.path += ['/home/SomeUserName/SomeFolderPath/root']
from src.myapp.do_something_else import do_it
Please note that the second line will essentially modify the sys.path by adding the root folder path (I guess until the interpreter quits). If this is not what you can afford then I am sorry.
I am using poetry to create .whl files.
I have an ftp sever runing on a remote host.
I wrote a python script (log_revision.py) which save in a database the git commit, few more parameters and in the end send the the .whl(that poetry created) to the remote server ( each .whl in a different path in the server, the path is save in the db) .
At the moment I run the script manually after each time I run the poetry build commend.
I know the pyproject.toml has the [tool.poetry.scripts] but i dont get how can i use it to run a python script.
I tried
[tool.poetry.scripts]
my-script = "my_package_name:log_revision.py
and then poetry run my-script but I allways get an error
AttributeError: module 'my_package_namen' has no attribute 'log_revision'
1. can some one please help me understand how to run to wish commend?
as a short term option(with out git and params) i tried to use the poetry publish -r http://192.168.1.xxx/home/whl -u hello -p world but i get the following error
[RuntimeError]
Repository http://192.168.1.xxx/home/whl is not defined
2. what am i doing wring and how can i fix it?
would appricate any help, thx!
At the moment the [tool.poetry.scripts] sections is equivalent to setuptools console_scripts.
So the argument must be a valid module and method name. Let's imagine within your package my_package, you have log_revision.py, which has a method start(). Then you have to write:
[tool.poetry.scripts]
my-script = "my_package.log_revision:start"
Here's a complete example:
You should have this folder structure:
my_package
├── my_package
│ ├── __init__.py
│ └── log_revision.py
└── pyproject.toml
The content of pyproject.toml is:
[tool.poetry]
name = "my_package"
version = "0.1.0"
description = ""
authors = ["Your Name <you#example.com>"]
[tool.poetry.dependencies]
python = "^3.8"
[tool.poetry.scripts]
my-script = "my_package.log_revision:start"
[build-system]
requires = ["poetry_core>=1.0.0"]
build-backend = "poetry.core.masonry.api"
and of log_revision.py:
def start():
print("Hello")
After you have run poetry install once you should be able to do this:
$ poetry run my-script
Hello
You cannot pass something to the start() method directly. Instead you can use command line arguments and parse them, e.g. with pythons argparse.
Although the previous answers are correct, they are a bit complicated. The simplest way to run a python script with poetry is as follows:
poetry run python myscript.py
If you are using a dev framework like streamlit you can use
poetry run streamlit run myapp.py
Basically anything you put after poetry run will execute from the poetry virtual environment.
For future visitors, I think what OP is asking for (a post build hook?) isn't directly supported. But you might find satisfaction from using a tool I wrote called poethepoet which integrates with poetry to run arbitrary tasks defined in the pyproject.toml in terms of shell commands or by referencing python functions.
For example you could define something like the following in your pyproject.toml
[tool.poe.tasks.log_revision]
script = "my_package.log_revision:main" # where main is the name of the python function in the log_revision module
help = "Register this revision in the catalog db"
[tool.poe.tasks.build]
cmd = "poetry build"
help = "Build the project"
[tool.poe.tasks.shipit]
sequence = ["build", "log_revision"]
help = "Build and deploy"
And then execute and of the tasks with the poe CLI like the following which will run the other two tasks in sequence, thus building your project and running the deployment script in one go!
poe shipit
By default tasks are executed inside the poetry managed virtualenv (like using poetry run) so the python script can use dev dependencies.
If you need to define CLI arguments or load values into a task from a dotenv file (such as credentials) then this is also supported.
Update: poetry plugin support
poethepoet can now support post build hooks when used as a poetry plugin. For example when using poetry >=1.2.0b1 you can configure the following to run your log_revision task automatically after poetry build is run:
[tool.poe.poetry_hooks]
post_build = "log-revision"
[tool.poe.tasks.log-revision]
script = "scripts:log_revision"
Tinkering with such a problem for a couple of hours and found a solution
I had a task to start the django server via poetry script.
Here are the directories. manage.py is in test folder:
├── pyproject.toml
├── README.rst
├── runserver.py
├── test
│ ├── db.sqlite3
│ ├── manage.py
│ └── test
│ ├── asgi.py
│ ├── __init__.py
│ ├── __pycache__
│ │ ├── __init__.cpython-39.pyc
│ │ ├── settings.cpython-39.pyc
│ │ ├── urls.cpython-39.pyc
│ │ └── wsgi.cpython-39.pyc
│ ├── settings.py
│ ├── urls.py
│ └── wsgi.py
├── tests
│ ├── __init__.py
│ └── test_tmp.py
└── tmp
└── __init__.py
I had to create a file runserver.py:
import subprocess
def djtest():
cmd =['python', 'test/manage.py', 'runserver']
subprocess.run(cmd)
then write the script itself pyproject.toml:
[tool.poetry.scripts]
dj = "runserver:djtest"
and still make changes to pyproject.toml:
[tool.poetry.scripts]
dj = "runserver:djtest"
only then command poetry run dj started working
In the last few days, I was working on a python module. Until now, I used poetry as a packages management tool in many other projects, but it is my first time wanting to publish a package to PyPI.
I was able to run the poetry build and poetry publish commands. I was also able to also install the published package:
$ pip3 install git-profiles
Collecting git-profiles
Using cached https://files.pythonhosted.org/packages/0e/e7/bac9027effd1e34a5b5718f2b35c0b28b3d67f3809e2f2981b6c7b58963e/git_profiles-1.1.0-py3-none-any.whl
Installing collected packages: git-profiles
Successfully installed git-profiles-1.1.0
However, right after the install, I am not able to run my package:
$ git-profiles --help
git-profiles: command not found
My project has the following structure:
git-profiles/
├── src/
│ ├── commands/
│ ├── executor/
│ ├── git_manager/
│ ├── profile/
│ ├── utils/
│ ├── __init__.py
│ └── git_profiles.py
└── tests
I tried to work with different scripts configurations in the pyproject.toml file but I've never been able to make it work after install.
[tool.poetry.scripts]
poetry = "src:git_profiles.py"
or
[tool.poetry.scripts]
git-profile = "src:git_profiles.py"
I don't know if this is a python/pip path/version problem or I need to change something in the configuration file.
If it is helpful, this is the GitHub repository I'm talking about. The package is also published on PyPI.
Poetry's scripts sections wraps around the console script definition of setuptools. As such, the entrypoint name and the call path you give it need to follow the exact same rules.
In short, a console script does more or less this from the shell:
import my_lib # the module isn't called src, that's just a folder name
# the right name to import is whatever you put at [tool.poetry].name
my_lib.my_module.function()
Which, if given the name my-lib-call (the name can be the same as your module, but it doesn't need to be) would be written like this:
[tool.poetry.scripts]
my-lib-call = "my_lib.my_module:function"
Adapted to your project structure, the following should do the job:
[tool.poetry.scripts]
git-profile = "git-profiles:main"
I am having problems with importing modules/packages with Python. I noticed this problem when I ran it in my terminal (CMD),
rather than my IDE (I use PyCharm). In PyCharm, I use a virtual enviroment setting with Python 3.7 and everything works as a charm
and as intended.
For reference this is how the imports were done in test_suite.py:
...
from tests.scenarios.test_scenario_01 import TestScenario # They work perfectly fine
from tests.scenarios.test_scenario_02 import TestScenario2 # written like this in PyCharm venv Python 3.7, but why?
...
This is a simplified version of my directory (without the unneccesary files):
QA System/
├── locators/
│ ├── locators.py
│ ├── __init__.py
├── pages/
│ ├── pages.py
│ └── __init__.py
└── tests/
├── reports
├── test_scenarios
├── test_scenario_01.py
├── test_scenario_02.py
├── __init__.py
|── test_suite.py
|── __init__.py
However when running the file test_suite.py manually through my CMD (because I want to integrate it with Jenkins
eventually), I get this error (py -3 test_suite.py):
ModuleNotFoundError: No module named 'tests'
Note: I am using the newest Python 3.7
From what I know about Python imports, for a directory to be treated like a python module, there needs to be a '__init__.py' file
included in the same directory.
After a bit of research I found out that it is possible to do a different type of imports in Python 3 and tried it out (putting a .
before the name of the imports). Like this:
from .scenarios.test_scenario_01 import TestScenario
from .scenarios.test_scenario_02 import TestScenario2
But still, it didn't run successfully and this was the error I've gotten:
ModuleNotFoundError: No module named '__main__.scenarios'; '__main__' is not a package
Could you please help me out on this one?
TLDR: Imports work in a Python3.7 venv, but not outside it
I fixed the issue by setting up a PYTHONPATH in system environment variables with the path to the project. As a value, I put the dir to the project. Thanks for the tips.
In case of directory It is mandate to use init.py. In modules it does not require to include init.py. Please, check what is your main directory and which is your modules..Hope this helps....
I'm writing a couple of packages that I'd like to release on PyPi for other people to use.
I've not released to PyPi before so I have been mocking up a submission template: https://github.com/chris-brown-nz/pypi-package-template
Here's a tree of the project template:
| MANIFEST.in
| README.rst
| setup.cfg
| setup.py
|
\---package
module_one.py
module_three.py
module_two.py
__init__.py
In terms of interacting with the package, this is what I would usually do - is it the best way?
To run a method:
from package import module_one
module_one.ClassOne().method_a()
To get a value from a method:
from package import module_two
print(module_two.ClassFive().method_e())
To set then use an attribute of an instance:
from package import module_three
cls = module_three.ClassSeven("Hello World")
print(cls.value)
'package' is a reserved name obviously and won't be used in the final project.
I'd be grateful for some feedback on how I've structured my project and whether it is considered standard, or if it should be modified in some way.
There are different approaches to this, whether one or the other is better is depending on a how you want to develop, usage of the package (e.g. if you ever install it using pip install -e packag_name), etc.
What is missing from your tree is the name of the directory where the setup.py resides, and that is usually the package name:
└── package
├── package
│ ├── __init__.py
│ ├── module_one.py
│ ├── module_three.py
│ └── module_two.py
├── MANIFEST.in
├── README.rst
├── setup.cfg
└── setup.py
as you can see you are doubling the 'package' name, and that means that your setup.py has to be adapted for each package, or dynamically determine the name of the directory where the module.py files resides. If you go for this route, I would suggest you put the module.py files in a generically named directory 'src' or 'lib'.
I don't like the above "standard" setup for multiple reasons:
it doesn't map well to how python programs "grow" before they are split up into packages. Before splitting up having such a 'src' directory would mean using:
from package.src.module_one import MyModuleOneClass
Instead you would have your module.py files directly under package
Having a setup.py to control installation, a README.rst for documentation and an __init__.py to satisfy Python's import is one thing, but all other stuff, apart from your module.py files containing the actual functionality, is garbage. Garbage that might be needed at some point during the package creation process, but is not necessary for the package functionality.
There are other considerations, such as being able to access the version number of the package from the setup.py as well as from the program, without the former having to import the package itself (which may lead to install complications), nor having another extra version.py file that needs importing.
In particular I always found the transition from using a directory structure under site-packages that looked like:
└── organisation
├── package1
└── package2
├── subpack1
└── subpack2
and that could intuitively be used for both importing and navigation to source files, to something like:
├── organisation_package1
│ └── src
├── organisation_package2_subpack1
│ └── src
└── organisation_package2_subpack2
└── src
unnatural. To rearrange and break a working structure to be able to package things seems wrong.
For my set of published packages I followed another way:
- I kept the natural tree structure that you can use "before packaging", 'src' or 'lib' directories.
- I have a generic setup.py which reads and parses (it does not import) the metadata (such as version number, package name, license information, whether to install a utility (and its name)), from a dictionary in the __init__.py file. A file you need anyway.
- The setup.py is smart enough to distinguish subdirectories containing other packages from subdirectories that are part of the parent package.
- setup.py generates files that are needed during package generation only (like setup.cfg), on the fly, and deletes them when no longer needed.
The above allows you to have nested namespaced packages (i.e. package2 can be a package you upload to PyPI, in addition to package2.subpack1 and package2.subpack2). The major thing it (currently) doesn't allow is using pip install -e to edit a single package (and not have the others editable). Given the way I develop, that is not a restriction.
The above embraces namespace packages, where many other approaches have problems with these (remember the last line of Zen of Python: Namespaces are one honking great idea – let’s do more of those)
Examples of the above can e.g be found in my packages ruamel.yaml (and e.g. ruamel.yaml.cmd), or generically by searching PyPI for ruamel.
As is probably obvious, the standard disclaimer: I am the author of those packages
As I use a utility to start packaging, which also runs the tests and does other sanity checks, the generic setup.py could be removed from the setup and inserted by that utility as well. But since subpackage detection is based upon setup.py availability or not, this requires some rework of the generic setup.py.