I'm trying to integrate TravisCI into my workflow, and realized I had some dependencies because of my old directory structure (not having self-contained, virtualenv-able git repos).
When I try to run nosetests locally, it runs the tests just fine; when TravisCI tries to run them, it fails, with an import error. Specifically, I have, as one of the lines in my test script:
from myproject import something
My directory structure is inside my git repo myproject is something like:
.travis.yml
requirements.txt
something.py
tests/
test_something.py
I have tried getting this to fail locally (because then I'd understand the TravisCI issue, maybe), but cannot accomplish it.
I've tried running with regular python, and using a virtualenv which added nose to its requirements.txt, and the tests always pass locally.
I feel like I still haven't understood absolute-vs-relative imports, and I can't tell if that's coming in to play here, or if I'm just doing something obvious and dumb in my project.
Desired outcome: figure out why TravisCI is failing, and fix my repo accordingly, so that I can commit and have things build correctly, both locally and on TravisCI. If that requires more drastic changes like "you should have a setup.py that does blah-blah to the environment" or similar - please let me know. I'm new to this aspect of Python, and find the current documentation overwhelmingly unclear.
As an FYI, I found this question and adding --exe doesn't help, or seem to be the same issue.
I see there are no answer and I encountered the same issue, so I am posting here in hope to help somebody:
Solution 1
The quick fix for me was to add this line export PYTHONPATH=$PYTHONPATH:$(pwd) in the .travis.yml:
before_install:
- "pip install -U pip"
- "export PYTHONPATH=$PYTHONPATH:$(pwd)"
Solution 2
Having a setup.py which should be the default option as it is the most elegant, configured like:
from setuptools import setup, find_packages
setup(name='MyPythonProject',
version='0.0.1',
description='What it does',
author='',
author_email='',
url='',
packages=find_packages(),
)
And then add this line in .travis.yml
before_install:
- "pip install -U pip"
- "python setup.py install"
Solution 3:
Changing the layout of the project to have the test folder under the application one (the one with your core python code) such as:
.travis.yml
requirements.txt
app
|_ tests
| |_ test_application.py
|_ application.py
And running the test in travis with coverage and nosetest like:
script:
- "nosetests --with-coverage --cover-package app"
Related
I'm trying to let users write code as a python module (folder with __init__.py defined) under whatever folder name they see fit. After that I want to install that module as a python package but define the import name myself.
The folder structure would be like this:
project_name/
user_defined_name/
__init__.py
...
setup.py
According to this I should be able to add this to my setup.py to get it working:
setuptools.setup(
package_dir={'my_defined_name': 'user_defined_name'},
packages=['user_defined_name']
)
But the only way that I was able to access the code was by using import user_defined_name. I tried installing the package without -e but that gave the same result. Leaving the packages=['..'] out of the setup functions also did not change the result.
My question is kind of the same as this one and there the only answers seem to be to change folder names and that is something that I would like to avoid. That question mentioned that it might be due to a problem in setuptools but that seemed fixed 3 years ago.
In short, it looks like you need something like that in your setup.py:
setuptools.setup(
package_dir={
'my_defined_name': 'user_defined_name',
},
packages=[
'my_defined_name',
],
)
as Ilia Novoselov said in a comment to your question.
This should work, if you package and install the project normally. You would be able to import my_defined_name.
Although, note that as far as I can tell, this will not work if you use an editable installation (python setup.py develop or python -m pip install --editable .). It will be impossible to import my_defined_name, but you would be able to import user_defined_name, which is not what you want.
#Oliver's answer here clarified this for me.
My TLDR is that to support both releases (python setup.py install and pip install .) and editable installs (python setup.py develop and pip install -e .) you must change your file structure to
project_name
setup.py
user_defined_name
my_defined_name
__init__.py
...
docs
tests
and your setup.py to
setuptools.setup(
package_dir={'': 'user_defined_name'},
packages=['my_defined_name']
)
You can support just releases (NOT editable installs) with
project_name
setup.py
user_defined_name
__init__.py
...
docs
tests
setuptools.setup(
package_dir={'my_defined_name': 'user_defined_name'},
packages=['my_defined_name']
)
I've got a repo on GitHub, for which I wanted to implement Travis CI with pytest for basic testing. Currently the Travis CI build fails when loading data tables within a module, raising a FileNotFoundError.
To make it short, here is the imho most important information on the build:
directory of the data tables is included in MANIFEST.in with include mypkg/data_tables/* (see below for a detailed structure)
setuptools.setup method has the include_package_data=True parameter
additionally packages=setuptools.find_packages() is provided
Travis CI installs the package with install: pip install -e .
Travis CI pytest is invoked with script: pytest --import-mode=importlib
during testing the first tests succeed. But when it comes to loading the data tables, pytest raises the error FileNotFoundError: [Errno 2] No such file or directory: '/home/travis/build/myname/mypkg/mypkg/data_tables\\my_data.csv'
Interestingly the slashes before the file name are back-slashes, while the other are not, even though the final path is constructed with os.path.abspath().
Detailed description
Unluckily the repo is private and I'm not allowed to share it. Thus I'll try to describe the GitHub package layout as detailed as possible. So let's say my repo is built with a structure like this (general layout taken from this example):
setup.py
MANIFEST.in
mypkg/
some_data_tables/
my_data.csv
my_other_data.pkl
__init__.py
view.py
tests/
test_view.py
My minimum MANIFEST.in looks like this:
include mypkg/data_tables/*
With the setup.py fully reduced to a minimum working example like this:
from setuptools import find_packages, setup
setup(
name='Mypkg',
version='123.456',
description='some_text',
python_requires='>=3.7.7',
packages=find_packages( # <---- this should be sufficient, right?
exclude=["tests", "*.tests", "*.tests.*", "tests.*"]),
include_package_data=True, # <---- also this should work
)
And the .travis.yml file (omitting - pip install -r requirements.txt etc.):
language: python
python:
- "3.7.7"
dist: xenial
install:
- pip install -e .
script:
- pytest --import-mode=importlib
Checking the content of the .egg or tar.gz files, the data tables are included. So I have no idea, where the files are "getting lost".
Any idea how to solve this error?
If providing more information could help, f.i. on the class initialized in test_view, please tell me.
I have never really fully understood how packages are handled in Python and I'm having a problem with that right now. But googling doesn't seem to help as I find the topic really confusing.
I have a project with this structure:
project_name/
src/
main.py
utils/
string_utils.py
tests/
test_string_utils.py
I am using Pytest for running unit testing and currently inside the "test_string_utils.py" file I have the following:
from ..src.utils.string_utils import StringUtilsClass
But I go to the folder "project_name" and try to run tests with any of this command I get errors:
$ pytest tests/
ValueError: attempted relative import beyond top-level package
I know about the -m argument for python, but it seems that running "pytest -m" has a completely different behavior.
How can I solve this? Am I using the wrong folder architecture? I don't think what I'm building should be a pip package (which would simplify imports)
did you try : from src.utils.string_utils import StringUtilsClass without .. before src?
or from string_utils import StringUtilsClass
Travis CI showed a weird behaviour after I tried to integrate with coverage. Before trying to use coverage the build was okay with all the tests. Now suddenly it doesn't locate the file
here is the .travis.yml file
#language to use for app
language: python
-- "3.6"
script:
- virtualEnv/run_travis.sh
# whitelist
branches:
only:
- master
- flask_dev_branch
#dependacies and libraries to install
install: pip install -r virtualEnv/requirements.txt
after_success:
- coveralls
And the file run_travis.sh
#!/usr/bin/env bash
python tests/test_shopping_cart.py > /dev/null &
nosetests --with-coverage
Also an image of the directory with the files included
All this started happening after trying to configure coverage.
Your tests subdirectory is inside virtualEnv and the current directory is the parent of virtualEnv. Run
python virtualEnv/tests/test_shopping_cart.py
PS. And please don't show screenshots — copy text.
I guess the requirements.txt is not in the virtualEnv directory.
I have some Python package and some tests. The files are layed out following http://pytest.org/latest/goodpractices.html#choosing-a-test-layout-import-rules
Putting tests into an extra directory outside your actual application
code, useful if you have many functional tests or for other reasons
want to keep tests separate from actual application code (often a good
idea):
setup.py # your distutils/setuptools Python package metadata
mypkg/
__init__.py
appmodule.py
tests/
test_app.py
My problem is, when I run the tests py.test, I get an error
ImportError: No module named 'mypkg'
I can solve this by installing the package python setup.py install but this means the tests run against the installed package, not the local one, which makes development very tedious. Whenever I make a change and want to run the tests, I need to reinstall, else I am testing the old code.
What can I do?
I know this question has been already closed, but a simple way I often use is to call pytest via python -m, from the root (the parent of the package).
$ python -m pytest tests
This works because -m option adds the current directory to the python path, and hence mypkg is detected as a local package (not as the installed).
See:
https://docs.pytest.org/en/latest/usage.html#calling-pytest-through-python-m-pytest
The normal approach for development is to use a virtualenv and use pip install -e . in the virtualenv (this is almost equivalent to python setup.py develop). Now your source directory is used as installed package on sys.path.
There are of course a bunch of other ways to get your package on sys.path for testing, see Ensuring py.test includes the application directory in sys.path for a question with a more complete answer for this exact same problem.
On my side, while developing, I prefer to run tests from the IDE (using a runner extension) rather than using the command line. However, before pushing my code or prior to a release, I like to use the command line.
Here is a way to deal with this issue, allowing you to run tests from both the test runner used by your IDE and the command line.
My setup:
IDE: Visual Studio Code
Testing: pytest
Extension (test runner): https://marketplace.visualstudio.com/items?itemName=LittleFoxTeam.vscode-python-test-adapter
Work directory structure (my solution should be easily adaptable to your context):
project_folder/
src/
mypkg/
__init__.py
appmodule.py
tests/
mypkg/
appmodule_test.py
pytest.ini <- Use so pytest can locate pkgs from ./src
.env <- Use so VsCode and its extention can locate pkgs from ./src
.env:
PYTHONPATH="${PYTHONPATH};./src;"
pytest.ini (tried with pytest 7.1.2):
[pytest]
pythonpath = . src
./src/mypkg/appmodule.py:
def i_hate_configuring_python():
return "Finally..."
./tests/mypkg/appmodule_test.py:
from mypkg import app_module
def test_demo():
print(app_module.i_hate_configuring_python())
This should do the trick
Import the package using from .. import mypkg. For this to work you will need to add (empty) __init__.py files to the tests directory and the containing directory. py.test should take care of the rest.