We Keep Coding

Pytest Django Importing apps issues

I have a question regarding pytest and django 3.1. I have an app structure like that:
main_app
├── config
│   └── settings
│   └── base.py / local.py / prod.py
├── main_app
│   ├── sub_apps
│   │   └── models.py tests.py apps.py views.py etc...
│   ├── templates
│   │   ├── account
│   │   ├── sub_apps
│   │   ├── pages
│   │   ├── userpreferences
│   │   └── users
│   ├── userpreferences
│   │   └── models.py tests.py apps.py views.py etc...
│   ├── users
│   │   └── tests --> test_models.py test_views.py test_forms.py etc...
│   └── utils
└── requirements
Following the structure of pydanny/cookiecutter.
With this cookiecutter, the preferred test method is via pytest. But pytest is giving me a bunch of headaches:
In my settings, INSTALLED_APPS I register sub_apps within the
LOCAL_APPS =["sub_app_1", "sub_app_2", ...]
When I start django, everything is fine. But if I want to run tests, pytest is complaining heavily, it can't import the sub_app modules.
I suspect mixed up relative and absolute import paths but am not seeing how to solve this atm.
As a background: I use a separate userpreferences model which is imported in the settings via
LOCAL_APPS = ["main_app.userpreferences.apps.UserpreferencesConfig"]
In the apps.py I have to define the app name as
name = "main_app.userpreferences"
Otherwise I get an Runtime error stating the Model class does'nt declare an explicit app_label / is not installed in INSTALLED_APPS.
While django runs just fine with these imports, pytest exits before running tests with:
ModuleNotFoundError: No module named "sub_app_1"
I tried to change every import of modules in the sub_apps to e.g.:
from .sub_app_1 import models --> from main_app.sub_app_1 import models
but then django won't run anymore while the tests seem to run (pytest is starting and telling me 5/7 tests passed, while also occasionally printing import errors)
How do I resolve this?

Add dir to Django search path for tests

I am using a single django for various custom distributions.
I arrived at a solution where the apps unique to each distribution live outside of the django projects folder (lets call it DIR/), and added this outside DIR/ to the Django path using
How to keep all my django applications in specific folder (adding to the sys.path solution)
The project runs, however now it does not discover tests. Before moving the apps out of the Django project folder I could simply run all tests with:
manage.py test
However, now the tests aren't found. In addition to adding DIR/ to settings.py, I tried adding it to manage.py and it did not help. By reading the django docs I discovered I can specify a module like this:
manage.py test app_name.tests
The above works, but is impractical for many apps. How can I add a path for where to search for tests?
I read this but it only describes the problem, not the solution:
Django test runner not finding tests
Request for my project structure:
somefolder/
|-- dist/
| |-- dist1/apps/
| | |---- app11/
| | '---- app12/
| |
| '-- dist2/apps/
| |---- app21/
| '---- app22/
|-- src/
|-- manage.py
|-- project/settings.py
|-- appA/
'-- appB/
Since asking have found out this works (not ideal):
manage.py test app11 app12 app21 app22 --keepdb

The key is to add some empty __init__py files in your dist, dist/dist1 and dist/dist1/apps directories.
I just tried putting together a toy project with the following structure:
.
├── dist
│   ├── dist1
│   │   ├── apps
│   │   │   ├── app_external
│   │   │   │   ├── admin.py
│   │   │   │   ├── apps.py
│   │   │   │   ├── __init__.py
│   │   │   │   ├── migrations
│   │   │   │   │   └── __init__.py
│   │   │   │   ├── models.py
│   │   │   │   ├── tests.py
│   │   │   │   └── views.py
│   │   │   └── __init__.py
│   │   └── __init__.py
│   └── __init__.py
└── src
├── app_internal
│   ├── admin.py
│   ├── apps.py
│   ├── __init__.py
│   ├── migrations
│   │   └── __init__.py
│   ├── models.py
│   ├── tests.py
│   └── views.py
├── manage.py
└── project
├── __init__.py
├── settings.py
├── urls.py
└── wsgi.py
Notice that there is an empty __init__.py file in each directory under dist/. This is what makes each directory a Python package and make the Python test machinery to look for tests inside those directories.
Also notice that I have an internal app called app_internal and an external one called app_external in a directory structure similar to yours.
Both the app_internal and app_external have one fake test in the tests.py file.
app_external/tests.py content:
from django.test import TestCase
class ExternalTestCase(TestCase):
def test_fake(self):
self.assertTrue(False)
app_internal/tests.py content:
from django.test import TestCase
class InternalTestCase(TestCase):
def test_fake(self):
self.assertTrue(True)
Here we expect that the app_external test will fail and the app_internal test will succeed.
I can call the app_internal test by giving the usual command:
$ ./manage.py test
Creating test database for alias 'default'...
System check identified no issues (0 silenced).
.
----------------------------------------------------------------------
Ran 1 test in 0.000s
OK
Destroying test database for alias 'default'...
And I can call the app_external test by giving the following command:
$ ./manage.py test ..
Creating test database for alias 'default'...
System check identified no issues (0 silenced).
F
======================================================================
FAIL: test_fake (dist.dist1.apps.app_external.tests.ExternalTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/augusto/develop/apps_outside/dist/dist1/apps/app_external/tests.py", line 10, in test_fake
self.assertTrue(False)
AssertionError: False is not true
----------------------------------------------------------------------
Ran 1 test in 0.000s
FAILED (failures=1)
Destroying test database for alias 'default'...
Notice how I passed .. as the parameter to the test command.
This point to the parent directory with respect to src/, so it finds the dist package and every other package inside it. I expect that this will find all tests under your dist/ directory.
I can also run all internal and external tests at once by giving the following command:
$ ./manage.py test . ..
Creating test database for alias 'default'...
System check identified no issues (0 silenced).
.F
======================================================================
FAIL: test_fake (dist.dist1.apps.app_external.tests.ExternalTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/augusto/develop/apps_outside/dist/dist1/apps/app_external/tests.py", line 10, in test_fake
self.assertTrue(False)
AssertionError: False is not true
----------------------------------------------------------------------
Ran 2 tests in 0.001s
FAILED (failures=1)
Destroying test database for alias 'default'...
I tested this using Django 1.11.23 and Python 2.7.12.

Django cannot setup when not using manager

I am having a strange issue, hopefully someone has encountered it before.
I have a custom script located at my_project/my_app/scripts/custom_script.py -- My desired use case is to feed this script a filepath and have it process the file and load/modify some data within my database.
Here is a snippet from that script:
import django
def setup_django(env):
if env == 'dev':
settings = "my_project.my_project.dev_settings"
elif env == 'stg':
settings = "my_project.my_project.staging_settings"
elif env == 'prod':
settings = "my_project.my_project.prod_settings"
else:
raise ValueError("Invalid choice for --env argument: {}".format(env))
os.environ.setdefault('DJANGO_SETTINGS_MODULE', settings)
django.setup()
def main():
env = 'dev'
setup_django(env)
# Do stuff with my script
When I run the above from within the same virtualenv as my project, I get the error ModuleNotFoundError: No module named 'my_app'
Inside the settings file:
...
INSTALLED_APPS = [
'my_app',
'django.contrib.admin',
'django.contrib.auth',
...
]
...
It seems that, when I call django.setup(), it goes and parses my my dev_settings file and finds my_app in the INSTALLED_APPS list, then tries to import it directly (e.g. import my_app). This will never work, because my_app is a sub-module of my_project (should be import my_project.my_app). Here is a sample of the directory structure:
my_project
├── my_project
│   ├── __init__.py
│   ├── dev_settings.py
│   ├── prod_settings.py
│   ├── staging_settings.py
│   ├── urls.py
│   └── wsgi.py
├── my_app
│   ├── __init__.py
│   ├── admin.py
│   ├── apps.py
│   ├── models.py
│   ├── scripts
│   │   ├── __init__.py
│   │   ├── **custom_script.py**
│   │   └── utils.py
│   ├── tests.py
│   ├── urls.py
│   └── views.py
├── __init__.py
└── manage.py
This settings file seems to work fine with manage.py, both in runserver and shell (can find the my_app application just fine). However inside the custom_script.py all seems lost.
Any ideas?

Use management commands to add custom scripts to your app so that all django configs get loaded up correctly.
In your my_app folder, create folder structure below
management/
__init__.py
commands/
__init__.py
my_command.py
In my_command.py
from django.core.management import BaseCommand
class Command(BaseCommand):
def handle(self, *args, **options):
print('running custom command')
on your shell
python manage.py my_command

Run Django tests with nosetests

My python apps testing is performed on the remote server with command nosetests. I cannot modify the way tests are started nor can I add options to it. I have Django app with tests, but tests are not working properly.
My project structure:
project
├── README.md
├── setup.py
├── mysite
│   ├── blog
│   │   ├── __init__.py
│   │   ├── models.py
│   │   ├── tests.py
| | ├── ...
│   ├── db.sqlite3
│   ├── manage.py
│   ├── mysite
│   │   ├── __init__.py
│   │   ├── settings.py
| | ├── ...
Command nosetests is executed in project directory. I want it to properly run tests.py which has 2 Django testcases. I tried creating tests directory in project root and invoke tests with DiscoverRunner):
os.environ['DJANGO_SETTINGS_MODULE'] = 'mysite.settings'
test_dir = os.path.dirname(os.path.dirname(__file__)) # one level up
sys.path.insert(0, os.path.join(test_dir, 'mysite'))
class ServerTest(unittest.TestCase):
def test_runtests(self):
django.setup()
self.test_runner = DiscoverRunner(verbosity=1, interactive=True, failfast=False)
failures = self.test_runner.run_tests(['mysite'])
self.assertEqual(failures, 0)
It works but the problem is all the tests are considered as a single test and wrong reports are produced by the server.
Another solution: if I add empty __init__.py to project/mysite nose discovers tests.py but the tests fail because 'Apps are not loaded yet' which probably means I have to invoke django.setup() earlier but I don't know how to do it. I found a plugin for the nose which does it but I cannot install plugins or alter options on the remote machine.
Any ideas how to make any of my approaches solve the problem?

Firsts things first, you should install and configure django-nose if you haven't done so already:
pip install django-nose
Then add it on your INSTALLED_APPS:
INSTALLED_APPS = (
...
'django_nose'
)
TEST_RUNNER = 'django_nose.NoseTestSuiteRunner'
Now for the nosetests command to run correctly, you can create a folder named tests with the following structure:
project
├── mysite
│ ├── blog
│ │ ├── __init__.py
│ │ ├── models.py
| | ├── ...
│ ├── mysite
│ │ ├── __init__.py
│ │ ├── settings.py
| | ├── ...
│ ├── tests
│ │ ├── __init__.py
│ │ ├── test_a_whole_class_of_methods.py
│ │ ├── test_a_whole_view.py
│ │ ├── test_one_of_my_methods.py
│ │ ├── test_another_one_of_my_methods.py
| | ├── ...
│ ├── db.sqlite3
│ ├── manage.py
├── README.md
├── setup.py
DON'T FORGET THE __init__.py FILE INSIDE THE tests FOLDER!!!
Now when you run nosetests from the root of your project, it will run every test in the tests folder:
~ $ cd path/to/project
path/to/project $ nosetests

Since you can add files to the root of your project, you could try adding a setup.cfg file there that nose will look for when executing, and in it specify where to look for tests:
# setup.cfg
[nosetests]
where=mysite/blog/
(See the documentation for what parameters you can put here).
I'm not sure that this will work (it is possible that the command that starts nose has already specified a different configuration file to use), but it seems worth a shot.

Importing app when using Alembic raises ImportError

I am trying to study how to use alembic in flask, I want to import a method in flask app:
tree .
.
├── README.md
├── alembic
│   ├── README
│   ├── env.py
│   ├── env.pyc
│   ├── script.py.mako
│   └── versions
│   ├── 8f167daabe6_create_account_table.py
│   └── 8f167daabe6_create_account_table.pyc
├── alembic.ini
├── app
│   ├── __init__.py
│   ├── main
│   │   ├── __init__.py
│   │   ├── errors.py
│   │   ├── forms.py
│   │   └── views.py
│   ├── models.py
│   └── templates
│   ├── 404.html
│   ├── 500.html
│   ├── base.html
│   ├── index.html
│   └── user.html
├── config.py
├── data.sqlite
├── manage.py
└── requirements.txt
in app/__init__.py:
def create_app(config_name):
app = Flask(__name__)
I want to import create_app in env.py:
from app import create_app
but the error shows as below when I run the command alembic upgrade head:
File "alembic/env.py", line 5, in <module>
from app import create_app
ImportError: No module named app
Any idea for this?

I guess you are trying to run
python env.py
In this case, your app directory is not in PYTHONPATH.
solution 1
Run the app from parent dir:
python alembic/env.py
solution 2
Set the PYTHONPATH before running the app
PYTHONPATH=/path/to/parent/dir python env.py
edit
I read about alembic. As #mrorno said, just set the PYTHONPATH before running alembic:
PYTHONPATH=. alembic upgrade head

alembic just tries to load your env.py source code. It's not in your package, so it can't access your app module.
Use solution 2 #tomasz-jakub-rup suggested, you can execute like
$ PYTHONPATH=. alembic upgrade head
and you should get your result.

Create file .env and insert PYTHONPATH=.