I have a flask app with essentially the following structure:
app/
__init__.py
myapp.py
common/
tool1.py
tool2.py
web/
__init__.py
views.py
api/
api_impl.py
worker/
__init__.py
worker.py
tasks.py
I initialize in myapp.py an important object I use in several places and I can access it from common/tool1.py and web/api/api_impl.py with from myapp import object. I've been able to use tool1 and tool2 in multiple places in web/ and myapp.py importing with from common.tool1 import tool1_def.
Other relevant facts are in myapp.py there is an import web statement for the blueprints and app/__init__.py and worker/__init__.py are empty. web/__init__.py contains the blueprint definitions for the routes.
I can run the app with gunicorn with no issues, but when I try to run my worker with python app/worker/worker.py I get the error ModuleNotFoundError: No module named 'myapp'. The worker.py is trying to import the same object defined in myapp.py.
I just don't understand why I can run the app and it works but when I try to run the worker it doesn't! I'm definitely not fully understanding the import system in this case and everything I've read online doesn't seem to fully clarify this.
your working imports imply that the project root is the app folder. As such you need to lunch your worker from this folder (or add it to PYTHONPATH environment variable)
python worker/worker.py
Or
python -m worker.worker
In addition your __init__.py in the app folder should be removed as app is not a package but a project root.
One method we use regularly for development and debugging (but also works in production) is the following. You can add this to each (offending?) module, or all modules if you choose.
import os
import sys
# Add your project root to sys.path.
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.realpath(__file__))))
from common import tool1
from common import tool2
from worker import worker
etc ...
Here you are adding your project root to sys.path. When performing imports, Python starts by looking through sys.path for the module you're trying to import. So here, you are adding your project root as the first item in sys.path.
Another advantage is these imports are explicit to your project. For example, if you have a local workers module which you want to import, but also have a workers package in site-packages, this will look in your local project first.
Related
apologies if I'm going about this wrong, but I am quite new to python and can't quite figure out what the problem is! I have a simple Flask app that I'm trying to write tests using pytest for. The file structure is like this:
│── app.py
|── tests
│ ├── tests_app.py
I originally had app.py and tests_app.py in the same level and it all worked fine with tests passing, but now I have put tests in their own folder I can no longer import app without error.
I have tried the following in tests_app.py:
from app import app #this is what worked fine when app and tests were in the same folder
from ..app import app
from .. import app
and the linting error is 'Attempted relative import beyond top-level package' and when I run pytest it says 'ImportError: attempted relative import with no known parent package'.
Many thanks in advance!
p.s. I am using Python 3.8.5
You need to have app on the module search path. A quick fix would be to add the path to app.py to the PYTHONPATH environment variable:
export PYTHONPATH=/path/to/dir # where app.py is
If app.py is on the python path, then import app will work (barring any other import problems such as circular imports).
An effective and reproducible way to do this is to install your application. You can do this in develop mode, which means the code runs from its current directory. This is an understated, important part of developing a Python application, but to do it you will need to write code to package your application.
I am struggling to successfully import my project to the test-suite in my project, as well as being able to run the program from the command-line. I've been able to run my test-suite for some time, under the impression that if the tests work, so does the command-line stuff--evidently this isn't the case. I do not yet intend on using my program as a library. The api.py acts is the entry-point for the program.
I have a project with the following structure (the same directory hierarchy as requests):
myapp/
myapp/
__init__.py
api.py # depends on commands.py
commands.py # depends on utils.py
utils.py
tests/
context.py
test_api.py # depends on api.py
test_commands.py # depends on commands.py, utils.py
In the file context.py I have a path modification adding myapp to the PYTHONPATH, so I can successfully run the tests on my code. Here is the contents of that file
import os
import sys
sys.path.insert(0, os.path.abspath('..'))
import myapp
I've tried imaginable import combination I can think of. Far too many to list! I have also perused the Python reference import system page, and this tutorial.
How should I import my dependencies?
Turns out this was the correct layout, I mistook the error for something else. Although for future reference, relative imports in Python 3 must be explicit: when in the myapp package directory you can't say import commands, you must instead import it as from . import commands. This was defined in PEP 328 also see this SO post on the topic. Run your package with python -m mutil.api not python ./mutil/api.py as the latter won't give the interpreter context of the current path.
Small context: I'm writing a project in Python that uses Celery. I use a general framework of utility functions(that I wrote myself) that gets pip installed as a local module, the utility library is also used in other projects.
I have a project directory like this:
project/
__init__.py
celery.py
test_runner.py
some_more_files_that_dont_matter_for_this_example.py
And a utility project that I install as a python module that's like this:
__init__.py
utils/
__init__.py
utils.py
setup.py
Now, I have some code to initialize celery in the celery.py, this includes from utils.utils import Utils. The test_runner.py looks like this:
from utils.utils import Utils
if __name__ == '__main__':
print 1+1
And my utils.py has the following import in it: from celery import Celery which SHOULD refer to the celery module I installed with pip install celery.
Now what goes wrong when I do python test_runner.py is that it tries to import utils.py, which then apperantly imports project/celery.py(not the celery module I pip-installed) and that makes a circular dependency happen(since project/celery.py imports utils/utils.py as well).
Can someone explain to me how to make sure the from celery import Celery in utils/utils.py only imports the actualy pip-installed celery module, and more importantly: why the imports happen like this? I'm running Python 3.5 and I thought that it would only do these local imports if I used from . import x.
The python interpreter adds the script's directory to sys.path (sort of reference). Furthermore, it is put at the beginning of the path and therefore searched first. This is why the local celery.py overrides the installed module.
You could either change sys.path and remove the local directory or put it at the back, but I think the better solution is not to name your local modules after global ones.
The from . import xyz syntax applies only to importing subpackages within packages (docs)
I have read numerous SO questions and blogs. I am trying to structure my flask application. The current structure of my application is the following:
application
run_server.py
/config
__init__.py
production.py
staging.py
development.py
/app
__init__.py
/site
__init__.py
views.py
Now, inside app/__init__.py I want to access the config based on my environment( dev, staging, production).
from ..config import config
I am getting this error:
ValueError: Attempted relative import beyond toplevel package
I have tried using -m switch.
I have also tried to set PYTHONPATH as my root directory to tell interpreter what is top level package.
I think I am missing some fundamental in relative imports.
Try using absolute import. IMHO it makes things easier to grok
from __future__ import absolute_import
from application.config import production
This is absolute because you are specifying the exact path you are importing from which reduces ambiguity.
Also, you are missing __init__.py in the application folder
If you are running your application through run_server.py then there is no need of relative import in app/__init__.py. You can simply say,
from config import <production/staging/development>
This is because, when your interpreter interprets run_server.py, at the line say, from app import <something>, it will fetch app/__init__.py content and try to execute them at toplevel i.e. from directory application.
Now, assume you are running from ..config import config at toplevel. Obviously, it will throw an error.
Assume you have configin application/config/__init__.py.
You also need __init__.py under application directory, if not, then the application/app is your top level package, you can not access application/config.
In this structure:
app
-companies
--models.py
--__init__.py
-manage.py
-__init__.py
models.py
class Company():
pass
manage.py
from flask import Flask
app = Flask(__name__)
from app.companies.models import Company
if __name__ == '__main__':
manager.run()
ImportError: No module named app.companies.models
What is the reason? In the views.py, inside the companies folder, the import works without any problem.
The problem is not with your code, but with how you're running it:
I am running python manage.py
For this to work, your working directory must be app. The current working directory is automatically added to your sys.path. There are two possibilities here, both of them bad:
The more likely is that the directory containing app isn't on sys.path at all, so as far as Python is concerned, there is no top-level package named app.
The less likely is that the directory containing app is on sys.path, as well as app itself. In that case, you will confuse the hell out of the importer, because the same file or directory can end up imported as two or more separate objects with different names in sys.modules. It's hard to predict exactly what problems that can cause, but it will cause problems.
Either way, the solution is to not run it that way. Instead, run it from one directory up, the directory that app is in. Which means you have to run it one of the following ways:
python -m app.manage
python app/manage.py