I have read numerous SO questions and blogs. I am trying to structure my flask application. The current structure of my application is the following:
application
run_server.py
/config
__init__.py
production.py
staging.py
development.py
/app
__init__.py
/site
__init__.py
views.py
Now, inside app/__init__.py I want to access the config based on my environment( dev, staging, production).
from ..config import config
I am getting this error:
ValueError: Attempted relative import beyond toplevel package
I have tried using -m switch.
I have also tried to set PYTHONPATH as my root directory to tell interpreter what is top level package.
I think I am missing some fundamental in relative imports.
Try using absolute import. IMHO it makes things easier to grok
from __future__ import absolute_import
from application.config import production
This is absolute because you are specifying the exact path you are importing from which reduces ambiguity.
Also, you are missing __init__.py in the application folder
If you are running your application through run_server.py then there is no need of relative import in app/__init__.py. You can simply say,
from config import <production/staging/development>
This is because, when your interpreter interprets run_server.py, at the line say, from app import <something>, it will fetch app/__init__.py content and try to execute them at toplevel i.e. from directory application.
Now, assume you are running from ..config import config at toplevel. Obviously, it will throw an error.
Assume you have configin application/config/__init__.py.
You also need __init__.py under application directory, if not, then the application/app is your top level package, you can not access application/config.
Related
apologies if I'm going about this wrong, but I am quite new to python and can't quite figure out what the problem is! I have a simple Flask app that I'm trying to write tests using pytest for. The file structure is like this:
│── app.py
|── tests
│ ├── tests_app.py
I originally had app.py and tests_app.py in the same level and it all worked fine with tests passing, but now I have put tests in their own folder I can no longer import app without error.
I have tried the following in tests_app.py:
from app import app #this is what worked fine when app and tests were in the same folder
from ..app import app
from .. import app
and the linting error is 'Attempted relative import beyond top-level package' and when I run pytest it says 'ImportError: attempted relative import with no known parent package'.
Many thanks in advance!
p.s. I am using Python 3.8.5
You need to have app on the module search path. A quick fix would be to add the path to app.py to the PYTHONPATH environment variable:
export PYTHONPATH=/path/to/dir # where app.py is
If app.py is on the python path, then import app will work (barring any other import problems such as circular imports).
An effective and reproducible way to do this is to install your application. You can do this in develop mode, which means the code runs from its current directory. This is an understated, important part of developing a Python application, but to do it you will need to write code to package your application.
I have a flask app with essentially the following structure:
app/
__init__.py
myapp.py
common/
tool1.py
tool2.py
web/
__init__.py
views.py
api/
api_impl.py
worker/
__init__.py
worker.py
tasks.py
I initialize in myapp.py an important object I use in several places and I can access it from common/tool1.py and web/api/api_impl.py with from myapp import object. I've been able to use tool1 and tool2 in multiple places in web/ and myapp.py importing with from common.tool1 import tool1_def.
Other relevant facts are in myapp.py there is an import web statement for the blueprints and app/__init__.py and worker/__init__.py are empty. web/__init__.py contains the blueprint definitions for the routes.
I can run the app with gunicorn with no issues, but when I try to run my worker with python app/worker/worker.py I get the error ModuleNotFoundError: No module named 'myapp'. The worker.py is trying to import the same object defined in myapp.py.
I just don't understand why I can run the app and it works but when I try to run the worker it doesn't! I'm definitely not fully understanding the import system in this case and everything I've read online doesn't seem to fully clarify this.
your working imports imply that the project root is the app folder. As such you need to lunch your worker from this folder (or add it to PYTHONPATH environment variable)
python worker/worker.py
Or
python -m worker.worker
In addition your __init__.py in the app folder should be removed as app is not a package but a project root.
One method we use regularly for development and debugging (but also works in production) is the following. You can add this to each (offending?) module, or all modules if you choose.
import os
import sys
# Add your project root to sys.path.
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.realpath(__file__))))
from common import tool1
from common import tool2
from worker import worker
etc ...
Here you are adding your project root to sys.path. When performing imports, Python starts by looking through sys.path for the module you're trying to import. So here, you are adding your project root as the first item in sys.path.
Another advantage is these imports are explicit to your project. For example, if you have a local workers module which you want to import, but also have a workers package in site-packages, this will look in your local project first.
I am struggling to successfully import my project to the test-suite in my project, as well as being able to run the program from the command-line. I've been able to run my test-suite for some time, under the impression that if the tests work, so does the command-line stuff--evidently this isn't the case. I do not yet intend on using my program as a library. The api.py acts is the entry-point for the program.
I have a project with the following structure (the same directory hierarchy as requests):
myapp/
myapp/
__init__.py
api.py # depends on commands.py
commands.py # depends on utils.py
utils.py
tests/
context.py
test_api.py # depends on api.py
test_commands.py # depends on commands.py, utils.py
In the file context.py I have a path modification adding myapp to the PYTHONPATH, so I can successfully run the tests on my code. Here is the contents of that file
import os
import sys
sys.path.insert(0, os.path.abspath('..'))
import myapp
I've tried imaginable import combination I can think of. Far too many to list! I have also perused the Python reference import system page, and this tutorial.
How should I import my dependencies?
Turns out this was the correct layout, I mistook the error for something else. Although for future reference, relative imports in Python 3 must be explicit: when in the myapp package directory you can't say import commands, you must instead import it as from . import commands. This was defined in PEP 328 also see this SO post on the topic. Run your package with python -m mutil.api not python ./mutil/api.py as the latter won't give the interpreter context of the current path.
I want to have this structure for my project:
requirements.txt
README.md
.gitignore
project/
__init__.py
project.py
core/
__init__.py
base.py
engines/
__init__.py
engine1.py
engine2.py
utils/
__init__.py
refine_data.py
whatever.py
The application is run from project/project.py. However, I constantly get import errors when using relative or absolute imports.
Both engines need to import from project.core.base, the utils need to import from project.core.base as well, and project.py (the main file ran) needs to be able to import from engines.
Absolute imports don't work:
# engines/engine1.py
from project.core.base import MyBaseClass
which gives the error:
ImportError: No module named project.core.base
But if I try a relative import instead
# engines/engine1.py
from ..core.base import MyBaseClass
I get:
ValueError: Attempted relative import beyond toplevel package
I've seen other projects on Github structured similarly, but this seems to cause all sorts of problems. How do I get this to work?
Take a look at your sys.path. It's likely that the top project directory is in the python path, and it sees your sub-packages (ie. utils, engines, etc.) as separate packages, which is why it's giving you an error that you're trying to import from outside your package when doing relative imports, and absolute imports don't work because it can't find the top project directory because it's not under any of the python paths.
The directory above the top project directory is what needs to be added to the python path.
Ex.
/path/is/here/project/core/...
# Add this to the PYTHONPATH
/path/is/here
Try to use these imports:
engine1.py:
from core import base
refine_data.py:
from core import base
project.py
from engines import engine1
if you use pycharm mark project directory as sources root and then try to run project.py. If you don't use pycharm you can run project.py by going to project directory and running command:
python project.py
I have been looking for a way to solve this.
I have a python project, and this is the folder structure I want:
/project/main.py
/project/src/models.py
/project/test/tests.py
I want to be able to run the tests by executing the tests.py in terminal. tests.py imports modules in /project/src/ for testing. First I solved this by adding
sys.path.insert(0, '..') in tests.py. But then the paths used in models.py for opening text files had to be relative to the tests.py, etc. Which means the program wouldn't run when excecuted from main.py, cause of the paths.
I also tried with dots when importing modules into tests.py, like from ..src.models import *, but that gave error message saying "Attempted relative import in non-package".
What should I put in the top of tests.py to be able to import the modules from models.py?
The structure you're using is not one I would recommend, but I'm a comparaitive newb to how Python projects are usually structured. I believe this will do what you're after:
1) Place an __init__.py file inside /project, /project/src, and /project/test to make sure they're treated as packages.
2) Place from __future__ import absolute_import at the top of each Python file.
3) Then use relative imports:
test.py:
from ..src import models
main.py:
from .src import models
4) You'll need to start your application differently. Ensure your current directory is the parent of /project (which appears to be the file system root) and run your project this way:
python -m project.main
For my own project, I would definitely put main.py inside src if it's the start point of your application. I might put tests.py in src, too, but if not, I would add /project/src to the test runner's Python path on the command line instead of in code.
I would still use absolute_import regardless. In my experience, it's a very clean solution to module organization, and it's also how things work by default in Python 3.
first put your main.py in the src directory..
in your tests you can do , sys.path.append('the src directory')
if you like to force the execution to be in specific directory regardless from where you are executing the app i suggest you adding
import os
os.chdir('relative path to the src dir')
thisway your program will run in the directory you specified so it will respect the relative paths you have in your code