I'm reading env variables from .prod.env file in my config.py:
from pydantic import BaseSettings
class Settings(BaseSettings):
A: int
class Config:
env_file = ".prod.env"
env_file_encoding = "utf-8"
settings = Settings()
in my main.py I'm creating the app like so:
from fastapi import FastAPI
from app.config import settings
app = FastAPI()
print(settings.A)
I am able to override settings variables like this in my conftest.py:
import pytest
from fastapi.testclient import TestClient
from app.main import app
from app.config import settings
settings.A = 42
#pytest.fixture(scope="module")
def test_clinet():
with TestClient(app) as client:
yield client
This works fine, whenever I use settings.A I get 42.
But is it possible to override the whole env_file from .prod.env to another env file .test.env?
Also I probably want to call settings.A = 42 in conftest.py before I import app, right?
You can override the env file you use by creating a Settings instance with the _env_file keyword argument.
From documentation:
Passing a file path via the _env_file keyword argument on instantiation (method 2) will override the value (if any) set on the Config class. If the above snippets were used in conjunction, prod.env would be loaded while .env would be ignored.
For example, this should work for your test -
import pytest
from fastapi.testclient import TestClient
import app.config as conf
from app.config import Settings
# replace the settings object that you created in the module
conf.settings = Settings(_env_file='.test.env')
from app.main import app
# just to show you that you changed the module-level
# settings
from app.config import settings
#pytest.fixture(scope="module")
def test_client():
with TestClient(app) as client:
yield client
def test_settings():
print(conf.settings)
print(settings)
And you could create a .test.env, set A=10000000, and run with
pytest -rP conftest.py
# stuff
----- Captured stdout call -----
A=10000000
A=10000000
This looks a little messy (though this is probably only used for test purposes), so I'd recommend not creating a settings object that is importable by everything in your code, and instead making it something you create in, say, your __main__ that actually creates and runs the app, but that's a design choice for you to make.
I ran into the same issue today, really annoying. My goal was to set a different postgresql database for the unit tests. The default configuration comes from a .env file. But when you think of it thoroughly it is not that difficult to understand. It all boils down to the order of the imported modules in conftest.py. I based the example below on the answer of #wkl:
import pytest
from typing import Generator
from fastapi.testclient import TestClient
import app.core.config as config
from app.core.config import Settings
# Replace individual attribute in the settings object
config.settings = Settings(
POSTGRES_DB="test_db")
# Or replace the env file in the settings object
config.settings = Settings(_env_file='.test.env')
# All other modules that import settings are imported here
# This ensures that those modules will use the updated settings object
# Don't forget to use "noqa", otherwise a formatter might put it back on top
from app.main import app # noqa
from app.db.session import SessionLocal # noqa
#pytest.fixture(scope="session")
def db() -> Generator:
try:
db = SessionLocal()
yield db
finally:
db.close()
#pytest.fixture(scope="module")
def client() -> Generator:
with TestClient(app) as c:
yield c
One workaround I have found is to remove the env_file from Config completely and replace it's functionality with load_dotenv() from dotenv like this:
config.py:
from pydantic import BaseSettings
from dotenv import load_dotenv
load_dotenv(".prod.env")
class Settings(BaseSettings):
A: int
settings = Settings()
conftest.py:
import pytest
from fastapi.testclient import TestClient
from dotenv import load_dotenv
load_dotenv("test.env")
from app.config import settings
from app.main import app
#pytest.fixture(scope="module")
def test_clinet():
with TestClient(app) as client:
yield client
Please note, that calling load_dotenv("test.env") happens before importing the settings (from app.config import settings)
and also note that load_dotenv() will load environment variables globally for the whole python script.
Loading env variables like this will not override already exported variables, same as using the env_file in pydantic's BaseSettings
Related
I'm trying to make a Flask api :
app.py
routes
-> __init__.py
-> myobject.py
# app.py
from flask import Flask
from flask_restful import Api
from routes import MyObject
app = Flask(__name__)
api = Api(app)
api.add_resource(MyObject, '/')
if __name__ == '__main__':
app.run()
# __init__.py
from myobject import MyObject
# myobject.py
from flask_restful import Resource
class MyObject(Resource):
def get(self):
return {'hello': 'world'}
When I run my application (python app.py), I get a ModuleNotFoundError: No module named 'myobject'
I don't understand why python can't find my module myobject. Is there something i'm missing in my __init__.py
For __init__.py, it also needs the relative import or absolute import path in order to work correctly.
# __init__.py
# relative import
from .myobject import MyObject
# absolute import
from routes.myobject import MyObject
In another approach, you can write to import MyObject more specifically like this and leave __init__.py an empty file.
# app.py
# ...
from routes.myobject import MyObject
# ...
I believe the problem is due to your app running in the "main" directory and your import residing in a sub directory. Basically your app thinks you are trying to import "/main/myobject.py" instead of "/main/routes/myobject.py"
Change your import in __init__ to from .myobject import MyObject
The . means "this directory" (essentially).
I am having trouble with getting uvicorn to start. I am very new to python and fastapi so I am assuming I have doing something very silly.
I have isolated the problem to being in my api_router.py file
from fastapi import APIRouter
from API.endpoints import user_endpoints
api_router = APIRouter()
api_router.include_router(user_endpoints, prefix="/user", tags=["User"])
When I comment out from API.endpoints import user_endpoints and
api_router.include_router(user_endpoints, prefix="/user", tags=["User"]), the error does not occur. Am I trying to import my user_endpoints.py file incorrectly? I have attached an image of the directory structure.
user_endpoints.py looks like this:
from fastapi.routing import APIRouter
from typing import Optional, List
from API.schema.user import User
from models.user import Users
from main import db
router = APIRouter
#router.get('/users', response_model=List[User], status_code=200)
def get_all_users():
users=db.query(Users).all()
return users
#router.get('/users/{user_id}')
def get_user(user_id):
pass
#router.post('/users')
def create_user():
pass
#router.put('/users/{user_id}')
def update_user(user_id):
pass
#router.delete('/users/{user_id}')
def delete_user(user_id):
pass
Any help with this would be greatly appreciated.
Thanks,
Greg
I think it's talking about the current working directory of your terminal, when you feed it uvicorn main:app ... not being able to find main. Make your terminal's working directory same as main.py
I am preparing unittests for Airflow DAGs, and I have ran into below problem.
I have a module with common functions, which imports airflow variables into common global variable:
"""
Script for common functions used in airflow DAGs
"""
[...]
from airflow.models import Variable
[...]
config = Variable.get("some_dag_details")
[...]
I have code that import DAG files for test validation.
import unittest
from unittest.mock import patch
from airflow.models import DagBag
from some_package import common_functions
class DAGValidationTest(unittest.TestCase):
DETAILS = {
"some important data"
}
def setUp(self):
with patch.object(common_functions.Variable, 'get', return_value=self.DETAILS) as mock_get_variable:
[...]
Problem is that code for connecting to database is called before I am able to mock class Variable or variable config. Python is showing 4th line (from some_package import common_functions) as the error source. How can I mock this object before calling this script ?
Is it possible to mock a module in python using unittest.mock? I have a module named config, while running tests I want to mock it by another module test_config. how can I do that ? Thanks.
config.py:
CONF_VAR1 = "VAR1"
CONF_VAR2 = "VAR2"
test_config.py:
CONF_VAR1 = "test_VAR1"
CONF_VAR2 = "test_VAR2"
All other modules read config variables from the config module. While running tests I want them to read config variables from test_config module instead.
If you're always accessing the variables in config.py like this:
import config
...
config.VAR1
You can replace the config module imported by whatever module you're actually trying to test. So, if you're testing a module called foo, and it imports and uses config, you can say:
from mock import patch
import foo
import config_test
....
with patch('foo.config', new=config_test):
foo.whatever()
But this isn't actually replacing the module globally, it's only replacing it within the foo module's namespace. So you would need to patch it everywhere it's imported. It also wouldn't work if foo does this instead of import config:
from config import VAR1
You can also mess with sys.modules to do this:
import config_test
import sys
sys.modules["config"] = config_test
# import modules that uses "import config" here, and they'll actually get config_test
But generally it's not a good idea to mess with sys.modules, and I don't think this case is any different. I would favor all of the other suggestions made over it.
foo.py:
import config
VAR1 = config.CONF_VAR1
def bar():
return VAR1
test.py:
import unittest
import unittest.mock as mock
import test_config
class Test(unittest.TestCase):
def test_one(self):
with mock.patch.dict('sys.modules', config=test_config):
import foo
self.assertEqual(foo.bar(), 'test_VAR1')
As you can see, the patch works even for code executed during import foo.
If you want to mock an entire module just mock the import where the module is used.
myfile.py
import urllib
test_myfile.py
import mock
import unittest
class MyTest(unittest.TestCase):
#mock.patch('myfile.urllib')
def test_thing(self, urllib):
urllib.whatever.return_value = 4
Consider this following setup
configuration.py:
import os
class Config(object):
CONF_VAR1 = "VAR1"
CONF_VAR2 = "VAR2"
class TestConfig(object):
CONF_VAR1 = "test_VAR1"
CONF_VAR2 = "test_VAR2"
if os.getenv("TEST"):
config = TestConfig
else:
config = Config
now everywhere else in your code you can use:
from configuration import config
print config.CONF_VAR1, config.CONF_VAR2
And when you want to mock your coniguration file just set the environment variable "TEST".
Extra credit:
If you have lots of configuration variables that are shared between your testing and non-testing code, then you can derive TestConfig from Config and simply overwrite the variables that need changing:
class Config(object):
CONF_VAR1 = "VAR1"
CONF_VAR2 = "VAR2"
CONF_VAR3 = "VAR3"
class TestConfig(Config):
CONF_VAR2 = "test_VAR2"
# CONF_VAR1, CONF_VAR3 remain unchanged
If your application ("app.py" say) looks like
import config
print config.var1, config.var2
And gives the output:
$ python app.py
VAR1 VAR2
You can use mock.patch to patch the individual config variables:
from mock import patch
with patch('config.var1', 'test_VAR1'):
import app
This results in:
$ python mockimport.py
test_VAR1 VAR2
Though I'm not sure if this is possible at the module level.
I'm using WSGI/Apache2 and am trying to declare my database pool on init, to be accessible via a global var from my endpoints. I'm using Redis and Cassandra (DSE, specifically). It's my understanding that both the Redis and DSE libs offer pool management so this shouldn't be an issue.
My folder structure for my WSGI app looks something akin to
folder/
tp.wsgi
app/
__init__.py
decorators/
cooldec.py
mod_api/
controllers.py
tp.wsgi looks like the following
#! /usr/bin/env python2.7
import sys
import logging
logging.basicConfig(stream=sys.stderr)
sys.path.insert(0, "/opt/tp")
from app import app
def application(environ, start_response):
return app(environ, start_response)
__init__.py looks like the following
#! /usr/bin/env python2.7
from flask import Flask
from cassandra.cluster import Cluster
# Import our handlers
from app.mod_api.files import mod_files
# Setup routine
def setup():
# Instantiate Flask
app = Flask('app')
# Set up a connection to Cassandra
cassandraSession = Cluster(['an ip address', 'an ip address']).connect('keyspace')
cassandraSession.default_timeout = None
# Register our blueprints
app.register_blueprint(mod_files)
...
return app, cassandraSession
app, cassandraSession = setup()
I'm calling a decorator defined in cooldec.py that handles authentication (I use that term loosely, for a reason. I ask that we not go down the path of using Flask extensions for authentication, that's out of scope for this question and isn't applicable in my use-use [see: loose usage of the term 'authentication'])
In cooldec.py and controllers.py I'm trying to access the cassandraSession global but I keep getting global name 'cassandraSession' is not defined. I know what the error means, but I'm not sure why I'm seeing this. It's my understanding that the way I've set my WSGI app up allows for cassandraSession to be accessible within the scope of the app, no?
I found Preserving state in mod_wsgi Flask application but .. it hasn't really shed any light on to what I'm doing wrong.
My issue was the location of my imports. I made a few changes to tp.wsgi and __init__.py and I've got what I need working. That is, calling from app import cassandraSession from within cooldec.py and controllers.py
Below is how I've set up the aforementioned.
tp.wsgi
#! /usr/bin/env python2.7
import sys
import logging
logging.basicConfig(stream=sys.stderr)
sys.path.insert(0, "/opt/tp")
from app import app as application
__init__.py
#! /usr/bin/env python2.7
# API Module
from flask import Flask, jsonify
from cassandra.cluster import Cluster
# Create our API
app = Flask('app')
# Define a cassandra cluster/session we can use
cassandraSession = Cluster(['an ip address' 'an ip address']).connect('keyspace')
cassandraSession.default_timeout = None
... Register blueprints
These are overly simplified edits, but it gives the idea of what I was doing wrong (eg: declaring in wrong file and trying to import improperly.
In both cooldec.py and controllres.py we can now do
from app import cassandraSession
rows = cassandraSession.execute('select * from table')
Tip for new WSGI developers: Continue to think "in python".
+ WARNING +
I have yet to find an absolute answer on whether or not this is safe to do. Doing this using sqlalchemy is perfectly OK due to how sqlalchemy handles connection pooling. I am, as of yet, unaware if this is safe to do with Cassandra/DSE, so proceed with caution if you utilize this post.