Here is the project Structure.
|-- a_api/
| |- a1.py
|
|-- b_api/
| |-b1.py
|
|-- c_api/
| |-c1.py
| |-c2.py
|
|-- utils/
| |-db.py
|
|-- main.py
db.py connects to mongo and stores connection in g from flask.
from flask import g
from pymongo import MongoClient
mongo_db = 'mongo_db'
def get_mongo_db():
"""Function will create a connection to mongo db for the current Request
Returns:
mongo_db: THe connection to Mongo Db
"""
if mongo_db not in g:
print('New Connection Created for mongo db')
mongo_client = MongoClient('the_url')
# Store the Client
g.mongo_db = mongo_client
else:
print('Old Connection reused for mongo db')
# Return The db
return g.mongo_db['db_name']
main.py calls two functions a1.py and b1.py
for a1: it interacts directly with the db.py file and updates data, this happens without any error and task is completed successfully.
for b1: it first calls c1 in a separate process, which used db.py and updates data - but in this case an error is thrown set up an application context with app.app_context()
How do I pass the application context to db.py when it is called from c1, which is called from b1?
How Do I create a single connection point to mongodb and use it across all requests or process in flask?
Traceback (most recent call last):
File "C:\Users\kunda\AppData\Local\Programs\Python\Python310\lib\multiprocessing\process.py", line 315, in _bootstrap
self.run()
File "C:\Users\kunda\AppData\Local\Programs\Python\Python310\lib\multiprocessing\process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "W:\xx\offers_api\offer_ops.py", line 60, in update_offer
response = aser(id=id, d=d)
File "W:\xx\offers_api\offer_ops.py", line 86, in aser
x = get_mongo_db()
File "W:\xx\utils\db.py", line 13, in get_mongo_db
if mongo_db not in g:
File "C:\Users\kunda\AppData\Local\Programs\Python\Python310\lib\site-packages\werkzeug\local.py", line 278, in __get__
obj = instance._get_current_object()
File "C:\Users\kunda\AppData\Local\Programs\Python\Python310\lib\site-packages\werkzeug\local.py", line 407, in _get_current_object
return self.__local() # type: ignore
File "C:\Users\kunda\AppData\Local\Programs\Python\Python310\lib\site-packages\flask\globals.py", line 40, in _lookup_app_object
raise RuntimeError(_app_ctx_err_msg)
RuntimeError: Working outside of application context.
This typically means that you attempted to use functionality that needed
to interface with the current application object in some way. To solve
this, set up an application context with app.app_context(). See the
documentation for more information.
Try something like this:
from flask import g
from pymongo import MongoClient
# import main flask app
from X import app
mongo_db = 'mongo_db'
def get_mongo_db():
"""Function will create a connection to mongo db for the current Request
Returns:
mongo_db: THe connection to Mongo Db
"""
# if circular dependency error try importing app here
from X import app
with app.app_context():
if mongo_db not in g:
print('New Connection Created for mongo db')
mongo_client = MongoClient('the_url')
# Store the Client
g.mongo_db = mongo_client
else:
print('Old Connection reused for mongo db')
# Return The db
return g.mongo_db['db_name']
Related
I am writing an integration test where I am creating a rabbitMQ container using -
docker run -d --hostname localhost -p 5672:5672 --name rabbit-tox -e RABBITMQ_DEFAULT_USER=guest -e RABBITMQ_DEFAULT_PASS=guest rabbitmq:3
To test if I can connect to rabbitMQ from within the test, I created this test and it can send the data -
def test_rmq(self):
connection = pika.BlockingConnection(pika.ConnectionParameters('localhost', 5672))
channel = connection.channel()
channel.queue_declare(queue='hello')
channel.basic_publish(exchange='',
routing_key='hello',
body='Hello World!')
print(" [x] Sent 'Hello World!'")
connection.close()
Now I want to use rabbitMQ container as celery backend, this is the code I am using -
from celery import Celery
broker_url = 'amqp://guest:guest#localhost:5672//'
app = Celery('test', broker=broker_url, backend='amqp')
from celery.contrib.testing.worker import start_worker
from swordfish_app import tasks
# Testing the consumer logic
class ServiceAccountCeleryTestCase(TransactionTestCase):
#classmethod
def setUpClass(cls):
super().setUpClass()
cls.celery_worker = start_worker(app)
cls.celery_worker.__enter__()
#classmethod
def tearDownClass(cls):
super().tearDownClass()
cls.celery_worker.__exit__(None, None, None)
def setUp(self):
super().setUp()
self.task = tasks.delete_all_service_accounts_when_user_inactive()
fake_obj_meta_del = V1ObjectMeta(self_link="deleted_service_account_link")
self.delete_namespaced_service_account_fake_data = V1Status(metadata=fake_obj_meta_del)
self.results = self.task.get()
#patch('kubernetes.client.CoreV1Api.delete_namespaced_service_account')
#patch('app.k8s.serviceaccount.get_inactive_serviceaccounts')
def test_delete_all_service_accounts_when_user_inactive(self, k8s_get_inactive_patch, k8s_del_sa_patch):
k8s_get_inactive_patch.return_value = ["sf-user-1", "sf-user-2"]
k8s_del_sa_patch.return_value = self.delete_namespaced_service_account_fake_data
assert self.task.state == "SUCCESS"
When I execute the test I find this error --
Creating test database for alias 'default'...
System check identified no issues (0 silenced).
...... [x] Sent 'Hello World!'
.E
======================================================================
ERROR: setUpClass (tests.test_service_accounts.ServiceAccountCeleryTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/mlokur/swordfish/src/tests/test_service_accounts.py", line 124, in setUpClass
cls.celery_worker.__enter__()
File "/usr/local/lib/python3.7/contextlib.py", line 112, in __enter__
return next(self.gen)
File "/home/mlokur/venv/lib/python3.7/site-packages/celery/contrib/testing/worker.py", line 78, in start_worker
**kwargs) as worker:
File "/usr/local/lib/python3.7/contextlib.py", line 112, in __enter__
return next(self.gen)
File "/home/mlokur/venv/lib/python3.7/site-packages/celery/contrib/testing/worker.py", line 103, in _start_worker_thread
assert 'celery.ping' in app.tasks
AssertionError
----------------------------------------------------------------------
Ran 7 tests in 1.842s
FAILED (errors=1)
Destroying test database for alias 'default'...
I wrote a separate python file to connect and check -
from celery import Celery
import urllib.request
import os
# Where the downloaded files will be stored
BASEDIR="/home/celery/downloadedFiles"
# Create the app and set the broker location (RabbitMQ)
broker_url = 'amqp://guest:guest#localhost:5672//'
app = Celery('test', broker=broker_url, backend='amqp')
#app.task
def download(url, filename):
"""
Download a page and save it to the BASEDIR directory
url: the url to download
filename: the filename used to save the url in BASEDIR
"""
response = urllib.request.urlopen(url)
data = response.read()
with open("snap",'wb') as file:
file.write(data)
file.close()
and this example celery task connects to the rabbitMQ properly and then works.
Any help is appreciated. Thanks.
Just disable ping check
start_worker(app, perform_ping_check=False)
Otherwise it will look for task celery.ping and chances are you didn't register such task.
But I'd suggest adding it anyway
#shared_task(name='celery.ping')
def ping():
return 'pong'
As it will allow you to healtcheck your celery worker.
I have a project where I am trying to build an api using flask and python with the following structure:
graph:
-app.py
-server.py
-apis:
-__init__.py
-users.py
-transaction_functions.py
-neo4j_ops.py
In the server.py file I am trying to add authentification to the endpoints of my api which are coded in the users.py file. My server.py file looks like this:
import json
from six.moves.urllib.request import urlopen
from functools import wraps
from flask import Flask, request, jsonify, _request_ctx_stack
from flask_cors import cross_origin
from jose import jwt
AUTH0_DOMAIN = 'mydomain.eu'
API_AUDIENCE = 'https://my_audience.com'
ALGORITHMS = ["RS256"]
APP = Flask(__name__)
# Error handler
class AuthError(Exception):
def __init__(self, error, status_code):
self.error = error
self.status_code = status_code
#APP.errorhandler(AuthError)
def handle_auth_error(ex):
#some code
# Format error response and append status code
def get_token_auth_header():
"""Obtains the Access Token from the Authorization Header
"""
# some code
return token
def requires_auth(f):
"""Determines if the Access Token is valid
"""
#wraps(f)
def decorated(*args, **kwargs):
#some code
return decorated
def requires_scope(required_scope):
"""Determines if the required scope is present in the Access Token
Args:
required_scope (str): The scope required to access the resource
"""
#some code
And I keep getting this error:
Traceback (most recent call last):
File "C:\Python37\lib\site-packages\flask_restplus\api.py", line 183, in init_app
app.record(self._deferred_blueprint_init)
AttributeError: 'Flask' object has no attribute 'record'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "app.py", line 16, in <module>
api.init_app(app)
File "C:\Python37\lib\site-packages\flask_restplus\api.py", line 186, in init_app
self._init_app(app)
File "C:\Python37\lib\site-packages\flask_restplus\api.py", line 204, in _init_app
self._register_view(app, resource, *urls, **kwargs)
File "C:\Python37\lib\site-packages\flask_restplus\api.py", line 282, in _register_view
resource_func = self.output(resource.as_view(endpoint, self, *resource_class_args,
AttributeError: 'function' object has no attribute 'as_view'
As you can see the result of this printstack is not really useful at all since none of these calls comes from any of my files.
The only file involved in that is app.py and it looks like this:
from flask import Flask
from flask_restplus import Api
from apis import api
import config
import os
app = Flask(__name__)
api.init_app(app)#traceback comes from here.
app.run(host='0.0.0.0', port=8080)
The apis/__init__.py file looks like this:
from flask_restplus import Api, fields
from .users import api as users
from flask import Flask
api = Api(
title='Graph Api',
version='0.2',
)
api.add_namespace(users)
Any idea of what the issue is?
If I import app (the flask instance) from app.py into server.py and use that flask instance created in app rather than creating a new whole flask instance in server.py somehow the error goes away, but the issue is that I would then create a circular call of dependencies, so I cant do that.
You already have an app
APP = Flask(__name__)
And your error handler at least is using that
Yet, you defined a second one
app = Flask(__name__)
And __name__ here is app (the name of the file), which is probably what's breaking things and why the server file isn't broken
The problem stopped existing as soon as I went back to the version of the code of before adding the server.py file, and then I added it again. I have no clue of what the issue was though as the code is actually the same.
I'm trying to use redis cache with my python code, below code works fine and it sets the keys perfectly. I wanted to set timeout when its not able to connect to redis or if the ports are not open.
unfortunately I could not able to find any document on how to pass the timeout to the connection parameters.
Following is my code.
from flask import Flask, render_template
from flask_caching import Cache
app = Flask(__name__, static_url_path='/static')
config = {
"DEBUG": True,
"CACHE_TYPE": "redis",
"CACHE_DEFAULT_TIMEOUT": 300,
"CACHE_KEY_PREFIX": "inventory",
"CACHE_REDIS_HOST": "localhost",
"CACHE_REDIS_PORT": "6379",
"CACHE_REDIS_URL": 'redis://localhost:6379'
}
cache = Cache(app, config=config)
socket_timeout = 5
#app.route('/')
#cache.memoize()
def dev():
# some code
return render_template("index.html", data=json_data, columns=columns)
when its not able to connect it waits for long time and throws the following error:
Traceback (most recent call last):
File "/Users/amjad/.virtualenvs/inventory/lib/python3.7/site-packages/flask_caching/__init__.py", line 771, in decorated_function
f, *args, **kwargs
File "/Users/amjad/.virtualenvs/inventory/lib/python3.7/site-packages/flask_caching/__init__.py", line 565, in make_cache_key
f, args=args, timeout=_timeout, forced_update=forced_update
File "/Users/amjad/.virtualenvs/inventory/lib/python3.7/site-packages/flask_caching/__init__.py", line 524, in _memoize_version
version_data_list = list(self.cache.get_many(*fetch_keys))
File "/Users/amjad/.virtualenvs/inventory/lib/python3.7/site-packages/flask_caching/backends/rediscache.py", line 101, in get_many
return [self.load_object(x) for x in self._read_clients.mget(keys)]
File "/Users/amjad/.virtualenvs/inventory/lib/python3.7/site-packages/redis/client.py", line 1329, in mget
return self.execute_command('MGET', *args, **options)
File "/Users/amjad/.virtualenvs/inventory/lib/python3.7/site-packages/redis/client.py", line 772, in execute_command
connection = pool.get_connection(command_name, **options)
File "/Users/amjad/.virtualenvs/inventory/lib/python3.7/site-packages/redis/connection.py", line 994, in get_connection
connection.connect()
File "/Users/amjad/.virtualenvs/inventory/lib/python3.7/site-packages/redis/connection.py", line 497, in connect
raise ConnectionError(self._error_message(e))
redis.exceptions.ConnectionError: Error 60 connecting to localhost:6379. Operation timed out.
Thanks in advance.
This question is fairly old but came across this exact problem just now and I found a solution. Leaving here for posterity for future readers.
According to the documentation at https://flask-caching.readthedocs.io/en/latest/index.html, the CACHE_TYPE parameter:
Specifies which type of caching object to use. This is an import string that will be imported and instantiated. It is assumed that the import object is a function that will return a cache object that adheres to the cache API.
So make a modified version of their redis function, found in flask_caching.backends.cache like so:
def redis_with_timeout(app, config, args, kwargs):
try:
from redis import from_url as redis_from_url
except ImportError:
raise RuntimeError("no redis module found")
# [... extra lines skipped for brevity ...]
# kwargs set here are passed through to the underlying Redis client
kwargs["socket_connect_timeout"] = 0.5
kwargs["socket_timeout"] = 0.5
return RedisCache(*args, **kwargs)
And use it instead of the default redis like so:
CACHE_TYPE = 'path.to.redis_with_timeout'
And the library will use that one instead, with the custom kwargs passed into the underlying Redis client. Hope that helps.
From latest document, there is an CACHE_OPTIONS config passed to almost every types of cache backends as keyword arguments:
Entries in CACHE_OPTIONS are passed to the redis client as **kwargs
We can simply pass additional settings like this:
from flask import Flask
from flask_caching import Cache
app = Flask(__name__)
config = {
"CACHE_TYPE": "redis",
...
"CACHE_REDIS_HOST": "localhost",
"CACHE_REDIS_PORT": "6379",
"CACHE_REDIS_URL": 'redis://localhost:6379',
"CACHE_OPTIONS": {
"socket_connect_timeout": 5, # connection timeout in seconds
"socket_timeout": 5, # send/recv timeout in seconds
}
}
cache = Cache(app, config=config)
in my flask app app/__init__.py I have included the below function
def create_app(config_filename=None):
_app = Flask(__name__, instance_relative_config=True)
with _app.app_context():
print "Creating web app",current_app.name
cors = CORS(_app, resources={r'/*': {"origins": "*"}})
sys.path.append(_app.instance_path)
_app.config.from_pyfile(config_filename)
from config import app_config
config = app_config[_os.environ.get('APP_SETTINGS', app_config.development)]
_app.config.from_object(config)
register_blueprints(_app)
# _app.app_context().push()
return _app
now I also have /app/datastore/core.py in which I have
import peewee
import os
from flask import g, current_app
cfg = current_app.config
dbName = 'clinic_backend'
def connect_db():
"""Connects to the specific database."""
return peewee.MySQLDatabase(dbName,
user=cfg['DB_USER'],
host=cfg['DB_HOST'],
port=3306,
password=cfg['DB_PWD']
)
def get_db():
""" Opens a new database connection if there is none yet for the
current application context.
"""
if not hasattr(g, 'db'):
g.db = connect_db()
return g.db
when I create my run my app start.py it creates the app object but when I try access the URL in browser I get error saying
File "/Users/ciasto/Development/python/backend/app/datastore/core.py",
line 31, in get_db
g.db = connect_db() File "/Users/ciasto/Development/python/backend/venv/lib/python2.7/site-packages/werkzeug/local.py",
line 364, in <lambda>
__setattr__ = lambda x, n, v: setattr(x._get_current_object(), n, v) File
"/Users/ciasto/Development/python/backend/venv/lib/python2.7/site-packages/werkzeug/local.py",
line 306, in _get_current_object
return self.__local() File "/Users/ciasto/Development/python/backend/venv/lib/python2.7/site-packages/flask/globals.py",
line 44, in _lookup_app_object
raise RuntimeError(_app_ctx_err_msg) RuntimeError: Working outside of application context.
This typically means that you attempted to use functionality that
needed to interface with the current application object in some way.
To solve this, set up an application context with app.app_context().
See the documentation for more information.
what am I doing wrong here?
what am I doing wrong here?
Just about everything.
Peewee already exposes database connections as a threadlocal so what you're doing is senseless. Especially given from reading your comments you are only trying to add connection hooks.
The peewee docs are QUITE CLEAR: http://docs.peewee-orm.com/en/latest/peewee/database.html#flask
As an aside, read the damn docs. How many questions have you posted already that could have been easily answered just by reading the documentation?
I am attempting to deploy a Flask app to Heroku. I'm using Peewee as an ORM for a Postgres database. When I follow the standard Heroku steps to deploying Flask, the web process crashes after I enter heroku ps:scale web=1. Here's what the logs say:
Starting process with command `python app.py`
/app/.heroku/venv/lib/python2.7/site-packages/peewee.py:2434: UserWarning: Table for <class 'flask_peewee.auth.User'> ("user") is reserved, please override using Meta.db_table
cls, _meta.db_table,
Traceback (most recent call last):
File "app.py", line 167, in <module>
auth.User.create_table(fail_silently=True)
File "/app/.heroku/venv/lib/python2.7/site-packages/peewee.py", line 2518, in create_table if fail_silently and cls.table_exists():
File "/app/.heroku/venv/lib/python2.7/site-packages/peewee.py", line 2514, in table_exists return cls._meta.db_table in cls._meta.database.get_tables()
File "/app/.heroku/venv/lib/python2.7/site-packages/peewee.py", line 507, in get_tables ORDER BY c.relname""")
File "/app/.heroku/venv/lib/python2.7/site-packages/peewee.py", line 313, in execute cursor = self.get_cursor()
File "/app/.heroku/venv/lib/python2.7/site-packages/peewee.py", line 310, in get_cursor return self.get_conn().cursor()
File "/app/.heroku/venv/lib/python2.7/site-packages/peewee.py", line 306, in get_conn self.connect()
File "/app/.heroku/venv/lib/python2.7/site-packages/peewee.py", line 296, in connect self.__local.conn = self.adapter.connect(self.
database, **self.connect_kwargs)
File "/app/.heroku/venv/lib/python2.7/site-packages/peewee.py", line 199, in connect return psycopg2.connect(database=database, **kwargs)
File "/app/.heroku/venv/lib/python2.7/site-packages/psycopg2/__init__.py", line 179, in connect connection_factory=connection_factory, async=async)
psycopg2.OperationalError: could not connect to server: No such file or directory
Is the server running locally and accepting connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
Process exited with status 1
State changed from starting to crashed
I've tried a bunch of different things to get Heroku to allow my app to talk to a Postgres db, but haven't had any luck. Is there an easy way to do this? What do I need to do to configure Flask/Peewee so that I can use a db on Heroku?
According to the Peewee docs, you don't want to use Proxy() unless your local database driver is different than your remote one (i.e. locally, you're using SQLite and remotely you're using Postgres). If, however, you are using Postgres both locally and remotely it's a much simpler change. In this case, you'll want to only change the connection values (database name, username, password, host, port, etc.) at runtime and do not need to use Proxy().
Peewee has a built-in URL parser for database connections. Here's how to use it:
import os
from peewee import *
from playhouse.db_url import connect
db = connect(os.environ.get('DATABASE_URL'))
class BaseModel(Model):
class Meta:
database = db
In this example, peewee's db_url module reads the environment variable DATABASE_URL and parses it to extract the relevant connection variables. It then creates a PostgresqlDatabase object with those values.
Locally, you'll want to set DATABASE_URL as an environment variable. You can do this according to the instructions of whatever shell you're using. Or, if you want to use the Heroku toolchain (launch your local server using heroku local) you can add it to a file called .env at the top level of your project. For the remote setup, you'll want to add your database URL as a remote Heroku environment variable. You can do this with the following command:
heroku config:set DATABASE_URL=postgresql://myurl
You can find that URL by going into Heroku, navigating to your database, and clicking on "database credentials". It's listed under URI.
Are you parsing the DATABASE_URL environment variable? It will look something like this:
postgres://username:password#host:port/database_name
So you will want to pull that in and parse it before you open a connection to your database. Depending on how you've declared your database (in your config or next to your wsgi app) it might look like this:
import os
import urlparse
urlparse.uses_netloc.append('postgres')
url = urlparse.urlparse(os.environ['DATABASE_URL'])
# for your config
DATABASE = {
'engine': 'peewee.PostgresqlDatabase',
'name': url.path[1:],
'password': url.password,
'host': url.hostname,
'port': url.port,
}
See the notes here: https://devcenter.heroku.com/articles/django
heroku config:set HEROKU=1
import os
import urlparse
import psycopg2
from flask import Flask
from flask_peewee.db import Database
if 'HEROKU' in os.environ:
DEBUG = False
urlparse.uses_netloc.append('postgres')
url = urlparse.urlparse(os.environ['DATABASE_URL'])
DATABASE = {
'engine': 'peewee.PostgresqlDatabase',
'name': url.path[1:],
'user': url.username,
'password': url.password,
'host': url.hostname,
'port': url.port,
}
else:
DEBUG = True
DATABASE = {
'engine': 'peewee.PostgresqlDatabase',
'name': 'framingappdb',
'user': 'postgres',
'password': 'postgres',
'host': 'localhost',
'port': 5432 ,
'threadlocals': True
}
app = Flask(__name__)
app.config.from_object(__name__)
db = Database(app)
Modified coleifer's answer to answer hasenj's comment.
Please mark one of these as the accepted answer.
I have managed to get my Flask app which uses Peewee working on Heroku using the below code:
# persons.py
import os
from peewee import *
db_proxy = Proxy()
# Define your models here
class Person(Model):
name = CharField(max_length=20, unique=True)
age = IntField()
class Meta:
database = db_proxy
# Import modules based on the environment.
# The HEROKU value first needs to be set on Heroku
# either through the web front-end or through the command
# line (if you have Heroku Toolbelt installed, type the following:
# heroku config:set HEROKU=1).
if 'HEROKU' in os.environ:
import urlparse, psycopg2
urlparse.uses_netloc.append('postgres')
url = urlparse.urlparse(os.environ["DATABASE_URL"])
db = PostgresqlDatabase(database=url.path[1:], user=url.username, password=url.password, host=url.hostname, port=url.port)
db_proxy.initialize(db)
else:
db = SqliteDatabase('persons.db')
db_proxy.initialize(db)
if __name__ == '__main__':
db_proxy.connect()
db_proxy.create_tables([Person], safe=True)
You should already have a Postgres database add-on attached to your app. You can do this via the command line or through the web front-end. Assuming that the database is already attached to your app and you have already deployed your with the above changes, log-in to Heroku and create the table(s):
$ heroku login
$ heroku run bash
$ python persons.py
Check that the table was created:
$ heroku pg:psql
your_app_name::DATABASE=> \dt
You then import this file (persons.py in this example) in another Python script, e.g. a request handler. You need to manage the database connection explicitly:
# server.py
from flask import g
from persons import db_proxy
#app.before_request
def before_request():
g.db = db_proxy
g.db.connect()
#app.after_request
def after_request(response):
g.db.close()
return response
…
References:
https://devcenter.heroku.com/articles/heroku-postgresql
http://peewee.readthedocs.org/en/latest/peewee/database.html#dynamically-defining-a-database
http://peewee.readthedocs.org/en/latest/peewee/example.html#establishing-a-database-connection
https://stackoverflow.com/a/20131277/3104465
https://gist.github.com/fprieur/9561148