We Keep Coding

Python circular import error with flask and celery

How can I make #celery.task available on user resources package?
I'm new to python and confused about the "circular imports error" can somebody explain how it works? and how to handle this kind of errors in flask application. This is the part where I always get stuck.
This is my project current folder structure
code
|_resources
| |_user.py
|
|_utils
| |_flask_celery.py
|
|_flask_app.py
--- SOURCE CODE ---
flask_app.py
from flask import Flask
from flask_cors import CORS
from flask_restful import Api
from util.flask_celery import make_celery
from routes.endpoint import urls
from resources.user import Users
app = Flask(__name__)
CORS(app, resources={r"/*": {"origins": "*"}})
app.config['SQLALCHEMY_DATABASE_URI'] = 'mysql:///username:StrongPassword#localhost:3306/db'
app.config['SQLALCHEMY_TRACK_MODIFICATIONS'] = False
app.config['CELERY_BROKER_URL'] = 'amqp//admin:StrongPassword#localhost:5672'
app.config['CELERY_RESULT_BACKEND'] = 'db+mysql:///username:StrongPassword#localhost:3306/db'
app.config['PROPAGATE_EXCEPTIONS'] = True
api = Api(app)
celery = make_celery(app)
url = urls()
api.add_resource(Users, url.get('users'))
if __name__ == "__main__":
# sqlalchemy
from db import db
db.__init__(app)
app.run(host='0.0.0.0', debug=True, port=5000)
flask_celery.py
from celery import Celery
def make_celery(celery_app):
celery = Celery(
celery_app.import_name,
backend=celery_app.config['CELERY_RESULT_BACKEND'],
broker=celery_app.config['CELERY_BROKER_URL']
)
celery.conf.update(celery_app.config)
class ContextTask(celery.Task):
def __call__(self, *args, **kwargs):
with celery_app.app_context():
return self.run(*args, **kwargs)
celery.Task = ContextTask
return celery
user.py
from flask_restful import Resource, reqparse
from app import celery # <-- this cause the error
from util.zk_connector import zk_connect
from util.zk_error import error
class Users(Resource):
def post(self, ip, comkey):
zk = zk_connect(ip=ip, password=int(comkey))
try:
session = zk.connect()
session.disable_device()
users = session.get_users()
print(users)
session.enable_device()
session.disconnect()
return {'message': 'success'}, 200
except Exception as e:
return error(e)
#celery.task(name='user.reverse')
def reverse(string):
return string[::-1]
Error:
Traceback (most recent call last):
File ".\flask_app.py", line 6, in <module>
from resources.user import User
File "C:\Users\Gelo\Documents\Brand new clean arch pyzk\code\resources\user.py", line 2, in <module>
from flask_app import celery
File "C:\Users\Gelo\Documents\Brand new clean arch pyzk\code\flask_app.py", line 6, in <module>
from resources.user import User
ImportError: cannot import name 'User' from partially initialized module 'resources.user' (most likely due to a circular import) (C:\Users\Gelo\Documents\Brand new clean arch pyzk\code\resources\user.py)

Mock database connection python

I have a file called redis_db.py which has code to connect to redis
import os
import redis
import sys
class Database:
def __init__(self, zset_name):
redis_host = os.environ.get('REDIS_HOST', '127.0.0.1')
redis_port = os.environ.get('REDIS_PORT', 6379)
self.db = redis.StrictRedis(host=redis_host, port=redis_port)
self.zset_name = zset_name
def add(self, key):
try:
self.db.zadd(self.zset_name, {key: 0})
except redis.exceptions.ConnectionError:
print("Unable to connect to redis host.")
sys.exit(0)
I have another file called app.py which is like this
from flask import Flask
from redis_db import Database
app = Flask(__name__)
db = Database('zset')
#app.route('/add_word/word=<word>')
def add_word(word):
db.add(word)
return ("{} added".format(word))
if __name__ == '__main__':
app.run(host='0.0.0.0', port='8080')
Now I am writing unit test for add_word function like this
import unittest
import sys
import os
from unittest import mock
sys.path.append(os.path.dirname(os.path.realpath(__file__)) + "/../api/")
from api import app # noqa: E402
class Testing(unittest.TestCase):
def test_add_word(self):
with mock.patch('app.Database') as mockdb:
mockdb.return_value.add.return_value = ""
result = app.add_word('shivam')
self.assertEqual(result, 'shivam word added.')
Issue I am facing is that even though I am mocking the db method call it is still calling the actual method in the class instead of returning mocked values and during testing I am getting error with message Unable to connect to redis host..
Can someone please help me in figuring out how can I mock the redis database calls.
I am using unittest module

The issue is that db is defined on module import, so the mock.patch does not affect the db variable. Either you move the instantiation of
db in the add_word(word) function or you patch db instead of Database, e.g.
def test_add_word():
with mock.patch('api.app.db') as mockdb:
mockdb.add = mock.MagicMock(return_value="your desired return value")
result = app.add_word('shivam')
print(result)
Note that the call of add_word has to be in the with block, otherwise the unmocked version is used.

"flask.cli.NoAppException" while running flask app

I am fairly new to python development and have no idea about flask, I have been assigned a project that is developed using flask. After working for couple of weeks i am now able to resolve all the dependencies and project is now compiled successfully. But when I run the project using flask run and then enter the url in browser it throws "flask.cli.NoAppException". How can I run my project I have tried like this.
set FLASK_APP=init.py
set FLASK_ENV=develpment
flask run
Serving Flask app "init.py" (lazy loading)
Environment: develpment
Debug mode: on
Restarting with stat
Debugger is active!
Debugger PIN: 202-733-235
Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
here is the trackback
"FLASK_APP=myappnam:name to specify one.
Traceback (most recent call last)
File "C:\Program Files\Python38\Lib\site-packages\flask\_compat.py", line 39, in reraise
raise value
File "C:\Program Files\Python38\Lib\site-packages\flask\cli.py", line 97, in find_best_app
raise NoAppException(
flask.cli.NoAppException: Failed to find Flask application or factory in module "myappnam". Use "FLASK_APP=myappnam:name to specify one.
The debugger caught an exception in your WSGI application. You can now look at the traceback which led to the error.
To switch between the interactive traceback and the plaintext one, you can click on the "Traceback" headline. From the text traceback you can also create a paste of it. For code execution mouse-over the frame you want to debug and click on the console icon on the right side.
You can execute arbitrary Python code in the stack frames and there are some extra helpers available for introspection:
dump() shows all variables in the frame
dump(obj) dumps all that's known about the object
here is my --init--.py file
import os
import logging
import gevent
import datetime
import rollbar
from gevent.queue import Queue
from gevent.event import AsyncResult
import zmq.green as zmq
from werkzeug.contrib.fixers import ProxyFix
# Greens the postgress connector
try:
import psycogreen.gevent
psycogreen.gevent.patch_psycopg()
except ImportError:
pass
from rauth.service import OAuth2Service
from flask import Flask, Request
from flask_sqlalchemy import SQLAlchemy
from flask_bcrypt import Bcrypt
from flask_login import LoginManager, current_user
from flask_assets import Environment
from flask_uploads import UploadSet, configure_uploads, IMAGES
from app.util import setup_logging
from app.exceptions import TimeoutError, BackendError
import app.exceptions
flask_app = None
# Have to use an actor pattern because we cannot allow more than one request to
# be pending at a time.
class Backend(gevent.Greenlet):
def __init__(self):
super(Backend, self).__init__()
self.inbox = Queue()
self.zmq_context = zmq.Context()
self.zmq_socket = None
self.init_socket()
def init_socket(self):
zmq_socket = self.zmq_socket
if zmq_socket is not None:
zmq_socket.close(0)
zmq_socket = self.zmq_context.socket(zmq.REQ)
zmq_socket.connect(flask_app.config["SERVER_ZMQ_URI"])
self.zmq_socket = zmq_socket
def process(self, request):
zmq_socket = self.zmq_socket
poller = zmq.Poller()
poller.register(zmq_socket, zmq.POLLIN)
zmq_socket.send_json({
"command": request["command"],
"arguments": request["arguments"]
})
sockets = dict(poller.poll(10 * 1000))
if zmq_socket not in sockets:
self.init_socket()
result = request["result"]
result.set_exception(TimeoutError("The request to the backend timed out."))
return
received = zmq_socket.recv_json()
result = request["result"]
if received["success"]:
result.set(received["result"])
else:
result.set_exception(BackendError(received["result"]))
def _run(self):
while True:
self.process(self.inbox.get())
def send(self, command, **kwargs):
result = AsyncResult()
self.inbox.put({
"command": command,
"arguments": kwargs,
"result": result
})
return result.get()
class RollbarRequest(Request):
#property
def rollbar_person(self):
if current_user.is_anonymous:
return {
"id": 0,
"username": "anonymous"
}
return {
"id": current_user.id,
"username": current_user.name,
"email": current_user.email_address
}
def create_app(*args, **kwargs):
global flask_app
global l
app_mode = os.environ.get("APP_MODE")
assert app_mode is not None, "APP_MODE environment variable must be set"
flask_app = Flask(__name__)
flask_app.request_class = RollbarRequest
flask_app.config.from_object("config.mode_{}".format(app_mode))
flask_app.config["APP_MODE"] = app_mode
setup_logging(flask_app.config["LOGGING_LEVEL"])
l = logging.getLogger(__name__)
l.info("starting in mode {}".format(app_mode))
if not flask_app.config["DEBUG"]:
rollbar.init(
flask_app.config["ROLLBAR_API_KEY"],
app_mode,
allow_logging_basic_config=False
)
flask_app.jinja_env.globals.update(
current_year=lambda: datetime.datetime.now().year
)
# Have to do this so that redirects work in proxy mode behind NGINX.
if not flask_app.debug:
flask_app.wsgi_app = ProxyFix(flask_app.wsgi_app)
flask_app.db = SQLAlchemy(flask_app)
flask_app.bcrypt = Bcrypt(flask_app)
flask_app.assets = Environment(flask_app)
flask_app.images = UploadSet("images", IMAGES)
configure_uploads(flask_app, flask_app.images)
flask_app.photos = UploadSet("photos", IMAGES)
configure_uploads(flask_app, flask_app.photos)
login_manager = LoginManager()
login_manager.login_view = "signin"
login_manager.login_message_category = "alert" # Need newer release of Flask-Login for this to work.
login_manager.init_app(flask_app)
flask_app.facebook = OAuth2Service(
name="facebook",
base_url="https://graph.facebook.com/v2.8/",
client_id=flask_app.config["FACEBOOK_CLIENT_ID"],
client_secret=flask_app.config["FACEBOOK_CLIENT_SECRET"]
)
from app import views
from app import models
from app import commands
flask_app.backend = Backend()
flask_app.backend.start()
app.exceptions.register(flask_app)
return flask_app
and this is my project structure
I am using pycharm and widows 10.

In your create_app function you probably do not want to use the global keyword for late initialisation of the app.
In the examples you provided the create_app function is never called, and so the app instance is never created. It is more common to use the create_app function as follows:
def create_app():
app = flask.Flask(__name__)
# do some setup
return app
app = create_app()
The app instance should also be called app and not flask_app. Of course you can call it whatever you want, but by default flask looks for app. To specify your own change FLASK_APP=__init__.py to FLASK_APP=__init__:flask_app

Python Flask factory application problem with Flask-Mqtt has no attribute 'init_app'

I struggle with the implementation of the Flask-MQTT lib to my app. SQLAlchemy etc works fine, but
flask-mqtt throws the error AttributeError: module 'app.mqtt' has no attribute 'init_app'.
In the offical documentatio of Flask-MQTT they build up the create_app() Method the same way
( https://flask-mqtt.readthedocs.io/en/latest/usage.html )
Would be greate if someone can help me! Thank you very much
__init__.py
from flask import Flask
from flask_restful import Api
from flask_mqtt import Mqtt
from flask_sqlalchemy import SQLAlchemy
from flask_marshmallow import Marshmallow
from flask_jwt_extended import JWTManager
mqtt = Mqtt()
api = Api()
db = SQLAlchemy()
ma = Marshmallow()
jwt = JWTManager()
def create_app(config):
app = Flask(__name__)
app.config.from_object(config.DevelopmentConfig)
mqtt.init_app(app)
db.init_app(app)
api.init_app(app)
ma.init_app(app)
jwt.init_app(app)
return app
from app.mqtt import mqttclient
run.py
from app import create_app
import config
from flask_migrate import Migrate
app = create_app(config)
migrate = Migrate(app, db)
app.config['MQTT_BROKER_URL'] = 'hivemq'
app.config['MQTT_BROKER_PORT'] = 1883
app.config['MQTT_USERNAME'] = ''
app.config['MQTT_PASSWORD'] = ''
app.config['MQTT_KEEPALIVE'] = 5
app.config['MQTT_TLS_ENABLED'] = False
if __name__ == '__main__':
app.run(host='0.0.0.0', port=5003, threaded=True)
ERROR:
mqttservice | Traceback (most recent call last):
mqttservice | File "run.py", line 5, in <module>
mqttservice | app = create_app(config)
mqttservice | File "/code/app/__init__.py", line 18, in create_app
mqttservice | mqtt.init_app(app)
mqttservice | AttributeError: module 'app.mqtt' has no attribute 'init_app'

In that code fragment, mqtt has two meanings. The first is the variable assigned in
mqtt = Mqtt()
The second is a namespace (module)
from app.mqtt import mqttclient
The tell is in the error
AttributeError: module 'app.mqtt' has no attribute 'init_app'
which is happening because the import is overwriting the initial value, so by the time of that .init_app(), mqtt isn't what you expected.
You're going to have to change one of those names.

drop_all() freezes in Flask with SQLAlchemy

I am writing test cases for a Flask application.
I have a setUp method which drops the tables in the db before re-creating them again.
It looks like this:
def setUp(self):
# other stuff...
myapp.db.drop_all()
myapp.db.create_all()
# db creation...
This works fine for the first test, but it freezes at drop_all before the second test is run.
EDIT:
The stack trace looks like this when interrupting the process
File "populate.py", line 70, in create_test_db
print (myapp.db.drop_all())
File ".../flask_sqlalchemy/__init__.py", line 864, in drop_all
self._execute_for_all_tables(app, bind, 'drop_all')
File ".../flask_sqlalchemy/__init__.py", line 848, in _execute_for_all_tables
op(bind=self.get_engine(app, bind), tables=tables)
File ".../sqlalchemy/sql/schema.py", line 3335, in drop_all
....
File "/Library/Python/2.7/site-packages/MySQLdb/cursors.py", line 190, in execute
r = self._query(query)
Anybody has a clue how to fix this?

Oki, there might be other solutions but for now, after searching the interwebs, I found that the problem disappears if I prepend my code with a myapp.db.session.commit(). I guess, somewhere a transaction was waiting to be committed.
def setUp(self):
# other stuff...
myapp.db.session.commit() #<--- solution!
myapp.db.drop_all()
myapp.db.create_all()
# db creation...

Just close all sessions in your app and after that invoke drop_all
def __init__(self, conn_str):
self.engine = create_engine(conn_str)
self.session_factory = sessionmaker(engine)
def drop_all(self):
self.session_factory.close_all() # <- don't forget to close
Base.metadata.drop_all(self._engine)
more information about Sessions in SQLAlchemy
http://docs.sqlalchemy.org/en/latest/orm/session_api.html?highlight=close_all

I am a Flask developer and using flask_sqlalchemy and pytest to test my app server, I run into similar situation when I run the statement db.drop_all(), console shows that one of my table is locked.
I use db.session.remove()to remove the session before running db.drop_all().

I had the same problem, in my case I had 2 different sessions making queries to the same table. My solution was to use one scoped_session for both places.
I created it in a different module so I had no problem with circular dependencies, like this:
db.py:
from flask.ext.sqlalchemy import SQLAlchemy
db = SQLAlchemy()
models.py:
from .db import db
class User(db.Model):
id = db.Column(db.Integer, primary_key=True)
username = db.Column(db.String(80), unique=True)
app.py:
from flask import Flask
from .db import db
app = Flask(__name__)
db.init_app(app)
Using only the db.session in all your code will guarantee that you are in the same session. In the tests, make sure you execute the rollback at tearDown.