I am trying to create a REST API in Flask. The thing is it runs perfectly for a few days and then all of a sudden it STOPS receiving requests altogether. Forget about not responding to requests; it just doesn't receive any requests at the first place. This is my script:
from flask import Flask, jsonify
from flask_restful import Resource, Api
from flask_restful import reqparse
from sqlalchemy import create_engine
from flask.ext.httpauth import HTTPBasicAuth
from flask.ext.cors import CORS
conn_string = "mssql+pyodbc://x"
e = create_engine(conn_string)
auth = HTTPBasicAuth()
#auth.get_password
def get_password(username):
if username == 'x':
return 'x'
return None
app = Flask(__name__)
cors = CORS(app)
api = Api(app)
class Report(Resource):
decorators = [auth.login_required]
def get(self):
parser = reqparse.RequestParser()
parser.add_argument('start', type = str)
parser.add_argument('end', type = str)
args = parser.parse_args()
conn = e.connect()
stat = """
select a, b from report where c < ? and d > ?
"""
query = conn.execute(stat, [args['start'], args['end']])
json_dict = []
for i in query.cursor.fetchall():
res = {'aa': i[0], 'bb':i[1]}
json_dict.append(res)
conn.close()
return jsonify(results=json_dict)
api.add_resource(Report, '/report')
if __name__ == '__main__':
app.run(host='0.0.0.0')
I've tried to debug the issue and following are my observations:
1) Flask API is running on port 5000 and when I psping the VM on port 5000 I'm able to connect which means the process is actually running properly on the VM.
2) On checking my logs, the GET requests are not even being received by the API. If there was some db error then I'd have gotten a 500 error message but the requests are not even going to the API at the first place.
3) If I call the API locally then still the issue persists.
4) If I do a netstat for port 5000 (where my flask API is running on) I'm getting the following:
For some reason I think its not closing socket connections. I'm getting lots of "CLOSE_WAIT". Is this what is causing the problem? How can I fix this in my code?
Usually, when you get lots CLOSE_WAIT status, it means there are socket connections unclosed. And It seems that you have found the answer at Flask / Werkzeug - sockets stuck in CLOSE_WAIT , which leverages Tornado http://flask.pocoo.org/docs/0.10/deploying/wsgi-standalone/#tornado to build a non-blocking web server.
Related
Is it possible to use a secret key to secure just an API without a website or webpage?
I made an app that uses flask and when I test it from the client app, it works. However I want to secure the get request from the client to the server by using a secret key or token if possible.
The problem is that most examples I have seen assumed you are using this for a website with login credentials. I don't have any webpages or any routes in my flask app.
Here is the server side:
from flask import Flask, stream_with_context, request, Response
from flask_restful import Api, Resource
from flask_socketio import SocketIO
import intermedia_choose_action_flask
import subprocess
from io import StringIO
import sys
import sqlite_companysearch
import time
app = Flask(__name__)
api = Api(app)
SECRET_KEY = "a long set of strings I got from running a command in linux terminal"
app.secret_key = SECRET_KEY
class addspamblacklistspecific(Resource):
def get(self, emailordomain):
count = 0
sqlite_companysearch.comp_searchall_intermedia()
selection_cid = sqlite_companysearch.comp_searchall_intermedia.cid_selection_results
for cid in selection_cid:
subprocess.Popen(["python3", "/home/tech/scripts/Intermedia_automate/intermedia_choose_action.py", "--addblockspecific", "--cp", cid, "--ed", emailordomain], bufsize=10, errors='replace')
count = count + 1
if count == 3:
time.sleep(60)
count = 0
return "command completed succesfully"
api.add_resource(addspamblacklistspecific, "/addspamblacklistspecific/<string:emailordomain>")
if __name__ == "__main__":
app.run(debug=True)
Here is the client side:
from flask import json
import requests
#where do I put in a secret key?
def email_or_domain_selection():
email_or_domain_selection.email_select = input("""Enter an email or domain.
(NOTE: domains must have a "*" symbol infront of the name. For example *company.com)
Enter Email or Domain :""")
eselect = email_or_domain_selection.email_select
return email_or_domain_selection.email_select
email_or_domain_selection()
BASE = "http://127.0.0.1:5000/"
response = requests.get(BASE + "addspamblacklistspecific/"+email_or_domain_selection.email_select)
print(response.text)
I figure I should learn this before learning how to put my app in the cloud.
Thank you for your time,
Edit - I was told to read this: demystify Flask app.secret_key which I already did. That is for if you have webpages. I don't have any webpages and am just trying to secure an API only. It doesn't explain how or if I should be using session information for just calling an api from a client side. It doesn't explain how to use a secret key on the client side.
You could look into flask-httpauth. I used this a while back on one of my projects to add a layer of security to some API's running on flask. Keep in mind that this is only a basic authentication (base-64 encoded strings).
from flask import Flask, jsonify, request
from flask_restful import Resource, Api
from flask_httpauth import HTTPBasicAuth
# import credentials from env (suggested)
API_UNAME = "username"
API_PASS = "password"
USER_DATA = {API_UNAME: API_PASS}
# initialize flask/flask-restful instance
app = Flask(__name__)
api = Api(app)
auth = HTTPBasicAuth()
class API_Class(Resource):
#auth.login_required
def post(self):
# do api stuff
return jsonify("result")
# verify API authentication
#auth.verify_password
def verify(username, password):
if not (username and password):
return False
return USER_DATA.get(username) == password
api.add_resource(API_Class, "/post")
You might want to look into other methods like OAuth for extra security.
I am learning flask and PyMongo right now and came across ChangeStreams. I do understand how ChangeStreams work but I have only worked with them in Node and Express. I have implemented ChangeStreams in my Flask app as following:
with ms.db.collection.watch() as stream:
for change in stream:
print(change)
On the official docs pages, it says that it's a blocking method. But how would I go about in making it non-blocking? Because currently my ChangeStream logic is in a different file and I import it into the server.py file. So when it never goes past that import and the Flask App doesn't start at all. Below is my server.py
from flask import Flask, render_template, request
import mongo_starter as ms
import changestream as cs
app = Flask(__name__)
#app.route('/')
def home():
return render_template('index.html')
if __name__ == "__main__":
app.run(host="0.0.0.0", port="5000")
Below is my ChangeStream.py
import mongo_starter as ms
with ms.db.collection.watch() as stream:
for change in stream:
print(change)
Below is my MongoStarter.py that actually initiates the connection to Mongo
import pymongo
import mongo_config as mc
print(mc.data_header)
try:
print('Connecting to Database...')
mongo_client = pymongo.MongoClient(mc.mongo_url)
db = mongo_client['PyMongo']
collection = db['Test Data']
print("Connection to Database Successful!")
except pymongo.errors.InvalidURI:
print('Error Connecting to Database')
When I run the app using nodemon it prints the following to the output.
[nodemon] restarting due to changes...
[nodemon] starting `python server.py`
----------------- MONGO CONNECTION LOG --------------------
Connecting to Database...
Connection to Database Successful!
So it never actually goes past the change stream method. How can I make it so it worked in an async way? I have looked at asyncio, but wanted to see if there was any way to implement it without using asyncio.
I've used perfect Flask-SocketIO library with Python 3 for couple of months. Everything worked as expected until the last couple of days.
All works fine, if namespace for connection to websocket server stay default /socket.io. But I'm geting an error now if I trying to change namespace for connection to python flask-socketio backend.
My app.py:
from flask import Flask, session, request, render_template, copy_current_request_context
from flask_cors import CORS, cross_origin
import flask_socketio as socketio
import ujson
async_mode = 'threading'
namespace = '/mynamespace'
app = Flask(__name__)
CORS(app)
app.config['SECRET_KEY'] = 'secret!'
sio = socketio.SocketIO(app, async_mode=async_mode)
#sio.on('connect', namespace=namespace)
def connect():
logging.info('Connected')
#sio.on('disconnect', namespace=namespace)
def disconnect():
logging.info('Disconnected')
#app.route("/home/index")
def home():
return render_template('index.html',
async_mode=sio.async_mode)
I'm using ./main.py to run the server, main.py contains:
from app import app, sio
if __name__ == "__main__":
sio.run(app, debug=False)
My template/index.html contains ton of code js, but I think most valuable I loading the socketio from cdn in a head:
<script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/socket.io/1.3.5/socket.io.min.js"></script>
... and I using connect with custom namespace path, as in docs:
namespace = '/mynamespace/socket.io';
var socket = io.connect(location.protocol + '//' + document.domain + ':' + location.port,
{path: namespace});
socket.on('connect', function() {
socket.emit('Connected to server');
})
As I understand, By default socketio library trying to connect to backend with emitting connect message to namespace. During loading 'index.html' template on '/home/index' route, logging the errors to console:
Flask server also gives and 404 error:
My best guess: at this moment it looks like something changed in client-side JS library or in chrome browser itself(few days ago I updated Chrome).
Maybe I just understood wrong one small detail. I really appreciate some help with this problem.
Stack versions:
Python 3.7.2,
Flask 1.0.2,
Flask-SocketIO 3.3.1,
socketio.min.js
1.3.5,
Google Chrome 77.0.3865.90 (64 bit)
You are confusing namespace with path, which are completely different things. The path is the endpoint URL where the Socket.IO server is listening. The namespace is a protocol feature of Socket.IO that allows multiplexing of multiple logical connections into a single physical connection.
I need help in debugging -the Same Origin Policy disallows reading the remote resource at https://some-domain.com. (Reason: CORS request did not succeed) in python flask-socketio error.
I am working on a chat application using python flask-socketio. In previously I have created that application in local and it works fine as expected, while I move the below code to the server it shows the above error. The client code runs in the https servers and server code also runs on the https server I don't know why that error shows.
I have attached my code below and please give a better solution to me.
server.py
import json
import os
from flask import Flask, render_template, request,session
from flask_socketio import SocketIO, send, emit
from datetime import timedelta,datetime
from flask_cors import CORS
app = Flask(__name__)
app.config['SECRET_KEY'] = 'secretkey'
app.config['DEBUG'] = True
app.config['CORS_HEADERS'] = 'Content-Type'
cors = CORS(app, resources={r"/*": {"origins": "*"}})
socketio = SocketIO(app)
users = {}
#app.before_request
def make_session_permanent():
session.permanent = True
app.permanent_session_lifetime = timedelta(minutes=1)
#app.route('/')
##cross_origin(origin='*',headers=['Content- Type','Authorization'])
def index():
return render_template('index.html')
#socketio.on('connect')
def connect():
print("connected");
#app.route('/orginate')
def orginate():
socketio.emit('server orginated', 'Something happened on the server!')
return '<h1>Sent!</h1>'
#socketio.on('username')
def receive_username(username):
users[username] = request.sid
#users.append({username : request.sid})
#print(users)
emit('userList', users, broadcast=True)
print('Username added!')
print(users)
if _name_ == '__main__':
socketio.run(app,host='xxx.xxx.xx.x',port=5001)
client.js
var socket = io.connect("https://xxx.xxx.xx.x:5001/",{secure:false});
Screenshot 1:
This screenshot explains the access-control-allow-orgin works fine for images under static folder in flask framework
Screenshot 2:
This screenshot explains there is no access-control-orgin for socket call
You are using Flask-CORS to set up CORS on your Flask routes. You are missing a similar set up for Flask-SocketIO:
socketio = SocketIO(app, cors_allowed_origins=your_origins_here)
You can use '*' as the value to allow all origins (which I do not recommend), or set a single origin as a string, or a list of origins as a list of strings.
I have a flask app. I want the client-server connection to terminate if the server does not respond within a stipulated time (say 20 seconds). I read here that the session.permanent = True can be set. I am a bit unclear where this goes in the server side code (if at all this is the way??).
For simplicity I am including the minimal server side code I have. Actually the server is performing a File Read/Write operation and returning a result to the client.
from flask import Flask, session, app
from flask_restful import Api, Resource
from datetime import timedelta
app = Flask(__name__)
api = Api(app)
class GetParams(Resource):
def get(self):
print ("Hello.")
return 'OK'
api.add_resource(GetParams, '/data')
if __name__ == '__main__':
app.run(host='127.0.0.1', port=5002)
Can anyone tell me what should I do here so that the connection between my client and server is terminated if the server does not respond i.e., send data back to the client within 20 seconds?
Long running tasks should be dealt with in a different design because, if you allow your server to keep a request alive for 50 minutes, you can't force user browser to do so.
I would recommend implementing the long running task as a thread that notifies the user once it's done.
For more readings about the problem statement and suggested solutions:
timeout issue with chrome and flask
long request time patterns
I believe that the only thing you need is to put your connexion statement in a try/except block. So that you will be able to handle any kind of connexion error.
Furthermore, a session timeout and a connexion fail/unreachable server are different things. A session timeout disconnect a user from a server which is here for too long (usually used to avoid a user to forgot a session open). Whereas when a server is unreachable the user isn't connected so there is no session timeout.
from flask import Flask, session, app
from flask_restful import Api, Resource
from datetime import timedelta
app = Flask(__name__)
api = Api(app)
class GetParams(Resource):
def get(self):
print ("Hello.")
return 'OK'
api.add_resource(GetParams, '/data')
if __name__ == '__main__':
try:
app.run(host='130.0.1.1', port=5002)
except:
print("unexcepted error")
you could qualify the received exception, but you'll have to read a bit of doc http://flask.pocoo.org/docs/1.0/quickstart/#what-to-do-if-the-server-does-not-start