I am trying to have a Flask Server that allows me to launch a tweepy stream and on every message received in the stream listener, it sends that message to a socketio client. The Flask server is at the same time supposed to allow Twilio to post to it, and route that message to the client—so that the client is receiving messages from both Twilio and twitter.
I have been trying to get the server to send messages over to the client for the data incoming from twitter, the code for Twilio works just fine. It sends data over to the client on message receipt. The main loop in tweepy is also not locking up the program—I can test print statements and see tweets and the incoming sms's being printed in the handle_message(msg) function asynchronously. I feel like there must be something really simple that I am missing here since the SMS's are emitted to the client, but the incoming tweets are not, even though they are propagating through to the handle_message(msg) function. What gives?
server.py
from flask import Flask, json, request
from twilio.twiml.messaging_response import Message, MessagingResponse
from flask_socketio import SocketIO
import tweepy
import json
PATH = '/path/to/credentials/'
with open(PATH, "r") as file:
credentials = json.load(file)
app = Flask(__name__)
app.debug = True
app.config['SECRET_KEY'] = 'abc123'
sio = SocketIO(app, cors_allowed_origins="*")
auth = tweepy.OAuthHandler(credentials['CONSUMER_KEY'], credentials['CONSUMER_SECRET'])
auth.set_access_token(credentials['ACCESS_TOKEN'], credentials['ACCESS_SECRET'])
api = tweepy.API(auth)
class MyListener(tweepy.StreamListener):
def on_status(self, status):
print('status')
def on_data(self, data):
handle_message(data)
def on_error(self, status):
print('error')
print(status)
stream_listener = MyListener()
# twilio sms route
#app.route('/sms', methods=['POST'])
def sms():
number = request.form['From']
message_body = request.form['Body']
message_data = {"number": number, "msg": message_body}
resp = MessagingResponse()
resp.message('Hello {}, you said: {}'.format(number, message_body))
handle_message(message_data)
return str(resp)
# flask-socketio stuff
#sio.on('connect')
def connect():
print('connected')
sio.emit('client_connected', "you connected")
search_term = "#mysearchterm"
stream = tweepy.Stream(auth=api.auth, listener=stream_listener)
stream.filter(track=[search_term], is_async=True)
sio.emit('client_connected', "the search term is {}".format(search_term))
#sio.on('disconnect')
def disconnect():
print('Client Diconnected')
#sio.event
def handle_message(message):
print("This is the message received: ", message)
sio.emit('handle_message', message)
if __name__ == '__main__':
sio.run(app)
client.py
import socketio
client = socketio.Client()
#client.on('client_connected')
def on_connect(message):
print(message)
#client.on('handle_message')
def message(data):
print(data)
client.connect('http://localhost:5000/')
Twilio developer evangelist here.
You've decorated the handle_message function as #sio.event but as far as I can see in the docs, you should only do that to have the handle_message method respond to events on the socket called "handle_message".
I'd start be removing the #sio.event decorator.
I'm not a Python expert, but I also wonder whether there is a scope issue here. You define your MyListener class and create an instance of it before you define the handle_message method. Just to test, can you try emitting to the socket directly within the on_data method:
def on_data(self, data):
sio.emit('handle_message', data)
If that works, consider moving the definition of handle_message above the definition of MyListener.
I solved my problem! As I noted in this comment, the issue was with multithreading and passing information between the threads. With tweepy, the parameter is_async=True, which in 4.1.0 is threading=True, opens up a new thread once the stream is run.
Instead of trying to deal with passing information around, I exploited the extant flask-socketio functionality by using a local redis server as a message queue (start from the section "Using Multiple Workers" if you are setting this up for the first time, also be sure to install redis).
Here is the updated server.py code. The client.py code remained essentially unchanged:
import eventlet
eventlet.monkey_patch()
from flask import Flask, json, request
from twilio.twiml.messaging_response import Message, MessagingResponse
from flask_socketio import SocketIO
import tweepy
import json
PATH = '/PATH/TO/CREDENTIALS'
with open(PATH, "r") as file:
credentials = json.load(file)
app = Flask(__name__)
app.debug = True
app.config['SECRET_KEY'] = 'abc123'
sio = SocketIO(app, message_queue='redis://', cors_allowed_origins="*")
class MyStream(tweepy.Stream):
def __init__(self, consumer_key, consumer_secret, access_token, access_secret):
super(MyStream, self).__init__(consumer_key, consumer_secret, access_token, access_secret)
self.stream_sio = SocketIO(message_queue='redis://')
def on_status(self, status):
print('status')
def on_data(self, data):
json_data = json.loads(data)
self.stream_sio.emit('handle_message', json_data['text'])
# TODO: Send along all necessary information
#app.route('/sms', methods=['POST'])
def sms():
number = request.form['From']
message_body = request.form['Body']
message_data = {"number": number, "msg": message_body}
resp = MessagingResponse()
resp.message('Hello {}, you said: {}'.format(number, message_body))
handle_message(message_data)
return str(resp)
#sio.on('connect')
def connect():
print('connected')
sio.emit('client_connected', "you connected")
search_term = "#testingtesting123"
stream = MyStream(credentials['CONSUMER_KEY'], credentials['CONSUMER_SECRET'],
credentials['ACCESS_TOKEN'], credentials['ACCESS_SECRET'])
stream.filter(track=[search_term], threaded=True)
sio.emit('client_connected', "the search term is {}".format(search_term))
#sio.on('disconnect')
def disconnect():
print('Client disconnected')
def handle_message(message):
sio.emit('handle_message', message)
if __name__ == '__main__':
sio.run(app)
Related
I am trying to implement a client-server using tornado_http2 api in python but server never receive messages from the client.
I have checked that server is well started with this comm
and and I had this result:
(mmsx-TPjM8MGB-py3.9) xx#ITLP071: 7 (master) ~/dev/mmsx/tornado_http2/demo$ proxy=127.0.0.1:8443; curl --http2-prior-knowledge -d "bla bla" -X POST https://localhost:8443/ -E test.crt
curl: (60) SSL certificate problem: self signed certificate
More details here: https://curl.se/docs/sslcerts.html
curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
And from the output server :
(mmsx-TPjM8MGB-py3.9) xx#ITLP071: 130 (master) ~/dev/mmsx/tornado_http2/demo$ poetry run python server_test.py
[I 220722 04:02:37 server_test:30] starting
[W 220722 04:02:41 iostream:1517] SSL Error on 7 ('127.0.0.1', 60040): [SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:1123)
The connection is not perfectly done (that I do not succed to resolve for now) but at least I have a reaction from the server.
With request from the client, I have no response.
Please find my server code below:
import logging
import os
import ssl
from tornado.ioloop import IOLoop
from tornado.options import parse_command_line
from tornado.web import Application, RequestHandler
from tornado_http2.server import Server
class MainHandler(RequestHandler):
def get(self):
self.write("Hello world")
def post(self):
self.write("bla bla")
def main():
parse_command_line()
ssl_ctx = ssl.SSLContext(ssl.PROTOCOL_SSLv23)
ssl_ctx.load_cert_chain(
os.path.join(os.path.dirname(__file__), 'test.crt'),
os.path.join(os.path.dirname(__file__), 'test.key'))
app = Application([('/hello', MainHandler)], debug=True)
server = Server(app, ssl_options=ssl_ctx)
port = 8443
address = "127.0.0.1"
server.listen(port, address)
logging.info("starting")
IOLoop.instance().start()
if __name__ == '__main__':
main()
And my client code:
from tornado_http2.curl import CurlAsyncHTTP2Client as HTTP2Client
import asyncio
URI = "http:127.0.0.1:8443/hello"
class Test():
def __init__(self):
self.__client = HTTP2Client(force_instance=True)
async def send(self):
global URI
body = "body"
response = await self.__client.fetch(URI, method='POST', body=body,
validate_cert=False)
print(response)
def main():
asyncio.run(Test().send())
if __name__ == "__main__":
main()
I started the server in a terminal and then the client in another one and for me, it should displayed in the client console the result of the request.
Thanks for your help !
OK, I have found.
It is a bug in tornado_http2 api. The event loop has to be created before the instanciation of the class HTTP2Client, else this does not work.
If the client code is remplaced bu this, it will work :
from tornado_http2.curl import CurlAsyncHTTP2Client as HTTP2Client
import asyncio
from tornado.httpclient import AsyncHTTPClient
from tornado.ioloop import IOLoop
class Test():
def __init__(self):
self.__client = HTTP2Client(force_instance=True)
async def send(self):
uri = "https://127.0.0.1:8443/hello"
response = await self.__client.fetch(uri, validate_cert=False)
print(response.body.decode('utf-8'))
def run_asyncio():
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
try:
return loop.run_until_complete(Test().send())
finally:
loop.close()
asyncio.set_event_loop(None)
def main():
run_asyncio()
if __name__ == "__main__":
main()
Hopefully it will help someone =).
I am trying to send data from server to flutter app using socketIO. Although I am able to connect and emit, the server is not able to send data to client side.
Server side code:
import cv2
import numpy as np
from flask import Flask, render_template
from flask_socketio import SocketIO, emit
from threading import Lock,Timer as tmr
from engineio.payload import Payload
import base64
import io
app = Flask(__name__)
app.config['SECRET_KEY'] = 'secret!'
socketio = SocketIO(app)
someList = ['apple', 'peas', 'juice','orange']
i=0
#socketio.on('connect')
def connect():
print("a client connected")
#socketio.on('disconnect')
def disconnect():
print('Client disconnected')
#socketio.on('msg')
def handlemsg(msg):
print (msg)
socketio.send("msg from server")
#app.route('/')
def hello():
return "hii"
if __name__ == '__main__':
socketio.run(app,host= '0.0.0.0')
Client side (flutter)
#override
void initState() {
super.initState();
IO.Socket socket = IO.io('http://x.x.x.x:5000', <String, dynamic>{
'transports': ['websocket', 'polling']});
socket.connect();
socket.emit('msg', 'test');
socket.onConnect((_) {
print('connect');
socket.emit('msg', 'testing');
});
socket.onDisconnect((_) => print('disconnect'));
socket.on('*', (data) => print(data)); //nothing is printed
}
The result I get on the server-side:
a client connected
testing
However, I get no data on the client side. Where am I going wrong? Please help
I can't test it with flutter but I tested it with client create with python-socketio
Main problem can be that send() sends message with name "message" like emit("message", ...) but your on("msg", ...) expects message with name "msg", not "message".
So you should use emit("msg", ...) in Python and on("msg", ...) in flutter.
Or you should use send() in Python and on("message", ...) in flutter.
Other problem can be that it may need some time to send message and receive it - and it may need extra time after connecting and extra time befor disconnecting - at least in my example I had to sleep to get results.
Full working code.
I added more emit() with different names.
server.py
from flask import Flask
from flask_socketio import SocketIO, emit
app = Flask(__name__)
app.config['SECRET_KEY'] = 'secret!'
socketio = SocketIO(app)
#socketio.on('connect')
def connect():
print("client connected")
#socketio.on('disconnect')
def disconnect():
print('client disconnected')
#socketio.on('question')
def handle_questio(msg):
print('question msg:', msg)
socketio.emit("answer", "msg from server")
#socketio.on('help')
def handle_help(msg):
print('help msg:', msg)
socketio.emit("support", "help from server")
#app.route('/')
def hello():
return "hii"
if __name__ == '__main__':
print('start')
socketio.run(app, host='0.0.0.0')
client.py
import socketio
sio = socketio.Client()
#sio.on('connect')
def connect():
print('connected')
#sio.on('disconnect')
def disconnect():
print('disconnected')
#sio.on('answer')
def answer(data):
print('answer:', data)
#sio.on('support')
def support(data):
print('support:', data)
# --- main ---
print('start')
sio.connect('http://localhost:5000')
print('sleep')
sio.sleep(1)
print('emit question')
sio.emit('question', {'foo': 'bar'})
print('emit help')
sio.emit('help', 'can you help me')
print('sleep')
sio.sleep(1)
sio.disconnect()
I am creating an API to send message to echo server
For this, I am using echo chat server URL ws://echo.websocket.org/ from this site https://www.websocket.org/echo.html. This connection will echo our input.
First time when the user request the API, I need to establish connection with echo server and send the user message to echo server. When second time user request the same API, this time connection is already established. So, I just need to send user message to echo server.
For this, I am using python session to store the connection details. First time when the connection in made I am trying to save it in session. session['ConnetionMade'] = "True" by default it is false
So, second time when the user request the API, this time ConnetionMade is True. So, I don't to make connection again.
But here session variable is not updated when the connection is made. It is always false. But we set to True.
Below is the full working code. Kindly help me to update session variable.
Note : Session variable works when we skips socket connection code
from flask import Flask
from flask import request, session
from config import config
import websocket
try:
import thread
except ImportError:
import _thread as thread
SECRET_KEY = 'a secret key'
app = Flask(__name__)
app.config.from_object(__name__)
#app.route('/')
def root():
return 'Hello NLP....!'
userMessage = ""
#app.route('/whatsapp', methods=['POST'])
def echo():
global userMessage
userMessage = request.json['message']
# session.clear()
print("\n\nuserMessage: ", userMessage, "\n\n")
print("ConnetionMade--: ", session.get('ConnetionMade', "False"))
if session.get('ConnetionMade', "False") == "False":
session['ConnetionMade'] = "True"
print('True set to ConnetionMade ', session.get('ConnetionMade', "False"))
echoConnection()
else:
session['ConnetionMade'] = "False"
print('False set to ConnetionMade ', session.get('ConnetionMade', "False"))
return ""
def echoConnection():
if __name__ == "__main__":
websocket.enableTrace(True)
ws = websocket.WebSocketApp("ws://echo.websocket.org/",
on_open = on_open,
on_message = on_message,
on_error = on_error,
on_close = on_close)
ws.run_forever()
return ""
def on_message(ws, message):
print("\n\nMessage received from Echo Socket server:", message, '\n\n')
return
def on_error(ws, error):
print("on_error ", error)
return
def on_close(ws):
print("on_close")
return
def on_open(ws):
def run(*args):
print("\n\nSocket connection made. Now sending this message ("+userMessage+") to Echo Socket server\n\n")
ws.send(userMessage)
print("\nsent...\n")
print("thread terminating...")
thread.start_new_thread(run, ())
return
if __name__ == "__main__":
app.run(host='0.0.0.0', port=config['server']['port'])
In your case, the ws.run_forever() blocks the thread and then the further call's to the API doesn't get captured.
You can run the websocket in a deamon thread and ensure to use its send method to communicate with the websocket server.
Something like this:
from flask import Flask
from flask import request, session
import websocket
import threading
SECRET_KEY = 'a secret key'
app = Flask(__name__)
app.config.from_object(__name__)
websocket_client = None
#app.route('/')
def root():
return 'Hello NLP....!'
#app.route('/whatsapp', methods=['POST'])
def echo():
userMessage = request.json['message']
# session.clear()
print("userMessage: ", userMessage, "\n")
print("ConnetionMade--: ", session.get('ConnetionMade', "False"))
if session.get('ConnetionMade', "False") == "False":
session['ConnetionMade'] = "True"
print('True set to ConnetionMade ', session.get('ConnetionMade', "False"))
send_message_to_websocket_server(userMessage)
return ""
def send_message_to_websocket_server(message):
print("Sending message to WebSocket Server")
websocket_client.send(message)
def createConnection():
ws = websocket.WebSocketApp("ws://echo.websocket.org/",
on_open = on_open,
on_message = on_message,
on_error = on_error,
on_close = on_close)
# Initialise the run_forever inside a thread and make this thread as a daemon thread
wst = threading.Thread(target=ws.run_forever)
wst.daemon = True
wst.start()
return ws
def on_message(ws, message):
print("Message received from Echo Socket server:", message)
def on_error(ws, error):
print("on_error ", error)
def on_close(ws):
print("on_close")
def on_open(ws):
print("Socket connection opened")
if __name__ == "__main__":
websocket_client = createConnection()
app.run(host='0.0.0.0', port=8001) # I have hardcoded the port to test this
Now, using Curl if I take session into account and hit it with the following command, it only sends an echo to the websocket server if the session is new:
curl -s -i -X POST http://0.0.0.0:8001/whatsapp\
-d '{"message":"Sample"}' -H "Content-Type: application/json"\
--cookie cookie.txt --cookie-jar cookie.txt
It will always take the latest message and echo it to the console.
I'm trying to make a Flask app that uses WebSockets. The example from Flask-sockets works but how would I send a message from a regular view?
Similarly to how Flask-SocketIO use .emit() and .send()-methods.
In the example below (from the Flask-Sockets example) I would for instance like to be able to broadcast a message from the hello-view.
from flask import Flask
from flask_sockets import Sockets
app = Flask(__name__)
sockets = Sockets(app)
#sockets.route('/echo')
def echo_socket(ws):
while not ws.closed:
message = ws.receive()
ws.send(message)
#app.route('/')
def hello():
# How can I send a WebSocket message from here?
return 'Hello World!'
if __name__ == "__main__":
from gevent import pywsgi
from geventwebsocket.handler import WebSocketHandler
server = pywsgi.WSGIServer(('', 5000), app, handler_class=WebSocketHandler)
server.serve_forever()
You can use a global socket list of all client. Traverse all list and send message to all ws instance.
Example code;
from flask import Flask, render_template
from flask_sockets import Sockets
app = Flask(__name__)
sockets = Sockets(app)
ws_list = []
#sockets.route('/echo')
def echo_socket(ws):
ws_list.append(ws)
while not ws.closed:
message = ws.receive()
ws.send(message)
#app.route('/')
def hello():
# How can I send a WebSocket message from here?
return render_template('index.html')
#app.route('/send_message_to_all_client')
def broadcast():
for ws in ws_list:
if not ws.closed:
ws.send("broadcast message")
else:
# Remove ws if connection closed.
ws_list.remove(ws)
return "ok"
if __name__ == "__main__":
from gevent import pywsgi
from geventwebsocket.handler import WebSocketHandler
server = pywsgi.WSGIServer(('', 5000), app, handler_class=WebSocketHandler)
server.serve_forever()
Im trying to programming a simplest client-server application with SSL, and i have some issues with this:
1) How i can decrease time for serialization at JSON?
2) Maybe something exists for better than LineReciver, for creating communication between server and client? Or i can increase length of received lines?
Source code:
a) ServerSLL
import server
from twisted.internet.protocol import Factory
from twisted.internet import reactor
from OpenSSL import SSL
class ServerSSL(object):
def getContext(self):
ctx = SSL.Context(SSL.SSLv23_METHOD)
ctx.use_certificate_file('server_cert.pem')
ctx.use_privatekey_file('server_key.pem')
return ctx
if __name__ == '__main__':
factory = Factory()
factory.protocol = server.Server
reactor.listenSSL(8000, factory, ServerSSL())
reactor.run()
b) Server
from json import dumps, loads
import sqlalchemy
from sqlalchemy.orm import sessionmaker
from db.create_db import Users
from twisted.internet.protocol import Protocol, Factory
from twisted.internet import reactor
engine = sqlalchemy.create_engine('postgresql://user:test#localhost/csan', pool_size=20, max_overflow=0)
class Server(Protocol):
def __init__(self):
self.Session = sessionmaker(bind=engine)
def __del__(self):
self.session.close()
def authorization(self, data):
"""
Checking user with DB
"""
session = self.Session()
result = session.execute(sqlalchemy.select([Users]).where(Users.name == data['user']))
result = result.fetchone()
if result is None:
data['error'] = 404
else:
if result['name'] == data['user']:
# correct users info --> real user
if result['password'] == data['pswd']:
data['auth'] = 1
# incorrect password --> fake user
else:
data['error'] = 403
session.close()
return data
def dataReceived(self, data):
"""
Processing request from user and send response
"""
new_data = loads(data)
if new_data['cmd'] == 'AUTH':
response = self.authorization(new_data)
self.transport.write(str(dumps(new_data)))
if __name__ == '__main__':
f = Factory()
f.protocol = Server
reactor.listenTCP(8000, f)
reactor.run()
c) client_console
from json import dumps, loads
from twisted.internet.protocol import ClientFactory
from twisted.protocols.basic import LineReceiver
from twisted.internet import ssl, reactor
class ServerClientSSL(LineReceiver):
"""
Basic client for talking with server under SSL
"""
def connectionMade(self):
"""
Send auth request to serverSLL.py
"""
login = raw_input('Login:')
password = raw_input('Password:')
hash_password = str(hash(password))
data = dumps({'cmd': 'AUTH', 'user': login, 'pswd': hash_password, 'auth': 0, 'error': 0})
self.sendLine(str(data))
def connectionLost(self, reason):
"""
Says to client, why we are close connection
"""
print 'connection lost (protocol)'
def lineReceived(self, data):
"""
Processing responses from serverSSL.py and send new requests to there
"""
new_data = loads(data)
if new_data['cmd'] == 'BBYE':
self.transport.loseConnection()
else:
print new_data
class ServerClientSLLFactory(ClientFactory):
protocol = ServerClientSSL
def clientConnectionFailed(self, connector, reason):
print 'connection failed:', reason.getErrorMessage()
reactor.stop()
def clientConnectionLost(self, connector, reason):
print 'connection lost:', reason.getErrorMessage()
reactor.stop()
if __name__ == '__main__':
import sys
if len(sys.argv) < 3:
print 'Using: python client_console.py [IP] [PORT] '
else:
ip = sys.argv[1]
port = sys.argv[2]
factory = ServerClientSLLFactory()
reactor.connectSSL(ip, int(port), factory, ssl.ClientContextFactory())
reactor.run()
class ServerSSL(object):
...
Don't write your own context factory. Use twisted.internet.ssl.CertificateOptions instead. It has fewer problems than what you have here.
def __del__(self):
self.session.close()
First rule of __del__: don't use __del__. Adding this method doesn't give you automatic session cleanup. Instead, it almost certainly guarantees your session will never be be cleaned up. Protocols have a method that gets called when they're done - it's called connectionLost. Use that instead.
result = session.execute(sqlalchemy.select([Users]).where(Users.name == data['user']))
result = result.fetchone()
Twisted is a single-threaded multi-tasking system. These statements block on network I/O and database operations. While they're running your server isn't doing anything else.
Use twisted.enterprise.adbapi or twext.enterprise.adbapi2 or alchimiato perform database interactions asynchronously instead.
class ServerClientSSL(LineReceiver):
...
There are lots of protocols better than LineReceiver. The simplest improvement you can make is to switch to Int32StringReceiver. A more substantial improvement would be to switch to twisted.protocols.amp.
1) How i can decrease time for serialization at JSON?
Use a faster JSON library. After you fix the blocking database code in your application, though, I doubt you'll still need a faster JSON library.
2) Maybe something exists for better than LineReciver, for creating communication between server and client? Or i can increase length of received lines?
LineReceiver.MAX_LENGTH. After you switch to Int32StringReceiver or AMP you won't need this anymore though.