We Keep Coding

How to fix an error when trying to get the live stream of data from FastAPI using websockets (Unsupported upgrade request.)?

How can I get the response for every iteration, like a live stream feed?
Here is the RestAPI main.py:
from fastapi import FastAPI
from fastapi import Request
from fastapi import WebSocket
import asyncio
app = FastAPI()
#app.get("/ws_res")
async def websoc(websocket: WebSocket):
await websocket.accept()
for i in range(100000):
i = "John"
await asyncio.sleep(0.01)
await websocket.send_json({"msg": i})
await websocket.close()
Now, I am trying to get the response from python code but I'm receiving an error that says Unsupported upgrade request.
Here is the output from API side :
INFO: Started server process [67680]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
WARNING: Unsupported upgrade request.
Here is the python code that I used to access the API (test.py):
import asyncio
import websockets
async def hello():
uri = "ws://127.0.0.1:8000/ws_res"
async with websockets.connect(uri) as websocket:
greeting = await websocket.recv()
print(f"< {greeting['msg']}")
asyncio.get_event_loop().run_until_complete(hello())
Traceback (most recent call last):
File "test.py", line 12, in <module>
asyncio.get_event_loop().run_until_complete(hello())
File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
return future.result()
File "test.py", line 7, in hello
async with websockets.connect(uri) as websocket:
File "/home/test/env/lib/python3.8/site-packages/websockets/client.py", line 517, in __aenter__
return await self
File "/home/test/env/lib/python3.8/site-packages/websockets/client.py", line 542, in __await_impl__
await protocol.handshake(
File "/home/test/env/lib/python3.8/site-packages/websockets/client.py", line 296, in handshake
raise InvalidStatusCode(status_code)
websockets.exceptions.InvalidStatusCode: server rejected WebSocket connection: HTTP 400

Fastapi websocket endpoints need to be defined using the websocket decorator, not the get decorator:
#app.websocket("/ws_res")
async def websoc(websocket: WebSocket):
await websocket.accept()
for i in range(100000):
i = "John"
await asyncio.sleep(0.01)
await websocket.send_json({"msg": i})
await websocket.close()
See the documentation for full details: https://fastapi.tiangolo.com/advanced/websockets/

aiohttp ConnectionError: Cannot send requests while disconnected

I'm using aiohttp as server and telethon client for sending messages.
Running this edited code from telegrammodule.py is working fine:
from telethon import TelegramClient
import asyncio
api_id = censored
api_hash = "censored"
client = TelegramClient('anon', api_id, api_hash)
async def sendMessageToUser(user, message):
await client.send_message(user, message)
def sendMessage(user, message):
with client:
client.loop.run_until_complete(sendMessageToUser(user, message))
sendMessage("censored", "Hello")
This is the error I get when I try to call the function sendMessage() from server.py:
======== Running on http://0.0.0.0:5000 ========
(Press CTRL+C to quit)
Error handling request
Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/aiohttp/web_protocol.py", line 418, in start
resp = await task
File "/usr/local/lib/python3.7/dist-packages/aiohttp/web_app.py", line 458, in _handle
resp = await handler(request)
File "server.py", line 9, in send
response = await sendMessage(user, message)
File "/home/user/telegram_user_bot/telegrammodule.py", line 13, in sendMessage
client.loop.run_until_complete(sendMessageToUser(user, message))
File "/usr/lib/python3.7/asyncio/base_events.py", line 571, in run_until_complete
self.run_forever()
File "/usr/lib/python3.7/asyncio/base_events.py", line 526, in run_forever
raise RuntimeError('This event loop is already running')
RuntimeError: This event loop is already running
Task exception was never retrieved
future: <Task finished coro=<sendMessageToUser() done, defined at /home/user/telegram_user_bot/telegrammodule.py:8> exception=ConnectionError('Cannot send requests while disconnected')>
Traceback (most recent call last):
File "/home/user/telegram_user_bot/telegrammodule.py", line 9, in sendMessageToUser
await client.send_message(user, message)
File "/usr/local/lib/python3.7/dist-packages/telethon/client/messages.py", line 752, in send_message
result = await self(request)
File "/usr/local/lib/python3.7/dist-packages/telethon/client/users.py", line 53, in __call__
future = self._sender.send(request, ordered=ordered)
File "/usr/local/lib/python3.7/dist-packages/telethon/network/mtprotosender.py", line 170, in send
raise ConnectionError('Cannot send requests while disconnected')
ConnectionError: Cannot send requests while disconnected
Here's the code from server.py:
from telegrammodule import sendMessage
from aiohttp import web
import asyncio
async def send(request):
data = await request.post()
user = data['user']
message = data['message']
response = await sendMessage(user, message)
app = web.Application()
app.router.add_post("/send", send)
if __name__ == "__main__":
web.run_app(app, port=5000)
And here from telegrammodule.py:
from telethon import TelegramClient
import asyncio
api_id = censored
api_hash = "censored"
client = TelegramClient('anon', api_id, api_hash)
async def sendMessageToUser(user, message):
await client.send_message(user, message)
async def sendMessage(user, message):
async with client:
client.loop.run_until_complete(sendMessageToUser(user, message))
I don't know how to solve the problem. How can I fix it?

If a loop is already running, you can't start it one more time. Use create_task, to add a task to a running loop.
For your case
client.loop.create_task(sendMessageToUser(user, message))
would be a fix, I guess.
Also, if the client.loop is in another thread, you might use run_coroutine_threadsafe:
asyncio.run_coroutine_threadsafe(sendMessageToUser(user, message), client.loop)
And one more important thing - if client.loop is the loop you're currently in, you can await sendMessageToUser(user, message) until message is sent.

Handling websockets module exceptions and errors

I'm trying to create a simple WebSocket server using websockets module in Python.
My task is to create a WebSocket server that sends a message to all the clients that are listening to it. The problem is that everytime a client disconnects, websockets launches an exception (ConnectionClosedOK or ConnectionClosedError, depending of what happened) that I need to catch because I have to unregister the clients that have disconnected so that the WebSocket stops to send messages in the ether (and stopping the WebSocket server throwing these exceptions everytime that he tries to send a message to a closed client).
Here's the server's code:
async def foo_call():
await asyncio.sleep(3)
print('Hi!')
async def register(websocket):
users.add(websocket)
await notify_users()
async def unregister(websocket):
users.remove(websocket)
async def start(websocket, path):
await register(websocket)
try:
await foo_call()
except websockets.ConnectionClosedError:
await unregister(websocket)
# WebSocket clients that are listening this WebSocket server
users = set()
# Open WebSocket server
start_server = websockets.serve(start, "localhost", 8765)
# Run asyncio event loop
asyncio.get_event_loop().run_until_complete(start_server)
asyncio.get_event_loop().run_forever()
And here's the traceback:
Task exception was never retrieved
future: <Task finished coro=<WebSocketCommonProtocol.send() done, defined at /home/vlado/Projects/dahua/venv/lib/python3.6/site-packages/websockets/protocol.py:521> exception=ConnectionClosedError('code = 1006 (connection closed abnormally [internal]), no reason',)>
Traceback (most recent call last):
File "/home/vlado/Projects/dahua/venv/lib/python3.6/site-packages/websockets/protocol.py", line 827, in transfer_data
message = await self.read_message()
File "/home/vlado/Projects/dahua/venv/lib/python3.6/site-packages/websockets/protocol.py", line 895, in read_message
frame = await self.read_data_frame(max_size=self.max_size)
File "/home/vlado/Projects/dahua/venv/lib/python3.6/site-packages/websockets/protocol.py", line 971, in read_data_frame
frame = await self.read_frame(max_size)
File "/home/vlado/Projects/dahua/venv/lib/python3.6/site-packages/websockets/protocol.py", line 1051, in read_frame
extensions=self.extensions,
File "/home/vlado/Projects/dahua/venv/lib/python3.6/site-packages/websockets/framing.py", line 105, in read
data = await reader(2)
File "/usr/lib/python3.6/asyncio/streams.py", line 672, in readexactly
raise IncompleteReadError(incomplete, n)
asyncio.streams.IncompleteReadError: 0 bytes read on a total of 2 expected bytes
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/vlado/Projects/dahua/venv/lib/python3.6/site-packages/websockets/protocol.py", line 555, in send
await self.ensure_open()
File "/home/vlado/Projects/dahua/venv/lib/python3.6/site-packages/websockets/protocol.py", line 803, in ensure_open
raise self.connection_closed_exc()
websockets.exceptions.ConnectionClosedError: code = 1006 (connection closed abnormally [internal]), no reason
I don't really understand how can I catch those exceptions thrown by websockets module.

asyncio client server does not work inside docker

I have client and server (send json messages over http) works ok on my standart ubuntu 16.04
But then I try to run it client and server inside docker or client outside docker and server inside i got errors.
My docker command:
sudo docker run -p 127.0.0.1:8888:8888 -i -t seo_server
Here is my server and client code and errors:
server
import asyncio
import json
import aiohttp
import re
async def get_cy(domain):
return {'result': 'ok','value':10}
async def handle(reader, writer):
# data = await reader.read(1000)
data = bytearray()
while not reader.at_eof():
chunk = await reader.read(2 ** 12)
data += chunk
# print(json.loads(data))
# https://aiomas.readthedocs.io/en/latest/
message = data.decode()
addr = writer.get_extra_info('peername')
print("Received %r from %r" % (message, addr))
m = json.loads(message)
f = m['method']
del m['method']
r = await globals()[f](**m)
r = json.dumps(r)
# await asyncio.sleep(int(m['time']))
print("Send: %r" % r)
writer.write(r.encode())
await writer.drain()
print("Close the client socket")
writer.close()
if __name__ == "__main__":
loop = asyncio.get_event_loop()
coro = asyncio.start_server(handle, '127.0.0.1', 8888, loop=loop)
server = loop.run_until_complete(coro)
# Serve requests until Ctrl+C is pressed
print('Serving cy microservice on {}'.format(server.sockets[0].getsockname()))
try:
loop.run_forever()
except KeyboardInterrupt:
pass
# Close the server
server.close()
loop.run_until_complete(server.wait_closed())
loop.close()
client
import asyncio
import json
from itertools import zip_longest
import time
def to_json(func):
def wrap(**kwargs):
message = kwargs
message['method'] = func.__name__
print(message)
return asyncio.ensure_future(tcp_send(json.dumps(message)))
return wrap
#to_json
def get_cy(domain):
pass
async def tcp_send(message):
loop = asyncio.get_event_loop()
reader, writer = await asyncio.open_connection('127.0.0.1', 8888,
loop=loop)
print('Send: %r' % message)
writer.write(message.encode())
writer.write_eof()
data = await reader.read()
data = data.decode()
print('Received: %r' % data)
print('Close the socket')
writer.close()
return json.loads(data)
def grouper(iterable, n, fillvalue=None):
"Collect data into fixed-length chunks or blocks"
# grouper('ABCDEFG', 3, 'x') --> ABC DEF Gxx"
args = [iter(iterable)] * n
return zip_longest(*args, fillvalue=fillvalue)
def async_map(loop, f, iterable, chunk_size=2):
for chunk in grouper(iterable, chunk_size):
future = asyncio.gather(*(f(param) for param in chunk if param))
loop.run_until_complete(future)
if __name__ == '__main__':
loop = asyncio.get_event_loop()
async def cy(site):
cy = await get_cy(domain=site)
print(site + " cy =", cy)
#update site here
while True:
sites = ('site1.ru', 'site2.ru', 'site3.ru', 'site4.ru', 'site5.ru')
async_map(loop, cy, sites)
time.sleep(100) #if not sites
error's then i try client and server inside docker:
client error
root#341fdee56d6d:/seo_server# python client.py
{'domain': 'site1.ru', 'method': 'get_cy'}
{'domain': 'site2.ru', 'method': 'get_cy'}
Send: '{"domain": "site2.ru", "method": "get_cy"}'
Send: '{"domain": "site1.ru", "method": "get_cy"}'
Received: ''
Close the socket
Traceback (most recent call last):
File "client.py", line 63, in <module>
async_map(loop, cy, sites)
File "client.py", line 49, in async_map
loop.run_until_complete(future)
File "/usr/local/lib/python3.5/asyncio/base_events.py", line 341, in run_until_complete
return future.result()
File "/usr/local/lib/python3.5/asyncio/futures.py", line 276, in result
raise self._exception
File "/usr/local/lib/python3.5/asyncio/tasks.py", line 236, in _step
result = coro.throw(exc)
File "client.py", line 57, in cy
cy = await get_cy(domain=site)
File "/usr/local/lib/python3.5/asyncio/futures.py", line 387, in __iter__
yield self # This tells Task to wait for completion.
File "/usr/local/lib/python3.5/asyncio/tasks.py", line 287, in _wakeup
value = future.result()
File "/usr/local/lib/python3.5/asyncio/futures.py", line 276, in result
raise self._exception
File "/usr/local/lib/python3.5/asyncio/tasks.py", line 238, in _step
result = coro.send(value)
File "client.py", line 36, in tcp_send
return json.loads(data)
File "/usr/local/lib/python3.5/json/__init__.py", line 319, in loads
return _default_decoder.decode(s)
File "/usr/local/lib/python3.5/json/decoder.py", line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/local/lib/python3.5/json/decoder.py", line 357, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Task was destroyed but it is pending!
task: <Task pending coro=<tcp_send() running at client.py:30> wait_for=<Future pending cb=[Task._wakeup()]> cb=[Task._wakeup()]>
server error
sudo docker run -p 127.0.0.1:8888:8888 -i -t seo_server
Serving cy microservice on ('127.0.0.1', 8888)
Received '{"domain": "site2.ru", "method": "get_cy"}' from ('127.0.0.1', 47768)
http://bar-navig.yandex.ru/u?ver=2&show=31&url=http://site2.ru
Received '{"domain": "site1.ru", "method": "get_cy"}' from ('127.0.0.1', 47770)
http://bar-navig.yandex.ru/u?ver=2&show=31&url=http://site1.ru
Send: '{"result": "ok", "value": "50"}'
Task exception was never retrieved
future: <Task finished coro=<handle() done, defined at seo_server.py:18> exception=ConnectionResetError('Connection lost',)>
Traceback (most recent call last):
File "/usr/local/lib/python3.5/asyncio/tasks.py", line 238, in _step
result = coro.send(value)
File "seo_server.py", line 40, in handle
await writer.drain()
File "/usr/local/lib/python3.5/asyncio/streams.py", line 304, in drain
yield from self._protocol._drain_helper()
File "/usr/local/lib/python3.5/asyncio/streams.py", line 195, in _drain_helper
raise ConnectionResetError('Connection lost')
ConnectionResetError: Connection lost
Send: '{"result": "ok", "value": "50"}'
Task exception was never retrieved
future: <Task finished coro=<handle() done, defined at seo_server.py:18> exception=ConnectionResetError('Connection lost',)>
Traceback (most recent call last):
File "/usr/local/lib/python3.5/asyncio/tasks.py", line 238, in _step
result = coro.send(value)
File "seo_server.py", line 40, in handle
await writer.drain()
File "/usr/local/lib/python3.5/asyncio/streams.py", line 304, in drain
yield from self._protocol._drain_helper()
File "/usr/local/lib/python3.5/asyncio/streams.py", line 195, in _drain_helper
raise ConnectionResetError('Connection lost')
ConnectionResetError: Connection lost
If i try client outside docker i got this client error and no actions at server:
outside client error
/usr/bin/python3.5 /home/se7en/examples/python_3.5/seo_server/client.py
{'method': 'get_cy', 'domain': 'site1.ru'}
{'method': 'get_cy', 'domain': 'site2.ru'}
Send: '{"method": "get_cy", "domain": "site1.ru"}'
Send: '{"method": "get_cy", "domain": "site2.ru"}'
Traceback (most recent call last):
File "/home/se7en/examples/python_3.5/seo_server/client.py", line 63, in <module>
async_map(loop, cy, sites)
File "/home/se7en/examples/python_3.5/seo_server/client.py", line 49, in async_map
loop.run_until_complete(future)
File "/usr/lib/python3.5/asyncio/base_events.py", line 373, in run_until_complete
return future.result()
File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
raise self._exception
File "/usr/lib/python3.5/asyncio/tasks.py", line 242, in _step
result = coro.throw(exc)
File "/home/se7en/examples/python_3.5/seo_server/client.py", line 57, in cy
cy = await get_cy(domain=site)
File "/usr/lib/python3.5/asyncio/futures.py", line 361, in __iter__
yield self # This tells Task to wait for completion.
File "/usr/lib/python3.5/asyncio/tasks.py", line 297, in _wakeup
future.result()
File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
raise self._exception
File "/usr/lib/python3.5/asyncio/tasks.py", line 240, in _step
result = coro.send(None)
File "/home/se7en/examples/python_3.5/seo_server/client.py", line 28, in tcp_send
writer.write_eof()
File "/usr/lib/python3.5/asyncio/streams.py", line 294, in write_eof
return self._transport.write_eof()
File "/usr/lib/python3.5/asyncio/selector_events.py", line 745, in write_eof
self._sock.shutdown(socket.SHUT_WR)
OSError: [Errno 107] Transport endpoint is not connected
my docker file:
FROM davidjfelix/python3.5
RUN pip3 install aiohttp
ADD . /seo_server
WORKDIR /seo_server
CMD python seo_server.py
docker version:
$ sudo docker version
Client:
Version: 1.11.2
API version: 1.23
Go version: go1.5.4
Git commit: b9f10c9
Built: Wed Jun 1 22:00:43 2016
OS/Arch: linux/amd64
Server:
Version: 1.11.2
API version: 1.23
Go version: go1.5.4
Git commit: b9f10c9
Built: Wed Jun 1 22:00:43 2016
OS/Arch: linux/amd64
Please help to find a problem and fix

Because of container isolation python threads in different container don't see each other. So for asyncio loop you need to start server and worker/client in the same container. You can do it via launch .sh script as described here https://docs.docker.com/config/containers/multi-service_container/
or same via Supervisord.

I had this same issue.
async server and async client outside docker refusing to connect,
this solved my problem: https://forums.docker.com/t/python-asyncio-container-is-not-receiving-socket-connections/34018
u need to change:
coro = asyncio.start_server(handle, '127.0.0.1', 8888, loop=loop)
to
coro = asyncio.start_server(handle, '0.0.0.0', 8888, loop=loop)
here quoting "currently you just bind the local interface."

Python asyncio: yield from wasn't used with future?

I'm attempting to use asyncio to make an asynchronous client/server setup.
For some reason I'm getting AssertionError: yield from wasn't used with future when running the client.
Searching for this error didn't turn up much.
What does this error mean and what's causing it?
#!/usr/bin/env python3
import asyncio
import pickle
import uuid
port = 9999
class ClientProtocol(asyncio.Protocol):
def __init__(self, loop):
self.loop = loop
self.conn = None
self.uuid = uuid.uuid4()
self.other_clients = []
def connection_made(self, transport):
print("Connected to server")
self.conn = transport
m = "hello"
self.conn.write(m)
def data_received(self, data):
print('Data received: {!r}'.format(data))
def connection_lost(self, exc):
print('The server closed the connection')
print('Stop the event loop')
self.loop.stop()
# note that in my use-case, main() is called continuously by an external game engine
client_init = False
def main():
# use a global here only for the purpose of providing example code runnable outside of aforementioned game engine
global client_init
if client_init != True:
loop = asyncio.get_event_loop()
coro = loop.create_connection(lambda: ClientProtocol(loop), '127.0.0.1', port)
task = asyncio.Task(coro)
transport, protocol = loop.run_until_complete(coro)
client_init = True
# to avoid blocking the execution of main (and of game engine calling it), only run one iteration of the event loop
loop.stop()
loop.run_forever()
if transport:
transport.write("some data")
if __name__ == "__main__":
main()
Traceback:
Traceback (most recent call last):
File "TCPclient.py", line 57, in <module>
main()
File "TCPclient.py", line 45, in main
transport, protocol = loop.run_until_complete(coro)
File "/usr/lib/python3.5/asyncio/base_events.py", line 337, in run_until_complete
return future.result()
File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
raise self._exception
File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step
result = coro.send(None)
File "/usr/lib/python3.5/asyncio/base_events.py", line 599, in create_connection
yield from tasks.wait(fs, loop=self)
File "/usr/lib/python3.5/asyncio/tasks.py", line 341, in wait
return (yield from _wait(fs, timeout, return_when, loop))
File "/usr/lib/python3.5/asyncio/tasks.py", line 424, in _wait
yield from waiter
File "/usr/lib/python3.5/asyncio/futures.py", line 359, in __iter__
assert self.done(), "yield from wasn't used with future"
AssertionError: yield from wasn't used with future

The problem seems to be that you create a task from your coroutine, but then pass the coroutine to run_until_complete instead:
coro = loop.create_connection(lambda: ClientProtocol(loop), '127.0.0.1', port)
task = asyncio.Task(coro)
transport, protocol = loop.run_until_complete(coro)
Either pass the task:
coro = loop.create_connection(lambda: ClientProtocol(loop), '127.0.0.1', port)
task = asyncio.Task(coro)
transport, protocol = loop.run_until_complete(task)
Or don't create the task and pass the coroutine. run_until_complete will create a task for you
coro = loop.create_connection(lambda: ClientProtocol(loop), '127.0.0.1', port)
transport, protocol = loop.run_until_complete(coro)
In addition, you need to ensure the strings you are writing are byte strings. String literals in Python 3 default to unicode. You can either encode these, or just write byte strings in the first place
transport.write("some data".encode('utf-8'))
transport.write(b"some data")
EDIT It's not clear to me why this is an issue, however the source for run_until_complete has this to say:
WARNING: It would be disastrous to call run_until_complete()
with the same coroutine twice -- it would wrap it in two
different Tasks and that can't be good.
I suppose creating a task and then passing in the coroutine (which causes a task to be created) has the same effect.