Minimal demonstration example:
import asyncio
async def main():
c1_reader, c1_writer = await asyncio.open_connection(host='google.com', port=80)
c1_socket = c1_writer.get_extra_info('socket')
c1_socket.close()
c2_reader, c2_writer = await asyncio.open_connection(host='google.com', port=80)
asyncio.run(main())
Running this program gives this error:
$ python3 asyncio_fd_used.py
Traceback (most recent call last):
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/selector_events.py", line 469, in _sock_connect
sock.connect(address)
BlockingIOError: [Errno 36] Operation now in progress
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "asyncio_fd_used.py", line 11, in <module>
asyncio.run(main())
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/runners.py", line 43, in run
return loop.run_until_complete(main)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/base_events.py", line 579, in run_until_complete
return future.result()
File "asyncio_fd_used.py", line 9, in main
c2_reader, c2_writer = await asyncio.open_connection(host='google.com', port=80)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/streams.py", line 77, in open_connection
lambda: protocol, host, port, **kwds)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/base_events.py", line 941, in create_connection
await self.sock_connect(sock, address)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/selector_events.py", line 463, in sock_connect
self._sock_connect(fut, sock, address)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/selector_events.py", line 477, in _sock_connect
self.add_writer(fd, self._sock_connect_cb, fut, sock, address)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/selector_events.py", line 333, in add_writer
self._ensure_fd_no_transport(fd)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/selector_events.py", line 244, in _ensure_fd_no_transport
f'File descriptor {fd!r} is used by transport '
RuntimeError: File descriptor 8 is used by transport <_SelectorSocketTransport fd=8 read=polling write=<idle, bufsize=0>>
Just for explanation why I am doing the low-level socket.close() and not asyncio-level writer.close(): I was trying some code to send RST packet. But I can imagine other reasons why people would call socket.close(), maybe even unintentionally.
The problem is that the low-level socket is closed, but asyncio doesn't know about that and thinks it is still open. For some reason (performance?) asyncio remembers the socket file descriptor (fileno).
When a new connection is opened, operating system gives to it the same file descriptor number, and asyncio starts panicking, because it has the same exact fd number associated with that previous connection.
Solution: tell asyncio the socket is closed :)
import asyncio
async def main():
c1_reader, c1_writer = await asyncio.open_connection(host='google.com', port=80)
c1_socket = c1_writer.get_extra_info('socket')
c1_socket.close()
c1_writer.close() # <<< here
c2_reader, c2_writer = await asyncio.open_connection(host='google.com', port=80)
asyncio.run(main())
This code runs without raising an error.
Related
I've been trying to write code that collects crypto data from Binance. Binance auto disconnects after 24 hours. Is there any way for me to reconnect after disconnection? I believe running forever should take care of that for me, but it dies when an error is thrown. I will be running this program on a server 24/7. I will also need a way to be notified maybe telegram/discord bot that I can build where do I type the code to send when it is disconnected
This is the error I get.
Traceback (most recent call last):
File "exchanges/binance/binance_ticker.py", line 97, in <module>
start()
File "exchanges/binance/binance_ticker.py", line 94, in start
rel.dispatch()
File "/home/pyjobs/.local/lib/python3.8/site-packages/rel/rel.py", line 205, in dispatch
registrar.dispatch()
File "/home/pyjobs/.local/lib/python3.8/site-packages/rel/registrar.py", line 72, in dispatch
if not self.loop():
File "/home/pyjobs/.local/lib/python3.8/site-packages/rel/registrar.py", line 81, in loop
e = self.check_events()
File "/home/pyjobs/.local/lib/python3.8/site-packages/rel/registrar.py", line 232, in check_events
self.callback('read', fd)
File "/home/pyjobs/.local/lib/python3.8/site-packages/rel/registrar.py", line 125, in callback
self.events[etype][fd].callback()
File "/home/pyjobs/.local/lib/python3.8/site-packages/rel/listener.py", line 108, in callback
if not self.cb(*self.args) and not self.persist and self.active:
File "/home/pyjobs/.local/lib/python3.8/site-packages/websocket/_app.py", line 349, in read
op_code, frame = self.sock.recv_data_frame(True)
File "/home/pyjobs/.local/lib/python3.8/site-packages/websocket/_core.py", line 401, in recv_data_frame
frame = self.recv_frame()
File "/home/pyjobs/.local/lib/python3.8/site-packages/websocket/_core.py", line 440, in recv_frame
return self.frame_buffer.recv_frame()
File "/home/pyjobs/.local/lib/python3.8/site-packages/websocket/_abnf.py", line 352, in recv_frame
payload = self.recv_strict(length)
File "/home/pyjobs/.local/lib/python3.8/site-packages/websocket/_abnf.py", line 373, in recv_strict
bytes_ = self.recv(min(16384, shortage))
File "/home/pyjobs/.local/lib/python3.8/site-packages/websocket/_core.py", line 524, in _recv
return recv(self.sock, bufsize)
File "/home/pyjobs/.local/lib/python3.8/site-packages/websocket/_socket.py", line 122, in recv
raise WebSocketConnectionClosedException(
websocket._exceptions.WebSocketConnectionClosedException: Connection to remote host was lost.
My code:
import websocket
import rel
uri = "wss://stream.binance.com:9443/ws/!ticker#arr"
def on_message(ws, message):
print(message)
def on_error(ws, error):
print(error)
write_logs(error)
def on_close(ws, close_status_code, close_msg):
print("### closed ###")
write_logs(str(close_status_code) + str(close_msg))
start(
def on_open(ws):
print("Opened connection")
start()
websocket.enableTrace(True)
ws = websocket.WebSocketApp(uri,
on_open = on_open,
on_message=on_message,
on_error = on_error,
on_close (on_close)
ws.run_forever(dispatcher=rel) #Set the dispatcher to automatic reconnection.
rel.signal(2, rel.abort) # Keyboard Interrupt
rel.dispatch()
start()
The comment in this line of code ws.run_forever(dispatcher=rel) #Set the dispatcher to automatic reconnection. could auto reconnection depending on rel module? And how the module rel and func dispatcher work together?
There are lots of coroutines in my production code, which are stuck at unknown position while processing request. I attached gdb with Python support extension to the process, but it doesn't show the exact line in the coroutine where the process is stuck, only primary stack trace. Here is a minimal example:
import asyncio
async def hello():
await asyncio.sleep(30)
print('hello world')
asyncio.run(hello())
(gdb) py-bt
Traceback (most recent call first):
File "/usr/lib/python3.8/selectors.py", line 468, in select
fd_event_list = self._selector.poll(timeout, max_ev)
File "/usr/lib/python3.8/asyncio/base_events.py", line 2335, in _run_once
File "/usr/lib/python3.8/asyncio/base_events.py", line 826, in run_forever
None, getaddr_func, host, port, family, type, proto, flags)
File "/usr/lib/python3.8/asyncio/base_events.py", line 603, in run_until_complete
self.run_forever()
File "/usr/lib/python3.8/asyncio/runners.py", line 299, in run
File "main.py", line 7, in <module>
GDB shows a trace that ends on line 7, but the code is obviously stuck on line 4. How to make it show a more complete trace with nested coroutines?
You can use the aiodebug.log_slow_callbacks.enable(0.05)
Follow for more : https://pypi.org/project/aiodebug/
I am using Python 3.6.2, on Fedora 26 Workstation.
Below is some scrapbook code which demonstrates my issue:
EDIT: added Sam Hartman's suggestion to code.
import asyncio, json
from autobahn.asyncio.websocket import WebSocketClientProtocol, WebSocketClientFactory
class MyClientProtocol(WebSocketClientProtocol):
def onConnect(self, response):
print(response.peer)
def onOpen(self):
print("open")
self.sendMessage(json.dumps({'command': 'subscribe', 'channel': "1010"}).encode("utf8"))
def onMessage(self, payload, isBinary):
print("message")
print(json.loads(payload))
factory1 = WebSocketClientFactory("wss://api2.poloniex.com:443")
factory1.protocol = MyClientProtocol
loop1 = asyncio.get_event_loop()
loop1.run_until_complete(loop1.create_connection(factory1, "api2.poloniex.com", 443, ssl=True))
try:
loop1.run_forever()
except KeyboardInterrupt:
pass
loop1.close()
asyncio.set_event_loop(asyncio.new_event_loop())
factory2 = WebSocketClientFactory("wss://api2.poloniex.com:443")
factory2.protocol = MyClientProtocol
loop2 = asyncio.get_event_loop()
loop2.run_until_complete(loop2.create_connection(factory2, "api2.poloniex.com", 443, ssl=True))
try:
loop2.run_forever()
except KeyboardInterrupt:
pass
loop2.close()
After having closed an initial asyncio event loop, creating another and setting it as the global event loop, attempting to use the new event loop yields the following errors:
Fatal write error on socket transport
protocol: <asyncio.sslproto.SSLProtocol object at 0x7f8a84ed4748>
transport: <_SelectorSocketTransport fd=6>
Traceback (most recent call last):
File "/usr/lib64/python3.6/asyncio/selector_events.py", line 762, in write
n = self._sock.send(data)
OSError: [Errno 9] Bad file descriptor
Fatal error on SSL transport
protocol: <asyncio.sslproto.SSLProtocol object at 0x7f8a84ed4748>
transport: <_SelectorSocketTransport closing fd=6>
Traceback (most recent call last):
File "/usr/lib64/python3.6/asyncio/selector_events.py", line 762, in write
n = self._sock.send(data)
OSError: [Errno 9] Bad file descriptor
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib64/python3.6/asyncio/sslproto.py", line 648, in _process_write_backlog
self._transport.write(chunk)
File "/usr/lib64/python3.6/asyncio/selector_events.py", line 766, in write
self._fatal_error(exc, 'Fatal write error on socket transport')
File "/usr/lib64/python3.6/asyncio/selector_events.py", line 646, in _fatal_error
self._force_close(exc)
File "/usr/lib64/python3.6/asyncio/selector_events.py", line 658, in _force_close
self._loop.call_soon(self._call_connection_lost, exc)
File "/usr/lib64/python3.6/asyncio/base_events.py", line 574, in call_soon
self._check_closed()
File "/usr/lib64/python3.6/asyncio/base_events.py", line 357, in _check_closed
raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
Exception in callback _SelectorSocketTransport._read_ready()
handle: <Handle _SelectorSocketTransport._read_ready()>
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/txaio/_common.py", line 63, in call_later
self._buckets[real_time][1].append(call)
KeyError: 412835000
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib64/python3.6/asyncio/events.py", line 127, in _run
self._callback(*self._args)
File "/usr/lib64/python3.6/asyncio/selector_events.py", line 731, in _read_ready
self._protocol.data_received(data)
File "/usr/lib64/python3.6/asyncio/sslproto.py", line 503, in data_received
ssldata, appdata = self._sslpipe.feed_ssldata(data)
File "/usr/lib64/python3.6/asyncio/sslproto.py", line 204, in feed_ssldata
self._handshake_cb(None)
File "/usr/lib64/python3.6/asyncio/sslproto.py", line 619, in _on_handshake_complete
self._app_protocol.connection_made(self._app_transport)
File "/usr/lib/python3.6/site-packages/autobahn/asyncio/websocket.py", line 97, in connection_made
self._connectionMade()
File "/usr/lib/python3.6/site-packages/autobahn/websocket/protocol.py", line 3340, in _connectionMade
WebSocketProtocol._connectionMade(self)
File "/usr/lib/python3.6/site-packages/autobahn/websocket/protocol.py", line 1055, in _connectionMade
self.onOpenHandshakeTimeout,
File "/usr/lib/python3.6/site-packages/txaio/_common.py", line 72, in call_later
self._notify_bucket, real_time,
File "/usr/lib/python3.6/site-packages/txaio/aio.py", line 382, in call_later
return self._config.loop.call_later(delay, real_call)
File "/usr/lib64/python3.6/asyncio/base_events.py", line 543, in call_later
timer = self.call_at(self.time() + delay, callback, *args)
File "/usr/lib64/python3.6/asyncio/base_events.py", line 553, in call_at
self._check_closed()
File "/usr/lib64/python3.6/asyncio/base_events.py", line 357, in _check_closed
raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
It seems reasonable that one might need to reopen an event loop after having closed an earlier one. Indeed this question even shows how: Asyncio Event Loop is Closed
The code below should achieve this:
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
so I am clearly doing something wrong. Can somebody see something missing?
I have fairly high confidence that your factory object is maintaining a reference to the old event loop presumably that it gets from asyncio.get_event_loop. Asyncio consumers are bad about getting hidden references to loops.
My recommendation is to reconstruct the web socket factory after closing the loop
I was learning some async/await in python, and i wanted to try it, but
I'm getting this error while trying to connect to chatango via websocket and i don't know what means.
I'm using python 3.6.1 and aiohttp 2.2.3
This is my code:
import asyncio
import aiohttp
msgs = []
async def main():
async with aiohttp.ClientSession() as session:
async with session.ws_connect("ws://s12.chatango.com:8081/") as ws:
for msg in ws:
msgs.append(msg)
print(msg)
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
loop.close()
Full traceback:
Traceback (most recent call last):
File "C:\Program Files\Python36\lib\site-packages\aiohttp\client_reqrep.py", line 559, in start
(message, payload) = yield from self._protocol.read()
File "C:\Program Files\Python36\lib\site-packages\aiohttp\streams.py", line 509, in read
yield from self._waiter
File "C:\Program Files\Python36\lib\site-packages\aiohttp\client_proto.py", line 165, in data_received
messages, upgraded, tail = self._parser.feed_data(data)
File "aiohttp\_http_parser.pyx", line 274, in aiohttp._http_parser.HttpParser.feed_data (aiohttp/_http_parser.c:4364)
aiohttp.http_exceptions.BadHttpMessage: 400, message='invalid constant string'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "C:/Users/joseh/Desktop/a.ws.py", line 42, in <module>
loop.run_until_complete(main())
File "C:\Program Files\Python36\lib\asyncio\base_events.py", line 466, in run_until_complete
return future.result()
File "C:/Users/joseh/Desktop/a.ws.py", line 34, in main
async with session.ws_connect("ws://s12.chatango.com:8081/") as ws:
File "C:\Program Files\Python36\lib\site-packages\aiohttp\client.py", line 603, in __aenter__
self._resp = yield from self._coro
File "C:\Program Files\Python36\lib\site-packages\aiohttp\client.py", line 390, in _ws_connect
proxy_auth=proxy_auth)
File "C:\Program Files\Python36\lib\site-packages\aiohttp\helpers.py", line 91, in __iter__
ret = yield from self._coro
File "C:\Program Files\Python36\lib\site-packages\aiohttp\client.py", line 241, in _request
yield from resp.start(conn, read_until_eof)
File "C:\Program Files\Python36\lib\site-packages\aiohttp\client_reqrep.py", line 564, in start
message=exc.message, headers=exc.headers) from exc
aiohttp.client_exceptions.ClientResponseError: 400, message='invalid constant string'
invalid constant string is a custom response from chatango, they probably want a protocol or some kind of auth header.
If you don't know much about how chatango uses websockets, reverse engineering their system is probably not a good task for learning asyncio and aiohttp.
Better to use something like httparrot which just echos back the message you send it.
Here's your code modified to use httparrot and send 5 messages, get 5 responses, then exit.
import asyncio
import aiohttp
msgs = []
async def main():
async with aiohttp.ClientSession() as session:
async with session.ws_connect('ws://httparrot.herokuapp.com/websocket') as ws:
ws.send_str('hello')
async for msg in ws:
msgs.append(msg)
print(msg)
ws.send_str('hello')
if len(msgs) >= 5:
break
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
loop.close()
print(msgs)
While trying to set up a Websocket server, I have encountered the following error.
The same code works fine under LAN IP '192.168.x.x', but fails to work with a public ip/domain name
Here is the error trace
Traceback (most recent call last):
File "WSServer.py", line 19, in <module>
server = loop.run_until_complete(coro)
File "/usr/lib64/python3.4/asyncio/base_events.py", line 208, in run_until_complete
return future.result()
File "/usr/lib64/python3.4/asyncio/futures.py", line 243, in result
raise self._exception
File "/usr/lib64/python3.4/asyncio/tasks.py", line 319, in _step
result = coro.send(value)
File "/usr/lib64/python3.4/asyncio/base_events.py", line 579, in create_server
% (sa, err.strerror.lower()))
OSError: [Errno 99] error while attempting to bind on address ('121.6.x.x', 9000): cannot assign requested address
Python Server Code:
from autobahn.asyncio.websocket import WebSocketServerProtocol
class MyServerProtocol(WebSocketServerProtocol):
def onMessage(self, payload, isBinary):
print("message received")
self.sendMessage(payload, isBinary)
if __name__ == '__main__':
import asyncio
from autobahn.asyncio.websocket import WebSocketServerFactory
factory = WebSocketServerFactory()
factory.protocol = MyServerProtocol
loop = asyncio.get_event_loop()
coro = loop.create_server(factory, '121.6.x.x', 9000)
server = loop.run_until_complete(coro)
try:
loop.run_forever()
except KeyboardInterrupt:
pass
finally:
server.close()
loop.close()
Could the issue be related with the server setting? e.g. hostname, SELinux