We Keep Coding

Receiving EPIPE error when streaming from PSQL copy function

I am trying to write a streaming implementation of dumping a table from psql into a pre-signed URL on S3. Unfortunately, it seems to error out at a seemingly random time in the upload. I have tried many combinations of opening/closing the file descriptors at different times. I for the life of me cannot figure out why this is occurring.
The strangest thing is when I mock the requests library and analyze the sent data, it works as intended. The socket is raising an EPIPE error at a certain amount through the stream
from psycopg2 import connect
import threading
import requests
import requests_mock
import traceback
from base64 import b64decode
from boto3 import session
r_fd, w_fd = os.pipe()
connection = connect(host='host', database='db',
user='user', password='pw')
cursor = connection.cursor()
b3_session = session.Session(profile_name='profile', region_name='us-east-1')
url = b3_session.client('s3').generate_presigned_url(
ClientMethod='put_object',
Params={'Bucket': 'bucket', 'Key': 'test_streaming_upload.txt'},
ExpiresIn=3600)
rd = os.fdopen(r_fd, 'rb')
wd = os.fdopen(w_fd, 'wb')
def stream_data():
print('Starting stream')
with os.fdopen(r_fd, 'rb') as rd:
requests.put(url, data=rd, headers={'Content-type': 'application/octet-stream'})
print('Ending stream')
to_thread = threading.Thread(target=stream_data)
to_thread.start()
print('Starting copy')
with os.fdopen(w_fd, 'wb') as wd:
cursor.copy_expert('COPY table TO STDOUT WITH CSV HEADER', wd)
print('Ending copy')
to_thread.join()
The output is always the same:
Starting stream
Starting copy
Exception in thread Thread-1:
Traceback (most recent call last):
File "/venv/lib/python3.9/site-packages/urllib3/contrib/pyopenssl.py", line 342, in _send_until_done
return self.connection.send(data)
File "/venv/lib/python3.9/site-packages/OpenSSL/SSL.py", line 1718, in send
self._raise_ssl_error(self._ssl, result)
File "/venv/lib/python3.9/site-packages/OpenSSL/SSL.py", line 1624, in _raise_ssl_error
raise SysCallError(errno, errorcode.get(errno))
OpenSSL.SSL.SysCallError: (32, 'EPIPE')
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/venv/lib/python3.9/site-packages/requests/adapters.py", line 473, in send
low_conn.send(b'\r\n')
File "/Users/me/.pyenv/versions/3.9.7/lib/python3.9/http/client.py", line 995, in send
self.sock.sendall(data)
File "/venv/lib/python3.9/site-packages/urllib3/contrib/pyopenssl.py", line 354, in sendall
sent = self._send_until_done(
File "/venv/lib/python3.9/site-packages/urllib3/contrib/pyopenssl.py", line 349, in _send_until_done
raise SocketError(str(e))
OSError: (32, 'EPIPE')
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/me/.pyenv/versions/3.9.7/lib/python3.9/threading.py", line 973, in _bootstrap_inner
self.run()
File "/Users/me/.pyenv/versions/3.9.7/lib/python3.9/threading.py", line 910, in run
self._target(*self._args, **self._kwargs)
File "/Users/me/Library/Application Support/JetBrains/PyCharm2021.2/scratches/scratch_60.py", line 37, in stream_data
requests.put(url, data=rd, headers={'Content-type': 'application/octet-stream'})
File "/venv/lib/python3.9/site-packages/requests/api.py", line 131, in put
return request('put', url, data=data, **kwargs)
File "/venv/lib/python3.9/site-packages/requests/api.py", line 60, in request
return session.request(method=method, url=url, **kwargs)
File "/venv/lib/python3.9/site-packages/requests/sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "/venv/lib/python3.9/site-packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/venv/lib/python3.9/site-packages/requests/adapters.py", line 498, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: (32, 'EPIPE')
Am I missing something obvious? Is this a memory error? I appreciate any insight I can get because this is killing me. I can verify that the socket is being written to anywhere from 1.5 to 2.5k times before this error occurs.

pymongo: Resolver configuration could not be read or specified no nameservers

It's my first time using MongoDB but I can't seem to fix this one issue, my friend who uses MongoDB doesn't know how to use python so he can't really help me.
Here's my code:
import pymongo
# Replace the uri string with your MongoDB deployment's connection string.
conn_str = "mongodb+srv://sqdnoises:{mypass}#sqd.d4kjb.mongodb.net/myFirstDatabase?retryWrites=true&w=majority"
# set a 5-second connection timeout
client = pymongo.MongoClient(conn_str, serverSelectionTimeoutMS=5000)
try:
print(client.server_info())
print('\n\n\n aka connected')
except Exception:
print("Unable to connect to the server.")
Where {mypass} is my MongoDB password.
I keep getting this error:
Traceback (most recent call last):
File "/data/data/com.termux/files/usr/lib/python3.9/site-packages/dns/resolver.py", line 782, in read_resolv_conf
f = stack.enter_context(open(f))
FileNotFoundError: [Errno 2] No such file or directory: '/etc/resolv.conf'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/data/data/com.termux/files/usr/lib/python3.9/site-packages/pymongo/srv_resolver.py", line 88, in _resolve_uri
results = _resolve('_' + self.__srv + '._tcp.' + self.__fqdn,
File "/data/data/com.termux/files/usr/lib/python3.9/site-packages/pymongo/srv_resolver.py", line 41, in _resolve
return resolver.resolve(*args, **kwargs)
File "/data/data/com.termux/files/usr/lib/python3.9/site-packages/dns/resolver.py", line 1305, in resolve
return get_default_resolver().resolve(qname, rdtype, rdclass, tcp, source,
File "/data/data/com.termux/files/usr/lib/python3.9/site-packages/dns/resolver.py", line 1278, in get_default_resolver
reset_default_resolver()
File "/data/data/com.termux/files/usr/lib/python3.9/site-packages/dns/resolver.py", line 1290, in reset_default_resolver
default_resolver = Resolver()
File "/data/data/com.termux/files/usr/lib/python3.9/site-packages/dns/resolver.py", line 734, in __init__
self.read_resolv_conf(filename)
File "/data/data/com.termux/files/usr/lib/python3.9/site-packages/dns/resolver.py", line 785, in read_resolv_conf
raise NoResolverConfiguration
dns.resolver.NoResolverConfiguration: Resolver configuration could not be read or specified no nameservers.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/storage/emulated/0/! workspace/mongolearn/main.py", line 7, in <module>
client = pymongo.MongoClient(conn_str, serverSelectionTimeoutMS=5000)
File "/data/data/com.termux/files/usr/lib/python3.9/site-packages/pymongo/mongo_client.py", line 677, in __init__
res = uri_parser.parse_uri(
File "/data/data/com.termux/files/usr/lib/python3.9/site-packages/pymongo/uri_parser.py", line 532, in parse_uri
nodes = dns_resolver.get_hosts()
File "/data/data/com.termux/files/usr/lib/python3.9/site-packages/pymongo/srv_resolver.py", line 119, in get_hosts
_, nodes = self._get_srv_response_and_hosts(True)
File "/data/data/com.termux/files/usr/lib/python3.9/site-packages/pymongo/srv_resolver.py", line 99, in _get_srv_response_and_hosts
results = self._resolve_uri(encapsulate_errors)
File "/data/data/com.termux/files/usr/lib/python3.9/site-packages/pymongo/srv_resolver.py", line 95, in _resolve_uri
raise ConfigurationError(str(exc))
pymongo.errors.ConfigurationError: Resolver configuration could not be read or specified no nameservers.
How do I fix this?
I am following https://docs.mongodb.com/drivers/pymongo/

indeed, the problem is that dnspython tries to open /etc/resolv.conf
import dns.resolver
dns.resolver.default_resolver=dns.resolver.Resolver(configure=False)
dns.resolver.default_resolver.nameservers=['8.8.8.8']
Just adde this code to the top of your main code, and that should be sufficient to get you past this hurdle..

How to debug a stuck asyncio coroutine in Python?

There are lots of coroutines in my production code, which are stuck at unknown position while processing request. I attached gdb with Python support extension to the process, but it doesn't show the exact line in the coroutine where the process is stuck, only primary stack trace. Here is a minimal example:
import asyncio
async def hello():
await asyncio.sleep(30)
print('hello world')
asyncio.run(hello())
(gdb) py-bt
Traceback (most recent call first):
File "/usr/lib/python3.8/selectors.py", line 468, in select
fd_event_list = self._selector.poll(timeout, max_ev)
File "/usr/lib/python3.8/asyncio/base_events.py", line 2335, in _run_once
File "/usr/lib/python3.8/asyncio/base_events.py", line 826, in run_forever
None, getaddr_func, host, port, family, type, proto, flags)
File "/usr/lib/python3.8/asyncio/base_events.py", line 603, in run_until_complete
self.run_forever()
File "/usr/lib/python3.8/asyncio/runners.py", line 299, in run
File "main.py", line 7, in <module>
GDB shows a trace that ends on line 7, but the code is obviously stuck on line 4. How to make it show a more complete trace with nested coroutines?

You can use the aiodebug.log_slow_callbacks.enable(0.05)
Follow for more : https://pypi.org/project/aiodebug/

RuntimeError: File descriptor 8 is used by transport

Minimal demonstration example:
import asyncio
async def main():
c1_reader, c1_writer = await asyncio.open_connection(host='google.com', port=80)
c1_socket = c1_writer.get_extra_info('socket')
c1_socket.close()
c2_reader, c2_writer = await asyncio.open_connection(host='google.com', port=80)
asyncio.run(main())
Running this program gives this error:
$ python3 asyncio_fd_used.py
Traceback (most recent call last):
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/selector_events.py", line 469, in _sock_connect
sock.connect(address)
BlockingIOError: [Errno 36] Operation now in progress
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "asyncio_fd_used.py", line 11, in <module>
asyncio.run(main())
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/runners.py", line 43, in run
return loop.run_until_complete(main)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/base_events.py", line 579, in run_until_complete
return future.result()
File "asyncio_fd_used.py", line 9, in main
c2_reader, c2_writer = await asyncio.open_connection(host='google.com', port=80)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/streams.py", line 77, in open_connection
lambda: protocol, host, port, **kwds)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/base_events.py", line 941, in create_connection
await self.sock_connect(sock, address)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/selector_events.py", line 463, in sock_connect
self._sock_connect(fut, sock, address)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/selector_events.py", line 477, in _sock_connect
self.add_writer(fd, self._sock_connect_cb, fut, sock, address)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/selector_events.py", line 333, in add_writer
self._ensure_fd_no_transport(fd)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/selector_events.py", line 244, in _ensure_fd_no_transport
f'File descriptor {fd!r} is used by transport '
RuntimeError: File descriptor 8 is used by transport <_SelectorSocketTransport fd=8 read=polling write=<idle, bufsize=0>>
Just for explanation why I am doing the low-level socket.close() and not asyncio-level writer.close(): I was trying some code to send RST packet. But I can imagine other reasons why people would call socket.close(), maybe even unintentionally.

The problem is that the low-level socket is closed, but asyncio doesn't know about that and thinks it is still open. For some reason (performance?) asyncio remembers the socket file descriptor (fileno).
When a new connection is opened, operating system gives to it the same file descriptor number, and asyncio starts panicking, because it has the same exact fd number associated with that previous connection.
Solution: tell asyncio the socket is closed :)
import asyncio
async def main():
c1_reader, c1_writer = await asyncio.open_connection(host='google.com', port=80)
c1_socket = c1_writer.get_extra_info('socket')
c1_socket.close()
c1_writer.close() # <<< here
c2_reader, c2_writer = await asyncio.open_connection(host='google.com', port=80)
asyncio.run(main())
This code runs without raising an error.

Restart python file after error

I have a program that streams prices and is getting a badstatusline error during slow hours. This causes issues with other files that need to interact with the stream. I am having much trouble simply catching the exceptions, leading to other exceptions that I cannot catch for some reason BadStatusLine leads to CannotSendRequest leads to ResponseNotReady. How can I simply restart (in this case) trading.py when execution.py raises the exception BadStatusLine?
Here is how I'm handling it now..
while True:
try:
response = self.conn.getresponse().read()
print response
except Exception:
pass
else:
break
Its a stream using Httplib if thats of importance
Thanks for the help
Here is the error as well:
Exception in thread Thread-1:
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810, in __bootstrap_inner
self.run()
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763, in run
self.__target(*self.__args, **self.__kwargs)
File "/Users/mattduhon/trading4.py", line 30, in trade
execution.execute_order(event)
File "/Users/mattduhon/execution.py", line 34, in execute_order
response = self.conn.getresponse().read()
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1073, in getresponse
response.begin()
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 415, in begin
version, status, reason = self._read_status()
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 379, in _read_status
raise BadStatusLine(line)
BadStatusLine: ''

If you are file continuously then you can put it in supervisor and add
auto_start = True
Or In your code you can do something like that
import os
while True:
try:
response = self.conn.getresponse().read()
print response
except:
os.system("python trading.py")
I added broad exception because you don't know which exception is occuring

Create another script to run your main script, and try and except the whole thing:
try:
execfile('main.py')
except:
pass