We Keep Coding

RuntimeWarning: coroutine 'UnaryStreamCall._send_unary_request' was never awaited

I am trying to make some API calls to firebase using the google.cloud.firestore.AsyncClient module. Fire2 is just a wrapper for this object.
The traceback suggests its a problem with asyncio, but if I make a dummy await it actually runs just fine. What is the problem and why does it occur?
Sample code:
import Fire2
class Test:
def __init__(self):
doc = asyncio.run(self.wait())
async def wait(self):
doc = await Fire2("test1").get(g1) # gives the error
# doc = await asyncio.sleep(1) # runs without error
return doc
def test(self):
x = Test2(p1)
class Test2:
def __init__(self, p):
doc = asyncio.run(self.run(p))
print(doc.to_dict())
async def run(self, p):
doc = await Fire2('test2').get(p)
return doc
p1 = 'foo'
g1 = 'bar'
h = Test()
h.test()
Traceback:
Traceback (most recent call last):
File "<project path>\scratch.py", line 137, in <module>
h.test()
File "<project path>\scratch.py", line 123, in test
x = Test2(p1)
File "<project path>\scratch.py", line 127, in __init__
doc = asyncio.run(self.run(p))
File "<user AppData>\Local\Programs\Python\Python39\lib\asyncio\runners.py", line 44, in run
return loop.run_until_complete(main)
File "<user AppData>\Local\Programs\Python\Python39\lib\asyncio\base_events.py", line 642, in run_until_complete
return future.result()
File "<project path>\scratch.py", line 131, in run
doc = await Fire2('test2').get(p)
File "<project path>\<Fire2 file>", line 422, in get
res = await self._collection.document(doc_id).get()
File "<project path>\venv\lib\site-packages\google\cloud\firestore_v1\async_document.py", line 364, in get
response_iter = await self._client._firestore_api.batch_get_documents(
File "<project path>\venv\lib\site-packages\google\api_core\grpc_helpers_async.py", line 171, in error_remapped_callable
call = callable_(*args, **kwargs)
File "<project path>\venv\lib\site-packages\grpc\aio\_channel.py", line 165, in __call__
call = UnaryStreamCall(request, deadline, metadata, credentials,
File "<project path>\venv\lib\site-packages\grpc\aio\_call.py", line 553, in __init__
self._send_unary_request_task = loop.create_task(
File "<user AppData>\Local\Programs\Python\Python39\lib\asyncio\base_events.py", line 431, in create_task
self._check_closed()
File "<user AppData>\Local\Programs\Python\Python39\lib\asyncio\base_events.py", line 510, in _check_closed
raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
sys:1: RuntimeWarning: coroutine 'UnaryStreamCall._send_unary_request' was never awaited
Process finished with exit code 1

You are making two separate calls to asyncio.run. We can see from the stack trace that the error is coming from inside the grpc library. I suspect that the same grpc connection is being used both times, and that during its initialization it integrates itself with the async event loop in order to maintain the connection in the background between calls.
The problem is that every time you call asyncio.run, you will create a new event loop, and every time it returns that event loop will be closed. So by the time you call this function again, and the grpc layer tries to do its work, it realizes that the original event loop has been closed and it returns this error.
To fix this, You can have a single call to asyncio.run, typically calling a "main" function, or similar. This would then make calls to async functions which perform the actual work.
If the problem persists then there might be some issues in the “Fire2”class.
For more reference you can check this link which outlines high-level asyncio APIs to work with coroutines and Tasks with the help of various examples.

Use pytest with using fixture with scope="session" in conftest.py
#pytest.fixture(scope="session")
def event_loop():
loop = get_event_loop()
yield loop
loop.close()

TypeError during the call python grpc method without arguments using google/protobuf/empty.proto

I have a proto scheme like this:
import "google/protobuf/empty.proto";
...
service NodeInfoService {
rpc NodeConfig (google.protobuf.Empty) returns (NodeConfigResponse);
}
Using grpc_tools I got classes and now, when I'm trying to send request from py client, but catching the error in "stub.NodeConfig()" call.
Even If I call it like "stub.NodeConfig({})" or "stub.NodeConfig("")" I have the same TypeError. Full code of client:
import grpc
import logging
from util import node_info_service_pb2_grpc
from util import node_info_service_pb2
def run():
with grpc.insecure_channel('ip:port') as channel:
stub = node_info_service_pb2_grpc.NodeInfoServiceStub(channel)
response = stub.NodeConfig(node_info_service_pb2.google_dot_protobuf_dot_empty__pb2.Empty)
print("Echo client received: " + response.message)
if __name__ == '__main__':
logging.basicConfig()
run()
Error:
ERROR:grpc._common:Exception serializing message!
Traceback (most recent call last):
File "/Users/user/p/p/venv/lib/python3.8/site-packages/grpc/_common.py", line 86, in _transform
return transformer(message)
TypeError: descriptor 'SerializeToString' for 'google.protobuf.pyext._message.CMessage' objects doesn't apply to a 'GeneratedProtocolMessageType' object
Traceback (most recent call last):
File "/Users/user/p/scripts/grpc/protobuf/client.py", line 15, in <module>
run()
File "/Users/user/p/scripts/grpc/protobuf/client.py", line 9, in run
response = stub.NodeConfig(node_info_service_pb2.google_dot_protobuf_dot_empty__pb2.Empty)
File "/Users/user/p/p/venv/lib/python3.8/site-packages/grpc/_channel.py", line 921, in __call__
state, call, = self._blocking(request, timeout, metadata, credentials,
File "/Users/user/p/p/venv/lib/python3.8/site-packages/grpc/_channel.py", line 901, in _blocking
raise rendezvous # pylint: disable-msg=raising-bad-type
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.INTERNAL
details = "Exception serializing request!"
debug_error_string = "None"

Solution:
empty = node_info_service_pb2.google_dot_protobuf_dot_empty__pb2.Empty()
response = stub.NodeConfig(empty)

sanic: how to convert markdown content asynchronously

I have the following Sanic route:
md = Markdown()
#app.route('/md_file')
async def md_file(request):
async with aiofiles.open('./file.md')) as f:
content = await f.read()
content = md.convert(content)
return html(content)
This works just fine but the conversion takes a very long time and blocks the endpoint. When benchmarking, the endpoint can only handle 4 requests per second.
Since there is no asyncio markdown library, I figured I would offload the conversion into a separate thread to free up the blocking code:
loop = asyncio.get_event_loop()
content = await loop.run_in_executor(ThreadPoolExecutor(), md.convert(content))
However, this throws a traceback:
2017-07-22 12:02:24 - (sanic)[ERROR]: Traceback (most recent call last):
File "/home/user/app/venv/lib64/python3.5/site-packages/sanic/app.py", line 471, in handle_request
response = await response
File "app.py", line 127, in blog_posts
content = await loop.run_in_executor(ThreadPoolExecutor(), md.convert(content))
File "uvloop/future.pyx", line 241, in __await__ (uvloop/loop.c:110786)
File "uvloop/future.pyx", line 432, in uvloop.loop.BaseTask._fast_wakeup (uvloop/loop.c:113980)
File "uvloop/future.pyx", line 101, in uvloop.loop.BaseFuture._result_impl (uvloop/loop.c:108900)
File "/opt/rh/rh-python35/root/usr/lib64/python3.5/concurrent/futures/thread.py", line 55, in run
result = self.fn(*self.args, **self.kwargs)
TypeError: 'str' object is not callable
Is it not possible to use the event loop from within Sanic? Are there any other options to make the conversion non-blocking?

md.convert(content) actually runs the function. This:
content = await loop.run_in_executor(ThreadPoolExecutor(), md.convert(content))
is the same as this:
content = await loop.run_in_executor(ThreadPoolExecutor(), "some HTML")
And that is quite obviously wrong. You don't want to run the function. You want to pass the function; the executor will handle running it. The signature of run_in_executor is
coroutine AbstractEventLoop.run_in_executor(executor, func, *args)
So use this instead
content = await loop.run_in_executor(ThreadPoolExecutor(), md.convert, content)

How to Mock/Patch App Engine's Oauth Decorator?

App Engine's python client library has made oauth flow really easy with the following decorator.
#decorator.oauth_required
But it's really not straightforward to mock/patch for unit testing. For example in the following get handler, I need to stub out oauth decorator.
from auth import decorator
class ListUsersHandler(webapp2.RequestHandler):
#decorator.oauth_required
def get(self):
self.response.write(_RenderUserListTemplate())
I have tried something like below.
from mock import patch
patch('decorator.oauth_required', lambda x: x).start()
import user
class MyTest(unittest.TestCase):
def setUp(self):
app = webapp2.WSGIApplication([('/', user.ListUsersHandler)])
self.testapp = webtest.TestApp(app)
def testListUsersHandler(self):
response = self.testapp.get('/')
self.assertTrue(('list tokens' in response))
But, what I'm seeing this error, which doesn't seem to give much clue.
Traceback (most recent call last):
File "user_test.py", line 44, in testAbc
response = self.testapp.get('/')
File "/usr/local/lib/python2.7/dist-packages/webtest/app.py", line 322, in get
expect_errors=expect_errors)
File "/usr/local/lib/python2.7/dist-packages/webtest/app.py", line 605, in do_request
res = req.get_response(app, catch_exc_info=True)
File "/google_appengine/lib/webob-1.2.3/webob/request.py", line 1292, in send
application, catch_exc_info=True)
File "/google_appengine/lib/webob-1.2.3/webob/request.py", line 1269, in call_application
return (captured[0], captured[1], app_iter, captured[2])
IndexError: list index out of range

build a simple remote dispatcher using multiprocessing.Managers

Consider the following code :
Server :
import sys
from multiprocessing.managers import BaseManager, BaseProxy, Process
def baz(aa) :
l = []
for i in range(3) :
l.append(aa)
return l
class SolverManager(BaseManager): pass
class MyProxy(BaseProxy): pass
manager = SolverManager(address=('127.0.0.1', 50000), authkey='mpm')
manager.register('solver', callable=baz, proxytype=MyProxy)
def serve_forever(server):
try :
server.serve_forever()
except KeyboardInterrupt:
pass
def runpool(n):
server = manager.get_server()
workers = []
for i in range(int(n)):
Process(target=serve_forever, args=(server,)).start()
if __name__ == '__main__':
runpool(sys.argv[1])
Client :
import sys
from multiprocessing.managers import BaseManager, BaseProxy
import multiprocessing, logging
class SolverManager(BaseManager): pass
class MyProxy(BaseProxy): pass
def main(args) :
SolverManager.register('solver')
m = SolverManager(address=('127.0.0.1', 50000), authkey='mpm')
m.connect()
print m.solver(args[1])._getvalue()
if __name__ == '__main__':
sys.exit(main(sys.argv))
If I run the server using only one process as python server.py 1
then the client works as expected. But if I spawn two processes (python server.py 2) listening for connections, I get a nasty error :
$python client.py ping
Traceback (most recent call last):
File "client.py", line 24, in <module>
sys.exit(main(sys.argv))
File "client.py", line 21, in main
print m.solver(args[1])._getvalue()
File "/usr/lib/python2.6/multiprocessing/managers.py", line 637, in temp
authkey=self._authkey, exposed=exp
File "/usr/lib/python2.6/multiprocessing/managers.py", line 894, in AutoProxy
incref=incref)
File "/usr/lib/python2.6/multiprocessing/managers.py", line 700, in __init__
self._incref()
File "/usr/lib/python2.6/multiprocessing/managers.py", line 750, in _incref
dispatch(conn, None, 'incref', (self._id,))
File "/usr/lib/python2.6/multiprocessing/managers.py", line 79, in dispatch
raise convert_to_error(kind, result)
multiprocessing.managers.RemoteError:
---------------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/lib/python2.6/multiprocessing/managers.py", line 181, in handle_request
result = func(c, *args, **kwds)
File "/usr/lib/python2.6/multiprocessing/managers.py", line 402, in incref
self.id_to_refcount[ident] += 1
KeyError: '7fb51084c518'
---------------------------------------------------------------------------
My idea is pretty simple. I want to create a server that will spawn a number of workers that will share the same socket and handle requests independently. Maybe I'm using the wrong tool here ?
The goal is to build a 3-tier structure where all requests are handled via an http server and then dispatched to nodes sitting in a cluster and from nodes to workers via the multiprocessing managers...
There is one public server, one node per machine and x number of workers on each machine depending on the number of cores... I know I can use a more sophisticated library, but for such a simple task (I'm just prototyping here) I would just use the multiprocessing library... Is this possible or I should explore directly other solutions ? I feel I'm very close to have something working here ... thanks.

You're trying to invent a wheel, many have invented before.
It sounds to me that you're looking for task queue where your server dispatches tasks to, and your workers execute this tasks.
I would recommend you to have a look at Celery.