I am trying to get some data from Oracle, using Twisted and runQuery and keep getting Deferred instead of actual data.
How can this be solved?
Some code (I excluded some unnecessary parts, but the idea should be clear):
from twisted.enterprise import adbapi
from twisted.internet import defer
import service_config
ORACLE_DSN = service_config.oracle_dsn
ORACLE_USER = service_config.oracle_user
ORACLE_PASSWORD = service_config.oracle_password
dbpool = adbapi.ConnectionPool('cx_Oracle',
user=ORACLE_USER,
password=ORACLE_PASSWORD,
dsn=ORACLE_DSN, port='49161')
#defer.inlineCallbacks
def ask_db():
data = yield dbpool.runQuery("SELECT * FROM customer")
a = ask_db()
print(a)
I got reactor running in other module, if that is important.
Thank you in advance.
UPDATE:
With help of #notorious.no got working code, returning data instead of Deferred with Python 3.5:
#defer.inlineCallbacks
def ask_db(request):
data = yield dbpool.runQuery(request)
return defer.returnValue(data)
You get a Deferred because you're calling an inlineCallback which always returns a Deferred. You're also misinterpreting what yield does. It doesn't actually return a value from an inlinceCallback it just wait's for a result. Use defer.returnValue() to return a value (you can use a simple return if you're using Python 3.4+). This is what your code should look like:
from __future__ import print_function
#...
#defer.inlineCallbacks
def ask_db():
data = yield dbpool.runQuery("SELECT * FROM customer")
defer.returnValue(data) # actually return a value
a = ask_db() # this returns a Deferred so add callbacks!
a.addCallback(print) # add a useful callback to processes query list
reactor.run()
The difference between what you had previously and this answer is that a callback is added so when the runQuery() returns with a value, the callback is executed and ask_db() actually returns a value you care about.
References
Database Usage with Klein/Twisted
Related
I am using cocotb version 1.5.2, and I would like to write an utility function to create reports/plots per test.
MWE: Implementing the get_test_name function so that the following test will print my_wonderful_test.
import cocotb
#cocotb.test()
async def my_wonderful_test(dut):
print(get_test_name(dut));
def get_test_name(dut):
pass # how do I get the current test from here?
You can use "name" attribute :
import cocotb
#cocotb.test()
async def my_wonderful_test(dut):
print(my_wonderful_test.name);
But not sure that exactly you want.
Thank you for the commenters, and thank you #FabienM for trying to give an answer. Thank you for #Tzane for trying to find an answer. You were close.
If you want to know a one liner answer
import cocotb;
def get_test_name():
return cocotb.regression_manager._test.name
but the underline prefix in _test maybe it will break in the future, but for since I was only concerned about version 1.5.2 this is OK for me.
Any way I implemented another method that scans the stack one level at a time and check if the frame is in a cocotb.test decorated function. This is also the method that cocotb uses to _discover_tests
It won't work if the test is in a closure, but I never use that, and I don't know if it is even supported.
import cocotb
import inspect;
import sys
#cocotb.test()
async def test_get_testname(dut):
print('Runnign from test ', get_test_name())
def get_test_name():
try:
return cocotb.regression_manager._test.name
except:
pass
cocotbdir = '/'.join(cocotb.__file__.split('/')[:-1])
frame = sys._getframe();
prev_frame = None
while frame is not None:
try:
# the [documentation](https://docs.python.org/2/library/inspect.html#inspect.getmodule)
# says
# Try to guess which module an object was defined in.
# Implying it may fail, wrapp in a try block and everything is fine
module = inspect.getmodule(frame.f_code)
func_name = inspect.getframeinfo(frame).function
if hasattr(module, func_name):
ff = getattr(module, func_name)
if isinstance(ff, cocotb.test):
return func_name
except:
pass
prev_frame = frame;
frame = frame.f_back;
# return None if fail to determine the test name
I don't know why my questions are so badly received
It was something simple that I preferred to engage more people
I am trying to use twisted framework for some async programming task recently. One thing I do not quite understand is how to wrap a function which takes a call back function and make it an function that return a deferred object?
For example if I have a function like below:
def registerCallbackForData(callback):
pass # this is a function that I do not control, some library code
And now the way I use it is to just register a callback. But I want to be able to incorporate this into the twisted framework, returning a deferred object probably and use reactor.run() later. Is this possible?
def convert_callback_to_deferred(f):
def g():
d = Deferred()
d.addCallback(callback)
f(d.callback)
return d
return g
from somelib import registerCallbackForData
getSomeDeferredForData = convert_callback_to_deferred(registerCallbackForData)
d = getSomeDeferredForData()
d.addCallback(...)
...
However, bear in mind that a Deferred can produce at most one result. If registerCallbackForData(cb) will result in cb being called more than once, there is nowhere for the 2nd and following calls' data to go. Only if you have an at-most-once event source does it make sense to convert it to the Deferred interface.
defer.maybeDeferred will wrap a blocking function call to return a deferred. If you want the call to be non-blocking you can instead use threads.deferToThread.
You can see the difference by switching which call is commented out:
import time
from twisted.internet import reactor, defer, threads
def foo():
print "going to sleep"
time.sleep(1)
print "woke up"
return "a result"
def show_result_and_stop_reactor(result):
print "got result: %s, stopping the reactor" % result
reactor.stop()
print "making the deferred"
d = defer.maybeDeferred(foo)
# d = threads.deferToThread(foo)
print "adding the callback"
d.addCallback(show_result_and_stop_reactor)
print "running the reactor"
reactor.run()
I want to get value of a tornado object with key
This is my code :
beanstalk = beanstalkt.Client(host='host', port=port)
beanstalk.connect()
print("ok1")
beanstalk.watch('contracts')
stateTube = beanstalk.stats_tube('contracts', callback=show)
print("ok2")
ioloop = tornado.ioloop.IOLoop.instance()
ioloop.start()
print("ok3")
And this is the function `show()``
def show(s):
pprint(s['current-jobs-ready'])
ioloop.stop
When I look at the documentation I found this :
And when I excecute this code, I have this :
ok1
ok2
3
In fact I have the result I wanted "3" but I don't understand why my program continue to running? Whythe ioloop doesn't close? I don't have ok3when I compile how can I do to close the ioloop and have ok3?
beanstalk.stats_tube is async, it returns a Future which represents a future result that has not yet been resolved.
As the README says, Your callback show will be executed with a dict that contains the resolved result. So you could define show like:
def show(stateTube):
pprint(stateTube['current-job-ready'])
beanstalk.stats_tube('contracts', callback=show)
from tornado.ioloop import IOLoop
IOLoop.current().start()
Note that you pass show, not show(): you're passing the function itself, not calling the function and passing its return value.
The other way to resolve a Future, besides passing a callback, is to use it in a coroutine:
from tornado import gen
from tornado.ioloop import IOLoop
#gen.coroutine
def get_stats():
stateTube = yield beanstalk.stats_tube('contracts')
pprint(stateTube['current-job-ready'])
loop = IOLoop.current()
loop.spawn_callback(get_stats)
loop.start()
In the following code example I have a function do_async_thing which appears to return a Future, even though I'm not sure why?
import tornado.ioloop
import tornado.web
import tornado.httpclient
#tornado.gen.coroutine
def do_async_thing():
http = tornado.httpclient.AsyncHTTPClient()
response = yield http.fetch("http://www.google.com/")
return response.body
class MainHandler(tornado.web.RequestHandler):
def get(self):
x = do_async_thing()
print(x) # <tornado.concurrent.Future object at 0x10753a6a0>
self.set_header("Content-Type", "application/json")
self.write('{"foo":"bar"}')
self.finish()
if __name__ == "__main__":
app = tornado.web.Application([
(r"/foo/?", MainHandler),
])
app.listen(8888)
tornado.ioloop.IOLoop.current().start()
You'll see that I yield the call to fetch and in doing so I should have forced the value to be realised (and subsequently been able to access the body field of the response).
What's more interesting is how I can even access the body field on a Future and not have it error (as far as I know a Future has no such field/property/method)
So does anyone know how I can:
Resolve the Future so I get the actual value
Modify this example so the function do_async_thing makes multiple async url fetches
Now it's worth noting that because I was still getting a Future back I thought I would try adding a yield to prefix the call to do_async_thing() (e.g. x = yield do_async_thing()) but that gave me back the following error:
tornado.gen.BadYieldError: yielded unknown object <generator object get at 0x1023bc308>
I also looked at doing something like this for the second point:
def do_another_async_thing():
http = tornado.httpclient.AsyncHTTPClient()
a = http.fetch("http://www.google.com/")
b = http.fetch("http://www.github.com/")
return a, b
class MainHandler(tornado.web.RequestHandler):
def get(self):
y = do_another_async_thing()
print(y)
But again this returns:
<tornado.concurrent.Future object at 0x102b966d8>
Where as I would've expected a tuple of Futures at least? At this point I'm unable to resolve these Futures without getting an error such as:
tornado.gen.BadYieldError: yielded unknown object <generator object get at 0x1091ac360>
Update
Below is an example that works (as per answered by A. Jesse Jiryu Davis)
But I've also added another example where by I have a new function do_another_async_thing which makes two async HTTP requests (but evaluating their values are a little bit more involved as you'll see):
def do_another_async_thing():
http = tornado.httpclient.AsyncHTTPClient()
a = http.fetch("http://www.google.com/")
b = http.fetch("http://www.github.com/")
return a, b
#tornado.gen.coroutine
def do_async_thing():
http = tornado.httpclient.AsyncHTTPClient()
response = yield http.fetch("http://www.google.com/")
return response.body
class MainHandler(tornado.web.RequestHandler):
#tornado.gen.coroutine
def get(self):
x = yield do_async_thing()
print(x) # displays HTML response
fa, fb = do_another_async_thing()
fa = yield fa
fb = yield fb
print(fa.body, fb.body) # displays HTML response for each
It's worth clarifying: you might expect the two yield statements for do_another_async_thing to cause a blockage. But here is a breakdown of the steps that are happening:
do_another_async_thing returns immediately a tuple with two Futures
we yield the first tuple which causes the program to be blocked until the value is realised
the value is realised and so we move to the next line
we yield again, causing the program to block until the value is realised
but as both futures were created at the same time and run concurrently the second yield returns practically instantly
Coroutines return futures. To wait for the coroutine to complete, the caller must also be a coroutine, and must yield the future. So:
#gen.coroutine
def get(self):
x = yield do_async_thing()
For more info see Refactoring Tornado Coroutines.
First of all, here are my two python files:
sred.py:
import _thread,time
class Thread:
def __init__(self,time:int,say:str):
self.time=time
self.say=say
def create():
id = _thread.get_ident()
for i in range(5):
print("HALLO", id)
return
from sred import Thread
import time,_thread
_thread.start_new_thread(Thread.create,())
The second one:
main.py
from sred import Thread
import time,_thread
_thread.start_new_thread(Thread.create,())
when executing this it doesn't print anything out, why?
UPDATE:
import _thread
class Thread:
#classmethod
def create():
id = _thread.get_ident()
for i in range(5):
print("HALLO", id)
return
main.py:
from sred import Thread
import time,_thread
_thread.start_new_thread(Thread().create,())
Is this now right, or is there still something wrong?
The create method is missing self as a parameter -- it looks like it should also be a #classmethod if you want to call it as it's written now. Note that your __init__ method is never getting called, because you never instantiate any Thread objects. You may want it to read:
_thread.start_new_thread(Thread().create, ())
i.e., instantiate a thread, then pass its create method to be executed in the new thread. I'm not sure what's happening, but I suspect that something is erroring and the stacktrace is being suppressed by something.
Also, you need to delete the space after the for statement -- it's significant, and it should be throwing you a syntax error about an unexpected indent.
EDIT:
This version runs on my machine:
import _thread
class Thread:
def create(self):
id = _thread.get_ident()
for i in range(5):
print("HALLO", id)
return
_thread.start_new_thread(Thread().create, ())