I'm working on a project in Tornado that relies heavily on the asynchronous features of the library. By following the chat demo, I've managed to get long-polling working with my application, however I seem to have run into a problem with the way it all works.
Basically what I want to do is be able to call a function on the UpdateManager class and have it finish the asynchronous request for any callbacks in the waiting list. Here's some code to explain what I mean:
update.py:
class UpdateManager(object):
waiters = []
attrs = []
other_attrs = []
def set_attr(self, attr):
self.attrs.append(attr)
def set_other_attr(self, attr):
self.other_attrs.append(attr)
def add_callback(self, cb):
self.waiters.append(cb)
def send(self):
for cb in self.waiters:
cb(self.attrs, self.other_attrs)
class LongPoll(tornado.web.RequestHandler, UpdateManager):
#tornado.web.asynchronous
def get(self):
self.add_callback(self.finish_request)
def finish_request(self, attrs, other_attrs):
# Render some JSON to give the client, etc...
class SetSomething(tornado.web.RequestHandler):
def post(self):
# Handle the stuff...
self.add_attr(some_attr)
(There's more code implementing the URL handlers/server and such, however I don't believe that's necessary for this question)
So what I want to do is make it so I can call UpdateManager.send from another place in my application and still have it send the data to the waiting clients. The problem is that when you try to do this:
from update import UpdateManager
UpdateManager.send()
it only gets the UpdateManager class, not the instance of it that is holding user callbacks. So my question is: is there any way to create a persistent object with Tornado that will allow me to share a single instance of UpdateManager throughout my application?
Don't use instance methods - use class methods (after all, you're already using class attributes, you just might not realize it). That way, you don't have to instantiate the object, and can instead just call the methods of the class itself, which acts as a singleton:
class UpdateManager(object):
waiters = []
attrs = []
other_attrs = []
#classmethod
def set_attr(cls, attr):
cls.attrs.append(attr)
#classmethod
def set_other_attr(cls, attr):
cls.other_attrs.append(attr)
#classmethod
def add_callback(cls, cb):
cls.waiters.append(cb)
#classmethod
def send(cls):
for cb in cls.waiters:
cb(cls.attrs, cls.other_attrs)
This will make...
from update import UpdateManager
UpdateManager.send()
work as you desire it to.
Related
I am building a software and am using one class to only store data:
class Data():
data = [1,2,3]
Now the data in this class can be accessed and changed from other classes without instantiating the Data class which is exactly what I need.
In order to update the of the software properly I have to call functions in other classes whenever the data changes. I looked at the observer pattern in python but could not get it to work without making data an attribute of the class that's only available when instantiated. In other words all the observer pattern implementations I found required:
class Data():
def __init__(self):
self.data = [1,2,3]
Obviously, if Data is my publisher/observable it needs to be instantiated once to get the functionality (as far as I understand) but I am looking for an implementation like:
class Data():
data = [1,2,3]
def __init__(self):
self.subscribers = {}
def register(self, who, callback):
self.subscribers[who] = callback
def dispatch(self):
for susbriber, callback in self.subscribers.items():
callback()
For the sake of the example let's use this other class as Subscriber/Observer that can also change the data with another function. As this will be the main class handling the software, this is where I instantiate Data to get the observer behavior. It is important however that I would not have to instantiate it only to get data as data will be changed from a lot of other classes:
class B():
def __init__(self):
self.data = Data()
self.data.register(self, self.print_something)
def print_something(self):
print("Notification Received")
def change_data(self):
Data.data.append(100)
My question now is, how to automatically send the notification from the Publisher/Observable whenever data gets changed in any way?
I am running python 3.8 on Windows 10.
I am trying to run multiple tasks in queue. The tasks come on user input. What i tried was creating a singleton class with ThreadPoolExecutor property and adding tasks into it. The tasks are added fine, but it looks like only the first addition of set of tasks works. The following are added but not executed.
class WebsiteTagScrapper:
class __WebsiteTagScrapper:
def __init__(self):
self.executor = ThreadPoolExecutor(max_workers=5)
instance = None
def __new__(cls): # __new__ always a classmethod
if not WebsiteTagScrapper.instance:
WebsiteTagScrapper.instance = WebsiteTagScrapper.__WebsiteTagScrapper()
return WebsiteTagScrapper.instance
I used multiprocess in one of my project without using celery, cause i think it was overkill for my use.
Maybe you could do something like this:
from multiprocessing import Process
class MyQueuProcess(Process):
def __init__(self):
super(MyQueuProcess, self).__init__()
self.tasks = []
def add_task(self, task):
self.tasks.append(task)
def run(self):
for task in self.tasks:
#Do your task
You just have to create an instance in your view, set up your task and then run(). Also if you need to access your database, you will need to import django in your child and then make a django.setup().
As it was unclear earlier I am posting this scenario:
class Scraper:
def __init__(self,url):
self.start_page = url
def parse_html(self):
pass
def get_all_links(self):
pass
def run(self):
#parse html, get all links, parse them and when done...
return links
Now in a task queue like rq
from rq import Queue
from worker import conn
q = Queue(connection=conn)
result = q.enqueue(what_function, 'http://stackoverflow.com')
I want to know what this what_function would be? I remembered Django does something similar with their CBVs so I used that analogy but it wasn't so clear.
I have a class like
class A:
def run(self,arg):
#do something
I need to past this to a task queue, so I can do something like
a = A()
b = a.run
# q is the queue object
q.enqueue(b,some_arg)
I'd want to know what other method is there to do this, for example, Django does it in their Class Based Views,
class YourListView(ListView):
#code for your view
which is eventually passed as a function
your_view = YourListView.as_view()
How is it done?
Edit: to elaborate, django's class based views are converted to functions because the argument in the pattern function expects a function. Similarly, you might have a function which accepts the following argument
task_queue(callback_function, *parameters):
#add to queue and return result when done
but the functionality of callback_function might have been mostly implemented in a class, which has a run() method via which the process is ran.
I think you're describing a classmethod:
class MyClass(object):
#classmethod
def as_view(cls):
'''method intended to be called on the class, not an instance'''
return cls(instantiation, args)
which could be used like this:
call_later = MyClass.as_view
and later called:
call_later()
Most frequently, class methods are used to instantiate a new instance, for example, dict's fromkeys classmethod:
dict.fromkeys(['foo', 'bar'])
returns a new dict instance:
{'foo': None, 'bar': None}
Update
In your example,
result = q.enqueue(what_function, 'http://stackoverflow.com')
you want to know what_function could go there. I saw a very similar example from the RQ home page. That's got to be your own implementation. It's going to be something you can call with your code. It's only going to be called with that argument once, so if using a class, your __init__ should look more like this, if you want to use Scraper for your what_function replacement:
class Scraper:
def __init__(self,url):
self.start_page = url
self.run()
# etc...
If you want to use a class method, that might look like this:
class Scraper:
def __init__(self,url):
self.start_page = url
def parse_html(self):
pass
def get_all_links(self):
pass
#classmethod
def run(cls, url):
instance = cls(url)
#parse html, get all links, parse them and when done...
return links
And then your what_function would be Scraper.run.
I have a UUT class which instantiates Worker objects, and calls their do_stuff() method.
The Worker objects uses a Provider object for two things:
Calls methods on the provider object to do some stuff
Gets notifications from the provider by subscribing a method with the provider's events
When a worker gets a notification, it processes it, an notifies the UUT object, which in reponse can create more Worker objects.
I've already tested each class on its own, and I want to test UUT+Worker together. For that, I intend to mock-out Provider.
import mock
import unittest
import provider
class Worker():
def __init__(self, *args):
resource.default_resource.subscribe('on_spam', self._on_spam) # I'm going to patch 'resource.default_resource'
def do_stuff(self):
self.resource.do_stuff()
def _on_spam(self, message):
self._tell_uut_to_create_more_workers(message['num_of_new_workers_to_create'])
class UUT():
def __init__(self, *args):
self._workers = []
def gen_worker_and_do_stuff(self, *args)
worker = Worker(*args)
self._workers.append(resource)
worker.do_stuff()
class TestCase1(unittest.TestCase):
#mock.patch('resource.default_resource', spec_set=resource.Resource)
def test_1(self, mock_resource):
uut = UUT()
uut.gen_worker_and_do_stuff('Egg') # <-- say I automagically grabbed the resulting Worker into self.workers
self.workers[0]._on_spam({'num_of_new_workers_to_create':5}) # <-- I also want to get hold of the newly-created workers
Is there a way to grab the worker objects generated by uut, without directly accessing the _workers list in uut (which is an implementation detail)?
I guess I can do it in Worker.__init__, where the worker subscribes to provider events, so I guess the question reduces to:
How to I extract the self in the callee, when calling resource.default_resource.subscribe('on_spam', self._on_spam)?
As an application of the Dependency Inversion principle, I'd pass the Worker class as a dependency to UUT:
class UUT():
def __init__(self, make_worker=Worker):
self._workers = []
self._make_worker = make_worker
def gen_worker_and_connect(self, *args)
worker = self._make_worker(*args)
self._workers.append(resource)
worker.connect()
Then provide anything you want from the test instead of Worker. This own function could share the created object with the test scope. Besides solving this particular problem, that would also make the dependency explicit and independent of the UUT implementation. And you would not need to mock the resource thing as well, which makes the test dependent on things unrelated to the class under test.
I'm trying to write a library that will register an arbitrary list of service calls from multiple service endpoints to a container. I intend to implement the service calls in classes written one per service. Is there a way to maintain the boundedness of the methods from the service classes when registering them to the container (so they will still have access to the instance data of their owning object instance), or must I register the whole object then write some sort of pass through in the container class with __getattr__ or some such to access the methods within instance context?
container:
class ServiceCalls(object):
def __init__(self):
self._service_calls = {}
def register_call(self, name, call):
if name not in self._service_calls:
self._service_calls[name] = call
def __getattr__(self, name):
if name in self._service_calls:
return self._service_calls[name]
services:
class FooSvc(object):
def __init__(self, endpoint):
self.endpoint = endpoint
def fooize(self, *args, **kwargs):
#call fooize service call with args/kwargs utilizing self.endpoint
def fooify(self, *args, **kwargs):
#call fooify service call with args/kwargs utilizing self.endpoint
class BarSvc(object):
def __init__(self, endpoint):
self.endpoint = endpoint
def barize(self, *args, **kwargs):
#call barize service call with args/kwargs utilizing self.endpoint
def barify(self, *args, **kwargs):
#call barify service call with args/kwargs utilizing self.endpoint
implementation code:
foosvc = FooSvc('fooendpoint')
barsvc = BarSvc('barendpoint')
calls = ServiceCalls()
calls.register('fooize', foosvc.fooize)
calls.register('fooify', foosvc.fooify)
calls.register('barize', barsvc.barize)
calls.register('barify', barsvc.barify)
calls.fooize(args)
I think this answers your question:
In [2]: f = 1 .__add__
In [3]: f(3)
Out[3]: 4
You won't need the staticmethod function when adding these functions to classes, because they are effectively already "staticed".
What you are trying to do will work fine, as you can see by running your own code. :)
The object foosvc.fooize is called a "bound method" in Python, and it contains both, a reference to foosvc and to the function FooSvc.fooize. If you call the bound method, the reference to self will be passed implicitly as the first paramater.
On a side note, __getattr__() shouldn't silently return None for invalid attribute names. Better use this:
def __getattr__(self, name):
try:
return self._service_calls[name]
except KeyError:
raise AttributeError
I don't understand the use case for this -- it seems to me that the easy, simple, idiomatic way to accomplish this is to just pass in an object.
But: program to the interface, not the implementation. Only assume that the object has the method you need -- don't touch the internals or any other methods.