So this is kind of a python design question + multiple heritance. I'm working on a program of mine and I've ran into an issue I can't figure out a decent way of solving.
To keep it simple. The software scans a log event file generated from another program. Initially it creates and stores each event in a representative event object. But I want to access them quickly and with a more robust language so I'm loading them into a SQL DB after doing a little processing on each event, so theres more data than previous. When I query the DB I'm wanting to recreate an object for each entry representative of the event so its easier to work with.
The problem I'm running into is that I want to avoid a lot of duplicate code and technically I should be able to just reuse some of the code in the original classes for each event. Example:
class AbstractEvent:
__init__(user, time)
getTime()
getUser()
class MessageEvent(AbstractEvent):
__init__(user,time,msg)
getMessage()
class VideoEvent(AbstractEvent):
pass
But, there is extra data after its gone into the DB so there needs to be new subclasses:
class AbstractEventDB(AbstractEvent):
__init__(user, time, time_epoch)
getTimeEpoch()
(static/classmethod) fromRowResult(row)
class MessageEventDB(AbstractEventDB, MessageEvent):
__init__(user, time, msg, time_epoch, tags)
getTags()
(static/classmethod) fromRowResult(row)
class VideoEventDB(AbstractEventDB, VideoEvent):
pass
This is a simpler version than whats happening, but it shows some of what does happen. I change long form time stamps from the log file into epoch timestamps when they go into the DB and various tags are added on message events but other events have nothing extra really beyond the timestamp change.
The above is ideally how I would like to format it, but the problem I've ran into is that the call signatures are completely different on the DB object side compared to the Simple Event side; so when I try to call super() I get an error about expected arguements missing.
I was hoping someone might be able to offer some advice on how to structure it and avoid duplicating code 10-20 times over, particularly in the fromRowResult (a factory method). Help much appreciated.
I thin what you are looking for is a Python implementation for the decorator design pattern.
http://en.wikipedia.org/wiki/Decorator_pattern
The main idea is to replace multiple inheritance with inheritance + composition:
class AbstractEvent(object):
def __init__(self, user, time):
self.user = user
self.time = time
class MessageEvent(AbstractEvent):
def __init__(self, user, time, msg):
super(MessageEvent, self).__init__(user, time)
self.msg = msg
class AbstractEventDBDecorator(object):
def __init__(self, event, time_epoch):
# event is a member of the class. Using dynamic typing, the event member will
# be a AbstractEvent or a MessageEvent at runtime.
self.event = event
self.time_epoch = time_epoch
#classmethod
def fromRowResult(cls, row):
abstract_event = AbstractEvent(row.user, row.time)
abstract_event_db = AbstractEventDBDecorator(abstract_event, row.time_epoch)
return abstract_event_db
class MessageEventDB(AbstractEventDBDecorator):
def __init__(self, message_event, time_epoch, tags):
super(MessageEventDB, self).__init__(message_event, time_epoch)
self.tags = tags
#classmethod
def fromRowResult(cls, row):
message_event = MessageEvent(row.user, row.time, row.msg)
message_event_db = MessageEventDB(message_event, row.time_epoch, row.tags)
return message_event_db
class Row:
def __init__(self, user, time, msg, time_epoch, tags):
self.user = user
self.time = time
self.msg = msg
self.time_epoch = time_epoch
self.tags = tags
if __name__ == "__main__":
me = MessageEvent("user", "time", "msg")
r = Row("user", "time", "Message", "time_epoch", "tags")
med = MessageEventDB.fromRowResult(r)
print med.event.msg
Related
I am building a software and am using one class to only store data:
class Data():
data = [1,2,3]
Now the data in this class can be accessed and changed from other classes without instantiating the Data class which is exactly what I need.
In order to update the of the software properly I have to call functions in other classes whenever the data changes. I looked at the observer pattern in python but could not get it to work without making data an attribute of the class that's only available when instantiated. In other words all the observer pattern implementations I found required:
class Data():
def __init__(self):
self.data = [1,2,3]
Obviously, if Data is my publisher/observable it needs to be instantiated once to get the functionality (as far as I understand) but I am looking for an implementation like:
class Data():
data = [1,2,3]
def __init__(self):
self.subscribers = {}
def register(self, who, callback):
self.subscribers[who] = callback
def dispatch(self):
for susbriber, callback in self.subscribers.items():
callback()
For the sake of the example let's use this other class as Subscriber/Observer that can also change the data with another function. As this will be the main class handling the software, this is where I instantiate Data to get the observer behavior. It is important however that I would not have to instantiate it only to get data as data will be changed from a lot of other classes:
class B():
def __init__(self):
self.data = Data()
self.data.register(self, self.print_something)
def print_something(self):
print("Notification Received")
def change_data(self):
Data.data.append(100)
My question now is, how to automatically send the notification from the Publisher/Observable whenever data gets changed in any way?
I am running python 3.8 on Windows 10.
ok, I am not even entirely sure if my title is completely accurate as I completely do not understand class inheritance and instances at that moment but understand it is something that I need or should grasp moving forward.
Background: attempting to create a custom importer for my bank to be used with the popular Beancount/fava double entry ledger accounting system. I originally reported to fava as a bug but then realized its not a bug and its more my lack of general understanding of Python classes so thought it would be better to post here.
So...I have created the following import script file which as I understand is a sub class of beancount csv.Importer (https://github.com/beancount/beancount/blob/master/beancount/ingest/importers/csv.py) which is a sub class of beancount Importer (https://github.com/beancount/beancount/blob/master/beancount/ingest/importer.py)
In my importer I over ride 2 methods of csv.Importer, name() and file_account(). My goal is to derive the source account associated to input file based on file name and dictionary look-up. The extract() method I do not wish to over-ride in my sub class, however in the csv.Importer extract() method there is reference to self.account that represents the source account to use for extracted transactions. Currently the way my script is if I feed it a file named 'SIMPLII_9999_2018-01-01.csv' the account will be properly derived as 'Assets:Simplii:Chequing-9999'. However, if I stop short of actually importing the transactions in fava and instead attempt to extract the transactions again from the same file the derived account then becomes 'Assets:Simplii:Chequing-9999 :Chequing-9999'.
What I am trying to do is derive the source account from the input file and pass this information as the self.account variable in the parent class (csv.Importer) for my class instance (I think). What is it that I am doing wrong in my class that is causing the derived source account to be carried over to the next instance?
#!/usr/bin/env python3
from beancount.ingest import extract
from beancount.ingest.importers import csv
from beancount.ingest import cache
from beancount.ingest import regression
import re
from os import path
from smart_importer.predict_postings import PredictPostings
class SimpliiImporter(csv.Importer):
'''
Importer for the Simplii bank.
Note: This undecorated class can be regression-tested with
beancount.ingest.regression.compare_sample_files
'''
config = {csv.Col.DATE: 'Date',
csv.Col.PAYEE: 'Transaction Details',
csv.Col.AMOUNT_DEBIT: 'Funds Out',
csv.Col.AMOUNT_CREDIT: 'Funds In'}
account_map = {'9999':'Chequing-9999'}
def __init__(self, *, account, account_map=account_map):
self.account_map = account_map
self.account = 'Assets:Simplii'
super().__init__(
self.config,
self.account,
'CAD',
['Filename: .*SIMPLII_\d{4}_.*\.csv',
'Contents:\n.*Date, Transaction Details, Funds Out, Funds In'],
institution='Simplii'
)
def name(self):
cls = self.__class__
return '{}.{}'.format(cls.__module__, cls.__name__)
def file_account(self, file):
__account = None
if file:
m = re.match(r'.+SIMPLII_(\d{4})_.*', file.name)[1]
if m:
sub_account = self.account_map.get(m)
if sub_account:
__account = self.account + ':' + sub_account
return __account
def extract(self, file):
self.account = self.file_account(file)
return super().extract(file)
#PredictPostings(training_data='/beancount/personal.beancount')
class SmartSimpliiImporter(SimpliiImporter):
'''
A smart version of the Simplii importer.
'''
pass
so I have managed to get this working however I don't think its the proper way to do it...
I changed the extract function like this
def extract(self, file):
self.account = self.file_account(file)
postings = super().extract(file)
self.account = 'Assets:Simplii'
return postings
basically I set the self.account to the value I need to, call the parent class extract function saving results to variable, reset the self.account variable and return results. Seems more of a work around than the proper way but at least its here in case it helps someone else out...
I am using SimPy for a time-based simulation.
I have a Messager process which listens for a specific event, and returns the parameters of that event when it occurs:
class Messager(Process):
def __init__(self):
Process.__init__(self,'messager')
def monitor(self):
while True:
yield waitevent, self, messageEvent
print messageEvent.signalparam
messager = Messager()
activate(messager,messager.monitor())
this works fine. However, in reality there is more than one Event which should be handled by the Messager, and I want the Messager to return the signalparam for the causative Event in a list of events:
messageEvents = [Event1,Event2,Event3]
class Messager(Process):
def __init__(self):
Process.__init__(self,'messager')
def monitor(self):
while True:
yield waitevent, self, messageEvents
.....
how do I access the signalparam of the Event which has occurred?
Looking at the source of simpy, and specifically of the class Simevent, if you can somehow cast the event to the Simevent class then you should be able to get the attribute signal param out of it. Not sure if this is possible, but hopefully you know a way to do this. Good luck.
EDIT:
Just thought of this too, you could edit your simpy and make the Simevent class have a function to get the signalparam. FYI if you want to do this, the simevent class is located in the lib.py file.
I'm working on a project in Tornado that relies heavily on the asynchronous features of the library. By following the chat demo, I've managed to get long-polling working with my application, however I seem to have run into a problem with the way it all works.
Basically what I want to do is be able to call a function on the UpdateManager class and have it finish the asynchronous request for any callbacks in the waiting list. Here's some code to explain what I mean:
update.py:
class UpdateManager(object):
waiters = []
attrs = []
other_attrs = []
def set_attr(self, attr):
self.attrs.append(attr)
def set_other_attr(self, attr):
self.other_attrs.append(attr)
def add_callback(self, cb):
self.waiters.append(cb)
def send(self):
for cb in self.waiters:
cb(self.attrs, self.other_attrs)
class LongPoll(tornado.web.RequestHandler, UpdateManager):
#tornado.web.asynchronous
def get(self):
self.add_callback(self.finish_request)
def finish_request(self, attrs, other_attrs):
# Render some JSON to give the client, etc...
class SetSomething(tornado.web.RequestHandler):
def post(self):
# Handle the stuff...
self.add_attr(some_attr)
(There's more code implementing the URL handlers/server and such, however I don't believe that's necessary for this question)
So what I want to do is make it so I can call UpdateManager.send from another place in my application and still have it send the data to the waiting clients. The problem is that when you try to do this:
from update import UpdateManager
UpdateManager.send()
it only gets the UpdateManager class, not the instance of it that is holding user callbacks. So my question is: is there any way to create a persistent object with Tornado that will allow me to share a single instance of UpdateManager throughout my application?
Don't use instance methods - use class methods (after all, you're already using class attributes, you just might not realize it). That way, you don't have to instantiate the object, and can instead just call the methods of the class itself, which acts as a singleton:
class UpdateManager(object):
waiters = []
attrs = []
other_attrs = []
#classmethod
def set_attr(cls, attr):
cls.attrs.append(attr)
#classmethod
def set_other_attr(cls, attr):
cls.other_attrs.append(attr)
#classmethod
def add_callback(cls, cb):
cls.waiters.append(cb)
#classmethod
def send(cls):
for cb in cls.waiters:
cb(cls.attrs, cls.other_attrs)
This will make...
from update import UpdateManager
UpdateManager.send()
work as you desire it to.
In a twisted application I have a series of resource controller/manager classes that interact via the Observable pattern. Generally most observers will subscribe to a specific channel (ex. "foo.bar.entity2") but there are a few cases where I'd like to know about all event in a specific channel (ex. "foo.*" ) so I wrote something like the following:
from collections import defaultdict
class SimplePubSub(object):
def __init__(self):
self.subjects = defaultdict(list)
def subscribe(self, subject, callbackstr):
"""
for brevity, callbackstr would be a valid Python function or bound method but here is just a string
"""
self.subjects[subject].append(callbackstr)
def fire(self, subject):
"""
Again for brevity, fire would have *args, **kwargs or some other additional message arguments but not here
"""
if subject in self.subjects:
print "Firing callback %s" % subject
for callback in self.subjects[subject]:
print "callback %s" % callback
pubSub = SimplePubSub()
pubSub.subscribe('foo.bar', "foo.bar1")
pubSub.subscribe('foo.foo', "foo.foo1")
pubSub.subscribe('foo.ich.tier1', "foo.ich.tier3_1")
pubSub.subscribe('foo.ich.tier2', "foo.ich.tier2_1")
pubSub.subscribe('foo.ich.tier3', "foo.ich.tier2_1")
#Find everything that starts with foo
#say foo.bar is fired
firedSubject = "foo.bar"
pubSub.fire(firedSubject)
#outputs
#>>Firing callback foo.bar
#>>callback foo.bar1
#but let's say I want to add a callback for everything undr foo.ich
class GlobalPubSub(SimplePubSub):
def __init__(self):
self.globals = defaultdict(list)
super(GlobalPubSub, self).__init__()
def subscribe(self, subject, callback):
if subject.find("*") > -1:
#assumes global suscriptions would be like subject.foo.* and we want to catch all subject.foo's
self.globals[subject[:-2]].append(callback)
else:
super(GlobalPubSub, self).subscribe(subject, callback)
def fire(self, subject):
super(GlobalPubSub, self).fire(subject)
if self.globals:
for key in self.globals.iterkeys():
if subject.startswith(key):
for callback in self.globals[key]:
print "global callback says", callback
print "Now with global subscriptions"
print
pubSub = GlobalPubSub()
pubSub.subscribe('foo.bar', "foo.bar1")
pubSub.subscribe('foo.foo', "foo.foo1")
pubSub.subscribe('foo.ich.tier1', "foo.ich.tier3_1")
pubSub.subscribe('foo.ich.tier2', "foo.ich.tier2_1")
pubSub.subscribe('foo.ich.tier3', "foo.ich.tier2_1")
pubSub.subscribe("foo.ich.*", "All the ichs, all the time!")
#Find everything that starts with foo.ich
firedSubject = "foo.ich.tier2"
pubSub.fire(firedSubject)
#outputs
#>>Firing callback foo.bar
#>>callback foo.bar1
#>>Now with global subscriptions
#
#>>Firing callback foo.ich.tier2
#>>callback foo.ich.tier2_1
#>>global callback says All the ichs, all the time!
Is this as good as it gets without resorting to some sort of exotic construct ( tries for example )? I'm looking for an affirmation that I'm on the right track or a better alternative suggestion on a global subscription handler that's pure python ( no external libraries or services ).
This looks like you are on the right track to me. I was using PyPubSub with a wxPython app for a bit, and then ended up implementing my own "more simple" version that, at its root, looks very similar to what you've done here except with a few more bells and whistles that you'd probably end up implementing as you fill out your requirements.
The answer given here is also lot like what you have done as well.
This answer goes into examples that are a bit different approach.
There are a number existing libraries besides PyPubSub out there, such as pydispatch and blinker, that might be worth looking at for reference or ideas.