I have a main database and a replica (they are the same in the test environment):
core_db = PooledPostgresqlExtDatabase(**DB_COFIG)
replica_db = PooledPostgresqlExtDatabase(**DB_REPLICA_COFIG)
A controllers that executes a query in different databases depending on the model
class BaseController:
def _get_logs(self):
query = self.model.select()
if is_instance(self.model, ModelToReplica)
query = query.bind(replica_db)
return list(query)
class ReplicaExampleController(BaseLogsController):
model = ModelToReplica
def process(self):
return self._get_logs()
class BaseExampleController(BaseLogsController):
model = BaseModel
def process(self):
return self._get_logs()
Controllers are linked to two urls:
/get_core_result/ # Returns the result from BaseExampleController (core_db)
/get_replica_result/ # Returns the result from ReplicaExampleController (replica_db)
I want to check that each of the corners accesses the right base. And I know that the reference to the database object is stored in the request object. How do I get it from the test? I'm using a PyTest. I understand that I probably need to use mock, but I don't understand how.
Unfortunately, this is all I have so far:
class TestSwitchDB:
def test_switch_db_to_replica():
url_core = url_for('core_db_controller')
core_result = self.client.get(url_core)
url_replica = url_for('replica_db_controller')
replica_result = self.client.get(url_replica)
In test_switch_db_to_replica you can mock the db query and patch it, something
with patch(BaseController._get_logs) as mock_get_logs:
mock_get_logs.return_value="your expected return"
As a reference: https://docs.python.org/3/library/unittest.mock-examples.html
Related
In google app engine, say I have a Parent and a Child Entity:
class Parent(ndb.Model):
pass
class Child(ndb.Model):
parent_key = ndb.KeyProperty(indexed = True)
... other properties I don't need to fetch ...
I have a list of parents' keys, say parents_list, and I'm trying to answer efficiently: what parent in parents_list has a child.
Ideally, I would run this query:
children_list = Child.query().filter(Child.parent_key = parents_list).fetch(projection = 'parent_key')
It does not work because of the projection property (parent_key) being in the equality filter. So I would have to retrieve all properties, which seems inefficient.
Is there a way to efficiently solve this?
Your child model should actually be
class Child(ndb.Model):
parent_key = ndb.KeyProperty(kind="Parent", indexed = True)
If you were doing this in Python2, you could use an ndb_tasklet (see code below; note that I haven't executed this code myself so it's not guaranteed to work; it's just here to serve as a guide but I have used tasklets in the past before). If python3, try and create async queries
class Parent(ndb.Model):
#classmethod
def parentsWithChildren(cls, parents_list):
#ndb.tasklet
def child_callback(parent_key):
q = Child.query(Child.parent_key == parent_key)
output = yield q.fetch_async(projection = 'parent_key')
raise ndb.Return ((parentKey, output))
# For each key in parents_list, invoke the callback - child_callback which returns a result only if the parent_key matches
# ndb.tasklet is asynchronous so code is run in parallel
final_output = [ child_callback(a) for a in parents_list]
return final_output
I am building an Open Source Project, Python MongoDB ORM (for Flask especially) using flask_pymongo and I am kind of stuck at building dynamic conditions.
Below code is what I have written in corresponding files
Model.py
from app.database import Database
class Model:
conditions = {"and":[], "or":[], "in":[]}
operators = {
"!=": "$ne",
"<": "$lt",
">": "$gt",
"<=": "$lte",
">=": "$gte",
"in": "$in",
"not in":"$nin",
"and": "$and",
"or": "$or"
}
def __init__(self):
# collection property from User class
# Database class takes collection to fire MongoDB queries
self.db = Database(self.collection)
def where(self, field, operator, value=None):
if value is None:
# to enable Model.where("first_name", "John")
value = operator
operator = "="
self._handle_condition("and", field, operator, value)
# to enable Model.where().where_or() and etc
return self
def where_or(self, field, operator, value=None):
if value is None:
# to enable Model.where("first_name", "John")
value = operator
operator = "="
self._handle_condition("or", field, operator, value)
# to enable Model.where().where_or() and etc
return self
def _handle_condition(self, type, field, operator, value):
self.conditions[type].append({"field":field, "operator":operator, value:value})
def get(self):
filetrs = {}
for type in self.conditions:
filetrs[self.operators[type]] = []
for condition in self.conditions[type]:
if condition["operator"] == "=":
filter = {condition["field"]:condition["value"]}
else:
filter = {condition["field"]:{self.operators[condition["operator"]]:condition["value"]}}
filetrs[self.operators[type]].append(filter)
return self.db.find(filters)
User.py
from app.Model import Model
class UserModel(Model):
# MongoDB collection name
collection = "users"
def __init__(self):
Model.__init__(self)
User = UserModel()
What I want to achieve is, from UserController.py where User.py is imported and used like the mentioned code.
Where multiple conditions are being added using where and where_or Model methods, get methods is parsing all the conditions and passing as filter to find method
UserController.py
from app.User import User
class UserController:
def index(self):
# Should return all the users where _id is not blank or their first_name is equal to John
return User.where("_id", "!=", "").where_or("first_name", "John").get()
The problem is this is not working at it should be, it seems working fine for any one condition, where or where_or but when I try to add multiple where and where_or conditions it is not working.
Your help is really appreciated.
PS: This question seems to have lots of code but to make you understand the complete scenario I had to, please feel free to comment if you still need any clarifications.
Eagerly looking forward.
I am attempting to push data from a DJANGO view into the Tables object, passing it through as an argument. In this case, I would like to pass a variable called doc_id into a Tables2 object called tableName
In this example, I have set doc_id as 1, and pass it into the
View
def editorView(request):
doc_id = 1
table = tableName(UserProfile.objects.filter(), doc_id=doc_id)
Table
class tableName(tables.Table):
tbl_doc_id = None ## Creating a temporary variable
def __init__(self, *args, **kwargs):
temp = kwargs.pop("doc_id") ## Grab doc_ID from kwargs
super(tableName, self).__init__(*args, **kwargs)
self.tbl_doc_id = temp ## Assign to self.tbl_doc_id for use later
### Do something with tbl_doc_id
modelFilter = model.objects.filter(pk = tbl_doc_id)
When running the debugger, I can see that tbl_doc_id is still assigned as None, rather than 1.
What is the correct way to pass arguments into a Tables2 instance? Is it possible?
EDIT: Adding more information for context.
In the real world scenario, I have a view. That view takes an argument from the URL called doc_id. That doc_id is used to grab an object from a model called 'MaterialCollection', and return it as 'mc'.
'mc' is then passed into the table
View
def editorView(request, doc_id):
try:
mc = MaterialCollection.objects.get(pk = doc_id)
except Material.DoesNotExist:
raise Http404("Document does not exist")
config = RequestConfig(request)
unnassigned_User_Table = unassignedUserTable(UserProfile.objects.filter(), mc=mc)
... Other code + Render ...
From my table, I create a custom LinkColumn. That linkColumn is used to construct a URL based upon a number of Attributes from the model 'UserProfile', and from mc.
Table
class unassignedUserTable(tables.Table):
mc = None
def __init__(self, *args, **kwargs):
temp_mc = kwargs.pop("mc")
super(unassignedUserTable, self).__init__(*args, **kwargs)
self.mc = temp_mc
current_Assignment = "NONE"
new_Assignment = "AS"
assign_Reviewer = tables.LinkColumn('change_Review_AssignmentURL' , args=[ A('user'), current_Assignment, new_Assignment, mc, A('id')], empty_values=(), attrs={'class': 'btn btn-success'})
class Meta:
model = UserProfile
... Setup excludes/sequence/attributes...
In this particular instance. mc has a FK to UserProfile (in a 1:M) relationship.
I see that the name of your table class is tableName so if you want __init__ to work as expected please change the line:
super(unassignedUsers, self).__init__(*args, **kwargs)
to
super(tableName, self).__init__(*args, **kwargs)
Beyond this obvious problem, there are some more issues with your code:
Your classes must start with a capital letter (TableName instead of tableName)
Your table classes should end end with -Table (for example NameTable)
I am using django-tables2 for many years and never needed to pass something in __init__ as you are doing here. Are you sure that you really need to do this?
If you want to filter the table's data the filtering must be done to your view - the table will get the filtered data to display.
I have a script that takes a list of metrics as an input, and then fetches those metrics from the database to perform various operations with them.
My problem is that different clients get different subsets of the metrics, but I don't want to write a new IF block every time we add a new client. So right now, I have a large IF block that calls different functions based on whether the corresponding metric is in the list. What is the most elegant or Pythonic way of handling this?
Setup and function definitions:
clientOne = ['churn','penetration','bounce']
clientTwo = ['engagement','bounce']
def calcChurn(clientId):
churn = cursor.execute(sql to get churn)
[...]
return churn
def calcEngagement(clientId):
engagement = cursor.execute(sql to get engagement)
[...]
return engagement
Imagine three other functions in a similar format, so there is one function that corresponds to each unique metric. Now here is the block of code in the script that takes the list of metrics:
def scriptName(client, clientId):
if churn in client:
churn = calcChurn(clientId)
if engagement in client:
engagement = calcEngagement(clientId)
if penetration in client:
[...]
Generally, you'd create a mapping of names to functions and use that to calculate the stuff you want:
client_action_map = {
'churn': calcChurn,
'engagement': calcEngagement,
...
}
def scriptName(actions, clientId):
results = {}
for action in actions:
results[action] = client_action_map[action](clientId)
return results
You can create a class with static methods and use getattr to get the correct method. It's similar to what mgilson suggests but you essentially get the dict creation for free:
class Calculators:
#staticmethod
def calcChurn():
print("called calcChurn")
#staticmethod
def calcEngagement():
print("called calcEngagement")
#staticmethod
def calcPenetration():
print("called calcPenetration")
stats = ["churn", "engagement", "penetration", "churn", "churn", "engagement", "undefined"]
def capitalise(str):
return str[0].upper() + str[1:]
for stat in stats:
try:
getattr(Calculators, "calc" + capitalise(stat))()
except AttributeError as e:
print("Unknown statistic: " + stat)
called calcChurn
called calcEngagement
called calcPenetration
called calcChurn
called calcChurn
called calcEngagement
Unknown statistic: undefined
Perhaps it might make sense to encapsulate the required calls inside an object.
If it makes sense for your clients to be object and especially if many clients call the same set of functions to obtain metrics, then you could create a set of Client sub classes, which call a predefined set of the functions to obtain metrics.
It's a bit heavier than the mapping dict.
''' Stand alone functions for sql commands.
These definitions however dont really do anything.
'''
def calc_churn(clientId):
return 'result for calc_churn'
def calc_engagement(clientId):
return 'result for calc_engagement'
''' Client base object '''
class Client(object):
''' Base object allows list of functions
to be stored in client subclasses'''
def __init__(self, id):
self.id = id
self.metrics = []
self.args = []
def add_metrics(self, metrics, *args):
self.metrics.extend(metrics)
self.args = args
def execute_metrics(self):
return {m.__name__: m(*self.args) for m in self.metrics}
''' Specific sub classes '''
class Client1(Client):
def __init__(self, id):
''' define which methods are called for this class'''
super(Client1, self).__init__(id)
self.add_metrics([calc_churn], id)
class Client2(Client):
def __init__(self, id):
''' define which methods are called for this class'''
super(Client2, self).__init__(id)
self.add_metrics([calc_churn, calc_engagement], id)
''' create client objects and '''
c1 = Client1(1)
c2 = Client2(2)
for client in [c1, c2]:
print client.execute_metrics()
The result you will get from execute_metrics is a dict mapping the function name to its results for that client.
I have two implementations of database queue (they use different tables) and want them to use objects of the same class. So, they both look really similar:
class AbstractDBQueue(object):
def __init__(self, tablename):
self.tablename = tablename
self.metadata = MetaData()
self.engine = create_engine('mysql+mysqldb://%s:%s#%s:%d/%s' % (
settings.DATABASE.get('USER'),
settings.DATABASE.get('PASSWORD'),
settings.DATABASE.get('HOST') or '127.0.0.1',
settings.DATABASE.get('PORT') or 3306,
settings.DATABASE.get('NAME')
), encoding='cp1251', echo=True, pool_recycle=7200)
self.metadata.bind = self.engine
self.session = sessionmaker(bind=self.engine)()
def setup_table(self, table, entity_name):
self.table = table
newcls = type(entity_name, (SMSMessage, ), {})
mapper(newcls, table)
return newcls
def put(self, message=None, many_messages=[]):
if message:
self.session.add(message)
else:
for m in many_messages:
self.session.add(m)
self.session.commit()
def get(self, limit=None):
if limit:
q = self.session.query(self.SMSClass).limit(limit)
else:
q = self.session.query(self.SMSClass)
smslist = []
for sms in q:
smslist.append(sms)
self.session.expunge_all()
return smslist
class DBQueue(AbstractDBQueue):
"""
MySQL database driver with queue interface
"""
def __init__(self):
self.tablename = settings.DATABASE.get('QUEUE_TABLE')
super(DBQueue, self).__init__(self.tablename)
self.logger = logging.getLogger('DBQueue')
self.SMSClass = self.setup_table(Table(self.tablename, self.metadata, autoload=True), "SMSQueue")
class DBWorkerQueue(AbstractDBQueue):
"""
MySQL database driver with queue interface for separate workers queue
"""
def __init__(self):
self.tablename = settings.DATABASE.get('WORKER_TABLE')
super(DBWorkerQueue, self).__init__(self.tablename)
self.logger = logging.getLogger('DBQueue')
self.SMSClass = self.setup_table(Table(self.tablename, self.metadata, autoload=True), "SMSWorkerQueue")
def _install(self):
self.metadata.create_all(self.engine)
SMSMessage is the name of the class I want to use. The map_class_to_table() function is a hack I've found in SQLAlchemy documentation: http://www.sqlalchemy.org/trac/wiki/UsageRecipes/EntityName
But it doesn't seems to help - when the first queue instance maps SMSMessage to it's table, then all objects I pass to second queue's put() are implicitly casted to first queue's mapped class, and second database is still empty after session.commit().
I need to use both queues at the same time, maybe even using threads (I think, pool connection will be useful), but I just can't make this work. Could you help, please?
I think your problem relates to the tablename variable. It's a class variable which gets defined when you create the class, and then does not change. So, it will be the same for both of your instances, when they access it with self.tablename. To fix this, move it inside the init function, and make it a self.tablename. That will initialize it each time you create a new object.