We Keep Coding

Mocking return value of a nested call in Python mock library

Brand new to this library
Here is the call stack of my mocked object
[call(),
call('test'),
call().instance('test'),
call().instance().database('test'),
call().instance().database().snapshot(),
call().instance().database().snapshot().__enter__(),
call().instance().database().snapshot().__enter__().execute_sql('SELECT * FROM users'),
call().instance().database().snapshot().__exit__(None, None, None),
call().instance().database().snapshot().__enter__().execute_sql().__iter__()]
Here is the code I have used
#mock.patch('testmodule.Client')
def test_read_with_query(self, mock_client):
mock = mock_client()
pipeline = TestPipeline()
records = pipeline | ReadFromSpanner(TEST_PROJECT_ID, TEST_INSTANCE_ID, self.database_id).with_query('SELECT * FROM users')
pipeline.run()
print mock_client.mock_calls
exit()
I want to mock this whole stack that eventually it gives me some fake data which I will provide as a return value.
The code being tested is
spanner_client = Client(self.project_id)
instance = spanner_client.instance(self.instance_id)
database = instance.database(self.database_id)
with database.snapshot() as snapshot:
results = snapshot.execute_sql(self.query)
So my requirements is that the results variable should contain the data I will provide.
How can I provide a return value to such a nested calls
Thanks

Create separate MagicMock instances for the instance, database and snapshot objects in the code under test. Use return_value to configure the return values of each method. Here is an example. I simplified the method under test to just be a free standing function called mut.
# test_module.py : the module under test
class Client:
pass
def mut(project_id, instance_id, database_id, query):
spanner_client = Client(project_id)
instance = spanner_client.instance(instance_id)
database = instance.database(database_id)
with database.snapshot() as snapshot:
results = snapshot.execute_sql(query)
return results
# test code (pytest)
from unittest.mock import MagicMock
from unittest import mock
from test_module import mut
#mock.patch('test_module.Client')
def test_read_with_query(mock_client_class):
mock_client = MagicMock()
mock_instance = MagicMock()
mock_database = MagicMock()
mock_snapshot = MagicMock()
expected = 'fake query results'
mock_client_class.return_value = mock_client
mock_client.instance.return_value = mock_instance
mock_instance.database.return_value = mock_database
mock_database.snapshot.return_value = mock_snapshot
mock_snapshot.execute_sql.return_value = expected
mock_snapshot.__enter__.return_value = mock_snapshot
observed = mut(29, 42, 77, 'select *')
mock_client_class.assert_called_once_with(29)
mock_client.instance.assert_called_once_with(42)
mock_instance.database.assert_called_once_with(77)
mock_database.snapshot.assert_called_once_with()
mock_snapshot.__enter__.assert_called_once_with()
mock_snapshot.execute_sql.assert_called_once_with('select *')
assert observed == expected
This test is kind of portly. Consider breaking it apart by using a fixture and a before function that sets up the mocks.

Either set the value directly to your Mock instance (those enters and exit should have not seen) with:
mock.return_value.instance.return_value.database.return_value.snapshot.return_value.execute_sql.return_value = MY_MOCKED_DATA
or patch and set return_value to target method, something like:
#mock.patch('database_engine.execute_sql', return_value=MY_MOCKED_DATA)

class instance method takes exactly two args, one given

I am having a bit of an issue. First off, I know that this code is able to stand alone and not be in a class but I would prefer that it is in a class. Second, when I run the code, I get this error TypeError: set_options() takes exactly 2 arguments (1 given) .
Here is my code. If anyone could point me in the right direction, I would appreciate it. I'm assuming that the set_options method isn't getting my jobj instance. Am I correct in assuming that and how would one go about fixing this? ps. I do have the correct imports and here is my py command at terminal python test.py radar 127.0.0.1 hashNumber testplan:speed
class TransferStuff(object):
tool = sys.argv[1]
target = sys.argv[2]
hash = sys.argv[3]
options = sys.argv[4]
def set_options(self, test_options):
option_arr = test_options.split(',')
new_arr = [i + ':{}'.format(i) for i in option_arr if ':' not in i]
for i in option_arr:
if ':' in i:
new_arr.append(i)
d = {}
for i in new_arr:
temp = i.split(':')
d[temp[0]] = temp[1]
return d
data = {'target': target, 'test': tool, 'HASH': hash,
'options': set_options(options)}
def write_to_json(self):
"""Serialize cli args and tool options in json format.
Write stream to json file.
"""
with open('envs.json', 'w') as fi:
json.dump(TransferStuff.data, fi)
if __name__ == "__main__":
try:
jobj = TransferStuff()
jobj.write_to_json()

Your method is inside a class, you need to create a instance of the class:
transfer_stuff_instance = TransferStuff()
And call the method with this instance:
transfer_stuff_instance.ser_options(options)

Python Type Dispatching with variables, is it possible?

I am developing a script that parses through a large number of IDs and want to build an abstraction that can hold all of these IDs. I also want this object to be able to be easily modified to add new IDs.
This is what I have so far:
class action: # abstraction for all IDs in an action
def __init__(self, dID=".", cID='.', dTp='.', dVs='.', mcID='.', aID='.', lID='.', pID='.', uID='.', uSe='.', udTp='.', componentID='.', eCx='.', eUr='.', eTp='.', rUrl='.', sec='.', oID='.', oVa='.', oCu='.', sID='.', saID='.', socNetworkUserID='.'):
self._dID = dID
self._cID = cID
self._dTp = dTp
self._dVs = dVs
self._mcID = mcID
self._aID = aID
self._lID = lID
self._pID = pID
self._uID = uID #Display and Insights
self._uSe = uSe
self._udTp = udTp
self._componentID = componentID
self._eCx = eCx #Display and Insights
self._eUr = eUr
self._eTp = eTp #Display and Insights
self._rUrl = rUrl
self._sec = sec
self._oID = oID #Display and Insights
self._oVa = oVa
self._oCu = oCu
self._sID = sID
self._saID = saID
self._socNetworkUserID = socNetworkUserID
self._empty_params = []
def insert_id_val(self, name, value): #Utility method
return
def insert_id_val_display(self, name, value):
item_map = {"dID" : _dID, "cID" : _cID, "dTp" : _dTp, "dVs" : _dVs,
"mcID" : _mcID, "aID" : _aID, "lID" : _lID, "pID" : _pID,
"uID" : _uID, "uSe" : _uSe, "udTp" : _udTp, "componentID" : _componentID,
"eCx" : _eCx,"eUr" : _eUr, "eTp" : _eTp, "rUrl" : _rUrl,
"sec" : _sec, "oID" : _oID, "oVa" : _oVa, "oCu" : _oCu,
"socNetworkUserID" : _socNetworkUserID, "_" : _empty_params}
self.item_map[name] = value
def insert_id_val_insights(self, name, value):
item_map = {"eTp" : self._eTp, "eCx" : self._eCx, "uID" : self._uID, "oID" : self._oID,
"sID" : self._sID, "saID" : self._saID}
item_map[name] = value
I tried two ways of doing type dispatching with two different ways, in the two different insert functions, but they both error. I have seen examples where people do things like Type-Dispatching Example
but they don't show how to use type dispatching with setting, only getting.
My implementation gives:
global name '_dID' is not defined at line 193 of program.py
How does one use type dispatching with setting variables? Is it even possible? If not, how else can I quickly make an abstraction like the one above without having to type a bunch of if statements?
EDIT: To add, I cannot instantiate the object with all of the IDs as their values are not known at instantiation.

I'm really not sure what you're asking here, or what exactly you mean by type dispatching. If you're asking about how to dynamically set attributes in a class, you can easily do that with **kwargs:
def __init__(self, **kwargs):
self.__dict__.update(kwargs)
and now you can instantiate it with action(dID='foo', oID='bar') and refer internally to self.dID and self.oID.

Plone: Change default schemata of a content type

I am using the Products.FacultyStaffDirectory product to manage contacts in my website.
One of the content types, i.e. FacultyStaffDirectory, has the description field in the 'categorization' tab when you are in edit mode. Now I need to remove the description field and put it in the default tab.
To do this, I'm using archetypes.schemaextender product to edit the field. In particular, I am using ISchemaModifier.
I have implemented the code but the field is still being shown in the categorization tab. May be something I missed. Below is the code:
This is the class that contains the description field that I want to modify:
# -*- coding: utf-8 -*-
__author__ = """WebLion <support#weblion.psu.edu>"""
__docformat__ = 'plaintext'
from AccessControl import ClassSecurityInfo
from Products.Archetypes.atapi import *
from zope.interface import implements
from Products.FacultyStaffDirectory.interfaces.facultystaffdirectory import IFacultyStaffDirectory
from Products.FacultyStaffDirectory.config import *
from Products.CMFCore.permissions import View, ManageUsers
from Products.CMFCore.utils import getToolByName
from Products.ATContentTypes.content.base import ATCTContent
from Products.ATContentTypes.content.schemata import ATContentTypeSchema, finalizeATCTSchema
from Products.membrane.at.interfaces import IPropertiesProvider
from Products.membrane.utils import getFilteredValidRolesForPortal
from Acquisition import aq_inner, aq_parent
from Products.FacultyStaffDirectory import FSDMessageFactory as _
schema = ATContentTypeSchema.copy() + Schema((
LinesField('roles_',
accessor='getRoles',
mutator='setRoles',
edit_accessor='getRawRoles',
vocabulary='getRoleSet',
default = ['Member'],
multiValued=1,
write_permission=ManageUsers,
widget=MultiSelectionWidget(
label=_(u"FacultyStaffDirectory_label_FacultyStaffDirectoryRoles", default=u"Roles"),
description=_(u"FacultyStaffDirectory_description_FacultyStaffDirectoryRoles", default=u"The roles all people in this directory will be granted site-wide"),
i18n_domain="FacultyStaffDirectory",
),
),
IntegerField('personClassificationViewThumbnailWidth',
accessor='getClassificationViewThumbnailWidth',
mutator='setClassificationViewThumbnailWidth',
schemata='Display',
default=100,
write_permission=ManageUsers,
widget=IntegerWidget(
label=_(u"FacultyStaffDirectory_label_personClassificationViewThumbnailWidth", default=u"Width for thumbnails in classification view"),
description=_(u"FacultyStaffDirectory_description_personClassificationViewThumbnailWidth", default=u"Show all person thumbnails with a fixed width (in pixels) within the classification view"),
i18n_domain="FacultyStaffDirectory",
),
),
))
FacultyStaffDirectory_schema = OrderedBaseFolderSchema.copy() + schema.copy() # + on Schemas does only a shallow copy
finalizeATCTSchema(FacultyStaffDirectory_schema, folderish=True)
class FacultyStaffDirectory(OrderedBaseFolder, ATCTContent):
"""
"""
security = ClassSecurityInfo()
implements(IFacultyStaffDirectory, IPropertiesProvider)
meta_type = portal_type = 'FSDFacultyStaffDirectory'
# Make this permission show up on every containery object in the Zope instance. This is a Good Thing, because it easy to factor up permissions. The Zope Developer's Guide says to put this here, not in the install procedure (http://www.zope.org/Documentation/Books/ZDG/current/Security.stx). This is because it isn't "sticky", in the sense of being persisted through the ZODB. Thus, it has to run every time Zope starts up. Thus, when you uninstall the product, the permission doesn't stop showing up, but when you actually remove it from the Products folder, it does.
security.setPermissionDefault('FacultyStaffDirectory: Add or Remove People', ['Manager', 'Owner'])
# moved schema setting after finalizeATCTSchema, so the order of the fieldsets
# is preserved. Also after updateActions is called since it seems to overwrite the schema changes.
# Move the description field, but not in Plone 2.5 since it's already in the metadata tab. Although,
# decription and relateditems are occasionally showing up in the "default" schemata. Move them
# to "metadata" just to be safe.
if 'categorization' in FacultyStaffDirectory_schema.getSchemataNames():
FacultyStaffDirectory_schema.changeSchemataForField('description', 'categorization')
else:
FacultyStaffDirectory_schema.changeSchemataForField('description', 'metadata')
FacultyStaffDirectory_schema.changeSchemataForField('relatedItems', 'metadata')
_at_rename_after_creation = True
schema = FacultyStaffDirectory_schema
# Methods
security.declarePrivate('at_post_create_script')
def at_post_create_script(self):
"""Actions to perform after a FacultyStaffDirectory is added to a Plone site"""
# Create some default contents
# Create some base classifications
self.invokeFactory('FSDClassification', id='faculty', title='Faculty')
self.invokeFactory('FSDClassification', id='staff', title='Staff')
self.invokeFactory('FSDClassification', id='grad-students', title='Graduate Students')
# Create a committees folder
self.invokeFactory('FSDCommitteesFolder', id='committees', title='Committees')
# Create a specialties folder
self.invokeFactory('FSDSpecialtiesFolder', id='specialties', title='Specialties')
security.declareProtected(View, 'getDirectoryRoot')
def getDirectoryRoot(self):
"""Return the current FSD object through acquisition."""
return self
security.declareProtected(View, 'getClassifications')
def getClassifications(self):
"""Return the classifications (in brains form) within this FacultyStaffDirectory."""
portal_catalog = getToolByName(self, 'portal_catalog')
return portal_catalog(path='/'.join(self.getPhysicalPath()), portal_type='FSDClassification', depth=1, sort_on='getObjPositionInParent')
security.declareProtected(View, 'getSpecialtiesFolder')
def getSpecialtiesFolder(self):
"""Return a random SpecialtiesFolder contained in this FacultyStaffDirectory.
If none exists, return None."""
specialtiesFolders = self.getFolderContents({'portal_type': 'FSDSpecialtiesFolder'})
if specialtiesFolders:
return specialtiesFolders[0].getObject()
else:
return None
security.declareProtected(View, 'getPeople')
def getPeople(self):
"""Return a list of people contained within this FacultyStaffDirectory."""
portal_catalog = getToolByName(self, 'portal_catalog')
results = portal_catalog(path='/'.join(self.getPhysicalPath()), portal_type='FSDPerson', depth=1)
return [brain.getObject() for brain in results]
security.declareProtected(View, 'getSortedPeople')
def getSortedPeople(self):
""" Return a list of people, sorted by SortableName
"""
people = self.getPeople()
return sorted(people, cmp=lambda x,y: cmp(x.getSortableName(), y.getSortableName()))
security.declareProtected(View, 'getDepartments')
def getDepartments(self):
"""Return a list of FSDDepartments contained within this site."""
portal_catalog = getToolByName(self, 'portal_catalog')
results = portal_catalog(portal_type='FSDDepartment')
return [brain.getObject() for brain in results]
security.declareProtected(View, 'getAddableInterfaceSubscribers')
def getAddableInterfaceSubscribers():
"""Return a list of (names of) content types marked as addable using the
IFacultyStaffDirectoryAddable interface."""
return [type['name'] for type in listTypes() if IFacultyStaffDirectoryAddable.implementedBy(type['klass'])]
security.declarePrivate('getRoleSet')
def getRoleSet(self):
"""Get the roles vocabulary to use."""
portal_roles = getFilteredValidRolesForPortal(self)
allowed_roles = [r for r in portal_roles if r not in INVALID_ROLES]
return allowed_roles
#
# Validators
#
security.declarePrivate('validate_id')
def validate_id(self, value):
"""Ensure the id is unique, also among groups globally."""
if value != self.getId():
parent = aq_parent(aq_inner(self))
if value in parent.objectIds():
return _(u"An object with id '%s' already exists in this folder") % value
groups = getToolByName(self, 'portal_groups')
if groups.getGroupById(value) is not None:
return _(u"A group with id '%s' already exists in the portal") % value
registerType(FacultyStaffDirectory, PROJECTNAME)
Below is the portion of code from the class that I have implemented to change the description field's schemata:
from Products.Archetypes.public import ImageField, ImageWidget, StringField, StringWidget, SelectionWidget, TextField, RichWidget
from Products.FacultyStaffDirectory.interfaces.facultystaffdirectory import IFacultyStaffDirectory
from archetypes.schemaextender.field import ExtensionField
from archetypes.schemaextender.interfaces import ISchemaModifier, ISchemaExtender, IBrowserLayerAwareExtender
from apkn.templates.interfaces import ITemplatesLayer
from zope.component import adapts
from zope.interface import implements
class _ExtensionImageField(ExtensionField, ImageField): pass
class _ExtensionStringField(ExtensionField, StringField): pass
class _ExtensionTextField(ExtensionField, TextField): pass
class FacultyStaffDirectoryExtender(object):
"""
Adapter to add description field to a FacultyStaffDirectory.
"""
adapts(IFacultyStaffDirectory)
implements(ISchemaModifier, IBrowserLayerAwareExtender)
layer = ITemplatesLayer
fields = [
]
def __init__(self, context):
self.context = context
def getFields(self):
return self.fields
def fiddle(self, schema):
desc_field = schema['description'].copy()
desc_field.schemata = "default"
schema['description'] = desc_field
And here is the code from my configure.zcml:
<adapter
name="apkn-FacultyStaffDirectoryExtender"
factory="apkn.templates.extender.FacultyStaffDirectoryExtender"
provides="archetypes.schemaextender.interfaces.ISchemaModifier"
/>
Is there something that I'm missing in this?

def fiddle(self, schema):
schema['description'].schemata = 'default'
should be sufficient. The copy() operation does not make any sense here.
In order to check if the fiddle() method is actually used: use pdb or add debug print statements.

It turns out that there was nothing wrong with the code.
I got it to work by doing the following:
Restarted the plone website
Re-installed the product
Clear and rebuilt my catalog
This got it working somehow.

Python OOP Project Organization

I'm a bit new to Python dev -- I'm creating a larger project for some web scraping. I want to approach this as "Pythonically" as possible, and would appreciate some help with the project structure. Here's how I'm doing it now:
Basically, I have a base class for an object whose purpose is to go to a website and parse some specific data on it into its own array, jobs[]
minion.py
class minion:
# Empty getJobs() function to be defined by object pre-instantiation
def getJobs(self):
pass
# Constructor for a minion that requires site authorization
# Ex: minCity1 = minion('http://portal.com/somewhere', 'user', 'password')
# or minCity2 = minion('http://portal.com/somewhere')
def __init__(self, title, URL, user='', password=''):
self.title = title
self.URL = URL
self.user = user
self.password = password
self.jobs = []
if (user == '' and password == ''):
self.reqAuth = 0
else:
self.reqAuth = 1
def displayjobs(self):
for j in self.jobs:
j.display()
I'm going to have about 100 different data sources. The way I'm doing it now is to just create a separate module for each "Minion", which defines (and binds) a more tailored getJobs() function for that object
Example: minCity1.py
from minion import minion
from BeautifulSoup import BeautifulSoup
import urllib2
from job import job
# MINION CONFIG
minTitle = 'Some city'
minURL = 'http://www.somewebpage.gov/'
# Here we define a function that will be bound to this object's getJobs function
def getJobs(self):
page = urllib2.urlopen(self.URL)
soup = BeautifulSoup(page)
# For each row
for tr in soup.findAll('tr'):
tJob = job()
span = tr.findAll(['span', 'class="content"'])
# If row has 5 spans, pull data from span 2 and 3 ( [1] and [2] )
if len(span) == 5:
tJob.title = span[1].a.renderContents()
tJob.client = 'Some City'
tJob.source = minURL
tJob.due = span[2].div.renderContents().replace('<br />', '')
self.jobs.append(tJob)
# Don't forget to bind the function to the object!
minion.getJobs = getJobs
# Instantiate the object
mCity1 = minion(minTitle, minURL)
I also have a separate module which simply contains a list of all the instantiated minion objects (which I have to update each time I add one):
minions.py
from minion_City1 import mCity1
from minion_City2 import mCity2
from minion_City3 import mCity3
from minion_City4 import mCity4
minionList = [mCity1,
mCity2,
mCity3,
mCity4]
main.py references minionList for all of its activities for manipulating the aggregated data.
This seems a bit chaotic to me, and was hoping someone might be able to outline a more Pythonic approach.
Thank you, and sorry for the long post!

Instead of creating functions and assigning them to objects (or whatever minion is, I'm not really sure), you should definitely use classes instead. Then you'll have one class for each of your data sources.
If you want, you can even have these classes inherit from a common base class, but that isn't absolutely necessary.