We Keep Coding

Optimal way to add functionality to classes - PySpark

A while a go I was looking for how to rename several columns at once for a PySpark DF and came across something like the following:
import pyspark
def rename_sdf(df, mapper={}, **kwargs_mapper):
# Do something
# return something
pyspark.sql.dataframe.DataFrame.rename = rename_sdf
I am interested in that last bit where a method is added to the pyspark.DataFrame class through an assignment statement.
The thing is, I am creating a Github repo to store all my functions and ETLs and I thought that if I could apply the logic showed above it would be super easy to just create an __init__.py module where I instantiate all my functionalities like:
from funcs import *
pyspark.sql.dataframe.DataFrame.func1 = func1
pyspark.sql.dataframe.DataFrame.func2 = func2
.
.
.
pyspark.sql.dataframe.DataFrame.funcN = funcN
I guess my question is:
Is this useful? Is it good for performance? Is it wrong? Is it un-Pythonic?

That can definitely have its uses in certain scenarios. I would recommend putting this code into a function so the user must explicitly call it.
import funcs
def wrap_pyspark_dataframe():
pyspark.sql.dataframe.DataFrame.func1 = funcs.func1
pyspark.sql.dataframe.DataFrame.func2 = funcs.func2
...

trying to call package method using a variable

Im trying to store the method in a package in a variable as it might change and I dont want to manually update in multiple places in the code.
import hashlib as hashy
foo='hello world'
bar='hello world'
algo='md5'
hfoo=hashy.algo(foo.encode())
hbar=hashy.algo(bar.encode())

In this particular case you can use hashlib.new() to create a hasher by its name.
import hashlib # don't randomly rename standard libraries
ALGORITHM = 'md5'
h = hashlib.new(ALGORITHM)
h.update('hello world'.encode('utf-8'))
print(h.hexdigest())
If you think you might want to change which function in a module you're calling, you can wrap it in your own function, which is right generic answer to the question you're asking.
import hashlib
def hash(s):
hashlib.md5(s.encode('utf-8'))
print hash('hello world')
You can use getattr() on a module to retrieve a function by name, but that's not the usual way to do things.

Python: how to get a function based on whether it matches an assigned string to it [duplicate]

I have a function name stored in a variable like this:
myvar = 'mypackage.mymodule.myfunction'
and I now want to call myfunction like this
myvar(parameter1, parameter2)
What's the easiest way to achieve this?

funcdict = {
'mypackage.mymodule.myfunction': mypackage.mymodule.myfunction,
....
}
funcdict[myvar](parameter1, parameter2)

It's much nicer to be able to just store the function itself, since they're first-class objects in python.
import mypackage
myfunc = mypackage.mymodule.myfunction
myfunc(parameter1, parameter2)
But, if you have to import the package dynamically, then you can achieve this through:
mypackage = __import__('mypackage')
mymodule = getattr(mypackage, 'mymodule')
myfunction = getattr(mymodule, 'myfunction')
myfunction(parameter1, parameter2)
Bear in mind however, that all of that work applies to whatever scope you're currently in. If you don't persist them somehow, you can't count on them staying around if you leave the local scope.

def f(a,b):
return a+b
xx = 'f'
print eval('%s(%s,%s)'%(xx,2,3))
OUTPUT
5

Easiest
eval(myvar)(parameter1, parameter2)
You don't have a function "pointer". You have a function "name".
While this works well, you will have a large number of folks telling you it's "insecure" or a "security risk".

Why not store the function itself? myvar = mypackage.mymodule.myfunction is much cleaner.

modname, funcname = myvar.rsplit('.', 1)
getattr(sys.modules[modname], funcname)(parameter1, parameter2)

eval(compile(myvar,'<str>','eval'))(myargs)
compile(...,'eval') allows only a single statement, so that there can't be arbitrary commands after a call, or there will be a SyntaxError. Then a tiny bit of validation can at least constrain the expression to something in your power, like testing for 'mypackage' to start.

I ran into a similar problem while creating a library to handle authentication. I want the app owner using my library to be able to register a callback with the library for checking authorization against LDAP groups the authenticated person is in. The configuration is getting passed in as a config.py file that gets imported and contains a dict with all the config parameters.
I got this to work:
>>> class MyClass(object):
... def target_func(self):
... print "made it!"
...
... def __init__(self,config):
... self.config = config
... self.config['funcname'] = getattr(self,self.config['funcname'])
... self.config['funcname']()
...
>>> instance = MyClass({'funcname':'target_func'})
made it!
Is there a pythonic-er way to do this?

How to test methods that compute relative time in python using unitttest?

I have a method in a django model that does a computation relative to the current time. Here is a snippet:
def next_date():
now = datetime.now()
trial_expires = max(self.date_status_changed + timedelta(self.trial_days), now)
return timezone.datetime(trial_expires.year, trial_expires.month+1, 1, tzinfo=trial_expires.tzinfo)
What's the proper way to test this in django/python using unittest? What I'd like to do is be able to hard code some values for "now" in the test so I can try the various edge cases. Ideally, I'd like to avoid relying on the current time and date in the test.
One approach would be to modify my method to accept an optional parameter that would override the 'now' value it uses. Does python have any functions to do something similar without having to modify my method signature?

You could extract datetime.now() as a parameter:
def next_date(nowfunc=datetime.now):
now = nowfunc()
...
or as a class' dependency:
class X:
def __init__(self, nowfunc=datetime.now):
self._nowfunc = nowfunc
def next_date(self):
now = self._nowfunc()
...
And pass a mock function with the required result from your tests.
But if you don't want to modify the signature, use patches:
#patch.object(datetime, 'now')
def test_next_date(self, nowfunc):
nowfunc.return_value = ... # required result
# the rest of the test

How to change the date/time in Python for all modules?

When I write with business logic, my code often depends on the current time. For example the algorithm which looks at each unfinished order and checks if an invoice should be sent (which depends on the no of days since the job was ended). In these cases creating an invoice is not triggered by an explicit user action but by a background job.
Now this creates a problem for me when it comes to testing:
I can test invoice creation itself easily
However it is hard to create an order in a test and check that the background job identifies the correct orders at the correct time.
So far I found two solutions:
In the test setup, calculate the job dates relative to the current date. Downside: The code becomes quite complicated as there are no explicit dates written anymore. Sometimes the business logic is pretty complex for edge cases so it becomes hard to debug due to all these relative dates.
I have my own date/time accessor functions which I use throughout my code. In the test I just set a current date and all modules get this date. So I can simulate an order creation in February and check that the invoice is created in April easily. Downside: 3rd party modules do not use this mechanism so it's really hard to integrate+test these.
The second approach was way more successful to me after all. Therefore I'm looking for a way to set the time Python's datetime+time modules return. Setting the date is usually enough, I don't need to set the current hour or second (even though this would be nice).
Is there such a utility? Is there an (internal) Python API that I can use?

Monkey-patching time.time is probably sufficient, actually, as it provides the basis for almost all the other time-based routines in Python. This appears to handle your use case pretty well, without resorting to more complex tricks, and it doesn't matter when you do it (aside from the few stdlib packages like Queue.py and threading.py that do from time import time in which case you must patch before they get imported):
>>> import datetime
>>> datetime.datetime.now()
datetime.datetime(2010, 4, 17, 14, 5, 35, 642000)
>>> import time
>>> def mytime(): return 120000000.0
...
>>> time.time = mytime
>>> datetime.datetime.now()
datetime.datetime(1973, 10, 20, 17, 20)
That said, in years of mocking objects for various types of automated testing, I've needed this approach only very rarely, as most of the time it's my own application code that needs the mocking, and not the stdlib routines. After all, you know they work already. If you are encountering situations where your own code has to handle values returned by library routines, you may want to mock the library routines themselves, at least when checking how your own app will handle the timestamps.
The best approach by far is to build your own date/time service routine(s) which you use exclusively in your application code, and build into that the ability for tests to supply fake results as required. For example, I do a more complex equivalent of this sometimes:
# in file apptime.py (for example)
import time as _time
class MyTimeService(object):
def __init__(self, get_time=None):
self.get_time = get_time or _time.time
def __call__(self):
return self.get_time()
time = MyTimeService()
Now in my app code I just do import apptime as time; time.time() to get the current time value, whereas in test code I can first do apptime.time = MyTimeService(mock_time_func) in my setUp() code to supply fake time results.
Update: Years later there's an alternative, as noted in Dave Forgac's answer.

The freezegun package was made specifically for this purpose. It allows you to change the date for code under test. It can be used directly or via a decorator or context manager. One example:
from freezegun import freeze_time
import datetime
#freeze_time("2012-01-14")
def test():
assert datetime.datetime.now() == datetime.datetime(2012, 1, 14)
For more examples see the project: https://github.com/spulec/freezegun

You can patch the system, by creating a custom datetime module (even a fake one - see example below) acting as a proxy and then insert it in sys.modules dictionary. From there on, each import to the datetime module will return your proxy.
There is still the caveat of datetime class, especially when someone does from datetime import datetime; for that, you can simply add another proxy only for that class.
Here is an example of what I am saying - of course it is just something I've thrown in 5 minutes, and may have several issues (for instance, the type of datetime class is not correct); but hopefully it may already be of use.
import sys
import datetime as datetime_orig
class DummyDateTimeModule(sys.__class__):
""" Dummy class, for faking datetime module """
def __init__(self):
sys.modules["datetime"] = self
def __getattr__(self, attr):
if attr=="datetime":
return DummyDateTimeClass()
else:
return getattr(datetime_orig, attr)
class DummyDateTimeClass(object):
def __getattr__(self, attr):
return getattr(datetime_orig.datetime, attr)
dt_fake = DummyDateTimeModule()
Finally - is it worth?
Frankly speaking, I like our second solution much more than this one :-).
Yes, python is a very dynamic language, where you can do quite a lot of interesting things, but patching code in this way has always a certain degree of risk, even if we are talking here of test code.
But mostly, I think the accessory function would make test patching more explicit, and also your code would be more explicit in terms of what it is going to be tested, thus increasing readability.
Therefore, if the change is not too expensive, I would go for your second approach.

I would use the helpers from the 'testfixtures' package to mock out the date, datetime or time calls you're making:
http://packages.python.org/testfixtures/datetime.html

Well one way to do it is to dynamic patch the time /datetime module
something like
import time
import datetime
class MyDatetime:
def now(self):
return time.time()
datetime.datetime = MyDatetime
print datetime.datetime().now()

there might be few ways of doing this, like creating the orders (with the current timestamp) and then changing that value in the DB directly by some external process (assuming data is in the DB).
I'll suggest something else. Have you though about running your application in a virtual machine, setting the time to say Feb, creating orders, and then just changing the VMs time? This approach is the closest as you can get to the real-life situation.