We Keep Coding

Python: Overwrite a function/method with the content of a String

As said in the title, I want to overwrite a function/method with the content of a string.
Background
I want to use an extern library which handles a CNN-model. The problem is, that some of these models contain functions, which my library cant handle. So I investigated the structure of the model and found out that some layers contain operators like "+=", which my extern library can't handle, so I have to replace these lines every time I found them.
To realize this, I used the inspect module to generate the a string, which contains the function with the problematic lines.
Out of this string a generate an abstract syntax tree with the ast module.
In this structure I could change the function like I needed it.
Now I can parse this abstract syntax tree to a string if needed, but the problem is that a lot of techniques to transform this string into a function, so that I can overwrite the problematic functions with my generated code in the string, but I failed every time.
This is my code in the moment, and sorry for the bad style, I'm a neewby to that :(
import torchvision
import torch
import inspect
import textwrap
import ast
from pprint import pprint
import torch.nn as nn
# This is a model which i have to investigate
model = torchvision.models.resnet18()
# Here i extract all classes and functions to check them below
classes = [
x[0] for x in inspect.getmembers(nn, inspect.isclass)
]
# This class changed my abstract syntax tree into the form i need
class Trasformer(ast.NodeTransformer):
def visit_AugAssign(self,node):
return (ast.Assign(targets=[ast.Name(id='out', ctx=ast.Store())], value=ast.Call(func=ast.Attribute(value=ast.Name(id='torch', ctx=ast.Load()), attr='stack', ctx=ast.Load()), args=[ast.List(elts=[ast.Name(id='identity', ctx=ast.Load()), ast.Name(id='out', ctx=ast.Load())], ctx=ast.Load())], keywords=[ast.keyword(arg='dim', value=ast.UnaryOp(op=ast.USub(), operand=ast.Constant(value=1)))])), ast.Assign(targets=[ast.Name(id='out', ctx=ast.Store())], value=ast.Call(func=ast.Attribute(value=ast.Name(id='self', ctx=ast.Load()), attr='canonizer_sum', ctx=ast.Load()), args=[ast.Name(id='out', ctx=ast.Load())], keywords=[])))
# This function checks, if there
def recursive(model):
children = list(model.children())
print(f"Checking {model.__class__.__name__}\t{len(children)}")
if len(children) == 0:
# print(model)
pass
else:
if not model.__class__.__name__ in classes:
# print(model)
print(f"!!{model.__class__.__name__}")
print(textwrap.dedent(inspect.getsource(model.forward)))
"""Hier wird der Code der Funktion verändert"""
forward_code = textwrap.dedent(inspect.getsource(model.forward))
tree = ast.parse(forward_code)
#print(ast.dump(tree))
# Using the Transformer class, to visit all nodes in the abstract syntax tree, and if the node is "AugAssign (+=) then the node will changed into two nodes, with the correct statements
Trasformer().visit(tree)
# delivers the needed information of the new nodes, after the change
tree = ast.fix_missing_locations(tree)
# print(ast.dump(tree, indent=2))
# Here the changed AST is compiled into string
forward_code = ast.unparse(tree)
"""Up to this point everything works fine. But now the problems start."""
#At this point i tried different ways to overwrite the function "model.forward" which the function, hold in the string. But nothing worked. This is my last attempt
model.forward. = exec(compile(tree, filename='test', mode='exec'))
# here the first line of code is cut off, (the function definition, because just the code inside should be overwritten
# forward_code = forward_code.split("\n", 1)[1]
print("new string:\n")
#print(forward_code)
print(textwrap.dedent(inspect.getsource(model.forward)))
# TODO: Transform the string into the method -> i.e. overwrite the method is needed"""
# setattr(model,"forward", forward_code)
#print("new:\n")
#print(textwrap.dedent(inspect.getsource(model.forward)))
for module in children:
recursive(module)
recursive(model)
quit()
So my question is, is it possible to overwrite a function/method with the content of a string?
I had different ideas:
Overwrite it directly --> doesn't work because you can't assign a string to a function
Write the string into a file and load the content of the file into the function
I haven't tried it yet, because I thought that there has to be a better way.
use exec/eval/compile. I think this way it maybe could work but I have no idea how.
I tried it with the function head in the string and without in it
Could somebody please help me and tell me how I overwrite the function with a corrected ast tree or the string generated out of it?
Is there a module/ function from the python library which I could use?
Thanks in advance for all volunteer helpers :)

Python functions are immutable, like many other Python objects. Once created, you cannot modify a function. But you can normally replace it with a different function, by setting the attribute of the class or object from which the function was extracted.
By the way, if you have the AST of a function, there is no need to turn it into a string in order to compile it. You can just compile the AST itself.

Cannot use Kmeans Cluster inside a python function

As above - I'm trying to create a function for clustering specific data types and displaying them.
The function looks a bit like this at the moment,
def cluster(inputData):
variable_s= inputData.groupby(['x','z', 'c'])['w'].sum().unstack()
## 4 Clusters
model = cluster.MiniBatchKMeans(n_clusters=5)
model.fit(variable_s.fillna(0))
variable_s['kmeans_4'] = model.predict(variable_s.fillna(0))
## 8 Clusters
model = cluster.KMeans(n_clusters=8)
model.fit(variable_s.fillna(0))
variable_s['kmeans_8'] = model.predict(variable_s.fillna(0))
## Looking at hourly distribution.
variable_s_Hourly = variable_s.reset_index(1, inplace=True)
variable_s_Hourly['hour'] = variable_s_Hourly.index.hour
return variable_s, variable_s_Hourly
it uses
from sklearn import cluster
to do the clustering, and it's giving me an error like this,
AttributeError: 'function' object has no attribute 'MiniBatchKMeans'
Any clues on solving this issue? I would have thought the function would be fine as long as the library is imported into the file itself - this is in jupyter notebook :)
Cheers!

The function name ("cluster") shadows the import. Change the function name to solve it.
Alternatively, you can give the import an alias:
from sklearn import cluster as clstr

How to properly pickle sklearn pipeline when using custom transformer

I am trying to pickle a sklearn machine-learning model, and load it in another project. The model is wrapped in pipeline that does feature encoding, scaling etc. The problem starts when i want to use self-written transformers in the pipeline for more advanced tasks.
Let's say I have 2 projects:
train_project: it has the custom transformers in src.feature_extraction.transformers.py
use_project: it has other things in src, or has no src catalog at all
If in "train_project" I save the pipeline with joblib.dump(), and then in "use_project" i load it with joblib.load() it will not find something such as "src.feature_extraction.transformers" and throw exception:
ModuleNotFoundError: No module named 'src.feature_extraction'
I should also add that my intention from the beginning was to simplify usage of the model, so programist can load the model as any other model, pass very simple, human readable features, and all "magic" preprocessing of features for actual model (e.g. gradient boosting) is happening inside.
I thought of creating /dependencies/xxx_model/ catalog in root of both projects, and store all needed classes and functions in there (copy code from "train_project" to "use_project"), so structure of projects is equal and transformers can be loaded. I find this solution extremely inelegant, because it would force the structure of any project where the model would be used.
I thought of just recreating the pipeline and all transformers inside "use_project" and somehow loading fitted values of transformers from "train_project".
The best possible solution would be if dumped file contained all needed info and needed no dependencies, and I am honestly shocked that sklearn.Pipelines seem to not have that possibility - what's the point of fitting a pipeline if i can not load fitted object later? Yes it would work if i used only sklearn classes, and not create custom ones, but non-custom ones do not have all needed functionality.
Example code:
train_project
src.feature_extraction.transformers.py
from sklearn.pipeline import TransformerMixin
class FilterOutBigValuesTransformer(TransformerMixin):
def __init__(self):
pass
def fit(self, X, y=None):
self.biggest_value = X.c1.max()
return self
def transform(self, X):
return X.loc[X.c1 <= self.biggest_value]
train_project
main.py
from sklearn.externals import joblib
from sklearn.preprocessing import MinMaxScaler
from src.feature_extraction.transformers import FilterOutBigValuesTransformer
pipeline = Pipeline([
('filter', FilterOutBigValuesTransformer()),
('encode', MinMaxScaler()),
])
X=load_some_pandas_dataframe()
pipeline.fit(X)
joblib.dump(pipeline, 'path.x')
test_project
main.py
from sklearn.externals import joblib
pipeline = joblib.load('path.x')
The expected result is pipeline loaded correctly with transform method possible to use.
Actual result is exception when loading the file.

I found a pretty straightforward solution. Assuming you are using Jupyter notebooks for training:
Create a .py file where the custom transformer is defined and import it to the Jupyter notebook.
This is the file custom_transformer.py
from sklearn.pipeline import TransformerMixin
class FilterOutBigValuesTransformer(TransformerMixin):
def __init__(self):
pass
def fit(self, X, y=None):
self.biggest_value = X.c1.max()
return self
def transform(self, X):
return X.loc[X.c1 <= self.biggest_value]
Train your model importing this class from the .py file and save it using joblib.
import joblib
from custom_transformer import FilterOutBigValuesTransformer
from sklearn.externals import joblib
from sklearn.preprocessing import MinMaxScaler
pipeline = Pipeline([
('filter', FilterOutBigValuesTransformer()),
('encode', MinMaxScaler()),
])
X=load_some_pandas_dataframe()
pipeline.fit(X)
joblib.dump(pipeline, 'pipeline.pkl')
When loading the .pkl file in a different python script, you will have to import the .py file in order to make it work:
import joblib
from utils import custom_transformer # decided to save it in a utils directory
pipeline = joblib.load('pipeline.pkl')

Apparently this problem raises when you split definitions and saving code part in two different files. So I have found this workaround that has worked for me.
It consists in these steps:
Guess we have your 2 projects/repositories : train_project and use_project
train_project:
On your train_project create a jupyter notebook or .py
On that file lets define every Custom transformer in a class, and import all other tools needed from sklearn to design the pipelines. Then lets write the saving code to pickle just inside the same file.(Don't create an external .py file src.feature_extraction.transformers to define your customtransformers).
Then fit and dumb your pipeline by running that file.
On use_project:
Create a customthings.py file with all the functions and transformers defined inside.
Create another file_where_load.py where you wish load the pickle. Inside, make sure you have imported all the definitions from customthings.py . Ensure that functions and classes have the same name than the ones you've used on train_project.
I hope it works for everyone with same problem

I have created a workaround solution. I do not consider it a complete answer to my question, but non the less it let me move on from my problem.
Conditions for the workaround to work:
I. Pipeline needs to have only 2 kinds of transformers:
sklearn transformers
custom transformers, but with only attributes of types:
number
string
list
dict
or any combination of those e.g. list of dicts with strings and numbers. Generally important thing is that attributes are json serializable.
II. names of pipeline steps need to be unique (even if there is pipeline nesting)
In short model would be stored as a catalog with joblib dumped files, a json file for custom transformers, and a json file with other info about model.
I have created a function that goes through steps of a pipeline and checks __module__ attribute of transformer.
If it finds sklearn in it it then it runs joblib.dump function under a name specified in steps (first element of step tuple), to some selected model catalog.
Otherwise (no sklearn in __module__) it adds __dict__ of transformer to result_dict under a key equal to name specified in steps. At the end I json.dump the result_dict to model catalog under name result_dict.json.
If there is a need to go into some transformer, because e.g. there is a Pipeline inside a pipeline, you can probably run this function recursively by adding some rules to the beginning of the function, but it becomes important to have always unique steps/transformers names even between main pipeline and subpipelines.
If there are other information needed for creation of model pipeline then save them in model_info.json.
Then if you want to load the model for usage:
You need to create (without fitting) the same pipeline in target project. If pipeline creation is somewhat dynamic, and you need information from source project, then load it from model_info.json.
You can copy function used for serialization and:
replace all joblib.dump with joblib.load statements, assign __dict__ from loaded object to __dict__ of object already in pipeline
replace all places where you added __dict__ to result_dict with assignment of appropriate value from result_dict to object __dict__ (remember to load result_dict from file beforehand)
After running this modified function, previously unfitted pipeline should have all transformer attributes that were effect of fitting loaded, and pipeline as a whole ready to predict.
The main things I do not like about this solution is that it needs pipeline code inside target project, and needs all attrs of custom transformers to be json serializable, but I leave it here for other people that stumble on a similar problem, maybe somebody comes up with something better.

I was similarly surprised when I came across the same problem some time ago. Yet there are multiple ways to address this.
Best practice solution:
As others have mentioned, the best practice solution is to move all dependencies of your pipeline into a separate Python package and define that package as a dependency of your model environment.
The environment then has to be recreated whenever the model is deployed. In simple cases this can be done manually e.g. via virtualenv or Poetry. But model stores and versioning frameworks (MLflow being one example) typically provide a way to define the required Python environment (e.g. via conda.yaml). They often can automatically recreate the environment at deployment time.
Solution by putting code into main:
In fact, class and function declearations can be serialized, but only declarations in __main__ actually get serialized. __main__ is the entry point of the script, the file that is run. So if all the custom code and all of its dependencies are in that file, then custom objects can later be loaded in Python environments that do not include the code. This kind of solves the problem, but who wants to have all that code in __main__? (Note that this property also applies to cloudpickle)
Solution by "mainifying":
There is one other way which is to "mainify" the classes or function objects before saving. I came across that same problem some time ago and have written a function that does that. It essentially redefines an existing object's code in __main__. Its application is simple: Pass object to function, then serialize the object, voilà, it can be loaded anywhere. Like so:
# ------ In file1.py: ------
class Foo():
pass
# ------ In file2.py: ------
from file1 import Foo
foo = Foo()
foo = mainify(foo)
import dill
with open('path/file.dill', 'wb') as f
dill.dump(foo, f)
I post the function code below. Note that I have tested this with dill, but I think it should work with pickle as well.
Also note that the original idea is not mine, but came from a blog post that I cannot find right now. I will add the reference/acknowledgement when I find it.
Edit: Blog post by Oege Dijk by which my code was inspired.
def mainify(obj, warn_if_exist=True):
''' If obj is not defined in __main__ then redefine it in main. Allows dill
to serialize custom classes and functions such that they can later be loaded
without them being declared in the load environment.
Parameters
---------
obj : Object to mainify (function or class instance)
warn_if_exist : Bool, default True. Throw exception if function (or class) of
same name as the mainified function (or same name as mainified
object's __class__) was already defined in __main__. If False
don't throw exception and instead use what was defined in
__main__. See Limitations.
Limitations
-----------
Assumes `obj` is either a function or an instance of a class.
'''
if obj.__module__ != '__main__':
import __main__
is_func = True if isinstance(obj, types.FunctionType) else False
# Check if obj with same name is already defined in __main__ (for funcs)
# or if class with same name as obj's class is already defined in __main__.
# If so, simply return the func with same name from __main__ (for funcs)
# or assign the class of same name to obj and return the modified obj
if is_func:
on = obj.__name__
if on in __main__.__dict__.keys():
if warn_if_exist:
raise RuntimeError(f'Function with __name__ `{on}` already defined in __main__')
return __main__.__dict__[on]
else:
ocn = obj.__class__.__name__
if ocn in __main__.__dict__.keys():
if warn_if_exist:
raise RuntimeError(f'Class with obj.__class__.__name__ `{ocn}` already defined in __main__')
obj.__class__ = __main__.__dict__[ocn]
return obj
# Get source code and compile
source = inspect.getsource(obj if is_func else obj.__class__)
compiled = compile(source, '<string>', 'exec')
# "declare" in __main__, keeping track which key of __main__ dict is the new one
pre = list(__main__.__dict__.keys())
exec(compiled, __main__.__dict__)
post = list(__main__.__dict__.keys())
new_in_main = list(set(post) - set(pre))[0]
# for function return mainified version, else assign new class to obj and return object
if is_func:
obj = __main__.__dict__[new_in_main]
else:
obj.__class__ = __main__.__dict__[new_in_main]
return obj

Have you tried using cloud pickle?
https://github.com/cloudpipe/cloudpickle

Based on my research it seems that the best solution is to create a Python package that includes your trained pipeline and all files.
Then you can pip install it in the project where you want to use it and import the pipeline with from <package name> import <pipeline name>.

Credit to Ture Friese for mentioning cloudpickle >=2.0.0, but here's an example for your use case.
import cloudpickle
cloudpickle.register_pickle_by_value(FilterOutBigValuesTransformer)
with open('./pipeline.cloudpkl', mode='wb') as file:
pipeline.dump(
obj=Pipe
, file=file
)
register_pickle_by_value() is the key as it will ensure your custom module (src.feature_extraction.transformers) is also included when serializing your primary object (pipeline). However, this is not built for recursive module dependence, e.g. if FilterOutBigValuesTransformer also contains another import statement

Calling the location of the transform.py file with sys.path.append may resolve the issue.
import sys
sys.path.append("src/feature_extraction/transformers")

How to fix 'recursion error' when pickling Pyomo Expressions

I haven't had trouble with pickling a Pyomo model before, but now a recursion error is being raised when trying to pickle Expressions.
Interestingly, the error can be avoided in the example below via a couple of ways:
by removing the "mutable" flag from the parameter
by reducing the size of the set to a very small value, to e.g. range(0, 10)
...but I don't know why these would fix the error, nor are they workable solutions for the actual optimization model I am trying to pickle.
The following example generates the error from the pickling of just a single Expression.
(I am using pyomo=5.6.2, cloudpickle=0.6.1, and python=3.7.4)
import cloudpickle
import pyomo.environ as pyo
test_model = pyo.ConcreteModel()
# A set is added.
set_elements = list(range(0, 500))
test_model.my_set = pyo.Set(initialize=set_elements)
# A parameter is added.
param_values = dict()
for e in set_elements:
param_values[e] = 1
test_model.my_param = pyo.Param(test_model.my_set, initialize=param_values, mutable=True)
# An expression is added.
def calculation_rule(mdl):
return sum(mdl.my_param[e] for e in mdl.my_set)
test_model.calculation_expr = pyo.Expression(rule=calculation_rule)
# We attempt to pickle the expression.
pickle_str = cloudpickle.dumps(test_model.calculation_expr)
The last line of the above code raises the following exception:
PicklingError: Could not pickle object as excessively deep recursion required.
QUESTION: Do I need to modify the way the Expression is written in order to pickle the model, or should I save the model using something other than Cloudpickle?
Thanks in advance for any help!

One fix is to use Pyomo's quicksum instead of Python's sum. It leads to more compact expression trees and seems to fix the recursion issue you're seeing:
# An expression is added.
def calculation_rule(mdl):
return pyo.quicksum(mdl.my_param[e] for e in mdl.my_set)
test_model.calculation_expr = pyo.Expression(rule=calculation_rule)
Documentation here.

PyML: graphing the decision surface

PyML has a function for graphing decision surfaces.
First you need to tell PyML which data to use. Here I use a sparsevectordata with my feature vectors. This is the one I used to train my SVM.
demo2d.setData(training_vector)
Then you need to tell it which classifier you want to use. I give it a trained SVM.
demo2d.decisionSurface(best_svm, fileName = "dec.pdf")
However, I get this error message:
Traceback (most recent call last):
**deleted by The Unfun Cat**
demo2d.decisionSurface(best_svm, fileName = "dec.pdf")
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/PyML/demo/demo2d.py", line 140, in decisionSurface
results = classifier.test(gridData)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/PyML/evaluators/assess.py", line 45, in test
classifier.verifyData(data)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/PyML/classifiers/baseClassifiers.py", line 55, in verifyData
if len(misc.intersect(self.featureID, data.featureID)) != len(self.featureID) :
AttributeError: 'SVM' object has no attribute 'featureID'

I'm going to dive right into the source, because I have never used PyML. Tried to find it online, but I couldn't track down the verifyData method in the PyML 0.7.2 that was online, so I had to search through downloaded source.
A classifier's featureID is only set in the baseClassifier class's train method (lines 77-78):
if data.__class__.__name__ == 'VectorDataSet' :
self.featureID = data.featureID[:]
In your code, data.__class__.__name__ is evaluating to "SparseDataSet" (or what ever other class you are using) and the expression evaluates to False (never setting featureID).
Then in demo2d.decisionSurface:
gridData = VectorDataSet(gridX)
gridData.attachKernel(data.kernel)
results = classifier.test(gridData)
Which tries to test your classifier using a VectorDataSet. In this instance classifier.test is equivalent to a call to the assess.test method which tries to verify if the data has the same features the training data had by using baseClassifier.verifyData:
def verifyData(self, data) :
if data.__class__.__name__ != 'VectorDataSet' :
return
if len(misc.intersect(self.featureID, data.featureID)) != len(self.featureID) :
raise ValueError, 'missing features in test data'
Which then tests the class of the passed data, which is now "VectorDataSet", and proceeds to try to access the featureID attribute that was never created.
Basically, it's either a bug, or a hidden feature.
Long story short, You have to convert your data to a VectorDataSet because SVM.featureID is not set otherwise.
Also, you don't need to pass it a trained data set, the function trains the classifier for you.
Edit:
I would also like to bring attention to the setData method:
def setData(data_) :
global data
data = data_
There is no type-checking at all. So someone could potentially set data to anything, e.g. an integer, a string, etc., which will cause an error in decisionSurface.
If you are going to use setData, you must use it carefully (only with a VectorDataSet), because the code is not as flexible as you would like.