We Keep Coding

Python patching __new__ method

I am trying to patch __new__ method of a class, and it is not working as I expect.
from contextlib import contextmanager
class A:
def __init__(self, arg):
print('A init', arg)
#contextmanager
def patch_a():
new = A.__new__
def fake_new(cls, *args, **kwargs):
print('call fake_new')
return new(cls, *args, **kwargs)
# here I get error: TypeError: object.__new__() takes exactly one argument (the type to instantiate)
A.__new__ = fake_new
try:
yield
finally:
A.__new__ = new
if __name__ == '__main__':
A('foo')
with patch_a():
A('bar')
A('baz')
I expect the following output:
A init foo
call fake_new
A init bar
A init baz
But after call fake_new I get an error (see comment in the code).
For me It seems like I just decorate a __new__ method and propagate all args unchanged.
It doesn't work and the reason is obscure for me.
Also I can write return new(cls) and call A('bar') works fine. But then A('baz') breaks.
Can someone explain what is going on?
Python version is 3.8

You've run into a complicated part of Python object instantiation - in which the language opted for a design that would allow one to create a custom __init__ method with parameters, without having to touch __new__.
However, the in the base of class hierarchy, object, both __new__ and __init__ take one single parameter each.
IIRC, it goes this way: if your class have a custom __init__ and you did not touch __new__ and there are more any parameters to the class instantiation that would be passed to both __init__ and __new__, the parameters will be stripped from the call do __new__, so you don't have to customize it just to swallow the parameters you consume in __init__. The converse is also true: if your class have a custom __new__ with extra parameters, and no custom __init__, these are not passed to object.__init__.
With your design, Python sees a custom __new__ and passes it the same extra arguments that are passed to __init__ - and by using *args, **kw, you forward those to object.__new__ which accepts a single parameter - and you get the error you presented us.
The fix is to not pass those extra parameters to the original __new__ method - unless they are needed there - so you have to make the same check Python's type does when initiating an object.
And an interesting surprise to top it: while making the example work, I found out that even if A.__new__
is deleted when restoring the patch, it is still considered as "touched" by cPython's type instantiation, and the arguments are passed through.
In order to get your code working I needed to leave a permanent stub A.__new__ that will forward only the cls argument:
from contextlib import contextmanager
class A:
def __init__(self, arg):
print('A init', arg)
#contextmanager
def patch_a():
new = A.__new__
def fake_new(cls, *args, **kwargs):
print('call fake_new')
if new is object.__new__:
return new(cls)
return new(cls, *args, **kwargs)
# here I get error: TypeError: object.__new__() takes exactly one argument (the type to instantiate)
A.__new__ = fake_new
try:
yield
finally:
del A.__new__
if new is not object.__new__:
A.__new__ = new
else:
A.__new__ = lambda cls, *args, **kw: object.__new__(cls)
print(A.__new__)
if __name__ == '__main__':
A('foo')
with patch_a():
A('bar')
A('baz')
(I tried inspecting the original __new__ signature instead of the new is object.__new__ comparison - to no avail: object.__new__ signature is *args, **kwargs - possibly made so that it will never fail on static checking)

How to edit Python scripts with correct autocompletion?

I am trying to edit python scripts in first time and it drives me crazy :( .
I have some directory with *.py files, that I added to PyCharm as Interpreter Paths, for correct auto completion.
So, I have some class
class Sim():
def __init__(self, *args, **kwargs):
self._sim_info = None
I am Java and C++ programmer and I am used to class type variables.
I know, that in scripts this variable will have value with type SimInfo.
But, when indexer of the PyCharm indexed that *.py files, he knows, that Sim.sim_info var has value None. But how can specify, that in code
s1=Sim()
i=s1.sim_info
variable i has type of class SimInfo?
May be I should use something like "editor hits", to force auto completion for i.is_ghost?
For example, code
from simulation.sims.sim import Sim
from simulation.sims.sim_info import SimInfo
from simulation.sims.pregnancy.pregnancy_tracker import PregnancyOffspringData
s1=Sim()
i=s1.sim_info
i.is_ghos
where i.is_ghos must be auto completed to i.is_ghost().
How to specify variable types in this case (may be via something like editor hints)?
Thank you very much!

Python 3.6:
class Sim():
def __init__(self, *args, **kwargs):
self._sim_info: SimInfo = None
Other python:
class Sim():
def __init__(self, *args, **kwargs):
self._sim_info = None # type: SimInfo
It called "type hints"

You can use type hinting with pycharm using the #type docstring.
def __init__(self, *args, **kwargs):
# n.b., in your usage code you use `.sim_info`
# but in your constructor you used `._sim_info`
# since I didn’t see a prop get function, I assumed
# that you meant `.sim_info`
self.sim_info = None
"""#type: SimInfo"""
or
def __init__(self, *args, **kwargs):
self.sim_info = None
""":type SimInfo"""
You can also specify the full path to the class including the module if just the class name did not work. You can also use PEP484 in PyCharm to specify the type of member variables:
def __init__(self):
self.sim_info = None # type: SimInfo

Calling a decorated python function from robotframework script resulting in infinite recursing

I'm trying to invoke a python function from robotframework keyword. The python function has been decorated to be invoked using run_keyword from Builtin library. This is because robot logs appear well structured if library functions are invoked via run_keyword function from built in library. rather than invoked directly. However this is resulting in an infinite loop. Is there a solution to gracefully accomplish the goal?
robotkeyword :
do something
#creates a user by calling a function from python based library
create user
python function
#wrap_with_run_keyword
def create_user():
pass
def wrap_with_run_keyword(func):
def func_wrapper(*args, **kwargs):
return run_keyword(func, *args, **kwargs)
return func_wrapper
I couldn't solve the problem using partial application.
However, I broker the recursive loop by setting and unsetting an attribute as give below.
def wrap_with_run_keyword(func):
def func_wrapper(*args, **kwargs):
if not hasattr(func, 'second'):
setattr(func, "second", True)
return run_keyword(func, *args, **kwargs)
else:
delattr(func, "second")
return func(*args, **kwargs)
return func_wrapper
I have however run into another problem. I defined create_user as follows
def create_user(properties):
#some code
pass
On Calling this function in the way below
create_user("name=abc")
I'm getting the following error : got an unexpected keyword argument 'name'

I did run in the same issue, but solved it, only wondering if i can detect the caller...if the call is done from robotframework or by python in case that the call is done by the rf it should do only the second call
#wraps(function)
def wrapper(self, *args, **kwargs):
if not hasattr(function, 'second'):
setattr(function, 'second', True)
ar= list(args)
for key, value in kwargs.items():
ar.append(value)
return BuiltIn().run_keyword('Mylib.' + function.__name__, ar)
else:
delattr(function, 'second')
return function(self,*args[0])
return wrapper

Take a look at the partial class from the functools module. I think this might help you.
Or take a look at how decorators work in python.

TypeError: function()) takes exactly X arguments (1 given) with #property decorator

So I've looked around and read many postings covering the TypeError: message, where it "takes exactly X arguments but only 1 is given".
I know about self. I don't think I have an issue understanding self. Regardless, I was trying to to create a class with some properties and as long as I have #property in front of my function hwaddr, I get the following error:
Traceback (most recent call last):
File line 24, in <module>
db.hwaddr("aaa", "bbbb")
TypeError: hwaddr() takes exactly 3 arguments (1 given)
Here is the code. Why is #property messing me up? I take it out, and the code works as expected:
#!/usr/bin/env python2.7
class Database:
"""An instance of our Mongo systems database"""
#classmethod
def __init__(self):
pass
#property
def hwaddr(self, host, interface):
results = [ host, interface ]
return results
db = Database()
print db.hwaddr("aaa", "bbbb"
Process finished with exit code 1
With it gone, the output is:
File
['aaa', 'bbbb']
Process finished with exit code 0

Properties are used as syntactical sugar getters. So they expect that you only pass self. It would basically shorten:
print db.hwaddr()
to:
print db.hwaddr
There is no need to use a property here as you pass two arguments in.

Basically, what Dair said: properties don't take parameters, that's what methods are for.
Typically, you will want to use properties in the following scenarios:
To provide read-only access to an internal attribute
To implement a calculated field
To provide read/write access to an attribute, but control what happens when it is set
So the question would be, what does hwaddr do, and does it match any of those use cases? And what exactly are host and interface? What I think you want to do is this:
#!/usr/bin/env python2.7
class Database:
"""An instance of our Mongo systems database"""
def __init__(self, host, interface):
self._host = host
self._interface = interface
#property
def host(self):
return self._host
#property
def interface(self):
return self._interface
#property
def hwaddr(self):
return self._host, self._interface
db = Database("my_host", "my_interface")
print db.host
print db.interface
print db.hwaddr
Here, your Database class will have host and interface read-only properties, that can only be set when instantiating the class. A third property, hwaddr, will produce a tuple with the database full address, which may be convenient in some cases.
Also, note that I removed the classmethod decorator in init; constructors should be instance methods.

Using **kwargs with SimpleXMLRPCServer in python

I have a class that I wish to expose as a remote service using pythons SimpleXMLRPCServer. The server startup looks like this:
server = SimpleXMLRPCServer((serverSettings.LISTEN_IP,serverSettings.LISTEN_PORT))
service = Service()
server.register_instance(service)
server.serve_forever()
I then have a ServiceRemote class that looks like this:
def __init__(self,ip,port):
self.rpcClient = xmlrpclib.Server('http://%s:%d' %(ip,port))
def __getattr__(self, name):
# forward all calls to the rpc client
return getattr(self.rpcClient, name)
So all calls on the ServiceRemote object will be forwarded to xmlrpclib.Server, which then forwards it to the remote server. The problem is a method in the service that takes named varargs:
#useDb
def select(self, db, fields, **kwargs):
pass
The #useDb decorator wraps the function, creating the db before the call and opening it, then closing it after the call is done before returning the result.
When I call this method, I get the error "call() got an unexpected keyword argument 'name'". So, is it possible to call methods taking variable named arguments remotely? Or will I have to create an override for each method variation I need.
Thanks for the responses. I changed my code around a bit so the question is no longer an issue. However now I know this for future reference if I indeed do need to implement positional arguments and support remote invocation. I think a combination of Thomas and praptaks approaches would be good. Turning kwargs into positional args on the client through xmlrpclient, and having a wrapper on methods serverside to unpack positional arguments.

You can't do this with plain xmlrpc since it has no notion of keyword arguments. However, you can superimpose this as a protocol on top of xmlrpc that would always pass a list as first argument, and a dictionary as a second, and then provide the proper support code so this becomes transparent for your usage, example below:
Server
from SimpleXMLRPCServer import SimpleXMLRPCServer
class Server(object):
def __init__(self, hostport):
self.server = SimpleXMLRPCServer(hostport)
def register_function(self, function, name=None):
def _function(args, kwargs):
return function(*args, **kwargs)
_function.__name__ = function.__name__
self.server.register_function(_function, name)
def serve_forever(self):
self.server.serve_forever()
#example usage
server = Server(('localhost', 8000))
def test(arg1, arg2):
print 'arg1: %s arg2: %s' % (arg1, arg2)
return 0
server.register_function(test)
server.serve_forever()
Client
import xmlrpclib
class ServerProxy(object):
def __init__(self, url):
self._xmlrpc_server_proxy = xmlrpclib.ServerProxy(url)
def __getattr__(self, name):
call_proxy = getattr(self._xmlrpc_server_proxy, name)
def _call(*args, **kwargs):
return call_proxy(args, kwargs)
return _call
#example usage
server = ServerProxy('http://localhost:8000')
server.test(1, 2)
server.test(arg2=2, arg1=1)
server.test(1, arg2=2)
server.test(*[1,2])
server.test(**{'arg1':1, 'arg2':2})

XML-RPC doesn't really have a concept of 'keyword arguments', so xmlrpclib doesn't try to support them. You would need to pick a convention, then modify xmlrpclib._Method to accept keyword arguments and pass them along using that convention.
For instance, I used to work with an XML-RPC server that passed keyword arguments as two arguments, '-KEYWORD' followed by the actual argument, in a flat list. I no longer have access to the code I wrote to access that XML-RPC server from Python, but it was fairly simple, along the lines of:
import xmlrpclib
_orig_Method = xmlrpclib._Method
class KeywordArgMethod(_orig_Method):
def __call__(self, *args, **kwargs):
if args and kwargs:
raise TypeError, "Can't pass both positional and keyword args"
args = list(args)
for key in kwargs:
args.append('-%s' % key.upper())
args.append(kwargs[key])
return _orig_Method.__call__(self, *args)
xmlrpclib._Method = KeywordArgMethod
It uses monkeypatching because that's by far the easiest method to do this, because of some clunky uses of module globals and name-mangled attributes (__request, for instance) in the ServerProxy class.

As far as I know, the underlying protocol doesn't support named varargs (or any named args for that matter). The workaround for this is to create a wrapper that will take the **kwargs and pass it as an ordinary dictionary to the method you want to call. Something like this
Server side:
def select_wrapper(self, db, fields, kwargs):
"""accepts an ordinary dict which can pass through xmlrpc"""
return select(self,db,fields, **kwargs)
On the client side:
def select(self, db, fields, **kwargs):
"""you can call it with keyword arguments and they will be packed into a dict"""
return self.rpcClient.select_wrapper(self,db,fields,kwargs)
Disclaimer: the code shows the general idea, you can do it a bit cleaner (for example writing a decorator to do that).

Using the above advice, I created some working code.
Server method wrapper:
def unwrap_kwargs(func):
def wrapper(*args, **kwargs):
print args
if args and isinstance(args[-1], list) and len(args[-1]) == 2 and "kwargs" == args[-1][0]:
func(*args[:-1], **args[-1][1])
else:
func(*args, **kwargs)
return wrapper
Client setup (do once):
_orig_Method = xmlrpclib._Method
class KeywordArgMethod(_orig_Method):
def __call__(self, *args, **kwargs):
args = list(args)
if kwargs:
args.append(("kwargs", kwargs))
return _orig_Method.__call__(self, *args)
xmlrpclib._Method = KeywordArgMethod
I tested this, and it supports method with fixed, positional and keyword arguments.

As Thomas Wouters said, XML-RPC does not have keyword arguments. Only the order of arguments matters as far as the protocol is concerned and they can be called anything in XML: arg0, arg1, arg2 is perfectly fine, as is cheese, candy and bacon for the same arguments.
Perhaps you should simply rethink your use of the protocol? Using something like document/literal SOAP would be much better than a workaround such as the ones presented in other answers here. Of course, this may not be feasible.