I'm an experienced programmer, but completely new to Python.
I've resolved most difficulties, but I can't get the queue module to work.
Any help gratefully received. Python 3.2.
Reduced to its basic minimum, here's the issue:
>>>import queue
>>>q = queue.Queue
>>>q.qsize()
Traceback:
...
q.qsize()
...
TypeError: qsize() takes 1 argument exactly (0 given)
Documentation...
7.8.1. Queue Objects
Queue objects (Queue, LifoQueue, or PriorityQueue) provide the public methods described below.
Queue.qsize()
OK - what argument.... ?
You're not initializing an instance, you're just reassigning the class name to q. The "argument" that it's talking about is self, the explicit self-reference that all Python methods need. In other words, it's saying that you're trying to call an instance method with no instance.
>>> q = queue.Queue()
>>> q.qsize()
If you've never seen a Python method definition, it looks something like this:
class Queue(object):
# Note the explicit 'self' argument
def qsize(self):
# ...
You are simply renaming queue.Queue and not instantiating an object.
Try this
q = queue.Queue()
print q.qsize()
Related
I am trying to learn a bit about threading by building some code that will maintain a list and eventually send values from that list to a to-be-created function. When I call QueryReport.add I get the error TypeError: add() missing 1 required positional argument: 'user'. I am trying to pass in report and user list values. It is my belief the add method needs self but that self for some reason seems to be throwing off the call I made because now I dont have the right number of positional aruguments. In many other code examples I see methods defined with self and two other values with the idea that when the mthod is called two values are passed. Any thoughts on what I am doing wrong? Since I cant get past this syntax error, is there anything else as blantly wrong?
from threading import *
class QueueReport(Thread):
def __init__(self):
Thread.__init__(self)
self.reports_to_call = []
def add(self, report, user):
self.reports_to_call.append([report, user])
print("Added Value")
def run(self):
print("Running")
#print(reports_to_call)
#will loop and send values from this list to another function
if __name__ == '__main__':
t1 = QueueReport()
t1.run()
QueueReport.add("mainreport", "user")
You are attempting to add to the class QueueReport instead of the instance of the class t1. When you call the function on the class, there is no self so it treats "mainreport" as self and "user" as report.
What you want to do is t1.add("mainreport", "user").
The wrinkle is, if run is an infinite loop it will block execution and t1.add will never happen.
What you can do is create a separate thread for t1.run.
from threading import Thread
thread = Thread(target=t1.run)
thread.start()
t1.add("mainreport", "user")
Edit: silly me, I did not see that you were already using threading.
You are using the Class method instead of using the instantiation of the class.
Because of that, self is not given to the method, hence missing a parameter.
Try doing t1.add("mainreport", "user")
I have an large script where i found out that lot of connections to a machine are left open and the reason was that for one of the class destructor was never getting called.
below is a simplified version of script manifesting the issue.
I tiered searching around and found out that it could be because of GC and weakref does help but in this case no help.
2 cases where i can see that the destructor is getting called are
If i call B_class object without passing A_class function
self.b = B_class("AA")
I call the make the B_class objects not global i.e not use self
b = B_class("AA",self.myprint)
b.do_something()
Both of these cases will cause further issues for my case. Last resort will be to close/del the objects at the end myself but i don't want to go that way.
can anybody suggest a better way out of this and help me understand this issue? Thanks in advance.
import weakref
class A_class:
def __init__(self,debug_level=1,version=None):
self.b = B_class("AA",self.myprint)
self.b.do_something()
def myprint(self, text):
print text
class B_class:
def __init__(self,ip,printfunc=None):
self.ip=ip
self.new_ip =ip
#self.printfunc = printfunc
self.printfunc = weakref.ref(printfunc)()
def __del__(self):
print("##B_Class Destructor called##")
def do_something(self,timeout=120):
self.myprint("B_Class ip=%s!!!" % self.new_ip)
def myprint(self,text):
if self.printfunc:
print ("ExtenalFUNC:%s" %text)
else:
print ("JustPrint:%s" %text)
def _main():
a = A_class()
if __name__ == '__main__':
_main()
You're not using the weakref.ref object properly. You're calling it immediately after it is created, which returns the referred-to object (the function passed in as printref).
Normally, you'd want to save the weak reference and only call it when you're going to use the reffered-to object (e.g. in myprint). However, that won't work for the bound method self.myprint you're getting passed in as printfunc, since the bound method object doesn't have any other references (every access to a method creates a new object).
If you're using Python 3.4 or later and you know that the object passed in will always be a bound method, you can use the WeakMethod class, rather than a regular ref. If you're not sure what kind of callable you're going to get, you might need to do some type checking to see if WeakMethod is required or not.
Use Python's "with" statement (http://www.python.org/dev/peps/pep-0343/).
It creates a syntactic scope and the __exit__ function which it creates is guaranteed to get called as soon as execution leaves the scope. You can also emulate "__enter__/__exit__" behavior by creating a generator with "contextmanager" decorator from the contextlib module (python 2.6+ or 2.5 using "from __future__ import with_statement" see PEP for examples).
Here's an example from the PEP:
import contextlib
#contextlib.contextmanger
def opening(filename):
f = open(filename) # IOError is untouched by GeneratorContext
try:
yield f
finally:
f.close() # Ditto for errors here (however unlikely)
and then in your main code, you write
with opening(blahblahblah) as f:
pass
# use f for something
# here you exited the with scope and f.close() got called
In your case, you'll want to use a different name (connecting or something) instead of "opening" and do socket connecting/disconnecting inside of your context manager.
self.printfunc = weakref.ref(printfunc)()
isn't actually using weakrefs to solve your problem; the line is effectively a noop. You create a weakref with weakref.ref(printfunc), but you follow it up with call parens, which converts back from weakref to a strong ref which you store (and the weakref object promptly disappears). Apparently it's not possible to store a weakref to the bound method itself (because the bound method is its own object created each time it's referenced on self, not a cached object whose lifetime is tied to self), so you have to get a bit hacky, unbinding the method so you can take a weakref on the object itself. Python 3.4 introduced WeakMethod to simplify this, but if you can't use that, then you're stuck doing it by hand.
Try changing it to (on Python 2.7, and you must import inspect):
# Must special case printfunc=None, since None is not weakref-able
if printfunc is None:
# Nothing provided
self.printobjref = self.printfuncref = None
elif inspect.ismethod(printfunc) and printfunc.im_self is not None:
# Handling bound method
self.printobjref = weakref.ref(printfunc.im_self)
self.printfuncref = weakref.ref(printfunc.im_func)
else:
self.printobjref = None
self.printfuncref = weakref.ref(printfunc)
and change myprint to:
def myprint(self,text):
if self.printfuncref is not None:
printfunc = self.printfuncref()
if printfunc is None:
self.printfuncref = self.printobjref = None # Ref died, so clear it to avoid rechecking later
elif self.printobjref is not None:
# Bound method not known to have disappeared
printobj = self.printobjref()
if printobj is not None:
print ("ExtenalFUNC:%s" %text) # To call it instead of just saying you have it, do printfunc(printobj, text)
return
self.printobjref = self.printfuncref = None # Ref died, so clear it to avoid rechecking later
else:
print ("ExtenalFUNC:%s" %text) # To call it instead of just saying you have it, do printfunc(text)
return
print ("JustPrint:%s" %text)
Yeah, it's ugly. You could factor out the ugly if you like (borrowing the implementation of WeakMethod from Python 3.4's source code would make sense, but names would have to change; __self__ is im_self in Py2, __func__ is im_func), but it's unpleasant even so. It's definitely not thread safe if the weakrefs could actually go dark, since the checks and clears of the weakref members aren't protected.
util.py
def exec_multiprocessing(self, method, args):
with concurrent.futures.ProcessPoolExecutor() as executor:
results = pool.map(method, args)
return results
clone.py
def clone_vm(self, name, first_run, host, ip):
# clone stuff
invoke.py
exec_args = [(name, first_run, host, ip) for host, ip in zip(hosts, ips)]
results = self.util.exec_multiprocessing(self.clone.clone_vm, exec_args)
The above code gives the pickling error. I found that it is because we are passing instance method. So we should unwrap the instance method. But I am not able to make it work.
Note: I can not create top level method to avoid this. I have to use instance methods.
Let's start with an overview - why the error came up in the first place:
The multiprocessing must requires to pickle (serialize) data to pass them along processes or threads. To be specific, pool methods themselves rely on queue at the lower level, to stack tasks and pass them to threads/processes, and queue requires everything that goes through it must be pickable.
The problem is, not all items are pickable - list of pickables - and when one tries to pickle an unpicklable object, gets the PicklingError exception - exactly what happened in your case, you passed an instance method which is not picklable.
There can be various workarounds (as is the case with every problem) - the solution which worked for me is here by Dano - is to make pickle handle the methods and register it with copy_reg.
Add the following lines at the start of your module clone.py to make clone_vm picklable (do import copy_reg and types):
def _pickle_method(m):
if m.im_self is None:
return getattr, (m.im_class, m.im_func.func_name)
else:
return getattr, (m.im_self, m.im_func.func_name)
copy_reg.pickle(types.MethodType, _pickle_method)
Other useful answers - by Alex Martelli, mrule, by unutbu
You need to add support for pickling functions and methods for that to work as pointed out by Nabeel Ahmed. But his solution won't work with name-mangled methods -
import copy_reg
import types
def _pickle_method(method):
attached_object = method.im_self or method.im_class
func_name = method.im_func.func_name
if func_name.startswith('__'):
func_name = filter(lambda method_name: method_name.startswith('_') and method_name.endswith(func_name), dir(attached_object))[0]
return (getattr, (attached_object, func_name))
copy_reg.pickle(types.MethodType, _pickle_method)
This would work for name mangled methods as well. For this to work, you need to ensure this code is always ran before any pickling happens. Ideal place is settings file(if you are using django) or some package that is always imported before other code is executed.
Credits:- Steven Bethard (https://bethard.cis.uab.edu/)
I want to process a large for loop in parallel, and from what I have read the best way to do this is to use the multiprocessing library that comes standard with Python.
I have a list of around 40,000 objects, and I want to process them in parallel in a separate class. The reason for doing this in a separate class is mainly because of what I read here.
In one class I have all the objects in a list and via the multiprocessing.Pool and Pool.map functions I want to carry out parallel computations for each object by making it go through another class and return a value.
# ... some class that generates the list_objects
pool = multiprocessing.Pool(4)
results = pool.map(Parallel, self.list_objects)
And then I have a class which I want to process each object passed by the pool.map function:
class Parallel(object):
def __init__(self, args):
self.some_variable = args[0]
self.some_other_variable = args[1]
self.yet_another_variable = args[2]
self.result = None
def __call__(self):
self.result = self.calculate(self.some_variable)
The reason I have a call method is due to the post I linked before, yet I'm not sure I'm using it correctly as it seems to have no effect. I'm not getting the self.result value to be generated.
Any suggestions?
Thanks!
Use a plain function, not a class, when possible. Use a class only when there is a clear advantage to doing so.
If you really need to use a class, then given your setup, pass an instance of Parallel:
results = pool.map(Parallel(args), self.list_objects)
Since the instance has a __call__ method, the instance itself is callable, like a function.
By the way, the __call__ needs to accept an additional argument:
def __call__(self, val):
since pool.map is essentially going to call in parallel
p = Parallel(args)
result = []
for val in self.list_objects:
result.append(p(val))
Pool.map simply applies a function (actually, a callable) in parallel. It has no notion of objects or classes. Since you pass it a class, it simply calls __init__ - __call__ is never executed. You need to either call it explicitly from __init__ or use pool.map(Parallel.__call__, preinitialized_objects)
I've got a question about defining functions and the self-parameter in python.
There is following code.
class Dictionaries(object):
__CSVDescription = ["ID", "States", "FilterTime", "Reaction", "DTC", "ActiveDischarge"]
def __makeDict(Lst):
return dict(zip(Lst, range(len(Lst))))
def getDict(self):
return self.__makeDict(self.__CSVDescription)
CSVDescription = __makeDict(__CSVDescription)
x = Dictionaries()
print x.CSVDescription
print x.getDict()
x.CSVDescription works fine. But print x.getDict() returns an error.
TypeError: __makeDict() takes exactly 1 argument (2 given)
I can add the self-parameter to the __makeDict() method, but then print x.CSVDescription wouldn't work.
How do I use the self-parameter correctly?
In python, the self parameter is implicitly passed to instance methods, unless the method is decorated with #staticmethod.
In this case, __makeDict doesn't need a reference to the object itself, so it can be made a static method so you can omit the self:
#staticmethod
def __makeDict(Lst): # ...
def getDict(self):
return self.__makeDict(self.__CSVDescription)
A solution using #staticmethod won't work here because calling the method from the class body itself doesn't invoke the descriptor protocol (this would also be a problem for normal methods if they were descriptors - but that isn't the case until after the class definition has been compiled). There are four major options here - but most of them could be seen as some level of code obfuscation, and would really need a comment to answer the question "why not just use a staticmethod?".
The first is, as #Marcus suggests, to always call the method from the class, not from an instance. That is, every time you would do self.__makeDict, do self.__class__.__makeDict instead. This will look strange, because it is a strange thing to do - in Python, you almost never need to call a method as Class.method, and the only time you do (in code written before super became available), using self.__class__ would be wrong.
In similar vein, but the other way around, you could make it a staticmethod and invoke the descriptor protocol manually in the class body - do: __makeDict.__get__(None, Dictionaries)(__lst).
Or, you could detect yourself what context its being called from by getting fancy with optional arguments:
def __makeDict(self, Lst=None):
if Lst is None:
Lst = self
...
But, by far the best way is to realise you're working in Python and not Java - put it outside the class.
def _makeDict(Lst):
...
class Dictionaries(object):
def getDict(self):
return _makeDict(self.__CSVDescription)
CSVDescription = _makeDict(__CSVDescription)