We Keep Coding

Unpacking a python tuple in a python function

Using Visual Studio & http://pythontutor.com/visualize I am unable to get the result for filesize function because of the following Error: TypeError: file_size() takes 1 positional argument but 3 were given. Code below.
#this function stores information about a file, its name, its type and its size in bytes.
def file_size(file_info):
name, file_type, size = file_info
return("{:.2f}".format(size/1024))
print(file_size('Class Assignment','docx', 17875)) #Should print 17.46
print(file_size('Notes','txt', 496)) #Should print 0.48
print(file_size('Program','py', 1239)) #Should print 1.21
I though unpacking (file_info) within the function will override the 1 positional argument but 3 were given error. What am I doing wrong?

Currently you are passing three distinct arguments ... pass it instead as one argument which is a tuple:
print(file_size( ('Class Assignment','docx', 17875) ))
Alternatively, you could alter your function declaration to capture distinct arguments into a tuple when called:
def file_size(*file_info):
...
Then calling it as you are is valid.

Put the values into a tuple like this:
print(file_size(('Class Assignment','docx', 17875)))
print(file_size(('Notes','txt', 496)))
print(file_size(('Program','py', 1239)))
This way you are passing one variable

Please try this:
def file_size(file_info):
name, type, size= file_info
return("{:.2f}".format(size / 1024))

Please try this.
def file_size(file_info):
(name,file_type,size) = file_info
return("{:.2f}".format( size/ 1024))
You need to pass file_info in a tuple in python.

If you want to keep the function with one parameter, try assigning the tuple to a variable and pass it in:
file_size(*varname)

Passing multiple arguments to pool.map using class function

I'm trying to thread as described in this post, and also pass multiple arguments in Python 2.7 through a work-around described here.
Right now I have something like this, a function that is part of class pair_scraper:
def pool_threading(self):
pool = ThreadPool(4)
for username in self.username_list:
master_list = pool.map(self.length_scraper2,
itertools.izip(username*len(self.repo_list),
itertools.repeat(self.repo_list)))
def length_scraper2(self, username, repo):
#code
However, when I run my code I get the error:
TypeError: length_scraper2() takes exactly 3 arguments (2 given)
Which seems to be because it wants self passed as an argument, which is nonsensical given I'm using a class function within the class. Thoughts on how to fix?

itertools.izip(username*len(self.repo_list),itertools.repeat(self.repo_list)) yields a tuple.
You need to pass 2 arguments explicitly to your method (self is implicitly passed because it's a non-static method), but you only pass 1 tuple explicitly, plus the implicit self which makes 2 arguments, hence the confusing error message.
You have to use * to pass your tuple as 2 separate arguments, like this:
master_list = pool.map(self.length_scraper2,
*itertools.izip(username*len(self.repo_list),itertools.repeat(self.repo_list)))
simple test using the classical map on a simple function:
def function(b,c):
return (b,c)
print(list(map(function,zip([1,2],[4,5]))))
error:
print(list(map(function,zip([1,2],[4,5]))))
TypeError: function() missing 1 required positional argument: 'c'
now adding single asterisk to expand args:
print(list(map(function,*zip([1,2],[4,5]))))
works:
[(1, 2), (4, 5)]
same goes for class method:
class Foo:
def function(self,b,c):
return (b,c)
f = Foo()
print(list(map(f.function,*zip([1,2],[4,5]))))

Can anyone explain how this functional program work?

def apply_twice(func,arg):
return func(func(arg))
def add_five(x):
return x+5
print (apply_twice(add_five,10))
The output I get is 20.
This one is actually confusing me like how is it working.Can anybody explain me how this is working by breaking it down

The function apply_twice(func,arg) takes two arguments, a function object func and an argument to pass to the function func called arg.
In Python, functions can easily be passed around to other functions as arguments, they are not treated differently than any other argument type (i.e first class citizens).
Inside apply_twice, func is called twice in the line:
func(func(arg))
Which, alternatively, can be viewed in a more friendly way as:
res = func(arg)
func(res)
If you replace func with the name of the function passed in add_five you get the following:
res = add_five(arg) # equals: 15
add_five(res) # result: 20
which, of course, returns your expected result.
The key point to remember from this is that you shouldn't think of functions in Python as some special construct, functions are objects just like ints, listss and everything else is.

Expanding the code it executes as follows, starting with the print call:
apply_twice(add_five,10))
add_five(add_five(10)) # add_five(10) = 15
add_five(15) # add_five(15) = 20
Which gives you the result: 20.
When apply_twice is called, you are passing in a function object and a value. As you can see in the apply_twice definition, where you see func that is substituted with the function object passed to it (in this case, add_five). Then, starting with the inner func(arg) call, evaluate the result, which is then passed to add_five again, in the outer return func( ... ) call.

What you need to understand here is that
apply_twice(func,arg)
is a higher function which accepts two arguments (another function named func and an argument arg). The way it works is that it first evaluate the value of the other function, then use the value as an argument inside the higher function.
remember we have a function add_five(x) which add 5 to the argument supply in it...
then this function add_five(x) is then passed as an argument to another function called
apply_twice_(func,arg) which return func(func(arg)).
now splitting func(func(arg)) we have
func(arg) #lets called it a
then func(func(arg))==func(a) since a = func(agr)
and (a) is our add_five(x) function, after it add 5, then the value we got is re-used as another fresh argument to add another 5 to it, that is why we have 20 as our result.
Another example is:
def test(func, arg):
return func(func(arg))
def mult(x):
return x * x
print(test(mult, 2))
which give 16 as result.

How to correctly call function with optional parameters in python

I'm a beginner with python and I'm facing a problem with a function that requires optional parameters.
This function gets as parameters a variable number of file paths, that can be from 2 to n parameters.
After that, a certain number of optional parameters can be passed to this function.
I tried to do something like that:
def compareNfilesParameters(*args):
start_time = time.time()
listFiles = []
listParameters = []
for argument in args:
if str(argument).endswith(".vcf"):
listFiles.append(str(argument))
else:
listParameters.append(argument)
So if the parameters has the file extension it is considered as one of the file path parameters, the others are seen as the optional parameters.
What I want to do is letting the user call the function like:
function('a.vcf', 'b.vcf', 'c.vcf')
or
function('a.vcf', 'b.vcf', 'c.vcf', 0, 1)
or
function('a.vcf', 'b.vcf', 'c.vcf', 0, 1, 4,...,3)
I tried different approaches but none of them satisfies me.
The first approach is declaring the function as:
def compareNfilesParameters(*args)
but this way, if I get for example 3 parameters, 2 will certainly be the files path, and the last one I don't know on which variable it refers. So I need to specify every value and pass '-1' for the parameters that I want to use default value.
The 2nd approach is the following:
def compareNfilesParameters(*args, par1 = 10, par2 = 15 ..)
But this way I need to call the function like:
compareNfilesParameters(path1, path2, path3, par1 = 10)
and not like
compareNfilesParameters(path1, path2, path3, 10)
or the 10 will be considered in the args input, right? I wouldn't like to use this approach because it becomes very verbose to call the function.
How would you do this?

Make the user pass in the filenames as a sequence; don't try to cram everything into separate arguments:
def compareNfilesParameters(files, *params):
and call this as:
compareNfilesParameters(('a.vcf', 'b.vcf', 'c.vcf'), 0, 1, 4)
This makes the files explicit and removes the need to separate files from other parameters.
If your remaining parameters are distinct options (and not a homogenous series of integers), I'd use keyword arguments:
def compareNfilesParameters(files, op1=default_value, op2=default_value, op3=default_value):
You don't have to use keyword arguments with keywords when calling; you can still treat them as positional:
compareNfilesParameters(('a.vcf', 'b.vcf', 'c.vcf'), 0, 1, 4)
would give op1 the value 0, op2 the value 1, and op3 the value 4. Only if you want to specify values out of order or for a specific option do you have to use keyword arguments in the call:
compareNfilesParameters(('a.vcf', 'b.vcf', 'c.vcf'), op3=4)

Ok, I solved like using the keyword parameters as suggested.
def compareNfilesParameters(listFiles, **kwargs):
start_time = time.time()
if len(listFiles) < MINUMUM_FILES_NUMBER :
print "You need to specify at least "+ str(MINUMUM_FILES_NUMBER) +" files."
return
try:
operationType = int(kwargs.get("op", DEFAULT_OPERATION_TYPE))
except ValueError:
print "Operation type filter has to be an integer."
return
if operationType not in [0,1]:
print "Operation type must be 0 (intersection), 1 (union)"
return
and so on for all the parameters.
Like this I need to put all the files paths in a list and pass it as a single required parameter, and searching kwargs dictionary for optionals parameters setting the default values if not expressed.

Why is multiprocessing's apply_async so picky?

Sample code that works without issue:
from multiprocessing import *
import time
import random
def myfunc(d):
a = random.randint(0,1000)
d[a] = a
print("Process; %s" % a)
print("Starting mass threads")
man = Manager()
d = man.dict()
p = Pool(processes=8)
for i in range(0,100):
p.apply_async(myfunc, [d])
p.close()
p.join()
print(d)
print("Ending multiprocessing")
If you change p.apply_async(myfunc, [d]) to p.apply_async(myfunc, (d)) or p.apply_async(myfunc, d) then the pool will not work at all. If you add another arg to myfunc and then just pass in a None it'll work like this p.apply_async(myfunc, (None, d)) — but why?

The documentation for apply_async says the following:
apply(func[, args[, kwds]])
Call func with arguments args and keyword arguments kwds. It blocks until the result is ready. Given this blocks, apply_async() is better suited for performing work in parallel. Additionally, func is only executed in one of the workers of the pool.
Thus instead of taking star and double star arguments, it takes positional arguments and keyword arguments to be passed to the target function as the 2nd and 3rd arguments to the function; the second must be an iterable and the 3rd one a mapping, respectively.
Notice that since the apply works asynchronously, you won't see any exceptions, unless you .wait and .get them from the results;
You can try simply:
for i in range(0,100):
result = p.apply_async(myfunc, d)
print(result.get())
In the code above, the result.get() waits for the completion of the 100th thread and returns its returned value - or tries as it will fail, because the managed dictionary cannot be used as the positional arguments:
Traceback (most recent call last):
File "test.py", line 21, in <module>
print(result.get())
File "/usr/lib/pythonN.N/multiprocessing/pool.py", line 558, in get
raise self._value
KeyError: 0
Thus, looking at your original question: do note that [d] is a list of length 1; (d) is the same as d; to have a tuple of length 1 you need to type (d,). From the Python 3 tutorial section 5.3:
A special problem is the construction of tuples containing 0 or 1
items: the syntax has some extra quirks to accommodate these. Empty
tuples are constructed by an empty pair of parentheses; a tuple with
one item is constructed by following a value with a comma (it is not
sufficient to enclose a single value in parentheses). Ugly, but
effective. For example:
>>> empty = ()
>>> singleton = 'hello', # <-- note trailing comma
>>> len(empty)
0
>>> len(singleton)
1
>>> singleton
('hello',)
(d,), [d], {d}, or even iter(frozenset(d)) or {d: True} would work just nicely as your positional arguments; all these as args would result in an Iterable whose iterator yields exactly 1 value - that of d. On the other hand, if you had passed almost any other kind of value than that unfortunate managed dictionary, you would have gotten a much more usable error; say if the value was 42, you'd have got:
TypeError: myfunc() argument after * must be a sequence, not int