I want to use multiprocessing to accelerate multiple calls to a function, the function takes as argument two values, a variable that changes for every calculation and a constant variable.
Here is a code that emulates my problem:
import pathos.pools as pp
p = pp.ProcessPool(4)
def add(x,y):
return x+y
x = [0,1,2,3]
y = 5
result = p.map(add, x, y)
As y is not a list I get the following error:
TypeError: izip argument #2 must support iteration
In this simple case the straightforward solution would be to make y a list of the constant values:
y = [5 for value in x]
But I would like to avoid this solution as in my case y is a complex object that takes up quite a big chunk of memory.
Thanks for any suggestions
you can just use local variables for the function you declare. I often make a wrapper to pass non-iterables, e.g.
def add(x,y):
return x+y
def add_wrapper(x):
y = 5
return add(x,y)
x = [0,1,2,3]
result = p.map(add_wrapper, x)
or
def add(x,y):
return x+y
def add_wrapper(x):
return add(x,y)
y = 5
x = [0,1,2,3]
result = p.map(add_wrapper, x)
Related
From this answer: How do chained assignments work?, I understand that chained assignement in Python :
x = y = z # (1)
is equivalent to:
temp = z
x = temp
y = temp
But is (1) also equivalent to:
x = z
y = x
?
Or is there a slight difference (for example when z = some_function())? If so, which difference?
In the very example you give, yes, the effects of the two approaches are practically identical because both involve simply assigning the same reference to a number of names.
Be aware, however, that if the expressions in the assignment targets involve more complex evaluations, the two approaches could be different.
For example, consider the following chain expression, where x is initialized as a dict and expensive_func is a time-consuming function that returns a key:
x[expensive_func()] = y = some_function()
While it would be indeed equivalent to the following:
temp = some_function()
x[expensive_func()] = temp
y = temp
it would not be be equivalent to the second approach:
x[expensive_func()] = some_function()
y = x[expensive_func()]
since expensive_func would then have to be called twice, doubling the time taken, and triggering the side effect of the function twice, if it has any.
Also, consider the following code:
obj = []
x = []
x[:] = y = obj
print(id(obj), id(x), id(y))
where the output would show that y gets assigned the same reference as obj, while x is different.
That code is then indeed equivalent to:
obj = []
x = []
temp = obj
x[:] = temp
y = temp
print(id(obj), id(x), id(y))
But not equivalent to:
obj = []
x = []
x[:] = obj
y = x[:]
print(id(obj), id(x), id(y))
The latter of which would show y getting a different reference from both obj and x.
I always find using examples to be the best way to understand things (in general).
Let's say we have a func:
def function_sample ():
print(20)
If you print it:
print(function_sample)
<function function_sample at 0x7f8f840a01f0>
returns the function object.
When assigning to a variable a function without parentheses (without calling/running it).
x = function_sample
print(x)
you will get the same message: <function function_sample at 0x7f8f840a01f0>
However, if you run it (with parentheses).
print(x())
You will see :
20
None
Why None? It's because Python functions have a default return value, which is None if no return expression is given, or return is given on its own.
Another sample:
def another_sample(some):
print(some)
y = another_sample
print(y)
As you probably have guessed it : <function another_sample at 0x7f8f7e747700>
If you try to print y() you will get an error because the some argument is missing.
But if we add one:
print(y(5))
5
None
One last example:
def third_sample ():
return 20
aa = third_sample # without running the func
bb = third_sample() # calling/running the func
print(aa) # function object
print(bb) # 20
The 2 approaches you have shown are both functional and legit in terms of going about chaining and using previous variables. No difference at all
When assigning variables to the same number of variables, instead of doing the typical:
x = 0
y = 0
OR using tuple unpacking approach:
(x,y) = 0,0
You could just do like what you have (chained assignment):
x = y = 0
This could be used with any object (being called on) for the RHS, and that:
x = y = some_object()
is the same as:
tmp = some_object()
x = tmp
y = tmp
and when you del tmp, the xand y become useless or nothing.
I want to generate an array in python numpy based on equation; 1/x for x = 1,2,3,...10 and I wrote; But I wasn't getting any output, please help
def Number(x):
for x in range (1,11):
y = 1/x
return y
y = Number(10)
print y
If you want to return an list, you need to actually make a list and append to it (or use a list comprehension). You just create a variable and assign to it. Instead do:
def Number(x):
y = []
for x in range (1,11):
y.append(1./x)
return y
y = Number(10)
print y
Now if you want to, you could use a list comprehension. This is a pythonic way to generate a list in a single line. It would look something like this.
def Number(x):
y = [1./x for x in range(1,11)]
return y
y = Number(10)
print y
Another way to do this is to use the map builtin, which is different in python 3, but you are using 2.7, so we are good.
def Number(x):
y = map(lambda x: 1./x, range(1,11))
return y
y = Number(10)
print y
The map function is applies the function to the specified list. In this case, I use an anonymous lambda function lambda x:1./x which is a simple way of writing a function with x as an argument and returns 1/x. The map function applies the lambda function to each element in the list.
This can be done as follow
def Number(n):
return np.array([1/i for i in range(1, n)])
now you can do that
y = Number(10)
Say I have the following example, in python:
import numpy as np, PROGRAMS as prg
testlist = []
x = 0
n=0
y=[1,2,3,4,5]
x_fn = np.array=([prg.test1(x),prg.test2(x),prg.test3(x)])
for i in range(0,len(x_fn)):
for j in range(0, len(y)):
x = y[j]*2
z=x_fn[i]
testlist.append(z)
j = j+1
i = i+1
print testlist
#####PROGRAMS
def test1(x):
x=x**2
return x
def test2(x):
x=x**3
return x
def test3(x):
x=x+10
return x
If x isn't defined before x_fn then an error occurs but if I define it as zero then that is what is used in the calculations. I basically want this code to produce a list with the the defined value of x in the 2nd loop :
x = y[j]*2
for all values of y. I know there would be a way around this mathematically - but I would like to solve it by running the same function and not changing any of the values of y or any of the functions in PROGRAMS.
Basically, is it a good idea to put these functions in a array and run through it element by element or is there a better way to do it?
Thanks in advance for your replies,
Sven D.
Could this be what you want ?
def test1(x):
x=x**2
return x
def test2(x):
x=x**3
return x
def test3(x):
x=x+10
return x
testlist = []
n=0
y_vals=[1,2,3,4,5]
x_fn = [test1, test2, test3]
for fun in x_fn:
for y in y_vals:
x = y*2
z=fun(x)
testlist.append(z)
print testlist
Functions are objects that can be stored in containers and recalled for use later just like any other object in Python.
You don't even need to use numpy arrays. Just use a list (functions) and put the test functions in it. Note, that I have removed the argument. The elements of the functions array are references to the functions, so you can use them in your loop.
import PROGRAMS as prg
testlist = []
y=[1,2,3,4,5]
functions = [prg.test1, prg.test2, prg.test3]
for func in functions:
for j in y:
x = j*2
z = func(x)
testlist.append(z)
print testlist
#####PROGRAMS
def test1(x):
x=x**2
return x
def test2(x):
x=x**3
return x
def test3(x):
x=x+10
return x
I want to create functions and add them to a list, reusing the same name every time.
def fconstruct():
flist = []
for x in xrange(0,5):
def kol():
return x
flist.append(kol)
del kol #this doesn't fix the problem.
return flist
k = fconstruct()
However, this fails, even if i delete the function every loop, and no matter which of the functions in k i call, the result is the the same: 4, because the newest definition of kol has changed all the previous ones. For a simple function such as this,
kol = lambda: x
would work. However, i need to do this for a much more complex function
As solutions, i could store the function as a string in the list and use exec to call it.
I could generate disposable and random function names:
fname = '_'+str(random.random())
exec fname + ' = kol'
exec 'flist.append('+fname+')'
I could play around with this implementation of multiline lambdas: https://github.com/whaatt/Mu
None of these seem elegant, so what is the preferred way of doing this?
You have to use another function that generates the function you want with the x parameter set. Here I use the kol_factory (see also the answer to Closures in Python):
def fconstruct():
flist = []
# create a function g with the closure x
def kol_factory(y):
# y is local here
def kol():
# kol uses the y
return y
return kol
for x in xrange(0,5):
# we create a new g with x "burned-in"
flist.append(kol_factory(x))
return flist
for k in fconstruct():
print k()
You can define the factory function factory outside the fconstruct function:
def kol_factory(y):
# y is local here
def kol():
# kol uses the y
return y
return kol
def fconstruct():
flist = []
for x in xrange(0,5):
# we create a new kol_factory with x "burned-in"
flist.append(kol_factory(x))
return flist
for k in fconstruct():
print k()
When you're defining kol, you're establishing a closure around x. In fact, each time through the loop you're getting a closure around the same variable.
So while you have 5 different functions all named kol, they all return the value of the same variable. When the loop is finished, that variable's value is 4, so each function returns 4.
Consider this:
def fconstruct():
flist = []
for x in range(5):
def get_kol(y):
def kol():
return y
return kol
flist.append(get_kol(x))
return flist
In this case, the function get_kol() returns a function who's return value is get_kol()'s argument.
The closure in kol() is now around y, which is local to the get_kol() function, not the loop. Each time get_kol() is called, a new local y is created, so each kol() gets a different variable closure.
An alternative way is to create a partial function with functools.partial. This accomplishes the same thing (creates a function which, when called executes another function with arguments), and is a lot more powerful
def f(a): # whatever arguments you like
return a
# create a bunch of functions that return f(x)
flist = [functools.partial(f, x) for x in range(5)]
def g(a, b):
return a + b
# create a bunch of functions that take a single argument (b), but have a set to
# 0..4:
flist = [functools.partial(g, x) for x in range(5)]
some_g = flist[0]
some_g(1) # returns 0 + 1
Is there a shortcut in python for applying the same function multiple times to a variable (and its output)?
Something like:
# n times foo(x)
instead of
foo(foo(foo...(foo(x))))...)
for i in range(n):
x = foo(x)
If you need to preserve x, use a different name
baz = x
for i in range(n):
baz = foo(baz)
If you are desperate for a shortcut (eg. for codegolf)
reduce(lambda x,y:foo(x),[x]*n)
def double(x):
return x * 2
i = 44
for i in xrange(1,100):
i = double(i)
Could you mean a simple for loop?
One of the ways i can think of is creating a generic recursive function to do this
def repeatX(foo, output, count):
if not count:
return output
else:
return repeatX(foo, foo(output), count -1)