I'm not sure how verbose I should go so please ask for elaboration if this is too terse.
Is there a way to pack the for a,b,c in product(d['a'],d['b'],d['c']): in some syntactical sugar so I would only need to type mute variables a,b,c only once for this loop itself?
Something like this may be?
my_for_loop('a','b','c'):
API_Call1(a)
API_Call2(b,c)
instead of
for a,b,c in product(d['a'],d['b'],d['c']):
API_Call1(a)
API_Call2(b,c)
How could my_for_loop look like? I am a bit lost at how to approach this conceptually.
More detail:
I have an API that requires calling it for each cell of a cartesian product of some lists. I am using the product function to avoid nested loops. Let's say we have a list1 and a list2 it can be done in a following way
from itertools import product
for a,b in product(list1,list2):
API_Call(a,b)
I have created a dictionary_of_lists={'a':list1,'b':list2,'c':list3...}
to be able to write it like this
for a,b in product(dictionary_of_lists['a'],dictionary_of_lists['b']):
API_Call(a,b)
for c,b in product(dictionary_of_lists['c'],dictionary_of_lists['b']):
API_Call(c,b)
for e,f,g,h in product(dictionary_of_lists['e'],dictionary_of_lists['f'],dictionary_of_lists['g'],dictionary_of_lists['h'],):
API_Call1(e,f,g,h)
API_Call2(e,h)
...
So basically the variables that the loop creates are used in that API calls and they are mute otherwise, their name doesn't matter. There are many of these calls and there is some convoluted logic around them. So I would like to keep the loop itself simple and should I need to change the variables I won't have to to change them at three places for each such loop.
my_for_loop('a','b'):
API_Call(a,b)
my_for_loop('b','c'):
API_Call(c,b)
my_for_loop('e','f','g','h'):
API_Call1(e,f,g,h)
API_Call2(e,h)
...
ADDITION:
I have simplified a few things but was taken by surprise where exactly ambiguity was lurking :-)
Thanks for all the answers so far!
It's a good suggestion to have the dproduct wrapper. I have one indeed, just did not want to preempt your suggestions.
The variable names are mute for the code logic but they have some meaning for the sake of maintenance of the code. So they can not consist of a single letter each.
In an attempt to clarify further: I would like to avoid using the variable names three times - in the "for ..." part, in the call to dproduct wrapper and in the API calls. Two times - in the call to the wrapper and in the API calls is OK because it reflects the logic.
Below is a more elaborated example of the code I have now.
def dproduct(d, keys):
subset_d = dict((k, d[k]) for k in keys if k in d)
return product(*[subset_d.values()])
for foo, bar in dproduct(d, ['foo','bar',]):
some logic here
if API_Call1(foo,bar) == 123:
some other stuff, API_Call6(quux,baz,)
some more stuff and a call to another dproduct
for quux, sl, baz in dproduct(d, ['quux','sl','baz',]):
blah blah, API_Call2(quux,sl,baz)
other stuff
for pa, mf in dproduct(d, ['pa','mf',]):
API_Call4(pa,mf)
for quux, sl, baz in dproduct(d, ['quux','sl','baz',]):
further logic
if API_Call1(quux, sl, baz) == 342: some other stuff
some more stuff and a call to another dproduct
for pa,mf in dproduct(d, ['pa','mf',]):
API_Call3(pa,mf)
First, you can write a product wrapper like this:
def dproduct(d, keys):
return product(*(d[key] for key in keys))
for a, b, c in dproduct(d, 'abc'):
API_Call1(a)
API_Call2(b, c)
(Notice that you can write the same thing with operator.itemgetter instead of a genexpr; it depends on which you find more readable.)
Of course I'm taking advantage of the fact that all of your keys have single-character names, and a string is an iterable of its characters. If your real names aren't single-character, you have to be slightly more verbose:
for a, b, c in dproduct(d, ('spam', 'eggs', 'beans')):
… at which point you might want to consider taking *args instead of args in the parameters—or, if you don't mind being hacky:
def dproduct(d, keys):
return product(*(d[key] for key in keys.split()))
for a, b, c in dproduct(d, 'spam eggs beans'):
You can then go farther and write wrappers around your calls, too. For example, if you yield dicts of values instead of tuples of values, you can use those dicts the same way we used the original dict of lists:
def dproduct(d, keys):
for vals in product(*(d[key] for key in keys)):
yield dict(zip(keys, vals))
def call1(d, keys):
return API_Call1(*(d[key] for key in keys))
def call2(d, keys):
return API_Call2(*(d[key] for key in keys))
for vals in dproduct(d, 'abc'):
call1(vals, 'a')
call2(vals, 'bc')
Or, if your API_Call* functions can take keyword instead of positional arguments, you can make things a lot simpler and cleaner:
def dproduct(d, keys):
for vals in product(*(d[key] for key in keys)):
yield dict(zip(keys, vals))
def dselect(d, keys):
return {key: d[key] for key in keys}
for vals in dproduct(d, 'abc'):
API_Call1(**dselect(vals, 'ab'))
API_Call2(**dselect(vals, 'c'))
If you can't use keyword arguments, and you have a lot of those API_Call* functions, you can go farther and generate the wrappers dynamically:
def apiwrap(apicall):
#functools.wraps(apicall)
def wrapper(d, keys):
return apicall(*(d[key] for key in keys))
return wrapper
apicall1 = apiwrap(API_Call1)
apicall2 = apiwrap(API_Call2)
# etc.
Although if you have lots of these, you probably want to stick them in a list or a dict in the first place…
If you want to get way too clever, you can even split the tuples up dynamically based on the signatures of the API functions:
def dispatch(d, keys, *calls):
for vals in product(*(d[key] for key in keys)):
it = iter(vals)
for call in calls:
args = islice(it, len(signature(call).parameters))
call(*args)
dispatch(d, 'abc', API_Call1, API_Call2)
(If your function bodies are pretty minimal, you probably want to speed things up by doing argcounts = [len(signature(call).parameters for call in calls] at the top of the function and then using zip(calls, argcounts) rather than using inspect each time in the inner loop.)
Anyway, without knowing more about your program, it's hard to say exactly what you can do, and what you should do—most of these ideas are not very Pythonic in general, even if they might be useful in some particular unusual case.
But either way, they should serve as examples of the kinds of things you can do pretty easily without having to get into horrible hacks involving locals and globals or getframe.
You can't easily insert variables into the local namespace (probably can but shouldn't without good cause). So use a dictionary to hold your named values.
I've used the operator.itemgetter function to grab the ones you want for the various API calls, and wrapped your product function in a generator.
from operator import itemgetter
from itertools import product
class DictCaller(dict):
def __call__(self, fn, *args):
fn(*map(itemgetter(self), args))
def my_product(d, *args):
for xs in product(*map(itemgetter(d), args)):
yield DictCaller(zip(args, xs))
for caller in my_product(*'abc'):
caller(API_CALL, *'ab')
caller(API_CALL1, *'bc')
Related
def cons(a, b):
def pair(f):
return f(a, b)
return pair
def car(f):
def left(a, b):
return a
return f(left)
def cdr(f):
def right(a, b):
return b
return f(right)
Found this python code on git.
Just want to know what is f(a,b) in cons definition is, and how does it work?
(Not a function I guess)
cons is a function, that takes two arguments, and returns a function that takes another function, which will consume these two arguments.
For example, consider the following function:
def add(a, b):
return a + b
This is just a function that adds the two inputs, so, for instance, add(2, 5) == 7
As this function takes two arguments, we can use cons to call this function:
func_caller = cons(2, 5) # cons receives two arguments and returns a function, which we call func_caller
result = func_caller(add) # func_caller receives a function, that will process these two arguments
print(result) # result is the actual result of doing add(2, 5), i.e. 7
This technique is useful for wrapping functions and executing stuff, before and after calling the appropriate functions.
For example, we can modify our cons function to actually print the values before and after calling add:
def add(a, b):
print('Adding {} and {}'.format(a, b))
return a + b
def cons(a, b):
print('Received arguments {} and {}'.format(a, b))
def pair(f):
print('Calling {} with {} and {}'.format(f, a, b))
result = f(a, b)
print('Got {}'.format(result))
return result
return pair
With this update, we get the following outputs:
func_caller = cons(2, 5)
# prints "Received arguments 2 and 5" from inside cons
result = func_caller(add)
# prints "Calling add with 2 and 5" from inside pair
# prints "Adding 2 and 5" from inside add
# prints "Got 7" from inside pair
This isn't going to make any sense to you until you know what cons, car, and cdr mean.
In Lisp, lists are stored as a very simple form of linked list. A list is either nil (like None) for an empty list, or it's a pair of a value and another list. The cons function takes a value and a list and returns you another list just by making a pair:
def cons(head, rest):
return (head, rest)
And the car and cdr functions (they stand for "Contents of Address|Data Register", because those are the assembly language instructions used to implement them on a particular 1950s computer, but that isn't very helpful) return the first or second value from a pair:
def car(lst):
return lst[0]
def cdr(lst):
return lst[1]
So, you can make a list:
lst = cons(1, cons(2, cons(3, None)))
… and you can get the second value from it:
print(car(cdr(lst))
… and you can even write functions to get the nth value:
def nth(lst, n):
if n == 0:
return car(lst)
return nth(cdr(lst), n-1)
… or print out the whole list:
def printlist(lst):
if lst:
print(car(lst), end=' ')
printlist(cdr(lst))
If you understand how these work, the next step is to try them on those weird definitions you found.
They still do the same thing. So, the question is: How? And the bigger question is: What's the point?
Well, there's no practical point to using these weird functions; the real point is to show you that everything in computer science can be written with just functions, no built-in data structures like tuples (or even integers; that just takes a different trick).
The key is higher-order functions: functions that take functions as values and/or return other functions. You actually use these all the time: map, sort with a key, decorators, partial… they’re only confusing when they’re really simple:
def car(f):
def left(a, b):
return a
return f(left)
This takes a function, and calls it on a function that returns the first of its two arguments.
And cdr is similar.
It's hard to see how you'd use either of these, until you see cons:
def cons(a, b):
def pair(f):
return f(a, b)
return pair
This takes two things and returns a function that takes another function and applies it to those two things.
So, what do we get from cons(3, None)? We get a function that takes a function, and applies it to the arguments 3 and None:
def pair3(f):
return f(3, None)
And if we call cons(2, cons(3, None))?
def pair23(f):
return f(2, pair3)
And what happens if you call car on that function? Trace through it:
def left(a, b):
return a
return pair23(left)
That pair23(left) does this:
return left(2, pair3)
And left is dead simple:
return 2
So, we got the first element of (2, cons(3, None)).
What if you call cdr?
def right(a, b):
return a
return pair23(right)
That pair23(right) does this:
return right(2, pair3)
… and right is dead simple, so it just returns pair3.
You can work out that if we call car(cdr(pair23)), we're going to get the 3 out of it.
And now you can write lst = cons(1, cons(2, cons(3, None))), write the recursive nth and printlist functions above, and trace through how they work on lst.
I mentioned above that you can even get rid of integers. How do you do that? Read about Church numerals. You define zero and successor functions. Then you can define one as successor(zero) and two as successor(one). You can even recursively define add so that add(x, zero) is x but add(x, successor(y)) is successor(add(x, y)), and go on to define mul, etc.
You also need a special function you can use as a value for nil.
Anyway, once you've done that, using all of the other definitions above, you can do lst = cons(zero(cons(one, cons(two, cons(three, nil)))), and nth(lst, two) will give you back one. (Of course writing printlist will be a bit trickier…)
Obviously, this is all going to be a lot slower than just using tuples and integers and so on. But theoretically, it’s interesting.
Consider this: we could write a tiny dialect of Python that has only three kinds of statements—def, return, and expression statements—and only three kinds of expressions—literals, identifiers, and function calls—and it could do everything normal Python does. (In fact, you could get rid of statements altogether just by having a function-defining expression, which Python already has.) That tiny language would be a pain to use, but it would a lot easier to write a program to reason about programs in that tiny language. And we even know how to translate code using tuples, loops, etc. into code in this tiny subset language, which means we can write a program that reasons about that real Python code.
In fact, with a couple more tricks (curried functions and/or static function types, and lazy evaluation), the compiler/interpreter could do that kind of reasoning on the fly and optimize our code for us. It’s easy to tell programmatically that car(cdr(cons(2, cons(3, None)) is going to return 3 without having to actually evaluate most of those function calls, so we can just skip evaluating them and substitute 3 for the whole expression.
Of course this breaks down if any function can have side effects. You obviously can’t just substitute None for print(3) and get the same results. So instead, you need some clever trick where IO is handled by some magic object that evaluates functions to figure out what it should read and write, and then the whole rest of the program, the part that users write, becomes pure and can be optimized however you want. With a couple more abstractions, we can even make IO something that doesn’t have to be magical to do that.
And then you can build a standard library that gives you back all those things we gave up, written in terms of defining and calling functions, so it’s actually usable—but under the covers it’s all just reducing pure function calls, which is simple enough for a computer to optimize. And then you’ve basically written Haskell.
How can I dynamically get the names and values of all arguments to a class method? (For debugging).
The following code works, but it would need to be repeated a few dozen times (one for each method). Is there a simpler, more Pythonic way to do this?
class Foo:
def foo(self, a, b):
myself = getattr(self, inspect.stack()[0][3])
argnames = inspect.getfullargspec(myself).args[1:]
d = {}
for argname in argnames:
d[argname] = locals()[argname]
log.debug(d)
That's six lines of code for something that should be a lot simpler.
Sure, I can hardcode the debugging code separately for each method, but it seems easier to use copy/paste. Besides, it's way too easy to leave out an argument or two when hardcoding, which could make the debugging more confusing.
I would also prefer to assign local variables instead of accessing the values using a kwargs dict, because the rest of the code (not shown) could get clunky real fast, and is partially copied/pasted.
What is the simplest way to do this?
An alternative:
from collections import OrderedDict
class Foo:
def foo(self, *args):
argnames = 'a b'.split()
kwargs = OrderedDict(zip(argnames, args))
log.debug(kwargs)
for argname, argval in kwargs.items():
locals()[argname] = argval
This saves one line per method, but at the expense of IDE autocompete/intellisense when calling the method.
As wpercy wrote, you can reduce the last three lines to a single line using a dict comprehension. The caveat is that it only works in some versions of Python.
However, in Python 3, a dict comprehension has its own namespace and locals wouldn't work. So a workaround is to put the locals func after the in:
from itertools import repeat
class Foo:
def foo(self, a, b):
myname = inspect.stack()[0][3]
argnames = inspect.getfullargspec(getattr(self, myname)).args[1:]
args = [(x, parent[x]) for x, parent in zip(argnames, repeat(locals()))]
log.debug('{}: {!s}'.format(myname, args))
This saves two lines per method.
This may sound a bit funny, but this is what am trying to do:
I have two lists: myList1 and myList2. I am passing these lists to a function that will perform a specific task depending on which list it receives (i.e., based on the name of the list, not content).
def listProcessor(data):
if data == 'myList1':
perform task1
else: #meaning data is myList2
perform task 2
Is there a way in Python 3 to inquire of the name of a list (or a tuple, dictionary, etc.) and do a comparison like this? Obviously, the way I am doing it here isn't working since it's comparing the content of the list 'data' to the single string 'myList1', which isn't working! :-).
There are a few ways to do this:
You can create separate functions if what needs to be done for each list really is completely separate.
You can update your function to either:
Take two lists as an argument:
def list_processor(list1=None, list2=None):
if list1:
# do stuff for list1
if list2:
# do stuff for list2
You can add an extra flag, identifying what kind of action to be performed, and set a default as well:
def list_processor(some_list=None, type_of_list=1):
if type_of_list == 1:
# do stuff with some_list as if it was list1
if type_of_list == 2:
# do stuff with some_list as if it was list2
You do not want to do what you initially proposed, for various reasons. One key reason is that in Python, what you may call variables in other languages are not "boxes to put stuff in" (as most textbooks refer to them).
In Python variables are actually just names that point to an object. The key thing is that multiple names can point to the same object; which will easily confuse your function if you rely on the lists "name".
Here is an example:
>>> a = [1,2,3]
>>> b = a
>>> b.append('hello')
>>> b is a
True
>>> b == a
True
>>> b
[1, 2, 3, 'hello']
>>> a
[1, 2, 3, 'hello']
In this example, both a and b are pointing to the same list object. What you do with a affects b.
So let's start with this: you really shouldn't do this. Data is just data in Python -- the identifier (e.g. the name you're using to talk about it) means nothing to the program logic itself. It's only meaningful to the programmer.
That said, there's ways to do what you're trying to do, and they're all the wrong thing to do. But they're possible, so let's talk about them.
globals() will give you a dictionary with keys of identifiers and values of, um, values, for all objects currently in the global scope. This means that you can do:
def list_processor(data):
g = globals()
data_name = next((k for k,v in g.items() if v is data))
if data_name == 'myList1':
... # do whatever you want here
Note, however, that you're looking through EVERYTHING in the global scope. First off, that's dumb since it's slow, and secondly, it's dumb because it's buggy. What if myList1 isn't being passed from the global scope? What if it's a local variable inside a function that never hits the global scope? Now your tricky hack fails.
The correct way to do this is to perform some sort of introspection on the argument being passed in as "data". For instance if myList1 always has 8 elements and myList2 always has 10:
def f_if_myList1(lst):
"""Do whatever we do if we pass myList1"""
...
def f_if_myList2(lst):
"""Do whatever we do if we pass myList2"""
...
def list_processor(data):
if len(data) == 8: # myList1!
f_if_myList1(data)
elif len(data) == 10:
f_if_myList2(data)
else:
# not sure. raise ValueError()?
I have a working solution to this question, it just doesn't feel very pythonic. I am working in Python 2.7 and, thus, cannot use Python 3 solutions.
I have a dictionary that is regularly being updated. Eventually a key, let's call it "foo", with a value will appear in the dictionary. I want to keep polling that object and getting that dictionary until the key "foo" appears at which point I want to get the value associated with that key and use it.
Here is some psuedo code that is functioning right now:
polled_dict = my_object.get_dict()
while('foo' not in polled_dict.keys()):
polled_dict = my_object.get_dict()
fooValue = polled_dict['foo']
Let me emphasize that what the code is doing right now works. It feels gross but it works. A potential saolution I came up with is:
fooValue = None
While fooValue is None:
polled_dict = my_object.get_dict()
fooValue = polled_dict.get('foo')
This also works but it only seems a tiny bit better. Instead of calling polled_dict.get('foo') twice once it shows up in the dict(the key is accessed during the while loop and again on exiting the while loop) we only call it once. But, honestly, it doesn't seem much better and the gains are minimal.
As I look over the other solutions I've implemented I see that they're just different logical permutations of the two above examples (a not in a different place or something) but nothing feels pythonic. I seems like there would be an easy, cleaner way of doing this. Any suggestions? If not, is either of the above better than the other?
EDIT A lot of answers are recommending I override or otherwise change the dictionaries that the code is polling from. I agree that this would normally be a great solution but, to quote from some of my comments below:
"The code in question needs to exist separately from the API that updates the dictionary. This code needs to be generic and access the dictionary of a large number of different types of objects. Adding a trigger would ultimately require completely reworking all of those objects (and would not be nearly as generic as this function needs to be) This is grossly simplified obviously but, ultimately, I need to check values in this dict until it shows up instead of triggering something in the object. I'm unconvinced that making such a wide reaching and potentially damaging change is a pythonic solution(though should the API be rewritten from the ground up this will definitely be the solution and for something that does not need to be separated/can access the API this is definitely the pythonic solution.)"
You could always do something like subclass dict.
This is completely untested, but something to the effect of:
class NoisyDict(dict):
def __init__(self, *args, **kwargs):
self.handlers = {}
#Python 3 style
super().__init__(*args, **kwargs)
def add_handler(self, key, callback):
self.handlers[key] = self.handlers.get(key, [])
self.handlers[key].append(callback)
def __getitem__(self, key):
for handler in self.handlers.get(key, []):
handler('get', key, super().__getitem__(key))
return super().__getitem__(key)
def __setitem__(self, key, value):
for handler in self.handlers.get(key, []):
handler('set', key, value)
return super().__setitem(value)
Then you could do
d = NoisyDict()
d.add_handler('spam', print)
d['bar'] = 3
d['spam'] = 'spam spam spam'
Fun with generators:
from itertools import repeat
gen_dict = (o.get_dict() for o in repeat(my_object))
foo_value = next(d['foo'] for d in gen_dict if 'foo' in d)
Is it not possible to do something like this? (obviously not thread safe) The only catch is that the method below does not catch dictionary initialization via construction. That is it wouldn't catch keys added when the dictionary is created; eg MyDict(watcher=MyWatcher(), a=1, b=2) - the a and b keys would not be caught as added. I'm not sure how to implement that.
class Watcher(object):
"""Watches entries added to a MyDict (dictionary). key_found() is called
when an item is added whose key matches one of elements in keys.
"""
def __init__(self, *keys):
self.keys = keys
def key_found(self, key, value):
print key, value
class MyDict(dict):
def __init__(self, *args, **kwargs):
self.watcher = kwargs.pop('watcher')
super(MyDict, self).__init__(*args, **kwargs)
def __setitem__(self, key, value):
super(MyDict, self).__setitem__(key, value)
if key in self.watcher.keys:
self.watcher.key_found(key, value)
watcher = Watcher('k1', 'k2', 'k3')
d = MyDict(watcher=watcher)
d['a'] = 1
d['b'] = 2
d['k1'] = 'k1 value'
If your object is modifying the dictionary in place then you should only need to get it once. Then you and your object have a pointer to the same dictionary object. If you need to stick with polling then this is probably the cleanest solution:
polled_dict = my_object.get_dict()
while 'foo' not in polled_dict:
pass # optionally sleep
fooValue = polled_dict['foo']
The best overall way of doing this would be to push some type of event through a pipe/socket/thread-lock in some way.
Maybe Try/Except would be considered more 'Pythonic'?
A sleep statement in the while loop will stop it consuming all your resources as well.
polled_dict = my_object.get_dict()
while True:
time.sleep(0.1)
try:
fooValue = polled_dict['foo']
return (foovalue) # ...or break
except KeyError:
polled_dict = my_object.get_dict()
I think a defaultdict is great for this kind of job.
from collections import defaultdict
mydeafultdict = defaultdict(lambda : None)
r = None
while r is None:
r = mydeafultdict['foo']
a defaultdict works just like a regular dictionary, except when a key doesn't exist, it calls the function supplied to it in the constructor to return a value. In this case, I gave it a lambda that just returns None. With this, you can keep trying to get foo, when there is a value associated with it, it will be returned.
In Python, can one define a function (that can have statements in it, thus not a lambda) in a way similar to the following JavaScript example?
var func = function(param1, param2) {
return param1*param2;
};
I ask, since I'd like to have a dictionary of functions, and I wouldn't want to first define all the functions, and then put them in a dictionary.
The reason I want a dictionary of functions is because I will have another function that takes another dictionary as parameter, loops through its keys, and if it finds a matching key in the first dictionary, calls the associated function with the value of the second dictionary as parameter. Like this:
def process_dict(d):
for k, v in d.items():
if k in function_dict:
function_dict[k](v)
Maybe there is a more pythonic way to accomplish such a thing?
Use a class (with static methods) instead of a dictionary to contain your functions.
class MyFuncs:
#staticmethod
def func(a, b):
return a * b
# ... define other functions
In Python 3, you don't actually need the #staticmethod since class methods are simple functions anyway.
Now to iterate:
def process_dict(d):
for k, v in d.items():
getattr(MyFuncs, k, lambda *x: None)(*v)
N.B. You could also use a module for your functions and use import.