I'm working on a script using an API from a software I use. I lack documentation for the methods available so I have little control over what I have. I'm looking to call a variable that can be buried arbitrarily deep in an object. I know the depth before hand but it can range widely.
Example:
index = ['foo','bar']
object['foo']['bar'].method()
I've tried something like:
temp = object[index[0]]
for ind in index[1:]:
temp = temp[ind]
temp.method()
But this makes a copy (it does not according to the replies I've gotten) of the object and does not apply the method correctly. Index can be arbitrarily long.
My only working solution is to hardcode this by using:
if lengthIndex == 1:
object[index[0]].method()
if lengthIndex == 2:
object[index[0]][index[1]].method()
if lengthIndex == 3:
object[index[0]][index[1]][index[2]].method()
# and so on ...
What is the proper way to code this?
Your first code sample doesn't copy the object. In Python, variables are references (at least that's how it effectively works), and when you assign something to a variable, you're just copying a memory address, not copying the thing itself. So if your only reason not to use that was that you wanted to avoid making unnecessary copies of objects, you have nothing to worry about.
What you do in your code sample is the way I would handle repeated indexing.
If the main point of the question is asking for a function to index arbitrarily nested dictionaries then see the following:
def f():
print('nested func')
def g():
print('nested func 2')
def nested_value(nested_dict, keys):
for k in keys:
nested_dict = nested_dict[k]
return nested_dict
nested_dict = {
'foo': {
'bar': {
'baz': f
},
'taco': g
}
}
keys = ['foo', 'bar', 'baz']
val = nested_value(nested_dict, keys)
val()
keys = ['foo', 'taco']
val = nested_value(nested_dict, keys)
val()
Output:
nested func
nested func 2
In [5]: def get_deep_object(obj, index):
...: for i in index:
...: obj = obj[i]
...: return obj
...:
In [6]: l = ["foo", "bar"]
In [7]: get_deep_object(l, [1])
Out[7]: 'bar'
In [8]: get_deep_object(l, [1,1])
Out[8]: 'a'
In [9]: get_deep_object(l, [1,1]).capitalize()
Out[9]: 'A'
You can get a shorter (but not necessarily more readable) solution via functional programming:
from operator import getitem
from functools import reduce
reduce(getitem, index, object).method()
getitem from operator is equivalent to lambda data, key: data[key].
Related
I have an OrderedDict that I'm passing to a function. Somewhere in the function it changes the ordering, though I'm not sure why and am trying to debug it. Here is an example of the function and the function and output:
def unnest_data(data):
path_prefix = ''
UNNESTED = OrderedDict()
list_of_subdata = [(data, ''),] # (data, prefix)
while list_of_subdata:
for subdata, path_prefix in list_of_subdata:
for key, value in subdata.items():
path = (path_prefix + '.' + key).lstrip('.').replace('.[', '[')
if not (isinstance(value, (list, dict))):
UNNESTED[path] = value
elif isinstance(value, dict):
list_of_subdata.append((value, path))
elif isinstance(value, list):
list_of_subdata.extend([(_, path) for _ in value])
list_of_subdata.remove((subdata, path_prefix))
if not list_of_subdata: break
return UNNESTED
Then, if I call it:
from collections import OrderedDict
data = OrderedDict([('Item', OrderedDict([('[#ID]', '288917'), ('Main', OrderedDict([('Platform', 'iTunes'), ('PlatformID', '353736518')])), ('Genres', OrderedDict([('Genre', [OrderedDict([('[#FacebookID]', '6003161475030'), ('Value', 'Comedy')]), OrderedDict([('[#FacebookID]', '6003172932634'), ('Value', 'TV-Show')])])]))]))])
unnest_data(data)
I get an OrderedDict that doesn't match the ordering of my original one:
OrderedDict([('Item[#ID]', '288917'), ('Item.Genres.Genre[#FacebookID]', ['6003172932634', '6003161475030']), ('Item.Genres.Genre.Value', ['TV-Show', 'Comedy']), ('Item.Main.Platform', 'iTunes'), ('Item.Main.PlatformID', '353736518')])
Notice how it has "Genre" before "PlatformID", which is not the way it was sorted in the original dict. What seems to be my error here and how would I fix it?
It’s hard to say exactly what’s wrong without a complete working example. But based on the code you’ve shown, I suspect your problem isn’t with OrderedDict at all, but rather that you’re modifying list_of_subdata while iterating through it, which will result in items being unexpectedly skipped.
>>> a = [1, 2, 3, 4, 5, 6, 7]
>>> for x in a:
... print(x)
... a.remove(x)
...
1
3
5
7
Given your use, consider a deque instead of a list.
I need to modify a tuple during a for in loop, such that the iterator iterates on the tuple.
From my understanding, tuples are immutable; so tup = tup + (to_add,) is just reassigning tup, not changing the original tuple. So this is tricky.
Here is a test script:
tup = ({'abc': 'a'}, {'2': '2'})
blah = True
to_add = {'goof': 'abcde'}
for i in tup:
if blah:
tup = tup + (to_add,)
blah = False
print(i)
Which prints:
{'abc': 'a'}
{'2': '2'}
What I would like is for it to print:
{'abc': 'a'}
{'2': '2'}
{'goof': 'abcde'}
From what I understand, I need to "repoint" the implicit tuple iterator mid-script so that it is pointing at the new tuple instead. (I know this is a seriously hacky thing to be doing).
This script accesses the tuple_generator in question:
import gc
tup = ({'abc': 'a'}, {'2': '2'})
blah = True
to_add = {'goof': 'abcde'}
for i in tup:
if blah:
tup = tup + (to_add,)
blah = False
refs = gc.get_referrers(i)
for ref in refs:
if type(ref) == tuple and ref != tup:
refs_to_tup = gc.get_referrers(ref)
for j in refs_to_tup:
if str(type(j)) == "<class 'tuple_iterator'>":
tuple_iterator = j
print(i)
How can I modify this tuple_generator so that it points at the new tup, and not the old? Is this even possible?
I am aware that this is a really strange situation, I cannot change that tup is a tuple or that I need to use an implicit for in, as I am trying to plug into code that I cannot change.
There is no way-either portably or specifically within CPython—to do what you're trying to do from within Python, even via undocumented internals of the tuple_iterator object. The tuple reference is stored in a variable that isn't exposed to Python, and (unlike the stored index) isn't modified by __setstate__ or any other method.
However, if you're willing to start monkeying with C pointers behind CPython's back, and you know how to debug the inevitable segfaults…
Under the covers, there's a C struct representing tuple_iterator. I think it's either seqiterobject, or a struct with the exact same shape, but you should read through the tupleobject source code to make sure.
Here's what that type looks like in C:
typedef struct {
PyObject_HEAD
Py_ssize_t it_index;
PyObject *it_seq; /* Set to NULL when iterator is exhausted */
} seqiterobject;
So, what happens if you create a ctypes.Structure subclass that's the same size as this, something like this:
class seqiterobject(ctypes.Structure):
_fields_ = (
('ob_refcnt', ctypes.c_ssize_t),
('ob_type', ctypes.c_void_p),
('it_index', ctypes.c_ssize_t),
('it_seq', ctypes.POINTER(ctypes.pyobject)))
… and then do this:
seqiter = seqiterobject.from_address(id(j))
… and then do this:
seqiter.it_seq = id(other_tuple)
…? Well, you probably corrupt the heap by underreferencing the new value (and also leak the old one), so you'll need to incref the new value and decref the old value first.
But, if you do that… most likely, either it'll segfault the next time you call __next__, or it'll work.
If you want more example code that does similar things, see superhackyinternals. Other than the fact that seqiterobject is not even a public type, so this is even more hacky, everything else is basically the same.
You could write your own coroutine and send the new tup to it.
def coro(iterable):
iterable = iter(iterable)
while True:
try:
v = next(iterable)
i = yield v
except StopIteration:
break
if i:
yield v
iterable = it.chain(iterable, i)
Then this works as you describe:
In []:
blah = True
tup = ({'abc': 'a'}, {'2': '2'})
to_add = {'goof': 'abcde'}
c = coro(tup)
for i in c:
if blah:
i = c.send((to_add,))
blah = False
print(i)
Out[]:
{'abc': 'a'}
{'2': '2'}
{'goof': 'abcde'}
I'm sure there are lots of edge cases I'm missing in the above but it should give you an idea of how it can be done.
Since you plan on modifying the tuple inside the loop, you are probably better off using a while loop keeping track of the current index rather then relying on an iterator. Iterators are only good for looping through collections that do not get added/removed to in the loop.
If you run this below example, the resulting tup object has the items added to it, all while looping through 3 times.
tup = ({'abc': 'a'}, {'2': '2'})
blah = True
to_add = {'goof': 'abcde'}
i = 0
while i < len(tup):
cur = tup[i]
if blah:
tup = tup + (to_add,)
blah = False
i += 1
print(tup)
Is there a way, lib, or something in python that I can set value in list at an index that does not exist?
Something like runtime index creation at list:
l = []
l[3] = 'foo'
# [None, None, None, 'foo']
And more further, with multi dimensional lists:
l = []
l[0][2] = 'bar'
# [[None, None, 'bar']]
Or with an existing one:
l = [['xx']]
l[0][1] = 'yy'
# [['xx', 'yy']]
There isn't a built-in, but it's easy enough to implement:
class FillList(list):
def __setitem__(self, index, value):
try:
super().__setitem__(index, value)
except IndexError:
for _ in range(index-len(self)+1):
self.append(None)
super().__setitem__(index, value)
Or, if you need to change existing vanilla lists:
def set_list(l, i, v):
try:
l[i] = v
except IndexError:
for _ in range(i-len(l)+1):
l.append(None)
l[i] = v
Not foolproof, but it seems like the easiest way to do this is to initialize a list much larger than you will need, i.e.
l = [None for i in some_large_number]
l[3] = 'foo'
# [None, None, None, 'foo', None, None None ... ]
If you really want the syntax in your question, defaultdict is probably the best way to get it:
from collections import defaultdict
def rec_dd():
return defaultdict(rec_dd)
l = rec_dd()
l[3] = 'foo'
print l
{3: 'foo'}
l = rec_dd()
l[0][2] = 'xx'
l[1][0] = 'yy'
print l
<long output because of defaultdict, but essentially)
{0: {2: 'xx'}, 1: {0: 'yy'}}
It isn't exactly a 'list of lists' but it works more or less like one.
You really need to specify the use case though... the above has some advantages (you can access indices without checking whether they exist first), and some disadvantages - for example, l[2] in a normal dict will return a KeyError, but in defaultdict it just creates a blank defaultdict, adds it, and then returns it.
Other possible implementations to support different syntactic sugars could involve custom classes etc, and will have other tradeoffs.
You cannot create a list with gaps. You could use a dict or this quick little guy:
def set_list(i,v):
l = []
x = 0
while x < i:
l.append(None)
x += 1
l.append(v)
return l
print set_list(3, 'foo')
>>> [None, None, None, 'foo']
In my application I am receiving a string 'abc[0]=123'
I want to convert this string to an array of items. I have tried eval() it didnt work for me. I know the array name abc but the number of items will be different in each time.
I can split the string, get array index and do. But I would like to know if there is any direct way to convert this string as an array insert.
I would greately appreciate any suggestion.
are you looking for something like
In [36]: s = "abc[0]=123"
In [37]: vars()[s[:3]] = []
In [38]: vars()[s[:3]].append(eval(s[s.find('=') + 1:]))
In [39]: abc
Out[39]: [123]
But this is not a good way to create a variable
Here's a function for parsing urls according to php rules (i.e. using square brackets to create arrays or nested structures):
import urlparse, re
def parse_qs_as_php(qs):
def sint(x):
try:
return int(x)
except ValueError:
return x
def nested(rest, base, val):
curr, rest = base, re.findall(r'\[(.*?)\]', rest)
while rest:
curr = curr.setdefault(
sint(rest.pop(0) or len(curr)),
{} if rest else val)
return base
def dtol(d):
if not hasattr(d, 'items'):
return d
if sorted(d) == range(len(d)):
return [d[x] for x in range(len(d))]
return {k:dtol(v) for k, v in d.items()}
r = {}
for key, val in urlparse.parse_qsl(qs):
id, rest = re.match(r'^(\w+)(.*)$', key).groups()
r[id] = nested(rest, r.get(id, {}), val) if rest else val
return dtol(r)
Example:
qs = 'one=1&abc[0]=123&abc[1]=345&foo[bar][baz]=555'
print parse_qs_as_php(qs)
# {'abc': ['123', '345'], 'foo': {'bar': {'baz': '555'}}, 'one': '1'}
Your other application is doing it wrong. It should not be specifying index values in the parameter keys. The correct way to specify multiple values for a single key in a GET is to simply repeat the key:
http://my_url?abc=123&abc=456
The Python server side should correctly resolve this into a dictionary-like object: you don't say what framework you're running, but for instance Django uses a QueryDict which you can then access using request.GET.getlist('abc') which will return ['123', '456']. Other frameworks will be similar.
Is there a way to map a list onto a dictionary? What I want to do is give it a function that will return the name of a key, and the value will be the original value. For example;
somefunction(lambda a: a[0], ["hello", "world"])
=> {"h":"hello", "w":"world"}
(This isn't a specific example that I want to do, I want a generic function like map() that can do this)
In Python 3 you can use this dictionary comprehension syntax:
def foo(somelist):
return {x[0]:x for x in somelist}
I don't think a standard function exists that does exactly that, but it's very easy to construct one using the dict builtin and a comprehension:
def somefunction(keyFunction, values):
return dict((keyFunction(v), v) for v in values)
print somefunction(lambda a: a[0], ["hello", "world"])
Output:
{'h': 'hello', 'w': 'world'}
But coming up with a good name for this function is more difficult than implementing it. I'll leave that as an exercise for the reader.
If I understand your question correctly, I believe you can accomplish this with a combination of map, zip, and the dict constructor:
def dictMap(f, xs) :
return dict(zip(map(f, xs), xs)
And a saner implementation :
def dictMap(f, xs) :
return dict((f(i), i) for i in xs)
Taking hints from other answers I achieved this using map operation. I am not sure if this exactly answers your question.
mylist = ["hello", "world"]
def convert_to_dict( somelist ):
return dict( map( lambda x: (x[0], x), somelist ) )
final_ans = convert_to_dict( mylist )
print final_ans
If you want a general function to do this, then you're asking almost the right question. Your example doesn't specify what happens when the key function produces duplicates, though. Do you keep the last one? The first one? Do you actually want to make a list of all the words that start with the same letter? These questions are probably best answered by the user of the function, not the designer.
Parametrizing over these results in a more complicated, but very general, function. Here's one that I've used for several years:
def reduce_list(key, update_value, default_value, l):
"""Reduce a list to a dict.
key :: list_item -> dict_key
update_value :: key * existing_value -> updated_value
default_value :: initial value passed to update_value
l :: The list
default_value comes before l. This is different from functools.reduce,
because functools.reduce's order is wrong.
"""
d = {}
for k in l:
j = key(k)
d[j] = update_value(k, d.get(j, default_value))
return d
Then you can write your function by saying:
reduce_list(lambda s:s, lambda s,old:s[0], '', ['hello', 'world'])
# OR
reduce_list(lambda s:s, lambda s,old: old or s[0], '', ['hello', 'world'])
Depending on whether you want to keep the first or last word starting with, for example, 'h'.
This function is very general, though, so most of the time it's the basis for other functions, like group_dict or histogram:
def group_dict(l):
return reduce_list(lambda x:x, lambda x,old: [x] + old, [], l)
def histogram(l):
return reduce_list(lambda x:x, lambda x,total: total + 1, 0, l)
>>> dict((a[0], a) for a in "hello world".split())
{'h': 'hello', 'w': 'world'}
If you want to use a function instead of subscripting, use operator.itemgetter:
>>> from operator import itemgetter
>>> first = itemgetter(0)
>>> dict((first(x), x) for x in "hello world".split())
{'h': 'hello', 'w': 'world'}
Or as a function:
>>> dpair = lambda x : (first(x), x)
>>> dict(dpair(x) for x in "hello world".split())
{'h': 'hello', 'w': 'world'}
Finally, if you want more than one word per letter as a possibility, use collections.defaultdict
>>> from collections import defaultdict
>>> words = defaultdict(set)
>>> addword = lambda x : words[first(x)].add(x)
>>> for word in "hello house home hum world wry wraught".split():
addword(word)
>>> print words['h']
set(['house', 'hello', 'hum', 'home'])