We Keep Coding

Can I combine multiple dicts with the same name in Python?

I'm trying to create a new list from an API return in python. The purpose of this API call is to pull a list of driver's names, and pair them to a vehicle ID native to the API service. The code currently looks like this:
url = url
headers = {
"accept": "application/json",
"authorization": auth
}
response = requests.get(url, headers=headers)
response = response.json()
for doc in response['data']:
try:
doc['id'],
doc['staticAssignedDriver']['name']
except:
pass
else:
names = {
doc['staticAssignedDriver']['name']: doc['id']
}
names.update(names)
print(type(names))
print(names)
This prints a list of unique names and id's as individual dicts. IE:
{'name1':'id1'}
<class 'dict'>
{'name2':'id2'}
<class 'dict'>
Until I have all of my name:id pairs from my API.
But I'd like to make that a single dict, as such:
{'name1': 'id1', 'name2': 'id2'}
It seems like each new name/id pair ends up being its own var 'names'. Is there a way to make this its own singular dict, instead of individual?

When you do names = {whatever: whatever}, you always create a new dictionary with exactly one key and value. If you want to have only one dictionary that you update over and over, create the dictionary outside of the loop, and just assign a single value into it at a time:
names = {}
for doc in ...:
...
names[doc['staticAssignedDriver']['name']] = doc['id']

x = [{'name1':'id1'},
{'name2':'id2'}]
d = {}
for dct in x:
d.update({key: value for key, value in dct.items()})
print(d)
{'name1': 'id1', 'name2': 'id2'}

Python / Django: Get var from cookies

I tried to get distinct_id with request.COOKIES.get('distinct_id'). However Mixpanel saves the data in a not extractable way for me. Anyone knows why there are all these %22%3A%20%22 and how to extraxt distinct_id?
print(request.COOKIES):
{
'djdt': 'hide',
'cookie_bar': '1',
'mp_1384c4d0e46aaaaad007e3d8b5d6eda_mixpanel': '%7B%22distinct_id%22%3A%20%22165edf326870-00fc0e7eb72ed3-34677908-fa000-165e40c268947b%22%2C%22%24initial_referrer%22%3A%20%22%24direct%22%2C%22%24initial_referring_domain%22%3A%20%22%24direct%22%2C%22__alias%22%3A%20%22maz%2B1024%40gmail.com%22%7D',
'csrftoken': 'nvWzsrp3t6Sivkrsyu0gejjjjjiTfc36ZfkH7U7fgHaI40EF',
'sessionid': '7bkel6r27ebd55x262cv9lzv61gzoemw'
}

Check this code. You can run it because use the example you shared. First you must unquote the data in the mixpanel value. I used the suffix of the cookie key to get it. Then after the unquote you must load the json to get back a dictionary.
The code here prints all the keys in the dictionary, but you can easily get the distinct_id using mixpanel_dict.get('distinct_id')
Try it.
from urllib import parse
import json
cookie = {'djdt': 'hide',
'cookie_bar': '1',
'mp_1384c4d0e46aaaaad007e3d8b5d6eda_mixpanel': '%7B%22distinct_id%22%3A%20%22165edf326870-00fc0e7eb72ed3-34677908-fa000-165e40c268947b%22%2C%22%24initial_referrer%22%3A%20%22%24direct%22%2C%22%24initial_referring_domain%22%3A%20%22%24direct%22%2C%22__alias%22%3A%20%22maz%2B1024%40gmail.com%22%7D',
'csrftoken': 'nvWzsrp3t6Sivkrsyu0gejjjjjiTfc36ZfkH7U7fgHaI40EF',
'sessionid': '7bkel6r27ebd55x262cv9lzv61gzoemw'
}
def get_value_for_mixpanel(cookie):
mixpanel_dict = {}
for key in cookie.keys():
if '_mixpanel' in key:
value = parse.unquote(cookie.get(key))
mixpanel_dict = json.loads(value)
return mixpanel_dict
if __name__ == "__main__":
mixpanel_dict = get_value_for_mixpanel(cookie) # type: dict
for key,value in mixpanel_dict.items():
print("%s:%s" %(key, value))
Result
distinct_id:165edf326870-00fc0e7eb72ed3-34677908-fa000-165e40c268947b
$initial_referrer:$direct
$initial_referring_domain:$direct
__alias:maz+1024#gmail.com

Try unquote()
>>> s = '/path/to/my/handler/?action=query&id=112&type=vca&info=ch%3D0%26type%3Devent%26ev46[sts%3Dbegin'
>>> import urllib
>>> urllib.unquote(s)
>>> '/path/to/my/handler/?action=query&id=112&type=vca&info=ch=0&type=event&ev46[sts=begin'
Credits : https://stackoverflow.com/a/11215316/5647272

Can you use {} and .format to put values into a dictionary

I am writing a script to query an ArcGIS rest service and return records. I want to use {} and .format to allow a dictionary item to be changed a time. How do I write this:
time = '2016-10-06 19:18:00'
URL = 'http://XXXXXXXXX.gov/arcgis/rest/services/AGO_Street/StreetMaint_ServReqs/FeatureServer/10/query'
params = {'f': 'pjson', 'where': "CLOSE_DATE > '{}'", 'outfields' : 'OBJECTID, REPORTED_DATE, SUMMARY, ADDRESS1, REQUEST_STATUS, CLOSE_DATE, INCIDENT_NUMBER', 'returnGeometry' : 'false'}.format(time)
req = urllib2.Request(URL, urllib.urlencode(params))
if I use this for param it will work
params = {'f': 'pjson', 'where': "CLOSE_DATE > '2016-10-06 19:18:00'", 'outfields' : 'OBJECTID, REPORTED_DATE, SUMMARY, ADDRESS1, REQUEST_STATUS, CLOSE_DATE, INCIDENT_NUMBER', 'returnGeometry' : 'false'}
What is the proper python formatting to do this?

str.format is a string method, not a method on a dictionary. Just apply the method to that one string value:
params = {
'f': 'pjson',
'where': "CLOSE_DATE > '{}'".format(time),
'outfields' : 'OBJECTID, REPORTED_DATE, SUMMARY, ADDRESS1, REQUEST_STATUS, CLOSE_DATE, INCIDENT_NUMBER',
'returnGeometry' : 'false'
}
Each of the key and value parts in a dictionary definition is just another expression, you are free to use any valid Python expression to produce the value, including calling methods on the string and using the result as the value.

Try this:
'where': "CLOSE_DATE > '{}'".format(time)

How to decode dataTables Editor form in python flask?

I have a flask application which is receiving a request from dataTables Editor. Upon receipt at the server, request.form looks like (e.g.)
ImmutableMultiDict([('data[59282][gender]', u'M'), ('data[59282][hometown]', u''),
('data[59282][disposition]', u''), ('data[59282][id]', u'59282'),
('data[59282][resultname]', u'Joe Doe'), ('data[59282][confirm]', 'true'),
('data[59282][age]', u'27'), ('data[59282][place]', u'3'), ('action', u'remove'),
('data[59282][runnerid]', u''), ('data[59282][time]', u'29:49'),
('data[59282][club]', u'')])
I am thinking to use something similar to this really ugly code to decode it. Is there a better way?
from collections import defaultdict
# request.form comes in multidict [('data[id][field]',value), ...]
# so we need to exec this string to turn into python data structure
data = defaultdict(lambda: {}) # default is empty dict
# need to define text for each field to be received in data[id][field]
age = 'age'
club = 'club'
confirm = 'confirm'
disposition = 'disposition'
gender = 'gender'
hometown = 'hometown'
id = 'id'
place = 'place'
resultname = 'resultname'
runnerid = 'runnerid'
time = 'time'
# fill in data[id][field] = value
for formkey in request.form.keys():
exec '{} = {}'.format(d,repr(request.form[formkey]))

This question has an accepted answer and is a bit old but since the DataTable module seems being pretty popular among jQuery community still, I believe this approach may be useful for someone else. I've just wrote a simple parsing function based on regular expression and dpath module, though it appears not to be quite reliable module. The snippet may be not very straightforward due to an exception-relied fragment, but it was only one way to prevent dpath from trying to resolve strings as integer indices I found.
import re, dpath.util
rxsKey = r'(?P<key>[^\W\[\]]+)'
rxsEntry = r'(?P<primaryKey>[^\W]+)(?P<secondaryKeys>(\[' \
+ rxsKey \
+ r'\])*)\W*'
rxKey = re.compile(rxsKey)
rxEntry = re.compile(rxsEntry)
def form2dict( frmDct ):
res = {}
for k, v in frmDct.iteritems():
m = rxEntry.match( k )
if not m: continue
mdct = m.groupdict()
if not 'secondaryKeys' in mdct.keys():
res[mdct['primaryKey']] = v
else:
fullPath = [mdct['primaryKey']]
for sk in re.finditer( rxKey, mdct['secondaryKeys'] ):
k = sk.groupdict()['key']
try:
dpath.util.get(res, fullPath)
except KeyError:
dpath.util.new(res, fullPath, [] if k.isdigit() else {})
fullPath.append(int(k) if k.isdigit() else k)
dpath.util.new(res, fullPath, v)
return res
The practical usage is based on native flask request.form.to_dict() method:
# ... somewhere in a view code
pars = form2dict(request.form.to_dict())
The output structure includes both, dictionary and lists, as one could expect. E.g.:
# A little test:
rs = jQDT_form2dict( {
'columns[2][search][regex]' : False,
'columns[2][search][value]' : None,
'columns[2][search][regex]' : False,
} )
generates:
{
"columns": [
null,
null,
{
"search": {
"regex": false,
"value": null
}
}
]
}
Update: to handle lists as dictionaries (in more efficient way) one may simplify this snippet with following block at else part of if clause:
# ...
else:
fullPathStr = mdct['primaryKey']
for sk in re.finditer( rxKey, mdct['secondaryKeys'] ):
fullPathStr += '/' + sk.groupdict()['key']
dpath.util.new(res, fullPathStr, v)

I decided on a way that is more secure than using exec:
from collections import defaultdict
def get_request_data(form):
'''
return dict list with data from request.form
:param form: MultiDict from `request.form`
:rtype: {id1: {field1:val1, ...}, ...} [fieldn and valn are strings]
'''
# request.form comes in multidict [('data[id][field]',value), ...]
# fill in id field automatically
data = defaultdict(lambda: {})
# fill in data[id][field] = value
for formkey in form.keys():
if formkey == 'action': continue
datapart,idpart,fieldpart = formkey.split('[')
if datapart != 'data': raise ParameterError, "invalid input in request: {}".format(formkey)
idvalue = int(idpart[0:-1])
fieldname = fieldpart[0:-1]
data[idvalue][fieldname] = form[formkey]
# return decoded result
return data

using ConfigParser and dictionary in Python

I am trying some basic python scripts using ConfigParser and converting to a dictionary. I am reading a file named "file.cfg" which contains three sections - root, first, second. Currently the code reads the file and converts everything within the file to a dictionary.
My requirement is to convert only sections named "first" and "second" and so on, its key value pair to a dictionary. What would be best way of excluding the section "root" and its key value pair?
import urllib
import urllib2
import base64
import json
import sys
from ConfigParser import SafeConfigParser
parser = SafeConfigParser()
parser.read('file.cfg')
print parser.get('root', 'auth')
config_dict = {}
for sect in parser.sections():
config_dict[sect] = {}
for name, value in parser.items(sect):
config_dict[sect][name] = value
print config_dict
Contents of file.cfg -
~]# cat file.cfg
[root]
username = admin
password = admin
auth = http://192.168.1.1/login
[first]
username = pete
password = sEcReT
url = http://192.168.1.1/list
[second]
username = ron
password = SeCrET
url = http://192.168.1.1/status
Output of the script -
~]# python test4.py
http://192.168.1.1/login
{'second': {'username': 'ron', 'url': 'http://192.168.1.1/status', 'password': 'SeCrEt'}, 'root': {'username': 'admin', 'password': 'admin', 'auth': 'http://192.168.1.1/login'}, 'first': {'username': 'pete', 'url': 'http://192.168.1.1/list', 'password': 'sEcReT'}}

You can remove root section from parser.sections() as follows:
parser.remove_section('root')
Also you don't have to iterate over each pair in each section. You can just convert them to dict:
config_dict = {}
for sect in parser.sections():
config_dict[sect] = dict(parser.items(sect))
Here is one liner:
config_dict = {sect: dict(parser.items(sect)) for sect in parser.sections()}

Bypass the root section by comparison.
for sect in parser.sections():
if sect == 'root':
continue
config_dict[sect] = {}
for name, value in parser.items(sect):
config_dict[sect][name] = value
Edit after acceptance:
ozgur's one liner is a much more concise solution. Upvote from me. If you don't feel like removing sections from the parser directly, the entry can be deleted afterwards.
config_dict = {sect: dict(parser.items(sect)) for sect in parser.sections()} # ozgur's one-liner
del config_dict['root']

Maybe a bit off topic, but ConfigParser is a real pain when in comes to store int, floats and booleans. I prefer using dicts which I dump into configparser.
I also use funtcions to convert between ConfigParser objects and dicts, but those deal with variable type changing, so ConfigParser is happy since it requests strings, and my program is happy since 'False' is not False.
def configparser_to_dict(config: configparser.ConfigParser) -> dict:
config_dict = {}
for section in config.sections():
config_dict[section] = {}
for key, value in config.items(section):
# Now try to convert back to original types if possible
for boolean in ['True', 'False', 'None']:
if value == boolean:
value = bool(boolean)
# Try to convert to float or int
try:
if isinstance(value, str):
if '.' in value:
value = float(value)
else:
value = int(value)
except ValueError:
pass
config_dict[section][key] = value
# Now drop root section if present
config_dict.pop('root', None)
return config_dict
def dict_to_configparser(config_dict: dict) -> configparser.ConfigParser:
config = configparser.ConfigParser()
for section in config_dict.keys():
config.add_section(section)
# Now let's convert all objects to strings so configparser is happy
for key, value in config_dict[section].items():
config[section][key] = str(value)
return config