I am trying to create a new dictionary out of html form data that was submitted by the user. I end up writing repetitive if statements, checking if xyz key is in the dictionary in the form data. I know this is a quite suboptimal approach though I am not quite sure how to implement this using python.
This is the form data dictionary:
form_data = {
'urls': ['www.google.com', 'www.bing.com'],
'useremail': ['my#email.com'],
'emailfield': ['1'],
'addressfield': ['1'],
'addressfield_info':['Company'],
'addressfield_instruction': ['Please only if the company is a LLC'],
'phonefield': ['1'],
'phonefield_instruction': ['please include area code']
}
and I want to create a dictionary that looks like this:
new_dic = {
'urls': ['www.google.com', 'www.bing.com'],
'useremail': ['my#email.com'],
'infofield': [
{'field': 'email'},
{'field': 'address', 'info':'Company', 'instruction': 'Please only if the company is a LLC'},
{'field':'phone', 'instruction': 'please include area code'}
]
}
Important note: The 'xyzfield' is mandatory and the 'xyzfield_info' and 'xyzfield_instruction' are both optional. Also: the user can add more fields and create for instance an 'agefield', 'agefield_info' and 'agefield_instruction'.
The problem I have is about how to efficiently check if xyzfield (email, phone, etc) is in the dictionary. If it is in there, check also if any of the optional fields are in there as well. This looks currently something like this:
if 'emailfield' in form_data:
infofield = {'field': 'email'}
if 'emailfield_info' in form_data:
infofield['info'] = form_data['emailfield_info']
if 'emailfield_instruction' in form_data:
infofield['instruction'] = form_data['emailfield_instruction']
cleaned_data['infofields'].append(infofield)
...
and I do this for every field, hence I have 4-5 of this. Additional, I will not be able to process any of the fields that the user has created himself since I don't know the name upfront.
Long story short: How can I make this more efficient and dynamic?
The standard answer to how to avoid repeated code applies here --- extract the repeated code to a function:
def extract_field(form_data, clean, fieldname, optional=('info', 'instruction')):
if fieldname+'field' in form_data:
infofield = { 'field': fieldname }
for opt in optional:
optname = '{}field_{}'.format(fieldname, opt)
if optname in form_data:
infofield[opt] = form_data[optname]
clean.append(infofield)
extract_field(form_data, cleaned_data['infofields'], 'email')
extract_field(form_data, cleaned_data['infofields'], 'address')
extract_field(form_data, cleaned_data['infofields'], 'phone')
This assumes you just want to clean whatever is actually submitted. If you are looking for specific things to be there, I suggest making a list of things to look for, and iterating over the list and checking to see if the things are there.
form_data = {
'urls': ['www.google.com', 'www.bing.com'],
'useremail': ['my#email.com'],
'emailfield': ['1'],
'addressfield': ['1'],
'addressfield_info':['Company'],
'addressfield_instruction': ['Please only if the company is a LLC'],
'phonefield': ['1'],
'phonefield_instruction': ['please include area code']
}
def make_field_dict(form_data, base):
field_dict = {}
name_field = base + "field"
name_info = base + "field_info"
name_inst = base + "field_instruction"
if name_field not in form_data:
raise KeyError, "%s not found in form_data" % name_field
if form_data[name_field] != ['1']:
raise ValueError, "%s not valid in form_data" % name_field
field_dict["field"] = base
if name_info in form_data:
lst = form_data[name_info]
if len(lst) != 1:
raise ValueError, "%s not valid in form_data" % name_info
field_dict["info"] = lst[0]
if name_inst in form_data:
lst = form_data[name_inst]
if len(lst) != 1:
raise ValueError, "%s not valid in form_data" % name_inst
field_dict["instruction"] = lst[0]
return field_dict
def parse_form_data(form_data):
cleaned_data = {}
cleaned_data["infofield"] = []
seen = set()
for key, value in form_data.items():
if "field" not in key:
cleaned_data[key] = value
else:
base, _, tail = key.partition("field")
if base in seen:
continue
cleaned_data["infofield"].append(make_field_dict(form_data, base))
seen.add(base)
return cleaned_data
new_dic = {
'urls': ['www.google.com', 'www.bing.com'],
'useremail': ['my#email.com'],
'infofield': [
{'field': 'email'},
{'field': 'address', 'info':'Company', 'instruction': 'Please only if the company is a LLC'},
{'field':'phone', 'instruction': 'please include area code'}
]
}
clean_data = parse_form_data(form_data)
new_dic['infofield'].sort()
clean_data['infofield'].sort()
assert(new_dic == clean_data)
Related
Summary problem: Building an API endpoint and trying to push new key:values to the existing API. Not knowing if I am correctly adding key value pairs or not. Familiar with Ruby but first time Python user!
Context:
I currently have a method that will format given information into a Python Dictionary to be used as a JSON for my API. I have one method that pushes information to this method but another that is not functioning. Can anybody spot why?
Things I've tried:
Feature test using command line environment
Getting visibility - Printing
Environment:
MacOS 11.1, python 3.9.1, VSCode
Code:
METHOD THAT IS FORMATTING
class Database(object):
def __init__(self):
self.data = {}
def insert_entity(self, kind, entity):
kind_dict = self.data.get(kind, {})
entity_id = entity.get('id', str(uuid4()))
if not isinstance(entity_id, str):
raise Exception('Entity `id` must be a string')
entity['id'] = entity_id
kind_dict[entity_id] = entity
self.data[kind] = kind_dict
return entity
def get_entity(self, kind, entity_id):
entities = self.data.get(kind, {})
return entities.get(entity_id, None)
def get_all_entities(self, kind):
return list(self.data.get(kind, {}).values())
METHOD THAT IS WORKING:
def initialise_user_data():
first_names = ['Ron', 'Paul', 'Simon', 'David', 'Phil', 'Ada', 'Julia']
last_names = [
'Legend', 'Mac', 'Stuartson', 'Sili', 'Word', 'Nine',
'Smith'
]
for index in range(len(first_names)):
first_name = first_names[index]
last_name = last_names[index]
email = str(random.randint(0, 9999)) + "#email.com"
user_data = {
'firstName': first_name,
'lastName': last_name,
'email': email
}
database.insert_entity('User', user_data)
METHOD THAT IS NOT WORKING:
def initialise_event_data():
users = database.get_all_entities('User')
for user in users:
for _ in range(random.randint(0, 10)):
database.insert_entity(
'Event', {
'userId': user['id'],
'points': 100,
'eventName': 'levels_completed'
})
ALL METHODS ARE INVOKED AS SUCH:
database = InMemoryDatabase()
initialise_data()
def initialise_data():
initialise_user_data()
initialise_event_data()
initialise_follow_data()
I'm trying to import 5000+ rows in Odoo 12 it's basically a mapping from a CSV developed in a custom method in a module, the problem I'm getting timeout in the request, that's happening when writing to the database, I'm using the standard ERP methods create and write.
How can I work around a solution to this? I know bulk insert is not possible to this, any other solution to this?
is a SQL command for insertion OK to use?
class file_reader(models.TransientModel):
_name = "rw.file.reader"
csv_file = fields.Binary(string='CSV File', required=True)
#api.multi
def import_csv(self):
# csv importer handler
file = base64.b64decode(self.csv_file).decode().split('\n')
reader = csv.DictReader(file)
# account.analytic.line
ignored = []
time1 = datetime.now()
self._cr.execute('select id, name from project_project where active = true')
projects = self._cr.fetchall()
self._cr.execute('select id, login from res_users')
users = self._cr.fetchall()
self._cr.execute('select id, work_email from hr_employee')
employees = self._cr.fetchall()
LOG_EVERY_N = 100
for row in reader:
project_name = row['Project - Name']
email = row['User - Email Address']
project = [item for item in projects if item[1] == project_name]
if len(project) >0:
user = [item for item in users if item[1] == email]
employee = [item for item in employees if item[1] == email]
if len(user)>0 and len(employee)>0:
task = self.env['project.task'].search([['user_id','=',user[0][0]],
['project_id','=',project[0][0] ]],limit=1)
if task:
y = row['Duration'].split(':')
i, j = y[0], y[1]
model = {
'project_id': project[0][0],
'task_id': task['id'],
'employee_id':employee[0][0],
'user_id': user[0][0],
'date': row['Date'],
'unit_amount': int(i) + (float(j) / 60), # Time Spent convertion to float
'is_timesheet': True,
'billable': True if row['Billable'] == 'Yes' else False,
'nexonia_id':row['ID']
}
time_sheet = self.env['account.analytic.line'].search([['nexonia_id','=', row['ID']]],limit=1)
if time_sheet:
model.update({'id':time_sheet.id})
self.env['account.analytic.line'].sudo().write(model)
else:
self.env['account.analytic.line'].sudo().create(model)
else:
if email not in ignored:
ignored.append(email)
else:
if project_name not in ignored:
ignored.append(project_name)
all_text = 'Nothing ignored'
if ignored is not None:
all_text = "\n".join(filter(None, ignored))
message_id = self.env['message.wizard'].create({
'message': "Import data completed",
'ignored': all_text
})
time2 = datetime.now()
logging.info('total time ------------------------------------------ %s',time2-time1)
return {
'name': 'Successfull',
'type': 'ir.actions.act_window',
'view_mode': 'form',
'res_model': 'message.wizard',
# pass the id
'res_id': message_id.id,
'target': 'new'
}
I Enhanced your code a litle bit because you are searching for each project, user and employee using loop
for each row and for 5000+ row.
Using ORM method is always good because, they handle the stored compute fields and python constrains, but this will take time too
if you don't have any complex compute you can use INSERT or UPDATE query this will speed up the importion 100 times.
#api.multi
def import_csv(self):
# when you use env[model] for more than ones extract it to variable its better
# notice how I added sudo to the name of variable
AccountAnalyticLine_sudo =self.env['account.analytic.line'].sudo()
# csv importer handler
file = base64.b64decode(self.csv_file).decode().split('\n')
reader = csv.DictReader(file)
# account.analytic.line
ignored = []
time1 = datetime.now()
# convert result to dictionary for easy access later
self._cr.execute('select id, name from project_project where active = true order by name')
projects = {p[1]: p for p in self._cr.fetchall()}
self._cr.execute('select id, login from res_users order by login')
users = {u[1]: u for u in self._cr.fetchall()}
self._cr.execute('select id, work_email from hr_employee order by work_email')
employees = {emp[1]: emp for emp in self._cr.fetchall()}
LOG_EVERY_N = 100
for row in reader:
project_name = row['Project - Name']
email = row['User - Email Address']
# no need for loop and the dicionary loopkup is so fast
project = projects.get(project_name)
if project:
user = user.get(email)
employee = employees.get(email)
if user and employee:
task = self.env['project.task'].search([('user_id','=',user[0]),
('project_id','=',project[0])],
limit=1)
if task:
y = row['Duration'].split(':')
i, j = y[0], y[1]
# by convention dictionary that are passed to create or write should be named vals or values
vals = {
'project_id': project[0],
'task_id': task['id'],
'employee_id':employee[0],
'user_id': user[0],
'date': row['Date'],
'unit_amount': int(i) + (float(j) / 60), # Time Spent convertion to float
'is_timesheet': True,
'billable': True if row['Billable'] == 'Yes' else False,
'nexonia_id':row['ID']
}
time_sheet = AccountAnalyticLine_sudo.search([('nexonia_id','=', row['ID'])],limit=1)
# I think adding logger message here will be or create and update counters to know how much record record were updated or created
if time_sheet:
# I think you want to update the existing time sheet record so do this
# record.write(vals)
time_sheet.write(vals)
# you are updating an empty RecordSet
#self.env['account.analytic.line'].sudo().write(model)
else:
# create new one
AccountAnalyticLine_sudo.create(model)
else:
if email not in ignored:
ignored.append(email)
else:
if project_name not in ignored:
ignored.append(project_name)
all_text = 'Nothing ignored'
# ignored is not None is always True because ignored is a list
if ignored:
all_text = "\n".join(filter(None, ignored))
message_id = self.env['message.wizard'].create({
'message': "Import data completed",
'ignored': all_text
})
time2 = datetime.now()
logging.info('total time ------------------------------------------ %s',time2-time1)
return {
'name': 'Successfull',
'type': 'ir.actions.act_window',
'view_mode': 'form',
'res_model': 'message.wizard',
# pass the id
'res_id': message_id.id,
'target': 'new'
}
I hope this will help you a little bit even that the question is meant for somethinng else but I'm confused Odoo usually allow request to be handled
for 60 minutes.
While you are importing records through script, code optimization is very important.Try to reduce the number of search/read calls by using dictionary to save each result or use the SQL which i don't recommend.
I am pulling from a dictionary in Python. I have it setup like this:
entries = [
{'website': 'yahoo','username': 'jblumberg','password': 'sdkljfhwe'},
{'website': 'google','username': 'jblumberg1','password': 'CoIushujSetu'}
]
I am asking the user to give me a website so I can return the password as so:
def lookup_password(website):
if website in entries.keys():
encrypted_password = entries[website]
return password_encrypt(encrypted_password, -encryption_key)
pass
However that won't give me what I want. How do I set it up to give me the password value for the provided website?
First let's restructure your dict like this so that it's a dict with each key as a website, like it seems you treat it in your code:
entries = {
'yahoo': {'username': 'jblumberg', 'password': 'sdkljfhwe'},
'google': {'username': 'jblumberg1', 'passwprd': 'CoIushujSetu'}
}
Now a couple of changes to your original code gets us to what should be working:
if website in entries:
encrypted_password = entries[website]['password']
I should note that website in entries and website in entries.keys() do the same thing here.
There are multiples small problems in your conception of your data :
First entries is a list. Therefore you won't be able to call ".keys()"
Also when using
encrypted_password = entries[website]
this will store the whole dictionnary. Meaning that you would then be able to access the password via ['password']
encrypted_password = entries[website]['password']
To resume : if you change you data to look like this
entries = {
'yahoo':{
'username': 'jblumberg','password': 'sdkljfhwe'
},
'google':{
'username': 'jblumberg1','password': 'CoIushujSetu'
},
}
def lookup_password(website):
if website in entries:
encrypted_password = entries[website]['password']
return password_encrypt(encrypted_password, -encryption_key)
pass
But if you keep the same data, it will have to look like this:
def lookup_password(website):
for record in entries:
if record['website'] == website:
encrypted_password = record['website']
return password_encrypt(encrypted_password, -encryption_key)
pass
You could use another dictionary:
entries = [
{'yahoo':{'username': 'jblumberg','password': 'sdkljfhwe'},
{'google':{'username': 'jblumberg1','password': 'CoIushujSetu'}
]
Then the following would happen:
>> website = 'yahoo'
>> entries[website]
>> {'username': 'jblumberg','password': 'sdkljfhwe'}
so if you wanted the password:
>> entires[website]['password']
>> 'sdkljfhwe'
Try:
def lookup_password(website):
for entry in entries:
if entry['website'] == website:
return entry['password']
Output:
In[2]: lookup_password('google')
Out[2]: 'CoIushujSetu'
I have a flask application which is receiving a request from dataTables Editor. Upon receipt at the server, request.form looks like (e.g.)
ImmutableMultiDict([('data[59282][gender]', u'M'), ('data[59282][hometown]', u''),
('data[59282][disposition]', u''), ('data[59282][id]', u'59282'),
('data[59282][resultname]', u'Joe Doe'), ('data[59282][confirm]', 'true'),
('data[59282][age]', u'27'), ('data[59282][place]', u'3'), ('action', u'remove'),
('data[59282][runnerid]', u''), ('data[59282][time]', u'29:49'),
('data[59282][club]', u'')])
I am thinking to use something similar to this really ugly code to decode it. Is there a better way?
from collections import defaultdict
# request.form comes in multidict [('data[id][field]',value), ...]
# so we need to exec this string to turn into python data structure
data = defaultdict(lambda: {}) # default is empty dict
# need to define text for each field to be received in data[id][field]
age = 'age'
club = 'club'
confirm = 'confirm'
disposition = 'disposition'
gender = 'gender'
hometown = 'hometown'
id = 'id'
place = 'place'
resultname = 'resultname'
runnerid = 'runnerid'
time = 'time'
# fill in data[id][field] = value
for formkey in request.form.keys():
exec '{} = {}'.format(d,repr(request.form[formkey]))
This question has an accepted answer and is a bit old but since the DataTable module seems being pretty popular among jQuery community still, I believe this approach may be useful for someone else. I've just wrote a simple parsing function based on regular expression and dpath module, though it appears not to be quite reliable module. The snippet may be not very straightforward due to an exception-relied fragment, but it was only one way to prevent dpath from trying to resolve strings as integer indices I found.
import re, dpath.util
rxsKey = r'(?P<key>[^\W\[\]]+)'
rxsEntry = r'(?P<primaryKey>[^\W]+)(?P<secondaryKeys>(\[' \
+ rxsKey \
+ r'\])*)\W*'
rxKey = re.compile(rxsKey)
rxEntry = re.compile(rxsEntry)
def form2dict( frmDct ):
res = {}
for k, v in frmDct.iteritems():
m = rxEntry.match( k )
if not m: continue
mdct = m.groupdict()
if not 'secondaryKeys' in mdct.keys():
res[mdct['primaryKey']] = v
else:
fullPath = [mdct['primaryKey']]
for sk in re.finditer( rxKey, mdct['secondaryKeys'] ):
k = sk.groupdict()['key']
try:
dpath.util.get(res, fullPath)
except KeyError:
dpath.util.new(res, fullPath, [] if k.isdigit() else {})
fullPath.append(int(k) if k.isdigit() else k)
dpath.util.new(res, fullPath, v)
return res
The practical usage is based on native flask request.form.to_dict() method:
# ... somewhere in a view code
pars = form2dict(request.form.to_dict())
The output structure includes both, dictionary and lists, as one could expect. E.g.:
# A little test:
rs = jQDT_form2dict( {
'columns[2][search][regex]' : False,
'columns[2][search][value]' : None,
'columns[2][search][regex]' : False,
} )
generates:
{
"columns": [
null,
null,
{
"search": {
"regex": false,
"value": null
}
}
]
}
Update: to handle lists as dictionaries (in more efficient way) one may simplify this snippet with following block at else part of if clause:
# ...
else:
fullPathStr = mdct['primaryKey']
for sk in re.finditer( rxKey, mdct['secondaryKeys'] ):
fullPathStr += '/' + sk.groupdict()['key']
dpath.util.new(res, fullPathStr, v)
I decided on a way that is more secure than using exec:
from collections import defaultdict
def get_request_data(form):
'''
return dict list with data from request.form
:param form: MultiDict from `request.form`
:rtype: {id1: {field1:val1, ...}, ...} [fieldn and valn are strings]
'''
# request.form comes in multidict [('data[id][field]',value), ...]
# fill in id field automatically
data = defaultdict(lambda: {})
# fill in data[id][field] = value
for formkey in form.keys():
if formkey == 'action': continue
datapart,idpart,fieldpart = formkey.split('[')
if datapart != 'data': raise ParameterError, "invalid input in request: {}".format(formkey)
idvalue = int(idpart[0:-1])
fieldname = fieldpart[0:-1]
data[idvalue][fieldname] = form[formkey]
# return decoded result
return data
I have write the following function to build parameters, i want to know that is there any other way in python to do that so that the code efficient is improved...
build_parameters(users[keys]["email"], Null , xsmtpapi, Message_Subject, Message_Content, Message_Content, 'support#brightspyre.com', 'BrightSpyre', 'support#brightspyre.com', Null, Null, Null, Null)
Here is the function
def build_parameters(to = None, toname = None, x-smtpapi = None, subject = None, text = None, html = None, from = None, cc = None, ccname = None, bcc = None, bccname = None, fromname = None, replyto = None, date = None, files = None, content = None, headers = None):
param = {}
if headers:
param['headers'] = headers
if content:
param['content'] = content
if files:
param['files'] = files
if date:
param['date'] = date
if replyto:
param['replyto'] = replyto
if fromname:
param['fromname'] = fromname
if bccname:
param['bccname'] = bccname
if bcc:
param['bcc'] = bcc
if ccname:
param['ccname'] = ccname
if cc:
param['cc'] = cc
if from:
param['from'] = from
if html:
param['html'] = html
if text:
param['text'] = text
if subject:
param['subject'] = subject
if x-smtpapi:
param['x-smtpapi'] = x-smtpapi
if toname:
param['toname'] = toname
if to:
param['to'] = to
return param
UPDATED
I have updated the code as described by #J0HN
_allowed_keys = {'to', 'toname', 'x-smtpapi', 'subject', 'text', 'html', 'from', 'cc', 'ccname', 'bcc', 'bccname', 'fromname', 'replyto', 'date', 'files', 'content', 'headers'}
def build_parameter(**kwargs):
return {key:value for key, value in kwargs.items() if key in _allowed_keys}
params = build_parameter(to = users[keys]["email"], toname = users[keys]["name"], x-smtpapi = xsmtpapi, subject = Message_Subject,text = Message_Content, html = Message_Content, from = 'support#bs.com', fromname = 'BS', replyto = 'support#bs.com')
error
params = build_parameter(to = users[keys]["email"],toname = users[keys]["name"], x-smtpapi = xsmtpapi, subject = Message_Subject,text = Message_Content, html = Message_Content, from = '
support#bs.com', fromname = 'BSe', replyto = 'support#bs.com')
^
SyntaxError: invalid syntax
_allowed_keys = {'to', 'toname', 'x-smtpapi', ...]
def build_parameters(**kwargs):
return {key:value for key, value in kwargs.items() if key in _allowed_keys}
Step by step:
Define _allowed_keys to contain all keyword arguments' names. Refer to set documentation for details.
Replace parameters with **kwargs. Refer to Understanding kwargs in Python for details on what's that.
Use a dictionary comprehension to build a new dictionary out of kwargs
items are used to iterate over key-value pair of a dictionary
key in _allowed_keys is self-explanatory I believe
As a result, this function receives any number of keyword arguments, but filter out keys not in _allowed_keys.
UPD: ok, from is a keyword and x-smtpapi can't be a keyword argument as it contains -. It's an expected behavior, but the canonical way to workaround it renders the whole method useless.
So you'll need to represent them differently, e.g.:
_transforms = {'x_smtpapi'='x-smtpapi', `_from`='from'}
def build_parameter(**kwargs):
return {_transforms.get(key, key):value for key, value in kwargs.items() if _transforms.get(key, key) in _allowed_keys}
And use like this build_parameter(_from='from value', x_smtpapi: 'x-smtpapi value', ...)
However, I wouldn't recommend doing this is it might be quite confusing. Instead, consider alternative approaches, e.g. build a class to encapsulate creating params dict (and probably using it)
Just build a dictionary.
params = {"to": "to value",
"from": "from value}
There is no need to use a method for this.