We Keep Coding

Trying to compare 3 wtforms fields with custom functions and geting recursion problem

I have 3 fields that I want to compare salary "from" field and "to" field and also there is fixed salary field. I have no idea how to do it, since there is no documentation how to do it, so i created custom function that look to each other and trying to se if they have a value.
def validate_salarylow(self, salarylow):
if self.validate_salary_fixed(self.salary_fixed) != "":
salarylow.data = int(0)
else:
try:
salarylow.data = int(salarylow.data)
except:
raise ValidationError("value is not a number")
return salarylow.data
def validate_salary_high(self, salary_high):
if self.validate_salary_fixed(self.salary_fixed) != "":
salary_high.data = int(0)
else:
try:
salary_high.data = int(salary_high.data)
except:
raise ValidationError("value is not a number")
return salary_high.data
def validate_salary_fixed(self, salary_fixed):
if self.validate_salary_high(self.salary_high) != "":
salary_fixed.data = int(0)
try:
salary_fixed.data = int(salary_fixed.data)
except:
raise ValidationError("value is not a number")
return salary_fixed.data
if I don't set if self.validate_salary_high(self.salary_high) != "": everything works fine. but when i set it I'm getting "RecursionError: maximum recursion depth exceeded" error.validate_salary_fixed function looks to validate_salary_high function and vice versa. I'm new in Python and flask and I'm sure there is easy solution, but I cant find it so I would appreciate if anyone could help.

My suggestion is to suppress the error message of the integer field by overwriting it. Thus, the types of the inputs do not have to be converted.
For validation I use two custom validators, one of which checks whether a range or a fixed value has been entered and the second checks the range for its limits. In addition, pre-built validators are used to prohibit negative values.
I'm not sure if you really need the field for the fixed salary, because it is possible to define a fixed value by narrowing the range.
from flask_wtf import FlaskForm
from wtforms import IntegerField
from wtforms.validators import (
NumberRange,
Optional,
StopValidation,
ValidationError
)
class OptionalIntegerField(IntegerField):
def process_data(self, value):
try:
super().process_data(value)
except ValueError:
pass
def process_formdata(self, valuelist):
try:
super().process_formdata(valuelist)
except ValueError:
pass
def validate_salary(form, field):
range_fields = [form.salary_low, form.salary_high]
if all(f.data is None for f in [form.salary_low, form.salary_high, form.salary_fixed]) or \
(form.salary_fixed.data is not None and any(f.data is not None for f in range_fields)) or \
(form.salary_fixed.data is None and any(f.data is None for f in range_fields)):
raise StopValidation('Either state a range from low to high or a fixed salary.')
def validate_salary_range(form, field):
if form.salary_low.data and form.salary_high.data and \
form.salary_low.data > form.salary_high.data:
raise ValidationError('The lower value should be less than or equal to the higher one.')
class SalaryForm(FlaskForm):
salary_low = OptionalIntegerField(
validators=[
validate_salary,
validate_salary_range,
Optional(),
NumberRange(min=0)
]
)
salary_high = OptionalIntegerField(
validators=[
validate_salary,
validate_salary_range,
Optional(),
NumberRange(min=0)
]
)
salary_fixed = OptionalIntegerField(
validators=[
validate_salary,
Optional(),
NumberRange(min=0)
]
)
app = Flask(__name__)
app.secret_key = 'your secret here'
#app.route('/', methods=['GET', 'POST'])
def index():
form = SalaryForm(request.form)
if form.validate_on_submit():
print(form.salary_low.data, ' - ', form.salary_high.data, '||', form.salary_fixed.data)
return render_template('index.html', **locals())

Let's take a look at your code:
Your function validate_salary_high calls validate_salary_fixed.
But when you go to your function validate_salary_fixed it calls validate_salary_high.
So you go back to your function validate_salary_high which calls validate_salary_fixed.
Now in your function validate_salary_fixed, you call validate_salary_high.
Your functions repeatedly call each other over and over again, forever, until your computer eventually throws an error - and this is exactly what is happening to you.
The way to get around this is to remove one of your recursive calls. More specifically you should either
remove your call to validate_salary_fixed in the function validate_salary_high
or remove your call to validate_salary_high in the function validate_salary_fixed
You should chose which function call to remove depending on the goal of your code (which I don't fully understand.) Good luck!

Pydantic get a fields type hint

I want to store metadata for my ML models in pydantic. Is there a proper way to access a fields type? I know you can do BaseModel.__fields__['my_field'].type_ but I assume there's a better way.
I want to make it so that if a BaseModel fails to instantiate it is very clear what data is required to create this missing fields and which methods to use. Something like this :
from pydantic import BaseModel
import pandas as pd
# basic model
class Metadata(BaseModel):
peaks_per_day: float
class PeaksPerDayType(float):
data_required = pd.Timedelta("180D")
data_type = "foo"
#classmethod
def determine(cls, data):
return cls(data)
# use our custom float
class Metadata(BaseModel):
peaks_per_day: PeaksPerDayType
def get_data(data_type, required_data):
# get enough of the appropriate data type
return [1]
# Initial data we have
metadata_json = {}
try:
metadata = Metadata(**metadata_json)
# peaks per day is missing
except Exception as e:
error_msg = e
missing_fields = error_msg.errors()
missing_fields = [missing_field['loc'][0] for missing_field in missing_fields]
# For each missing field use its type hint to find what data is required to
# determine it and access the method to determine the value
new_data = {}
for missing_field in missing_fields:
req_data = Metadata[missing_field].data_required
data_type = Metadata[missing_field].data_type
data = get_data(data_type=data_type, required_data=req_data)
new_data[missing_field] = Metadata[missing_field].determine(data)
metadata = Metadata(**metadata_json, **new_data)

In the case you dont need to handle nested classes, this should work
from pydantic import BaseModel, ValidationError
import typing
class PeaksPerDayType(float):
data_required = 123.22
data_type = "foo"
#classmethod
def determine(cls, data):
return cls(data)
# use our custom float
class Metadata(BaseModel):
peaks_per_day: PeaksPerDayType
def get_data(data_type, required_data):
# get enough of the appropriate data type
return required_data
metadata_json = {}
try:
Metadata(**metadata_json)
except ValidationError as e:
field_to_type = typing.get_type_hints(Metadata)
missing_fields = []
for error in e.errors():
if error['type']=='value_error.missing':
missing_fields.append(error['loc'][0])
else:
raise
new_data = {}
for field in missing_fields:
type_ = field_to_type[field]
new_data[field] = get_data(type_.data_type, type_.data_required)
print(Metadata(**metadata_json, **new_data))
peaks_per_day=123.22
Im not really sure whats the point of data_type or get_data, but I assume its some internal logic that you want to add

functools stops working when switch to Python 3

We have migrated our Django project code base from Python 2.7 to 3.6 and suddenly what used to work stopped. Specifically, this:
map(functools.partial(self._assocUser, user=user), persistedGroupIds)
needed to be replaced with:
for group_id in persistedGroupIds:
self._assocUser(group_id, user)
and this:
persistedGroupIds = map(functools.partial(self._persistGroup, grp_mappings=attrAll.entitlements), saml_authorization_attributes)
needed to go to:
persistedGroupIds = []
for idp_group_name in saml_authorization_attributes:
persistedGroupIds.append(self._persistGroup(idp_group_name, attrAll.entitlements))
before the old functionality reappeared. Python 3's functools don't seem to work.
Here's the full listing of the code that works fine under Python 2:
from django.contrib.auth.models import User
from django.contrib.auth.models import Group
import functools
from mappings import SAMLAttributesConfig
from django.conf import settings
import logging
log = logging.getLogger(__name__)
class SAMLServiceProviderBackend(object):
empty_entitlements_message="IdP supplied incorrect authorization entitlements. Please contact their support."
def _assocUser(self, group_id, user):
group = Group.objects.get(id=group_id)
group.user_set.add(user)
return None
def _persistGroup(self,idp_group_name, grp_mappings):
group_name = grp_mappings[idp_group_name]
try:
group = Group.objects.get(name=group_name)
except Group.DoesNotExist:
group = Group(name=group_name)
group.save()
return group.id
def _extract_grp_entitlements(self,saml_authentication_attributes,groups):
result = []
input_length = len(saml_authentication_attributes[groups])
if input_length == 0:
log.error(self.empty_entitlements_message)
raise RuntimeError(self.empty_entitlements_message)
if input_length == 1:
result = [t.strip() for t in saml_authentication_attributes[groups][0].split(',')]
elif input_length:
result = saml_authentication_attributes[groups]
return result
# return [t.strip() for t in saml_authentication_attributes[groups][0].split(',')] \
# if len(saml_authentication_attributes[groups]) == 1\
# else saml_authentication_attributes[groups]
def authenticate(self, saml_authentication=None):
if not saml_authentication: # Using another authentication method
return None
attrAll = SAMLAttributesConfig(mappings_file_name=settings.AUTH_MAPPINGS_FILE).get_config()
groups = attrAll.entitlements.containerName
if saml_authentication.is_authenticated():
saml_authentication_attributes = saml_authentication.get_attributes()
saml_authorization_attributes = self._extract_grp_entitlements(saml_authentication_attributes,groups)
persistedGroupIds = map(functools.partial(self._persistGroup, grp_mappings=attrAll.entitlements), saml_authorization_attributes)
try:
user = User.objects.get(username=saml_authentication.get_nameid())
except User.DoesNotExist:
user = User(username=saml_authentication.get_nameid())
user.set_unusable_password()
try:
user.first_name = saml_authentication_attributes['samlNameId'][0]
except KeyError:
pass
try:
setattr(user, "first_name", saml_authentication_attributes[attrAll.subject.first_name][0])
except KeyError:
pass
#user.last_name = attributes['Last name'][0]
user.save()
map(functools.partial(self._assocUser, user=user), persistedGroupIds)
user.save()
return user
return None
def get_user(self, user_id):
try:
return User.objects.get(pk=user_id)
except User.DoesNotExist:
return None
The above code no longer works under Python 3 environment and only starts working to something like this, with a functools.partial() calls spelled out in a for loop:
from django.contrib.auth.models import User
from django.contrib.auth.models import Group
import functools
from .mappings import SAMLAttributesConfig
from django.conf import settings
import logging
log = logging.getLogger(__name__)
class SAMLServiceProviderBackend(object):
empty_entitlements_message="IdP supplied incorrect authorization entitlements. Please contact their support."
def _assocUser(self, group_id, user):
group = Group.objects.get(id=group_id)
group.user_set.add(user)
return None
def _persistGroup(self,idp_group_name, grp_mappings):
group_name = grp_mappings[idp_group_name]
try:
group = Group.objects.get(name=group_name)
except Group.DoesNotExist:
group = Group(name=group_name)
group.save()
return group.id
def _extract_grp_entitlements(self,saml_authentication_attributes,groups):
result = []
input_length = len(saml_authentication_attributes[groups])
if input_length == 0:
log.error(self.empty_entitlements_message)
raise RuntimeError(self.empty_entitlements_message)
if input_length == 1:
result = [t.strip() for t in saml_authentication_attributes[groups][0].split(',')]
elif input_length:
result = saml_authentication_attributes[groups]
return result
# return [t.strip() for t in saml_authentication_attributes[groups][0].split(',')] \
# if len(saml_authentication_attributes[groups]) == 1\
# else saml_authentication_attributes[groups]
def authenticate(self, saml_authentication=None):
if not saml_authentication: # Using another authentication method
return None
attrAll = SAMLAttributesConfig(mappings_file_name=settings.AUTH_MAPPINGS_FILE).get_config()
groups = attrAll.entitlements.containerName
if saml_authentication.is_authenticated():
saml_authentication_attributes = saml_authentication.get_attributes()
saml_authorization_attributes = self._extract_grp_entitlements(saml_authentication_attributes,groups)
persistedGroupIds = map(functools.partial(self._persistGroup, grp_mappings=attrAll.entitlements), saml_authorization_attributes)
try:
user = User.objects.get(username=saml_authentication.get_nameid())
except User.DoesNotExist:
user = User(username=saml_authentication.get_nameid())
user.set_unusable_password()
try:
user.first_name = saml_authentication_attributes['samlNameId'][0]
except KeyError:
pass
try:
setattr(user, "first_name", saml_authentication_attributes[attrAll.subject.first_name][0])
except KeyError:
pass
#user.last_name = attributes['Last name'][0]
user.save()
for group_id in persistedGroupIds:
self._assocUser(user = user, group_id = group_id)
# map(functools.partial(self._assocUser, user=user), persistedGroupIds)
user.save()
return user
return None
def get_user(self, user_id):
try:
return User.objects.get(pk=user_id)
except User.DoesNotExist:
return None
What can be wrong?
I'm using PyDev plugin in Eclipse. Here's how my Python interpreter is configured there:
Here's Eclipse's .pydevproject file:
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<?eclipse-pydev version="1.0"?><pydev_project>
<pydev_property name="org.python.pydev.PYTHON_PROJECT_INTERPRETER">venv3.6</pydev_property>
<pydev_property name="org.python.pydev.PYTHON_PROJECT_VERSION">python interpreter</pydev_property>
<pydev_variables_property name="org.python.pydev.PROJECT_VARIABLE_SUBSTITUTION">
<key>DJANGO_SETTINGS_MODULE</key>
<value>reporting.settings</value>
<key>DJANGO_MANAGE_LOCATION</key>
<value>./manage.py</value>
<key>SAML_PLUGIN</key>
<value>/Users/sl/abc/venv3.6/lib/python3.6/site-packages/onelogin/saml2</value>
<key>PY</key>
<value>36</value>
</pydev_variables_property>
<pydev_pathproperty name="org.python.pydev.PROJECT_SOURCE_PATH">
<path>/${PROJECT_DIR_NAME}</path>
</pydev_pathproperty>
<pydev_pathproperty name="org.python.pydev.PROJECT_EXTERNAL_SOURCE_PATH">
<path>${SAML_PLUGIN}</path>
</pydev_pathproperty>
</pydev_project>

In Python 3, the map function returns an iterator instead of a list.
This means that if you call map on a collection, the effects of the call are not materialised until you iterate over the resulting iterator.
Consider this class:
>>> class C:
... def __init__(self, x):
... self.x = x
... def double(self):
... self.x *= 2
... def __repr__(self):
... return '<C:{}>'.format(self.x)
...
Let's make a list of instances:
>>> cs = [C(x) for x in range(1, 4)]
>>> cs
[<C:1>, <C:2>, <C:3>]
Now use map to call each instance's double method:
>>> res = map(C.double, cs)
Note the result is not a list:
>>> res
<map object at 0x7ff276350470>
And the instances have not changed:
>>> cs
[<C:1>, <C:2>, <C:3>]
if we call next on the iterator, the instances are updated in turn.
>>> next(res)
>>> cs
[<C:2>, <C:2>, <C:3>]
>>> next(res)
>>> cs
[<C:2>, <C:4>, <C:3>]
>>> next(res)
>>> cs
[<C:2>, <C:4>, <C:6>]
In the code samples that you have provided, the result of calling map is not assigned to a variable, so map is being used for its side effects rather than its output. In Python 3 the correct way to do this is to loop over the iterable and call the function on each element:
>>> for c in cs:
c.double()
As the linked doc puts it:
Particularly tricky is map() invoked for the side effects of the function; the correct transformation is to use a regular for loop (since creating a list would just be wasteful).

How to validate fields and get validated output

I'm trying to validate the values in the model and get as output the validated values. The only example in the documentation is the following code, which doesn't show how exactly this is done so I can't extend on it.
>>> from schematics.models import Model
>>> from schematics.types import StringType, BooleanType
>>> from schematics.exceptions import ValidationError
>>>
>>> class Signup(Model):
... name = StringType()
... call_me = BooleanType(default=False)
... def validate_call_me(self, data, value):
... if data['name'] == u'Brad' and data['call_me'] is True:
... raise ValidationError(u'He prefers email.')
... return value
...
>>> Signup({'name': u'Brad'}).validate()
>>> Signup({'name': u'Brad', 'call_me': True}).validate()
Traceback (most recent call last):
...
ModelValidationError: {'call_me': [u'He prefers email.']}
I made a version of it but removed data and value from arguments. Client is the name of my model. So when I do the following I get the desired result as output:
client.validate_client(client.to_native())
However,
first of all, this doesn't seem a clean way. client already has all the values and so I wouldn't need to do this.
Also, I like to get this to update the values of client as an outcome of validation.
For the first part, I did something line this:
def validate_client(self):
data = self.to_native()
...
return data
But I don't think this is the best way to do this and I am not sure about the second issue of updating the values. Is there a way to do this?
EDIT:
This is the code I have and I want the client value for employer to be set to 'unspecified' and client full_name to be set as specified in the function.
class LowerCaseEmailType(EmailType):
def convert(self, value, context=None):
value = super(LowerCaseEmailType, self).convert(value, context)
return value.lower()
class CleanedStringType(StringType):
converters = []
def __init__(self, **kwargs):
"""
This takes in all the inputs as String Type, but takes in an extra
input called converters.
Converters must be a list of functions, and each of those functions
must take in exactly 1 value , and return the transformed input
"""
if 'converters' in kwargs:
self.converters = kwargs['converters']
del kwargs['converters']
super(CleanedStringType, self).__init__(**kwargs)
def convert(self, value, context=None):
value = super(CleanedStringType, self).convert(value, context)
for func in self.converters:
value = func(value)
return value # will have a value after going through all the conversions in order
class Client(Model):
"""
Client Model
"""
first_name = CleanedStringType(required=False,
converters=[lambda x: re.sub(r"[!##$%&\(\)\^]+", ' ', x),
lambda x: x.strip()])
last_name = CleanedStringType(required=False,
converters=[lambda x: re.sub(r"[!##$%&\(\)\^]+", ' ', x),
lambda x: x.strip()])
full_name = CleanedStringType(required=False,
converters=[lambda x: re.sub(r"[!##$%&\(\)\^]+", ' ', x),
lambda x: x.strip()])
employer = StringType(required=False)
created = StringType(default=" ")
updated = StringType(default=" ")
email = LowerCaseEmailType(required=False)
def validate_client(self):
data = self.to_native()
if data.get('first_name') and data.get('last_name'):
data['full_name'] = ' '.join([data['first_name'], data['last_name']])
if data.get('full_name') is None:
if data.get('first_name') is None:
error_message = 'first name missing'
else:
error_message = 'last name missing'
logger.error('info: {} for {}'.format(error_message, data))
raise ValidationError(error_message)
if data.get('employer') is None:
logger.warning('info: employer missing for {}'.format(data))
data['employer'] = 'unspecified'
return data

I think you want pre_setattr. It is called with the value when you set a value of a model (code in question).
For the eymployer I think you want to set a default. To me this seems to behave exactly as you want.
I think fullname should not be on the model, because you can have inconsistent data where firstname + lastname != fullname. If you need it later you could implement that via #serialized.

method reverse django error

I'm calling a method using reverse but I have problem argument that I am not able to understand.
My error :
Reverse for 'shopping.views.payment_confirmation' with arguments '(35,)' and keyword arguments '{}' not found.
My url:
url(r'^payment_confirmation/(?P<id>\d+\d{2\})/$', 'payment_confirmation', name='payment_confirmation'),
My view:
def show(request):
...
...
payment.submit(settings.SITE_URL + reverse("shopping.views.payment_confirmation", args=[payment.id]))
My model Payment:
class Payment(models.Model):
...
...
def submit(self, redirect_url):
'''
Sends self as a Payment through PagSeguro API.
Payment instance MUST BE SAVED before calling this method.
'''
if not self.id:
#Temporary to identify which problem caused the crash.
raise ValidationError
#creating a reference if its None
if self.reference is None:
self.reference = configs.PAGSEGURO_REFERENCE_PREFIX + str(self.id)
document = Document()
document.appendChild(self.to_element(document, redirect_url))
response = document2dict(api.submit_payment(document))
try:
self.code = response['checkout']['code']
self.answered_on = datetime.datetime.now()
self.save()
except:
error_str = ""
if type(response["errors"]["error"]) != list:
response["errors"]["error"] = [response["errors"]["error"]]
for error in response["errors"]["error"]:
error_payment = ErrorPayment()
error_payment.code = int(error['code'])
error_payment.payment = self
error_payment.save()
error_str += "[%s: %s]" % (error_payment.code,
error_payment.get_code_display())
raise Exception(error_str)
the error is here payment.submit (settings.SITE_URL + reverse ("shopping.views.payment_confirmation", args = [payment.id]))
I', using this api https://bitbucket.org/leonardosantos/django-pagseguro2/

This line: reverse("shopping.views.payment_confirmation", args=[payment.id]) tells Django to find a url that matches a method called payment_confirmation in the views.py file in the shopping app that will accept a payment ID parameter.
In the error that you shared, the payment.id was 35. The error is saying that either there is no method called payment_confirmation in your shopping.views or the method does not accept a single int as a parameter.
You didn't share a payment_confirmation method in your views file so it seems that that is the issue. You need to add:
def payment_confirmation(payment_id):
#do stuff
to your views.