We Keep Coding

Python automatically adds singlequotes around float value in List [duplicate]

This question already has answers here:
Possible to enforce type hints?
(2 answers)
Closed 12 months ago.
I struggle with Python(3.9.7) automatically adding singlequotes around a float-value, which leads to a failing assertion.
My goal is to use pytest to assert an expected output of a parsing-function. The parser takes a json-object (shortened code sample below) and returns a list of Signal objects. I copied the print-out from logging.debug(f"{signal_list}") inside the parsing function, and assigned it to expected_result in my test-function:
#Arrange
message = json.dumps({"data":{"name":"battery_volt","alias":"Volt","unit":"volt","value":12.0,"time":1644587969}})
expected_result = [Signal(id=None, name='battery_volt', description=None, value=12.0, type=None, unit='volt', time='2022-02-11T13:59:29')]
print(expected_result)
p = Parser(message)
#Act
result = p.parse_message()
#Assert
assert result == expected_result
Irritatingly, pytest -vv throws an AssertionError:
E Full diff:
E - [Signal(id=None, name='battery_volt', description=None, value='12.0', type=None, unit='volt', time='2022-02-11T13:59:29')]
E ? - -
E + [Signal(id=None, name='battery_volt', description=None, value=12.0, type=None, unit='volt', time='2022-02-11T13:59:29')]
The upper line seems to be the value of expected_result, because print(expected_result)
also adds the singlequotes around the 12.0
I assume the copied output from logging.debug(f"{signal_list}") isn't the same as the real value of result. I tried typecasting expected_result as list, str()-converting both result and expected_result inside the test, but expected_result always has '12.0' and result has 12.0.
I desperatly need a hint how to do this kind of assertion the correct way.
EDIT:
Here is the parsing function:
def parse_message(self):
message = json.loads(self.message)
#logging.debug(f"message is: {message}")
message_data = message.get('data', {})
parsed_data = []
try:
device = message_data.get('device', None)
if device is not None:
vehicle = self.parse_vehicle(device)
parsed_data.append(vehicle)
else:
logging.error("'device' was None!")
except Exception as e:
logging.error(e)
signals = []
try:
data = message_data.get('data', None)
if data is not None:
signals = self.parse_signals(data)
gps = message_data.get('gps', None)
if gps is not None:
gps_signal = self.parse_gps(gps)
signals.append(gps_signal)
parsed_data.append(signals)
except Exception as e:
logging.error(e)
return parsed_data
if __name__ == "__main__":
setup_logging()
message = json.dumps({"consumerid":redacted,"data":{"device":{"time":1644587969,"imei":"redacted","man_id":redacted,"car_id":999,"vin":"redacted"},"data":[{"name":"battery_volt","alias":"Volt","unit":"volt","value":12.0,"time":1644587969}],"gps":{"lat":51.437515,"lon":6.9281199,"dir":252,"alt":88,"sat":19,"time":1644587969}},"event":"redacted","itemid":redacted,"itemtype":1,"senderip":"redacted"})
p = Parser(message)
signal_list = p.parse_message()
logging.debug(f"{signal_list}")
Please note that the passed json-objects are more complex than the code-sample in the original post.
class Signal(BaseModel):
id: int = None
name: str = None
description: str = None
value: str = None
type: str = None
unit: str = None
time: str = None
EDIT2 - Assignment of Signal.value happens here:
def parse_signals(self, data):
signals = []
#logging.debug(f"data is : {data}")
for data_object in data:
signal = Signal()
try:
signal.name = data_object.get('name', None)
#get dtc-count value as seperate return-element, needed by the controller to handle dtc-codes
if signal.name == 'dtc_count':
self.dtc_count == data_object.get('value', None)
signal.description = data_object.get('description', None)
signal.value = data_object.get('value', None)
signal.time = datetime.datetime.utcfromtimestamp(data_object.get('time', None)).isoformat()
signal.unit = data_object.get('unit', None)
if signal.unit == "dtc":
signal.type = "1"
if signal.name is not None:
signals.append(signal)
#logging.debug(signal.__dict__)
except Exception as e:
logging.error(f"While parsing signal {data_object}, the following error occured: {e}")
return signals
When parse_message is called as __name__ == "main":, the testcode beneath outputs value=12.0

Ok, turns out python type hints don't enforce like I expected:
When parse_message is called by way of __name__ == "main":, the testcode beneath outputs value=12.0 Apparently, despite the type-hint :str for Signal.value, 12.0 was assigned as float. When I tried to sys.stdout.write(signal_list), I got a TypeError.
I now simply str()-convert in parse_signals() like this
signal.value = str(data_object.get('value', None))
resulting in having my value in single quotes consistently.

Identify Linux passwd file

I need help in writing a function (python preferably) to identify if a file is /etc/passwd or etc/shadow. So far I have tried using print(pw.getpwall()) but this reads the file from the os env. I need a library that takes input and can tell if a file is passwd/shadow file or not

passwd and shadow file format differs.
You can write a short function or class. First iteration would be:
Find root user, almost 100% true that root is the first entry
Check 2nd, 6th and 7th column (separator is : sign)
If 2nd is x and 6th is /root and 7th is /bin/*sh then it is a password file almost in 100%
If 2nd is a salt and hash (format: $salt$hash) and 6th is a number and 7th is empy then it is a shadow file almost in 100%
Naturally there could be problems:
Linux is configured not to use shadow file. In this case pasword file 2nd column contains the password
Linux is configured not to use salt (I guess is it possible or not)
Please check manuals: man 5 passwd and man 5 shadow
EDIT, 2020-04-24:
Here is the my corrected pwd.py:
#!/usr/bin/env python3
import os
import sys
passwd_file=('./passwd')
# path conversion handlers
def __nullpathconv(path):
return path
def __unixpathconv(path):
return path
# decide what field separator we can try to use - Unix standard, with
# the platform's path separator as an option. No special field conversion
# handler is required when using the platform's path separator as field
# separator, but are required for the home directory and shell fields when
# using the standard Unix (":") field separator.
__field_sep = {':': __unixpathconv}
if os.pathsep:
if os.pathsep != ':':
__field_sep[os.pathsep] = __nullpathconv
# helper routine to identify which separator character is in use
def __get_field_sep(record):
fs = None
for c in list(__field_sep.keys()):
# there should be 6 delimiter characters (for 7 fields)
if record.count(c) == 6:
fs = c
break
if fs:
return fs
else:
raise KeyError
# class to match the new record field name accessors.
# the resulting object is intended to behave like a read-only tuple,
# with each member also accessible by a field name.
class Passwd:
def __init__(self, name, passwd, uid, gid, gecos, dir, shell):
self.__dict__['pw_name'] = name
self.__dict__['pw_passwd'] = passwd
self.__dict__['pw_uid'] = uid
self.__dict__['pw_gid'] = gid
self.__dict__['pw_gecos'] = gecos
self.__dict__['pw_dir'] = dir
self.__dict__['pw_shell'] = shell
self.__dict__['_record'] = (self.pw_name, self.pw_passwd,
self.pw_uid, self.pw_gid,
self.pw_gecos, self.pw_dir,
self.pw_shell)
def __len__(self):
return 7
def __getitem__(self, key):
return self._record[key]
def __setattr__(self, name, value):
raise AttributeError('attribute read-only: %s' % name)
def __repr__(self):
return str(self._record)
def __cmp__(self, other):
this = str(self._record)
if this == other:
return 0
elif this < other:
return -1
else:
return 1
# read the whole file, parsing each entry into tuple form
# with dictionaries to speed recall by UID or passwd name
def __read_passwd_file():
if passwd_file:
passwd = open(passwd_file, 'r')
else:
raise KeyError
uidx = {}
namx = {}
sep = None
while 1:
entry = passwd.readline().strip()
if len(entry) > 6:
if sep is None:
sep = __get_field_sep(entry)
fields = entry.split(sep)
for i in (2, 3):
fields[i] = int(fields[i])
for i in (5, 6):
fields[i] = __field_sep[sep](fields[i])
record = Passwd(*fields)
if fields[2] not in uidx:
uidx[fields[2]] = record
if fields[0] not in namx:
namx[fields[0]] = record
elif len(entry) > 0:
pass # skip empty or malformed records
else:
break
passwd.close()
if len(uidx) == 0:
raise KeyError
return (uidx, namx)
# return the passwd database entry by UID
def getpwuid(uid):
u, n = __read_passwd_file()
return u[uid]
# return the passwd database entry by passwd name
def getpwnam(name):
u, n = __read_passwd_file()
return n[name]
# return all the passwd database entries
def getpwall():
u, n = __read_passwd_file()
return list(n.values())
# test harness
if __name__ == '__main__':
print(getpwall())

Language: Python(vers3.3); If statements inside while and if statements

My program is unable to go into the last of my if/else statement.
My code:
def main_prog():
while True:
data_dict = {'123': 'viksun', '111': ['tord'], '333': ['test']} # Data storage.
print (data_dict) # Track-print
prompt = input('Enter command >>> ').split(" ")
if prompt[0] == 'lookup':
B_value = name_lookup_return_boolean(data_dict, prompt[1])
if B_value == True:
print (prompt[1],"'s number is:", name_lookup(data_dict, prompt[1]))
else:
print("The name:" ,prompt[1], "do not exist, please try a different name")
Data struction of the dictionary: data_dict
data_dict = {'123': ['name1'], '111': ['tord'], '333': ['test']} # Data storage.
Function descriptions:
- name_lookup_returns_boolean(data_dict, name) returns a boolean value: True if the name is located in the dictionary and False if the name does not exist.
- name_lookup returns the key that corresponds to the specific name.
The functions have been tested separately from the whole program, so they are working. If you need them, I can also post them.
I can't get the program to run the last else statement in the program. This is going to run if the name does not exist in the dictionary: data_dict.
Here is the implementation of the two functions:
def name_lookup(data_dict, name):
for key, value in data_dict.items():
if name in value:
return key
def name_lookup_return_boolean(data_dict, name):
for value in data_dict.items():
if name in value:
return True
else:
return False
This is the variation I have tried to use_but with no sucess:
def name_lookup_version_02(data_dict, name):
for value in data_dict.values():
if name in value:
return True
else:
return False

In this line:
if B_value == True:
I'm guessing that B_value holds a true value, but not True. Try this instead:
if B_value:

The problem is in your name_lookup_return_boolean function. You are returning True for both conditions. Also you should be enumerating both key and value otherwise value will be assigned a tuple.
It should look like this:
def name_lookup_return_boolean(data_dict, name):
for key,value in data_dict.items():
if name in value:
return True
UPDATE: After testing, I realised previous answer was wrong - was only matching the first value

Python string assignment issue!

So I'm fairly new to Python but I have absolutely no idea why this strong oldUser is changing to current user after I make the parse call. Any help would be greatly appreciated.
while a < 20:
f = urllib.urlopen("SITE")
a = a+1
for i, line in enumerate(f):
if i == 187:
print line
myparser.parse(line)
if fCheck == 1:
result = oldUser[0] is oldUser[1]
print oldUser[0]
print oldUser[1]
else:
result = user is oldUser
fCheck = 1
print result
user = myparser.get_descriptions(firstCheck)
firstCheck = 1
print user
if result:
print "SAME"
array[index+1] = array[index+1] +0
else:
oldUser = user
elif i > 200:
break
myparser.reset()
I don't understand why result doesn't work either... I print out both values and when they're the same it's telling me they're not equal... Also, why does myparser.parse(line) turn oldUser into a size 2 array? Thanks!
** Here's the definition for myparse...
class MyParser(sgmllib.SGMLParser):
"A simple parser class."
def parse(self, s):
"Parse the given string 's'."
self.feed(s)
self.close()
def __init__(self, verbose=0):
"Initialise an object, passing 'verbose' to the superclass."
sgmllib.SGMLParser.__init__(self, verbose)
self.divs = []
self.descriptions = []
self.inside_div_element = 0
def start_div(self, attributes):
"Process a hyperlink and its 'attributes'."
for name, value in attributes:
if name == "id":
self.divs.append(value)
self.inside_div_element = 1
def end_div(self):
"Record the end of a hyperlink."
self.inside_div_element = 0
def handle_data(self, data):
"Handle the textual 'data'."
if self.inside_div_element:
self.descriptions.append(data)
def get_div(self):
"Return the list of hyperlinks."
return self.divs
def get_descriptions(self, check):
"Return a list of descriptions."
if check == 1:
self.descriptions.pop(0)
return self.descriptions

Don’t compare strings with is. That checks if they’re the same object, not two copies of the same string. See:
>>> string = raw_input()
hello
>>> string is 'hello'
False
>>> string == 'hello'
True
Also, the definition of myparser would be useful.

I'm not quite sure what your code is doing, but I suspect you want to use == instead of is. Using is compares object identity, which is not the same as string equality. Two different string objects may contain the same sequence of characters.
result = oldUser[0] == oldUser[1]
If you're curious, for more information on the behaviour of the is operator see Python “is” operator behaves unexpectedly with integers.

python list mysteriously getting set to something within my django/piston handler

Note: (I've updated this since the first two suggestions... you can view the old post in txt form here: http://bennyland.com/old-2554127.txt). The update I made was to better understand what was going wrong - and now I at least sort of know what's happening but I have no clue how to fix it.
Anyway, using Django and Piston, I've set up a new BaseHandler class named BaseApiHandler which does most of the work I was doing across all of my handlers. This worked great until I added the ability to limit the filters being applied to my results (for instance 'give me the first result only').
Examples (had to remove ":" because i can't submit more urls):
- http//localhost/api/hours_detail/empid/22 gives me all hours_detail rows from employee # 22
- http//localhost/api/hours_detail/empid/22/limit/first gives me the first hours_detail row from employee # 22
What's happening is that when I run /limit/first several times in succession, the first example is then broken, pretending it's a /limit/ url when it isn't.
Right now I'm storing whether or not it's a limit and what the limit is in a new class - prior to this stackoverflow edit, I was just using a list with two entries (limit = [] when initialized, limit = [0,1] when set). Prior to this stackoverflow edit, once you spammed /limit/first, when going to the first example 'limit' would be pre-set to [0,1] and the handler would then limit the query because of this. With the debug data I've added, I can say for certain that the list was pre-set and not getting set during the execution of the code.
I'm adding debug info into my response so I can see what's happening. Right now when you first ask for Example 1's url, you get this CORRECT statusmsg response:
"statusmsg": "2 hours_detail found with query: {'empid':'22','datestamp':'2009-03-02',}",
When you ask for Example 2's url, you get this CORRECT statusmsg response:
"statusmsg": "1 hours_detail found with query: {'empid':'22','datestamp':'2009-03-02','limit','first',with limit[0,1](limit,None... limit set 1 times),}",
However, if you refresh a bunch of times, the limit set value starts increasing (incrementing this value was something a friend of mine suggested to see if this variable was somehow getting kept around)
"statusmsg": "1 hours_detail found with query: {'empid':'22','datestamp':'2009-03-02','limit','first',with limit[0,1](limit,None... limit set 10 times),}",
Once that number goes above '1 times', you can start trying to get Example 1's url. Each time I now refresh example 1, i get odd results. Here are 3 different status messages from different refreshes (Notice that from each one, 'limit':'first' is CORRECTLY missing from the kwarg's debug output while the actual value of islimit is hovering between 8 and 10):
"statusmsg": "1 hours_detail found with query: {'empid':'22','datestamp':'2009-03-02',with limit[0,1](limit,None... limit set 10 times),}",
"statusmsg": "1 hours_detail found with query: {'empid':'22','datestamp':'2009-03-02',with limit[0,1](limit,None... limit set 8 times),}",
"statusmsg": "1 hours_detail found with query: {'empid':'22','datestamp':'2009-03-02',with limit[0,1](limit,None... limit set 9 times),}",
So it would appear that this object is getting cached. Prior to changing 'limit' form a list to a class, it also appeared that the list version of 'limit' was getting cached as after going to Example 2's url, i would sometimes have [0,1] as the limit.
Here are the updated snippets of the code (remember, you can view the first post here: bennyland.com/old-2554127.txt)
URLS.PY - inside 'urlpatterns = patterns('
#hours_detail/id/{id}/empid/{empid}/projid/{projid}/datestamp/{datestamp}/daterange/{fromdate}to{todate}
#empid is required
url(r'^api/hours_detail/(?:' + \
r'(?:[/]?id/(?P<id>\d+))?' + \
r'(?:[/]?empid/(?P<empid>\d+))?' + \
r'(?:[/]?projid/(?P<projid>\d+))?' + \
r'(?:[/]?datestamp/(?P<datestamp>\d{4,}[-/\.]\d{2,}[-/\.]\d{2,}))?' + \
r'(?:[/]?daterange/(?P<daterange>(?:\d{4,}[-/\.]\d{2,}[-/\.]\d{2,})(?:to|/-)(?:\d{4,}[-/\.]\d{2,}[-/\.]\d{2,})))?' + \
r')+' + \
r'(?:/limit/(?P<limit>(?:first|last)))?' + \
r'(?:/(?P<exact>exact))?$', hours_detail_resource),
HANDLERS.PY
class ResponseLimit(object):
def __init__(self):
self._min = 0
self._max = 0
self._islimit = 0
#property
def min(self):
if self.islimit == 0:
raise LookupError("trying to access min when no limit has been set")
return self._min
#property
def max(self):
if self.islimit == 0:
raise LookupError("trying to access max when no limit has been set")
return self._max
#property
def islimit(self):
return self._islimit
def setlimit(self, min, max):
self._min = min
self._max = max
# incrementing _islimit instead of using a bool so I can try and see why it's broken
self._islimit += 1
class BaseApiHandler(BaseHandler):
limit = ResponseLimit()
def __init__(self):
self._post_name = 'base'
#property
def post_name(self):
return self._post_name
#post_name.setter
def post_name(self, value):
self._post_name = value
def process_kwarg_read(self, key, value, d_post, b_exact):
"""
this should be overridden in the derived classes to process kwargs
"""
pass
# override 'read' so we can better handle our api's searching capabilities
def read(self, request, *args, **kwargs):
d_post = {'status':0,'statusmsg':'Nothing Happened'}
try:
# setup the named response object
# select all employees then filter - querysets are lazy in django
# the actual query is only done once data is needed, so this may
# seem like some memory hog slow beast, but it's actually not.
d_post[self.post_name] = self.queryset(request)
s_query = ''
b_exact = False
if 'exact' in kwargs and kwargs['exact'] <> None:
b_exact = True
s_query = '\'exact\':True,'
for key,value in kwargs.iteritems():
# the regex url possibilities will push None into the kwargs dictionary
# if not specified, so just continue looping through if that's the case
if value is None or key == 'exact':
continue
# write to the s_query string so we have a nice error message
s_query = '%s\'%s\':\'%s\',' % (s_query, key, value)
# now process this key/value kwarg
self.process_kwarg_read(key=key, value=value, d_post=d_post, b_exact=b_exact)
# end of the kwargs for loop
else:
if self.limit.islimit > 0:
s_query = '%swith limit[%s,%s](limit,%s... limit set %s times),' % (s_query, self.limit.min, self.limit.max, kwargs['limit'],self.limit.islimit)
d_post[self.post_name] = d_post[self.post_name][self.limit.min:self.limit.max]
if d_post[self.post_name].count() == 0:
d_post['status'] = 0
d_post['statusmsg'] = '%s not found with query: {%s}' % (self.post_name, s_query)
else:
d_post['status'] = 1
d_post['statusmsg'] = '%s %s found with query: {%s}' % (d_post[self.post_name].count(), self.post_name, s_query)
except:
e = sys.exc_info()[1]
d_post['status'] = 0
d_post['statusmsg'] = 'error: %s %s' % (e, traceback.format_exc())
d_post[self.post_name] = []
return d_post
class HoursDetailHandler(BaseApiHandler):
#allowed_methods = ('GET', 'PUT', 'POST', 'DELETE',)
model = HoursDetail
exclude = ()
def __init__(self):
BaseApiHandler.__init__(self)
self._post_name = 'hours_detail'
def process_kwarg_read(self, key, value, d_post, b_exact):
# each query is handled slightly differently... when keys are added
# handle them in here. python doesn't have switch statements, this
# could theoretically be performed using a dictionary with lambda
# expressions, however I was affraid it would mess with the way the
# filters on the queryset work so I went for the less exciting
# if/elif block instead
# querying on a specific row
if key == 'id':
d_post[self.post_name] = d_post[self.post_name].filter(pk=value)
# filter based on employee id - this is guaranteed to happen once
# per query (see read(...))
elif key == 'empid':
d_post[self.post_name] = d_post[self.post_name].filter(emp__id__exact=value)
# look for a specific project by id
elif key == 'projid':
d_post[self.post_name] = d_post[self.post_name].filter(proj__id__exact=value)
elif key == 'datestamp' or key == 'daterange':
d_from = None
d_to = None
# first, regex out the times in the case of range vs stamp
if key == 'daterange':
m = re.match('(?P<daterangefrom>\d{4,}[-/\.]\d{2,}[-/\.]\d{2,})(?:to|/-)(?P<daterangeto>\d{4,}[-/\.]\d{2,}[-/\.]\d{2,})', \
value)
d_from = datetime.strptime(m.group('daterangefrom'), '%Y-%m-%d')
d_to = datetime.strptime(m.group('daterangeto'), '%Y-%m-%d')
else:
d_from = datetime.strptime(value, '%Y-%m-%d')
d_to = datetime.strptime(value, '%Y-%m-%d')
# now min/max to get midnight on day1 through just before midnight on day2
# note: this is a hack because as of the writing of this app,
# __date doesn't yet exist as a queryable field thus any
# timestamps not at midnight were incorrectly left out
d_from = datetime.combine(d_from, time.min)
d_to = datetime.combine(d_to, time.max)
d_post[self.post_name] = d_post[self.post_name].filter(clock_time__gte=d_from)
d_post[self.post_name] = d_post[self.post_name].filter(clock_time__lte=d_to)
elif key == 'limit':
order_by = 'clock_time'
if value == 'last':
order_by = '-clock_time'
d_post[self.post_name] = d_post[self.post_name].order_by(order_by)
self.limit.setlimit(0, 1)
else:
raise NameError
def read(self, request, *args, **kwargs):
# empid is required, so make sure it exists before running BaseApiHandler's read method
if not('empid' in kwargs and kwargs['empid'] <> None and kwargs['empid'] >= 0):
return {'status':0,'statusmsg':'empid cannot be empty'}
else:
return BaseApiHandler.read(self, request, *args, **kwargs)

I would say that there is a basic flaw in your code, if has_limit() can return True when limit is a list of length 2, but this line will fail if limit is shorter than 3 elements long:
s_query = '%swith limit[%s,%s](limit,%s > traceback:%s),' %
(s_query, self.limit[0], self.limit[1], kwargs['limit'],
self.limit[2])
Why are you initializing self.limit to an invalid length list? You could also make this code a little more defensive:
if self.has_limit():
s_query += 'with limit[%s,%s]' % self.limit[0:1]
if 'limit' in kwargs and len(self.limit) > 2:
s_query += '(limit,%s > traceback:%s),' %
(kwargs['limit'], self.limit[2])

I think you may be creating an alias to your internal limit list, via the get_limit property accessor. Try removing (or at least adding a print statement) inside this accessor. If you have code externally that binds a local list to get_limit, then it can update the contents using append, del, or assignment to slices, such as [:]. Or try this:
def get_limit(self):
return self._limit[:]
Instead of binding your internal list to an external name, it will make a copy of your internal list.