Related
This question already has answers here:
Possible to enforce type hints?
(2 answers)
Closed 12 months ago.
I struggle with Python(3.9.7) automatically adding singlequotes around a float-value, which leads to a failing assertion.
My goal is to use pytest to assert an expected output of a parsing-function. The parser takes a json-object (shortened code sample below) and returns a list of Signal objects. I copied the print-out from logging.debug(f"{signal_list}") inside the parsing function, and assigned it to expected_result in my test-function:
#Arrange
message = json.dumps({"data":{"name":"battery_volt","alias":"Volt","unit":"volt","value":12.0,"time":1644587969}})
expected_result = [Signal(id=None, name='battery_volt', description=None, value=12.0, type=None, unit='volt', time='2022-02-11T13:59:29')]
print(expected_result)
p = Parser(message)
#Act
result = p.parse_message()
#Assert
assert result == expected_result
Irritatingly, pytest -vv throws an AssertionError:
E Full diff:
E - [Signal(id=None, name='battery_volt', description=None, value='12.0', type=None, unit='volt', time='2022-02-11T13:59:29')]
E ? - -
E + [Signal(id=None, name='battery_volt', description=None, value=12.0, type=None, unit='volt', time='2022-02-11T13:59:29')]
The upper line seems to be the value of expected_result, because print(expected_result)
also adds the singlequotes around the 12.0
I assume the copied output from logging.debug(f"{signal_list}") isn't the same as the real value of result. I tried typecasting expected_result as list, str()-converting both result and expected_result inside the test, but expected_result always has '12.0' and result has 12.0.
I desperatly need a hint how to do this kind of assertion the correct way.
EDIT:
Here is the parsing function:
def parse_message(self):
message = json.loads(self.message)
#logging.debug(f"message is: {message}")
message_data = message.get('data', {})
parsed_data = []
try:
device = message_data.get('device', None)
if device is not None:
vehicle = self.parse_vehicle(device)
parsed_data.append(vehicle)
else:
logging.error("'device' was None!")
except Exception as e:
logging.error(e)
signals = []
try:
data = message_data.get('data', None)
if data is not None:
signals = self.parse_signals(data)
gps = message_data.get('gps', None)
if gps is not None:
gps_signal = self.parse_gps(gps)
signals.append(gps_signal)
parsed_data.append(signals)
except Exception as e:
logging.error(e)
return parsed_data
if __name__ == "__main__":
setup_logging()
message = json.dumps({"consumerid":redacted,"data":{"device":{"time":1644587969,"imei":"redacted","man_id":redacted,"car_id":999,"vin":"redacted"},"data":[{"name":"battery_volt","alias":"Volt","unit":"volt","value":12.0,"time":1644587969}],"gps":{"lat":51.437515,"lon":6.9281199,"dir":252,"alt":88,"sat":19,"time":1644587969}},"event":"redacted","itemid":redacted,"itemtype":1,"senderip":"redacted"})
p = Parser(message)
signal_list = p.parse_message()
logging.debug(f"{signal_list}")
Please note that the passed json-objects are more complex than the code-sample in the original post.
class Signal(BaseModel):
id: int = None
name: str = None
description: str = None
value: str = None
type: str = None
unit: str = None
time: str = None
EDIT2 - Assignment of Signal.value happens here:
def parse_signals(self, data):
signals = []
#logging.debug(f"data is : {data}")
for data_object in data:
signal = Signal()
try:
signal.name = data_object.get('name', None)
#get dtc-count value as seperate return-element, needed by the controller to handle dtc-codes
if signal.name == 'dtc_count':
self.dtc_count == data_object.get('value', None)
signal.description = data_object.get('description', None)
signal.value = data_object.get('value', None)
signal.time = datetime.datetime.utcfromtimestamp(data_object.get('time', None)).isoformat()
signal.unit = data_object.get('unit', None)
if signal.unit == "dtc":
signal.type = "1"
if signal.name is not None:
signals.append(signal)
#logging.debug(signal.__dict__)
except Exception as e:
logging.error(f"While parsing signal {data_object}, the following error occured: {e}")
return signals
When parse_message is called as __name__ == "main":, the testcode beneath outputs value=12.0
Ok, turns out python type hints don't enforce like I expected:
When parse_message is called by way of __name__ == "main":, the testcode beneath outputs value=12.0 Apparently, despite the type-hint :str for Signal.value, 12.0 was assigned as float. When I tried to sys.stdout.write(signal_list), I got a TypeError.
I now simply str()-convert in parse_signals() like this
signal.value = str(data_object.get('value', None))
resulting in having my value in single quotes consistently.
I am trying to write a function that validates a package that contains 2 dates (beginning and end), a destination and a price
Initially, I tried to write a function that "creates" dates and puts them in a list and then compares them to find out if the end date was lower than the beginning but I figured that was too complicated so I resorted to the datetime inbuilt module
However, when I try to run the test function, it fails and it outputs this error message
File "C:\Users\Anon\Desktop\Fac\FP\pythonProject\main.py", line 65, in valideaza_pachet
raise Exception(err)
Exception: wrong dates!
I assume that I must have messed up a condition in the valideaza_pachet() function, but I don't understand what I did wrong
The code:
import time
import calendar
from datetime import date, datetime
def creeaza_pachet(data_i, data_s, dest, pret):
# function that creates a tourism package, data_i = beginning date, data_s = end date, dest = destination and pret = price
return {
"data_i": data_i,
"data_s": data_s,
"dest": dest,
"pret": pret
}
def get_data_i(pachet):
# functie that returns the beginning date of the package
return pachet["data_i"]
def get_data_s(pachet):
# functie that returns the end date of the package
return pachet["data_s"]
def get_destinatie(pachet):
# functie that returns the destination of the package
return pachet["dest"]
def get_pret(pachet):
# functie that returns the price of the package
# input: pachet - un pachet
# output: pretul float > 0 al pachetului
return pachet["pret"]
def valideaza_pachet(pachet):
#functie that validates if a package was correctly introduced or not
#it raises an Exception as ex if any of the conditions aren't met
err = ""
if get_data_i(pachet) < get_data_s(pachet):
err += "wrong dates!" # if the end date is lower than the beginning date
if get_destinatie(pachet) == " ":
err += "wrong destination!"
if get_pret(pachet) <= 0:
err += "wrong price!"
if len(err) > 0:
raise Exception(err)
def test_valideaza_pachet():
pachet = creeaza_pachet(datetime.strptime('24/08/2021',"%d/%m/%Y"), datetime.strptime('26/08/2021',"%d/%m/%Y"), "Galati", 9000.1)
valideaza_pachet(pachet)
pachet_gresit = creeaza_pachet(datetime.strptime('24/08/2021',"%d/%m/%Y"), datetime.strptime('22/08/2021',"%d/%m/%Y"), "Galati", 9000.1)
try:
valideaza_pachet(pachet_gresit)
assert False
except Exception as ex:
assert (str(ex) == "wrong dates!\n")
alt_pachet = creeaza_pachet(datetime.strptime('24/08/2021',"%d/%m/%Y"), datetime.strptime('22/08/2021',"%d/%m/%Y"), " ", -904)
try:
valideaza_pachet(alt_pachet)
assert False
except Exception as ex:
assert(str(ex) == "wrong dates!\nwrong destination!\nwrong price!\n")
def test_creeaza_pachet():
data_i_string = '24/08/2021'
data_i = datetime.strptime(data_i_string, "%d/%m/%Y")
data_s_string = '26/08/2021'
data_s = datetime.strptime(data_s_string, "%d/%m/%Y")
dest = "Galati"
pret = 9000.1
pachet = creeaza_pachet(data_i,data_s,dest,pret)
assert (get_data_i(pachet) == data_i)
assert (get_data_s(pachet) == data_s)
assert (get_destinatie(pachet) == dest)
assert (abs(get_pret(pachet) - pret) < 0.0001)
def run_teste():
test_creeaza_pachet()
test_valideaza_pachet()
def run():
pass
def main():
run()
run_teste()
main()
This is more of a code review and kind of off-topic here but... First,
drop all assert False - these won't do anything useful
drop the getter functions, that just makes things convoluted (just use dict[key] in the code as long as you don't use custom class)
drop unnecessary imports like calendar
you might also want to drop the run and main functions (again convoluted) - just call the functions you need
Then you could change valideaza_pachet to raise specific value errors:
def valideaza_pachet(pachet):
if pachet["data_i"] >= pachet["data_s"]:
raise ValueError("start date must be < end date")
if pachet["dest"] == " ":
raise ValueError("destination must not be empty")
if pachet["pret"] <= 0:
raise ValueError("price must be >= 0")
# returns None if no exception was raised
Now to test a valid package, you would just do
valideaza_pachet(pachet) # no exception expected
Testing an invalid package is a bit more complicated without a class that can be unit-tested (see e.g. here) - but you can catch the exception and use the else clause of the try/except to raise an AssertionError that says you wanted an exception:
try:
valideaza_pachet(pachet_gresit)
except Exception as ex:
print(f"successfully received exception: {ex}")
else:
raise AssertionError("validation should have raised an Exception")
or even
assert str(ex) == "start date must be < end date", f"got '{ex}', expected 'start date must be < end date'"
in place of the print statement.
I have already written a simple python cmd program using the cmd module. However, I would like it to be modular. For Example, there would be a folder named script and any .py that contains a command would add the command to the application. How would I go about that?
NOTE:
I have figured out how to find and load modules that are within a folder using importlib.
First, you must understand how the cmd module works. I would not go into it here but the gist is command is entered, then split up into the actual command itself and its arguments. The command is then searched in the implementation of the cmd module using getattr(). The resulting function returned is then executed. Raises an error if the function(attribute) is not found.
Add the list of modules added to a list.
lst.append(importlib.importlib.import_module(<module path here>))
When cmd is finding a command, modify the code to get it to run through the list of imported modules and see if the function/command exists in that module. If so, execute it.
rough code to find func
def findfunc(funcname):
for module in lst:
if hasattr(<module class where func is stored>, "<funcname>"):
return getattr(<module class where func is stored>,"<funcname>")
However, I will leave my implementation here. It also includes the capability to reload modules(limited to a certain extent) and private commands that do not show up in help or autocomplete(I am using py prompt toolkit). A fair bit of warning, no docs or help in understanding this code as it is only meant for my personal use only. Meaning don't directly copy it and also It was a modification to the python cmd module.
import importlib
import os
import string
import sys
import time
from prompt_toolkit import PromptSession
from prompt_toolkit.completion import WordCompleter
from prompt_toolkit.styles import Style
from internal.logger import logger # my logger Implementation, ignore it
__all__ = ["cli2"]
PROMPT = '>>> '
IDENTCHARS = string.ascii_letters + string.digits + '_'
style = Style.from_dict(
{
"completion-menu.completion": "bg:#008888 #ffffff",
"completion-menu.completion.current": "bg:#00aaaa #000000",
"scrollbar.background": "bg:#88aaaa",
"scrollbar.button": "bg:#222222",
}
)
def parse(arg):
return arg.split()
class Interpreter:
identchars = IDENTCHARS
ruler = '='
lastcmd = ''
doc_leader = ""
doc_header = "Available Commands (type help <topic> to get documentation for <topic> ):"
misc_header = "Miscellaneous help topics:"
nohelp = "*** No help on %s"
use_rawinput = 1
modulelst = []
internalcmdlst = ["help", "exit", "reload", "listmodules", "listvar"]
def __init__(self, intro=None, prompt=PROMPT, stdin=None, stdout=None):
"""Instantiate a line-oriented interpreter framework.
The optional arguments stdin and stdout
specify alternate input and output file objects; if not specified,
sys.stdin and sys.stdout are used.
"""
self.prompt = prompt
self.intro = intro
if stdin is not None:
self.stdin = stdin
else:
self.stdin = sys.stdin
if stdout is not None:
self.stdout = stdout
else:
self.stdout = sys.stdout
self.cmdqueue = []
def cmdloop(self):
self.preloop()
try:
if self.intro:
self.stdout.write(str(self.intro) + "\n")
stop = None
while not stop:
if self.cmdqueue:
line = self.cmdqueue.pop(0)
else:
line = self.session.prompt(self.prompt)
line = self.precmd(line)
stop = self.onecmd(line)
stop = self.postcmd(stop, line)
self.postloop()
finally:
pass
def precmd(self, line):
"""Hook method executed just before the command line is
interpreted, but after the input prompt is generated and issued.
"""
# Parse command to argument
return self.parseline(line)
def postcmd(self, stop, line):
"""Hook method executed just after a command dispatch is finished."""
return stop
def preloop(self):
"""Hook method executed once when the cmdloop() method is called."""
self.autocommands = [item[3:] for item in self.get_names() if (item[-5:] != '__pvt') and (item[:3] == 'do_')]
self.session = PromptSession(completer=WordCompleter(self.autocommands, ignore_case=False))
def postloop(self):
"""Hook method executed once when the cmdloop() method is about to
return.
"""
self.autocommands, self.session = None, None
def parseline(self, line):
"""Parse the line into a command name and a string containing
the arguments. Returns a tuple containing (command, args, line).
'command' and 'args' may be None if the line couldn't be parsed.
"""
line = line.strip()
if not line:
return None, None, line
elif line[0] == '?':
line = 'help ' + line[1:]
elif line[0] == '!':
if hasattr(self, 'do_shell'):
line = 'shell ' + line[1:]
else:
return None, None, line
i, n = 0, len(line)
while i < n and line[i] in self.identchars: i = i + 1
cmd, arg = line[:i], line[i:].strip()
return cmd, arg, line
def onecmd(self, linecommand):
"""Interpret the argument as though it had been typed in response
to the prompt.
This may be overridden, but should not normally need to be;
see the precmd() and postcmd() methods for useful execution hooks.
The return value is a flag indicating whether interpretation of
commands by the interpreter should stop.
"""
cmd, arg, line = linecommand
if not line:
return self.emptyline()
if cmd is None:
return self.default(line)
self.lastcmd = line
if line == 'EOF':
self.lastcmd = ''
if cmd == '':
return self.default(line)
if cmd in self.internalcmdlst:
return self.getfunc(cmd)(arg)
else:
try:
terminalclass, func = self.getfunc(cmd)
except AttributeError:
return self.default(line)
return func(terminalclass, arg)
def emptyline(self):
"""Called when an empty line is entered in response to the prompt.
If this method is not overridden, it repeats the last nonempty
command entered.
"""
if self.lastcmd:
return self.onecmd(self.lastcmd)
def default(self, line):
"""Called on an input line when the command prefix is not recognized.
If this method is not overridden, it prints an error message and
returns.
"""
self.stdout.write('*** Unknown syntax: %s\n' % line.split(" ")[0])
def get_names(self):
# This method used to pull in base class attributes
# at a time dir() didn't do it yet.
lst = [*dir(self.__class__)]
for modulejs in self.modulelst:
terminalclass = getattr(getattr(modulejs["module"], 'cli_helper'), "Terminal")
lst += dir(terminalclass)
return lst
def print_topics(self, header, cmds, maxcol):
if cmds:
self.stdout.write("%s\n" % str(header))
if self.ruler:
self.stdout.write("%s\n" % str(self.ruler * len(header)))
self.columnize(cmds, maxcol - 1)
self.stdout.write("\n")
def columnize(self, lst, displaywidth=80):
"""Display a list of strings as a compact set of columns.
Each column is only as wide as necessary.
Columns are separated by two spaces (one was not legible enough).
"""
if not lst:
self.stdout.write("<empty>\n")
return
nonstrings = [i for i in range(len(lst))
if not isinstance(lst[i], str)]
if nonstrings:
raise TypeError("list[i] not a string for i in %s"
% ", ".join(map(str, nonstrings)))
size = len(lst)
if size == 1:
self.stdout.write('%s\n' % str(lst[0]))
return
# Try every row count from 1 upwards
for nrows in range(1, len(lst)):
ncols = (size + nrows - 1) // nrows
colwidths = []
totwidth = -2
for col in range(ncols):
colwidth = 0
for row in range(nrows):
i = row + nrows * col
if i >= size:
break
x = lst[i]
colwidth = max(colwidth, len(x))
colwidths.append(colwidth)
totwidth += colwidth + 2
if totwidth > displaywidth:
break
if totwidth <= displaywidth:
break
else:
nrows = len(lst)
ncols = 1
colwidths = [0]
for row in range(nrows):
texts = []
for col in range(ncols):
i = row + nrows * col
if i >= size:
x = ""
else:
x = lst[i]
texts.append(x)
while texts and not texts[-1]:
del texts[-1]
for col in range(len(texts)):
texts[col] = texts[col].ljust(colwidths[col])
self.stdout.write("%s\n" % str(" ".join(texts)))
def addmodulesfromdirectory(self, directory):
try:
directory_contents = os.listdir(directory)
for item in directory_contents:
self.addmodule(os.path.join(directory, item), "".join([directory[2:], ".", item]))
except OSError as e:
print("Something Happened\nPlease try again :)")
logger.error("OSErrot, {}".format(e))
except ImportError as e:
print("Something Went Wrong while importing modules\nPlease try again :)")
logger.error("ImportError, {}".format(e))
except Exception as e:
print("Oops, my fault\nPlease try again :)")
logger.error("Exception, {}".format(e))
finally:
logger.info("Done importing cli modules")
def addmodule(self, pathdir, importdir):
if os.path.isdir(pathdir):
importedmodule = importlib.import_module(importdir)
modulejs = {
"module": importedmodule,
"name": importedmodule.__name__,
"importdir": importdir,
"pathdir": pathdir
}
self.modulelst.append(modulejs)
def removemodule(self, modulename):
if modulename in [x["name"] for x in self.modulelst]:
self.modulelst.remove(self.getmodulejs(modulename))
del sys.modules[modulename]
else:
raise ModuleNotFoundError
def replacemodule(self, modulename):
module = self.getmodulejs(modulename)
self.removemodule(modulename)
self.addmodule(module["pathdir"], module["importdir"])
def getmodulejs(self, modulename):
for i in self.modulelst:
if i['name'] == modulename:
return i
#staticmethod
def getmoduledep(modulename):
unsortedscriptsysmodules = [module for module in sys.modules if (("script" in module) and ("script" != module))]
sortedlst = []
for scriptmodules in unsortedscriptsysmodules:
mod = sys.modules[scriptmodules]
if not (hasattr(mod, "__path__") and getattr(mod, '__file__', None) is None) and (
scriptmodules != "script.{}".format(modulename)):
sortedlst.append(sys.modules[scriptmodules])
return sortedlst
def reloadmoduledep(self, modulename):
for dep in self.getmoduledep(modulename):
try:
importlib.reload(dep)
except ModuleNotFoundError as e:
print(e)
def getfunc(self, command):
if "do_" in command:
command = command[3:]
if command in self.internalcmdlst:
return getattr(self, "_".join(['do', command]))
else:
for modulejs in self.modulelst:
terminalclass = getattr(getattr(modulejs["module"], 'cli_helper'), "Terminal")
if hasattr(terminalclass, "_".join(['do', command])):
return terminalclass, getattr(terminalclass, "_".join(['do', command]))
raise AttributeError
def do_help(self, arg):
"""
List available commands with "help" or detailed help with "help cmd".
"""
if arg:
# XXX check arg syntax
try:
func = self.getfunc('help_' + arg)
except AttributeError:
try:
if arg[-5:] == '__pvt':
raise AttributeError
doc = self.getfunc('do_' + arg).__doc__
if doc:
self.stdout.write("%s\n" % str(doc))
return
except AttributeError:
pass
self.stdout.write("%s\n" % str(self.nohelp % (arg,)))
return
func()
else:
names = self.get_names()
cmds_doc = []
funchelp = {}
for name in names:
if name[:5] == 'help_':
funchelp[name[5:]] = 1
names.sort()
# There can be duplicates if routines overridden
prevname = ''
for name in names:
if name[:3] == 'do_' and name[-5:] != '__pvt':
if name == prevname:
continue
prevname = name
cmd = name[3:]
if cmd in funchelp:
cmds_doc.append(cmd)
del funchelp[cmd]
elif self.getfunc(name).__doc__:
cmds_doc.append(cmd)
self.stdout.write("%s\n" % str(self.doc_leader))
self.print_topics(self.doc_header, cmds_doc, 80)
self.print_topics(self.misc_header, list(funchelp.keys()), 80)
def do_exit(self, arg):
"""
Exit
"""
if arg:
print(self.prompt + "No arguments please")
print("Exiting")
time.sleep(1)
return True
def do_reload(self, arg):
if arg == "":
print("No Arguments found")
return
arg = parse(arg)
if arg[0] == "all":
localmodulelst = self.modulelst.copy()
self.modulelst = []
for i in localmodulelst:
print("Reloading", ".".join(i["name"].split(".")[1:]))
importlib.invalidate_caches()
self.reloadmoduledep(i["name"])
self.addmodule(i["pathdir"], i["importdir"])
self.autocommands = [item[3:] for item in self.get_names() if
(item[-5:] != '__pvt') and (item[:3] == 'do_')]
self.session = PromptSession(completer=WordCompleter(self.autocommands, ignore_case=False))
else:
print("Only argument \'all\' is accepted")
def do_listmodules(self, arg):
arg = parse(arg)
if arg[0] == "sys":
for i in sys.modules:
print(i)
elif len(arg) != 0:
print("No Argument Please")
else:
print("Listing all imported Modules")
for i in self.modulelst:
print(".".join(i["module"].__name__.split(".")[1:]))
Hi I'm pretty new to Python and I've just started to learn about errors and exceptions.I have this function in a class that inserts a line at a given index called num.I know python will raise an error if no num is given but I want to raise my own error.How do I do that?This is what I tried. But the error raised is still the default error?
def insertNum(self, num, line):
if num== None:
raise Exception("Num not specified.")
else:
self.list.insert(num, line)
return self.list
You can use try...except statement.
def insertNum(num, line):
try:
list.insert(num, line)
return list
except:
print('custom error')
You can set the default value of num to None and then check if the value is None.
def insertNum(self, line, num=None):
if num is None:
raise Exception("Num not specified.")
else:
self.list.insert(num, line)
return self.list
If you pass only one parameter to the insertNum method, num will be set the None (the default value) and will raise the exception.
If you don't want to change the order of the arguments, you can use this:
def insertNum(self, num, line=None):
if line is None:
raise Exception("Num not specified.")
else:
self.list.insert(num, line)
return self.list
A simple demonstration for how default arguments work:
>>> def foo(bar, baz=None):
... print(bar, baz)
...
>>> foo(1, 2)
1 2
>>> foo(2)
2 None
I suggest you read about exceptions and errors
But the main idea is that you catch errors and then you handle them the way you like.
try:
#do something
except Exception as e:
# error occured
print("A wild error appeared")
wrap your function with another function that will have try and except` and there you could raise what ever error/exception you want.
def wrapper_func(self, num, line):
try:
self.insertNum(num, line)
except Exception as e:
raise Exception("whatever you want")
I am trying to run below python udf in Pig
#outputSchema("word:chararray")
def get(s):
out = s.lower()
return out;
I am getting below error :
File "/home/test.py", line 3, in get
out = s.lower()
AttributeError: 'NoneType' object has no attribute 'lower'
You should handle the case when s is none. In most of the examples such as:
from pig_util import outputSchema
#outputSchema('decade:chararray')
def decade(year):
"""
Get the decade, given a year.
e.g. for 1998 -> '1990s'
"""
try:
base_decade_year = int(year) - (int(year) % 10)
decade_str = '%ss' % base_decade_year
print 'input year: %s, decade: %s' % (year, decade_str)
return decade_str
except ValueError:
return None
You need to handle the case when the value is None. So, one possible fix would be to try:
#outputSchema("word:chararray")
def get(s):
if s is None:
return None
return str(s).lower()