Related
I am a beginner in Python and I need some help on decorators. I am writing a few methods which call other methods that are generated using swagger. Basically all these swagger methods have GET APIs. All I need to do in my code is to call those swagger methods and return the value. I am looking for ways to optimize this instead of writing the same kind of method for each API. I came across decorators which can be used in this case. But my implementation is not giving the desired result
def get_component_info(self, func):
def inner():
data = None
try:
ret = func()
if ret.status == 200:
log.info('ret ' + str(ret))
else:
logging.error('Error: ' + str(ret.text))
except Exception as e:
logging.error(" failed with error " + str(e.reason) +
" and error code " + str(e.status))
finally:
return data
return inner()
def get_firewall_info(self):
return self._monitor.list_firewall_monitors_with_http_info() <-- swagger method
def get_firewall_info_caller(self):
get_firewall_info = self.get_component_info(self.get_firewall_info())
But the above implementation always return None because it never executes self._monitor.list_firewall_monitors_with_http_info(), but the test isn't failing
If you help me fix this, then I can use the same for getting server info, auth info, network info, etc. If decorators can't be used, what else I can use to optimize this..?
Thanks
Decorators are usually functions which take a second function as an argument and then define and return a third function which calls the second function while changing its input and/or output. It looks like you have a better handle on this than I did as a beginner.
def decorator(func):
def inner(*args, **kwargs):
# Do stuff here...
value = func(*args, **kwargs)
# ...or here.
return value
return inner
The only change I would recommend to your decorator is not to call inner and return the result, but to return the function itself. When you make this change you'll have to call the function you are returning now after it is returned.
def get_component_info(self, func):
def inner():
# ...
return inner
def get_firewall_info_caller(self):
# You will now want to call the `inner` function after you get
# it from `get_component_info`.
get_firewall_info = self.get_component_info(...)()
It looks like the core of your bug is that you aren't providing a function to get_component_info; you're providing the result of calling that function. I think that changing the code to not call get_firewall_info should fix your code.
def get_firewall_info_caller(self):
# You don't want to call the function you're providing to a
# decorator, since it's expecting the function not the result.
get_firewall_info = self.get_component_info(self.get_firewall_info)()
I resolved it as shown below. Not sure if it is the right approach. Please correct me
def get_component_info(self, func):
def inner():
data = None
try:
ret = func()
if ret.status == 200:
log.info('ret ' + str(ret))
else:
logging.error('Error: ' + str(ret.text))
except Exception as e:
logging.error(" failed with error " + str(e.reason) +
" and error code " + str(e.status))
finally:
return data
return inner
def get_firewall_info(self):
data = self.get_component_info(self._monitor.list_firewall_monitors_with_http_info)()
return data
Thats my first question on Stackoverflow and im a totally Python beginner.
I want to write, to get firm with python, a small Backup-Programm, the main part is done, but now i want to make it a bit "portable" and use a Config file, which i want to Validate.
My class "getBackupOptions" should be give Back a validate dict which should be enriched with "GlobalOptions" and "BackupOption" so that i finally get an fully "BackupOption" dict when i call "getBackupOptions.BackupOptions".
My Question now is, (in this Example is it easy, because its only the Function which check if the Path should be Recursive searched or not) how to simplify my Code?
For each (possible) Error i must write a new "TryExcept" Block - Can i Simplify it?
Maybe is there another way to Validate Config Files/Arrays?
class getBackupOptions:
def __init__(self,BackupOption,GlobalOptions):
self.BackupOption = BackupOption
self.GlobalOptions = GlobalOptions
self.getRecusive()
def getRecusive(self):
try:
if self.BackupOption['recursive'] != None:
pass
else:
raise KeyError
except KeyError:
try:
if self.GlobalOptions['recursive'] != None:
self.BackupOption['recursive'] = self.GlobalOptions['recursive']
else:
raise KeyError
except KeyError:
print('Recusive in: ' + str(self.BackupOption) + ' and Global is not set!')
exit()
Actually i only catch an KeyError, but what if the the Key is there but there is something else than "True" or "False"?
Thanks a lot for you help!
You may try this
class getBackupOptions:
def __init__(self,BackupOption,GlobalOptions):
self.BackupOption = BackupOption
self.GlobalOptions = GlobalOptions
self.getRecusive()
def getRecusive(self):
if self.BackupOption.get('recursive') == 'True' and self.GlobalOptions.get('recursive') == 'True':
self.BackupOption['recursive'] = self.GlobalOptions['recursive']
else:
print('Recusive in: ' + str(self.BackupOption) + ' and Global is not set!')
exit()
Here get method is used, therefore KeyError will not be faced.
If any text other than True comes in the field it will be considered as False.
Can I limit the value range of eval()?
eval('12345678**9') returns a large number
eval('1234567**8**9') neither returns nor throws an exception
I could live with #1 but not with #2. All I need is results in the range of 32bit integers. I fear there is no way to tell eval to stop calculating too large numbers, or is there?
I've written "calculators" before using ast to parse the string into a tree and then walk the tree. In this case, if you want to do some trickery, you can make this work:
import ast
import ctypes
import operator
def _pow(a, b):
if isinstance(a, (ctypes.c_int, ctypes.c_float, ctypes.c_double)):
a = float(a.value)
if isinstance(b, (ctypes.c_int, ctypes.c_float, ctypes.c_double)):
b = float(b.value)
return ctypes.c_double(a ** b)
def _wrap_bin_op(op):
def wrapper(a, b):
if isinstance(a, (ctypes.c_int, ctypes.c_float, ctypes.c_double)):
a = float(a.value)
if isinstance(b, (ctypes.c_int, ctypes.c_float, ctypes.c_double)):
b = float(b.value)
return ctypes.c_double(op(a, b))
return wrapper
def _wrap_unary_op(op):
def wrapper(a):
if isinstance(a, (ctypes.c_int, ctypes.c_float)):
a = float(a.value)
return ctypes.c_double(op(a))
return wrapper
_OP_MAP = {
ast.Add: _wrap_bin_op(operator.add),
ast.Sub: _wrap_bin_op(operator.sub),
ast.Pow: _wrap_bin_op(operator.pow),
ast.Mult: _wrap_bin_op(operator.mul),
ast.Div: _wrap_bin_op(operator.truediv),
ast.Invert: _wrap_unary_op(operator.neg),
}
class Calc(ast.NodeVisitor):
def visit_BinOp(self, node):
left = self.visit(node.left)
right = self.visit(node.right)
return _OP_MAP[type(node.op)](left, right)
def visit_Num(self, node):
if isinstance(node.n, int):
val = ctypes.c_int(node.n)
elif isinstance(node.n, float):
val = ctypes.c_double(node.n)
return val
def visit_Expr(self, node):
return self.visit(node.value)
#classmethod
def evaluate(cls, expression):
tree = ast.parse(expression)
calc = cls()
return calc.visit(tree.body[0])
print(Calc.evaluate('12345678**8'))
print(Calc.evaluate('5 * 8'))
Note that unlike eval, I'm specifically picking and choosing what operations I want to allow -- and I have control over how they behave. In this case, I'm doing all of my math with ctypes to avoid HUGE numbers. I'm also preventing integer __pow__ and forcing those arguments to become floats before raising to a specific power.
As John Coleman recommended, I'll add my own solution here. Thanks for the discussion, I learned a lot about pythons capabilities.
As I commented already:
I found a solution by making any number a float via concatenating '.0', eval('1234567.0**8.0**9.0') throws an exception, that's fine.
Here's the bigger context, where this evaluation is embedded:
import itertools
digits1to8 = list(str(i+1) for i in range(8)) #('1','2','3','4','5','6','7','8')
with open("expressions.txt", "w") as outfile:
for operators in itertools.product(['','.0+','.0-','.0*','.0/','.0**'], repeat=8):
calculation = zip(digits1to8,operators)
expression = (''.join(list(itertools.chain(*calculation))))+'9.0'
try:
out = str(eval(expression))+','
expression = expression.replace('.0','')
out = out.replace('.0,',',') + expression
if (not out.find('.')>0):
print(out, file=outfile)
except:
pass
beforehand I had ['','+','-','*','/','**'] instead of ['','.0+','.0-','.0*','.0/','.0**']. Overall this just is a little mathematical experimentation in response to https://www.youtube.com/watch?v=-ruC5A9EzzE
Something along these lines:
from math import log
def bounded_eval(expression, bits = 32):
nums = expression.split('**')
if len(nums) == 1:
val = eval(expression)
if log(val,2) > bits:
return "too large"
else:
return val
else:
base = nums[0]
power = '**'.join(nums[1:])
base = eval(base)
power = eval(power)
if power*log(base,2) > bits:
return "too large"
else:
return pow(base,power)
This does use eval() which is potentially a security risk, but if you are just calling it on arithmetical expressions that your own code generates then it isn't really a problem. Obviously, you can replace the code which returns "too large" by code which raises an error.
So I'm working on a chemistry project for fun, and I have a function that initializes a list from a text file. What I want to do s make it so the function replaces itself with a list. So here's my first attempt at it which randomly will or won't work and I don't know why:
def periodicTable():
global periodicTable
tableAtoms = open('/Users/username/Dropbox/Python/Chem Project/atoms.csv','r')
listAtoms = tableAtoms.readlines()
tableAtoms.close()
del listAtoms[0]
atoms = []
for atom in listAtoms:
atom = atom.split(',')
atoms.append(Atom(*atom))
periodicTable = atoms
It gets called in in this way:
def findAtomBySymbol(symbol):
try:
periodicTable()
except:
pass
for atom in periodicTable:
if atom.symbol == symbol:
return atom
return None
Is there a way to make this work?
Don't do that. The correct thing to do would be using a decorator that ensures the function is only executed once and caches the return value:
def cachedfunction(f):
cache = []
def deco(*args, **kwargs):
if cache:
return cache[0]
result = f(*args, **kwargs)
cache.append(result)
return result
return deco
#cachedfunction
def periodicTable():
#etc
That said, there's nothing stopping you from replacing the function itself after it has been called, so your approach should generally work. I think the reason it doesn't is because an exception is thrown before you assign the result to periodicTable and thus it never gets replaced. Try removing the try/except block or replacing the blanket except with except TypeError to see what exactly happens.
This is very bad practice.
What would be better is to have your function remember if it has already loaded the table:
def periodicTable(_table=[]):
if _table:
return _table
tableAtoms = open('/Users/username/Dropbox/Python/Chem Project/atoms.csv','r')
listAtoms = tableAtoms.readlines()
tableAtoms.close()
del listAtoms[0]
atoms = []
for atom in listAtoms:
atom = atom.split(',')
atoms.append(Atom(*atom))
_table[:] = atoms
The first two lines check to see if the table has already been loaded, and if it has it simply returns it.
Have Python iterators got a has_next method?
There's an alternative to the StopIteration by using next(iterator, default_value).
For exapmle:
>>> a = iter('hi')
>>> print next(a, None)
h
>>> print next(a, None)
i
>>> print next(a, None)
None
So you can detect for None or other pre-specified value for end of the iterator if you don't want the exception way.
No, there is no such method. The end of iteration is indicated by an exception. See the documentation.
If you really need a has-next functionality, it's easy to obtain it with a little wrapper class. For example:
class hn_wrapper(object):
def __init__(self, it):
self.it = iter(it)
self._hasnext = None
def __iter__(self): return self
def next(self):
if self._hasnext:
result = self._thenext
else:
result = next(self.it)
self._hasnext = None
return result
def hasnext(self):
if self._hasnext is None:
try: self._thenext = next(self.it)
except StopIteration: self._hasnext = False
else: self._hasnext = True
return self._hasnext
now something like
x = hn_wrapper('ciao')
while x.hasnext(): print next(x)
emits
c
i
a
o
as required.
Note that the use of next(sel.it) as a built-in requires Python 2.6 or better; if you're using an older version of Python, use self.it.next() instead (and similarly for next(x) in the example usage). [[You might reasonably think this note is redundant, since Python 2.6 has been around for over a year now -- but more often than not when I use Python 2.6 features in a response, some commenter or other feels duty-bound to point out that they are 2.6 features, thus I'm trying to forestall such comments for once;-)]]
===
For Python3, you would make the following changes:
from collections.abc import Iterator # since python 3.3 Iterator is here
class hn_wrapper(Iterator): # need to subclass Iterator rather than object
def __init__(self, it):
self.it = iter(it)
self._hasnext = None
def __iter__(self):
return self
def __next__(self): # __next__ vs next in python 2
if self._hasnext:
result = self._thenext
else:
result = next(self.it)
self._hasnext = None
return result
def hasnext(self):
if self._hasnext is None:
try:
self._thenext = next(self.it)
except StopIteration:
self._hasnext = False
else: self._hasnext = True
return self._hasnext
In addition to all the mentions of StopIteration, the Python "for" loop simply does what you want:
>>> it = iter("hello")
>>> for i in it:
... print i
...
h
e
l
l
o
Try the __length_hint__() method from any iterator object:
iter(...).__length_hint__() > 0
You can tee the iterator using, itertools.tee, and check for StopIteration on the teed iterator.
hasNext somewhat translates to the StopIteration exception, e.g.:
>>> it = iter("hello")
>>> it.next()
'h'
>>> it.next()
'e'
>>> it.next()
'l'
>>> it.next()
'l'
>>> it.next()
'o'
>>> it.next()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
StopIteration
StopIteration docs: http://docs.python.org/library/exceptions.html#exceptions.StopIteration
Some article about iterators and generator in python: http://www.ibm.com/developerworks/library/l-pycon.html
No. The most similar concept is most likely a StopIteration exception.
I believe python just has next() and according to the doc, it throws an exception is there are no more elements.
http://docs.python.org/library/stdtypes.html#iterator-types
The use case that lead me to search for this is the following
def setfrom(self,f):
"""Set from iterable f"""
fi = iter(f)
for i in range(self.n):
try:
x = next(fi)
except StopIteration:
fi = iter(f)
x = next(fi)
self.a[i] = x
where hasnext() is available, one could do
def setfrom(self,f):
"""Set from iterable f"""
fi = iter(f)
for i in range(self.n):
if not hasnext(fi):
fi = iter(f) # restart
self.a[i] = next(fi)
which to me is cleaner. Obviously you can work around issues by defining utility classes, but what then happens is you have a proliferation of twenty-odd different almost-equivalent workarounds each with their quirks, and if you wish to reuse code that uses different workarounds, you have to either have multiple near-equivalent in your single application, or go around picking through and rewriting code to use the same approach. The 'do it once and do it well' maxim fails badly.
Furthermore, the iterator itself needs to have an internal 'hasnext' check to run to see if it needs to raise an exception. This internal check is then hidden so that it needs to be tested by trying to get an item, catching the exception and running the handler if thrown. This is unnecessary hiding IMO.
Maybe it's just me, but while I like https://stackoverflow.com/users/95810/alex-martelli 's answer, I find this a bit easier to read:
from collections.abc import Iterator # since python 3.3 Iterator is here
class MyIterator(Iterator): # need to subclass Iterator rather than object
def __init__(self, it):
self._iter = iter(it)
self._sentinel = object()
self._next = next(self._iter, self._sentinel)
def __iter__(self):
return self
def __next__(self): # __next__ vs next in python 2
if not self.has_next():
next(self._iter) # raises StopIteration
val = self._next
self._next = next(self._iter, self._sentinel)
return val
def has_next(self):
return self._next is not self._sentinel
No, there is no such method. The end of iteration is indicated by a StopIteration (more on that here).
This follows the python principle EAFP (easier to ask for forgiveness than permission). A has_next method would follow the principle of LBYL (look before you leap) and contradicts this core python principle.
This interesting article explains the two concepts in more detail.
Suggested way is StopIteration.
Please see Fibonacci example from tutorialspoint
#!usr/bin/python3
import sys
def fibonacci(n): #generator function
a, b, counter = 0, 1, 0
while True:
if (counter > n):
return
yield a
a, b = b, a + b
counter += 1
f = fibonacci(5) #f is iterator object
while True:
try:
print (next(f), end=" ")
except StopIteration:
sys.exit()
It is also possible to implement a helper generator that wraps any iterator and answers question if it has next value:
Try it online!
def has_next(it):
first = True
for e in it:
if not first:
yield True, prev
else:
first = False
prev = e
if not first:
yield False, prev
for has_next_, e in has_next(range(4)):
print(has_next_, e)
Which outputs:
True 0
True 1
True 2
False 3
The main and probably only drawback of this method is that it reads ahead one more element, for most of tasks it is totally alright, but for some tasks it may be disallowed, especially if user of has_next() is not aware of this read-ahead logic and may missuse it.
Code above works for infinite iterators too.
Actually for all cases that I ever programmed such kind of has_next() was totally enough and didn't cause any problems and in fact was very helpful. You just have to be aware of its read-ahead logic.
The way has solved it based on handling the "StopIteration" execption is pretty straightforward in order to read all iterations :
end_cursor = False
while not end_cursor:
try:
print(cursor.next())
except StopIteration:
print('end loop')
end_cursor = True
except:
print('other exceptions to manage')
end_cursor = True
I think there are valid use cases for when you may want some sort of has_next functionality, in which case you should decorate an iterator with a has_next defined.
Combining concepts from the answers to this question here is my implementation of that which feels like a nice concise solution to me (python 3.9):
_EMPTY_BUF = object()
class BufferedIterator(Iterator[_T]):
def __init__(self, real_it: Iterator[_T]):
self._real_it = real_it
self._buf = next(self._real_it, _EMPTY_BUF)
def has_next(self):
return self._buf is not _EMPTY_BUF
def __next__(self) -> _T_co:
v = self._buf
self._buf = next(self._real_it, _EMPTY_BUF)
if v is _EMPTY_BUF:
raise StopIteration()
return v
The main difference is that has_next is just a boolean expression, and also handles iterators with None values.
Added this to a gist here with tests and example usage.
With 'for' one can implement his own version of 'next' avoiding exception
def my_next(it):
for x in it:
return x
return None
very interesting question, but this "hasnext" design had been put into leetcode:
https://leetcode.com/problems/iterator-for-combination/
here is my implementation:
class CombinationIterator:
def __init__(self, characters: str, combinationLength: int):
from itertools import combinations
from collections import deque
self.iter = combinations(characters, combinationLength)
self.res = deque()
def next(self) -> str:
if len(self.res) == 0:
return ''.join(next(self.iter))
else:
return ''.join(self.res.pop())
def hasNext(self) -> bool:
try:
self.res.insert(0, next(self.iter))
return True
except:
return len(self.res) > 0
The way I solved my problem is to keep the count of the number of objects iterated over, so far. I wanted to iterate over a set using calls to an instance method. Since I knew the length of the set, and the number of items counted so far, I effectively had an hasNext method.
A simple version of my code:
class Iterator:
# s is a string, say
def __init__(self, s):
self.s = set(list(s))
self.done = False
self.iter = iter(s)
self.charCount = 0
def next(self):
if self.done:
return None
self.char = next(self.iter)
self.charCount += 1
self.done = (self.charCount < len(self.s))
return self.char
def hasMore(self):
return not self.done
Of course, the example is a toy one, but you get the idea. This won't work in cases where there is no way to get the length of the iterable, like a generator etc.