We Keep Coding

Can Factory Boy fields be set to require user input?

I thought this was a trivial question but to my surprise, I couldn't find an answer.
Consider I have a factory named Foo. It has a normal field named "name" and another field named user_input for which I want to force the user to pass a value on initialization, and get an error if they don't do so. How can I achieve this?
import factory
class FooFactory(factory.Factory):
name = factory.Faker("name")
user_input = ?
# I want this to work:
foo = FooFactory(user_input="blah blah")
# and this to throw an error:
foo = FooFactory()

The solution would be to use a factory.lazy_attribute:
class FooFactory(factory.Factory):
name = factory.Faker("name")
#factory.lazy_attribute
def user_input(self):
raise ValueError("FooFactory.user_input is required")
>>> FooFactory()
Traceback (most recent call last):
File "...py", line 7, in user_input
ValueError: FooFactory.user_input is required
>>> FooFactory(user_input=42)
{"name": "John Doe", "user_input": 42}

Is it really impossible to unpickle a Python class if the original python file has been deleted?

Suppose you have the following:
file = 'hey.py'
class hey:
def __init__(self):
self.you =1
ins = hey()
temp = open("cool_class", "wb")
pickle.dump(ins, temp)
temp.close()
Now suppose you delete the file hey.py and you run the following code:
pkl_file = open("cool_class", 'rb')
obj = pickle.load(pkl_file)
pkl_file.close()
You'll get an error. I get that it's probably the case that you can't work around the problem of if you don't have the file hey.py with the class and the attributes of that class in the top level then you can't open the class with pickle. But it has to be the case that I can find out what the attributes of the serialized class are and then I can reconstruct the deleted file and open the class. I have pickles that are 2 years old and I have deleted the file that I used to construct them and I just have to find out what what the attributes of those classes are so that I can reopen these pickles
#####UPDATE
I know from the error messages that the module that originally contained the old class, let's just call it 'hey.py'. And I know the name of the class let's call it 'you'. But even after recreating the module and building a class called 'you' I still can't get the pickle to open. So I wrote this code on the hey.py module like so:
class hey:
def __init__(self):
self.hey = 1
def __setstate__(self):
self.__dict__ = ''
self.you = 1
But I get the error message: TypeError: init() takes 1 positional argument but 2 were given
#########UPDATE 2:
I Changed the code from
class hey:
to
class hey():
I then got an AttributeError but it doesn't tell me what attribute is missing. I then performed
obj= pickletools.dis(file)
And got an error on the pickletools.py file here
def _genops(data, yield_end_pos=False):
if isinstance(data, bytes_types):
data = io.BytesIO(data)
if hasattr(data, "tell"):
getpos = data.tell
else:
getpos = lambda: None
while True:
pos = getpos()
code = data.read(1)
opcode = code2op.get(code.decode("latin-1"))
if opcode is None:
if code == b"":
raise ValueError("pickle exhausted before seeing STOP")
else:
raise ValueError("at position %s, opcode %r unknown" % (
"<unknown>" if pos is None else pos,
code))
if opcode.arg is None:
arg = None
else:
arg = opcode.arg.reader(data)
if yield_end_pos:
yield opcode, arg, pos, getpos()
else:
yield opcode, arg, pos
if code == b'.':
assert opcode.name == 'STOP'
break
At this line:
code = data.read(1)
saying: AttributeError: 'str' object has no attribute 'read'
I will now try the other methods in the pickletools
########### UPDATE 3
I wanted to see what happened when I saved an object composed mostly of dictionary but some of the values in the dictionaries were classes. This is the class that was saved:
so here is the class in question:
class fss(frozenset):
def __init__(self, *args, **kwargs):
super(frozenset, self).__init__()
def __str__(self):
str1 = lbr + "{}" + rbr
return str1.format(','.join(str(x) for x in self))
Now keep in mind that the object pickled is mostly a dictionary and that class exists within the dictionary. After performing
obj= pickletools.genops(file)
I get the following output:
image
image2
I don't see how I would be able to construct the class referred to with that data if I hadn't known what the class was.
############### UPDATE #4
#AKK
Thanks for helping me out. I am able to see how your code works but my pickled file saved from 2 years ago and whose module and class have long since been deleted, I cannot open it into a bytes-like object which to me seems to be a necessity.
So the path of the file is
file ='hey.pkl'
pkl_file = open(file, 'rb')
x = MagicUnpickler(io.BytesIO(pkl_file)).load()
This returns the error:
TypeError: a bytes-like object is required, not '_io.BufferedReader'
But I thought the object was a bytes object since I opened it with open(file, 'rb')
############ UPDATE #5
Actually, I think with AKX's help I've solved the problem.
So using the code:
pkl_file = open(name, 'rb')
x = MagicUnpickler(pkl_file).load()
I then created two blank modules which once contained the classes found in the save pickle, but I did not have to put the classes on them. I was getting an error in the file pickle.py here:
def load_reduce(self):
stack = self.stack
args = stack.pop()
func = stack[-1]
try:
stack[-1] = func(*args)
except TypeError:
pass
dispatch[REDUCE[0]] = load_reduce
So after excepting that error, everything worked. I really want to thank AKX for helping me out. I have actually been trying to solve this problem for about 5 years because I use pickles far more often than most programmers. I used to not understand that if you alter a class then that ruins any pickled files saved with that class so I ran into this problem again and again. But now that I'm going back over some code which is 2 years old and it looks like some of the files were deleted, I'm going to need this code a lot in the future. So I really appreciate your help in getting this problem solved.

Well, with a bit of hacking and magic, sure, you can hydrate missing classes, but I'm not guaranteeing this will work for all pickle data you may encounter; for one, this doesn't touch the __setstate__/__reduce__ protocols, so I don't know if they work.
Given a script file (so72863050.py in my case):
import io
import pickle
import types
from logging import Formatter
# Create a couple empty classes. Could've just used `class C1`,
# but we're coming back to this syntax later.
C1 = type('C1', (), {})
C2 = type('C2', (), {})
# Create an instance or two, add some data...
inst = C1()
inst.child1 = C2()
inst.child1.magic = 42
inst.child2 = C2()
inst.child2.mystery = 'spooky'
inst.child2.log_formatter = Formatter('heyyyy %(message)s') # To prove we can unpickle regular classes still
inst.other_data = 'hello'
inst.some_dict = {'a': 1, 'b': 2}
# Pickle the data!
pickle_bytes = pickle.dumps(inst)
# Let's erase our memory of these two classes:
del C1
del C2
try:
print(pickle.loads(pickle_bytes))
except Exception as exc:
pass # Can't get attribute 'C1' on <module '__main__'> – yep, it certainly isn't there!
we now have successfully created some pickle data that we can't load anymore, since we forgot about those two classes. Now, since the unpickling mechanism is customizable, we can derive a magic unpickler, that in the face of certain defeat (or at least an AttributeError), synthesizes a simple class from thin air:
# Could derive from Unpickler, but that may be a C class, so our tracebacks would be less helpful
class MagicUnpickler(pickle._Unpickler):
def __init__(self, fp):
super().__init__(fp)
self._magic_classes = {}
def find_class(self, module, name):
try:
return super().find_class(module, name)
except AttributeError:
return self._create_magic_class(module, name)
def _create_magic_class(self, module, name):
cache_key = (module, name)
if cache_key not in self._magic_classes:
cls = type(f'<<Emulated Class {module}:{name}>>', (types.SimpleNamespace,), {})
self._magic_classes[cache_key] = cls
return self._magic_classes[cache_key]
Now, when we run that magic unpickler against a stream from the aforebuilt pickle_bytes that plain ol' pickle.loads() couldn't load...
x = MagicUnpickler(io.BytesIO(pickle_bytes)).load()
print(x)
print(x.child1.magic)
print(x.child2.mystery)
print(x.child2.log_formatter._style._fmt)
prints out
<<Emulated Class __main__:C1>>(child1=<<Emulated Class __main__:C2>>(magic=42), child2=<<Emulated Class __main__:C2>>(mystery='spooky'), other_data='hello', some_dict={'a': 1, 'b': 2})
42
spooky
heyyyy %(message)s
Hey, magic!

The error in function load_reduce(self) can be re-created by:
class Y(set):
pass
pickle_bytes = io.BytesIO(pickle.dumps(Y([2, 3, 4, 5])))
del Y
print(MagicUnpickler(pickle_bytes).load())
AKX's answer do not solve cases when the class inherit from base classes as set, dict, list,...

SQLAlchemy one(): No Results Found

I keep getting a Traceback complaining about the system not finding a result. This should not be the case given that the object is actually going through (I know this as I have a print command to confirm what is going through the query). I believe the issue is found in the method of create_category_rating(..).
When I use .one(), it throws the complaint I have below. When I use .all, it returns AttributeError: 'list' object has no attribute '_sa_instance_state' (some one else had this issue). The docs at SQLAlchemy does a good job defining these methods, but for some reason my code crashes.
The issue: What is causing the program to break if the object is going through the retrieve method? There is a bug somewhere and I have been struggling with squashing it.
I am using SQLAlchemy and python2.7:
Traceback:
ERROR: notssdb.test.test.test1
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
self.test(*self.arg)
File "/Users/ack/code/venv/NotssDB/notssdb/test/test.py", line 111, in test1
api.create_category_rating(2, 'Decision-Making', 'baseball', 'Becoming a Leader')
File "/Users/ack/code/venv/NotssDB/notssdb/api/convenience.py", line 41, in create_category_rating
assessment_results = self.retrieve_assessment_results(owner, assessment)
File "/Users/ack/code/venv/NotssDB/notssdb/api/object.py", line 324, in retrieve_assessment_results
filter(Assessment_Results.owner == owner).one()
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2478, in one
raise orm_exc.NoResultFound("No row was found for one()")
NoResultFound: No row was found for one()
# output from Traceback
Decision-Making
baseball
Becoming a Leader
Becoming a Leader
test1 <Assessment(name='Becoming a Leader', text='better decisions')>
convenience.py
(create method):
def create_category_rating(self, category_rating_int, category_name, username, name):
category = self.retrieve_category(category_name)
owner = self.retrieve_user(username) # added
assessment = self.retrieve_assessment(name) #added
assessment_results = self.retrieve_assessment_results(owner, assessment)
return super(ConvenienceAPI, self).create_category_rating(category_rating_int, category, assessment_results)
object.py
(retrieve method):
def retrieve_assessment_results(self, *args):
id, assessment, owner = None, None, None
if len(args) == 1:
id, = args[0]
elif len(args) == 2:
assessment, owner = args
else:
raise ValueError('Value being passed is an object')
if id is not None:
return self.session.query(Assessment_Results).\
filter(Assessment_Results.id == id).one()
elif owner is not None:
print 'test1', owner
return self.session.query(Assessment_Results).\
filter(Assessment_Results.owner == owner).one()
elif assessment is not None:
print 'test2', assessment
return self.session.query(Assessment_Results).\
filter(Assessment_Results.assessment == assessment).one()

You have the following:
self.retrieve_assessment_results(owner, assessment)
...but the arguments for "retrieve_assessment_results" is the following (with my comment added)...
def retrieve_assessment_results(self, *args):
id, assessment, owner = None, None, None
if len(args) == 1:
id, = args[0]
elif len(args) == 2:
assessment, owner = args # Note this line!
else:
raise ValueError('Value being passed is an object')
You pass owner, then assessment. But if you detect two objects, you will read assessment, then owner. In other words, you are calling the method with one order of objects, and retrieve assuming the reverse order. You can even see this occur in the debug line:
print 'test1', owner
You'd expect this to print something like test1 <User(...)>, but instead it printed...
test1 <Assessment(name='Becoming a Leader', text='better decisions')>
Also, consider using Python keyword arguments instead:
def retrieve_assessment_results(self, id=None, assessment=None, owner=None):
...
Then your call becomes less ambiguous:
self.retrieve_assessment_results(owner=owner, assessment=assessment)

one() will throw an exception if the result set is empty. Use first() and you will get the first result or Python None if no results are present.
Ditto about all() because the question does not contain relevant traceback and code.

Faking a traceback in Python

I'm writing a test runner. I have an object that can catch and store exceptions, which will be formatted as a string later as part of the test failure report. I'm trying to unit-test the procedure that formats the exception.
In my test setup, I don't want to actually throw an exception for my object to catch, mainly because it means that the traceback won't be predictable. (If the file changes length, the line numbers in the traceback will change.)
How can I attach a fake traceback to an exception, so that I can make assertions about the way it's formatted? Is this even possible? I'm using Python 3.3.
Simplified example:
class ExceptionCatcher(object):
def __init__(self, function_to_try):
self.f = function_to_try
self.exception = None
def try_run(self):
try:
self.f()
except Exception as e:
self.exception = e
def format_exception_catcher(catcher):
pass
# No implementation yet - I'm doing TDD.
# This'll probably use the 'traceback' module to stringify catcher.exception
class TestFormattingExceptions(unittest.TestCase):
def test_formatting(self):
catcher = ExceptionCatcher(None)
catcher.exception = ValueError("Oh no")
# do something to catcher.exception so that it has a traceback?
output_str = format_exception_catcher(catcher)
self.assertEquals(output_str,
"""Traceback (most recent call last):
File "nonexistent_file.py", line 100, in nonexistent_function
raise ValueError("Oh no")
ValueError: Oh no
""")

Reading the source of traceback.py pointed me in the right direction. Here's my hacky solution, which involves faking the frame and code objects which the traceback would normally hold references to.
import traceback
class FakeCode(object):
def __init__(self, co_filename, co_name):
self.co_filename = co_filename
self.co_name = co_name
class FakeFrame(object):
def __init__(self, f_code, f_globals):
self.f_code = f_code
self.f_globals = f_globals
class FakeTraceback(object):
def __init__(self, frames, line_nums):
if len(frames) != len(line_nums):
raise ValueError("Ya messed up!")
self._frames = frames
self._line_nums = line_nums
self.tb_frame = frames[0]
self.tb_lineno = line_nums[0]
#property
def tb_next(self):
if len(self._frames) > 1:
return FakeTraceback(self._frames[1:], self._line_nums[1:])
class FakeException(Exception):
def __init__(self, *args, **kwargs):
self._tb = None
super().__init__(*args, **kwargs)
#property
def __traceback__(self):
return self._tb
#__traceback__.setter
def __traceback__(self, value):
self._tb = value
def with_traceback(self, value):
self._tb = value
return self
code1 = FakeCode("made_up_filename.py", "non_existent_function")
code2 = FakeCode("another_non_existent_file.py", "another_non_existent_method")
frame1 = FakeFrame(code1, {})
frame2 = FakeFrame(code2, {})
tb = FakeTraceback([frame1, frame2], [1,3])
exc = FakeException("yo").with_traceback(tb)
print(''.join(traceback.format_exception(FakeException, exc, tb)))
# Traceback (most recent call last):
# File "made_up_filename.py", line 1, in non_existent_function
# File "another_non_existent_file.py", line 3, in another_non_existent_method
# FakeException: yo
Thanks to #User for providing FakeException, which is necessary because real exceptions type-check the argument to with_traceback().
This version does have a few limitations:
It doesn't print the lines of code for each stack frame, as a real
traceback would, because format_exception goes off to look for the
real file that the code came from (which doesn't exist in our case).
If you want to make this work, you need to insert fake data into
linecache's
cache (because traceback uses linecache to get hold of the source
code), per #User's answer
below.
You also can't actually raise exc and expect the fake traceback
to survive.
More generally, if you have client code that traverses tracebacks in
a different manner than traceback does (such as much of the inspect
module), these fakes probably won't work. You'd need to add whatever
extra attributes the client code expects.
These limitations are fine for my purposes - I'm just using it as a test double for code that calls traceback - but if you want to do more involved traceback manipulation, it looks like you might have to go down to the C level.

EDIT2:
That is the code of linecache.. I will comment on it.
def updatecache(filename, module_globals=None): # module_globals is a dict
# ...
if module_globals and '__loader__' in module_globals:
name = module_globals.get('__name__')
loader = module_globals['__loader__']
# module_globals = dict(__name__ = 'somename', __loader__ = loader)
get_source = getattr(loader, 'get_source', None)
# loader must have a 'get_source' function that returns the source
if name and get_source:
try:
data = get_source(name)
except (ImportError, IOError):
pass
else:
if data is None:
# No luck, the PEP302 loader cannot find the source
# for this module.
return []
cache[filename] = (
len(data), None,
[line+'\n' for line in data.splitlines()], fullname
)
return cache[filename][2]
That means before you testrun just do:
class Loader:
def get_source(self):
return 'source of the module'
import linecache
linecache.updatecache(filename, dict(__name__ = 'modulename without <> around',
__loader__ = Loader()))
and 'source of the module' is the source of the module you test.
EDIT1:
My solution so far:
class MyExeption(Exception):
_traceback = None
#property
def __traceback__(self):
return self._traceback
#__traceback__.setter
def __traceback__(self, value):
self._traceback = value
def with_traceback(self, tb_or_none):
self.__traceback__ = tb_or_none
return self
Now you can set the custom tracebacks of the exception:
e = MyExeption().with_traceback(1)
What you usually do if you reraise an exception:
raise e.with_traceback(fake_tb)
All exception prints walk through this function:
import traceback
traceback.print_exception(_type, _error, _traceback)
Hope it helps somehow.

You should be able to simply raise whatever fake exception you want where you want it in your test runs. The python exception docs suggest you create a class and raise that as your exception. It's section 8.5 of the docs.
http://docs.python.org/2/tutorial/errors.html
Should be pretty straightforward once you've got the class created.

Does python allow calling of an instance variable name from an instance method?

I want to know if there is a way in python to call the name of an instance variable? For example, if I define a class
>>>class A(object):
... def get_instance_name(self):
... return # The name of the instance variable
>>>obj = A()
>>>obj.get_instance_name()
obj
>>>blah = A()
>>>blah.get_instance_name()
blah

Raise an exception. Not only is it the appropriate way to signal an error, it's also more useful for debugging. The traceback includes the line which did the method call but also additional lines, line numbers, function names, etc. which are more useful for debugging than just a variable name. Example:
class A:
def do(self, x):
if x < 0:
raise ValueError("Negative x")
def wrong(a, x):
a.do(-x)
wrong(A(), 1)
This gives a traceback similar to this, if the exception isn't caught:
Traceback (most recent call last):
File "...", line 1, in <module>
wrong(A(), 1)
File "...", line 7, in wrong
a.do(-x)
File "...", line 4, in do
raise ValueError("Negative x")
ValueError: Negative x
You can also use the traceback module to get this information programmatically, even without an exception (print_stack and friends).

globals() return a dictionary that represents the namespace of the module (the namespace is not this dictionary, this latter only represents it)
class A(object):
def get_instance_name(self):
for name,ob in globals().iteritems():
if ob is self:
return name
obj = A()
print obj.get_instance_name()
blah = A()
print blah.get_instance_name()
tu = (obj,blah)
print [x.get_instance_name() for x in tu]
result
obj
blah
['obj', 'blah']
.
EDIT
Taking account of the remarks, I wrote this new code:
class A(object):
def rondo(self,nameinst,namespace,li,s,seen):
for namea,a in namespace.iteritems():
if a is self:
li.append(nameinst+s+namea)
if namea=='__builtins__':
#this condition prevents the execution to go
# in the following section elif, so that self
# isn't searched among the cascading attributes
# of the builtin objects and the attributes.
# This is to avoid to explore all the big tree
# of builtin objects and their cascading attributes.
# It supposes that every builtin object has not
# received the instance, of which the names are
# searched, as a new attribute. This makes sense.
for bn,b in __builtins__.__dict__.iteritems():
if b is self:
li.append(nameinst+'-'+b)
elif hasattr(a,'__dict__') \
and not any(n+s+namea in seen for n in seen)\
and not any(n+s+namea in li for n in li):
seen.append(nameinst+s+namea)
self.rondo(nameinst+s+namea,a.__dict__,li,'.')
else:
seen.append(nameinst+s+namea)
def get_instance_name(self):
li = []
seen = []
self.rondo('',globals(),li,'')
return li if li else None
With the following
bumbum = A()
blah = A()
print "bumbum's names:\n",bumbum.get_instance_name()
print "\nmap(lambda y:y.get_instance_name(), (bumbum,blah) :\n",map(lambda y:y.get_instance_name(), (bumbum,blah))
print "\n[y.get_instance_name() for y in (bumbum,blah)] :\n",[y.get_instance_name() for y in (bumbum,blah)]
the result is
bumbum's names:
['bumbum']
map(lambda y:y.get_instance_name(), (bumbum,blah) :
[['bumbum'], ['blah']]
[y.get_instance_name() for y in (bumbum,blah)] :
[['bumbum', 'y'], ['blah', 'y']]
The second list comprehension shows that the function get_instance_name() must be used with care. In the list comp, identifier y is assigned in turn to every element of (bumbum,blah) then the finction finds it out as a name of the instance !
.
Now, a more complex situation:
ahah = A() # ahah : first name for this instance
class B(object):
pass
bobo = B()
bobo.x = ahah # bobo.x : second name for ahah
jupiter = bobo.x # jupiter : third name for ahah
class C(object):
def __init__(self):
self.azerty = jupiter # fourth name for ahah
ccc = C()
kkk = ccc.azerty # kkk : fifth name for ahah
bobo.x.inxnum = 1005
bobo.x.inxwhat = kkk # bobo.x.inxwhat : fifth name for ahah
# Since bobo.x is instance ahah, this instruction also
# creates attribute inxwhat in ahah instance's __dict__ .
# Consequently, instance ahah having already 5 names,
# this instruction adds 5 additional names, each one
# ending with .inxwhat
# By the way, this kkk being ahah itself, it results that ahah
# is the value of its own attribute inxwhat.
print ahah.get_instance_name()
result
['bobo.x', 'bobo.x.inxwhat',
'ahah', 'ahah.inxwhat',
'jupiter', 'jupiter.inxwhat',
'kkk', 'kkk.inxwhat',
'ccc.azerty', 'ccc.azerty.inxwhat']
I concur to judge this solution a little heavy and that if a coder thinks he needs such a heavy function, it is probably because the algorithm isn't optimal. But I find interesting to see that it's possible to do this in Python though it doesn't seem evident.
I say heavy, not hacky, I don't find it's hacky, by the way.

No, you can't. Objects can have any number of names, so the question doesn't even make sense. Consider:
a1 = a2 = a3 = A()
What is the name of the instance of A()?