I'm not really that knowledgeable about thread-safe vs non-thread-safe operations but am wondering if a problem I'm seeing might be because of that.
In my modules directory I've created a module that has a class defined.
Then in db.py I have an _after_insert trigger like so:
db.workorder._after_insert.append(lambda s,f: workorderAfterInsert(s,f))
In my _after_insert trigger I'm instantiating the class from my module like this:
import workorder.sequencer as sequencer
workorderId = id
wo = db.workorder(workorderId)
sequencer = sequencer.Sequencer(workorder_id=workorderId, db=db)
sequencer.build_bom()
sequencer.sequence()
sequencer.save_sequenced_workorder()
db.commit()
I'm not sure how to describe it, but I'm seeing random errors popping up in the execution of sequencer.sequence(). My only thought at this point is that there are thread-safe or concurrency issues going on.
I'd really appreciate it if someone could tell me whether or not this is safe (or wise). Any input would be appreciated.
-Jim
My problem was that I had my variables defined as class variables instead of instance variables. Changed that up and now it all works.
Related
I'd like to dynamically create a module from a dictionary, and I'm wondering if adding an element to sys.modules is really the best way to do this. EG
context = { a: 1, b: 2 }
import types
test_context_module = types.ModuleType('TestContext', 'Module created to provide a context for tests')
test_context_module.__dict__.update(context)
import sys
sys.modules['TestContext'] = test_context_module
My immediate goal in this regard is to be able to provide a context for timing test execution:
import timeit
timeit.Timer('a + b', 'from TestContext import *')
It seems that there are other ways to do this, since the Timer constructor takes objects as well as strings. I'm still interested in learning how to do this though, since a) it has other potential applications; and b) I'm not sure exactly how to use objects with the Timer constructor; doing so may prove to be less appropriate than this approach in some circumstances.
EDITS/REVELATIONS/PHOOEYS/EUREKA:
I've realized that the example code relating to running timing tests won't actually work, because import * only works at the module level, and the context in which that statement is executed is that of a function in the testit module. In other words, the globals dictionary used when executing that code is that of __main__, since that's where I was when I wrote the code in the interactive shell. So that rationale for figuring this out is a bit botched, but it's still a valid question.
I've discovered that the code run in the first set of examples has the undesirable effect that the namespace in which the newly created module's code executes is that of the module in which it was declared, not its own module. This is like way weird, and could lead to all sorts of unexpected rattlesnakeic sketchiness. So I'm pretty sure that this is not how this sort of thing is meant to be done, if it is in fact something that the Guido doth shine upon.
The similar-but-subtly-different case of dynamically loading a module from a file that is not in python's include path is quite easily accomplished using imp.load_source('NewModuleName', 'path/to/module/module_to_load.py'). This does load the module into sys.modules. However this doesn't really answer my question, because really, what if you're running python on an embedded platform with no filesystem?
I'm battling a considerable case of information overload at the moment, so I could be mistaken, but there doesn't seem to be anything in the imp module that's capable of this.
But the question, essentially, at this point is how to set the global (ie module) context for an object. Maybe I should ask that more specifically? And at a larger scope, how to get Python to do this while shoehorning objects into a given module?
Hmm, well one thing I can tell you is that the timeit function actually executes its code using the module's global variables. So in your example, you could write
import timeit
timeit.a = 1
timeit.b = 2
timeit.Timer('a + b').timeit()
and it would work. But that doesn't address your more general problem of defining a module dynamically.
Regarding the module definition problem, it's definitely possible and I think you've stumbled on to pretty much the best way to do it. For reference, the gist of what goes on when Python imports a module is basically the following:
module = imp.new_module(name)
execfile(file, module.__dict__)
That's kind of the same thing you do, except that you load the contents of the module from an existing dictionary instead of a file. (I don't know of any difference between types.ModuleType and imp.new_module other than the docstring, so you can probably use them interchangeably) What you're doing is somewhat akin to writing your own importer, and when you do that, you can certainly expect to mess with sys.modules.
As an aside, even if your import * thing was legal within a function, you might still have problems because oddly enough, the statement you pass to the Timer doesn't seem to recognize its own local variables. I invoked a bit of Python voodoo by the name of extract_context() (it's a function I wrote) to set a and b at the local scope and ran
print timeit.Timer('print locals(); a + b', 'sys.modules["__main__"].extract_context()').timeit()
Sure enough, the printout of locals() included a and b:
{'a': 1, 'b': 2, '_timer': <built-in function time>, '_it': repeat(None, 999999), '_t0': 1277378305.3572791, '_i': None}
but it still complained NameError: global name 'a' is not defined. Weird.
After checking at least 20 stackoverflow posts, I decided I'd share my problem here, maybe somebody can help me out.
It's quite simple, I am trying to mock a Database (to connect to another DB in the test), so I use unittest.mock.patch
self.mocks = [
"api.database.database.db",
"api.utils.checker.Checker.check_db_reachability"
]
self.mocks = [patch(mocked_func, return_value=self.db) for mocked_func in self.mocks]
After running (Console Output I printed after starting the mocks) the db import does not get mocked, however the class method I mocked the same way, got changed into a magic mock, if called.
dbtest_1 | DEBUG:test.postgres.test_crud:Changed DB:
dbtest_1 | DEBUG:test.postgres.test_crud:<databases.core.Database object at 0x7f7465e6c040>
dbtest_1 | DEBUG:test.postgres.test_crud:Changed Checker:
dbtest_1 | DEBUG:test.postgres.test_crud:<MagicMock name='check_db_reachability' id='140137474713264'>
Does somebody know what I'm doing wrong? The expected output would be, that the databases.core.Database object also gets mocked.
Found the answer myself - sharing it here, might help people who face the same out.
The problem was:
I was importing a variable / function from another module (file), which wasn't contained in a class or something similar
The problem was, I was patching the wrong location (the location where the object was defined (e.g. db = Database()) - however: You have to patch the location, where the object / variable is being imported!
Read more about the topic here: https://docs.python.org/3/library/unittest.mock.html#where-to-patch
I hope I can save you some time, I wish I found out sooner. Peace!
Introduction
Pydev is a great eclipse plugin that let us write python code easily.
It can even give autocompletion suggestion when I do this:
from package.module import Random_Class
x = Random_Class()
x. # the autocompletion will be popped up,
# showing every method & attribute inside Random_Class
That is great !!!
The Problem (And My Question)
However, when I don't use explicit import, and use __import__ for example, I can't have the same autocompletion effect.
import_location = ".".join(('package', 'module'))
__import__(import_location, globals(), locals(), ['*'])
My_Class = getattr(sys.modules[import_location], 'Random_Class')
x = My_Class()
x. # I expect autocompletion, but nothing happened
Question: is there any way (in pydev or any IDE) to make the second one also
show autocompletion?
Why do I need to do this?
Well, I make a simple MVC framework, and I want to provide something like load_model, load_controller, and load_view which is still work with autocompletion (or at least possible to work)
So, instead of leave users do this (although I don't forbid them to do so):
from applications.the_application.models.the_model import The_Model
x = The_Model()
x. # autocompletion works
I want to let users do this:
x = load_model('the_application', 'the_model')()
x. # autocompletion still works
The "applications" part is actually configurable by another script, and I don't want users to change all of their importing model/controller part everytime they change the configuration. Plus, I think load_model, load_controller, and load_view make MVC pattern shown more obvious.
Unexpected Answer
I know some tricks such as doing this (as what people do with
web2py):
import_location = ".".join(('package', 'module'))
__import__(import_location, globals(), locals(), ['*'])
My_Class = getattr(sys.modules[import_location], 'Random_Class')
x = My_Class()
if 0:
from package.module import Random_Class
x = Random_Class()
x. # Now autocompletion is going to work
and I don't expect to do this, since it will only add unnecessary
extra work.
I don't expect any don't try to be clever comments. I have enough of them
I don't expect dynamic import is evil comments. I'm not a purist.
I don't expect any just use django, or pylons, or whatever comments. Such as comments even unrelated to my question.
I have done this before. This may be slightly different from your intended method, so let me know if it doesn't apply.
I dynamically import different modules that all subclass a master class, using similar code to your example. Because the subclassing module already imports the master, I don't need to import it in the main module.
To get highlighting, the solution was to import the master class into the main module first, even though it wasn't used directly. In my case it was a good fallback if the particular subclass didn't exist, but that's an implementation detail.
This only works if your classes all inherit from one parent.
Not really an answer to my own question. However, I can change the approach. So, instead of provide "load_model()", I can use relative import. Something like this:
from ..models.my_model import Model_Class as Great_Model
m = Great_Model()
I am writing a Python wrapper for a C library using the cffi.
The C library has to be initialized and shut down. Also, the cffi needs some place to save the state returned from ffi.dlopen().
I can see two paths here:
Either I wrap this whole stateful business in a class like this
class wrapper(object):
def __init__(self):
self.c = ffi.dlopen("mylibrary")
self.c.initialize()
def __del__(self):
self.c.terminate()
Or I provide two global functions that hide the state in a global variable
def initialize():
global __library
__library = ffi.dlopen("mylibrary")
__library.initialize()
def terminate():
__library.terminate()
del __library
The first path is somewhat cumbersome in that it requires the user to always create an object that really serves no other purpose other than managing the library state. On the other hand, it makes sure that terminate() is actually called every time.
The second path seems to result in a somewhat easier API. However, it exposes some hidden global state, which might be a bad thing. Also, if the user forgets to call terminate(), the C library is not unloaded correctly (which is not a big problem on the C side).
Which one of these paths would be more pythonic?
Exposing a wrapper object only makes sense in python if the library actually supports something like multiple instances in one application. If it doesn't support that or it's not really relevant go for kindall's suggestion and just initialize the library when imported and add an atexit handler for cleanup.
Adding wrappers around a stateless api or even an api without support for keeping different sets of state is not really pythonic and would raise expectations that different instances have some kind of isolation.
Example code:
import atexit
# Normal library initialization
__library = ffi.dlopen("mylibrary")
__library.initialize()
# Private library cleanup function
def __terminate():
__library.terminate()
# register function to be called on clean interpreter termination
atexit.register(__terminate)
For more details about atexit this question has some more details, as has the python documentation of course.
I'm kinda new to Python, so i'm still lost in the whole namespace thing.
I've created a package, with the init file in it and also a classname.py file, with the class, obviously.
For instance:
from parachute import container, emitter
I tried to instance the class Container directly, but it gave me an error, so i had to instance it as container.Container(). How can i avoid doing this?
Basically, what i want to do is to import a class from a package and avoid typing the package name and/or the file name.
Thanks in advance, and please let me know if the question isn't clear enough.
UPDATE
The structure i have is:
- parachute
-- init.py
-- container.py
Serves as a controller, i'd say, instancing, calling and glueing all the other parts together.
-- sparkles.py
Has two classes: Sparkle and Sparkles. Sparkle is a single element, with only one property so far, and Sparkles serves as a collection. Sparkles() belongs to the Emitter, and Sparkle() belongs to Sparkles().
-- emitter.py
Emitter could be seen as the user entity. It has a name and an uid, it belongs to a Container and the Container belongs to it.
Now, from outside the package i'm calling Container and passing some arguments, and the Container instances and distributes the arguments as it needs.
I have the impression that this isn't the best way to do what i need to do, which is: Create a collection of sparkles, owned by the emitter.
Don't put the class in it's own file. Put Container and Emitter directly in parachute.py.
You can then do
from parachute import Container, Emitter
or
import parachute
container = parachute.Container()
This essentially boils down to "Python isn't Java so for best results, don't treat it like it is" ;)
from module import Class
classInst = Class()
This will work if your class is in module.py