I'm designing an add-on for blender, that changes the location of certain vertices of an object. Every object in blender has a matrix_world atribute, that holds a matrix that transposes the coordinates of the vertices from the object to the world frame.
print(object.matrix_world) # unit matrix (as expected)
object.location += mathutils.Vector((5,0,0))
object.rotation_quaternion *= mathutils.Quaternion((0.0, 1.0, 0.0), math.radians(45))
print(object.matrix_world) # Also unit matrix!?!
The above snippet shows that after the translation, you still have the same matrix_world. How can I force blender to recalculate the matrix_world?
You should call Scene.update after changing those values, otherwise Blender won't recalculate matrix_world until it's needed [somewhere else]. The reason, according to the "Gotcha's" section in the API docs, is that this re-calc is an expensive operation, so it's not done right away:
Sometimes you want to modify values from python and immediately access the updated values, eg:
Once changing the objects bpy.types.Object.location you may want to access its transformation right after from bpy.types.Object.matrix_world, but this doesn’t work as you might expect.
Consider the calculations that might go into working out the objects final transformation, this includes:
animation function curves.
drivers and their pythons expressions.
constraints
parent objects and all of their f-curves, constraints etc.
To avoid expensive recalculations every time a property is modified, Blender defers making the actual calculations until they are needed.
However, while the script runs you may want to access the updated values.
This can be done by calling bpy.types.Scene.update after modifying values which recalculates all data that is tagged to be updated.
Calls to bpy.context.scene.update() can become expensive when called within a loop.
If your objects have no complex constraints (e.g. plain or parented), the following can be used to recompute the world matrix after changing object's .location, .rotation_euler\quaternion, or .scale.
def update_matrices(obj):
if obj.parent is None:
obj.matrix_world = obj.matrix_basis
else:
obj.matrix_world = obj.parent.matrix_world * \
obj.matrix_parent_inverse * \
obj.matrix_basis
Some notes:
Immediately after setting object location/rotation/scale the object's matrix_basis is updated
But matrix_local (when parented) and matrix_world are only updated during scene.update()
When matrix_world is manually recomputed (using the code above), matrix_local is recomputed as well
If the object is parented, then its world matrix depends on the parent's world matrix as well as the parent's inverse matrix at the time of creation of the parenting relationship.
I needed to do this too but needed this value to be updated whilst I imported a large scene with tens of thousands of objects.
Calling 'scene.update()' became exponentially slower, so I needed to find a way to do this without calling that function.
This is what I came up with:
def BuildScaleMatrix(s):
return Matrix.Scale(s[0],4,(1,0,0)) * Matrix.Scale(s[1],4,(0,1,0)) * Matrix.Scale(s[2],4,(0,0,1))
def BuildRotationMatrixXYZ(r):
return Matrix.Rotation(r[2],4,'Z') * Matrix.Rotation(r[1],4,'Y') * Matrix.Rotation(r[0],4,'X')
def BuildMatrix(t,r,s):
return Matrix.Translation(t) * BuildRotationMatrixXYZ(r) * BuildScaleMatrix(s)
def UpdateObjectTransform(ob):
ob.matrix_world = BuildMatrix(ob.location, ob.rotation_euler, ob.scale)
This isn't most efficient way to build a matrix (if you know of a better way in blender, please add) and this only works for XYZ order transforms but this avoids exponential slow downs when dealing with large data sets.
Accessing Object.matrix_world is causing it to "freeze" even though you don't do anything to it, eg:
m = C.active_object.matrix_world
causes the matrix to be stuck. Whenever you want to access the matrix use
Object.matrix_world.copy()
only if you want to write the matrix, use
C.active_object.matrix_world = m
Related
In this video, Justin has written the following in the compute function of his component.
def compute(self, inputs, outputs):
self.alp_sc = 0.91
T0 = 28. #reference temperature
eff0 = .285 #efficiency at ref temp
T1 = -150.
eff1 = 0.335
delta_T = inputs['T'] - T0
self.slope = (eff1 - eff0) / (T1 - T0)
outputs['eta'] = (eff0 + self.slope * delta_T)/self.alp_sc
Why does he choose to use self.slope and self.alp_sc instead of just plain variables? Is it something important for openMDAO/vectorised components, or just an arbitrary choice?
The attributes (e.g. self.alp_sc) are constants that were never intended to change for that sample component. They do not need to be input variables, because they will never be connected to.
It is considered a best practice to define partial derivatives for all outputs with respect to all inputs in any component. Therefore, by making these values attributes, we can respect the best practice and avoid declaring partials for things we know will never change.
If you continue to watch the video, self.slope and self.alp_sc are also used in the compute_partials method. self.alp_sc is constant, so I think it could have been declared in the initialize method of the class (if it is a parameter, that you want to potentially change for different instances), or outside of the class (if it is really a constant). self.slope in this case is also a constant, so the same applies. But imagine, the slope would depend on the input of your component, and you would need to recalculate it in each iteration (and let's say it is also very computationally expensive, which in the example is clearly not). In this case you could save some computations by storing the value in a class attribute (self.slope) and just reusing it in the derivative computation.
One thing, that must to be ensured is, that in each iteration compute is called before compute_partials (otherwise you could end up using an obsolete value from the previous iteration in the derivative calculation), but I think that is always true in the current OpenMDAO 3.0.
It is quite common that you need to calculate the same quantities for the function and for the derivatives. Storing it in an attribute is one way to do it (less computation), or calling the same function within compute and compute_partials (twice as much computation, less memory) is another way.
From the snippet of code it looks like def compute(self, inputs, outputs) is a method inside of an object/class. This can be seen from self being conatined in its initialisation.
self.slope and 'self.alp_scwill be variables that are declared within the scope of the object that thecompute()` method is part of.
See classes in Python for more on this.
I recently found a very strange pattern in the "Pipeline" API from Quantopian/Zipline: they have a CustomFactor class, in which you will find a compute() method to be overriden when implementing your own Factor model.
The signature of compute() is: def compute(self, today, assets, out, *inputs), with the following comment for parameter "out":
Output array of the same shape as assets. compute should write its desired return values into out.
When I asked why the function could not simply return an output array instead of writing into an input parameter, I received the following answer:
"If the API required that the output array be returned by compute(), we'd end up doing a copy of the array into the actual output buffer which means an extra copy would get made unnecessarily."
I fail to understand why they end up doing so... Obviously in Python there is no issues about passing by value and there is no risk of unnecessarily copying data. This is really painful because this is the kind of implementations they are recommending people to code:
def compute(self, today, assets, out, data):
out[:] = data[-1]
So my question is, why could it not simply be:
def compute(self, today, assets, data):
return data[-1]
(I designed and implemented the API in question here.)
You're right that Python objects aren't copied when passed into and out of functions. The reason there's a difference between returning a row out of your CustomFactor and writing values into a provided array has to do with copies that would be made in the code that's calling your CustomFactor compute method.
When the CustomFactor API was originally designed, the code that calls your compute method looked roughly like this:
def _compute(self, windows, dates, assets):
# `windows` here is list of iterators yielding 2D slices of
# the user's requested inputs
# `dates` and `assets` are row/column labels for the final output.
# Allocate a (dates x assets) output array.
# Each invocation of the user's `compute` function
# corresponds to one row of output.
output = allocate_output()
for i in range(len(dates)):
# Grab the next set of input arrays.
inputs = [next(w) for w in windows]
# Call the user's compute, which is responsible for writing
# values into `out`.
self.compute(
dates[i],
assets,
# This index is a non-copying operation.
# It creates a view into row `i` of `output`.
output[i],
*inputs # Unpack all the inputs.
)
return output
The basic idea here is that we've pre-fetched a sizeable amount of data, and we're now going to loop over windows into that data, call the user's compute function on the data, and write the result into a pre-allocated output array, which is then passed along to further transformations.
No matter what we do, we have to pay the cost of at least one copy to get the result of the user's compute function into the output array.
The most obvious API, as you point out, is to have the user simply return the output row, in which case the calling code would look like:
# Get the result row from the user.
result_row = self.compute(dates[i], assets, *inputs)
# Copy the user's result into our output buffer.
output[i] = result_row
If that were the API, then we're locked into paying at least the following costs for each invocation of the user's compute
Allocating the ~64000 byte array that the user will return.
A copy of the user's computed data into the user's output array.
A copy from the user's output array into our own, larger array.
With the existing API, we avoid costs (1) and (3).
With all that said, we've since made changes to how CustomFactors work that make some of the above optimizations less useful. In particular, we now only pass data to compute for assets that weren't masked out on that day, which requires a partial copy of the output array before and after the call to compute.
There are still some design reasons to prefer the existing API though. In particular, leaving the engine in control of the output allocation makes it easier for us to do things like pass recarrays for multi-output factors.
I have a nested dictionary containing a bunch of data on a number of different objects (where I mean object in the non-programming sense of the word). The format of the dictionary is allData[i][someDataType], where i is a number designation of the object that I have data on, and someDataType is a specific data array associated with the object in question.
Now, I have a function that I have defined that requires a particular data array for a calculation to be performed for each object. The data array is called cleanFDF. So I feed this to my function, along with a bunch of other things it requires to work. I call it like this:
rm.analyze4complexity(allData[i]['cleanFDF'], other data, other data, other data)
Inside the function itself, I straight away re-assign the cleanFDF data to another variable name, namely clFDF. I.e. The end result is:
clFDF = allData[i]['cleanFDF']
I then have to zero out all of the data that lies below a certain threshold, as such:
clFDF[ clFDF < threshold ] = 0
OK - the function works as it is supposed to. But now when I try to plot the original cleanFDF data back in the main script, the entries that got zeroed out in clFDF are also zeroed out in allData[i]['cleanFDF']. WTF? Obviously something is happening here that I do not understand.
To make matters even weirder (from my point of view), I've tried to do a bodgy kludge to get around this by 'saving' the array to another variable before calling the function. I.e. I do
saveFDF = allData[i]['cleanFDF']
then run the function, then update the cleanFDF entry with the 'saved' data:
allData[i].update( {'cleanFDF':saveFDF} )
but somehow, simply by performing clFDF[ clFDF < threshold ] = 0 within the function modifies clFDF, saveFDF and allData[i]['cleanFDF'] in the main friggin' script, zeroing out all the entires at the same array indexes! It is like they are all associated global variables somehow, but I've made no such declarations anywhere...
I am a hopeless Python newbie, so no doubt I'm not understanding something about how it works. Any help would be greatly appreciated!
You are passing the value at allData[i]['cleanFDF'] by reference (decent explanation at https://stackoverflow.com/a/430958/337678). Any changes made to it will be made to the object it refers to, which is still the same object as the original, just assigned to a different variable.
Making a deep copy of the data will likely fix your issue (Python has a deepcopy library that should do the trick ;)).
Everything is a reference in Python.
def function(y):
y.append('yes')
return y
example = list()
function(example)
print(example)
it would return ['yes'] even though i am not directly changing the variable 'example'.
See Why does list.append evaluate to false?, Python append() vs. + operator on lists, why do these give different results?, Python lists append return value.
I'm running a model in Python and I'm trying to speed up the execution time. Through profiling the code I've found that a huge amount of the total processing time is spent in the cell_in_shadow function below. I'm wondering if there is any way to speed it up?
The aim of the function is to provide a boolean response stating whether the specified cell in the NumPy array is shadowed by another cell (in the x direction only). It does this by stepping backwards along the row checking each cell against the height it must be to make the given cell in shadow. The values in shadow_map are calculated by another function not shown here - for this example, take shadow_map to be an array with values similar to:
[0] = 0 (not used)
[1] = 3
[2] = 7
[3] = 18
The add_x function is used to ensure that the array indices loop around (using clock-face arithmetic), as the grid has periodic boundaries (anything going off one side will re-appear on the other side).
def cell_in_shadow(x, y):
"""Returns True if the specified cell is in shadow, False if not."""
# Get the global variables we need
global grid
global shadow_map
global x_len
# Record the original length and move to the left
orig_x = x
x = add_x(x, -1)
while x != orig_x:
# Gets the height that's needed from the shadow_map (the array index is the distance using clock-face arithmetic)
height_needed = shadow_map[( (x - orig_x) % x_len)]
if grid[y, x] - grid[y, orig_x] >= height_needed:
return True
# Go to the cell to the left
x = add_x(x, -1)
def add_x(a, b):
"""Adds the two numbers using clockface arithmetic with the x_len"""
global x_len
return (a + b) % x_len
I do agree with Sancho that Cython will probably be the way to go, but here are a couple of small speed-ups:
A. Store grid[y, orig_x] in some variable before you start the while loop and use that variable instead. This will save a bunch of look-up calls to the grid array.
B. Since you are basically just starting at x_len - 1 in shadow_map and working down to 1, you can avoid using the modulus so much. Basically, change:
while x != orig_x:
height_needed = shadow_map[( (x - orig_x) % x_len)]
to
for i in xrange(x_len-1,0,-1):
height_needed = shadow_map[i]
or just get rid of the height_needed variable all together with:
if grid[y, x] - grid[y, orig_x] >= shadow_map[i]:
These are small changes, but they might help a little bit.
Also, if you plan on going the Cython route, I would consider having your function do this process for the whole grid, or at least a row at a time. That will save a lot of the function call overhead. However, you might not be able to really do this depending on how you are using the results.
Lastly, have you tried using Psyco? It takes less work than Cython though it probably won't give you quite as big of a speed boost. I would certainly try it first.
If you're not limited to strict Python, I'd suggest using Cython for this. It can allow static typing of the indices and efficient, direct access to a numpy array's underlying data buffer at c speed.
Check out a short tutorial/example at http://wiki.cython.org/tutorials/numpy
In that example, which is doing operations very similar to what you're doing (incrementing indices, accessing individual elements of numpy arrays), adding type information to the index variables cut the time in half compared to the original. Adding efficient indexing into the numpy arrays by giving them type information cut the time to about 1% of the original.
Most Python code is already valid Cython, so you can just use what you have and add annotations and type information where needed to give you some speed-ups.
I suspect you'd get the most out of adding type information your indices x, y, orig_x and the numpy arrays.
The following guide compares several different approaches to optimising numerical code in python:
Scipy PerformancePython
It is a bit out of date, but still helpful. Note that it refers to pyrex, which has since been forked to create the Cython project, as mentioned by Sancho.
Personally I prefer f2py, because I think that fortran 90 has many of the nice features of numpy (e.g. adding two arrays together with one operation), but has the full speed of compiled code. On the other hand if you don't know fortran then this may not be the way to go.
I briefly experimented with cython, and the trouble I found was that by default cython generates code which can handle arbitrary python types, but which is still very slow. You then have to spend time adding all the necessary cython declarations to get it to be more specific and fast, whereas if you go with C or fortran then you will tend to get fast code straight out of the box. Again this is biased by me already being familiar with these languages, whereas Cython may be more appropriate if Python is the only language you know.
I'm doing a Simulated Annealing algorithm to optimise a given allocation of students and projects.
This is language-agnostic pseudocode from Wikipedia:
s ← s0; e ← E(s) // Initial state, energy.
sbest ← s; ebest ← e // Initial "best" solution
k ← 0 // Energy evaluation count.
while k < kmax and e > emax // While time left & not good enough:
snew ← neighbour(s) // Pick some neighbour.
enew ← E(snew) // Compute its energy.
if enew < ebest then // Is this a new best?
sbest ← snew; ebest ← enew // Save 'new neighbour' to 'best found'.
if P(e, enew, temp(k/kmax)) > random() then // Should we move to it?
s ← snew; e ← enew // Yes, change state.
k ← k + 1 // One more evaluation done
return sbest // Return the best solution found.
The following is an adaptation of the technique. My supervisor said the idea is fine in theory.
First I pick up some allocation (i.e. an entire dictionary of students and their allocated projects, including the ranks for the projects) from entire set of randomised allocations, copy it and pass it to my function. Let's call this allocation aOld (it is a dictionary). aOld has a weight related to it called wOld. The weighting is described below.
The function does the following:
Let this allocation, aOld be the best_node
From all the students, pick a random number of students and stick in a list
Strip (DEALLOCATE) them of their projects ++ reflect the changes for projects (allocated parameter is now False) and lecturers (free up slots if one or more of their projects are no longer allocated)
Randomise that list
Try assigning (REALLOCATE) everyone in that list projects again
Calculate the weight (add up ranks, rank 1 = 1, rank 2 = 2... and no project rank = 101)
For this new allocation aNew, if the weight wNew is smaller than the allocation weight wOld I picked up at the beginning, then this is the best_node (as defined by the Simulated Annealing algorithm above). Apply the algorithm to aNew and continue.
If wOld < wNew, then apply the algorithm to aOld again and continue.
The allocations/data-points are expressed as "nodes" such that a node = (weight, allocation_dict, projects_dict, lecturers_dict)
Right now, I can only perform this algorithm once, but I'll need to try for a number N (denoted by kmax in the Wikipedia snippet) and make sure I always have with me, the previous node and the best_node.
So that I don't modify my original dictionaries (which I might want to reset to), I've done a shallow copy of the dictionaries. From what I've read in the docs, it seems that it only copies the references and since my dictionaries contain objects, changing the copied dictionary ends up changing the objects anyway. So I tried to use copy.deepcopy().These dictionaries refer to objects that have been mapped with SQLA.
Questions:
I've been given some solutions to the problems faced but due to my über green-ness with using Python, they all sound rather cryptic to me.
Deepcopy isn't playing nicely with SQLA. I've been told thatdeepcopy on ORM objects probably has issues that prevent it from working as you'd expect. Apparently I'd be better off "building copy constructors, i.e. def copy(self): return FooBar(....)." Can someone please explain what that means?
I checked and found out that deepcopy has issues because SQLAlchemy places extra information on your objects, i.e. an _sa_instance_state attribute, that I wouldn't want in the copy but is necessary for the object to have. I've been told: "There are ways to manually blow away the old _sa_instance_state and put a new one on the object, but the most straightforward is to make a new object with __init__() and set up the attributes that are significant, instead of doing a full deep copy." What exactly does that mean? Do I create a new, unmapped class similar to the old, mapped one?
An alternate solution is that I'd have to "implement __deepcopy__() on your objects and ensure that a new _sa_instance_state is set up, there are functions in sqlalchemy.orm.attributes which can help with that." Once again this is beyond me so could someone kindly explain what it means?
A more general question: given the above information are there any suggestions on how I can maintain the information/state for the best_node (which must always persist through my while loop) and the previous_node, if my actual objects (referenced by the dictionaries, therefore the nodes) are changing due to the deallocation/reallocation taking place? That is, without using copy?
I have another possible solution: use transactions. This probably still isn't the best solution but implementing it should be faster.
Firstly create your session like this:
# transactional session
Session = sessionmaker(transactional=True)
sess = Session()
That way it will be transactional. The way transactions work is that sess.commit() will make your changes permanent while sess.rollback() will revert them.
In the case of simulated annealing you want to commit when you find a new best solution. At any later point, you can invoke rollback() to revert the status back to that position.
You don't want to copy sqlalchemy objects like that. You could implement your own methods which make the copies easily enough, but that is probably not want you want. You don't want copies of students and projects in your database do you? So don't copy that data.
So you have a dictionary which holds your allocations. During the process you should never modify the SQLAlchemy objects. All information that can be modified should be stored in those dictionaries. If you need to modify the objects to take that into account, copy the data back at the end.