We Keep Coding

Maya Python Script Job Attribute Change

I am trying to get one boolean attribute (A) to change another(B). The one to be controlled (B) already has a script job running it and so I can't create a set driven key, direct connection, or expression to control it, so I'm trying another script job, since running the script function by itself achieves the desired result. I just can't figure out how to tie that script to run to the attribute change (B) that I want to drive it by (A).
This is placed in a script node set to the open gui trigger (to load when maya opens as I understand it). Here's a screenshot.
What am I missing here?
import maya.cmds as cmds
def togglePicker(pickerAttr):
cmds.setAttr(pickerAttr, not 0)
nameSpace = cmds.ls(sl=True)[0].rpartition(':')[0]
if len(nameSpace) > 0:
pickerAttr = nameSpace + ':Main.picker'
myPickerAttr = nameSpace + ':MoverMain_Cntrl.Picker'
else:
pickerAttr = 'Main.picker'
myPickerAttr = 'MoverMain_Cntrl.Picker'
cmds.scriptJob(attributeChange=[myPickerAttr,togglePicker])

Your script node is executed every time when maya loads a scene, not when it is started, at least that's what the docs say. So every time you load a scene, a new scriptJob is created.
Your script should show an error message since the togglePicker() function is called without an argument, but it requires an argument. Even if it works, it will not work.. what you do at the moment is the following:
As soon as you turn on the MoverMain_Cntrl.Picker attribute, the togglePicker() function is called and turns it on, even if you turn it off. The pickerAttrvariable is not used. So you should have a look at your program logic.
You can solve the agrument problem by using the partial function like this:
import maya.cmds as cmds
from functools import partial
def togglePicker(pickerAttr):
cmds.setAttr(pickerAttr, not 0)
nameSpace = cmds.ls(sl=True)[0].rpartition(':')[0]
if len(nameSpace) > 0:
pickerAttr = nameSpace + ':Main.picker'
myPickerAttr = nameSpace + ':MoverMain_Cntrl.Picker'
else:
pickerAttr = 'Main.picker'
myPickerAttr = 'MoverMain_Cntrl.Picker'
cmds.scriptJob(attributeChange=[myPickerAttr,partial(togglePicker, pickerAttr)])

I got it to work! (previously I had switched to the script node to MEL so I could test the mel command mentioned in the comments that did work, but I forgot to switch back to python when I realized the selection issue I also mentioned in the comments).
So here's what worked, where I know I'll have to manually change the namespace name in case the scene file name changes:
import maya.cmds as cmds
def togglePicker():
cmds.setAttr(pickerAttr, not 0)
if cmds.namespace(exists='ExtremeBallRig_v008'):
pickerAttr = 'ExtremeBallRig_v008:Main.picker'
myPickerAttr = 'ExtremeBallRig_v008:MoverMain_Cntrl.Picker'
else:
pickerAttr = 'Main.picker'
myPickerAttr = 'MoverMain_Cntrl.Picker'
cmds.scriptJob(attributeChange=[myPickerAttr,togglePicker])

python multiprocessing - OverflowError('cannot serialize a bytes object larger than 4GiB')

We are running a script using the multiprocessing library (python 3.6), where a big pd.DataFrame is passed as an argument to a function :
from multiprocessing import Pool
import time
def my_function(big_df):
# do something time consuming
time.sleep(50)
if __name__ == '__main__':
with Pool(10) as p:
res = {}
output = {}
for id, big_df in some_dict_of_big_dfs:
res[id] = p.apply_async(my_function,(big_df ,))
output = {id : res[id].get() for id in id_list}
The problem is that we are getting an error from the pickle library.
Reason: 'OverflowError('cannot serialize a bytes objects larger than
4GiB',)'
We are aware than pickle v4 can serialize larger objects question related, link, but we don't know how to modify the protocol that multiprocessing is using.
does anybody know what to do?
Thanks !!

Apparently is there an open issue about this topic , and there is a few related initiatives described on this particular answer. I Found a way to change the default pickle protocol that is used in the multiprocessing library based on this answer. As was pointed out in the comments this solution Only works with Linux and OS multiprocessing lib
Solution:
You first create a new separated module
pickle4reducer.py
from multiprocessing.reduction import ForkingPickler, AbstractReducer
class ForkingPickler4(ForkingPickler):
def __init__(self, *args):
if len(args) > 1:
args[1] = 2
else:
args.append(2)
super().__init__(*args)
#classmethod
def dumps(cls, obj, protocol=4):
return ForkingPickler.dumps(obj, protocol)
def dump(obj, file, protocol=4):
ForkingPickler4(file, protocol).dump(obj)
class Pickle4Reducer(AbstractReducer):
ForkingPickler = ForkingPickler4
register = ForkingPickler4.register
dump = dump
And then, in your main script you need to add the following:
import pickle4reducer
import multiprocessing as mp
ctx = mp.get_context()
ctx.reducer = pickle4reducer.Pickle4Reducer()
with mp.Pool(4) as p:
# do something
That will probably solve the problem of the overflow.
But, warning, you might consider reading this before doing anything or you might reach the same error as me:
'i' format requires -2147483648 <= number <= 2147483647
(the reason of this error is well explained in the link above). Long story short, multiprocessing send data through all its process using the pickle protocol, if you are already reaching the 4gb limit, that probably means that you might consider redefining your functions more as "void" methods rather than input/output methods. All this inbound/outbound data increase the RAM usage, is probably inefficient by construction (my case) and it might be better to point all process to the same object rather than create a new copy for each call.
hope this helps.

Supplementing answer from Pablo
The following problem can be resolved be Python3.8, if you are okay to use this version of python:
'i' format requires -2147483648 <= number <= 2147483647

Gio.MemoryInputStream does not free memory when closed

Running Python 3.4 on Windows 7, the close function of Gio.MemoryInputStream does not free the memory, as it should. The test code is :
from gi.repository import Gio
import os, psutil
process = psutil.Process(os.getpid())
for i in range (1,10) :
input_stream = Gio.MemoryInputStream.new_from_data(b"x" * 10**7)
x = input_stream.close_async(2)
y = int(process.memory_info().rss / 10**6) # Get the size of memory used by the program
print (x, y)
This returns :
True 25
True 35
True 45
True 55
True 65
True 75
True 85
True 95
True 105
This shows that on each loop, the memory used by the program increases of 10 MB, even if the close function returned True.
How is it possible to free the memory, once the Stream is closed ?
Another good solution would be to reuse the stream. But set_data or replace_data raises the following error :
'Data access methods are unsupported. Use normal Python attributes instead'
Fine, but which property ?
I need a stream in memory in Python 3.4. I create a Pdf File with PyPDF2, and then I want to preview it with Poppler. Due to a bug in Poppler (see Has anyone been able to use poppler new_from_data in python?) I cannot use the new_from_data function and would like to use the new_from_stream function.

This is a bug in GLib’s Python bindings which can’t be trivially fixed.
Instead, you should use g_memory_input_stream_new_from_bytes(), which handles freeing memory differently, and shouldn’t suffer from the same bug.
In more detail, the bug with new_from_data() is caused by the introspection annotations, which GLib uses to allow language bindings to automatically expose all of its API, not supporting the GDestroyNotify parameter for new_from_data() which needs to be set to a non-NULL function to free the allocated memory which is passed in to the other arguments. Running your script under gdb shows that pygobject passes NULL to the GDestroyNotify parameter. It can’t do any better, since there is currently no way of expressing that the memory management semantics of the data parameter depend on what’s passed to destroy.

Thanks for your answer, #Philip Withnall. I tested the solution you propose, and it works. To help others to understand, here is my test code :
from gi.repository import Gio, GLib
import os, psutil
process = psutil.Process(os.getpid())
for i in range (1,10) :
input_stream = Gio.MemoryInputStream.new_from_bytes(GLib.Bytes(b"x" * 10**7))
x = input_stream.close()
y = int(process.memory_info().rss / 10**6) # Get the size of memory used by the program
print (x, y)
Now y non longer grows.

Python mmap ctypes - read only

I think I have the opposite problem as described here. I have one process writing data to a log, and I want a second process to read it, but I don't want the 2nd process to be able to modify the contents. This is potentially a large file, and I need random access, so I'm using python's mmap module.
If I create the mmap as read/write (for the 2nd process), I have no problem creating ctypes object as a "view" of the mmap object using from_buffer. From a cursory look at the c-code, it looks like this is a cast, not a copy, which is what I want. However, this breaks if I make the mmap ACCESS_READ, throwing an exception that from_buffer requires write privileges.
I think I want to use ctypes from_address() method instead, which doesn't appear to need write access. I'm probably missing something simple, but I'm not sure how to get the address of the location within an mmap. I know I can use ACCESS_COPY (so write operations show up in memory, but aren't persisted to disk), but I'd rather keep things read only.
Any suggestions?

I ran into a similar issue (unable to setup a readonly mmap) but I was using only the python mmap module. Python mmap 'Permission denied' on Linux
I'm not sure it is of any help to you since you don't want the mmap to be private?

Ok, from looking at the mmap .c code, I don't believe it supports this use case. Also, I found that the performance pretty much sucks - for my use case. I'd be curious what kind performance others see, but I found that it took about 40 sec to walk through a binary file of 500 MB in Python. This is creating a mmap, then turning the location into a ctype object with from_buffer(), and using the ctypes object to decipher the size of the object so I could step to the next object. I tried doing the same thing directly in c++ from msvc. Obviously here I could cast directly into an object of the correct type, and it was fast - less than a second (this is with a core 2 quad and ssd).
I did find that I could get a pointer with the following
firstHeader = CEL_HEADER.from_buffer(map, 0) #CEL_HEADER is a ctypes Structure
pHeader = pointer(firstHeader)
#Now I can use pHeader[ind] to get a CEL_HEADER object
#at an arbitrary point in the file
This doesn't get around the original problem - the mmap isn't read-only, since I still need to use from_buffer for the first call. In this config, it still took around 40 sec to process the whole file, so it looks like the conversion from a pointer into ctypes structs is killing the performance. That's just a guess, but I don't see a lot of value in tracking it down further.
I'm not sure my plan will help anyone else, but I'm going to try to create a c module specific to my needs based on the mmap code. I think I can use the fast c-code handling to index the binary file, then expose only small parts of the file at a time through calls into ctypes/python objects. Wish me luck.
Also, as a side note, Python 2.7.2 was released today (6/12/11), and one of the changes is an update to the mmap code so that you can use a python long to set the file offset. This lets you use mmap for files over 4GB on 32-bit systems. See Issue #4681 here

Ran into this same problem, we needed the from_buffer interface and wanted read only access. From the python docs https://docs.python.org/3/library/mmap.html "Assignment to an ACCESS_COPY memory map affects memory but does not update the underlying file."
If it's acceptable for you to use an anonymous file backing you can use ACCESS_COPY
An example: open two cmd.exe or terminals and in one terminal:
mm_file_write = mmap.mmap(-1, 4096, access=mmap.ACCESS_WRITE, tagname="shmem")
mm_file_read = mmap.mmap(-1, 4096, access=mmap.ACCESS_COPY, tagname="shmem")
write = ctypes.c_int.from_buffer(mm_file_write)
read = ctypes.c_int.from_buffer(mm_file_read)
try:
while True:
value = int(input('enter an integer using mm_file_write: '))
write.value = value
print('updated value')
value = int(input('enter an integer using mm_file_read: '))
#read.value assignment doesnt update anonymous backed file
read.value = value
print('updated value')
except KeyboardInterrupt:
print('got exit event')
In the other terminal do:
mm_file = mmap.mmap(-1, 4096, access=mmap.ACCESS_WRITE, tagname="shmem")
i = None
try:
while True:
new_i = struct.unpack('i', mm_file[:4])
if i != new_i:
print('i: {} => {}'.format(i, new_i))
i = new_i
time.sleep(0.1)
except KeyboardInterrupt:
print('Stopped . . .')
And you will see that the second process does not receive updates when the first process writes using ACCESS_COPY

Python, ArcObjects, and .AppRef: how to get from IAppROT to IMxDocument?

I am writing an external Python/comtypes script (in PythonWin) that needs to get a reference to the current ArcGIS 10.0 ArcMap session (through the ArcObjects COM). Because the script is outside the application boundary, I am getting the application reference through the AppROT (running object table). The first code snippet below is the main Python driver module. In it is a function GetApp() to grab an application reference from the AppROT. This code works fine and returns IApplication on the singleton AppRef object. Makes sense, and that's what the ArcObjects reference seems to indicate. Now, my main goal is to get to an IMxDocument. In the main loop, I get to an IDocument successfully and can print the title. The next line, though, a Query Interface, throws an error - NameError: name 'esriArcMapUI' is not defined. The error occurs consistently, even after closing PythonWin and reopening (which you always want to try before you conclude that you have a problem). [BTW, the second code snippet is the CType() function for QI, defined in and imported from the SignHelpers.py module.] So, here are my questions:
(1) What COM object is the IDocument on?
(2) How do I get from my IDocument to the intended IMxDocument? Do I need to create a new MxDocument object first? [Sorry. Two questions there.]
(3) If I don't have to create a new object, then how do I do the QI?
I did a lot of ArcObjects work in VB6 quite a few years ago, so explicit QI's and namespaces are putting the screws to me at the moment. Once I can get to an IMxDocument I will be home free. I would appreciate any help anyone can give me with this.
Also, I apologize for the formatting of the code below. I could not figure out how to get Python code to format correctly. Indentation doesn't work, and some of the Python code is interpreted as formatting characters.
=== code: main py module ===
import sys, os
sys.path.append('C:\GISdata\E_drive\AirportData\ATL\Scripts')
import comtypes
from SignHelpers import *
def GetApp(app):
"""Get a hook into the current session of ArcMap; \n\
Execute GetDesktopModules() and GetStandaloneModules() first"""
print "GetApp called" ####
import comtypes.gen.esriFramework as esriFramework
import comtypes.gen.esriArcMapUI as esriArcMapUI
import comtypes.gen.esriCarto as esriCarto
pAppROT = NewObj(esriFramework.AppROT, esriFramework.IAppROT)
iCount = pAppROT.Count
print "appROT count = ", iCount ####
if iCount == 0:
print 'No ArcGIS application currently running. Terminating ...'
return None
for i in range(iCount):
pApp = pAppROT.Item(i) #returns IApplication on AppRef
if pApp.Name == app:
print "Application: ", pApp.Name ####
return pApp
print 'No ArcMap session is running at this time.'
return None
if __name__ == "__main__":
#Wrap needed ArcObjects type libraries (.olb)...
... code omitted ...
GetDesktopModules(dtOLB) #These force comtypes to generate
GetStandaloneModules(saOLB) #the Python wrappers for .olb's
#Connect to ArcMap
pApp = GetApp("ArcMap")
pDoc = pApp.Document #IDocument on current Document object
print pDoc.Title
pMxDoc = CType(pDoc, esriArcMapUI.IMxDocument) #QI to IMxDocument on MxDocument
=== code for CType() ===
def CType(obj, interface):
try:
newobj = obj.QueryInterface(interface)
return newobj
except:
return None

Scoping error (per the comments):
The import comtypes.gen.esriArcMapUI as esriArcMapUI statement needed to define the esriArcMapUI namespace was being run within the GetApp() function, so the namespace was local to the function.