We Keep Coding

python importlib seems to be sharing data between instances

Ok, so I am having a weird one. I am running python in a SideFX Hython (their custom build) implementation that is using PDG. The only real difference between Hython and vanilla Python is some internal functions for handling geometry data and compiled nodes, which shouldn't be an issue even though they are being used.
The way the code runs, I am generating a list of files from the disk which creates PDG work items. Those work items are then processed in parallel by PDG. Here is the code for that:
import importlib.util
import pdg
import os
from pdg.processor import PyProcessor
import json
class CustomProcessor(PyProcessor):
def __init__(self, node):
PyProcessor.__init__(self,node)
self.extractor_module = 'GeoExtractor'
def onGenerate(self, item_holder, upstream_items, generation_type):
for upstream_item in upstream_items:
new_item = item_holder.addWorkItem(parent=upstream_item, inProcess=True)
return pdg.result.Success
def onCookTask(self, work_item):
spec = importlib.util.spec_from_file_location("callback", "Geo2Custom.py")
GE = importlib.util.module_from_spec(spec)
spec.loader.exec_module(GE)
GE.convert(f"{work_item.attribValue('directory')}/{work_item.attribValue('filename')}{work_item.attribValue('extension')}", work_item.index, f'FRAME { work_item.index }', self.extractor_module)
return pdg.result.Success
def bulk_convert (path_pattern, extractor_module = 'GeoExtractor'):
type_registry = pdg.TypeRegistry.types()
try:
type_registry.registerNode(CustomProcessor, pdg.nodeType.Processor, name="customprocessor", label="Custom Processor", category="Custom")
except Exception:
pass
whereItWorks = pdg.GraphContext("testBed")
whatWorks = whereItWorks.addScheduler("localscheduler")
whatWorks.setWorkingDir(os.getcwd (), '$HIP')
whereItWorks.setValues(f'{whatWorks.name}', {'maxprocsmenu':-1, 'tempdirmenu':0, 'verbose':1})
findem = whereItWorks.addNode("filepattern")
whereItWorks.setValue(f'{findem.name}', 'pattern', path_pattern, 0)
generic = whereItWorks.addNode("genericgenerator")
whereItWorks.setValue(generic.name, 'itemcount', 4, 0)
custom = whereItWorks.addNode("customprocessor")
custom.extractor_module = extractor_module
node1 = [findem]
node2 = [custom]*len(node1)
for n1, n2 in zip(node1, node2):
whereItWorks.connect(f'{n1.name}.output', f'{n2.name}.input')
n2.cook(True)
for node in whereItWorks.graph.nodes():
node.dirty(False)
whereItWorks.disconnect(f'{n1.name}.output', f'{n2.name}.input')
print ("FULLY DONE")
import os
import hou
import traceback
import CustomWriter
import importlib
def convert (filename, frame_id, marker, extractor_module = 'GeoExtractor'):
Extractor = importlib.__import__ (extractor_module)
base, ext = os.path.splitext (filename)
if ext == '.sc':
base = os.path.splitext (base)[0]
dest_file = base + ".custom"
geo = hou.Geometry ()
geo.loadFromFile (filename)
try:
frame = Extractor.extract_geometry (geo, frame_id)
except Exception as e:
print (f'F{ frame_id } Geometry extraction failed: { traceback.format_exc () }.')
return None
print (f'F{ frame_id } Geometry extracted. Writing file { dest_file }.')
try:
CustomWriter.write_frame (frame, dest_file)
except Exception as e:
print (f'F{ frame_id } writing failed: { e }.')
print (marker + " SUCCESS")
The onCookTask code is run when the work item is processed.
Inside of the GeoExtractor.py program I am importing the geometry file defined by the work item, then converting it into a couple Pandas dataframes to collate and process the massive volumes of data quickly, which is then passed to a custom set of functions for writing binary files to disk from the Pandas data.
Everything appears to run flawlessly, until I check my output binaries and see that they escalate in file size much more than they should, indicating that either something is being shared between instances or not cleared from memory and subsequent loads of the extractor code is appending the dataframes which are named the same.
I have run the GeoExtractor code sequentially with the python instance closing between each file conversion using the exact same code and the files are fine, growing only very slowly as the geometry data volume grows, so the issue has to lie somewhere in the parallelization of it using PDG and calling the GeoExtractor.py code over and over for each work item.
I have contemplated moving the importlib stuff to the __init__() for the class leaving only the call to the member function in the onCookTask() function. Maybe even going so far as to pass a unique variable for each work item which is used inside GeoExtractor to create a closure of the internal functions so they are unique instances in memory.
I tried to do a stripped down version of GeoExtractor and since I'm not sure where the leak is, I just ended up pulling out comments with proprietary or superfluous information and changing some custom library names, but the file ended up kinda long so I am including a pastebin: https://pastebin.com/4HHS8D2W
As for CustomGeometry and CustomWriter, there is no working form of either of those libraries that will be NDA safe, so unfortunately they have to stay blackboxed. The CustomGeometry is a handful of container classes which organize all of the data coming out of the geometry, and the writer is a formatter/writer for the binary format we are utilizing. I am hoping the issue wouldn't be in either of them.
Edit 1: I fixed an issue in the example code.
Edit 2: Added larger examples.

Instance attributes in Python and __init__ method

I am trying to writing a program to read a configuration file but while testing it am having this error:
self.connection_attempts = self.config_file.get('CONNECTION_ATTEMPTS', 'TIME')
AttributeError: 'list' object has no attribute 'get'
I ma pretty sure it is something I don't get, but it is few hours I am trying to understand where the problem is.
My __init__ method looks like this:
import simpleconfigparser
class ReportGenerator:
def __init__(self):
self.config_parser = simpleconfigparser.configparser()
self.config_file = config_parser.read('config.ini')
self.connection_attempts = config_file.get('CONNECTION_ATTEMPTS', 'TIME')
self.connection_timeout = config_file.get('CONNECTION_TIMEOUT', 'TIMEOUT')
self.report_destination_path = config_file.get('REPORT', 'REPORT_PATH')
This code uses the SimpleConfigParser package.

You want config_parser.get() not config_file.get(). config_parser.read() simply returns the list of config files successfully read after populating the config object. (Usually it is called config or cfg, not config_parser).
This list (config_file) serves no purpose in your code and you might as well not capture it at all.
from simpleconfigparser import simpleconfigparser
TIME = 5
TIMEOUT = 10
REPORT_PATH = '/tmp/'
class ReportGenerator:
def __init__(self):
self.config = simpleconfigparser()
config.read('config.ini')
self.connection_attempts = config.get('CONNECTION_ATTEMPTS', TIME)
self.connection_timeout = config.get('CONNECTION_TIMEOUT', TIMEOUT)
self.report_destination_path = config.get('REPORT', REPORT_PATH)
My guess would also be, that you use the default value in .get() the wrong way, but i cannot be certain with the information you have given.

Call a function through a variable in Python

I'm trying to make a game engine in Python as a "weekend project" but I have a problem.
I'm trying to make it that a user can declare a keybind by typing the key and the function into a text file what they want to run but when I run the code from that sheet with exec it runs the function and I have no idea on how to call the function through the variable. (Also I don't want it to run the function when it runs the code.)
Here is the code I use for executing the code from binds.zdata
for line in bind:
try:
exec line
except:
try:
error.write(localtime + " : " + "ERROR: Could not bind key from the bind file : " + line)
except:
pass
Here's the text in binds.zdata
_w = Functions.motion.move().forward()
_a = Functions.motion.move().left()
_s = Functions.motion.move().back()
_d = Functions.motion.move().right()

You want to lose the () at the end of each line:
_w = Functions.motion.move().forward
_a = Functions.motion.move().left
_s = Functions.motion.move().back
_d = Functions.motion.move().right
now you can call the function through the variable by simply applying parenthesis, such as:
_w()

I'm not sure I recommend "exec" since somebody could put any code in there.
But here is your problem.
_w = Functions.motion.move().forward()
Calls the function 'forward' and puts the results in _w.
_w = Functions.motion.move().forward
Assigns the function 'forward' to the variable '_w'.
Since you asked what I would do, I would create a set of tokens that represent the various functions and then let them do the mapping in side a config file (see: ConfigParser). Then parse the config file. It is a little more work, but a lot more secure.

I'd first change the lines in the binds.zdata file like this so they assign a function to each of the variables:
_w = Functions.motion.move().forward
_a = Functions.motion.move().left
_s = Functions.motion.move().back
_d = Functions.motion.move().right
Then, assuming that Functions is defined, I'd execute all the lines in the file at once with something like:
try:
execfile('binds.zdata')
except Exception as exc:
error.write(localtime + " : " + "ERROR: executing binds file")
Afterwards, the _w, _a, _s, _d variables will have the desired function assigned to them, which you can call as usual — i.e. _w(), _a(), etc.

Reading the config like that with exec is going to open you to all kinds of pain at runtime, as the user's config file is going to be executable code that you are just trusting. Malicious or unintentionally just bad commands will get execute. You could template it like this:
def read_config(config_file):
user_button_map = {}
with open(config_file, 'r') as fh:
for line in fh.readlines():
key,func_key = line.strip().split(',')
assert(key in BUTTON_MAP_FUNCTIONS),"no such action"
user_button_map[key] = BUTTON_MAP_FUNCTIONS[func_key]
return user_button_map
BUTTON_MAP_FUNCTIONS = {
"forward" : Functions.motion.move().forward,
"left" : Functions.motion.move().left,
# etc...
}
# sample config file
# w, forward
# a, left
# etc...

Get file syntax selection in sublime text 3 plugin

I have a very small plugin to open a perl file module starting from the use statement. It's really basic and it just replaces '::' with '/' and then if the file exists in one of the paths specified in PERL5LIB, it opens it.
I want it to run only when the open file syntax is selected as perl.
Is there any API to get that information?
This is the code that I have now:
class OpenPerlModule(sublime_plugin.TextCommand):
def run(self, edit=None, url=None):
perl_file = url.replace("::", "/")
perl_dirs = os.environ.get('PERL5LIB')
for perl_dir in perl_dirs.split(':'):
if (os.path.exists(perl_dir + '/' + perl_file + '.pm')):
self.view.window().open_file(perl_dir + '/' + perl_file + '.pm')
return
(The OS is Ubuntu)

Here is the code snippet you're looking for
self.view.settings().get("syntax")
You should check whether it's a syntax related to Perl or not. I suggest something like this:
syntax = self.view.settings().get("syntax")
syntax.endswith("Perl.tmLanguage") or syntax.endswith("Perl.sublime-syntax")
The second or clause is to cover the new syntax that's introduced in >=3080

Aside from self.view.settings().get("syntax") as described in Allen Bargi's answer, you could also get the scope for the current cursor position and check for source.perl in it:
import sublime_plugin
class FindScopeCommand(sublime_plugin.TextCommand):
def run(self, edit):
# `sel()` returns a list of Regions that are selected.
# Grab the beginning point of the first Region in the list.
first_point = self.view.sel()[0].a
# now, get the full scope name for that point
scope = self.view.scope_name(first_point)
if "source.perl" in scope:
print("You're using Perl. Yay!")
else:
print("Why don't you love Perl?")

How would I separate my Python File to multiple plugins?

So first thing I want to say: I have been looking into modules and such, I just don't quiet know how I would rewrite it to fit this in.
Project: What I have is a Skype Bot using the Skype4Py module. I have about 11 commands I noticed the one script is getting a little large.
I'm trying to think about how to link one main.py file to a Plugin Folder, which contains each and every bot function in it's own respectable Python file. It sounds simple an all, except when it comes to how the function is called.
Here is just a basic look at my Skype bot, missing some of the larger functions.
import Skype4Py, random
class SkypeBot():
def __init__(self):
self.skype = Skype4Py.Skype()
if self.skype.Client.IsRunning == False:
self.skype.Client.Start()
self.skype.Attach()
self.results = ['Yes', 'No', 'Maybe', 'Never']
def main(self):
print ' Skype Bot currently running on user: %s' % self.skype.CurrentUserHandle
print "\n\nCommands Called:\n"
while True:
self.skype.OnMessageStatus = self.RunFunction
def RunFunction(self, Message, Status):
if Status == 'SENT' or Status == 'RECEIVED':
cmd = Message.Body.split(' ')[0]
if cmd in self.functions.keys():
self.context = Message
self.caller = self.context.FromHandle
self.functions[cmd](self)
def ping(self):
print " %s : Ping" % self.caller
self.context.Chat.SendMessage('Pong')
def say(self):
try:
response = self.context.Body.split(' ', 1)
if response[1] == "-info":
print " %s : say -info" % self.caller
self.context.Chat.SendMessage("Resends the message entered. \n"
"Usage: !say Hello. \n"
"Example: Bot: Hello.")
else:
say = response[1]
print " %s : Say [%s]" % (self.caller, say)
self.context.Chat.SendMessage(say)
except:
self.context.Chat.SendMessage("Please use -info to properly use the !say command")
def eightball(self):
try:
question = self.context.Body.split(' ', 1)
if question[1] == "-info":
print " %s : 8Ball -info" % self.caller
self.context.Chat.SendMessage("Responds with an answer.\n"
"Usage: !8ball 'Do I have swag?'\n"
"Example: !8Ball Response: 'Yes'")
else:
random.shuffle(self.results)
answer = self.results[3]
print " %s : 8Ball [%s]" % (self.caller, question[1])
self.context.Chat.SendMessage("!8Ball Response: %s" % answer)
except:
self.context.Chat.SendMessage("Please use -info to properly use the !8ball command")
#FUNCTIONS LIST
#********************
functions = {
"!ping": ping,
"!say": say,
"!8ball": eightball,
}
if __name__ == "__main__":
snayer = SkypeBot()
snayer.main()
So basically, what I am wondering, how can I change
self.skype.OnMessageStatus = self.RunFunction
so that it'll run functions from another file?

For a program of this size it's not really necessary to put your command functions into separate files, but I guess it is good organization. And good practice for when you write a program that has thousands of lines of code. :)
One way to do this is to create a basic SkypeBot class without any command methods and then import the command methods from your plugins directory and add them to the class. It's easy enough to add new attributes to an existing class, and it doesn't matter if the new attributes are properties or methods, the syntax to add them is identical. (With a tiny bit more work it's even possible to add new attributes to an instance, so you can have multiple instances, each with their own individual set of commands. But I guess that's not necessary here, since a program that uses the SkypeBot class will normally only create a single instance).
So we can break your question into two parts:
How to add methods to an existing class.
How to import those methods
from other source files.
As I said, 1) is easy. 2) is quite easy as well, but I've never done it before, so I had to do a little bit of research and testing, and I can't promise that what I've done is best practice, but it works. :)
I don't know much about Skype, and I don't have that Skype4Py module, and as you said, the code above is not the complete program, so I've written some fairly simple code to illustrate the process of adding plugin methods from separate files to an existing class.
The name of the main program is "plugin_demo.py". To keep things neat, it lives in its own directory, "plugintest/", which you should create somewhere in your Python path (eg where you normally keep your Python programs). This path must be specified in your PYTHONPATH environment variable.
"plugintest/" has the following structure:
plugintest/
__init__.py
plugin_demo.py
plugins/
__init__.py
add.py
multiply.py
The __init__.py files are used by Python's import machinery to let it know that a directory contains a Python package, see 6.4. Packages in the Python docs for further details.
Here are the contents of those files. Firstly, the files that go into "plugintest/" itself:
__init__.py
__all__ = ['plugin_demo', 'plugins']
from plugintest import *
plugin_demo.py
#! /usr/bin/env python
#A simple class that will get methods added later from plugins directory
class Test(object):
def __init__(self, data):
self.data = data
def add_plugins(cls):
import plugins
print "Adding plugin methods to %s class" % cls.__name__
for name in plugins.__all__:
print name
plug = getattr(plugins, name)
print plug
method = getattr(plug, name)
print method
setattr(cls, name, method)
print
print "Done\n"
add_plugins(Test)
def main():
#Now test it!
t = Test([1, 2, 3]); print t.data
t.multiply(10); print t.data
t.add(5); print t.data
if __name__ == '__main__':
main()
And now the contents of the "plugintest/plugins/" directory:
__init__.py
__all__ = ['add', 'multiply']
from plugintest.plugins import *
add.py
#A method for the Test class of plugin_demo.py
def add(self, m):
self.data = [m + i for i in self.data]
multiply.py
#A method for the Test class of plugin_demo.py
def multiply(self, m):
self.data = [m * i for i in self.data]
If you cd to the directory containing the "plugintest/" folder, you should be able to run it with
python plugintest/plugin_demo.py
and if you cd to "plugintest/" itself
python plugin_demo.py
Also, in the interpreter (or another Python program), you should be able to do
import plugintest
and then run the main() function of "plugin_demo.py" with
plugintest.plugin_demo.main()
The other usual variations of from ... import ... etc should also work as expected.
The function in "plugin_demo.py" that performs the magic of adding the imported methods to the Test class is add_plugins(). When it runs it prints out each method name, its module, and its function. This could be handy during development, but you'd probably comment out some of those print statements once the program's working properly.
I hope this helps, and if you have any questions please don't hesitate to ask.