i have created a pretty simple MapReduce pipeline, but i am having a cryptic:
PipelineSetupError: Error starting production.cron.pipelines.ItemsInfoPipeline(*(), **{})#a741186284ed4fb8a4cd06e38921beff:
when i try to start it. This is the pipeline code:
class ItemsInfoPipeline(base_handler.PipelineBase):
"""
"""
def run(self):
output = yield mapreduce_pipeline.MapreducePipeline(
job_name="items_job",
mapper_spec="production.cron.mappers.items_info_mapper",
input_reader_spec="mapreduce.input_readers.DatastoreInputReader",
mapper_params={
"input_reader": {
"entity_kind": "production.models.Transaction"
}
}
)
yield ItemsInfoStorePipeline(output)
class ItemsInfoStorePipeline(base_handler.PipelineBase):
"""
"""
def run(self, statistics):
print statistics
return "OK"
Of course i have double checked that the mapper path is right, and take into account that ItemsInfoStorePipeline is not doing anything because i am focusing the have the pipeline started, which is not happening.
It is all triggered by a Flask view, the following:
class ItemsInfoMRJob(views.MethodView):
"""
It's based on transacions.
"""
def get(self):
"""
:return:
"""
pipeline = ItemsInfoPipeline()
pipeline.start()
redirect_url = "%s/status?root=%s" % (pipeline.base_path, pipeline.pipeline_id)
return flask.redirect(redirect_url)
I am using GoogleAppEngineMapReduce==1.9.22.0
Thanks for any help.
UPDATE
The above code works once deployed.
UPDATE 2
Apparently there's more people dealing with this:
https://github.com/GoogleCloudPlatform/appengine-mapreduce/issues/103
I'm updating this. I have a code base that's using pipelines and works fine in OSX. I had another developer using OSX that simply nothing I do seems to get it working, he gets this:
Encountered unexpected error from ProtoRPC method implementation: PipelineSetupError
I've tried swapping versions around and making our PC's match perfectly and it continue to happen. I finally broke down and built an Ubuntu image in docker. I'm also doing my best to match our versions of AppEngine and libraries perfectly.
It also refuses to start with the same message. I started working through the libraries uncommenting the part that's swallowing the error but that was a long rabbit hole I started down because lots of stuff above it also seems to be swallowing up whatever is going on.
Related
I have a Flyte task function like this:
#task
def do_stuff(framework_obj):
framework_obj.get_outputs() # This calls Types.Blob.fetch(some_uri)
Trying to load a blob URI using flytekit.sdk.types.Types.Blob.fetch, but getting this error:
ERROR:flytekit: Exception when executing No temporary file system is present. Either call this method from within the context of a task or surround with a 'with LocalTestFileSystem():' block. Or specify a path when calling this function. Note: Cleanup is not automatic when a path is specified.
I can confirm I can load blobs using with LocalTestFileSystem(), in tests, but when actually trying to run a workflow, I'm not sure why I'm getting this error, as the function that calls blob-processing is decorated with #task so it's definitely a Flyte Task. I also confirmed that the task node exists on the Flyte web console.
What path is the error referencing and how do I call this function appropriately?
Using Flyte Version 0.16.2
Could you please give a bit more information about the code? This is flytekit version 0.15.x? I'm a bit confused since that version shouldn't have the #task decorator. It should only have #python_task which is an older API. If you want to use the new python native typing API you should install flytekit==0.17.0 instead.
Also, could you point to the documentation you're looking at? We've updated the docs a fair amount recently, maybe there's some confusion around that. These are the examples worth looking at. There's also two new Python classes, FlyteFile and FlyteDirectory that have replaced the Blob class in flytekit (though that remains what the IDL type is called).
(would've left this as a comment but I don't have the reputation to yet.)
Some code to help with fetching outputs and reading from a file output
#task
def task_file_reader():
client = SynchronousFlyteClient("flyteadmin.flyte.svc.cluster.local:81", insecure=True)
exec_id = WorkflowExecutionIdentifier(
domain="development",
project="flytesnacks",
name="iaok0qy6k1",
)
data = client.get_execution_data(exec_id)
lit = data.full_outputs.literals["o0"]
ctx = FlyteContext.current_context()
ff = TypeEngine.to_python_value(ctx, lv=lit,
expected_python_type=FlyteFile)
with open(ff, 'rb') as fh:
print(fh.readlines())
Morning folks,
I'm trying to get a few unit tests going in Python to confirm my code is working, but I'm having a real hard time getting a Mock anything to fit into my test cases. I'm new to Python unit testing, so this has been a trying week thus far.
The summary of the program is I'm attempting to do serial control of a commercial monitor I got my hands on and I thought I'd use it as a chance to finally use Python for something rather than just falling back on one of the other languages I know. I've got pyserial going, but before I start shoving a ton of commands out to the TV I'd like to learn the unittest part so I can write for my expected outputs and inputs.
I've tried using a library called dummyserial, but it didn't seem to be recognising the output I was sending. I thought I'd give mock_open a try as I've seen it works like a standard IO as well, but it just isn't picking up on the calls either. Samples of the code involved:
def testSendCmd(self):
powerCheck = '{0}{1:>4}\r'.format(SharpCodes['POWER'], SharpCodes['CHECK']).encode('utf-8')
read_text = 'Stuff\r'
mo = mock_open(read_data=read_text)
mo.in_waiting = len(read_text)
with patch('__main__.open', mo):
with open('./serial', 'a+b') as com:
tv = SharpTV(com=com, TVID=999, tvInput = 'DVI')
tv.sendCmd(SharpCodes['POWER'], SharpCodes['CHECK'])
com.write(b'some junk')
print(mo.mock_calls)
mo().write.assert_called_with('{0}{1:>4}\r'.format(SharpCodes['POWER'], SharpCodes['CHECK']).encode('utf-8'))
And in the SharpTV class, the function in question:
def sendCmd(self, type, msg):
sent = self.com.write('{0}{1:>4}\r'.format(type,msg).encode('utf-8'))
print('{0}{1:>4}\r'.format(type,msg).encode('utf-8'))
Obviously, I'm attempting to control a Sharp TV. I know the commands are correct, that isn't the issue. The issue is just the testing. According to documentation on the mock_open page, calling mo.mock_calls should return some data that a call was made, but I'm getting just an empty set of []'s even in spite of the blatantly wrong com.write(b'some junk'), and mo().write.assert_called_with(...) is returning with an assert error because it isn't detecting the write from within sendCmd. What's really bothering me is I can do the examples from the mock_open section in interactive mode and it works as expected.
I'm missing something, I just don't know what. I'd like help getting either dummyserial working, or mock_open.
To answer one part of my question, I figured out the functionality of dummyserial. The following works now:
def testSendCmd(self):
powerCheck = '{0}{1:>4}\r'.format(SharpCodes['POWER'], SharpCodes['CHECK'])
com = dummyserial.Serial(
port='COM1',
baudrate=9600,
ds_responses={powerCheck : powerCheck}
)
tv = SharpTV(com=com, TVID=999, tvInput = 'DVI')
tv.sendCmd(SharpCodes['POWER'], SharpCodes['CHECK'])
self.assertEqual(tv.recv(), powerCheck)
Previously I was encoding the dictionary values as utf-8. The dummyserial library decodes whatever you write(...) to it so it's a straight string vs. string comparison. It also encodes whatever you're read()ing as latin1 on the way back out.
I've been struggling with this question for a while now. My question is very specific so please don't post a link to the ROS-tutorial page with the "!Note:" paragraph showing how to pass mutable objects (unless you show me something I have missed). I would like to know if anyone has been able to correctly pass mutable objects back and forth in SMACH states, without encountering any errors.
I wrote a particularly simple and useless program to explain what I am attempting to do. I could post the code of this example (unfortunately however, not surprisingly to those that have used SMACH before, it is a long piece of code). So for now I will just try my best to explain it and include a [link] to an image of my example. I created two python scripts. Each script contains a single class and object of that class (with some basic methods). I create a publisher and subscriber in each script, one of the scripts sends messages (talks) while the other script listens to (hears) the messages. At the end the talker flags both FSMs to shutdown. If anyone would like the full code example let me know...
Code snippet below showing smach states and transitions:
# begin sm
with sm:
smach.StateMachine.add('LOAD', loadFSM(),
transitions={'LOADED':'SENDMSG'})
smach.StateMachine.add('SENDMSG', startMSG(),
transitions={'SENT':'SENDMSG',
'ENDING':'END'})
smach.StateMachine.add('END', stopFSM(),
transitions={'DONE':'complete',
'ERRED':'incomplete'})
Code snippet below showing a smach state (loadFSM):
class loadFSM(smach.State):
def __init__(self):
smach.State.__init__(self, outcomes=['LOADED'],
output_keys=['talker_obj'],
input_keys=['talker_obj'])
# Initialise our talker object
self.talker = Talk()
def execute(self, userdata):
rospy.loginfo("talker state: Loading fsm")
self.talker.init_publish()
self.talker.init_subscribe()
userdata.talker_obj = self.talker
return 'LOADED'
The errors I receive (using Ubuntu 14.04, ROS indigo and python 2.7, not too sure but I believe the same errors occur in kinetic as well) only occur during state transitions and of course the introspective server does not work (does not show state transitions). The errors are;
1. "Exception in thread sm_introViewer:status_publisher:"
2. "Could not execute transition callback: Traceback (most recent call
last): File
"/opt/ros/indigo/lib/python2.7/dist-packages/smach/container.py",
line 175, in call_transition_cbs cb(self.userdata,
self.get_active_states(), *args)"
I also need to add, that my simple finite state machine example actually works and completes successfully, even my project's 2 larger FSMs complete. However, when the FSM has many states like in my project, sometimes my simulations fail. I would like to know from anyone who has used SMACH extensively if they think these errors are the cause or if they know for a fact that I am not passing the object correctly between states.
Thanks in Advance,
Tiz
I had similar problems in the past. If I remember correctly, userdata can only handle basic types (int, string, list, dict, ...) but not arbitrary objects.
I solved it by passing the objects to the constructor of state classes, instead of using the userdata, i.e. something like:
class myState(smach.State):
def __init__(self, obj):
smach.State.__init__(self, outcomes=['foo'], ...)
self.obj = obj
...
and then initialize like follows:
obj = SomeClass()
with sm:
smach.StateMachine.add('MY_STATE', myState(obj),
transitions={'foo':'bar'})
It is not so nice as it circumvents the userdata concept but it works.
Im using python 2.7.9, windows 7...
The overall goal: Have another application access our custom com server (already running at this point) and send it a message to be displayed. Obviously, there needs to be a single server, multiple clients.
Im trying to use some custom code as a com server. The class was created as:
class StatusServerClass:
_public_methods_ = ...
_reg_progid_ = "CseStatusServerLib.CseStatusServer"
_reg_verprogid_ = "CseStatusServerLib.CseStatusServer"
_reg_progid_ = "CseStatusServerLib.CseStatusServer"
_reg_clsid_ = "{<GUID here>}"
_reg_desc_ = 'CSE Status Server'
_typelib_guid_ = "{<typelib GUID here>}"
_typelib_version_ = 1, 0 # Version 1.0
_reg_clsctx_ = pythoncom.CLSCTX_LOCAL_SERVER
_reg_threading_ = "Apartment" # Not used?
def __init__.....
and registered using:
win32com.server.register.UseCommandLine(StatusServerClass)
I can see it in regedit and as far as i know, it looks ok.
The GUID is right, name is right.
Now when i go to use it, this works just fine:
self.StatusClient = Dispatch('CseStatusServerLib.CseStatusServer')
but when i want to attach to a running instance from another exe (or even another python window for debug) using:
win32com.client.GetActiveObject("CseStatusServerLib.CseStatusServer")
it just gives me:
dispatch = pythoncom.GetActiveObject(resultCLSID)
com_error: (-2147221021, 'Operation unavailable', None, None)
Tells me that its not registered?
Ive tried using the GUID, Ive tried using pythoncom.GetObject with both the ID and the GUID... no luck.
Ive tried comtypes package and get the same thing.
Any ideas on what im doing wrong? Why does Dispatch find it by name, but GetActiveObject gets mad?
Seems that Dispatch working by name would suggest that the registering worked?
What else can i verify in the regedit?
thanks!!!
UPDATED 6/6/2016
In case you haven't realized yet, I know very little about this. But I have read that for win32com.client.GetActiveObject() to work, the server needs to be in the "running object table"... and its not.
So, I found some more example code that i used to register the class this way:
import win32com.server.util
wrapped = win32com.server.util.wrap(StatusServerClass)
flags = pythoncom.REGCLS_MULTIPLEUSE|pythoncom.REGCLS_SUSPENDED
handle = pythoncom.RegisterActiveObject(wrapped,
StatusServerClass._reg_clsid_,flags)
and that does allow the server to show in the running object table, and i can get this:
testSW = win32com.client.GetActiveObject("CseStatusServerLib.CseStatusServer")
to return without error.
So now, I can use Dispatch or GetActiveObject just fine in python, pythonWin, and even interact with the server in Excel/VB <-> python and it appears to share namespsace.
BUT...
I still cant get this IE-based third party app to use the existing server... even with GetActiveObject. Always wants to create new instance and use its own namespace... not good
Is there something with IE or Chrome that would prevent the existing server from being used? Again, it works fine in Excel/VB. The application is supposed to execute "python myscript.py" (which works fine in idle, pythonwin and cmdline), but doesnt execute the com server stuff when called from IE/Chrome App (although other python functions like file writing work just fine).
Also, seeing as how i know very little about this, by all means, suggest a better way of doing this: starting a server in python as a singleton and then accessing it from another application.
thanks!!
I have been developing a PyQt5 software for a while now. I have managed to package my Python 3 PyQt5 software with py2exe fine in past with everything working perfectly.
However, now I have encountered an issue where the packaged exe-program will crash when user exits. More specifically I get APPCRASH with following details
Problem signature:
Problem Event Name: APPCRASH
Application Name: Sotilasmatrikkelit.exe
Application Version: 0.0.0.0
Application Timestamp: 54467a51
Fault Module Name: PyQt5.QtCore.pyd
Fault Module Version: 0.0.0.0
Fault Module Timestamp: 549be77e
Exception Code: c0000005
Exception Offset: 0010c185
OS Version: 6.1.7601.2.1.0.256.4
Locale ID: 1035
Additional Information 1: 0a9e
Additional Information 2: 0a9e372d3b4ad19135b953a78882e789
Additional Information 3: 0a9e
Additional Information 4: 0a9e372d3b4ad19135b953a78882e789
I never get this crash when running the software from Pycharm during the development. Any idea of how to debug this or what could be the cause in general?
I suspect this might have something to do with memory management (that PyQt doesn't delete all the resources properly on exit and therefore segfaults) but does anyone have any good suggestions to figure out the exact problem without better debug information? Should I try to do some kind of cleanup on exit? Atm I start the software like this:
def start():
import sys
app = QApplication(sys.argv)
fixingtool = Mainwindow(app)
fixingtool.show()
sys.exit(app.exec_())
Additional investigation seems to suggest that I get Access Violation which is caused by C++ null-pointer. Sounds scary since I don't know too much of PyQt debugging. Anyway, I found an area on my businesslogic code which if removed will remove the problem. However, this code has nothing to do with PyQt and is just regular Python code and shouldn't differ in any way. Strangest part is that if I remove certain functions from the code, problem disappears even though functions are not called during runtime meaning that just importing the file with those functions cause the problem. Below is a code sample:
import re
from books.karelians.extraction.extractors.baseExtractor import BaseExtractor
from books.karelians.extraction.extractionExceptions import *
from books.karelians.extraction.extractors.dateExtractor import DateExtractor
from shared import textUtils
from books.karelians.extractionkeys import KEYS
from interface.valuewrapper import ValueWrapper
from shared import regexUtils
from books.karelians.extraction.extractors.professionextractor import ProfessionExtractor
class SpouseExtractor(BaseExtractor):
def extract(self, text, entry):
super(SpouseExtractor, self).extract(text)
self.entry = entry
self.PATTERN = r"Puol\.?(?P<spousedata>[A-ZÄ-Öa-zä-ö\s\.,\d-]*)(?=(Lapset|poika|tytär|asuinp))"
self.NAMEPATTERN = r"(?P<name>^[\w\s\.-]*)"
self.OPTIONS = (re.UNICODE | re.IGNORECASE) #TODO: TRY IGNORE CASE?
self.REQUIRES_MATCH_POSITION = False
self.SUBSTRING_WIDTH = 100
self.hasSpouse = False
self.spouseName = ""
self.profession = {KEYS["profession"] : ValueWrapper("")}
self.initVars(text)
self._findSpouse(text)
return self._constructReturnDict()
def initVars(self,text):
pass
def _findSpouse(self, text):
try:
self.foundSpouse = regexUtils.safeSearch(self.PATTERN, text, self.OPTIONS)
self.hasSpouse = True
self._findSpouseName(self.foundSpouse.group("spousedata"))
self._setFinalMatchPosition()
except regexUtils.RegexNoneMatchException:
pass
def _findSpouseName(self, text):
try:
name = regexUtils.safeSearch(self.NAMEPATTERN, text, self.OPTIONS)
self.spouseName = name.group("name").strip()
self._findProfession(text[name.end():])
except regexUtils.RegexNoneMatchException:
self.errorLogger.logError(SpouseNameException.eType, self.currentChild)
def _findProfession(self, text):
professionExt = ProfessionExtractor(self.entry, self.errorLogger, self.xmlDocument)
professionExt.setDependencyMatchPositionToZero()
self.profession = professionExt.extract(text, self.entry)
def _setFinalMatchPosition(self):
#Dirty fix for inaccuracy in positions which would screw the Location extraction
self.matchFinalPosition = self.foundSpouse.end() + self.matchStartPosition - 4
def _constructReturnDict(self):
print(self.profession)
return {KEYS["spouse"]: ValueWrapper({ KEYS["hasSpouse"]: ValueWrapper(self.hasSpouse),KEYS["spouseName"]: ValueWrapper(self.spouseName), KEYS["spouseProfession"]: ValueWrapper(self.profession[KEYS["profession"]].value) })}
Here if I remove or comment away all functions "initVars()" program exits properly. What gives?
This isn't really a solution to a problem itself but I'm leaving this here in case someone encounters a similar issue.
Today I decided to try cx_freeze instead of py2exe figuring that maybe the problem is with py2exe since the whole issue is not happening when running the application with normal python interpreter.
Turns out I was right and problem seemed to magically disappear after I packaged the app using cx_freeze instead of py2exe. I didn't do any changes to code. Someone more knowledgeable than me about how py2exe and cx_freeze work might be able to explain the difference. My wild guess is that for some reason the exiting from the Python interpreter is not handled perfectly in py2exe case somehow messing the end garbage cleanup. I have no idea if this is py2exe's or my fault by not configuring py2exe properly.
In any case I'm happy it works now since yesterday was a really frustrating day.