U-SQL, Python, local execution, "device not found" error - python

I'm trying to run a U-SQL job with Python extension locally using VS 2017.
I followed these steps:
https://1drv.ms/w/s!AvdZLquGMt47g0NultCKgm38sejs
https://blogs.msdn.microsoft.com/azuredatalake/2017/02/20/enabling-u-sql-advanced-analytics-for-local-execution/
And then I tried to run this:
https://learn.microsoft.com/en-us/azure/data-lake-analytics/data-lake-analytics-u-sql-python-extensions
It works fine if I run it in Azure, but if I try to run it locally, the error I get is: "The device is not read".
Details:
Start : 2017-08-16 14:35:13
Initialize : 2017-08-16 14:35:13
GraphParse : 2017-08-16 14:35:13
Run : 2017-08-16 14:35:13
Start 'Root' : 2017-08-16 14:35:13
End 'Root(Success)' : 2017-08-16 14:35:13
Start '1_SV1_Extract' : 2017-08-16 14:35:13
End '1_SV1_Extract(Error)' : 2017-08-16 14:35:14
Completed with 'Error' : 2017-08-16 14:35:14
Execution failed with error '1_SV1_Extract Error : '{"diagnosticCode":195887147,"severity":"Error","component":"RUNTIME","source":"User","errorId":"E_RUNTIME_USER_UNHANDLED_EXCEPTION_FROM_USER_CODE","message":"An unhandled exception from user code has been reported","description":"Unhandled exception from user code: \"The device is not ready.\r\n\"\nThe details includes more information including any inner exceptions and the stack trace where the exception was raised.","resolution":"Make sure the bug in the user code is fixed.","helpLink":"","details":"==== Caught exception System.IO.IOException\n\n at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)\r\n at System.IO.Directory.InternalCreateDirectory(String fullPath, String path, Object dirSecurityObj, Boolean checkHost)\r\n at System.IO.Directory.InternalCreateDirectoryHelper(String path, Boolean checkHost)\r\n at System.IO.Compression.ZipFileExtensions.ExtractToDirectory(ZipArchive source, String destinationDirectoryName)\r\n at System.IO.Compression.ZipFile.ExtractToDirectory(String sourceArchiveFileName, String destinationDirectoryName, Encoding entryNameEncoding)\r\n at Microsoft.MetaAnalytics.LanguageWorker.UsqlPyExecution.LocatePython(String version) in C:\\Users\\shravan\\Source\\Repos\\VSTS\\USqlExtensions\\lang\\python\\AFx\\Product\\Source\\Modules\\LanguageWorker\\LanguageWorker.Dll\\UsqlExecution.cs:line 146\r\n at Microsoft.MetaAnalytics.LanguageWorker.UsqlPyExecution.InvokeLanguage(String version, String scriptname, IList`1 infiles, IList`1 outfiles, IObserver`1 stringLogger) in C:\\Users\\shravan\\Source\\Repos\\VSTS\\USqlExtensions\\lang\\python\\AFx\\Product\\Source\\Modules\\LanguageWorker\\LanguageWorker.Dll\\UsqlExecution.cs:line 89\r\n at Microsoft.MetaAnalytics.LanguageWorker.UsqlPyExecution.Run(IRowset input, IUpdatableRow output, String script, String version) in C:\\Users\\shravan\\Source\\Repos\\VSTS\\USqlExtensions\\lang\\python\\AFx\\Product\\Source\\Modules\\LanguageWorker\\LanguageWorker.Dll\\UsqlExecution.cs:line 42\r\n at Extension.Python.Reducer.<Reduce>d__6.MoveNext() in C:\\Users\\shravan\\Source\\Repos\\VSTS\\USqlExtensions\\lang\\python\\ExtPy\\PyReducer.cs:line 56\r\n at ScopeEngine.SqlIpReducer<Extract_0_Data0,Process_1_Data0,ScopeEngine::KeyComparePolicy<Extract_0_Data0,3> >.GetNextRow(SqlIpReducer<Extract_0_Data0\\,Process_1_Data0\\,ScopeEngine::KeyComparePolicy<Extract_0_Data0\\,3> >* , Process_1_Data0* output) in c:\\users\\e\\source\\repos\\usqlapplication1\\usqlapplication1\\bin\\debug\\1b720f51a8b3caea\\script_fe316531c87f021f\\sqlmanaged.h:line 2788\r\n at std._Func_class<void>.()(_Func_class<void>* )\r\n at RunAndHandleClrExceptions(function<void __cdecl(void)>* code)","internalDiagnostics":""}
'
'
Execution failed !
I'm aware that the blog post mentions that running Python extensions locally is not officially supported, but they do make it sound like it should at least be possible somehow?
I don't get any errors if I run U-SQL scripts without using the Python extension locally.
Is there anything I'm missing? Is there any logging I could turn on to find out more? Has anyone had success running Python with U-SQL locally?

(Azure Data Lake team here)
There was a recent update in how the Python distribution is located in the Azure Data Lake Analytics service. While the change improved vertex startup times, it also broke some basic assumptions on how local execution of U-SQL scripts works.
The team is working on an alternative solution that will let a locally executing U-SQL script use an existing Python distribution that is installed on the same local machine.

Related

Error : : (-2147417848, 'The object invoked has disconnected from its clients.', None, None) Python2.7 Win-32com

I'm trying to run a python2.7 code to open a Excel file and pass args to Macro then to fetch the output and save it in csv file. This runs successfully in local windows 10 machine. I shifted the code from local machine to a windows server recently. when I try to compile it there it throws error.
Code :
import win32com.client
from win32com.client.gencache import EnsureDispatch
# send message to excel macro, save data as csv for each Ts
loc = path
#print loc
# open excel, run macro called "external_Run" with argList
#xls=win32com.client.Dispatch("Excel.Application")
xls = EnsureDispatch("Excel.Application")
wb = xls.Workbooks.Open(Filename=loc)
try:    
xls.Application.Run("external_Run",argList)
except Exception as e:    
print "--------------- ERROR ------------------"    
print(e)    
print "=> No data was found, please check your input file"    
raise        
#xls.Visible = True
#wb.close()
# disable asking dialog when close excel
xls.DisplayAlerts = False
# export first sheet (macro output) to csv
w=wb.Worksheets(1)
w.SaveAs(path +'#'+'.csv')
#w.close()
xls.Application.Quit()
del xls
In windows local machine code runs successfully with xls=win32com.client.Dispatch("Excel.Application") this option. But in server this option throws Attribute Error. So I changed Client.Dispatch to EnsureDispatch. After changed to EnsureDispatch the code always appears to work on couple of files for the first time that it is run, and it throws errors or the unexpected behaviors during subsequent calls to the same code. Like if the code has to run for 10 times it runs for 3 or 4 times successfully and creates output.But on 5th time it throws error.
Error:
File "C:\Users\subrta1\Documents\01_OMT_Calculator\callLedCalculator.py", line 27, in messageToCal
w=wb.Worksheets(1)
File "C:\Python27\Scripts\MHEGA\lib\site-packages\win32com\client\__init__.py", line 513, in __getattr__
if d is not None: return getattr(d, attr)
File "C:\Python27\Scripts\MHEGA\lib\site-packages\win32com\client\__init__.py", line 474, in __getattr__
return self._ApplyTypes_(*args)
File "C:\Python27\Scripts\MHEGA\lib\site-packages\win32com\client\__init__.py", line 467, in _ApplyTypes_
self._oleobj_.InvokeTypes(dispid, 0, wFlags, retType, argTypes, *args),
com_error: (-2147417848, 'The object invoked has disconnected from its clients.', None, None)
Kindly Help me finding out why this is happening and how to work around it?
This runs successfully in local windows 10 machine. I shifted the code from local machine to a windows server recently.
Microsoft does not currently recommend, and does not support, Automation of Microsoft Office applications from any unattended, non-interactive client application or component (including ASP, ASP.NET, DCOM, and NT Services), because Office may exhibit unstable behavior and/or deadlock when Office is run in this environment.
If you are building a solution that runs in a server-side context, you should try to use components that have been made safe for unattended execution. Or, you should try to find alternatives that allow at least part of the code to run client-side. If you use an Office application from a server-side solution, the application will lack many of the necessary capabilities to run successfully. Additionally, you will be taking risks with the stability of your overall solution.
Read more about that in the Considerations for server-side Automation of Office article.
As a possible workaround you may consider using the Open XML SDK if you deal with open XML documents only, see Welcome to the Open XML SDK 2.5 for Office for more information. Also consider third-party components designed for the server-side execution.

Metatrader5 - Python Integration - symbol_total() returns None

I'm trying to get the number of symbols of metatrader5 and I'm getting an error
TypeError: '>' not supported between instances of 'NoneType' and 'int'
Link to the documentation: https://www.mql5.com/en/docs/integration/python_metatrader5/mt5symbolstotal_py
code:
import MetaTrader5 as mt5
print("MetaTrader5 package author: ",mt5.__author__)
print("MetaTrader5 package version: ",mt5.__version__)
if not mt5.initialize():
print("initialize() failed, error code =",mt5.last_error())
quit()
symbols=mt5.symbols_total()
if symbols>0:
print("Total symbols =",symbols)
else:
print("symbols not found")
mt5.shutdown()
The problem is that the function is returning NoneType instead of a number.
Why it's returning a NoneType? How can i get the list of Symbols/Stocks?
Any clue?
I had the same problem too. If you are currently using a downloaded MT5 terminal from your broker, you can try using the official MT5 terminal instead. That seemed to have fixed my issue. Don't forget to specify the path to the correct MT5 terminal.exe afterwards within the initialization function initialize(path=...).
As to why this was causing an issue, I'm unsure myself. I happened across this post and it mentioned that there may have been modifications made by brokers.
Anyway, hope this works for you too!
To connect to your broker's server afterwards, within the MT5 terminal under Navigator->Accounts (Right Click)->Open an account->Search for your broker and enter your credentials.

Microsoft Face API Python poll_for_event: Assertion `!xcb_xlib_threads_sequence_lost' failed

I am trying to run the sample of Face API from https://github.com/Microsoft/Cognitive-Face-Python on Ubuntu 12.04 running on the virtual machine (VB). When I select the image to detect, it will send a request and will receive the results but after that:
sample/util.py:176: wxPyDeprecationWarning: Call to deprecated item EmptyImage. Use :class:`Image` instead.
wx_image = wx.EmptyImage(pil_image.width, pil_image.height)
[xcb] Unknown sequence number while processing queue
[xcb] Most likely this is a multi-threaded client and XInitThreads has not been called
[xcb] Aborting, sorry about that.
python: ../../src/xcb_io.c:274: poll_for_event: Assertion `!xcb_xlib_threads_sequence_lost' failed.
Aborted (core dumped)`
I have detected the place where it stops:
try:
attributes = (
'age,gender,headPose,smile,facialHair,glasses,emotion,hair,'
'makeup,occlusion,accessories,blur,exposure,noise'
)
# the call
res = util.CF.face.detect(path, False, False, attributes)
faces = [model.Face(face, path) for face in res]
# ... WON'T GET HERE ...
self.face_list.SetItems(faces)
util.draw_bitmap_rectangle(self.bitmap, faces)
log_text = 'Response: Success. Detected {} face(s) in {}'.format(
len(res), path)
self.log.log(log_text)
text = '{} face(s) has been detected.'.format(len(res))
self.result.SetLabelText(text)
except util.CF.CognitiveFaceException as exp:
self.log.log('Response: {}. {}'.format(exp.code, exp.msg))
This snippet is from the sample/view/panel_detection.py
I failed to reproduce your error with the latest version of code on Windows 10 & macOS 10.12.5 with Python 2.7 and wxPython 3.0.2.0.
The error is not reported directly from the sample code but somewhere inside. And wx.EmptyImage should not be the key point of the error. So I suspect it might has something to do with the environments.
Please ensure you are using the latest version of sample code and corresponding packages. For example, 3.0.2.0 for the wxPython. Ubuntu 12.04 is too old and may suffer on outdated dependencies.
I have to admit that we have limited effort to make it back compatible so please ensure everything is up-to-date. Also feel free to update if there is any further problems.

Named Parameters in SQL Queries - cx_Oracle - ORA-01460: unimplemented or unreasonable conversion requested

I have encountered a problem after implementing the named parameters in RAW SQL Queries as per Python DB-API.
Earlier, my code was as follows (and this works fine, both on my DEV Server and my Client's test server)
cursor.execute("SELECT DISTINCT(TAG_STATUS) FROM TAG_HIST WHERE TAG_NBR = '%s' " %(TAG_NBR))
I changed it to the following
cursor.execute("SELECT DISTINCT(TAG_STATUS) FROM TAG_HIST WHERE TAG_NBR = :TAG_NBR " ,{'TAG_NBR':TAG_NBR})
This changed version (with named parameters) works fine on my Development Server
Windows XP Oracle XE
SQL*Plus: Release 11.2.0.2.0
cx_Oracle-5.1.2-11g.win32-py2.7
However, when deployed on my Client's Test Server, it does not.... execution of all queries fail.
Characteristics of my client's server are as follows
Windows Server 2003
Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - 64bi
cx_Oracle-5.1.2-10g.win32-py2.7
The error that I get is as follows
Traceback (most recent call last):
File "C:/Program Files/App_Logic/..\apps\views.py", line 400, in regularize_TAG
T_cursor.execute("SELECT DISTINCT(TAG_STATUS) FROM TAG_HIST WHERE TAG_NBR = :TAG_NBR " ,{'TAG_NBR':TAG_NBR})
DatabaseError: ORA-01460: unimplemented or unreasonable conversion requested
Appreciate if someone could help me through this issue.
This issue presents itself only when the cx_Oracle code is run inside the Web App (Hosted on Apache).
If i run the same code with named parameters from within the python command line then the query runs just fine.
Here is how this got solved.
I tried typecasting unicode to str and the results were positive.
This one worked for example
T_cursor.execute("SELECT DISTINCT(TAG_STATUS) FROM TAG_HIST WHERE TAG_NBR = :TAG_NBR", {'TAG_NBR': str(TAG_NBR)})
So in effect, unicode was getting mangled by getting encoded into the potentially non-unicode database character set.
To solve that, here is another option
import os
os.environ.update([('NLS_LANG', '.UTF8'),('ORA_NCHAR_LITERAL_REPLACE', 'TRUE'),])
import cx_Oracle
Above guarantees that we are really in UTF8 mode.
Second environment variable one is not an absolute necessity. And AFAIK there is no other way to set these variables (except before running app itself) due the fact that NLS_LANG is
read by OCI libs from the environment.

Cant run Python Views on Couchdb

i started using couchdb with python-couchdb recently. The problem is when i use futon run my views written in python i get the following error message:
Error: os_process_error
{exit_status,1}
even for the default view it crashes.
def fun(doc):
yield None, doc
i haven't yet found much information regarding this issue so at this point im really lost. This is the log i get from couchdb:
{<0.3907.0>,crash_report,
[[{initial_call,{couch_file,init,['Argument__1']}},
{pid,<0.3907.0>},
{registered_name,[]},
{error_info,
{exit,
{os_process_error,{exit_status,1}},
[{gen_server,terminate,6},{proc_lib,init_p_do_apply,3}]}},
{ancestors,
[<0.3906.0>,couch_view,couch_secondary_services,couch_server_sup,
<0.33.0>]},
{messages,[]},
{links,[#Port<0.1483>,<0.3910.0>]},
{dictionary,[]},
{trap_exit,true},
{status,running},
{heap_size,377},
{stack_size,24},
{reductions,1423}],
[{neighbour,
[{pid,<0.3910.0>},
{registered_name,[]},
{initial_call,{couch_ref_counter,init,['Argument__1']}},
{current_function,{gen_server,loop,6}},
{ancestors,
[<0.3906.0>,couch_view,couch_secondary_services,
couch_server_sup,<0.33.0>]},
{messages,[]},
{links,[#Port<0.1483>,<0.3910.0>]},
{dictionary,[]},
{trap_exit,true},
{status,running},
{heap_size,377},
{stack_size,24},
{reductions,1423}],
[{neighbour,
[{pid,<0.3910.0>},
{registered_name,[]},
{initial_call,{couch_ref_counter,init,['Argument__1']}},
{current_function,{gen_server,loop,6}},
{ancestors,
[<0.3906.0>,couch_view,couch_secondary_services,
couch_server_sup,<0.33.0>]},
{messages,
[{'DOWN',#Ref<0.0.0.16475>,process,<0.3906.0>,
{os_process_error,{exit_status,1}}}]},
{links,[<0.3907.0>]},
{dictionary,[]},
{trap_exit,false},
{status,runnable},
{heap_size,233},
{stack_size,9},
{reductions,47}]}]]}}
Im running this on Ubuntu 10.04, with Django, couchdb and python-couchdb. The views on javascript works fine.
For couchdb-python query server exit status 1 means some error that.
What version of couchdb/couchdb-python you're uses?
Whats output will be if you run couchpy (or /usr/local/bin/couchpy or whatever you have setted in query_servers section for python key) directly from command line? Example as it should be:
$
~ $ couchpy
["reset"]
true
["add_fun", "def fun(doc): yield None, None"]
true
["map_doc", {}]
[[[null, null]]]
If p.2 works fine, try to enable CouchDB debug log level to trace query server commands and notice when it falls. If you're sure that this is a python query server bug, please write some story about it. Thanks(:
P.S. I hope that you have setup python query server correctly, but double things check never was useless(;

Categories