We Keep Coding

ModuleNotFoundError while using AzureML pipeline with yml file based RunConfiguration and environment.yml

I am running into a ModuleNotFoundError for pandas while using the following code to orchestrate my Azure Machine Learning Pipeline:
# Loading run config
print("Loading run config")
task_1_run_config = RunConfiguration.load(
os.path.join(WORKING_DIR + '/pipeline/task_runconfigs/T01_Test_Task.yml')
)
task_1_script_run_config = ScriptRunConfig(
source_directory=os.path.join(WORKING_DIR + '/pipeline/task_scripts'),
run_config=task_1_run_config
)
task_1_py_script_step = PythonScriptStep(
name='Task_1_Step',
script_name=task_1_script_run_config.script,
source_directory=task_1_script_run_config.source_directory,
compute_target=compute_target
)
pipeline_run_config = Pipeline(workspace=workspace, steps=[task_1_py_script_step])#, task_2])
pipeline_run = Experiment(workspace, 'Test_Run_New_Pipeline').submit(pipeline_run_config)
pipeline_run.wait_for_completion()
The environment.yml
name: phinmo_pipeline_env
dependencies:
- python=3.8
- pip:
- pandas
- azureml-core==1.43.0
- azureml-sdk
- scipy
- scikit-learn
- numpy
- pyyaml==6.0
- datetime
- azure
channels:
- conda-forge
The loaded RunConfiguration in T01_Test_Task.yml looks like this:
# The script to run.
script: T01_Test_Task.py
# The arguments to the script file.
arguments: [
"--test", False,
"--date", "2022-07-26"
]
# The name of the compute target to use for this run.
compute_target: phinmo-compute-cluster
# Framework to execute inside. Allowed values are "Python", "PySpark", "CNTK", "TensorFlow", and "PyTorch".
framework: Python
# Maximum allowed duration for the run.
maxRunDurationSeconds: 6000
# Number of nodes to use for running job.
nodeCount: 1
#Environment details.
environment:
# Environment name
name: phinmo_pipeline_env
# Environment version
version:
# Environment variables set for the run.
#environmentVariables:
# EXAMPLE_ENV_VAR: EXAMPLE_VALUE
# Python details
python:
# user_managed_dependencies=True indicates that the environmentwill be user managed. False indicates that AzureML willmanage the user environment.
userManagedDependencies: false
# The python interpreter path
interpreterPath: python
# Path to the conda dependencies file to use for this run. If a project
# contains multiple programs with different sets of dependencies, it may be
# convenient to manage those environments with separate files.
condaDependenciesFile: environment.yml
# The base conda environment used for incremental environment creation.
baseCondaEnvironment: AzureML-sklearn-0.24-ubuntu18.04-py37-cpu
# Docker details
# History details.
history:
# Enable history tracking -- this allows status, logs, metrics, and outputs
# to be collected for a run.
outputCollection: true
# Whether to take snapshots for history.
snapshotProject: true
# Directories to sync with FileWatcher.
directoriesToWatch:
- logs
# data reference configuration details
dataReferences: {}
# The configuration details for data.
data: {}
# Project share datastore reference.
sourceDirectoryDataStore:
I already tried a few things like overwriting the environment attribute in the RunConfiguration object with a environment.python.conda_dependencies object or assigning a version number to pandas in the environment.yml, changing the location of the environment.yml. But I am at a loss at what else to try. the T01_Test_Task.py runs without issues on its own. But putting it into a pipeline just does not seem to work.

Okay I found the issue.
I am unnecessarily using the ScriptRunConfig which overwrites the assigned environment with some default azureml environment. I was able to see that only in the Task description in the Azure Machine Learning Studio UI.
I was able to just remove that part and now it works:
task_1_run_config = RunConfiguration.load(
os.path.join(WORKING_DIR + '/pipeline/task_runconfigs/T01_Test_Task.yml')
)
task_1_py_script_step = PythonScriptStep(
name='Task_1_Step',
script_name='T01_Test_Task.py',
source_directory=os.path.join(WORKING_DIR + '/pipeline/task_scripts'),
runconfig=task_1_run_config,
compute_target=compute_target
)

PyInstaller & Pymeasure : NotImplementedError

I am currently experiencing difficulties using PyInstaller on a code relying on Pymeasure library. The program is working fine from the prompt, but not when started from the executable generated by PyInstaller.
Here is an simple example of a code working from prompt but not when frozen:
import visa
from pymeasure.instruments.keithley import Keithley2000, Keithley2400
rm = visa.ResourceManager()
list_available = rm.list_resources()
print(list_available)
keithley = Keithley2400("GPIB1::23")
keithley.apply_current() # Sets up to source current
keithley.source_current_range = 10e-3 # Sets the source current range to 10 mA
keithley.compliance_voltage = 10 # Sets the compliance voltage to 10 V
keithley.source_current = 0 # Sets the source current to 0 mA
keithley.enable_source() # Enables the source output
keithley.measure_voltage() # Sets up to measure voltage
keithley.ramp_to_current(5e-3) # Ramps the current to 5 mA
print(keithley.voltage) # Prints the voltage in Volts
keithley.shutdown() # Ramps the current to 0 mA and disables output
Here is the output when I run the executable:
Please note that I have PyVISA 1.9.1 installed.
Why do I get this error and how do I fix that ?

You need to make sure you include the package metadata for PyVisa in your PyInstaller project. PyInstaller has a utility hook for that job; create a hook-pyvista.py hook file (if you don’t already have one) with:
from PyInstaller.utils.hooks import copy_metadata
datas = copy_metadata("pyvisa")
and tell PyInstaller about it with the --additional-hooks-dir command-line switch. See the documentation on how hooks work for more details.
pymeasurement relies on the pyvisa.__version__ attribute to determine if you have installed the correct version of that project. But pyvisa.__version__ defaults to "unknown" unless it can locate its metadata files, which would provide pkg_resources with the required metadata to retrieve the version for it.
You can verify that PyVisa was installed correctly by importing it yourself and testing the __version__ attribute:
import pyvisa
print("PyVisa version", pyvisa.__version__)

Are you sure you are connected to the instrument, your code references "GPIB1::23" but your print(list_available) returns "GPIB1::24"?

How does one set specific vim-bindings in Ipython 5.0.0

I understand that because Ipython 5.0.0 uses a new input library (prompt_toolkit) it no longer defaults to the editor mode specified in .inputrc (*nix). This option has to be set in an Ipython profile configuration file (see https://stackoverflow.com/a/38329940/2915339).
My question is: having set vi-mode in the profile configuration file, how does one specify a particular keybinding? I like to use 'jk' for escape, for instance.

You're right. prompt_toolkit ignores .inputrc. There does not seem to be a way to define custom keybindings for the vi mode in the IPython 5.0.0 profile configuration file.
Here's workaround I'm currently using. It's not pretty, but it works for now.
According to the IPython docs, you can specify Keyboard Shortcuts in a startup configuration script.
Instead of rebinding jk to ESC, I'm making a unicode "j" (u'j') followed by a unicode "k" (u'k') inside of VimInsertMode() a shortcut for a prompt_toolkit event that switches to navigation mode.
I created a .ipython/profile_default/startup/keybindings.py with the following code:
from IPython import get_ipython
from prompt_toolkit.enums import DEFAULT_BUFFER
from prompt_toolkit.filters import HasFocus, ViInsertMode
from prompt_toolkit.key_binding.vi_state import InputMode
ip = get_ipython()
def switch_to_navigation_mode(event):
vi_state = event.cli.vi_state
vi_state.reset(InputMode.NAVIGATION)
if getattr(ip, 'pt_cli'):
registry = ip.pt_cli.application.key_bindings_registry
registry.add_binding(u'j',u'k',
filter=(HasFocus(DEFAULT_BUFFER)
& ViInsertMode()))(switch_to_navigation_mode)
The prompt_toolkit source will help you implement other shortcuts as needed.

This is an old post but it helped me find my answer so I thought I would post how I added a couple of bindings to vi mode in ipython. I added the following code in ~/.ipython/profile_default/startup/00-keybindings.py to bind to K and J in vi navigation mode.
"""Improve history access so I can skip over functions"""
from IPython import get_ipython
from prompt_toolkit.enums import DEFAULT_BUFFER
from prompt_toolkit.filters import HasFocus, ViNavigationMode
from prompt_toolkit.key_binding.bindings.named_commands import get_by_name
ip = get_ipython()
registry = ip.pt_app.key_bindings
ph = get_by_name('previous-history')
nh = get_by_name('next-history')
registry.add_binding('K',
filter=(HasFocus(DEFAULT_BUFFER) &
ViNavigationMode()))(ph)
registry.add_binding('J',
filter=(HasFocus(DEFAULT_BUFFER) &
ViNavigationMode()))(nh)

Cannot properly create a spark context in ipython to link to MySQL - com.mysql.jdbc.Driver

I am running a local environment with Spark, PySpark, Ipython and mysql. I am strugling to be able to launch a mysql query via spark. The main issue is including the proper jdbc jar in order to be able to perform the query.
Here is what I have so far :
import pyspark
conf = (pyspark.SparkConf()
.setMaster('local')
.setAppName('Romain_DS')
.set("spark.executor.memory", "1g")
.set("spark.driver.extraLibraryPath","mysql-connector-java-5.1.37/mysql-connector-java-5.1.37-bin.jar")
.set("spark.driver.extraClassPath","mysql-connector-java-5.1.37/mysql-connector-java-5.1.37-bin.jar")
)
sc = pyspark.SparkContext(conf=conf)
This is in order to properly create the spark context, and properly show the path to the jar including the jdbc driver.
Then I create an SQLContext :
from pyspark.sql import SQLContext
sqlsc=SQLContext(sc)
And finally the query :
MYSQL_USERNAME = "root";
MYSQL_PWD = "rootpass";
MYSQL_CONNECTION_URL = "jdbc:mysql://127.0.0.1:33060/O_Tracking?user=" + MYSQL_USERNAME + "&password=" + MYSQL_PWD;
query = 'Select * from tracker_action'
dataframe_mysql = sqlsc.read.format("jdbc").options(
url = MYSQL_CONNECTION_URL,
dbtable = "tracker_action",
driver = "com.mysql.jdbc.Driver",
user="root",
password="rootpass").load()
If I run this in the ipython notebook I get the error :
An error occurred while calling o198.load. :
java.lang.ClassNotFoundException: com.mysql.jdbc.Driver
However if I do everything from the shell (and not ipython), by initializing the spark context this way :
pyspark --driver-library-path './mysql-connector-java-5.1.37-bin.jar' --driver-class-path './mysql-connector-java-5.1.37-bin.jar'
It does work... I looked into the UI in Spark the configurations are the same. So I don't understand why would one work and not the other one ... Is there something to do with the runtime setting before the JVM ?
If I cannot find a proper solution, we could potentially think of running the sc in the shell and then use it from the ipython but I have no idea how to do that.
If someone can help me on that that would be great.
---- Hardware / Software
Mac OSX
Spark 1.5.2
Java 1.8.0
Python 2.7.10 :: Anaconda 2.3.0 (x86_64)
---- Sources to help:
https://gist.github.com/ololobus/4c221a0891775eaa86b0
http://spark.apache.org/docs/latest/configuration.html
Following the comments here is my conf file :
# Default system properties included when running spark-submit.
# This is useful for setting default environmental settings.
spark.driver.extraLibraryPath /Users/romainbui/mysql-connector-java-5.1.37/mysql-connector-java-5.1.37-bin.jar
spark.driver.extrClassPath /Users/romainbui/mysql-connector-java-5.1.37/mysql-connector-java-5.1.37-bin.jar
spark.AppName PySpark
spark.setMaster Local
--------- Solution ---------
Thanks to the comments I finally was able to properly have a working solution (and a clean one).
Step 1 : Creating a profile :
ipython profile create pyspark
Step 2 : Edit the profile startup script :
touch ~/.ipython/profile_pyspark/startup/00-pyspark-setup.py
Step 3 : Fill the file. Here I did something custom (thanks to the comments) :
import findspark
import os
import sys
findspark.init()
spark_home = findspark.find()
#spark_home = os.environ.get('SPARK_HOME', None)
sys.path.insert(0, spark_home + "/python")
# Add the py4j to the path.
# You may need to change the version number to match your install
sys.path.insert(0, os.path.join(spark_home, 'python/lib/py4j-0.8.2.1-src.zip'))
# Adding the library to mysql connector
packages = "mysql:mysql-connector-java:5.1.37"
os.environ["PYSPARK_SUBMIT_ARGS"] = "--packages {0} pyspark-shell".format(
packages
)
# Initialize PySpark to predefine the SparkContext variable 'sc'
execfile(os.path.join(spark_home, 'python/pyspark/shell.py'))
Then you can simply run the notebook with :
ipython notebook --profile=pyspark

I don't understand why would one work and not the other one ... Is there something to do with the runtime setting before the JVM ?
More or less. IPython configuration you've shown executes python/pyspark/shell.py which creates SparkContext (and some other stuff) and creates a JVM instance. When you create another context later it is using the same JVM and parameters like spark.driver.extraClassPath won't be used.
There are multiple ways can you handle this including passing arguments using PYSPARK_SUBMIT_ARGS or setting spark.driver.extraClassPath in $SPARK_HOME/conf/spark-defaults.conf.
Alternatively you can add following lines to 00-pyspark-setup.py before shell.py is executed:
packages = "mysql:mysql-connector-java:5.1.37"
os.environ["PYSPARK_SUBMIT_ARGS"] = "--packages {0} pyspark-shell".format(
packages
)
Setting --driver-class-path / --driver-library-path there should work as well.

Programmatically detect system-proxy settings on Windows XP with Python

I develop a critical application used by a multi-national company. Users in offices all around the globe need to be able to install this application.
The application is actually a plugin to Excel and we have an automatic installer based on Setuptools' easy_install that ensures that all a project's dependancies are automatically installed or updated any time a user switches on their Excel. It all works very elegantly as users are seldom aware of all the installation which occurs entirely in the background.
Unfortunately we are expanding and opening new offices which all have different proxy settings. These settings seem to change from day to day so we cannot keep up with the outsourced security guys who change stuff without telling us. It sucks but we just have to work around it.
I want to programatically detect the system-wide proxy settings on the Windows workstations our users run:
Everybody in the organisazation runs Windows XP and Internet Explorer. I've verified that everybody can download our stuff from IE without problems regardless of where they are int the world.
So all I need to do is detect what proxy settings IE is using and make Setuptools use those settings. Theoretically all of this information should be in the Registry.. but is there a better way to find it that is guaranteed not to change with people upgrade IE? For example is there a Windows API call I can use to discover the proxy settings?
In summary:
We use Python 2.4.4 on Windows XP
We need to detect the Internet Explorer proxy settings (e.g. host, port and Proxy type)
I'm going to use this information to dynamically re-configure easy_install so that it can download the egg files via the proxy.
UPDATE0:
I forgot one important detail: Each site has an auto-config "pac" file.
There's a key in Windows\CurrentVersion\InternetSettings\AutoConfigURL which points to a HTTP document on a local server which contains what looks like a javascript file.
The pac script is basically a series of nested if-statements which compare URLs against a regexp and then eventually return the hostname of the chosen proxy-server. The script is a single javascript function called FindProxyForURL(url, host)
The challenge is therefore to find out for any given server which proxy to use. The only 100% guaranteed way to do this is to look up the pac file and call the Javascript function from Python.
Any suggestions? Is there a more elegant way to do this?

Here's a sample that should create a bullet green (proxy enable) or red (proxy disable) in your systray
It shows how to read and write in windows registry
it uses gtk
#!/usr/bin/env python
import gobject
import gtk
from _winreg import *
class ProxyNotifier:
def __init__(self):
self.trayIcon = gtk.StatusIcon()
self.updateIcon()
#set callback on right click to on_right_click
self.trayIcon.connect('popup-menu', self.on_right_click)
gobject.timeout_add(1000, self.checkStatus)
def isProxyEnabled(self):
aReg = ConnectRegistry(None,HKEY_CURRENT_USER)
aKey = OpenKey(aReg, r"Software\Microsoft\Windows\CurrentVersion\Internet Settings")
subCount, valueCount, lastModified = QueryInfoKey(aKey)
for i in range(valueCount):
try:
n,v,t = EnumValue(aKey,i)
if n == 'ProxyEnable':
return v and True or False
except EnvironmentError:
break
CloseKey(aKey)
def invertProxyEnableState(self):
aReg = ConnectRegistry(None,HKEY_CURRENT_USER)
aKey = OpenKey(aReg, r"Software\Microsoft\Windows\CurrentVersion\Internet Settings", 0, KEY_WRITE)
if self.isProxyEnabled() :
val = 0
else:
val = 1
try:
SetValueEx(aKey,"ProxyEnable",0, REG_DWORD, val)
except EnvironmentError:
print "Encountered problems writing into the Registry..."
CloseKey(aKey)
def updateIcon(self):
if self.isProxyEnabled():
icon=gtk.STOCK_YES
else:
icon=gtk.STOCK_NO
self.trayIcon.set_from_stock(icon)
def checkStatus(self):
self.updateIcon()
return True
def on_right_click(self, data, event_button, event_time):
self.invertProxyEnableState()
self.updateIcon()
if __name__ == '__main__':
proxyNotifier = ProxyNotifier()
gtk.main()

As far as I know, In a Windows environment, if no proxy environment variables are set, proxy settings are obtained from the registry's Internet Settings section. .
Isn't it enough?
Or u can get something useful info from registry:
HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Internet Settings\ProxyServer
Edit:
sorry for don't know how to format comment's source code, I repost it here.
>>> import win32com.client
>>> js = win32com.client.Dispatch('MSScriptControl.ScriptControl')
>>> js.Language = 'JavaScript'
>>> js.AddCode('function add(a, b) {return a+b;}')
>>> js.Run('add', 1, 2)
3