We Keep Coding

Autoreload failed. ModuleNotFoundError : spec not found for the module

[autoreload of R17_Plot_profiles failed: Traceback (most recent call last):
File "/home/jdrevon/anaconda3/lib/python3.7/site-packages/IPython/extensions/autoreload.py", line 244, in check
superreload(m, reload, self.old_objects)
File "/home/jdrevon/anaconda3/lib/python3.7/site-packages/IPython/extensions/autoreload.py", line 378, in superreload
module = reload(module)
File "/home/jdrevon/anaconda3/lib/python3.7/imp.py", line 314, in reload
return importlib.reload(module)
File "/home/jdrevon/anaconda3/lib/python3.7/importlib/__init__.py", line 168, in reload
raise ModuleNotFoundError(f"spec not found for the module {name!r}", name=name)
ModuleNotFoundError: spec not found for the module 'R17_Plot_profiles'
]
This problem occurs in Python 3.7 on Spyder when I modify part of my code and I press F9 to execute some parts of it. Even if I execute my main python file which load every sub-python .py codes I have this error message. However, It does not stop the code from running but the file didn't save the modifications made. This is the first times it happens since I imported the 'R17_Plot_profiles.py' python file.
However in the case where I execute the main profile, I have the previous message but I have also the following one :
Reloaded modules: reconstruct, R2_Spice_Kernels, R4_Offset_Correction, R5_State_Vectors_twoway_v2, R6_Foot_Print, R7_Framework_Conversion,
R8_BendAng_ImpParam_up, R8_BendAng_ImpParam_dn, R9_Avg_BendAng_ImpParam, R10_Altitude_above_Ellipsoid, R11_Refractivity_and_Bending_Radius_v2,
R12a_Electron_Density_up, R12b_Electron_Density_dn, R12_Electron_Density_avg, R13_Neutral_Number_Density,
R14_Pressure_and_Temperature, R16_Errors, R17_Plot_profiles,
R3_Read_L2_Data_v2
So I don't know how to solve the autoreload problem. I've tried to downgrade my IPython version to 7.5 like some people tried and succeed but for me It doesn't change anything.
Edit : It's important to notice that if I restart the kernel, the first execution do not show the error message until I modify directly the python code.

I did few modifications yesterday and it seems to be solved. Since the error occurence started to be random at some points, i'm not sure for 100% but it can maybe solve the issue.
The problem could be came from the use of os.chdir function, I replaced it by specifying the entire path to save my figures until to go to the path using os.chdir and then save my figures in it.
It can maybe produce some conflicts of path and repository to find the python script which is no longer in the path where the script is currently working and therefore can't find it in order to save the modifications.
I keep you informed if at some point the error comes back but for now after some trials the error didn't show up anymore.

Problem with using Python library - Pyotodom

I wanted to use this library to scrape data from otodom. I've read the docs however I'm stuck on a very basic level, trying to do basic import. The code I use is taken directly from the docs. On top of that I've tried to launch example.py from Github but I'm getting the same error.
Here's the excerpt from the code:
import otodom, os, logging
from otodom.category import get_category
from otodom.offer import get_offer_information
Error message I get:
Traceback (most recent call last):
File ".\otodom_import.py", line 2, in
from otodom.category import get_category
File "C:\Users\Dom\Anaconda3\lib\site-packages\otodom\category.py", line 9, in
from otodom.utils import get_response_for_url, get_url
File "C:\Users\Dom\Anaconda3\lib\site-packages\otodom\utils.py", line 14, in
from scrapper_helpers.utils import caching, normalize_text, key_sha1,
get_random_user_agent
File "C:\Users\Dom\Anaconda3\lib\site-packages\scrapper_helpers\utils.py",
line 22, in
MAX_FILENAME_LENGTH = subprocess.check_output("getconf NAME_MAX /",
shell=True).strip()
File "C:\Users\Dom\Anaconda3\lib\subprocess.py", line 336, in check_output
**kwargs).stdout
File "C:\Users\Dom\Anaconda3\lib\subprocess.py", line 418, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command 'getconf NAME_MAX /' returned non-zero
exit status 1.
Thanks in advance!
EDIT: Don't understand downvotes, I realize that this is a niche module but I did my homework. I've read the docs and tried to find help in other places. If there is a chance that as a noobie I'll find at least one person that can help me, why wouldn't I use this opportunity? Even info that module is buggy can help.

getconf, which the dependency scrapper_helpers module is trying to call, is a POSIX command not available on Windows.
You can track it down in C:\Users\Dom\Anaconda3\lib\site-packages\scrapper_helpers\utils.py and replace the line 22 with:
MAX_FILENAME_LENGTH = 255
But then the question is what other error might pop up given that the module obviously expects to run on a POSIX-compatible system - for example, it will attempt to use /var/tmp/scrapper-helpers/ as its temporary cache path which is also not available on Windows (but Python will interpret it as <DRIVE_LETTER>:\var\tmp\scrapper-helpers and attempt to create it).
Alternatively, looking at its code, you might get away with just declaring the following environment variables:
set MAX_FILENAME_LENGTH=255
set CACHE_DIR=%TEMP%\scapper-helpers\

Access Azure blob storage from within an Azure ML experiment

Azure ML Experiments provide ways to read and write CSV files to Azure blob storage through the Reader and Writer modules. However, I need to write a JSON file to blob storage. Since there is no module to do so, I'm trying to do so from within an Execute Python Script module.
# Import the necessary items
from azure.storage.blob import BlobService
def azureml_main(dataframe1 = None, dataframe2 = None):
account_name = 'mystorageaccount'
account_key='mykeyhere=='
json_string='{jsonstring here}'
blob_service = BlobService(account_name, account_key)
blob_service.put_block_blob_from_text("upload","out.json",json_string)
# Return value must be of a sequence of pandas.DataFrame
return dataframe1,
However, this results in an error: ImportError: No module named azure.storage.blob
This implies that the azure-storage Python package is not installed on Azure ML.
How can I write to Azure blob storage from inside an Azure ML Experiment?
Here's the fill error message:
Error 0085: The following error occurred during script evaluation, please view the output log for more information:
---------- Start of error message from Python interpreter ----------
data:text/plain,Caught exception while executing function: Traceback (most recent call last):
File "C:\server\invokepy.py", line 162, in batch
mod = import_module(moduleName)
File "C:\pyhome\lib\importlib\__init__.py", line 37, in import_module
__import__(name)
File "C:\temp\azuremod.py", line 19, in <module>
from azure.storage.blob import BlobService
ImportError: No module named azure.storage.blob
---------- End of error message from Python interpreter ----------
Start time: UTC 02/06/2016 17:59:47
End time: UTC 02/06/2016 18:00:00`
Thanks, everyone!
UPDATE: Thanks to Dan and Peter for the ideas below. This is the progress I've made using those recommendations. I created a clean Python 2.7 virtual environment (in VS 2005), and did a pip install azure-storage to get the dependencies into my site-packages directory. I then zipped the site-packages folder and uploaded as the Zip file, as per Dan's note below. I then included the reference to the site-packages directory and successfully imported the required items. This resulted in a time out error when writing to blog storage.
Here is my code:
# Get access to the uploaded Python packages
import sys
packages = ".\Script Bundle\site-packages"
sys.path.append(packages)
# Import the necessary items from packages referenced above
from azure.storage.blob import BlobService
from azure.storage.queue import QueueService
def azureml_main(dataframe1 = None, dataframe2 = None):
account_name = 'mystorageaccount'
account_key='p8kSy3F...elided...3plQ=='
blob_service = BlobService(account_name, account_key)
blob_service.put_block_blob_from_text("upload","out.txt","Test to write")
# All of the following also fail
#blob_service.create_container('images')
#blob_service.put_blob("upload","testme.txt","foo","BlockBlob")
#queue_service = QueueService(account_name, account_key)
#queue_service.create_queue('taskqueue')
# Return value must be of a sequence of pandas.DataFrame
return dataframe1,
And here is the new error log:
Error 0085: The following error occurred during script evaluation, please view the output log for more information:
---------- Start of error message from Python interpreter ----------
data:text/plain,C:\pyhome\lib\site-packages\requests\packages\urllib3\util\ssl_.py:79: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
InsecurePlatformWarning
Caught exception while executing function: Traceback (most recent call last):
File "C:\server\invokepy.py", line 169, in batch
odfs = mod.azureml_main(*idfs)
File "C:\temp\azuremod.py", line 44, in azureml_main
blob_service.put_blob("upload","testme.txt","foo","BlockBlob")
File ".\Script Bundle\site-packages\azure\storage\blob\blobservice.py", line 883, in put_blob
self._perform_request(request)
File ".\Script Bundle\site-packages\azure\storage\storageclient.py", line 171, in _perform_request
resp = self._filter(request)
File ".\Script Bundle\site-packages\azure\storage\storageclient.py", line 160, in _perform_request_worker
return self._httpclient.perform_request(request)
File ".\Script Bundle\site-packages\azure\storage\_http\httpclient.py", line 181, in perform_request
self.send_request_body(connection, request.body)
File ".\Script Bundle\site-packages\azure\storage\_http\httpclient.py", line 143, in send_request_body
connection.send(request_body)
File ".\Script Bundle\site-packages\azure\storage\_http\requestsclient.py", line 81, in send
self.response = self.session.request(self.method, self.uri, data=request_body, headers=self.headers, timeout=self.timeout)
File "C:\pyhome\lib\site-packages\requests\sessions.py", line 464, in request
resp = self.send(prep, **send_kwargs)
File "C:\pyhome\lib\site-packages\requests\sessions.py", line 576, in send
r = adapter.send(request, **kwargs)
File "C:\pyhome\lib\site-packages\requests\adapters.py", line 431, in send
raise SSLError(e, request=request)
SSLError: The write operation timed out
---------- End of error message from Python interpreter ----------
Start time: UTC 02/10/2016 15:33:00
End time: UTC 02/10/2016 15:34:18
Where my current exploration is leading is that there is a dependency on the requests Python package in azure-storage. requests has a known bug in Python 2.7 for calling newer SSL protocols. Not sure, but I'm digging around in that area now.
UPDATE 2: This code runs perfectly fine inside of a Python 3 Jupyter notebook. Additionally, if I make the Blob Container open to public access, I can directly READ from the Container through a URL. For instance: df = pd.read_csv("https://mystorageaccount.blob.core.windows.net/upload/test.csv") easily loads the file from blob storage. However, I cannot use the azure.storage.blob.BlobService to read from the same file.
UPDATE 3: Dan, in a comment below, suggested I try from the Jupyter notebooks hosted on Azure ML. I had been running it from a local Jupyter notebook (see update 2 above). However, it fails when run from an Azure ML Notebook, and the errors point to the requires package again. I'll need to find the known issues with that package, but from my reading, the known issue is with urllib3 and only impacts Python 2.7 and NOT any Python 3.x versions. And this was run in a Python 3.x notebook. Grrr.
UPDATE 4: As Dan notes below, this may be an issue with Azure ML networking, as Execute Python Script is relatively new and just got networking support. However, I have also tested this on an Azure App Service webjob, which is on an entirely different Azure platform. (It is also on an entirely different Python distribution and supports both Python 2.7 and 3.4/5, but only at 32 bit - even on 64 bit machines.) The code there also fails, with an InsecurePlatformWarning message.
[02/08/2016 15:53:54 > b40783: SYS INFO] Run script 'ListenToQueue.py' with script host - 'PythonScriptHost'
[02/08/2016 15:53:54 > b40783: SYS INFO] Status changed to Running
[02/08/2016 15:54:09 > b40783: INFO] test.csv
[02/08/2016 15:54:09 > b40783: ERR ] D:\home\site\wwwroot\env\Lib\site-packages\requests\packages\urllib3\util\ssl_.py:315: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#snimissingwarning.
[02/08/2016 15:54:09 > b40783: ERR ] SNIMissingWarning
[02/08/2016 15:54:09 > b40783: ERR ] D:\home\site\wwwroot\env\Lib\site-packages\requests\packages\urllib3\util\ssl_.py:120: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
[02/08/2016 15:54:09 > b40783: ERR ] InsecurePlatformWarning
[02/08/2016 15:54:09 > b40783: ERR ] D:\home\site\wwwroot\env\Lib\site-packages\requests\packages\urllib3\util\ssl_.py:120: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
[02/08/2016 15:54:09 > b40783: ERR ] InsecurePlatformWarning

Bottom Line Up Front: Use HTTP instead of HTTPS for accessing Azure storage.
When declaring BlobService pass in protocol='http' to force the service to communicate over HTTP. Note that you must have your container configured to allow requests over HTTP (which it does by default).
client = BlobService(STORAGE_ACCOUNT, STORAGE_KEY, protocol="http")
History and credit:
I posted a query on this topic to #AzureHelps and they opened a ticket on the MSDN forums: https://social.msdn.microsoft.com/Forums/azure/en-US/46166b22-47ae-4808-ab87-402388dd7a5c/trouble-writing-blob-storage-file-in-azure-ml-experiment?forum=MachineLearning&prof=required
Sudarshan Raghunathan replied with the magic. Here are the steps to make it easy for everyone to duplicate my fix:
Download azure.zip which provides the required libraries: https://azuremlpackagesupport.blob.core.windows.net/python/azure.zip
Upload them as a DataSet to the Azure ML Studio
Connect them to the Zip input on an Execute Python Script module
Write your script as you would normally, being sure to create your BlobService object with protocol='http'
Run the Experiment - you should now be able to write to blob storage.
Some example code can be found here: https://gist.github.com/drdarshan/92fff2a12ad9946892df
The code I used was the following, which doesn't first write the CSV to the file system, but sends as a text stream.
from azure.storage.blob import BlobService
def azureml_main(dataframe1 = None, dataframe2 = None):
account_name = 'mystorageaccount'
account_key='p8kSy3FACx...redacted...ebz3plQ=='
container_name = "upload"
json_output_file_name = 'testfromml.json'
json_orient = 'records' # Can be index, records, split, columns, values
json_force_ascii=False;
blob_service = BlobService(account_name, account_key, protocol='http')
blob_service.put_block_blob_from_text(container_name,json_output_file_name,dataframe1.to_json(orient=json_orient, force_ascii=json_force_ascii))
# Return value must be of a sequence of pandas.DataFrame
return dataframe1,
Some thoughts:
I would prefer if the azure Python libraries were imported by default. Microsoft imports hundreds of 3rd party libraries into Azure ML as part of the Anaconda distribution. They should also include those necessary to work with Azure. We're in Azure, we've committed to Azure. Embrace it.
I don't like that I have to use HTTP, instead of HTTPS. Granted, this is internal Azure communication, so it's likely no big deal. However, most of the documentation suggests the use of SSL / HTTPS when working with blob storage, so I'd prefer to be able to do that.
I still get random timeout errors in the Experiment. Sometimes the Python code will execute in milliseconds, other times it runs for several 60 or seconds and then times out. This makes running it in an experiment very frustrating at times. However, when published as a Web Service I do not seem to have this problem.
I would prefer that the experience from my local code matched more closely Azure ML. Locally, I can use HTTPS and never time out. It's blazing fast, and easy to write. But moving to an Azure ML experiment means some debugging, nearly every time.
Huge props to Dan, Peter and Sudarshan, all from Microsoft, for their help in resolving this. I very much appreciate it!

You are going down the correct path. The Execution Python Script module is meant for custom needs just like this. Your real issue is how to import existing Python script modules. The complete directions can be found here, but I will summarize for SO.
You will want to take the Azure Python SDK and zip it up, upload, then import into your module. I can look into why this is not there by default...
https://azure.microsoft.com/en-us/documentation/articles/machine-learning-execute-python-scripts/
Importing existing Python script modules
A common use-case for many data scientists is to incorporate existing Python scripts into Azure Machine Learning experiments. Instead of concatenating and pasting all the code into a single script box, the Execute Python Script module accepts a third input port to which a zip file that contains the Python modules can be connected. The file is then unzipped by the execution framework at runtime and the contents are added to the library path of the Python interpreter. The azureml_main entry point function can then import these modules directly.
As an example, consider the file Hello.py containing a simple “Hello, World” function.
Figure 4. User-defined function.
Next, we can create a file Hello.zip containing Hello.py:
Figure 5. Zip file containing user-defined Python code.
Then, upload this as a dataset into Azure Machine Learning Studio. If we then create and run a simple experiment a uses the module:
Figure 6. Sample experiment with user-defined Python code uploaded as a zip file.
The module output shows that the zip file has been unpackaged and the function print_hello has indeed been run.  
Figure 7. User-defined function in use inside the Execute Python Script module.

As I know, you can use other packages via a zip file which you provide to the third input. The comments in the Python template script in Azure ML say:
If a zip file is connected to the third input port is connected, it is unzipped under ".\Script Bundle". This directory is added to sys.path. Therefore, if your zip file contains a Python file mymodule.py you can import it using:
import mymodule
So you can package azure-storage-python as a zip file thru click New, click Dataset, and then select From local file and the Zip file option to upload a ZIP file to your workspace.
As reference, you can see more information at the section How to Use Execute Python Script of the doc Execute Python Script.

Python ActiveX automation

I am trying to use ActiveX automation with python to control Audio Precision ATS-2. I am using pywin32 32-bit Windows XP with Python 2.7.
I installed the audio Precision software. Then I used makepy utility which found Audio Precision in the list and created a wrapper for it. Everything is going well until I attempt to call dispatch function abd I am stuck.
from guides I found online,
win32com.client.Dispatch("Excel.Application")
What should I put in the area that says "Excel.Application"?
How can I find this?
When I used OLE/COM Object viewer, I see Audio Precision entries under type library, but I am unsure how does the information in it could help me.
I have attempted few things on my own, but got almost no result. The best result I got was when I entered the CLSID in dispatch function, which I found in the wrapper object makepy created.
win32com.client.Dispatch("{80EC6E76-D94F-48EB-8F4C-05BDD7850BF1}")
which gave me the following error:
Traceback (most recent call last):
File "C:\Python27\Lib\site-packages\win32com\gen_py\APtest.py", line 5, in <module>
xlApp = win32com.client.Dispatch("{80EC6E76-D94F-48EB-8F4C-05BDD7850BF1}")
File "C:\Python27\lib\site-packages\win32com\client\__init__.py", line 95, in Dispatch
dispatch, userName = dynamic._GetGoodDispatchAndUserName(dispatch,userName,clsctx)
File "C:\Python27\lib\site-packages\win32com\client\dynamic.py", line 108, in _GetGoodDispatchAndUserName
return (_GetGoodDispatch(IDispatch, clsctx), userName)
File "C:\Python27\lib\site-packages\win32com\client\dynamic.py", line 85, in _GetGoodDispatch
IDispatch = pythoncom.CoCreateInstance(IDispatch, None, clsctx, pythoncom.IID_IDispatch)
com_error: (-2147221164, 'Class not registered', None, None)
I have:
looked under the registry, and the ID is indeed not registered anywhere. There are few Audio Precision entries, but doesn't look like it has anything that could fix this.
attempted reinstalling the Audio Precision software, but it didn't help.
attempted to register the dll objects in the Audio Precision control(ATS 1.60) software with regsrv32, but all gave errors and would not register(which I suspect aren't the right ones I need to register)
How can I fix this?

Possible way to solve this:
Identify the Audio Precision ATS-2 dll you want to call. E.g. "C:\path\my.dll"
Now go to the registry and look for "C:\path\my.dll". You'll find a few things around but what you really want is an entry in CLSID...\InprocServer32 folder, or something similar
There will be a ProgID entry as well, something like "AudioPrecison.ATS2"
Now in Python try win32com.client.Dispatch("AudioPrecison.ATS2")

To use the pywin32 module with an activex class you need to generate a wrapper file. To do this locate your makepy.py file in C:\Python27\Lib\site-packages\win32com\client (or your simmilar dir). Then run it, locate the NAME of your activex class then run it. It should generate a wrapper file in the gen_py folder C:\Python27\Lib\site-packages\win32com\gen_py with the CLSID in it. You can then access it like the excel example and the same way you would in other languages.

I had the same issue. I didn't know where to get that, which from what I can gather is the name of the CoClass (which I believe is the wrapper for the dll) created by the makepy program (maybe I'm wrong). After I read the answer by #NotAUser I found it, but when I inspected the file created by the makepy program, and right before the last class, (if more than one are created) I found the following:
from win32com.client import CoClassBaseClass
# This CoClass is known by the name 'ZKFPEngXControl.ZKFPEngX'
class ZKFPEngX(CoClassBaseClass): # A CoClass
So, there, in the same file there was the name I needed to use.

How to port a Python application to Linux that works fine in Windows

I am having trouble porting a working, Windows Python application to Linux. I am having some problems, because I did not write the code and am just learning Python. I am having trouble fixing the issues that it keeps throwing up. So here is a kind of error that right now I am stuck with
Traceback (most recent call last):
File "alpha_beta", line 237, in <module>
main()
File "alpha_beta", line 185, in main
ABCCmd()
File "alpha_beta.py", line 74, in ABCCmd
File "C:\softs\Python\Lib\shutil.py", line 80, in copy
File "C:\softs\Python\Lib\shutil.py", line 47, in copyfile
IOError: [Errno 13] Permission denied: '/myPath/XFiles.bin.addr_patched
Any pointers on how to fix it will be much appreciated
Edit:
1) What I mean by I am stuck is, the traceback of the error goes to C:\softs\Python\Lib but actually I am currently executing this code in Ubuntu. Why would the traceback reference to windows library
2) Another thing that bothers me is it says there is an IOError.But when I try to add permission for the denied one it gives me a chmod: changing permissions of /myPath/xFiles.bin.addr_patched': Operation not permitted
Edit 2:
I had commented out a module because I thought it wasn't very useful. Since Now I am anyway discussing the porting issues, I thought I can bring up this additional problem as well since I think the issue is the same and the fix should be similar. On including #pdb module in the python code, I get the following error
traceback (most recent call last):
File "alpha_beta", line 6, in <module>
import pdb
File "C:\softs\Python\Lib\pdb.py", line 14, in <module>
File "C:\softs\Python\Lib\pprint.py", line 39, in <module>
ImportError: No module named cStringIO
I looked at the importer_Cache and looks like this
'': None, '/usr/lib/python2.6/plat-linux2': None, '/usr/lib/python2.6/': None
'/usr/lib/pymodules/python2.6/gtk-2.0': None, '/usr/lib/python2.6/lib-tk': None,
'/usr/lib/python2.6/lib-old': <imp.NullImporter object at 0x7f1269048070>, '/usr/
/python2.6/dist-packages/gtk-2.0': None, '/usr/lib/python2.6/dist-packages/PIL': None,
'/usr/local/lib/python2.6/dist-packages': None, '/usr/lib/python2.6/dist-packages':
None
*SOLVED : There was a weird wrapper in the script which was causing the python script to execute from a different location. Still don't understand how though. After modifying my script to just directly execute as python myScript.py it runs fine. Thank you all anyway for the help

The mixture of Windows and Unix style paths in your error messages makes me think that you may have some filenames that are hard coded or manually created using os specific path separators ('\' or '/'). If you can figure out where the gOptions.inputTf and gWorkingTfFile values are assigned you should look into using os.path.join to help you use the correct paths.
from os.path import join
file = join('A', 'B')
# 'A/B' on unix systems
# 'A\B' on windows systems