Is there no way to connect to a URL from azure ml and get it's content
my code:
import requests
def azureml_main(dataframe1 = None, dataframe2 = None):
b= requests.get("http://www.google.com",timeout=30)
dataframe1 = b.content
return dataframe1
Is there any change need to be made to connect to URL
ERROR:
Error 0085: The following error occurred during script evaluation, please view the output log for more information:
---------- Start of error message from Python interpreter ----------
data:text/plain,Caught exception while executing function: Traceback (most recent call last):
File "C:\server\invokepy.py", line 167, in batch
odfs = mod.azureml_main(*idfs)
File "C:\temp\azuremod.py", line 24, in azureml_main
b= requests.get("http://www.google.com",timeout=30)
File "C:\pyhome\lib\site-packages\requests\api.py", line 55, in get
return request('get', url, **kwargs)
File "C:\pyhome\lib\site-packages\requests\api.py", line 44, in request
return session.request(method=method, url=url, **kwargs)
File "C:\pyhome\lib\site-packages\requests\sessions.py", line 456, in request
resp = self.send(prep, **send_kwargs)
File "C:\pyhome\lib\site-packages\requests\sessions.py", line 559, in send
r = adapter.send(request, **kwargs)
File "C:\pyhome\lib\site-packages\requests\adapters.py", line 375, in send
raise ConnectionError(e, request=request)
ConnectionError: HTTPConnectionPool(host='www.google.com', port=80): Max retries exceeded with url: / (Caused by <class 'socket.gaierror'>: [Errno 11001] getaddrinfo failed)
---------- End of error message from Python interpreter ---------
Or is there any change needed to be made on the azure ml settings
UPDATE 1/28/2016
Network I/O for Execute Python Script is now supported.
Out of date
Network I/O is not support from Execute Python Modules. In order to execute such a program, you should instead launch a virtual machine(Windows or Linux your choice).
Windows:
RDP into Virtual Machine
Install your choice of Python
You can drag and drop your Python program from your Local Windows machine onto your RDP screen to transfer your code
Then run your program
Ubuntu:
SSH into your virtual machine using Cygwin or Putty(Windows) or Terminal SSH (mac) ssh yourUserName#yourAzureVM.cloudapps.net
install Python sudo apt-get install python
open your preferred Linux text editor vi myProgram.py
Copy and Paste your code into the editor (leave vi with esc :wq )
Run your code python myProgram.py
To move data from your VM to AzureML please check out the Azure-MachineLearning-ClientLibrary-Python on Github
Related
I have 2 servers. I am attempting to setup one for continuous integration for my main website server
Web server 1(cloud-hosting):
Python3.6
Django3.1
Ubuntu16.04
Webserver 2(VPS):
Python3.7
Django3.1
Ubuntu16.04
Jenkins
--ShiningPanda(plugin)
Im new to web development, so if it seems odd as far as my web server types, that is why. I have been following along in the book Test Driven Development with Python. My issue is that when running python manage.py test [app] My [app] inherits from the class StaticLiveSever to generate a testing environment. On webserver 1, this works fine. On webserver 2, i get an error that the request address cannot be assigned. I use jenkins to build the environment, but the error i get is OSerror[99]:cannot assign requested address. I dont understand why this is happening when i run the same commands in Web Sever 1. It runs fine. Although again, the commands are run by jenkins and jenkins is configured to run python3.7
Full Traceback(Main Issue)
Traceback (most recent call last):
File "/var/lib/jenkins/shiningpanda/jobs/ddc1aed1/virtualenvs/d41d8cd9/lib/python3.7/site-packages/django/test/testcases.py", line 1449, in setUpClass
raise cls.server_thread.error
File "/var/lib/jenkins/shiningpanda/jobs/ddc1aed1/virtualenvs/d41d8cd9/lib/python3.7/site-packages/django/test/testcases.py", line 1374, in run
self.httpd = self._create_server()
File "/var/lib/jenkins/shiningpanda/jobs/ddc1aed1/virtualenvs/d41d8cd9/lib/python3.7/site-packages/django/test/testcases.py", line 1389, in _create_server
return ThreadedWSGIServer((self.host, self.port), QuietWSGIRequestHandler, allow_reuse_address=False)
File "/var/lib/jenkins/shiningpanda/jobs/ddc1aed1/virtualenvs/d41d8cd9/lib/python3.7/site-packages/django/core/servers/basehttp.py", line 67, in __init__
super().__init__(*args, **kwargs)
File "/usr/lib/python3.7/socketserver.py", line 452, in __init__
self.server_bind()
File "/usr/lib/python3.7/wsgiref/simple_server.py", line 50, in server_bind
HTTPServer.server_bind(self)
File "/usr/lib/python3.7/http/server.py", line 137, in server_bind
socketserver.TCPServer.server_bind(self)
File "/usr/lib/python3.7/socketserver.py", line 466, in server_bind
self.socket.bind(self.server_address)
OSError: [Errno 99] Cannot assign requested address
After hardcoded host
Traceback (most recent call last):
File "/var/lib/jenkins/workspace/Superlists/functional_tests/base.py", line 47, in setUp
self.browser = webdriver.Firefox()
File "/var/lib/jenkins/shiningpanda/jobs/ddc1aed1/virtualenvs/d41d8cd9/lib/python3.7/site-packages/selenium/webdriver/firefox/webdriver.py", line 164, in __init__
self.service.start()
File "/var/lib/jenkins/shiningpanda/jobs/ddc1aed1/virtualenvs/d41d8cd9/lib/python3.7/site-packages/selenium/webdriver/common/service.py", line 100, in start
self.assert_process_still_running()
File "/var/lib/jenkins/shiningpanda/jobs/ddc1aed1/virtualenvs/d41d8cd9/lib/python3.7/site-packages/selenium/webdriver/common/service.py", line 113, in assert_process_still_running
% (self.path, return_code)
selenium.common.exceptions.WebDriverException: Message: Service geckodriver unexpectedly exited. Status code was: 69
geckodriver.log w/ hardcoded host ip in LiveTestServer
eckodriver: error: Address not available (os error 99)
geckodriver 0.27.0 (7b8c4f32cdde 2020-07-28 18:16 +0000)
WebDriver implementation for Firefox
USAGE:
geckodriver [FLAGS] [OPTIONS]
[...]
Hopefully the tracebacks above are not too confusing. Ultmiately what i did notice was the when im in webserver 2, i access the Django testcases.py module that has LiverServerThread and hardcode host=0.0.0.0 instead of host=localhost (1st traceback). The connection is then made although then the problem lies with geckodriver and the same thing (2nd traceback). I need to hard code ip 0.0.0.0 to be able to establish a connection, but then geckodriver is just listening, which i am assuming, at a completely different location (no error.log shown here).
So 1st id like to at least be able to make a connection to run the LiveServerThread class properly. Then try and resolve the issue with geckodriver. I also was not sure if the type of servers im running on poses as the problem.
I have a simple code running good using http (not secure) but then I update it to https a get an error about ssl.SSLCertVerificationError
I'm using a heroku server with a valid ssl of course, python 3.7.3 on a mac, ssl.OPENSSL_VERSION= 1.1.0, certifi=2019.3.9, Flask=1.0.2, Flask-SocketIO=3.3.2
Here is the code:
import socketio
sio = socketio.Client()
#sio.on('connect')
def on_connect():
print('Connected ...')
#sio.on('message')
def on_message(data):
print('I received a message!')
print(data)
if __name__ == '__main__':
sio.connect('https://heroku.server.url')
And this is what I get using https:
Traceback (most recent call last):
File "/Users/calavraian/Devel/Projects/FlaskTesting/Client.py", line 21, in <module>
sio.connect('https://heroku.server.url')
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/socketio/client.py", line 208, in connect
engineio_path=socketio_path)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/engineio/client.py", line 166, in connect
url, headers, engineio_path)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/engineio/client.py", line 303, in _connect_polling
if self._connect_websocket(url, headers, engineio_path):
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/engineio/client.py", line 341, in _connect_websocket
cookie=cookies)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/websocket/_core.py", line 514, in create_connection
websock.connect(url, **options)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/websocket/_core.py", line 223, in connect
options.pop('socket', None))
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/websocket/_http.py", line 126, in connect
sock = _ssl_socket(sock, options.sslopt, hostname)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/websocket/_http.py", line 260, in _ssl_socket
sock = _wrap_sni_socket(sock, sslopt, hostname, check_hostname)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/websocket/_http.py", line 239, in _wrap_sni_socket
server_hostname=hostname,
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ssl.py", line 412, in wrap_socket
session=session
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ssl.py", line 853, in _create
self.do_handshake()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ssl.py", line 1117, in do_handshake
self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1056)
Any ideas ?
Finally I made it work, I tried the same code on another Mac computer with the same software condition and it works fine, also on a PC with Ubuntu and a Raspberry Pi with Raspbian, everything works fine, so back again on the first Mac with the problem, I started to debug again, reinstalling and updating packages with pip and nothing happened.
Other solution that I have tried was to reinstall the certifi package with pip but nothing was fixed.
At the end, what solved the problem was running the script that comes with Python and is located in /Applications/Python 3.X/Install Certificates.command, you can run it directly from Finder with double clic on it or on a terminal go to the folder and execute it.
After doing this I tested python-socketio 3.1.2 and python-socketio 4.0.1 and everything works fine.
A final recommendation, in case that you are still getting the same error after try running the script, you can try an alternative to python-socketio, in the process to figure it out I have tried socketIO-client-2 when I was getting the error and this library worked good on the initial conditions (before running the script). After running the script above both are running perfect now.
I'm trying to create a data science environment in Google Cloud Platform, with Python, Miniconda and Jupyter. But I'm encountering an error when I try to access jupyter with an external IP
I followed these steps:
1 - I've accessed the Resource Management page (https://console.cloud.google.com/cloud-resource-manager). Create a new project (test_01)
2 - I opened the Dashboard (https://console.cloud.google.com/home/dashboard). I selected the project I just created (test_01)
3 - In the top left menu - Select "Compute Engine" and click on "VM instances. Click on the "Create" button
4 - I filled in: Name the instance (the-who), Select the region (in my case - northamerica-northeast1-a), Select the memory, storage and CPU, Select the boot disk, Remote access SSH, Make sure you can access the VM from the internet by allowing Http and Https traffic and Click on "Create"
5 - I opened Google's Cloud Shell. I pressed the button >_
6 - With the Ubuntu terminal open, I typed in to create a password for root: sudo passwd
7 - Enter the command to enter SSH: gcloud compute ssh the-who
8 - There is always a question about the zone: Did you mean zone [us-east1-c] for instance: [the-who] (Y/n)? - Since mine zone is different (northamerica-northeast1-a) I press n
9 - I begin to install packages (bzip2 git libxml2-dev): "sudo apt-get update" and "sudo apt-get install bzip2 git libxml2-dev"
10 - Now install the miniconda, scikit-learn, pandas, jupyter and ipython: "wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh", "bash Miniconda3-latest-Linux-x86_64.sh", "rm Miniconda3-latest-Linux-x86_64.sh", "source .bashrc" and "conda install scikit-learn pandas jupyter ipython"
11 - I check installation: "python --version" and "which python"
12 - Now I want to set up my VM to allow Web access to my Jupyter notebooks. I returned to the instance panel and in the top left menu I selected "VPC Network > Firewall rules". I clicked on the "CREATE FIREWALL RULE" link and filled in with the following values
13 - Name: jupyter-rule * Source IP ranges: 0.0.0.0/0 * Specified protocols and ports: tcp:8888 * Target tags: the-who * and leave all the other variables to their default values. As a result, this firewall rule allows all incoming traffic (of all IPs) to reach port 8888.
14 - I returned to the VM page (upper left menu> Compute Engine> VM instances) and clicked on the VM name. I wrote down the VM's external IP address
15 - Back to the terminal I configure the jupyter. I type "jupyter notebook --generate-config" to generate the configuration file and to generate a password "jupyter notebook password"
16 - Now edit the jupyter configuration file: vim .jupyter/jupyter_notebook_config.py
17 - In the first line write and then save and close the file: c.NotebookApp.ip = '*'
18 - The above command will allow the notebook to be available for all IP addresses on your VM. And then I started the Jupyter Notebook with the command line: jupyter-notebook --no-browser --port=8888
But I found this error message:
(base) reichaves#the-who:~$ jupyter-notebook --no-browser --port=8888
[I 11:51:06.234 NotebookApp] Writing notebook server cookie secret to /run/user/1001/jupyter/notebook_cookie_secret
Traceback (most recent call last):
File "/home/reichaves/miniconda3/lib/python3.7/site-packages/traitlets/traitlets.py", line 528, in get
value = obj._trait_values[self.name]
KeyError: 'allow_remote_access'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/reichaves/miniconda3/lib/python3.7/site-packages/notebook/notebookapp.py", line 864, in _default_allow_remote
addr = ipaddress.ip_address(self.ip)
File "/home/reichaves/miniconda3/lib/python3.7/ipaddress.py", line 54, in ip_address
address)
ValueError: '' does not appear to be an IPv4 or IPv6 address
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/reichaves/miniconda3/bin/jupyter-notebook", line 11, in <module>
sys.exit(main())
File "/home/reichaves/miniconda3/lib/python3.7/site-packages/jupyter_core/application.py", line 266, in launch_instance
return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
File "/home/reichaves/miniconda3/lib/python3.7/site-packages/traitlets/config/application.py", line 657, in launch_instance
app.initialize(argv)
File "</home/reichaves/miniconda3/lib/python3.7/site-packages/decorator.py:decorator-gen-7>", line 2, in initialize
File "/home/reichaves/miniconda3/lib/python3.7/site-packages/traitlets/config/application.py", line 87, in catch_config_error
return method(app, *args, **kwargs)
File "/home/reichaves/miniconda3/lib/python3.7/site-packages/notebook/notebookapp.py", line 1628, in initialize
self.init_webapp()
File "/home/reichaves/miniconda3/lib/python3.7/site-packages/notebook/notebookapp.py", line 1378, in init_webapp
self.jinja_environment_options,
File "/home/reichaves/miniconda3/lib/python3.7/site-packages/notebook/notebookapp.py", line 159, in __init__
default_url, settings_overrides, jinja_env_options)
File "/home/reichaves/miniconda3/lib/python3.7/site-packages/notebook/notebookapp.py", line 252, in init_settings
allow_remote_access=jupyter_app.allow_remote_access,
File "/home/reichaves/miniconda3/lib/python3.7/site-packages/traitlets/traitlets.py", line 556, in __get__
return self.get(obj, cls)
File "/home/reichaves/miniconda3/lib/python3.7/site-packages/traitlets/traitlets.py", line 535, in get
value = self._validate(obj, dynamic_default())
File "/home/reichaves/miniconda3/lib/python3.7/site-packages/notebook/notebookapp.py", line 867, in _default_allow_remote
for info in socket.getaddrinfo(self.ip, self.port, 0, socket.SOCK_STREAM):
File "/home/reichaves/miniconda3/lib/python3.7/socket.py", line 748, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known
Please, does anyone know if any setup details were missing?
Recent versions of Jupyter need to be told which IP addresses to listen to. You need in the config file:
c.NotebookApp.ip = '0.0.0.0'
Even with this change, and a password, I prefer to put Jupyter in a Singularity container, since people with web access to it can run any python code on the VM. For a recipe to do this, see:
https://bitbucket.org/account/user/stfcsciml/projects/JSV
I am using SSL Connection from Web Client (firefox) to connect to the local Server created via the code available in this link MDM server code I have all the required signed Certificates issued by Apple available for running the server, as well as all the depended python modules installed.
I ran my code using the following command on my Windows Machine.
python server.py .
This successfully start running the server at the port 8080.
C:\Users\Administrator\Desktop\MDMServer>python server.py <ip address>
Can't find MyApp.mobileprovision in current directory.
Need both MyApp.ipa and Manifest.plist to enable InstallCustomApp.
Starting Server
https://<ipaddress>:8080/
But whenever a Web Client tries to connect to this Server.
Upon launching the link https://<ip address>:8080 in the firefox. The Server reports the following error.
Traceback (most recent call last):
File "server.py", line 498, in <module>
app.run()
File "C:\Python27\lib\site-packages\web\application.py", line 313, in run
return wsgi.runwsgi(self.wsgifunc(*middleware))
File "C:\Python27\lib\site-packages\web\wsgi.py", line 54, in runwsgi
return httpserver.runsimple(func, validip(listget(sys.argv, 1, '')))
File "C:\Python27\lib\site-packages\web\httpserver.py", line 157, in runsimple
server.start()
File "C:\Python27\lib\site-packages\web\wsgiserver\__init__.py", line 1765, in
start
self.tick()
File "C:\Python27\lib\site-packages\web\wsgiserver\__init__.py", line 1815, in
tick
s, ssl_env = self.ssl_adapter.wrap(s)
File "C:\Python27\lib\site-packages\web\wsgiserver\ssl_builtin.py", line 42, i
n wrap
keyfile=self.private_key, ssl_version=ssl.PROTOCOL_SSLv23)
File "C:\Python27\lib\ssl.py", line 372, in wrap_socket
ciphers=ciphers)
File "C:\Python27\lib\ssl.py", line 134, in __init__
self.do_handshake()
File "C:\Python27\lib\ssl.py", line 296, in do_handshake
self._sslobj.do_handshake()
ssl.SSLError: [Errno 1] _ssl.c:503: error:1408A0C1:SSL routines:SSL3_GET_CLIENT_
HELLO:no shared cipher
Can anyone help me understand:
1) What these errors mean
2) Why I'm getting them, It was not coming with the older version of server.py code applicable to iphone 4s?
3) How I can go about fixing these
Thanks,
Vivek
This is because APNSWrapper hasn't been updated to TLSv1 yet.
Apple no longer supports sslv3 for MDM due to the POODLE vulnerability.
You can update APNSWrapper source yourself though pretty easily.
download the source and update connection.py
ssl_version = self.ssl_module.PROTOCOL_TLSv1,
the run
python setup.py install
I'm maintaining a Python mini-app that uses requests + HTTPS.
The app worked until the IP address of the hostname in the HTTPS URL changed (legitimately). If I point my browser to the URL I can retrieve it fine.
Where does Python/requests keep the analog of ssh's known_hosts and how do I clear it for this host?
$ python --version
Python 2.7.3
$ cat foo.py
import requests
url = "https://somehost/resource.json"
requests.get(url, timeout=5, config={'danger_mode': True})
$ source venv/bin/activate
$ python foo.py
Traceback (most recent call last):
File "foo.py", line 3, in <module>
requests.get(url, timeout=5, config={'danger_mode': True})
File "/home/dfukdev/corsair-scripts/alfred/venv/local/lib/python2.7/site-packages/requests/api.py", line 65, in get
return request('get', url, **kwargs)
File "/home/dfukdev/corsair-scripts/alfred/venv/local/lib/python2.7/site-packages/requests/safe_mode.py", line 39, in wrapped
return function(method, url, **kwargs)
File "/home/dfukdev/corsair-scripts/alfred/venv/local/lib/python2.7/site-packages/requests/api.py", line 51, in request
return session.request(method=method, url=url, **kwargs)
File "/home/dfukdev/corsair-scripts/alfred/venv/local/lib/python2.7/site-packages/requests/sessions.py", line 241, in request
r.send(prefetch=prefetch)
File "/home/dfukdev/corsair-scripts/alfred/venv/local/lib/python2.7/site-packages/requests/models.py", line 641, in send
raise SSLError(e)
requests.exceptions.SSLError: [Errno 1] _ssl.c:504: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
You're using an ancient version of requests. You'll get a more helpful message if you upgrade to 2.0 and if your site has a certificate mismatch you may be able to fix it by specifying the system certificates which will be able to verify an intermediate certificate. You can also just have requests not verify your certificate as Andre suggested.
It turned out that during the server upgrade mentioned in the question an incorrectly-signed certificate was installed. HTTPS in the browser worked because of the root certificate differences between the Windows browser machine and Ubuntu Python client. HTTPS via a browser from the same Ubuntu machine on which Python was run revealed the certificate problem details.
The IP change had little to do with the problem except to confuse things.
Promoting my comment to an answer as:
this answered my question
this question is getting enough traffic I'd like to share the knowledge.
If the above doesn't work, and you find out it is local, then this solution worked for me...
Essentially run the Install Certificates.command file within Python folder.
https://www.dev2qa.com/how-to-fix-python-error-certificate-verify-failed-unable-to-get-local-issuer-certificate-in-mac-os/