We Keep Coding

SSL Error in thread with simple post request in python

I'm trying to do a simple post request, I'm using a list because I want to send all my post request at the same time using thread. Here is an example of an url :
s = "https://emoncms.org/input/post.json?node="+str(test)+"&json={test_stack_overflow:0}&apikey="+str(apikey)
list.append(threading.Thread(target=requests.post, args=([s, ])))
I was using this code maybe 3 months ago and it worked perfectly.
I wanted to get back on this project this week and I realized that I got some errors, this one particularly :
Exception in thread Thread-14:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 763, in run
self.__target(*self.__args, **self.__kwargs)
File "/usr/lib/python2.7/dist-packages/requests/api.py", line 94, in post
return request('post', url, data=data, json=json, **kwargs)
File "/usr/lib/python2.7/dist-packages/requests/api.py", line 49, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 457, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 569, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 420, in send
raise SSLError(e, request=request)
SSLError: <unprintable SSLError object>
I got an other error, ConnectionError but I think it's due to the network or because the website can't follow it's activity or is down. I leave you the traceback if you want :
ConnectionError: ('Connection aborted.', error(101, 'Network is unreachable'))
This code is only a part of my project, the code is running every minutes and I don't know why but this issue (SSLError) comes only maybe 10 times a day. I got this script running on different Raspberry Pi and some have the same problem but not the same frequency, others don't have it at all.
Any ideas on what is going ?
Thanks in advance !

Use verify=False in the requests method like this
import requests
url="https://emoncms.org/input/post.json?node="+str(test)+"&json={test_stack_overflow:0}&apikey="+str(apikey)
requests.post(url,verify=False)
If you are using with threads then it will be like
list.append(threading.Thread(target=requests.post, args=(url,),kwargs={"verify":False})) #**kwargs should be passed seperately.
You are getting this error because python requests tries to verify certificate for https connections so you have to override it by passing verify=False or you can also provide certificate in verify like this requests.get(url,verify="/path/to/certificate.ext")
Also I doubt that this should be a get request because query parameters won't come in post request as of my knowledge. So if you use GET method same verify applies there too.

Python requests gives SSL unknown protocol

I am trying to send a request to an API I have set up on an AWS machine.
The code I use is as follows:
import requests
import json
report_dict = {
"client_name": "Wayne Enterprises",
"client_id": 123,
"report_type": "api_testing",
"timestamp_generated": "2015-07-29T11:00:00Z",
"report_data": {"revenue": 9000.00}
}
report_json = json.dumps(report_dict)
resp = requests.post("https://my-url.com:8080/my-api/reports", data=report_json,verify=False)
Doing this, I get:
Traceback (most recent call last):
File "art2_java_test.py", line 124, in <module>
main()
File "art2_java_test.py", line 9, in main
test_post_good_data()
File "art2_java_test.py", line 29, in test_post_good_data
resp = requests.post("https://my-url.com:8080/my-api/reports", data=report_json,verify=Fal
se)
File "C:\Python27\lib\site-packages\requests-2.7.0-py2.7.egg\requests\api.py",
line 109, in post
return request('post', url, data=data, json=json, **kwargs)
File "C:\Python27\lib\site-packages\requests-2.7.0-py2.7.egg\requests\api.py",
line 50, in request
response = session.request(method=method, url=url, **kwargs)
File "C:\Python27\lib\site-packages\requests-2.7.0-py2.7.egg\requests\sessions
.py", line 465, in request
resp = self.send(prep, **send_kwargs)
File "C:\Python27\lib\site-packages\requests-2.7.0-py2.7.egg\requests\sessions
.py", line 573, in send
r = adapter.send(request, **kwargs)
File "C:\Python27\lib\site-packages\requests-2.7.0-py2.7.egg\requests\adapters
.py", line 428, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: [SSL: UNKNOWN_PROTOCOL] unknown protocol (_ssl.c:5
90)
But when I send the request as http instead of https, it (usually) works fine. I've found some evidence that this can have to do with proxy servers, but I am not using one. Are there any other potential reasons for this error? This is a website only available on my company's local network, if that's relevant.

.... https://my-url.com:8080/my-api/reports
...But when I send the request as http instead of https, it (usually) works fine.
My guess is that you are trying the same port 8080 for http and https. But, servers usually listen on a single port either for http or https and not both. This means that if your client is trying to start the TLS handshake needed for https against this port it will get a plain error message back. The client then tries to interpret this error message as TLS and returns some weird error messages, because the response is not TLS at all.

Http Request through proxy in python having # in password

How to escape # character in the password of proxy. So that python can create the request correctly. I have tried \\ but still not able to hit the url correctly.
proxy = {
"http": "http://UserName:PassWord#X.X.X.X:Port_No"
}
Update question:
I am using python requests module for the http request. It split the string (to get host) from first occurrence of # where as it was suppose to split from second #.
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 55, in get
return request('get', url, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 44, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 335, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 438, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 327, in send
raise ConnectionError(e)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='XXXXXXXX#X.X.X.X', port=XXXXX): Max retries exceeded with url: http:/URL (Caused by <class 'socket.gaierror'>: [Errno -2] Name or service not known)

You have to do urlencoding like in this post:
Escaping username characters in basic auth URLs
This way the # in the PW becomes %40

You don't mention which library you are using to perform your HTTP requests, so you should consider using requests, not only to solve this problem, but because it is a great library.
Here is how to use a proxy with basic authentication:
import requests
proxy = {'http': 'http://UserName:PassWord#X.X.X.X:Port_No'}
r = requests.get("http://whereever.com", proxies=proxy)
Update
Successfully tested with requests and proxy URLs:
http://UserName:PassWord#127.0.0.1:1234
http://UserName:PassWord##127.0.0.1:1234
http://User#Name:PassWord#1234#127.0.0.1:1234
If, instead, you need to use Python's urllib2 library, you can do this:
import urllib2
handler = urllib2.ProxyHandler({'http': 'http://UserName:PassWord#X.X.X.X:Port_No'})
opener = urllib2.build_opener(handler)
r = opener.open('http://whereever.com')
Note that in neither case is it necessary to escape the #.
A third option is to set environment variables HTTP_PROXY and/or HTTPS_PROXY (in *nix).

Name or service not known error in EasyWebDav for Python

I'm trying to setup a WebDAV connection using easywebdav in Python. (Using 2.7.8 for now)
import csv, easywebdav
webdav=easywebdav.connect('https://sakai.rutgers.edu/dav/restoftheurl,username="",password="")
print webdav.ls()
Though when I run this I get the following error message. My guess is that it possibly has something to do with the URL using HTTPS?
Traceback (most recent call last):
File "/home/willkara/Development/SakaiStuff/WorkProjects/sakai-manager/file.py", line 4, in <module>
print webdav.ls()
File "build/bdist.linux-x86_64/egg/easywebdav/client.py", line 176, in ls
File "build/bdist.linux-x86_64/egg/easywebdav/client.py", line 97, in _send
File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 456, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 559, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 375, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='https', port=80): Max retries exceeded with url: //sakai.rutgers.edu/dav/url:80/. (Caused by <class 'socket.gaierror'>: [Errno -2] Name or service not known)
[Finished in 0.1s with exit code 1]

I find it strange that you combine HTTPS protocol and port 80. HTTPS uses port 443.
Though the error message "Name or service not known" would rather indicate that the hostname sakai.rutgers.edu is not recognized on your system. Try to ping the host.

I noticed that you shouldn't have http:// or https:// in the beginning of your adress, only the host name. You select protocol with protocol='https'. Also, I couln't get it to work if I added the path the url, I had to use it as argument to the operations like easywebdav.ls('/dav/restoftheurl') or easywebdav.cd('/dav/restoftheurl').

python-requests authenticated proxy httplib.BadStatusLine

Authenticated proxy through python-requests returns the following error:
>>> import requests
>>> proxies = {'https': 'http://username:password#proxy.company.com:8080',}
>>> requests.get('https://api.github.com/',proxies=proxies,verify=False)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.6/site-packages/requests/api.py", line 55, in get
return request('get', url, **kwargs)
File "/usr/lib/python2.6/site-packages/requests/api.py", line 44, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/lib/python2.6/site-packages/requests/sessions.py", line 335, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python2.6/site-packages/requests/sessions.py", line 438, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python2.6/site-packages/requests/adapters.py", line 327, in send
raise ConnectionError(e)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='proxy.company.com', port=8080): Max retries exceeded with url: https://api.github.com/ (Caused by <class 'httplib.BadStatusLine'>: )
Authenticated proxy works fine in curl:
$ curl --proxy-user username:password --proxy http://proxy.company.com:8080 -k https://api.github.com/
{
"current_user_url": "https://api.github.com/user",
"authorizations_url": "https://api.github.com/authorizations",
"emails_url": "https://api.github.com/user/emails",
"emojis_url": "https://api.github.com/emojis",
"events_url": "https://api.github.com/events",
"feeds_url": "https://api.github.com/feeds",
"following_url": "https://api.github.com/user/following{/target}",
"gists_url": "https://api.github.com/gists{/gist_id}",
"hub_url": "https://api.github.com/hub",
"issue_search_url": "https://api.github.com/legacy/issues/search/{owner}/{repo}/{state}/{keyword}",
"issues_url": "https://api.github.com/issues",
"keys_url": "https://api.github.com/user/keys",
"notifications_url": "https://api.github.com/notifications",
"organization_repositories_url": "https://api.github.com/orgs/{org}/repos/{?type,page,per_page,sort}",
"organization_url": "https://api.github.com/orgs/{org}",
"public_gists_url": "https://api.github.com/gists/public",
"rate_limit_url": "https://api.github.com/rate_limit",
"repository_url": "https://api.github.com/repos/{owner}/{repo}",
"repository_search_url": "https://api.github.com/legacy/repos/search/{keyword}{?language,start_page}",
"current_user_repositories_url": "https://api.github.com/user/repos{?type,page,per_page,sort}",
"starred_url": "https://api.github.com/user/starred{/owner}{/repo}",
"starred_gists_url": "https://api.github.com/gists/starred",
"team_url": "https://api.github.com/teams",
"user_url": "https://api.github.com/users/{user}",
"user_organizations_url": "https://api.github.com/user/orgs",
"user_repositories_url": "https://api.github.com/users/{user}/repos{?type,page,per_page,sort}",
"user_search_url": "https://api.github.com/legacy/user/search/{keyword}"
}
How do I troubleshoot?

Requests has support for HTTP proxies just since 2.0: https://github.com/kennethreitz/requests/pull/1515
BTW, it does respect proxy environment variables, so you can just set:
export http_proxy="http://username:password#proxy.company.com:8080"
export https_proxy=$http_proxy
More details about the state of proxy support in requests 1.x can be found in this blog post: https://lukasa.co.uk/2013/07/Python_Requests_And_Proxies/

Your proxy is indeed HTTP not HTTPS. To use proxy with HTTPS it needs to use connect protocol, not HTTP, with presumably CONNECT basic auth. Now I do not have a HTTPS proxy with auth available, and do not know how to install such system, but what happens if you have just "https://username:password#proxy.company.com:8080" there instead?
On the other hand, you might want to make sure that requests/urllib3 are up to date. See this bug report.