We Keep Coding

raspberry pi can't make any url requests, connection timed out

I need to update my sensor data to thingspeak , so I was doing it with the python thingspeak library. Two days ago it was working fine but now it doesn't work, the connection times out , I also tried to update it with urllib2 ,that too doesnt' work.
My net connection is fine, I am able to open webpages on Pi and I can update the channel from my laptop using thingspeak library as well as urllib2.
could someone kindly help me.
my code with thingspeak library:
node = False
channel_id = ""
write = ""
if data['MAC'] in ':ab:35':
channel_id = "XXXXX"
write = "XXXXXXXXXXX"
node = True
if node:
channel = thingspeak.Channel(id=channel_id,write_key=write)
try :
response=channel.update({1:data['TEMP'],2:data['VOLT'],3:data['PRES'],4:data['HUM']})
print response
print 'Thingspeak updated!!!'
except :
print "connection failed"
when i try urllib2 :
f = urllib2.urlopen(url)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.7/urllib2.py", line 154, in urlopen
return opener.open(url, data, timeout)
File "/usr/lib/python2.7/urllib2.py", line 431, in open
response = self._open(req, data)
File "/usr/lib/python2.7/urllib2.py", line 449, in _open
'_open', req)
File "/usr/lib/python2.7/urllib2.py", line 409, in _call_chain
result = func(*args)
File "/usr/lib/python2.7/urllib2.py", line 1240, in https_open
context=self._context)
File "/usr/lib/python2.7/urllib2.py", line 1197, in do_open
raise URLError(err)
urllib2.URLError: <urlopen error [Errno 110] Connection timed out>
when I try thingspeak library :
>>> channel = thingspeak.Channel(id=c,write_key=w)
>>> res = channel.update({1:50,2:30,3:70,4:20})
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "build/bdist.linux-armv7l/egg/thingspeak/thingspeak.py", line 116, in update
File "/usr/lib/python2.7/dist-packages/requests/api.py", line 94, in post
return request('post', url, data=data, json=json, **kwargs)
File "/usr/lib/python2.7/dist-packages/requests/api.py", line 49, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 457, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 569, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 407, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', error(110, 'Connection timed out'))

I am using the raspberry-pi to send data to thingspeak.
What I do is something like this:
import httplib
import urllib
#Preparing to send the information to the cloud
params=urllib.urlencode({'field1':liters,'key':key})
headers={"Content-typZZe": "application/x-www-form-urlencoded","Accept":"text/plain"}
conn=httplib.HTTPConnection("api.thingspeak.com:80")
#Sending the data to the thingspeak platform
try:
conn.request("POST", "/update", params, headers)
response=conn.getresponse()
print response.status, response.reason
data=response.read()
conn.close()
except:
print "connection failed"
And it works just fine.
The key is the string of your channel key.
Hope it helps.

If I use SSH and try to run the script, it works .

Add a new User-Agent, by creating Request objects & using them as arguments for urlopen:
import urllib2
request = urllib2.Request('http://www.example.com/')
request.add_header('User-agent', 'Mozilla/5.0 (Linux i686)')
response = urllib2.urlopen(request)

connecting to secure website using CmisClient library

In my project we are trying to use CMIS to get the folder repository and I use python script to test it; below is the piece of code I used
from cmislib.model import CmisClient
client = CmisClient('http://localhost/CMIS/Service/servicedoc', 's', 's')
repo = client.defaultRepository
info = repo.info
for k,v in info.items():
print "%s:%s" % (k,v)
somefld = repo.getObject('idf_96_Z2CMIS')
props = somefld.properties
for k,v in props.items():
print "%s:%s" % (k,v)
This code works perfectly fine. However now the service is SSL enabled so (https//localhost/CMIS/Service/servicedoc) when I change the URL in CmisClient it is throwing the below error
c:\Python27>python.exe cmis.py
CMIS client connection to https://localhost/Cmis/Service/servicedoc
Traceback (most recent call last):
File "cmis.py", line 4, in <module>
repo = client.defaultRepository
File "c:\Python27\lib\site-packages\cmislib-0.5.1-py2.7.egg\cmislib\model.py",
line 179, in getDefaultRepository
File "c:\Python27\lib\site-packages\cmislib-0.5.1-py2.7.egg\cmislib\model.py",
line 206, in get
File "c:\Python27\lib\site-packages\cmislib-0.5.1-py2.7.egg\cmislib\net.py", l
ine 145, in get
File "c:\Python27\lib\urllib2.py", line 404, in open
response = self._open(req, data)
File "c:\Python27\lib\urllib2.py", line 422, in _open
'_open', req)
File "c:\Python27\lib\urllib2.py", line 382, in _call_chain
result = func(*args)
File "c:\Python27\lib\urllib2.py", line 1222, in https_open
return self.do_open(httplib.HTTPSConnection, req)
File "c:\Python27\lib\urllib2.py", line 1184, in do_open
raise URLError(err)
urllib2.URLError: <urlopen error [Errno 10054] An existing connection was forcibly closed by the remote host>
How do I use CMISClient library to connect to SSL enabled website. Thanks in advance.

I changed my URL to have the <> instead of localhost https://<>/Cmis/Service/servicedoc and it worked.

urllib2 proxy does not work with tor

I want to write some script with python which uses tor/proxy addresses to access web, for the test I have the following script:
import urllib2
from BeautifulSoup import BeautifulSoup
protocol = 'socks4'
ip = '127.0.0.1:9050'
proxy = urllib2.ProxyHandler({protocol:ip})
opener = urllib2.build_opener(proxy)
urllib2.install_opener(opener)
page = urllib2.urlopen("http://www.ifconfig.me/ip").read()
print(page)
The problem is it shows my own IP address, while when run directly from terminal:
proxychains curl ifconfig.me/ip
shows different IP, how can I fix it?
when http used instead of socks 4 it gives the following error:
Traceback (most recent call last):
File "proxy_test.py", line 11, in <module>
page = urllib2.urlopen("http://www.ifconfig.me/ip").read()
File "/usr/lib/python2.7/urllib2.py", line 126, in urlopen
return _opener.open(url, data, timeout)
File "/usr/lib/python2.7/urllib2.py", line 400, in open
response = meth(req, response)
File "/usr/lib/python2.7/urllib2.py", line 513, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib/python2.7/urllib2.py", line 438, in error
return self._call_chain(*args)
File "/usr/lib/python2.7/urllib2.py", line 372, in _call_chain
result = func(*args)
File "/usr/lib/python2.7/urllib2.py", line 521, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 501: Tor is not an HTTP Proxy

I use http (not sock) and it works
import urllib2
from BeautifulSoup import BeautifulSoup
protocol = 'http'
ip = '127.0.0.1:8118'
proxy = urllib2.ProxyHandler({protocol:ip})
opener = urllib2.build_opener(proxy)
urllib2.install_opener(opener)
page = urllib2.urlopen("http://www.ifconfig.me/ip").read()
print(page)

Program abruptly stops and throws URLERROR

from poster.encode import multipart_encode
from poster.streaminghttp import register_openers
def picscrazy(str,int):
register_openers()
datagen, headers = multipart_encode({"imagefile[]": open(str, "rb")})
request = urllib2.Request("http://www.picscrazy.com/process.php", datagen, headers)
Str is the filename and the int is just another flag.
The code is to upload a file to a image hosting website .I am using poster Poster for the post requests. The program stops after the request statement and gives an error .I cant understand the error whether its a problem in my network or in the program.
Below is the traceback of the error:
Traceback (most recent call last):
File "C:\Documents and Settings\Administrator\Desktop\for exbii\res.py", line 42, in <module>
picscrazy(fname,1)
File "C:\Documents and Settings\Administrator\Desktop\for exbii\res.py", line 14, in picscrazy
print(urllib2.urlopen(request).read())
File "C:\Python25\Lib\urllib2.py", line 121, in urlopen
return _opener.open(url, data)
File "C:\Python25\Lib\urllib2.py", line 374, in open
response = self._open(req, data)
File "C:\Python25\Lib\urllib2.py", line 392, in _open
'_open', req)
File "C:\Python25\Lib\urllib2.py", line 353, in _call_chain
result = func(*args)
File "C:\Python25\lib\poster\streaminghttp.py", line 142, in http_open
return self.do_open(StreamingHTTPConnection, req)
File "C:\Python25\Lib\urllib2.py", line 1076, in do_open
raise URLError(err)
URLError: <urlopen error (10054, 'Connection reset by peer')>

If you can't display the header coming back from the server, then, your server has simply cut you off.
It may be your request is bad -- but that's unlikely.
It may be that you've exceeded bandwidth restrictions.
It may be that your requests appear to be a DDOS attack because they're happening too frequently.

Browsing a NTLM protected website using python with python NTLM

I have been tasked with creating a script that logs on to a corporate portal goes to a particular page, downloads the page, compares it to an earlier version and then emails a certain person depending on changes that have been made. The last parts are easy enough but it has been the first step that is giving me the most trouble.
After unsuccessfully using urllib2(I am trying to do this in python) to connect and about 4 or 5 hours of googling I have determined that the reason I can't connect is due to NTLM authentication on the web page. I have tried a bunch of different processes for connecting found on this site and others to no avail. Based on the NTLM example I have done:
import urllib2
from ntlm import HTTPNtlmAuthHandler
user = 'username'
password = "password"
url = "https://portal.whatever.com/"
passman = urllib2.HTTPPasswordMgrWithDefaultRealm()
passman.add_password(None, url, user, password)
# create the NTLM authentication handler
auth_NTLM = HTTPNtlmAuthHandler.HTTPNtlmAuthHandler(passman)
# create and install the opener
opener = urllib2.build_opener(auth_NTLM)
urllib2.install_opener(opener)
# create a header
user_agent = 'Mozilla/4.0 (compatible; MSIE 5.5; Windows NT)'
header = { 'Connection' : 'Keep-alive', 'User-Agent' : user_agent}
response = urllib2.urlopen(urllib2.Request(url, None, header))
When I run this (with a real username, password and url) I get the following:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "ntlm2.py", line 21, in <module>
response = urllib2.urlopen(urllib2.Request(url, None, header))
File "C:\Python27\lib\urllib2.py", line 126, in urlopen
return _opener.open(url, data, timeout)
File "C:\Python27\lib\urllib2.py", line 400, in open
response = meth(req, response)
File "C:\Python27\lib\urllib2.py", line 513, in http_response
'http', request, response, code, msg, hdrs)
File "C:\Python27\lib\urllib2.py", line 432, in error
result = self._call_chain(*args)
File "C:\Python27\lib\urllib2.py", line 372, in _call_chain
result = func(*args)
File "C:\Python27\lib\urllib2.py", line 619, in http_error_302
return self.parent.open(new, timeout=req.timeout)
File "C:\Python27\lib\urllib2.py", line 400, in open
response = meth(req, response)
File "C:\Python27\lib\urllib2.py", line 513, in http_response
'http', request, response, code, msg, hdrs)
File "C:\Python27\lib\urllib2.py", line 432, in error
result = self._call_chain(*args)
File "C:\Python27\lib\urllib2.py", line 372, in _call_chain
result = func(*args)
File "C:\Python27\lib\urllib2.py", line 619, in http_error_302
return self.parent.open(new, timeout=req.timeout)
File "C:\Python27\lib\urllib2.py", line 400, in open
response = meth(req, response)
File "C:\Python27\lib\urllib2.py", line 513, in http_response
'http', request, response, code, msg, hdrs)
File "C:\Python27\lib\urllib2.py", line 438, in error
return self._call_chain(*args)
File "C:\Python27\lib\urllib2.py", line 372, in _call_chain
result = func(*args)
File "C:\Python27\lib\urllib2.py", line 521, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 401: Unauthorized
The thing that is most interesting about this trace to me is that the final line says a 401 error was sent back. From what I have read the 401 error is the first message sent back to the client when NTLM is started. I was under the impression that the purpose of python-ntml was to handle the NTLM process for me. Is that wrong or am I just using it incorrectly? Also I'm not bounded to using python for this, so if there is an easier way to do this in another language let me know (From what I seen a-googling there isn't).
Thanks!

If the site is using NTLM authentication, the headers attribute of the resulting HTTPError should say so:
>>> try:
... handle = urllib2.urlopen(req)
... except IOError, e:
... print e.headers
...
<other headers>
WWW-Authenticate: Negotiate
WWW-Authenticate: NTLM