I have a command-line Python script that is looping through the rows of a table, using the python-requests library to make PUT requests to an API for each row. The syntax looks like:
updateResponse = requests.put(updateRowUrl, data=json.dumps(payload), headers=headers)
The process works well for me, but I have a user getting the following error after the process makes several successful calls:
File "C:\Python27\lib\site-packages\requests\api.py", line 105, in put
return request('put', url, data=data, **kwargs)
File "C:\Python27\lib\site-packages\requests\api.py", line 49, in request
return session.request(method=method, url=url, **kwargs)
File "C:\Python27\lib\site-packages\requests\sessions.py", line 457, in reques
t
resp = self.send(prep, **send_kwargs)
File "C:\Python27\lib\site-packages\requests\sessions.py", line 569, in send
r = adapter.send(request, **kwargs)
File "C:\Python27\lib\site-packages\requests\adapters.py", line 407, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', error(10060, 'A con
nection attempt failed because the connected party did not properly respond afte
r a period of time, or established connection failed because connected host has
failed to respond'))
Judging by the 10060 error message I thought that the connection was timing out, so I suggested that the user add a timeout=None argument to the put call, but this doesn't seem to have solved this issue.
Are there any suggestions on what to try next?
Related
import requests
r = requests.get('https://www.whosampled.com/search/?q=marvin+gaye')`
This returns the following error
Traceback (most recent call last):
File "C:\Users\thoma\Downloads\RealisticMellowProfile\Python\New folder\Term project demo.py", line 8, in <module>
r = requests.get('https://www.whosampled.com/search/?q=marvin+gaye')
File "c:\users\thoma\miniconda3\lib\site-packages\requests\api.py", line 75, in get
return request('get', url, params=params, **kwargs)
File "c:\users\thoma\miniconda3\lib\site-packages\requests\api.py", line 60, in request
return session.request(method=method, url=url, **kwargs)
File "c:\users\thoma\miniconda3\lib\site-packages\requests\sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "c:\users\thoma\miniconda3\lib\site-packages\requests\sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "c:\users\thoma\miniconda3\lib\site-packages\requests\adapters.py", line 498, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
You can change the user agent so the server does not close the connection:
import requests
headers = {"User-Agent": "Mozilla/5.0"}
r = requests.get('https://www.whosampled.com/search/?q=marvin+gaye', headers=headers)
The url is broken (or the server serving this url)
Try to get it with
wget https://www.whosampled.com/search/?q=marvin+gaye
or with
curl https://www.whosampled.com/search/?q=marvin+gaye
Use try / except to handle such situations.
However you wnat be able to gat data from it (same as with wget or curl)
import requests
try:
r = requests.get('https://www.whosampled.com/search/?q=marvin+gaye')`
except requests.exceptions.ConnectionError:
print("can't get data from this server")
r = None
if r is not None:
# handle succesful request
else:
# handler error situation
i've ran a python bot on raspberry with raspbian for over a month without any problem.
Now, since some days, i'm facing this error, and i'm pretty sure i didn't touch anything in bot code and raspberry system.
File "/home/pi/Desktop/cugino/wrapper.py", line 92, in sendMessage
'reply_to_message_id':reply_to_message_id
File "/home/pi/Desktop/cugino/wrapper.py", line 55, in post
r = requests.post("%s/bot%s/%s" % (self.api_url,self.token,method),params,files=files,timeout=60)
File "/home/pi/.local/lib/python2.7/site-packages/requests/api.py", line 109, in post
return request('post', url, data=data, json=json, **kwargs)
File "/home/pi/.local/lib/python2.7/site-packages/requests/api.py", line 50,
in request
response = session.request(method=method, url=url, **kwargs)
File "/home/pi/.local/lib/python2.7/site-packages/requests/sessions.py",
line 465, in request
resp = self.send(prep, **send_kwargs)
File "/home/pi/.local/lib/python2.7/site-packages/requests/sessions.py",
line 573, in send
r = adapter.send(request, **kwargs)
File "/home/pi/.local/lib/python2.7/site-packages/requests/adapters.py",
line
415, in send
raise ConnectionError(err, request=request)
ConnectionError: ('Connection aborted.', error(101, 'Network is unreachable'))
The code that trigger the error is
r = requests.post("%s/bot%s/%s" % (self.api_url,self.token,method),params,files=files,timeout=60)
all was working, then suddently this error pops out.
The real strange thing is that this error occours occasionally, the bot run fine, it works, but it have alot of delay when this error occour, then it start to work normally, erro 101, and so on.
I use no proxy, no ipv6. Browser, internet and another bot runs fine without problems.
I'm trying to do a simple post request, I'm using a list because I want to send all my post request at the same time using thread. Here is an example of an url :
s = "https://emoncms.org/input/post.json?node="+str(test)+"&json={test_stack_overflow:0}&apikey="+str(apikey)
list.append(threading.Thread(target=requests.post, args=([s, ])))
I was using this code maybe 3 months ago and it worked perfectly.
I wanted to get back on this project this week and I realized that I got some errors, this one particularly :
Exception in thread Thread-14:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 763, in run
self.__target(*self.__args, **self.__kwargs)
File "/usr/lib/python2.7/dist-packages/requests/api.py", line 94, in post
return request('post', url, data=data, json=json, **kwargs)
File "/usr/lib/python2.7/dist-packages/requests/api.py", line 49, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 457, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 569, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 420, in send
raise SSLError(e, request=request)
SSLError: <unprintable SSLError object>
I got an other error, ConnectionError but I think it's due to the network or because the website can't follow it's activity or is down. I leave you the traceback if you want :
ConnectionError: ('Connection aborted.', error(101, 'Network is unreachable'))
This code is only a part of my project, the code is running every minutes and I don't know why but this issue (SSLError) comes only maybe 10 times a day. I got this script running on different Raspberry Pi and some have the same problem but not the same frequency, others don't have it at all.
Any ideas on what is going ?
Thanks in advance !
Use verify=False in the requests method like this
import requests
url="https://emoncms.org/input/post.json?node="+str(test)+"&json={test_stack_overflow:0}&apikey="+str(apikey)
requests.post(url,verify=False)
If you are using with threads then it will be like
list.append(threading.Thread(target=requests.post, args=(url,),kwargs={"verify":False})) #**kwargs should be passed seperately.
You are getting this error because python requests tries to verify certificate for https connections so you have to override it by passing verify=False or you can also provide certificate in verify like this requests.get(url,verify="/path/to/certificate.ext")
Also I doubt that this should be a get request because query parameters won't come in post request as of my knowledge. So if you use GET method same verify applies there too.
I am writing a python script that does these steps below.
Query a MongoDB database
Parse and aggregate results
Upload data to a ServiceNow table via a REST API
This works most of the time but occasionally I see this error:
requests.exceptions.ChunkedEncodingError: ("Connection broken: error(104, 'Connection reset by peer')", error(104, 'Connection reset by peer'))
This error stops the script and prevents the entire data set from being captured.
What can I do to alleviate this issue?
Python 2.7.5
Code
#!/usr/bin/env python
from config import *
import os, sys
mypath = os.path.dirname(os.path.realpath(__file__))
sys.path.append(os.path.join(mypath, "api-python-client"))
from apiclient.mongo import *
from pymongo import MongoClient
import json
import requests
from bson.json_util import dumps
client = MongoClient(mongo_uri)
#Create ServiceNow URL
svcnow_url = create_svcnow_url('u_imp_cmps')
#BITSDB Nmap Collection
db = client[mongo_db]
#Aggregate - RDBMS equivalent to Alias select x as y
#Rename fields to match ServiceNow field names
computers = db['computer'].aggregate([
{"$unwind": "$hostnames"},
{"$project" : {
"_id":0,
"u_hostname": "$hostnames.name",
"u_ipv4": "$addresses.ipv4",
"u_status": "$status.state",
"u_updated_timestamp": "$last_seen"
}}
])
j = dumps({"records":computers})
#print(j)
#Set proper headers
headers = {"Content-Type":"application/json","Accept":"application/json"}
#Build HTTP Request
response = requests.post(url=svcnow_url, auth=(svcnow_user, svcnow_pwd), headers=headers ,data=j)
#Check for HTTP codes other than 200
if response.status_code != 200:
print('Status:', response.status_code, 'Headers:', response.headers, 'Response Text', response.text, 'Error Response:',response.json())
exit()
#Decode the JSON response into a dictionary and use the data
print('Status:',response.status_code,'Headers:',response.headers,'Response:',response.json())
Error
Traceback (most recent call last):
File "/usr/src/computer_pingable_import.py", line 50, in <module>
response = requests.post(url=svcnow_url, auth=(svcnow_user, svcnow_pwd), headers=headers ,data=j)
File "/usr/local/lib/python2.7/site-packages/requests/api.py", line 107, in post
return request('post', url, data=data, json=json, **kwargs)
File "/usr/local/lib/python2.7/site-packages/requests/api.py", line 53, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 468, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 608, in send
r.content
File "/usr/local/lib/python2.7/site-packages/requests/models.py", line 737, in content
self._content = bytes().join(self.iter_content(CONTENT_CHUNK_SIZE)) or bytes()
File "/usr/local/lib/python2.7/site-packages/requests/models.py", line 663, in generate
raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ("Connection broken: error(104, 'Connection reset by peer')", error(104, 'Connection reset by peer'))
Another attempt and a slightly different error message:
Traceback (most recent call last):
File "/usr/src/computer_pingable_import.py", line 50, in <module>
response = requests.post(url=svcnow_url, auth=(svcnow_user, svcnow_pwd), headers=headers ,data=j)
File "/usr/local/lib/python2.7/site-packages/requests/api.py", line 107, in post
return request('post', url, data=data, json=json, **kwargs)
File "/usr/local/lib/python2.7/site-packages/requests/api.py", line 53, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 468, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 576, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python2.7/site-packages/requests/adapters.py", line 426, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', error(104, 'Connection reset by peer'))
ServiceNow prevents REST transactions from running for longer than 60 seconds.
I'm not too sure how large your dataset is, but you will want to chunk the data into smaller pieces to ensure the transaction always runs.
I am trying to send a request to an API I have set up on an AWS machine.
The code I use is as follows:
import requests
import json
report_dict = {
"client_name": "Wayne Enterprises",
"client_id": 123,
"report_type": "api_testing",
"timestamp_generated": "2015-07-29T11:00:00Z",
"report_data": {"revenue": 9000.00}
}
report_json = json.dumps(report_dict)
resp = requests.post("https://my-url.com:8080/my-api/reports", data=report_json,verify=False)
Doing this, I get:
Traceback (most recent call last):
File "art2_java_test.py", line 124, in <module>
main()
File "art2_java_test.py", line 9, in main
test_post_good_data()
File "art2_java_test.py", line 29, in test_post_good_data
resp = requests.post("https://my-url.com:8080/my-api/reports", data=report_json,verify=Fal
se)
File "C:\Python27\lib\site-packages\requests-2.7.0-py2.7.egg\requests\api.py",
line 109, in post
return request('post', url, data=data, json=json, **kwargs)
File "C:\Python27\lib\site-packages\requests-2.7.0-py2.7.egg\requests\api.py",
line 50, in request
response = session.request(method=method, url=url, **kwargs)
File "C:\Python27\lib\site-packages\requests-2.7.0-py2.7.egg\requests\sessions
.py", line 465, in request
resp = self.send(prep, **send_kwargs)
File "C:\Python27\lib\site-packages\requests-2.7.0-py2.7.egg\requests\sessions
.py", line 573, in send
r = adapter.send(request, **kwargs)
File "C:\Python27\lib\site-packages\requests-2.7.0-py2.7.egg\requests\adapters
.py", line 428, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: [SSL: UNKNOWN_PROTOCOL] unknown protocol (_ssl.c:5
90)
But when I send the request as http instead of https, it (usually) works fine. I've found some evidence that this can have to do with proxy servers, but I am not using one. Are there any other potential reasons for this error? This is a website only available on my company's local network, if that's relevant.
.... https://my-url.com:8080/my-api/reports
...But when I send the request as http instead of https, it (usually) works fine.
My guess is that you are trying the same port 8080 for http and https. But, servers usually listen on a single port either for http or https and not both. This means that if your client is trying to start the TLS handshake needed for https against this port it will get a plain error message back. The client then tries to interpret this error message as TLS and returns some weird error messages, because the response is not TLS at all.