it is the first time that I am working with a REST API in a jupyter notebook and I don't know what I am doing wrong here. When I try to execute the following code in a cell, the cell runs forever without throwing any errors. First I did not include the close method from the request package, but then I thought the problem might be the open connection. However including the close method also did not help. Do you know what could be the reason?
api_key = "exampletoken"
header = {'authorization':"Bearer {}".format(api_key)}
payload = {}
r = request.post('exampleurl', headers = header, data = payload)
r.close()
Thanks in advance!
runs forever without throwing any errors.
By default requests does not timeout, so it can wait infinite amount of time. This might cause behavior you described and mean server did not respond. To figure if that is cause, please set timeout for example
r = request.post('exampleurl', headers = header, data = payload, timeout=180)
would raise Exception after 180 seconds (i.e. 3 minutes) if it do not get response. If you want to know more about timeouts in requests I suggest reading realpython.com tutorial
Related
I have a program like this:
for i in range(25200):
time.sleep(1)
with requests.Session() as s:
data = {'ContractCode' : 'SAFMO98' }
r = s.post('http://cdn.ime.co.ir/Services/Fut_Live_Loc_Service.asmx/GetContractInfo', json = data ).json()
for key, value in r.items():
plt.clf()
last_prices = (r[key]['LastTradedPrice'])
z.append(last_prices)
plt.figure(1)
plt.plot(z)
Sometimes server rejects the connection and gives Exceeds request message. Or sometimes I lost my connection, etc.
Then I must re run my program and I will loose my plotted graph, and also the time my program was disconnected and the data I lost through this time. So what I like to do is add something to my program to keep my connection against interupts/desconnections. I mean my program wouldn't stop when it lost the connection or rejected from server side and will keep it's work when it connected again.
How is it possible?
EDIT: I edited my code like following but don't know how good is this way?
try:
for i in range(25200):
time.sleep(1)
with requests.Session() as s:
data = {'ContractCode' : 'SAFMO98' }
r =s.post('http://cdn.ime.co.ir/Services/Fut_Live_Loc_Service.asmx/GetContractInfo', json = data ).json()
for key, value in r.items():
plt.clf()
last_prices = (r[key]['LastTradedPrice'])
z.append(last_prices)
plt.figure(1)
plt.plot(z)
except:
pass
You have at least two connection failure events here, and either might result in an inability to connect for undefined amounts of time. A good option here is exponential backoff.
Basically, you attempt an operation, detect failures you know will require retrying, and wait. Each subsequent time the operation fails (in this case, presumably throwing an exception), you wait a multiple of the previous wait time. The idea is that, if you're being rate limited, you'll wait longer and longer until the API you're connecting to stops rejecting your requests. Also, if you've been physically disconnected, you'll attempt fewer connections over time, rather than spamming requests at a dead adapter.
There's a Python library, backoff, that handles most of the work involved in this for you with a decorator.
I know that requests.get() provides an HTTP interface so that the programmer can make various requests to a HTTP server.
That tells me that somewhere a port must be opened so that the request can happen.
Taking that into account, what would happen if the script is stopped (say, by a Key Board Interrupt, so the machine that is executing the script remains connected to the internet) before the request is answered/complete?
Would the port/connection remain opened?
Does the port/connection close automatically?
The short answer to the question is: requests will close a connection in the case of any exception, including KeyboardInterrupt and SystemExit.
A little digging into the requests source code reveals that requests.get ultimately calls the HTTPAdapter.send method (which is where all the magic happens).
There are two ways in which a request might be made within the send method: chunked or not chunked. Which send we perform depends on the value of the request.body and the Content-Length header:
chunked = not (request.body is None or 'Content-Length' in request.headers)
In the case where the request body is None or the Content-Length is set, requests will make use of the high-level urlopen method of urllib3:
if not chunked:
resp = conn.urlopen(
method=request.method,
url=url,
body=request.body,
# ...
)
The finally block of the urllib3.PoolManager.urlopen method has code that handles closing the connection in the case where the try block didn't execute successfully:
clean_exit = False
# ...
try:
# ...
# Everything went great!
clean_exit = True
finally:
if not clean_exit:
# We hit some kind of exception, handled or otherwise. We need
# to throw the connection away unless explicitly told not to.
# Close the connection, set the variable to None, and make sure
# we put the None back in the pool to avoid leaking it.
conn = conn and conn.close()
release_this_conn = True
In the case where the response can be chunked, requests goes a bit lower level and uses the underlying low level connection provided by urllib3. In this case, requests still handles the exception, it does this with a try / except block that starts immediately after grabbing a connection, and finishes with:
low_conn = conn._get_conn(timeout=DEFAULT_POOL_TIMEOUT)
try:
# ...
except:
# If we hit any problems here, clean up the connection.
# Then, reraise so that we can handle the actual exception.
low_conn.close()
raise
Interestingly the connection may not be closed if there are no errors, depending on how you have configured connection pooling for urllib3. In the case of a successful execution, the connection is put back into the connection pool (though I cannot find a _put_conn call in the requests source for the chunked send, which might be a bug in the chunked work-flow).
On a much lower level, when a program exits, the OS kernel closes all file descriptors opened by that program. These include network sockets.
I've written a simple python code to login to a forum, in order to keep alive and gain the online time. The code is as following:
logPara = {'username':user,'password':pwd}
s = requests.Session()
s.post(forumUrl,data=logPara)
homePage = requests.get(pageUrl)
I can get the correct homePage and am sure the login is successful. While I'm curious how long will this Session() last? If my program only contains theses four lines, will the Session() close thus the online status is lost?
Yes, definitely the session will be lost.
So, you two options for making the session last longer, One is as the answer posted by #Seekheart. Second is to save the session state in a file using python's pickle and load it again when needed. But this also will depend on the cookie expiration etc.
This is how you can do it.
When making the session request:
import pickle
import requests
logPara = {'username':user,'password':pwd}
s = requests.Session()
s.post(forumUrl,data=logPara)
homePage = requests.get(pageUrl)
with open('temp.dat', 'w') as f:
pickle.dump(s, f)
When you want to get the state back later:
import pickle
with open('temp.dat', 'r') as f:
s = pickle.load(f)
When you run the script, unless the script is being told to run endlessly or after a certain condition, it'll terminate almost immediately. So for your script it'll end after you run it. To continue running it you can put in a condition like for example.
while 1:
#Run your code
In the following code snippet, I know for a fact that https://asdasdasdasd.vm:8080/v2/api-docs does not exist. It fails a DNS lookup. Unfortunately, the get() never seems to return, raise, or timeout. My logs have only "A" in them. I would expect A C D or A B D. But I only ever see A in the logs.
try:
sys.stderr.write("A")
resp = requests.get("https://asdasdasdasd.vm:8080/v2/api-docs", timeout=1.0)
sys.stderr.write("B")
except:
sys.stderr.write("C")
sys.stderr.write("D")
sys.stderr.flush()
return swag
(That URL is not sanitized for this post. That's actually the URL I'm trying to use while working on this question.)
What am I missing here?
EDIT - I have also tried specifying the timeout as (1.0,1.0) but the behavior did not change.
EDIT2 - Per suggestions below, I ran my code from the python and ipython consoles. The code behaves as I expect (ACD). Of course, in my real application, I am not running this code from the command line. I don't know how this matters, but the method containing the code is being invoked by a web service. Specifically, a Swagger endpoint. With my browser, I hit an endpoint that's supposed to return our Swagger documentation. The endpoint (which uses flask_swagger) invokes init_swagger(...). init_swagger() calls my method with a Swagger object. That's it. How this matters, I cannot say. It doesn't make any sense to me that something outside of my method should somehow be able to mess with my exception handling.
The only thing I can think of is that Swagger has jacked with the requests class. But now it is dinner time and I am going home.
The following code for me returns A, C, D
import requests
from requests.exceptions import ConnectionError
try:
print("A")
resp = requests.get("https://asdasdasdasd.vm:8080/v2/api-docs", timeout=1.0)
print("B")
except ConnectionError:
print("C")
print("D")
This is because the host cannot be resolved for me, if I swap it out for localhost...
resp = requests.get("http://localhost/v2/api-docs", timeout=1.0)
...then I see an A, followed by a period of time before C and D show.
From reading the comments, I know what is up...
builtins has a ConnectionError that can be used (without importing anything). Requests doesn't use this exception, instead it uses the one found in requests.exceptions if you wish to catch the ConnectionError you must catch the correct exception, or it will drop out and not execute the except clause.
I have a python script that pulls from various internal network sources. With how our systems are set up we will initiate a urllib pull from a network location and it will get hung up waiting forever for a response on certain parts of the network. I would like my script to check that if it hasnt finished the pull in lets say 5 minutes it will pass the function and attempt to pull from the next address, and record it to a bad directory repository(so we can go check out which systems get hung up, there's like over 20,000 IP addresses we are checking some with some older scripts running on them that no longer work but will still try and run when requested, and they never stop trying to run)
Im familiar with having a script pause at a certain point
import time
time.sleep(300)
What Im thinking from a psuedo code perspective (not proper python just illustrating the idea)
import time
import urllib2
url_dict = ['http://1', 'http://2', 'http://3', ...]
fail_log_path = 'C:/Temp/fail_log.txt'
for addresses in url_dict:
clock_value = time.start()
while clock_value <= 300:
print str(clock_value)
res = urllib2.retrieve(url)
if res != []:
pass
else:
fail_log = open(fail_log_path, 'a')
fail_log.write("Failed to pull from site location: " + str(url) + "\n")
faile_log.close
Update: a specific option for this dealing with urls timeout for urllib2.urlopen() in pre Python 2.6 versions
Found this answer which is more in line with the overall problem of my question:
kill a function after a certain time in windows
Your code as is doesn't seem to describe what you were saying. It seems you want the if/else check inside your while loop. On top of that, you would want to loop over the ip addresses and not over a time period as your code is currently written (otherwise you will keep requesting the same ip address every time). Instead of keeping track of time yourself, I would suggest reading up on urllib.request.urlopen - specifically the timeout parameter. Once set, that function call will throw a socket.timeout exception once the time limit is reached. Surround that with a try/except block catching that error and then handle it appropriately.