I have a program like this:
for i in range(25200):
time.sleep(1)
with requests.Session() as s:
data = {'ContractCode' : 'SAFMO98' }
r = s.post('http://cdn.ime.co.ir/Services/Fut_Live_Loc_Service.asmx/GetContractInfo', json = data ).json()
for key, value in r.items():
plt.clf()
last_prices = (r[key]['LastTradedPrice'])
z.append(last_prices)
plt.figure(1)
plt.plot(z)
Sometimes server rejects the connection and gives Exceeds request message. Or sometimes I lost my connection, etc.
Then I must re run my program and I will loose my plotted graph, and also the time my program was disconnected and the data I lost through this time. So what I like to do is add something to my program to keep my connection against interupts/desconnections. I mean my program wouldn't stop when it lost the connection or rejected from server side and will keep it's work when it connected again.
How is it possible?
EDIT: I edited my code like following but don't know how good is this way?
try:
for i in range(25200):
time.sleep(1)
with requests.Session() as s:
data = {'ContractCode' : 'SAFMO98' }
r =s.post('http://cdn.ime.co.ir/Services/Fut_Live_Loc_Service.asmx/GetContractInfo', json = data ).json()
for key, value in r.items():
plt.clf()
last_prices = (r[key]['LastTradedPrice'])
z.append(last_prices)
plt.figure(1)
plt.plot(z)
except:
pass
You have at least two connection failure events here, and either might result in an inability to connect for undefined amounts of time. A good option here is exponential backoff.
Basically, you attempt an operation, detect failures you know will require retrying, and wait. Each subsequent time the operation fails (in this case, presumably throwing an exception), you wait a multiple of the previous wait time. The idea is that, if you're being rate limited, you'll wait longer and longer until the API you're connecting to stops rejecting your requests. Also, if you've been physically disconnected, you'll attempt fewer connections over time, rather than spamming requests at a dead adapter.
There's a Python library, backoff, that handles most of the work involved in this for you with a decorator.
Related
it is the first time that I am working with a REST API in a jupyter notebook and I don't know what I am doing wrong here. When I try to execute the following code in a cell, the cell runs forever without throwing any errors. First I did not include the close method from the request package, but then I thought the problem might be the open connection. However including the close method also did not help. Do you know what could be the reason?
api_key = "exampletoken"
header = {'authorization':"Bearer {}".format(api_key)}
payload = {}
r = request.post('exampleurl', headers = header, data = payload)
r.close()
Thanks in advance!
runs forever without throwing any errors.
By default requests does not timeout, so it can wait infinite amount of time. This might cause behavior you described and mean server did not respond. To figure if that is cause, please set timeout for example
r = request.post('exampleurl', headers = header, data = payload, timeout=180)
would raise Exception after 180 seconds (i.e. 3 minutes) if it do not get response. If you want to know more about timeouts in requests I suggest reading realpython.com tutorial
Hi guys I'm working a on script that will get data from a host using the Data Communications Standard (Developed by: Data Communication Standard Committee Lens Processing Division of The Vision Council), by serial port and pass the data into ModBus Protocol for the device to perform it's operations.
Since I don't fiscally have access to the host machine I'm trying to develop a secondary script to emulate the host. I am currently on the stage where I need to read a lot of information from the serial port and I get only part of the data. I was hoping to get the whole string sent on the send_job() function on my host emulator script.
Guys also can any of you tell me if this would be a good approach? the only thing the machine is supposed to do is grab 2 values from the host response and assign them to two modbus holding registers.
NOTE: the initialization function is hard coded because it will always be the same and the actual response data will not matter except for status. Also the job request is hard coded i only pass the job # that i get from a modbus holding register, the exact logic on how the host resolved this should not matter i only need to send the job number scanned from the device in this format.
main script:
def request_job_modbus(job):
data = F'[06][1c]req=33[0d][0a]job={job}[0d][0a][1e][1d]'.encode('ascii')
writer(data)
def get_job_from_serial():
response = serial_client.read_all()
resp = response.decode()
return resp
# TODO : SEND INIT SEQUENCE ONCE AND VERIFY IF REQUEST status=0
initiation_request()
init_response_status = get_init_status()
print('init method being active')
print(get_init_status())
while True:
# TODO: get job request data
job_serial = get_job_from_serial()
print(job_serial)
host emulation script:
def send_job():
job_response = '''[06][1c]ans=33[0d]job=30925[0d]status=0;"ok"[0d]do=l[0d]add=;2.50[0d]ar=1[0d]
bcerin=;3.93[0d]bcerup=;-2.97[0d]crib=;64.00[0d]do=l[0d]ellh=;64.00[0d]engmask=;613l[0d]
erdrin=;0.00[0d]erdrup=;10.00[0d]ernrin=;2.00[0d]ernrup=;-8.00[0d]ersgin=;0.00[0d]
ersgup=;4.00[0d]gax=;0.00[0d]gbasex=;-5.30[0d]gcrosx=;-7.96[0d]kprva=;275[0d]kprvm=;0.55[0d]
ldpath=\\uscqx-tcpmain-at\lds\iot\do\800468.sdf[0d]lmatid=;151[0d]lmatname=;f50[0d]
lnam=;vsp_basic_fh15[0d]sgerin=;0.00[0d]sgerup=;0.00[0d]sval=;5.18[0d]text_11=;[0d]
text_12=;[0d]tind=;1.53[0d][1e][1d]'''.encode('ascii')
writer(job_response)
def get_init_request():
req = p.readline()
print(req)
request = req.decode()[4:11]
# print(request)
if request == 'req=ini':
print('request == req=ini??? <<<<<<< cumple condicion y enviala respuesta')
send_init_response()
send_job()
while True:
# print(get_init_request())
get_init_request()
what I get in screen: main script
init method being active
bce
erd
condition was met init status=0
outside loop
ers
condition was met init status=0
inside while loop
trigger reset <<<--------------------
5782
`:lmatid=;151[0d]lmatname=;f50[0d]
lnam=;vsp_basic_fh15[0d]sgerin=;0.00[0d]sgerup=;0.00[0d]sval=;5.18[0d]text_11=;[0d]
text_12=;[0d]tind=;1.53[0d][1e][1d]
outside loop
condition was met init status=0
outside loop
what I get in screen: host emulation script
b'[1c]req=ini[0d][0a][1e][1d]'
request == req=ini??? <<<<<<< cumple condicion y enviala respuesta
b''
b'[06][1c]req=33[0d][0a]job=5782[0d][0a][1e][1d]'
b''
b''
b''
b''
b''
b''
I'm suspect you're trying to write too much at once to a hardware buffer that is fairly small. Especially when dealing with low power hardware, assuming you can stuff an entire message into a buffer is not often correct. Even full modern PC's sometimes have very small buffers for legacy hardware like serial ports. You may find when you switch from development to actual hardware, that the RTS and DTR lines need to be used to determine when to send or receive data. This will be up to whoever designed the hardware unfortunately, as they are often also ignored.
I would try chunking your data transfer into smaller bits as a test to see if the whole message gets through. This is a quick and dirty first attempt that may have bugs, but it should get you down the right path:
def get_job_from_serial():
response = b'' #buffer for response
while True:
try:
response += serial_client.read() #read any available data or wait for timeout
#this technically could only be reading 1 char at a time, but any
#remotely modern pc should easily keep up with 9600 baud
except serial.SerialTimeoutException: #timeout probably means end of data
#you could also presumably check the length of the buffer if it's always
#a fixed length to determine if the entire message has been sent yet.
break
return response
def writer(command):
written = 0 #how many bytes have we actually written
chunksize = 128 #the smaller you go, the less likely to overflow
# a buffer, but the slower you go.
while written < len(command):
#you presumably might have to wait for p.dtr() == True or similar
#though it's just as likely to not have been implemented.
written += p.write(command[written:written+chunksize])
p.flush() #probably don't actually need this
P.S. I had to go to the source code for p.read_all (for some reason I couldn't find it online), and it does not do what I think you expect it does. The exact code for it is:
def read_all(self):
"""\
Read all bytes currently available in the buffer of the OS.
"""
return self.read(self.in_waiting)
There is no concept of waiting for a complete message, it just a shorthand for grab everything currently available.
I have a process that runs data acquisition using PySerial. It's working fine now, but there's a weird thing I had to do to make it work continuously, and I'm not sure this is normal, so I'm asking this question.
What happens: It looks like that the connection drops now and then! Around once every 30-60 minutes, with big error bars (could go for hours and be OK, but sometimes happens often).
My question: Is this standard?
My temporary solution: I wrote a simple "reopen" function that looks like this:
def ReopenDevice(devObject):
try:
devObject.close()
devObject.open()
except Exception as e:
print("Error while trying to connect to device " + devObject.port + ". The error says: " + str(e))
time.sleep(2)
And what I do is that if data pulling fails for 2 minutes, I reopen the device with this function, and it continues working well with no problems.
My program model: It's a GUI program, where the user clicks something like "Start", and that button does some preparations and runs a function through multiprocessing.Process() that starts with:
devObj = serial.Serial()
#... other params
devObj.open()
and that function then runs a while loop that keeps polling data with something like:
bytesToRead = devObj.inWaiting()
if bytesToRead != 0:
buffer = decodeString(devObj.read(bytesToRead))
#process buffer and push it to a list...
The way I know that the problem happened, is that devObj.inWaiting() Keeps returning zero... no matter how much data there's on the device!
Is this behavior expected and should always be considered whether it happens or doesn't happen?
The problem reduced a lot after not calling inWaiting() very frequently. Anyway, I kept the reconnect part to ensure that my program never fails. Thanks for "Kobi K" for suggesting the possible cause of the problem.
I have a python script that pulls from various internal network sources. With how our systems are set up we will initiate a urllib pull from a network location and it will get hung up waiting forever for a response on certain parts of the network. I would like my script to check that if it hasnt finished the pull in lets say 5 minutes it will pass the function and attempt to pull from the next address, and record it to a bad directory repository(so we can go check out which systems get hung up, there's like over 20,000 IP addresses we are checking some with some older scripts running on them that no longer work but will still try and run when requested, and they never stop trying to run)
Im familiar with having a script pause at a certain point
import time
time.sleep(300)
What Im thinking from a psuedo code perspective (not proper python just illustrating the idea)
import time
import urllib2
url_dict = ['http://1', 'http://2', 'http://3', ...]
fail_log_path = 'C:/Temp/fail_log.txt'
for addresses in url_dict:
clock_value = time.start()
while clock_value <= 300:
print str(clock_value)
res = urllib2.retrieve(url)
if res != []:
pass
else:
fail_log = open(fail_log_path, 'a')
fail_log.write("Failed to pull from site location: " + str(url) + "\n")
faile_log.close
Update: a specific option for this dealing with urls timeout for urllib2.urlopen() in pre Python 2.6 versions
Found this answer which is more in line with the overall problem of my question:
kill a function after a certain time in windows
Your code as is doesn't seem to describe what you were saying. It seems you want the if/else check inside your while loop. On top of that, you would want to loop over the ip addresses and not over a time period as your code is currently written (otherwise you will keep requesting the same ip address every time). Instead of keeping track of time yourself, I would suggest reading up on urllib.request.urlopen - specifically the timeout parameter. Once set, that function call will throw a socket.timeout exception once the time limit is reached. Surround that with a try/except block catching that error and then handle it appropriately.
I coded a basic socket system with "select". I want get the list of connected clients instantly.
When the timeout of "select" has passed and several clients come after, it's the drama..
Example - Concerns:
I have 3 clients with one that connects before the timeout, 2 others are connected after the timeout, so I'm going to refresh my list if it took into account two other clients after the timeout.
1st result: I display my variable "list", I see the first socket that is connected before the timeout + one of the other socket who is connected after the timeout. Total: 2 of 3 clients
2nd result: I still re-display my variable "list", and the three clients are there ....
But I want the list without having to re-display the list every time for every customer you can imagine I have 10 clients and I have to show my liste10 times
So I thought to use the asyncore module who is more fluid, what do you think? Do you have a solution for me (easier)? Should I use the multi-threading or stayed on asyncore or select module?
EDIT CODE SOURCE:
import socket, select
hote = ''
port = 81
mainConnection = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
mainConnection.bind((hote, port))
mainConnection.listen(5)
print("Listen to {}".format(port))
client_online = []
while True:
connection_access, wlist, xlist = select.select([mainConnection], [], [], 10)
for connexion in connection_access:
connection_client, infos_connexion = connexion.accept()
client_online.append(connection_client)
refresh = input(">>> ")
while True:
try:
refresh = int(refresh)
except ValueError:
print("Not allowed")
refresh = int(refresh)
else:
break
if refresh == 1:
print("List client : {}".format(client_online))
There are three major problems with your code:
You call input in your loop. This function will block until ENTER is pressed.
If a non-integer is input from the console, you will get an exception. You handle that exception, but you handle it wrongly. Instead or asking for input again, you simply try to perform the same operation that caused the exception again.
You only check for incoming connection in your select call. You never check if any of the connected sockets have sent anything.
The major problem here for you is the call to input as it will completely stop your program until input from the console is entered.
Your post is very unclear but I can tell you that the problem is that you aren't understanding how to use select.
The code you posted only calls select one time. The program gets to the select() call and waits for mainConnection to be readable (or for the timeout). If mainConnection becomes readable before the timeout, select() returns with one readable file descriptor which you then process in your for loop. But that's it. select is never called again and so your program never checks for any more incoming connections.
In almost every application select should be in a loop. Each time through the loop the program waits in the select() call until one or more sockets is ready for reading or writing. When that happens, select gives you the file descriptors that are ready and it's your job to have other code actually do something. For example, if select returns a socket's file descriptor as readable it's your job to call .recv() on that socket.
You can certainly use asyncore. In fact, I think you should study the source code for asyncore to learn how to properly use select.