I have been building my own python (version 3.2.1) trading application in a practice account of a Forex provider (OANDA) but I am having some issues in receiving the streaming prices with a Linux debian-based OS.
In particular, I have followed their "Python streaming rates" guide available here: http://developer.oanda.com/rest-live/sample-code/.
I have a thread calling the function 'connect_to_stream' which prints out all the ticks received from the server:
streaming_thread = threading.Thread(target=streaming.connect_to_stream, args=[])
streaming_thread.start()
The streaming.connect_to_stream function is defined as following:
def connect_to_stream():
[..]#provider-related info are passed here
try:
s = requests.Session()
url = "https://" + domain + "/v1/prices"
headers = {'Authorization' : 'Bearer ' + access_token,
'Connection' : 'keep-alive'
}
params = {'instruments' : instruments, 'accountId' : account_id}
req = requests.Request('GET', url, headers = headers, params = params)
pre = req.prepare()
resp = s.send(pre, stream = True, verify = False)
return resp
except Exception as e:
s.close()
print ("Caught exception when connecting to stream\n%s" % str(e))
if response.status_code != 200:
print (response.text)
return
for line in response.iter_lines(1):
if line:
try:
msg = json.loads(line)
print(msg)
except Exception as e:
print ("Caught exception when connecting to stream\n%s" % str(e))
return
The msg variable contains the tick received for the streaming.
The problem is that I receive ticks for three hours on average after which the connection gets dropped and the script either hangs without receiving any ticks or throws an exception with reason "Connection Reset by Peer".
Could you please share any thoughts on where I am going wrong here? Is it anything related to the requests library (iter_lines maybe)?
I would like to receive ticks indefinitely unless a Keyboard exception is raised.
Thanks
That doesn't seem too weird to me that a service would close connections living for more than 3 hours.
That's probably a safety on their side to make sure to free their server sockets from ghost clients.
So you should probably just reconnect when you are disconnected.
try:
s = requests.Session()
url = "https://" + domain + "/v1/prices"
headers = {'Authorization' : 'Bearer ' + access_token,
'Connection' : 'keep-alive'
}
params = {'instruments' : instruments, 'accountId' : account_id}
req = requests.Request('GET', url, headers = headers, params = params)
pre = req.prepare()
resp = s.send(pre, stream = True, verify = False)
return resp
except SocketError as e:
if e.errno == errno.ECONNRESET:
pass # connection has been reset, reconnect.
except Exception as e:
pass # other exceptions but you'll probably need to reconnect too.
Related
I am using StormProxies to access Etsy data but despite using proxies and implementing retries I am getting 429 Too Many Requests error most of the time(~80%+). Here is my code to access data:
import requests
def create_request(url, logging, headers={}, is_proxy=True):
r = None
try:
proxies = {
'http': 'http://{}'.format(PROXY_GATEWAY_IP),
'https': 'http://{}'.format(PROXY_GATEWAY_IP),
}
with requests.Session() as s:
retries = Retry(total=5, backoff_factor=1, status_forcelist=[502, 503, 504, 429])
s.mount('http://', HTTPAdapter(max_retries=retries))
if is_proxy:
r = s.get(url, proxies=proxies, timeout=30, headers=headers)
else:
r = s.get(url, headers=headers, timeout=30)
r.raise_for_status()
if r.status_code != 200:
print('Status Code = ', r.status_code)
if logging is not None:
logging.info('Status Code = ' + str(r.status_code))
except Exception as ex:
print('Exception occur in create_request for the url:- {url}'.format())
crash_date = time.strftime("%Y-%m-%d %H:%m:%S")
crash_string = "".join(traceback.format_exception(etype=type(ex), value=ex, tb=ex.__traceback__))
exception_string = '[' + crash_date + '] - ' + crash_string + '\n'
print('Could not connect. Proxy issue or something else')
print('==========================================================')
print(exception_string)
finally:
return r
StormProxies guys say that I implement retries, this is how I have done but it is not working for me.
I am using Python multiprocessing and spawning 30+ threads at a time.
My recommendation is, remove huge overhead based on thread management in one process (30+ is really lot).
It is more efficiency to use more processes with only a few threads (2-4 threads, based on delay with I/O) because threads in one process have to play with GIL (Global Interpreter Lock). In this case all will be only about configuration for your Python code.
I am trying to fetch the json response using the VirusTotal API. To fetch the results as a response I have created a python script which is as shown below : Then error show like this : [Winerror]:10054 An existing connection was forcibly closed by the remote host
import requests
url = "https://www.virustotal.com/vtapi/v2/file/scan"
api_key = "MyApiKey"
params = {"apikey": api_key}
files = {"file": ("app.exe", open("C:/Users/Administrator/Desktop/malware detection/app.exe", "rb")}
try:
response = requests.post(url, files=files, params=params)
print(response.json())
except Exception as e:
print(e)
To provide more information, please try the following:
import requests
url = "https://www.virustotal.com/vtapi/v2/file/scan"
api_key = "MyApiKey"
params = {"apikey": api_key}
files = {"file": ("app.exe", open("C:/Users/Administrator/Desktop/malware detection/app.exe", "rb"))}
try:
response = requests.post(url, files=files, params=params)
print(response.json())
except Exception as e:
print(e)
And post the error response you receive from the updated code.
This issue is resolved, The issue was encountered only for the skype.exe file. while scanning using malware machine learning.
I would like the following script to try every url in url_list, and if it exist print it exist(url) and if not print don't(url) and if request timeout skip to the next url using "requests" lib:
url_list = ['www.google.com','www.urlthatwilltimeout.com','www.urlthatdon\'t exist']
def exist:
if request.status_code == 200:
print"exist{0}".format(url)
else:
print"don\'t{0}".format(url)
a = 0
while (a < 2):
url = urllist[a]
try:
request = requests.get(url, timeout=10)
except request.timeout:#any option that is similar?
print"timed out"
continue
validate()
a+=1
Based on this SO answer
below is code which will limit the total time taken by a GET request as well
as discern other exceptions that may happen.
Note that in requests 2.4.0 and later you may specify a connection timeout and read timeout
by using the syntax:
requests.get(..., timeout=(...conn timeout..., ...read timeout...))
The read timeout, however, only specifies the timeout between individual
read calls, not a timeout for the entire request.
Code:
import requests
import eventlet
eventlet.monkey_patch()
url_list = ['http://localhost:3000/delay/0',
'http://localhost:3000/delay/20',
'http://localhost:3333/', # no server listening
'http://www.google.com'
]
for url in url_list:
try:
with eventlet.timeout.Timeout(1):
response = requests.get(url)
print "OK -", url
except requests.exceptions.ReadTimeout:
print "READ TIMED OUT -", url
except requests.exceptions.ConnectionError:
print "CONNECT ERROR -", url
except eventlet.timeout.Timeout, e:
print "TOTAL TIMEOUT -", url
except requests.exceptions.RequestException, e:
print "OTHER REQUESTS EXCEPTION -", url, e
And here is an express server you can use to test it:
var express = require('express');
var sleep = require('sleep')
var app = express();
app.get('/delay/:secs', function(req, res) {
var secs = parseInt( req.params.secs )
sleep.sleep(secs)
res.send('Done sleeping for ' + secs + ' seconds')
});
app.listen(3000, function () {
console.log('Example app listening on port 3000!');
});
I'm using the following python script to connect with adsl modem and retrive wifi information
host = 192.168.1.1
pwd = **********
conn = httplib.HTTPConnection(host)
auth = b64encode("admin:"+pwd).decode("ascii")
head = { 'Authorization' : 'Basic %s' % auth }
conn.request("GET", "/basic/home_wlan.htm", headers = head)
resp = conn.getresponse()
#Do something ....
conn.close()
Most of the times evereything is OK, but sometimes happemes to receive 401 - Unauthorized http error.
When I retry executing immidiatly after this error, everything works ok. Now, to overcome this I added the following code immidiatly before #Do something...:
if resp.status == 401:
conn = httplib.HTTPConnection(host)
auth = b64encode("admin:"+pwd).decode("ascii")
head = { 'Authorization' : 'Basic %s' % auth }
conn.request("GET", "/basic/home_wlan.htm", headers = head)
resp = conn.getresponse()
In other terms I'm repeating the connection negotiation if 401 occurs. In my opinion this is not so elegant to do this. Can someone give me a more acceptable solution?
When I use the following function with the Python 3.2.3 package in cygwin it hangs on any request to any https host. It will throw with this error: [Errno 104] Connection reset by peer, after 60 seconds.
UPDATE: I thought it was limited to only cygwin, but this also happens in Windows 7 64bit with Python 3.3. I'll try 3.2 right now. The error when using the windows command shell is:
urlopen error [WinError 10054] An existing connection was forcibly closed by the remote host
UPDATE2(Electric-Bugaloo): This is limited to a couple of sites that I'm trying to use. I tested against google and other major sites with no issue. It appears it's related to this bug:
http://bugs.python.org/issue16361
Specifically, the server is hanging after the client-hello. It's due to the version of openssl that shipped with the compiled versions of python3.2 and 3.3. It's mis-identifying the ssl version of the server. Now I need code to auto downgrade my version of ssl to sslv3 when opening a connection to the affected sites like in this post:
How to use urllib2 to get a webpage using SSLv3 encryption
but I can't get it to work.
def worker(url, body=None, bt=None):
'''This function does all the requests to wherever for data
takes in a url, optional body utf-8 encoded please, and optional body type'''
hdrs = {'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
'Accept-Language': 'en-us,en;q=0.5',
'Accept-Encoding': 'gzip,deflate',
'User-Agent': "My kewl Python tewl!"}
if 'myweirdurl' in url:
hdrs = {'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
'Accept-Language': 'en-us,en;q=0.5',
'Accept-Encoding': 'gzip,deflate',
'User-Agent': "Netscape 6.0"}
if bt:
hdrs['Content-Type'] = bt
urlopen = urllib.request.urlopen
Request = urllib.request.Request
start_req = time.time()
logger.debug('request start: {}'.format(datetime.now().ctime()))
if 'password' not in url:
logger.debug('request url: {}'.format(url))
req = Request(url, data=body, headers=hdrs)
try:
if body:
logger.debug("body: {}".format(body))
handle = urlopen(req, data=body, timeout=298)
else:
handle = urlopen(req, timeout=298)
except socket.error as se:
logger.error(se)
logger.error(se.errno)
logger.error(type(se))
if hasattr(se, 'errno') == 60:
logger.error("returning: Request Timed Out")
return 'Request Timed Out'
except URLError as ue:
end_time = time.time()
logger.error(ue)
logger.error(hasattr(ue, 'code'))
logger.error(hasattr(ue, 'errno'))
logger.error(hasattr(ue, 'reason'))
if hasattr(ue, 'code'):
logger.warn('The server couldn\'t fulfill the request.')
logger.error('Error code: {}'.format(ue.code))
if ue.code == 404:
return "Resource Not Found (404)"
elif hasattr(ue, 'reason') :
logger.warn('We failed to reach a server with {}'.format(url))
logger.error('Reason: {}'.format(ue.reason))
logger.error(type(ue.reason))
logger.error(ue.reason.errno)
if ue.reason == 'Operation timed out':
logger.error("Arrggghh, timed out!")
else:
logger.error("Why U no match my reason?")
if ue.reason.errno == 60:
return "Operation timed out"
elif hasattr(ue, 'errno'):
logger.warn(ue.reason)
logger.error('Error code: {}'.format(ue.errno))
if ue.errno == 60:
return "Operation timed out"
logger.error("req time: {}".format(end_time - start_req))
logger.error("returning: Server Error")
return "Server Error"
else:
resp_headers = dict(handle.info())
logger.debug('Here are the headers of the page : {}'.format(resp_headers))
logger.debug("The true URL in case of redirects {}".format(handle.geturl()))
try:
ce = resp_headers['Content-Encoding']
except KeyError as ke:
ce = None
else:
logger.debug('Content-Encoding: {}'.format(ce))
try:
ct = resp_headers['Content-Type']
except KeyError as ke:
ct = None
else:
logger.debug('Content-Type: {}'.format(ct))
if ce == "gzip":
logger.debug("Unzipping payload")
bi = BytesIO(handle.read())
gf = GzipFile(fileobj=bi, mode="rb")
if "charset=utf-8" in ct.lower() or ct == 'text/html' or ct == 'text/plain':
payload = gf.read().decode("utf-8")
else:
logger.debug("Unknown content type: {}".format(ct))
sys.exit()
return payload
else:
if ct is not None and "charset=utf-8" in ct.lower() or ct == 'text/html' or ct == 'text/plain':
return handle.read().decode("utf-8")
else:
logger.debug("Unknown content type: {}".format(ct))
sys.exit()
I figured it out, here's the code block necessary to make this work on Windows:
'''had to add this windows specific block to handle this bug in urllib2:
http://bugs.python.org/issue11220
'''
if "windows" in platform().lower():
if 'my_wacky_url' or 'my_other_wacky_url' in url.lower():
import ssl
ssl_context = urllib.request.HTTPSHandler(
context=ssl.SSLContext(ssl.PROTOCOL_TLSv1))
opener = urllib.request.build_opener(ssl_context)
urllib.request.install_opener(opener)
#end of urllib workaround
I added this blob right before the first try: block and it worked like a charm. Thanks for the assistance andrean!