I am making an API call to the ECB and try to return an url for a csv file, but i always get a xml file any help would be appreciated.
My reference: https://www.datacareer.de/blog/accessing-ecb-exchange-rate-data-in-python/
My code:
import requests # 2.18.4
import pandas as pd # 0.23.0
import io
from datetime import datetime, timedelta
# Building blocks for the URL
entrypoint = 'https://sdw-wsrest.ecb.europa.eu/service/' # Using protocol 'https'
resource = 'data' # The resource for data queries is always'data'
flowRef ='EXR' # Dataflow describing the data that needs to be returned, exchange rates in this case
key = 'D.CHF.EUR.SP00.A' # Defining the dimension values, explained below
# Define the parameters
parameters = {
'startPeriod': (datetime.now()-timedelta(1)).strftime("%Y-%m-%d"), # Start date of the time series
'endPeriod': datetime.now().strftime("%Y-%m-%d") # End of the time series
}
# Construct the URL: https://sdw-wsrest.ecb.europa.eu/service/data/EXR/D.CHF.EUR.SP00.A
request_url = entrypoint + resource + '/' + flowRef + '/' + key
# Make the HTTP request again, now requesting for CSV format
response = requests.get(request_url, params=parameters, headers={'Accept': 'text/csv'})
# Response succesful? (Response code 200)
print(response)
print(response.url)
Related
I have written a sample python code to post log data to Fluntd endpoint of EFK stack. When I send 400 logs at a time, status_code is 200 and can see all the logs at Kibana Dashboard, But when I send 500 logs at a time, the status_code is 414.
Here is the sample python code:
import sys
import json
from datetime import datetime
import random
import requests
f = open('/etc/td-agent/data2.json',)
data = json.load(f)
input = file(sys.argv[-1])
actions = []
url = ''
u_name = ''
p_word = ''
for line in input:
temp = json.loads(line)
tenantid = temp['HTTP_FLUENT_TAG']
message = temp['message']
message_json = json.loads(message)
h_name = data['account_details'][tenantid]['hostname']
u_name = data['account_details'][tenantid]['username']
p_word = data['account_details'][tenantid]['password']
url = 'https://' + h_name
for element in message_json:
temp = str(element['date'])
url = url + '?time=' + temp
action = {
"msg": element['log'],
"id": element['ID']
}
actions.append(action)
r = requests.post(url, auth=(u_name, p_word), json=actions)
print(r.status_code)
f.close()
Can anyone please help how to send huge load at a time at the Fluentd endpoint.
For Elasticsearch endpoint, we can use the elasticsearch api and it has also bulk feature, which helps to send huge amount of data at a time. I am looking for if there is any such way for Fluentd Endpoint.
There are two ways this can be done, either there are two ways.
create a JSON file of request body JSON objects and zip them to send them all in a single request
Call APIs multiple times with maximum possible JSONs as you mentioned above.
According to the documentation for the Cosmos DB REST API, with every API call, the Authorization header must be set. The value for this is constructed as described here: https://learn.microsoft.com/en-us/rest/api/cosmos-db/access-control-on-cosmosdb-resources
I am implementing this in Python as follows:
def get_authorisation_token(verb, resource_type, resource_id, date, master_key):
key = base64.b64decode(master_key)
text = f"""{verb.lower()}\n{resource_type.lower()}\n{resource_id.lower()}\n{date.lower()}\n\n"""
text_encoded = text.encode('utf-8')
signature_hash = hmac.new(key, text_encoded, digestmod=hashlib.sha256).digest()
signature = base64.b64encode(signature_hash).decode()
key_type = 'master'
version = '1.0'
uri = f'type={key_type}&ver={version}&sig={signature}'
uri_encoded = urllib.parse.quote(uri)
return uri_encoded
Since this is sent with every call, the auth token needs to be re-created to match the request URL. So, for example to get a list of databases, one must provide the resource type to be dbs and the resource link/ID to be an empty string with the URL: https://{databaseaccount}.documents.azure.com/dbs/
The part I cannot figure out, is the correct combination of resource type and resource ID/link to get all users from a particular database. Documentation can be found here: https://learn.microsoft.com/en-us/rest/api/cosmos-db/list-users
I have tried some combinations but nothing returns the users, I just get a 401:
{
"code": "Unauthorized",
"message": "The input authorization token can't serve the request. Please check that the expected payload is built as per the protocol, and check the key being used. Server used the following payload to sign: 'get\nusers\ndbs/<db_name>\nmon, 09 nov 2020 23:37:24 gmt\n\n'\r\nActivityId: 697a4159-f160-4aab-ae90-6cb5eaadb710, Microsoft.Azure.Documents.Common/2.11.0"
}
Regarding the issue, please refer to the following code
from wsgiref.handlers import format_date_time
from datetime import datetime
from time import mktime
import base64
from urllib.parse import quote
import hmac
from hashlib import sha256
import requests
from azure.cosmos.auth import GetAuthorizationHeader
from azure.cosmos.cosmos_client import CosmosClientConnection
master_key = ''
database_name = ''
key = base64.b64decode(master_key)
verb = 'GET'
resource_type = 'users'
resource_id = f'dbs/{database_name}'
now = datetime.now()
stamp = mktime(now.timetuple())
date = format_date_time(stamp)
print(date)
text = "{verb}\n{resource_type}\n{resource_id}\n{date}\n{other}\n".format(
verb=(verb.lower() or ''),
resource_type=(resource_type.lower() or ""),
resource_id=(resource_id or ""),
date=date.lower(),
other="".lower())
body = text.encode("utf-8")
digest = hmac.new(key, body, sha256).digest()
signature = base64.encodebytes(digest).decode("utf-8")
key_type = 'master'
version = '1.0'
uri = f'type={key_type}&ver={version}&sig={signature[:-1]}'
uri_encoded = quote(uri)
url = "https://<>.documents.azure.com:443/dbs/<>/users"
payload = {}
headers = {
'Authorization': uri_encoded,
'x-ms-date': date,
'x-ms-version': '2018-12-31'
}
response = requests.request("GET", url, headers=headers, data=payload)
print(response.text)
I'm not sure why I am receiving this error. There is a decoder.py file in my python folder.
import requests
import json
import common
session = requests.Session()
uri = "http://www.missingkids.com"
json_srv_uri = uri + "/missingkids/servlet/JSONDataServlet"
search_uri = "?action=publicSearch"
child_detail_uri = "?action=childDetail"
session.get(json_srv_uri + search_uri + "&searchLang=en_US&search=new&subjToSearch=child&missState=CA&missCountry=US") #Change missState=All for all states
response = session.get(json_srv_uri + search_uri + "&searchLang=en_US&goToPage=1")
dct = json.loads(response.text)
pgs = int(dct["totalPages"])
print("found {} pages".format(pgs))
missing_persons = {}
The URL http://www.missingkids.com/missingkids/servlet/ returns a 404 Error. Thus, there is no JSON data for Requests to return. Fixing the URL so that it points to a valid destination will allow Requests to return page content.
To make a search for a missing child registered in that website's database, try this URL: http://www.missingkids.com/gethelpnow/search
After every HTTP call you need to check the status code.
Example
import requests
r = requests.get('my_url')
# status code 'OK' is very popular and its numeric value is 200
# note that there are other status codes as well
if r.status_code == requests.codes.ok:
# do your thing
else:
# we have a problem
Alright, so I'm a little outside of my league on this one I think.
I'm attempting to facilitate custom HTTP headers what is noted here:
API-Key = API key
API-Sign = Message signature using HMAC-SHA512 of (URI path + SHA256(nonce + POST data)) and base64 decoded secret API key
from https://www.kraken.com/help/api
I'm trying to work solely out of urllib if at all possible.
Below is one of many attempts to get it encoded like required:
import os
import sys
import time
import datetime
import urllib.request
import urllib.parse
import json
import hashlib
import hmac
import base64
APIKey = 'ThisKey'
secret = 'ThisSecret'
data = {}
data['nonce'] = int(time.time()*1000)
data['asset'] = 'ZUSD'
uripath = '/0/private/TradeBalance'
postdata = urllib.parse.urlencode(data)
encoded = (str(data['nonce']) + postdata).encode()
message = uripath.encode() + hashlib.sha256(encoded).digest()
signature = hmac.new(base64.b64decode(secret),
message, hashlib.sha512)
sigdigest = base64.b64encode(signature.digest())
#this is purely for checking how things are coming along.
print(sigdigest.decode())
headers = {
'API-Key': APIKey,
'API-Sign': sigdigest.decode()
}
The above may be working just fine, where I'm struggling now is appropriately getting it to the site.
This is my most recent attempt:
myBalance = urllib.urlopen('https://api.kraken.com/0/private/TradeBalance', urllib.parse.urlencode({'asset': 'ZUSD'}).encode("utf-8"), headers)
Any help is greatly appreciated.
Thanks!
urlopen doesn't support adding headers, so you need to create a Request object and pass it to urlopen:
url = 'https://api.kraken.com/0/private/TradeBalance'
body = urllib.parse.urlencode({'asset': 'ZUSD'}).encode("utf-8")
headers = {
'API-Key': APIKey,
'API-Sign': sigdigest.decode()
}
request = urllib.request.Request(url, data=body, headers=headers)
response = urllib.request.urlopen(request)
I'm trying to use Python to access the trading API at poloniex.com, a cryptocurrency exchange. To do this I must follow this prescription:
All calls to the trading API are sent via HTTP POST to https://poloniex.com/tradingApi and must contain the following headers:
Key - Your API key.
Sign - The query's POST data signed by your key's "secret" according to the HMAC-SHA512 method.
Additionally, all queries must include a "nonce" POST parameter. The nonce parameter is an integer which must always be greater than the previous nonce used.
Here is what I have so far. My current issue is that I do not know how to compile the POST url so that it can be signed without sending the incomplete request first. This obviously doesn't work.
import requests
import hmac
import hashlib
import time
headers = { 'nonce': '',
'Key' : 'myKey',
'Sign': '',}
payload = { 'command': 'returnCompleteBalances',
'account': 'all'}
secret = 'mySecret'
headers['nonce'] = int(time.time())
response = requests.post( 'https://poloniex.com/tradingApi', params= payload, headers= headers )
headers['Sign'] = hmac.new( secret, response.url, hashlib.sha512)
Create a prepared request; you can add headers to that after the body has been created:
import requests
import hmac
import hashlib
request = requests.Request(
'POST', 'https://poloniex.com/tradingApi',
data=payload, headers=headers)
prepped = request.prepare()
signature = hmac.new(secret, prepped.body, digestmod=hashlib.sha512)
prepped.headers['Sign'] = signature.hexdigest()
with requests.Session() as session:
response = session.send(prepped)
I changed your params argument to data; for a POST request it is customary to send the parameters in the body, not the URL.
For the nonce, I'd use a itertools.count() object, seeded from the current time so restarts don't affect it. According to the Poloniex API documentation (which you quoted in your question), the nonce is part of the POST body, not the headers, so put it in the payload dictionary:
from itertools import count
import time
# store as a global variable
NONCE_COUNTER = count(int(time.time() * 1000))
# then every time you create a request
payload['nonce'] = next(NONCE_COUNTER)
Using int(time.time()) would re-use the same number if you created more than one request per second. The example code provided by Poloniex uses int(time.time()*1000) to make it possible to create a request every microsecond instead, but using your own monotonically increasing counter (seeded from time.time()) is far more robust.
You can also encapsulate the digest signing process in a custom authentication object; such an object is passed in the prepared request as the last step in preparation:
import hmac
import hashlib
class BodyDigestSignature(object):
def __init__(self, secret, header='Sign', algorithm=hashlib.sha512):
self.secret = secret
self.header = header
self.algorithm = algorithm
def __call__(self, request):
body = request.body
if not isinstance(body, bytes): # Python 3
body = body.encode('latin1') # standard encoding for HTTP
signature = hmac.new(self.secret, body, digestmod=self.algorithm)
request.headers[self.header] = signature.hexdigest()
return request
Use this with your requests calls:
response = requests.post(
'https://poloniex.com/tradingApi',
data=payload, headers=headers, auth=BodyDigestSignature(secret))
The argument passed in is the secret used in the HMAC digest; you can also pass in a different header name.