I did my best writing this API, but I'm now stuck on the while loop.
My main goal is to scan batch by batch the API results and write it in the database.
The way the code is written is bringing me straight the results as offset=4000
Don't know what am I doing wrong.
Follow my code
#!/bin/env python
from asyncio.windows_events import NULL
from re import X
from typing import ItemsView
import requests
import json
import hashlib
import base64
import time
import hmac
import pandas as pd
import datetime
import pyodbc
#Account Info
AccessId = ''
AccessKey = ''
Company = ''
#Request Info
httpVerb ='GET'
resourcePath = '/alert/alerts'
offset = 0
while offset < 5000:
# Query parameters
queryParams ='?size=1000&offset=' + str(offset) + '&sort=-startEpoch&filter=cleared:*,rule:critical'
offset += 1000
data = ''
#Construct URL
url = 'https://'+ Company +'.logicmonitor.com/santaba/rest' + resourcePath + queryParams
print(url)
#Get current time in milliseconds
epoch = str(int(time.time() * 1000))
#Concatenate Request details
requestVars = httpVerb + epoch + data + resourcePath
#Construct signature
hmac1 = hmac.new(AccessKey.encode(),msg=requestVars.encode(),digestmod=hashlib.sha256).hexdigest()
signature = base64.b64encode(hmac1.encode())
#Construct headers
auth = 'LMv1 ' + AccessId + ':' + signature.decode() + ':' + epoch
headers = {'Content-Type':'application/json','Authorization':auth}
#Make request
response = requests.get(url, data=data, headers=headers)
data = response.json()
alerts_df = pd.DataFrame(data['data']['items'])
alerts_df = alerts_df[['id','internalId','rule','monitorObjectName','startEpoch','endEpoch','cleared','resourceTemplateName']]
alerts_df['startEpoch'] = pd.to_datetime(alerts_df['startEpoch'],unit='s')
alerts_df['endEpoch'] = alerts_df['endEpoch'].apply(lambda x: pd.to_datetime(x,unit='s') if x !=0 else x)
print(alerts_df)
You are overwriting the variable constantly before even making one request.
Also, the while loop should be <= 5000 if you want to include the 5000 in the loop.
There are two ways to achieve it.
append the results to a list for later use
make the request after you generated the string
Related
I currently have a script pulling data from Instagram that looks like the code block posted below. As long as you enter your plug in your Instagram credentials under user_name and password, it should be fully reproducible.
It is taking the account listed in player_df, pulling a list of all their followers on Instagram, and taking that list of followers and pulling all of their bio information. But when I run it, I get the following error:
ClientConnectionError: timeout The read operation timed out
You can find the entire error log here, I just didn't want to post it in the original question because it would exceed the character limit.
As an attempt to fix this, I added in the sleep(300) functions to lessen the stress between API calls, but that doesn't seem to do the trick. What would be the best way to get around this so it doesn't timeout while trying to run?
from ftplib import error_proto
from hashlib import new
from multiprocessing.spawn import import_main_path
from time import sleep
from instagram_private_api import Client, ClientCompatPatch
from operator import itemgetter
import pandas as pd
import json
import requests
from collections import Counter
import datetime
import os.path
user_name = "XXXXX"
password = "XXXXX"
players = [['hannahkshepherd', '201683404']]
player_df = pd.DataFrame(players, columns=['username', 'userId'])
def pull_followers(username_instagram, userid_instagram):
followers = []
combinacao = []
results = api.user_followers(userid_instagram, rank_token=api.generate_uuid())
followers.extend(results.get('users', []))
next_max_id = results.get('next_max_id')
while next_max_id:
results = api.user_followers(userid_instagram, rank_token=api.generate_uuid(), max_id=next_max_id)
followers.extend(results.get('users', []))
next_max_id = results.get('next_max_id')
userid = [followers[i]['pk'] for i in range(0,len(followers))]
full_names = [followers[i]['full_name'] for i in range(0,len(followers))]
usernames = [followers[i]['username'] for i in range(0,len(followers))]
profile_pic_url = [followers[i]['profile_pic_url'] for i in range(0,len(followers))]
followers_text = ['follower' for i in range(0,len(followers))]
following_username = [str(username_instagram) for i in range(0,len(followers))]
following_userid = [str(userid_instagram) for i in range(0,len(followers))]
combinacao.extend([list(i) for i in zip(userid, full_names,
usernames, profile_pic_url, followers_text,
following_username, following_userid)])
combinacao = sorted(combinacao, key=itemgetter(2), reverse=False)
return combinacao
all_followers = []
for i in range(len(player_df)):
all_followers += pull_followers(player_df['username'][i], player_df["userId"][i])
def get_bios(followers):
bios = []
for follower in followers:
follower_id = follower[0]
bios += [[follower_id, api.user_info(follower_id)['user']['biography']]]
return bios
#sleep(300)
bios = get_bios(all_followers)
#sleep(300)
def print_bios():
s = ''
for row in bios:
s += '\n' + 'user_id: ' + str(row[0]) + ', bio: ' + str(row[1])
print(s)
The Problem:
I am trying to get a requests.get to print from a website and it needs a signature. From what I read on the website it says that it needs to take my secret key and make a signature out of it using HMAC-SHA1 base 64.
I know I need to import hmac into my python script but other than that I am not sure if my signature will be what the website is looking for.
This is a link to how the authentication is suppose to be setup: https://www.ninjarmm.com/dev-api/#the_authentication_header
On the website it gives an example of how the signature should be but I don't know how to apply this in python:
### How do I apply this in my python script's header?
Signature = Base64( HMAC-SHA1( YourSecretAccessKeyID, Base64( UTF-8- Encoding-Of( StringToSign ) ) ) ); StringToSign = HTTP-Verb + "n" + Content-MD5 + "n" + Content-Type + "n" + Date + "n" + CanonicalizedResource;
### How do I apply this in my python script's header?
Here is my Code:
import requests
import os
import json
from datetime import date
from email.utils import formatdate
ninjaapi = "some data"
ninjasak = "some data"
ninjaurl = "https://api.ninjarmm.com/v1/devices"
ninjaheaders = {"authorization":ninjaapi + ninjasak, "Date":now,}
ninjastatusresponse = requests.get(ninjaurl, headers=ninjaheaders)
ninja_json = ninjastatusresponse.json()
print(ninja_json)
Here is the results I get so far:
{'error': 'invalid_header', 'error_description': "Invalid 'Authorization' header", 'error_code': 1}
If anyone has any ideas that I can test let me know!
Thank you for your time,
So I found out there was a nice person who made a module package for everything already. I used his code in my program and it worked for building the signature.
Its called ninjarmm_api; here is the link: https://bitbucket.org/raptus-it/py-ninjarmm-api-client/src/master/ninjarmm_api/
Here is the link from the ninjadojo community.
https://ninjarmm.zendesk.com/hc/en-us/community/posts/360037165831-Python-client-library-for-API
Specifically, here is the code for the his auth.py which is what I was trying to figure out how to do.
from requests.auth import AuthBase
from email.utils import formatdate
import base64
import hmac
import hashlib
class auth(AuthBase):
NINJA_HDR_AUTH = "Authorization"
NINJA_HDR_DATE = "Date"
NINJA_ENCODING = "utf-8"
def __init__(self, accessKeyId, secretAccessKey):
self.access_key_id = accessKeyId
self.secret_access_key = secretAccessKey
self.tstamp = formatdate(timeval=None, localtime=False, usegmt=True)
def __call__(self, request):
sts_clear = request.method + "\n" # HTTP verb
sts_clear += "\n" # Content MD5
sts_clear += "\n" # Content type
sts_clear += self.tstamp + "\n" # Date
sts_clear += request.path_url # Canonicalized resource
sts_base64 = base64.b64encode(sts_clear.encode(self.NINJA_ENCODING))
sts_digest = hmac.new(self.secret_access_key.encode(self.NINJA_ENCODING), sts_base64, hashlib.sha1)
signature = base64.b64encode(sts_digest.digest())
request.headers[self.NINJA_HDR_AUTH] = "NJ " + self.access_key_id + ":" + signature.decode(self.NINJA_ENCODING)
request.headers[self.NINJA_HDR_DATE] = self.tstamp
return request
This my main Program to get data from URL and write in google sheet-
import gspread
from oauth2client.service_account import ServiceAccountCredentials
scope = ["https://spreadsheets.google.com/feeds",'https://www.googleapis.com/auth/spreadsheets',"https://www.googleapis.com/auth/drive.file","https://www.googleapis.com/auth/drive"]
import requests
import datetime
import pandas as pd
def writedata():
# Connect to Google Sheet
credentials = ServiceAccountCredentials.from_json_keyfile_name('creds.json',scope)
client = gspread.authorize(credentials)
sheet = client.open("Min15Scan").sheet1
symlist = sheet.col_values(1)
i = 2
while i <= len(symlist):
symbolfromexcel = symlist[i-1] + "EQN"
# URL of NSEIndia chart data
headers = {'User-Agent': 'Mozilla/5.0'}
try:
url = 'https://www.nseindia.com/api/chart-databyindex?index=' + symbolfromexcel
r = requests.get(url, headers=headers)
data = r.json()
prices = data['grapthData']
# Create Empty Dataframe/List to add data
rawdata = []
# Loop to get data & write into list (rawdata) as list of list
for item in prices:
dt = datetime.datetime.utcfromtimestamp(item[0] / 1000)
value = item[1]
rawdata.append([dt.strftime('%H-%M'), value])
# Create dataframe
data_df = pd.DataFrame(rawdata, columns=["Time", "ltp"])
min15 = data_df.loc[data_df["Time"] <= "09-30"]
min15 = min15["ltp"]
min15max = min15.max()
min15min = min15.min()
total = data_df["ltp"]
totalmax = total.max()
totalmin = total.min()
# Write data on raw sheet.update('B1', 'text')
sheet.update('B' + str(i) + ':E' + str(i), [[min15max, min15min, totalmax, totalmin]])
except:
sheet.update('B' + str(i) + ':E' + str(i), [[0, 0, 0, 0]])
pass
i = i + 1
Clock file-
from apscheduler.schedulers.blocking import BlockingScheduler
from GoogleSheetAPI import writedata
sched = BlockingScheduler()
sched.add_job(writedata, 'interval', minutes=5)
sched.start()
It's working fine if I run this on my PC but its writing Except part (sheet.update('B' + str(i) + ':E' + str(i), [[0, 0, 0, 0]])) every time, I am unable to understand what is error.
You code is encountering an exception and executing the "except:" statement. So it executes sheet.update('B' + str(i) + ':E' + str(i), [[0, 0, 0, 0]]) and then pass which is not very useful if you don't know what exception you could be getting.
Try printing the exception for example as follows:
except Exception as e:
print(e)
To find out more about the error.
import re
import time
import urllib
import base64
import hmac
import hashlib
import requests
import xml.etree.ElementTree as ET
from xml.etree import ElementTree as et
import codecs
import xml.etree.cElementTree as EF
from datetime import date
import bitly_api
def aws_signed_request(region, params, public_key, private_key, associate_tag=None, version='2011-08-01'):
# some paramters
method = 'GET'
host = 'webservices.amazon.' + region
uri = '/onca/xml'
# additional parameters
params['Service'] = 'AWSECommerceService'
params['AWSAccessKeyId'] = public_key
params['Timestamp'] = time.strftime('%Y-%m-%dT%H:%M:%SZ', time.gmtime())
params['Version'] = version
if associate_tag:
params['AssociateTag'] = associate_tag
# create the canonicalized query
canonicalized_query = [urllib.quote(param).replace('%7E', '~') + '=' + urllib.quote(params[param]).replace('%7E', '~')
for param in sorted(params.keys())]
canonicalized_query = '&'.join(canonicalized_query)
# create the string to sign
string_to_sign = method + '\n' + host + '\n' + uri + '\n' + canonicalized_query
#print string_to_sign
# calculate HMAC with SHA256 and base64-encoding
signature = base64.b64encode(hmac.new(key=private_key, msg=string_to_sign, digestmod=hashlib.sha256).digest())
# encode the signature for the request
signature = urllib.quote(signature).replace('%7E', '~')
print 'http://' + host + uri + '?' + canonicalized_query + '&Signature=' + signature
return 'http://' + host + uri + '?' + canonicalized_query + '&Signature=' + signature
This is part of my code that used to be functional. It would get the best selling Amazon items in several categories. However, this script is not longer functioning. Does anyone notice anything wrong?
Fixed, my public and private key was expired
I have spent hours in frustration and now I have this:
import requests, json, urllib
import time
import string, random
from hashlib import sha1
import hmac, binascii
def twitterIDGenerator(length):
toRet = ""
for i in range(0, length):
toRet = toRet + random.choice(string.hexdigits)
return toRet
def twitterSignatureGenerator(baseString, keyString):
hashed = hmac.new(keyString, baseString, sha1)
return binascii.b2a_base64(hashed.digest()).rstrip('\n')
OAUTH_CONSUMER_KEY = ''
OAUTH_NONCE = twitterIDGenerator(32)
OAUTH_SIGNATURE_METHOD = 'HMAC-SHA1'
OAUTH_TIMESTAMP = str(int(time.time()))
OAUTH_VERSION = '1.0'
# Get request token from Twitter
request_tokenURL = 'https://api.twitter.com/oauth/request_token'
request_tokenParameterString = ("oauth_consumer_key=" + OAUTH_CONSUMER_KEY +
"&oauth_nonce=" + OAUTH_NONCE + "&oauth_signature_method=" +
OAUTH_SIGNATURE_METHOD + "&oauth_timestamp=" + OAUTH_TIMESTAMP +
"&oauth_version=" + OAUTH_VERSION)
request_tokenSigBaseString = ("POST&https%3A%2F%2Fapi.twitter.com%2Foauth%2Frequest_token&" +
urllib.quote(request_tokenParameterString))
request_tokenSignature = twitterSignatureGenerator(request_tokenSigBaseString,
'[REDACTED consumer secret key]')
request_tokenHeaders = {'oauth_nonce': OAUTH_NONCE,
'oauth_callback': 'oob',
'oauth_signature_method': OAUTH_SIGNATURE_METHOD,
'oauth_timestamp': OAUTH_TIMESTAMP,
'oauth_consumer_key': OAUTH_CONSUMER_KEY,
'oauth_signature': urllib.quote(request_tokenSignature),
'oauth_version': OAUTH_VERSION}
request_tokenResponse = requests.post(request_tokenURL, headers=request_tokenHeaders)
print request_tokenResponse.text
So far, it is supposed to return a request_token so I can have my user go to the PIN website so I can get the access_token. But I just get "Failed to validate oauth signature and token" from Twitter.
A possible reason for this is wrong URL encoding. I see that Twitter needs RFC3986 encoding. Is there a way to do this in Python? If yes, should I do it only at the two locations I am currently using urllib.quote? Is my oauth_signature generated correctly?
The documentation is annoyingly convoluted.