We Keep Coding

Discord API: Providing messages from incorrect time period

Im trying to get all messages from a channel from 2 days ago till now. instead, it's giving me the messages from the inception of the channel. Don't know what I'm doing wrong here...
here is the code
import requests
import json
import datetime
def retrieve_messages(channelid, after=None, before=None):
headers = {
'authorization': "<TOKEN>"
}
# Build the query string, including the after and before parameters
# if they were specified
query_string = '?'
if after:
query_string += f'&after={after}'
if before:
query_string += f'&before={before}'
# Make the API call, including the query string
r = requests.get(
f'https://discord.com/api/v9/channels/{channelid}/messages{query_string}', headers=headers)
jsonn = json.loads(r.text)
for value in jsonn:
print(value['content'], '\n')
# Retrieve messages from the past two days
two_days_ago = datetime.datetime.utcnow() - datetime.timedelta(days=2)
after = int(two_days_ago.timestamp())
retrieve_messages("<CHANNEL_ID>", after=after)

I finally figured it out. It is in the documentation
# Define the DISCORD_EPOCH constant
DISCORD_EPOCH = 1420070400000
# Convert the x_days_ago variable to a timestamp in milliseconds
timestamp_ms = int(x_days_ago.timestamp() * 1000)
# Subtract the DISCORD_EPOCH constant from the timestamp to get the correct format for the after parameter
after = (timestamp_ms - DISCORD_EPOCH) << 22

The discord.py API uses timestamps in milliseconds, not seconds. So:
# ...
if after:
query_string += f'&after={after * 1000}' # Multiply by 1000 (convert to ms)
if before:
query_string += f'&before={before}'

Python API script

I am making a python script using API of a free test automation website called TestProject.
Link to their API: https://api.testproject.io/docs/v2/
Basically what i want to do is grab pdf of reports of all tests and save them somewhere.
But to make the GET request to do that i first need projectID and jobID which i already wrote functions getting them and saving them in the array.
But now i have a problem where its looping through both lists and not using correct projectID and jobID and its throwing errors because it does not exist.
So what i need is something to check if jobID is in projectID so that way i can make a GET request to get all the executionID's to get the PDF of the report.
I am kinda new to programming so i would love any help i can get. If anyone has any better solutions please feel free to let me know.
My script:
import requests
import json
import csv
from datetime import datetime
from jsonpath_ng import jsonpath, parse
API_key = 'api_key'
headers = {'Authorization':'{}'.format(API_key)}
list_projectId = []
list_jobId = []
list_executionId = []
ParseData_projectId = parse('$..id')
ParseData_jobId = parse('$..id')
ParseData_executionId = parse('$..id')
def parsing (response,ParseData,list_data):
# parses data and appends it to the list
Data = json.loads(response)
Parsaj = ParseData
Podatki = Parsaj.find(Data)
for i in range(0, len(Podatki)):
vrednost = Podatki[i].value
list_data.append(vrednost)
def projectId():
# gets all projectId's and saves them in list_projectId
url = 'https://api.testproject.io/v2/projects?_start=0'
response = requests.get(url,headers=headers)
response_json = response.json()
converted = json.dumps(response_json)
parsing(converted,ParseData_projectId,list_projectId)
def jobId():
# gets all jobId's and saves them in list_jobId
for i in range(0, len(list_projectId)):
id = list_projectId[i]
url = 'https://api.testproject.io/v2/projects/{}'.format(id) + '/jobs?onlyScheduled=false&_start=0'
response = requests.get(url,headers=headers)
response_json = response.json()
converted = json.dumps(response_json)
parsing(converted,ParseData_jobId,list_jobId)
def executionId():
# Their API link:
# https://api.testproject.io/v2/projects/{projectId}/jobs/{jobId}/reports?_start=0
# the for loop below does not work here is where i need the help:
for i in range(0, len(list_projectId)):
project_id = list_projectId[i]
job_id = list_jobId[i]
url = 'https://api.testproject.io/v2/projects/{}'.format(project_id) + '/jobs/{}'.format(job_id) + '/reports?_start=0'
response = requests.get(url,headers=headers)
response_json = response.json()
converted = json.dumps(response_json)
parsing(converted,ParseData_executionId,list_executionId)
projectId()
print("----------LIST PROJECT ID: ----------")
print(list_projectId)
print("")
jobId()
print("----------LIST JOB ID: ----------")
print(list_jobId)
executionId()
print("----------LIST EXECUTION ID: ----------")
print(list_executionId)

you have to use 'in' operator to check the value exist in the list data structure.

How can I bulk upload JSON records to AWS OpenSearch index using a python client library?

I have a sufficiently large dataset that I would like to bulk index the JSON objects in AWS OpenSearch.
I cannot see how to achieve this using any of: boto3, awswrangler, opensearch-py, elasticsearch, elasticsearch-py.
Is there a way to do this without using a python request (PUT/POST) directly?
Note that this is not for: ElasticSearch, AWS ElasticSearch.
Many thanks!

I finally found a way to do it using opensearch-py, as follows.
First establish the client,
# First fetch credentials from environment defaults
# If you can get this far you probably know how to tailor them
# For your particular situation. Otherwise SO is a safe bet :)
import boto3
credentials = boto3.Session().get_credentials()
region='eu-west-2' # for example
auth = AWSV4SignerAuth(credentials, region)
# Now set up the AWS 'Signer'
from opensearchpy import OpenSearch, RequestsHttpConnection, AWSV4SignerAuth
auth = AWSV4SignerAuth(credentials, region)
# And finally the OpenSearch client
host=f"...{region}.es.amazonaws.com" # fill in your hostname (minus the https://) here
client = OpenSearch(
hosts = [{'host': host, 'port': 443}],
http_auth = auth,
use_ssl = True,
verify_certs = True,
connection_class = RequestsHttpConnection
)
Phew! Let's create the data now:
# Spot the deliberate mistake(s) :D
document1 = {
"title": "Moneyball",
"director": "Bennett Miller",
"year": "2011"
}
document2 = {
"title": "Apollo 13",
"director": "Richie Cunningham",
"year": "1994"
}
data = [document1, document2]
TIP! Create the index if you need to -
my_index = 'my_index'
try:
response = client.indices.create(my_index)
print('\nCreating index:')
print(response)
except Exception as e:
# If, for example, my_index already exists, do not much!
print(e)
This is where things go a bit nutty. I hadn't realised that every single bulk action needs an, er, action e.g. "index", "search" etc. - so let's define that now
action={
"index": {
"_index": my_index
}
}
You can read all about the bulk REST API, there.
The next quirk is that the OpenSearch bulk API requires Newline Delimited JSON (see https://www.ndjson.org), which is basically JSON serialized as strings and separated by newlines. Someone wrote on SO that this "bizarre" API looked like one designed by a data scientist - far from taking offence, I think that rocks. (I agree ndjson is weird though.)
Hideously, now let's build up the full JSON string, combining the data and actions. A helper fn is at hand!
def payload_constructor(data,action):
# "All my own work"
action_string = json.dumps(action) + "\n"
payload_string=""
for datum in data:
payload_string += action_string
this_line = json.dumps(datum) + "\n"
payload_string += this_line
return payload_string
OK so now we can finally invoke the bulk API. I suppose you could mix in all sorts of actions (out of scope here) - go for it!
response=client.bulk(body=payload_constructor(data,action),index=my_index)
That's probably the most boring punchline ever but there you have it.
You can also just get (geddit) .bulk() to just use index= and set the action to:
action={"index": {}}
Hey presto!
Now, choose your poison - the other solution looks crazily shorter and neater.
PS The well-hidden opensearch-py documentation on this are located here.

conn = wr.opensearch.connect(
host=self.hosts, # URL
port=443,
username=self.username,
password=self.password
)
def insert_index_data(data, index_name='stocks', delete_index_data=False):
""" Bulk Create
args: body [{doc1}{doc2}....]
"""
if delete_index_data:
index_name = 'symbol'
self.delete_es_index(index_name)
resp = wr.opensearch.index_documents(
self.conn,
documents=data,
index=index_name
)
print(resp)
return resp

I have used below code to bulk insert records from postgres into OpenSearch ( ES 7.2 )
import sqlalchemy as sa
from sqlalchemy import text
import pandas as pd
import numpy as np
from opensearchpy import OpenSearch
from opensearchpy.helpers import bulk
import json
engine = sa.create_engine('postgresql+psycopg2://postgres:postgres#127.0.0.1:5432/postgres')
host = 'search-xxxxxxxxxx.us-east-1.es.amazonaws.com'
port = 443
auth = ('username', 'password') # For testing only. Don't store credentials in code.
# Create the client with SSL/TLS enabled, but hostname verification disabled.
client = OpenSearch(
hosts = [{'host': host, 'port': port}],
http_compress = True,
http_auth = auth,
use_ssl = True,
verify_certs = True,
ssl_assert_hostname = False,
ssl_show_warn = False
)
with engine.connect() as connection:
result = connection.execute(text("select * from account_1_study_1.stg_pred where domain='LB'"))
records = []
for row in result:
record = dict(row)
record.update(record['item_dataset'])
del record['item_dataset']
records.append(record)
df = pd.DataFrame(records)
#df['Date'] = df['Date'].astype(str)
df = df.fillna("null")
print(df.keys)
documents = df.to_dict(orient='records')
#bulk(es ,documents, index='search-irl-poc-dump', raise_on_error=True)\
#response=client.bulk(body=documents,index='sample-index')
bulk(client, documents, index='search-irl-poc-dump', raise_on_error=True, refresh=True)

REST request to App Store Connect API response with '401' | 'NOT_AUTHORIZED'

we launched an iOS-App and I want to grab some Information (e.g. Installations, Updates, Reviews) from the App Store Connect API.
I create an JSON Web Token as described in the official Apple documentation: Link
Afterwards I make a request with the token in the header. Now I get an '401' | 'NOT_AUTHORIZED' each time as an answer, see the following picture:
REST Response
In the following snippets you can see my python code (I tried to solve it in Python and R, but the result is always the same).
First, I create an JWT:
from datetime import datetime, timedelta
from jose import jwt, jws
import ecdsa
KEY_ID = "XXXXXXXXXX"
ISSUER_ID = "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX"
PRIVATE_KEY = open('AuthKey_XXXXXXXXXX.p8', 'r').read()
TIMESTAMP = int( (datetime.now() - timedelta(minutes = 45)).timestamp() * 1000)
claim = {"iss" : ISSUER_ID,
"exp" : TIMESTAMP,
"aud" : "appstoreconnect-v1"}
header = {
"alg": "ES256",
"kid": KEY_ID,
"typ": "JWT"
}
# Create the JWT
encoded = jwt.encode(claim, PRIVATE_KEY, algorithm='ES256', headers=header)
Now when I print encoded, I get to following JWT (looks valid for me):
'eyJ0eXAiOiJKV1QiLCJhbGciOiJFUzI1NiIsImtpZCI6IlhYWFhYWFhYWFgifQ.eyJpc3MiOiJYWFhYWFhYWC1YWFhYLVhYWFgtWFhYWC1YWFhYWFhYWFhYWFgiLCJleHAiOjE1NDUzOTc1MTQ1ODAsImF1ZCI6ImFwcHN0b3JlY29ubmVjdC12MSJ9.eTl6iaAW-Gp67FNmITrWCpLTtJzVdLYXIl5_KKgqaNgzwyGo7npBOBo9_u5PtLNnssQFEwJWbPND-6Ww5ACgEg'
Even if I decode the first two parts of the JWT via Base64 I get the right Header (it also contains the right algorithm for encoding: 'alg': 'ES256') and Claim:
from jose.utils import base64url_decode
print(base64url_decode(b'eyJ0eXAiOiJKV1QiLCJhbGciOiJFUzI1NiIsImtpZCI6IlhYWFhYWFhYWFgifQ'))
print(base64url_decode(b'eyJpc3MiOiJYWFhYWFhYWC1YWFhYLVhYWFgtWFhYWC1YWFhYWFhYWFhYWFgiLCJleHAiOjE1NDUzOTc1MTQ1ODAsImF1ZCI6ImFwcHN0b3JlY29ubmVjdC12MSJ9'))
See the following picture: Output Base64 Decoding
So now, that I'm think that the JWT-Object is ready I send the request to the API:
import requests
JWT = 'Bearer ' + encoded
URL = 'https://api.appstoreconnect.apple.com/v1/apps'
HEAD = {'Authorization': JWT}
print(HEAD)
R = requests.get(URL, headers=HEAD)
R.json()
And now we can see my problem, see the picture: Header | REST Response
Please note that I have hidden the KEY_ID, ISSUER_ID and PRIVATE_KEY for the example.

Your token contains an expiry time
"exp": 1545397514580,
which equals September 12th, 50941.
When I delete the last three digits
"exp": 1545397514,
I get December 21st, 2018
which makes much more sense.
Change that line
TIMESTAMP = int( (datetime.now() - timedelta(minutes = 45)).timestamp() * 1000)
to
TIMESTAMP = int( (datetime.now() - timedelta(minutes = 45)).timestamp())
exp is a timestamp that is defined as seconds since 01.01.1970 00:00
See also here

First of all, please don't open files without context managers. This string:
PRIVATE_KEY = open('AuthKey_XXXXXXXXXX.p8', 'r').read()
should be:
with open('AuthKey_XXXXXXXXXX.p8', 'r') as f:
PRIVATE_KEY = f.read()
It save you from many problems with unclosed files in future.
Then, check what token you've read from file. Is it correct?
The next problem I see is timestamp. "The token's expiration time, in Unix epoch time;" You provide it in milliseconds, I guess.

Here is working solution for me. Without delay it return 401 error
KEY_ID = "xxxxx"
ISSUER_ID = "xxxxx"
EXPIRATION_TIME = int(round(time.time() + (20.0 * 60.0))) # 20 minutes timestamp
PATH_TO_KEY = '../credentials/AuthKey_xxxxx.p8'
with open(PATH_TO_KEY, 'r') as f:
PRIVATE_KEY = f.read()
header = {
"alg": "ES256",
"kid": KEY_ID,
"typ": "JWT"
}
payload = {
"iss": ISSUER_ID,
"exp": EXPIRATION_TIME,
"aud": "appstoreconnect-v1"
}
# Create the JWT
token = jwt.encode(header, payload, PRIVATE_KEY)
JWT = 'Bearer ' + token.decode()
HEAD = {'Authorization': JWT}
# Without delay I got 401 error
time.sleep(5)
URL = 'https://api.appstoreconnect.apple.com/v1/apps';
r = requests.get(URL, params={'limit': 200}, headers=HEAD)

GoogleMaps API -address to coordinates (latitude,longitude)

This is driving me crazy.
I have deleted this key 1000 times so far.
Yesterday it worked like a charm, today not anymore
Here is the python code:
from googlemaps import GoogleMaps
gmaps = GoogleMaps("AIzaSyBIdSyB_td3PE-ur-ISjwFUtBf2O0Uo0Jo")
exactaddress ="1 Toronto Street Toronto"
lat, lng = gmaps.address_to_latlng(exactaddress)
print lat, lng
GoogleMapsError: Error 610: G_GEO_BAD_KEY
It is now returning the above error for no obvious reasons.
I don't think I have reached the request limit or the maximum rate
To stay on the safe side I even introduced delays (1sec) ...stil getting the same error
Does anybody have any idea how I can solve this?
Having to work with a different python module is fine if you can indicate an alternative to the one that I am currently using.
thanks
C
PS: the key is valid, it is a client key and it was automatically enabled when I enabled GoogleMAP API3 in the App console. No restrictions for domains or IPs
EDIT: So here is what I ended up using
def decodeAddressToCoordinates( address ):
urlParams = {
'address': address,
'sensor': 'false',
}
url = 'http://maps.google.com/maps/api/geocode/json?' + urllib.urlencode( urlParams )
response = urllib2.urlopen( url )
responseBody = response.read()
body = StringIO.StringIO( responseBody )
result = json.load( body )
if 'status' not in result or result['status'] != 'OK':
return None
else:
return {
'lat': result['results'][0]['geometry']['location']['lat'],
'lng': result['results'][0]['geometry']['location']['lng']
}
The library that Jason pointed me to is also interesting but since my code was intended to fix something (one time use) I have not tried his solution. I will definitely consider that if I get to write code again :-)

Although Google deprecated the V2 calls with googlemaps (which is why you're seeing the broken calls), they just recently announced that they are giving developers a six-month extension (until September 8, 2013) to move from the V2 to V3 API. See Update on Geocoding API V2 for details.
In the meantime, check out pygeocoder as a possible Python V3 solution.

Since September 2013, Google Maps API v2 no longer works. Here is the code working for API v3 (based on this answer):
import urllib
import simplejson
googleGeocodeUrl = 'http://maps.googleapis.com/maps/api/geocode/json?'
def get_coordinates(query, from_sensor=False):
query = query.encode('utf-8')
params = {
'address': query,
'sensor': "true" if from_sensor else "false"
}
url = googleGeocodeUrl + urllib.urlencode(params)
json_response = urllib.urlopen(url)
response = simplejson.loads(json_response.read())
if response['results']:
location = response['results'][0]['geometry']['location']
latitude, longitude = location['lat'], location['lng']
print query, latitude, longitude
else:
latitude, longitude = None, None
print query, "<no results>"
return latitude, longitude
See official documentation for the complete list of parameters and additional information.

Did some code golfing and ended up with this version. Depending on your need you might want to distinguish some more error conditions.
import urllib, urllib2, json
def decode_address_to_coordinates(address):
params = {
'address' : address,
'sensor' : 'false',
}
url = 'http://maps.google.com/maps/api/geocode/json?' + urllib.urlencode(params)
response = urllib2.urlopen(url)
result = json.load(response)
try:
return result['results'][0]['geometry']['location']
except:
return None