Overall, I'm trying to invoke the MS Cognitive key phrases API from Python 3.5.1 :: Anaconda 4.0.0 (32-bit). I looked everywhere and tried to incorporate this stackoverflow response.
To call the API your account key below marked as ## needs to be added from here,
however to format the body correctly you probably don't need the account key. A good portion of the code below is from sample code.
Request body should look like
body = {
"documents": [
{
"language": "en",
"id": "1",
"text": "One line of text."
},
{
"language": "en",
"id": "2",
"text": "another line of text."
}
]
}
my code <it now works!!>
import sys
import os.path
import http.client
import urllib.request
import urllib.parse
import urllib.error
import base64
import json
subscription_key = '##'
headers = {
'Content-Type': 'application/json',
'Ocp-Apim-Subscription-Key': subscription_key
}
#input text is: ID | text to analyze. How my input file is formatted.
input_text = ["100|One line of text.", "101|another line of text."]
# Inputs holds the params to call the web service in bulk.
body = []
indx = 1
for line in input_text:
input_text = line.split("|")
print ('-----\n')
print ("Input text is:", input_text)
input_text_analyze = input_text[1]
print ('\nInput text to be analyzed:', input_text_analyze)
body.append({ "language" : "en", "id" : str(indx), "text" : input_text_analyze })
indx = indx + 1
print ('-----\n')
print ('\nBody has', body)
print ("Calling API to get keywords...")
body_documents = { 'documents': body }
print ("\nParams:", body_documents)
params = urllib.parse.urlencode({ })
try:
conn = http.client.HTTPSConnection('westus.api.cognitive.microsoft.com')
conn.request("POST", "/text/analytics/v2.0/keyPhrases?%s" % params, str(body_documents), headers)
response = conn.getresponse()
keyword_obj = response.read()
print("Returned keyword_obj is: ", keyword_obj)
conn.close()
except Exception as e:
print("[Errno {0}] {1}".format(e.errno, e.strerror))
I made 2 changed to the code above that allow it to work. 1) I was getting my params and body mixed up. 2) I needed to add str(body_documents) in my post. Both beginner mistakes.
Related
I'm trying to make decoder, data-->json. You paste raw data, and it decodes it into json. Here are 2 scenarios:
FIRST input is a correct data, it then correctly decodes it into json.
FIRST input is random string, like "abcd" - code then says that data is invalid.
After second scenario, even when you try to put VALID data, it will say "invalid data" forever. How can make it work, so when you first write "abcd", and then paste valid data, it will just decode it?! I tried loops, breaks, continues... Nothing works, and typing invalid input ruins whole code - you have to restart program and paste VALID data.
import requests
import json
while True:
try:
raw = input('Please paste raw data: '+"\n")
url = 'https://platform.lobaro.com/api/mbus?raw={}'.format(raw)
print("\n")
get = requests.get(url)
json_body = get.json()
parsed = json.dumps(json_body)
parsed2 = json.loads(parsed)
formatted = json.dumps(json.loads(parsed), indent=2)
print(formatted+"\n")
print("INFORMATIONS:")
name = parsed2["MFieldLongString"]
print(f'Producer: {name}')
l1 = parsed2["Length"]
print(f'Lenght: {l1}'+'\n')
print('============================================================================')
except requests.exceptions.JSONDecodeError:
print('Invalid data! Could not parse to JSON.'+"\n")
continue
The site uses other API URL to decode wmbus messages:
import json
import requests
api_url = "https://platform.lobaro.com/gqlgen/query"
payload = {
"operationName": None,
"query": "query ($raw: String!, $key: String) {\n wmbus: parseWmbus(raw: $raw, key: $key) {\n data\n parseError\n __typename\n }\n}\n",
"variables": {
"key": "0102030405060708090A0B0C0D0E0F11", # <-- change this field
"raw": "2e44931578563412330333637a2a0020055923c95aaa26d1b2e7493b2a8b013ec4a6f6d3529b520edff0ea6defc955b29d6d69ebf3ec8a", # <-- change this field
},
}
data = requests.post(api_url, json=payload).json()
if data['data']['wmbus']['parseError'] == "":
data = json.loads(data["data"]["wmbus"]["data"])
print(data)
Prints:
{
"Raw": "0x2e4493157856341233037a2a0020055923c95aaa26d1b2e7493b013ec4a6f6d3529b520edff0ea6defc99d6d69ebf3",
"RawWithCrc": "0x2e44931578563412330333637a2a0020055923c95aaa26d1b2e7493b2a8b013ec4a6f6d3529b520edff0ea6defc955b29d6d69ebf3ec8a",
"FrameFormat": "A",
"Length": 46,
"CField": "0x44",
"CFieldString": "0x44 (SND_NR)",
"MField": "0x9315",
"MFieldCodeString": "ELS",
"MFieldLongString": "Elster GmbH, Germany, Europe",
"Id": 305419896,
"IdString": "12345678",
"Version": 51,
"Device": "0x03",
"DeviceString": "Gas",
"CiField": "0x7a",
"HeaderKnown": True,
"PayloadKnown": True,
"CrcValid": True,
"HasCrc": True,
"SourceType": "WMBUS",
"IsCompactFrame": False,
"FormatSignature": 61330,
"FormatFrame": "0x0c14046d02fd17",
"Header": {
"Serial": 0,
"IdString": "",
"ManufacturerCode": 0,
"MFieldCodeString": "",
"MFieldLongString": "",
"Version": 0,
"DeviceType": "0x00",
"DeviceString": "",
"EncryptionMode": 5,
"EncryptionModeString": "AES with CBC",
"EncryptedBlocks": 2,
"HopCount": 0,
"IsAccessible": True,
"IsBidirectionalMode": False,
"IsSynchronous": False,
"ReservedBit": False,
"TelegramType": 0,
"AccessNumber": 42,
"StatusByte": 0,
"ConfigField": [32, 5],
...and so on.
This is because the response could have been cached. One way to disable the caching is to pass the appropriate headers in the request. Something like
get = requests.get(url, headers={
'Cache-Control': 'no-cache',
'Pragma': 'no-cache',
'Expires': '0' })
See 'Pedro Lobitos' answer to this question here for a good explanation here How do I clear cache with Python Requests?. (Good explanation and very good discussion/comments too)
You could try to restart a method after the Exception, so you delete all local variables temporarily and restart from scratch:
import gc
def myMethod():
gc.collect(generation=2)
while True:
try:
raw = input('Please paste raw data: '+"\n")
url = 'https://platform.lobaro.com/api/mbus?raw={}'.format(raw)
print("\n")
get = requests.get(url)
json_body = get.json()
parsed = json.dumps(json_body)
parsed2 = json.loads(parsed)
formatted = json.dumps(json.loads(parsed), indent=2)
print(formatted+"\n")
print("INFORMATIONS:")
name = parsed2["MFieldLongString"]
print(f'Producer: {name}')
l1 = parsed2["Length"]
print(f'Lenght: {l1}'+'\n')
print('=====')
except requests.exceptions.JSONDecodeError:
print('Invalid data! Could not parse to JSON.'+"\n")
break
myMethod()
So everytime it raises an Exception, the "old execution" is killed and a new one starts. The gc is there for automatically garbage-collect the leftovers from the old execution.
I understand how it is possible to use speech to record data into a backend database. However, I am unsure about how I may allow this to be done from a web application. As I am not dealing with text, how do I offload the speech activation options and consequently allow text to be pushed from the web application?
main.py
import speech_recognition as sr
import gtts
from playsound import playsound
import os
from datetime import datetime
from notion import NotionClient
r = sr.Recognizer()
token = "YOUR NOTION TOKEN HERE"
database_id = "YOUR NOTION DATABASE_ID HERE"
client = NotionClient(token, database_id)
ACTIVATION_COMMAND = "hey sam"
def get_audio():
with sr.Microphone() as source:
print("Say something")
audio = r.listen(source)
return audio
def audio_to_text(audio):
text = ""
try:
text = r.recognize_google(audio)
except sr.UnknownValueError:
print("Speech recognition could not understand audio")
except sr.RequestError:
print("could not request results from API")
return text
def play_sound(text):
try:
tts = gtts.gTTS(text)
tempfile = "./temp.mp3"
tts.save(tempfile)
playsound(tempfile)
os.remove(tempfile)
except AssertionError:
print("could not play sound")
if __name__ == "__main__":
while True:
a = get_audio()
command = audio_to_text(a)
if ACTIVATION_COMMAND in command.lower():
print("activate")
play_sound("What can I do for you?")
note = get_audio()
note = audio_to_text(note)
if note:
play_sound(note)
now = datetime.now().astimezone().isoformat()
res = client.create_page(note, now, status="Active")
if res.status_code == 200:
play_sound("Stored new item")
backend.py
import json
import requests
class NotionClient:
def __init__(self, token, database_id) -> None:
self.database_id = database_id
self.headers = {
"Authorization": "Bearer " + token,
"Content-Type": "application/json",
"Notion-Version": "2021-08-16"
}
# read, update
def create_page(self, description, date, status):
create_url = 'https://api.notion.com/v1/pages'
data = {
"parent": { "database_id": self.database_id },
"properties": {
"Description": {
"title": [
{
"text": {
"content": description
}
}
]
},
"Date": {
"date": {
"start": date,
"end": None
}
},
"Status": {
"rich_text": [
{
"text": {
"content": status
}
}
]
}
}}
data = json.dumps(data)
res = requests.post(create_url, headers=self.headers, data=data)
print(res.status_code)
return res
I have a json (url = http://open.data.amsterdam.nl/ivv/parkeren/locaties.json) and I want to print all 'title', 'adres', 'postcode'. How can I do that?
I want to print it like this:
title.
adres.
postcode.
title.
adres.
postcode.
so among themselves
I hope you can help me with this
import urllib, json
url = "http://open.data.amsterdam.nl/ivv/parkeren/locaties.json"
import requests
search = requests.get(url).json()
print(search['title'])
print(search['adres'])
print(search['postcode'])
Using print(json.dumps(r, indent=4)) you can see that the structure is
{
"parkeerlocaties": [
{
"parkeerlocatie": {
"title": "Fietsenstalling Tolhuisplein",
"Locatie": "{\"type\":\"Point\",\"coordinates\":[4.9032801,52.3824545]}",
...
}
},
{
"parkeerlocatie": {
"title": "Fietsenstalling Paradiso",
"Locatie": "{\"type\":\"Point\",\"coordinates\":[4.8833735,52.3621851]}",
...
}
},
So to access the inner properties, you need to follow the JSON path
import requests
url = ' http://open.data.amsterdam.nl/ivv/parkeren/locaties.json'
search = requests.get(url).json()
for parkeerlocatie in search["parkeerlocaties"]:
content = parkeerlocatie['parkeerlocatie']
print(content['title'])
print(content['adres'])
print(content['postcode'])
print()
I am trying to build a python script that access the email and last logged in time data of all my users on my auth0 portal. How should i approach this?
this is the code i have so far
import sys
sys.path.append("D:\home\site\wwwroot\env\Lib\site-packages")
import quickbase
import urllib2
import pypyodbc
import timestring
import datetime
import time
import math
import sys
import locale
import smtplib
import pypyodbc
import httplib
import json
import urllib
import urllib2
from dateutil.relativedelta import relativedelta
#Activate Webhook with Campaign is added or modified.
#0.1 establish connection to SQL server
connection = pypyodbc.connect('Driver={SQL Server Native Client 11.0};'
'Server=tcp:xxxxe.database.windows.net,1433;'
'Database=LQDC;'
'uid=Admin#lxx;pwd=xxxx')
cursor = connection.cursor()
#0.2 establish connection to Auth0
def main():
#Configuration Values
AUDIENCE = "https://xxe.auth0.com/api/v2/"
DOMAIN = "lxxx.auth0.com"
CLIENT_ID = "xxxxxL"
CLIENT_SECRET = "xxxxxxr"
GRANT_TYPE = "client_credentials" # OAuth 2.0 flow to use
#Get an Access Token from Auth0
base_url = "https://{domain}".format(domain=DOMAIN)
data = urllib.urlencode([('client_id', CLIENT_ID),
('client_secret', CLIENT_SECRET),
('audience', AUDIENCE),
('grant_type', GRANT_TYPE)])
req = urllib2.Request(base_url + "/oauth/token", data)
response = urllib2.urlopen(req)
oauth = json.loads(response.read())
access_token = oauth['access_token']
#Get all Applications using the token
req = urllib2.Request(base_url + "/api/v2/clients")
req.add_header('Authorization', 'Bearer ' + access_token)
req.add_header('Content-Type', 'application/json')
try:
response = urllib2.urlopen(req)
res = json.loads(response.read())
print res
conn = httplib.HTTPSConnection(DOMAIN)
payload = "{\"connection_id\":\"not sure which API/token/code to put here\",\"format\": \"csv\", \"limit\": 4, \"fields\": [{ \"name\": \"email\"},{\"name\": \"last_ip\"}, { \"name\": \"created_at\"}, { \"name\": \"last_login\"}]}"
headers = {
'authorization': "Bearer "+access_token,
'content-type': "application/json"
}
conn.request("POST", "/xxxx.auth0.com/api/v2/jobs/users-exports", payload, headers)
res = conn.getresponse()
data = res.read()
print(data.decode("utf-8"))
except urllib2.HTTPError, e:
print 'HTTPError = ' + str(e.code) + ' ' + str(e.reason)
except urllib2.URLError, e:
print 'URLError = ' + str(e.reason)
except urllib2.HTTPException, e:
print 'HTTPException'
except Exception:
print 'Generic Exception'
#Standard boilerplate to call the main() function.
if name == 'main':
main()
the first part works fine, second part returns the error below
im trying to have this return a csv file with all the information of users listed, and populate the database automatically with that.
also found some useful links here which i tried incorporating into my code:
https://auth0.com/docs/users/guides/bulk-user-exports
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Error</title>
</head>
<body>
<pre>Cannot POST /lqdfinance.auth0.com/api/v2/jobs/users-exports</pre>
</body>
</html>
As i can see i your code, you need a connection id so you can use
base_url + '/api/v2/connections'
to get your connection id
Next you need to modify your payload body as below
{
"connection_id": "your_id_as_above",
"format": "csv",
"limit": 5,
"fields": [
{
"name": "user_id"
},
{
"name": "name"
},
{
"name": "email"
},
{
"name": "identities[0].connection",
"export_as": "provider"
},
{
"name": "user_metadata.some_field"
}
]
}
In order to test your response data i suggest you test your api payload and response using Postman, as it gives you more clarity on how the request is working.
The example code for Google's YouTube Data API is a piece of junk. It's so complicated and tied to the oauth redirect flow that I can't use it. Trying to go raw with requests pip and not getting too far.
I've followed the instructions exactly (as far as I can tell), with the following code:
import json
import os
import sys
import urllib
import requests
payload_file = None
payload = None
print 'Loading Config'
# Get the directory path of this file. When using any relative file paths make
# sure they are relative to current_dir so that the script can be run from any CWD.
current_dir = os.path.dirname(os.path.abspath(__file__))
# Reads in the config.json file then parses it
config = json.loads(open(os.path.join(current_dir, '..', 'config.json')).read())
print 'Parsing Payload'
for i in range(len(sys.argv)):
if sys.argv[i] == "--json" and (i + 1) < len(sys.argv):
payload = json.loads(sys.argv[i + 1])
elif sys.argv[i] == "-payload" and (i + 1) < len(sys.argv):
payload_file = sys.argv[i + 1]
with open(payload_file,'r') as f:
payload = json.loads(f.read())
break
print 'Configuring youtube with token {0}'.format(payload['token'])
print 'Downloading video...'
# See how big it is
f = urllib.urlopen(payload['url'])
content_length = int(f.headers["Content-Length"])
# Download it
# urllib.urlretrieve(payload['url'], "video.mp4")
metadata = {
'snippet' : {
'title': payload['title'],
"categoryId": 22
},
'status' : {
"privacyStatus": "public",
"embeddable": True,
"license": "youtube"
}
}
if 'tags' in payload:
metadata['snippet']['tags'] = payload['tags']
if 'description' in payload:
metadata['snippet']['description'] = payload['description']
headers = {
'Authorization' : 'Bearer {0}'.format(payload['token']),
'Content-Type' : 'application/json; charset=UTF-8',
'Content-Length' : json.dumps(metadata).__len__(),
'X-Upload-Content-Length' : content_length,
'X-Upload-Content-Type' : 'video/*',
}
print 'Attempting to upload video'
print headers
# upload video file
r = requests.post('https://www.googleapis.com/upload/youtube/v3/videos?uploadType=resumable&part=snippet,status', data=metadata, headers=headers);
print "RESPONSE!"
print r.text
# files = {
# 'file': video_file,
# }
# r = requests.post('https://www.googleapis.com/upload/youtube/v3/videos', data={ "video" : video }, headers=headers);
Obviously its not finished, but its dying on the metadata upload request with the following output:
Loading Config
Parsing Payload
Configuring youtube with token <access-token>
Downloading video...
Attempting to upload video
{'X-Upload-Content-Length': 51998563, 'Content-Length': 578, 'Content-Type': 'application/json; charset=UTF-8', 'X-Upload-Content-Type': 'video/*', 'Authorization': 'Bearer <access-token>'}
RESPONSE!
{
"error": {
"errors": [
{
"domain": "global",
"reason": "parseError",
"message": "Parse Error"
}
],
"code": 400,
"message": "Parse Error"
}
}
This error is not even listed in their "Errors" docs.
What is wrong with my code?
Here is an example in python that works. It assumes you've already done the oauth part though.
import requests
from os import fstat
import json
fi = open('myvideo.mp4')
base_headers = {
'Authorization': '%s %s' % (auth_data['token_type'],
auth_data['access_token']),
'content-type': 'application/json'
}
initial_headers = base_headers.copy()
initial_headers.update({
'x-upload-content-length': fstat(fi.fileno()).st_size,
'x-upload-content-type': 'video/mp4'
})
initial_resp = requests.post(
'https://www.googleapis.com/upload/youtube/v3/videos?uploadType=resumable&part=snippet,status,contentDetails',
headers=initial_headers,
data=json.dumps({
'snippet': {
'title': 'my title',
},
'status': {
'privacyStatus': 'unlisted',
'embeddable': True
}
})
)
upload_url = initial_resp.headers['location']
resp = requests.put(
upload_url,
headers=base_headers,
data=fi
)
fi.close()
the above is graet, just adding: you can also get the youtube id from the response (for future use):
cont = json.loads(resp.content)
youtube_id = cont['id']