I am basing this task on the documentation provided here by Google:
https://cloud.google.com/appengine/docs/flexible/python/writing-and-responding-to-pub-sub-messages
I ended up coming with this code for my main that takes out the globally stored variable for messages, and instead writes each one out to a flat file.
# [START push]
#app.route('/pubsub/push', methods=['POST'])
def pubsub_push():
if (request.args.get('token', '') != current_app.config['PUBSUB_VERIFICATION_TOKEN']):
return 'Invalid request', 400
# Decode the data
envelope = json.loads(request.data.decode('utf-8'))
# Current time in UTC
current_date = datetime.utcnow()
payload = json.loads(base64.b64decode(envelope['message']['data']))
# Normalize and flatten the data
if "events" in payload and payload['events']:
payload['events'] = payload['events'][0]['data']
payload = parse_dict(init=payload, sep='_')
# Now jsonify all remaining lists and dicts
for key in payload.keys():
value = payload[key]
if isinstance(value, list) or isinstance(value, dict):
if value:
value = json.dumps(value)
else:
value = None
payload[key] = value
# Custom id with the message id and date string
id = "{}.{}".format(
payload['timestamp_unixtime_ms'],
payload['message_id']
)
filename_hourly = 'landing_path/{date}/{hour}/{id}.json'.format(
date=current_date.strftime("%Y%m%d"),
hour=current_date.strftime("%H"),
id=id
)
blob = bucket.get_blob(filename_hourly)
if blob: # We already have this file, skip this message
print('Already have {} stored.'.format(filename_hourly))
return 'OK', 200
blob_hourly = Blob(bucket=bucket, name=filename_hourly)
blob_hourly.upload_from_string(json.dumps(payload, indent=2, sort_keys=True))
# Returning any 2xx status indicates successful receipt of the message.
return 'OK', 200
# [END push]
This works perfectly, but I am getting a ton of 502 and 504 errors. It's displayed here in my stackdriver dashboard I created.
My guess is it's taking too long to upload the files, but I am unsure of what to do otherwise. Any suggestions? The resources on the appengine boxes are quite low, and my API limits are not even close.
Suggestions anyone?
This issue seemed to have been addressed on the Google Cloud Platform issue tracker link. As explained on that link, usually, 5xx responses are from nginx commonly caused when application is too busy to respond. The nginx webserver sits in front of App Engine application on Google App Engine Instance.
However, further review suggests to avoid the use gevent async workers with the Google Cloud Client Library as they cause requests to hang, but instead use more gunicorn workers.
Related
Apologies for any incorrect formatting, long time since I posted anything on stack overflow.
I'm looking to send a json payload of data to Azure IoT Hub which I am then going to process using an Azure Function App to display real-time telemetry data in Azure Digital Twin.
I'm able to post the payload to IoT Hub and view it using the explorer fine, however my function is unable to take this and display this telemetry data in Azure Digital Twin. From Googling I've found that the json file needs to be utf-8 encrypted and set to application/json, which I think might be the problem with my current attempt at fixing this.
I've included a snipped of the log stream from my azure function app below, as shown the "body" part of the message is scrambled which is why I think it may be an issue in how the payload is encoded:
"iothub-message-source":"Telemetry"},"body":"eyJwb3dlciI6ICIxLjciLCAid2luZF9zcGVlZCI6ICIxLjciLCAid2luZF9kaXJlY3Rpb24iOiAiMS43In0="}
2023-01-27T13:39:05Z [Error] Error in ingest function: Cannot access child value on Newtonsoft.Json.Linq.JValue.
My current test code is below for sending payloads to IoT Hub, with the potential issue being that I'm not encoding the payload properly.
import datetime, requests
import json
deviceID = "JanTestDT"
IoTHubName = "IoTJanTest"
iotHubAPIVer = "2018-04-01"
iotHubRestURI = "https://" + IoTHubName + ".azure-devices.net/devices/" + deviceID + "/messages/events?api-version=" + iotHubAPIVer
SASToken = 'SharedAccessSignature'
Headers = {}
Headers['Authorization'] = SASToken
Headers['Content-Type'] = "application/json"
Headers['charset'] = "utf-8"
datetime = datetime.datetime.now()
payload = {
'power': "1.7",
'wind_speed': "1.7",
'wind_direction': "1.7"
}
payload2 = json.dumps(payload, ensure_ascii = False).encode("utf8")
resp = requests.post(iotHubRestURI, data=payload2, headers=Headers)
I've attempted to encode the payload correctly in several different ways including utf-8 within request.post, however this produces an error that a dict cannot be encoded or still has the body encrypted within the Function App log stream unable to decipher it.
Thanks for any help and/or guidance that can be provided on this - happy to elaborate further on anything that is not clear.
is there any particular reason why you want to use Azure IoT Hub Rest API end point instead of using Python SDK? Also, even though you see the values in JSON format when viewed through Azure IoT Explorer, the message format when viewed through a storage end point such as blob reveals a different format as you pointed.
I haven't tested the Python code with REST API, but I have a Python SDK that worked for me. Please refer the code sample below
import os
import random
import time
from datetime import date, datetime
from json import dumps
from azure.iot.device import IoTHubDeviceClient, Message
def json_serial(obj):
"""JSON serializer for objects not serializable by default json code"""
if isinstance(obj, (datetime, date)):
return obj.isoformat()
raise TypeError("Type %s not serializable" % type(obj))
CONNECTION_STRING = "<AzureIoTHubDevicePrimaryConnectionString>"
TEMPERATURE = 45.0
HUMIDITY = 60
MSG_TXT = '{{"temperature": {temperature},"humidity": {humidity}, "timesent": {timesent}}}'
def run_telemetry_sample(client):
print("IoT Hub device sending periodic messages")
client.connect()
while True:
temperature = TEMPERATURE + (random.random() * 15)
humidity = HUMIDITY + (random.random() * 20)
x = datetime.now().isoformat()
timesent = dumps(datetime.now(), default=json_serial)
msg_txt_formatted = MSG_TXT.format(
temperature=temperature, humidity=humidity, timesent=timesent)
message = Message(msg_txt_formatted, content_encoding="utf-8", content_type="application/json")
print("Sending message: {}".format(message))
client.send_message(message)
print("Message successfully sent")
time.sleep(10)
def main():
print("IoT Hub Quickstart #1 - Simulated device")
print("Press Ctrl-C to exit")
client = IoTHubDeviceClient.create_from_connection_string(CONNECTION_STRING)
try:
run_telemetry_sample(client)
except KeyboardInterrupt:
print("IoTHubClient sample stopped by user")
finally:
print("Shutting down IoTHubClient")
client.shutdown()
if __name__ == '__main__':
main()
You can edit the MSG_TXT variable in the code to match the payload format and pass the values. Note that the SDK uses Message class from Azure IoT Device library which has an overload for content type and content encoding. Here is how I have passed the overloads in the code message = Message(msg_txt_formatted, content_encoding="utf-8", content_type="application/json")
I have validated the message by routing to a Blob Storage container and could see the telemetry data in the JSON format. Please refer below image screenshot referring the data captured at end point.
Hope this helps!
access_token = ''
import json
r = session.request('get', 'https://www.googleapis.com/drive/v3/files?access_token=%s' % access_token)
response_text = str(r.content, encoding='utf-8')
files_list = json.loads(response_text).get('files')
files_id_list = []
for item in files_list:
files_id_list.append(item.get('id'))
for item in files_id_list:
file_r = session.request('get', 'https://www.googleapis.com/drive/v3/files/%s?alt=media&access_token=%s' % (item, access_token))
print(file_r.content)
I use the above code and Google shows：
We're sorry ...
... but your computer or network may be sending automated queries. To protect our users, we can't process your request right now.
I do n’t know if this method ca n’t be downloaded originally, or where is the problem?
The reason you are getting this error is you are requesting the data in a Loop.
causes so many requests to Google's server.
And hence the error
We're sorry ... ... but your computer or network may be sending automated queries
access_token should not be placed in the request body,We should put access_token in the header.Can try on this site oauthplayground
I want to use Google Street View API to download some images in python.
Sometimes there is no image return in one area, but the API key can be used.
Other time API key is expired or invalid, and also cannot return image.
The Google Maps API server rejected your request. The provided API key is expired.
How to distinguish these two situations with code in Python?
Thank you very much.
One way to do this would be to make an api call with the requests library, and parse the JSON response:
import requests
url = 'https://maps.googleapis.com/maps/api/streetview?size=600x300&location=46.414382,10.013988&heading=151.78&pitch=-0.76&key=YOUR_API_KEY'
r = requests.get(url)
results = r.json()
error_message = results.get('error_message')
Now error_message will be the text of the error message (e.g., 'The provided API key is expired.'), or will be None if there is no error message. So later in your code you can check if there is an error message, and do things based on the content of the message:
if error_message and 'The provided API key is invalid.' in error_message:
do_something()
elif ...:
do_something_else()
You could also check the key 'status' if you just want to see if the request was successful or not:
status = results.get('status')
I'm using Google API v3 for Google Calendar. I'm able to add a recurrent event to Google Calendar but when trying to delete only the first occurrence of the series, the whole series gets deleted.
I'm using Python and authomatic_inst library and I'm passing the event_id as follows:
baseID_yyyymmddThhmmssZ
The delete request acts as if I only passed the baseID but I've checked the whole ID (ID with date and time) is passed.
I've used this reference (https://developers.google.com/google-apps/calendar/v3/reference/events) to retrieve the instances of the series and make sure the id of the first event is correct, but although deleting it from the try it! tool works, it does not work for me from Python.
def delete_gcal_event(google_event_id):
# google_event_id in the form baseID_yyyymmddThhmmssZ
# Deserialize the user's API credentials from the session storage
credentials = authomatic_inst.credentials(session['google_credentials'])
if not credentials.valid:
credentials.refresh()
session['google_credentials'] = credentials.serialize()
response = authomatic_inst.access(
credentials,
api_url('calendars/{}/events/{}', 'primary', google_event_id),
method='DELETE',
headers=JSON_HEADERS)
if (response.status == 204):
return dict(success = True,
message = "Event deleted successfully",
status = response.status)
else:
return dict(success = False,
message = "Error when deleting event from google",
status = response.status)
Did anyone came across this issue before?
Many thanks,
Javier
I have asked a few questions about this before, but still haven't solved my problem.
I am trying to allow Salesforce to remotely send commands to a Raspberry Pi via JSON (REST API). The Raspberry Pi controls the power of some RF Plugs via an RF Transmitter called a TellStick. This is all setup, and I can use Python to send these commands. All I need to do now is make the Pi accept JSON, then work out how to send the commands from Salesforce.
Someone kindly forked my repo on GitHub, and provided me with some code which should make it work. But unfortunately it still isn't working.
Here is the previous question: How to accept a JSON POST?
And here is the forked repo: https://github.com/bfagundez/RemotePiControl/blob/master/power.py
What do I need to do? I have sent test JSON messages n the Postman extension and in cURL but keep getting errors.
I just want to be able to send various variables, and let the script work the rest out.
I can currently post to a .py script I have with some URL variables, so /python.py?power=on&device=1&time=10&pass=whatever and it figures it out. Surely there's a simple way to send this in JSON?
Here is the power.py code:
# add flask here
from flask import Flask
app = Flask(__name__)
app.debug = True
# keep your code
import time
import cgi
from tellcore.telldus import TelldusCore
core = TelldusCore()
devices = core.devices()
# define a "power ON api endpoint"
#app.route("/API/v1.0/power-on/<deviceId>",methods=['POST'])
def powerOnDevice(deviceId):
payload = {}
#get the device by id somehow
device = devices[deviceId]
# get some extra parameters
# let's say how long to stay on
params = request.get_json()
try:
device.turn_on()
payload['success'] = True
return payload
except:
payload['success'] = False
# add an exception description here
return payload
# define a "power OFF api endpoint"
#app.route("/API/v1.0/power-off/<deviceId>",methods=['POST'])
def powerOffDevice(deviceId):
payload = {}
#get the device by id somehow
device = devices[deviceId]
try:
device.turn_off()
payload['success'] = True
return payload
except:
payload['success'] = False
# add an exception description here
return payload
app.run()
Your deviceID variable is a string, not an integer; it contains a '1' digit, but that's not yet an integer.
You can either convert it explicitly:
device = devices[int(deviceId)]
or tell Flask you wanted an integer parameter in the route:
#app.route("/API/v1.0/power-on/<int:deviceId>", methods=['POST'])
def powerOnDevice(deviceId):
where the int: part is a URL route converter.
Your views should return a response object, a string or a tuple instead of a dictionary (as you do now), see About Responses. If you wanted to return JSON, use the flask.json.jsonify() function:
# define a "power ON api endpoint"
#app.route("/API/v1.0/power-on/<int:deviceId>", methods=['POST'])
def powerOnDevice(deviceId):
device = devices[deviceId]
# get some extra parameters
# let's say how long to stay on
params = request.get_json()
try:
device.turn_on()
return jsonify(success=True)
except SomeSpecificException as exc:
return jsonify(success=False, exception=str(exc))
where I also altered the exception handler to handle a specific exception only; try to avoid Pokemon exception handling; do not try to catch them all!
To retrieve the Json Post values you must use request.json
if request.json and 'email' in request.json:
request.json['email']