We Keep Coding

Sending payload to IoT Hub for using in Azure Digital Twin using an Azure Function

Apologies for any incorrect formatting, long time since I posted anything on stack overflow.
I'm looking to send a json payload of data to Azure IoT Hub which I am then going to process using an Azure Function App to display real-time telemetry data in Azure Digital Twin.
I'm able to post the payload to IoT Hub and view it using the explorer fine, however my function is unable to take this and display this telemetry data in Azure Digital Twin. From Googling I've found that the json file needs to be utf-8 encrypted and set to application/json, which I think might be the problem with my current attempt at fixing this.
I've included a snipped of the log stream from my azure function app below, as shown the "body" part of the message is scrambled which is why I think it may be an issue in how the payload is encoded:
"iothub-message-source":"Telemetry"},"body":"eyJwb3dlciI6ICIxLjciLCAid2luZF9zcGVlZCI6ICIxLjciLCAid2luZF9kaXJlY3Rpb24iOiAiMS43In0="}
2023-01-27T13:39:05Z [Error] Error in ingest function: Cannot access child value on Newtonsoft.Json.Linq.JValue.
My current test code is below for sending payloads to IoT Hub, with the potential issue being that I'm not encoding the payload properly.
import datetime, requests
import json
deviceID = "JanTestDT"
IoTHubName = "IoTJanTest"
iotHubAPIVer = "2018-04-01"
iotHubRestURI = "https://" + IoTHubName + ".azure-devices.net/devices/" + deviceID + "/messages/events?api-version=" + iotHubAPIVer
SASToken = 'SharedAccessSignature'
Headers = {}
Headers['Authorization'] = SASToken
Headers['Content-Type'] = "application/json"
Headers['charset'] = "utf-8"
datetime = datetime.datetime.now()
payload = {
'power': "1.7",
'wind_speed': "1.7",
'wind_direction': "1.7"
}
payload2 = json.dumps(payload, ensure_ascii = False).encode("utf8")
resp = requests.post(iotHubRestURI, data=payload2, headers=Headers)
I've attempted to encode the payload correctly in several different ways including utf-8 within request.post, however this produces an error that a dict cannot be encoded or still has the body encrypted within the Function App log stream unable to decipher it.
Thanks for any help and/or guidance that can be provided on this - happy to elaborate further on anything that is not clear.

is there any particular reason why you want to use Azure IoT Hub Rest API end point instead of using Python SDK? Also, even though you see the values in JSON format when viewed through Azure IoT Explorer, the message format when viewed through a storage end point such as blob reveals a different format as you pointed.
I haven't tested the Python code with REST API, but I have a Python SDK that worked for me. Please refer the code sample below
import os
import random
import time
from datetime import date, datetime
from json import dumps
from azure.iot.device import IoTHubDeviceClient, Message
def json_serial(obj):
"""JSON serializer for objects not serializable by default json code"""
if isinstance(obj, (datetime, date)):
return obj.isoformat()
raise TypeError("Type %s not serializable" % type(obj))
CONNECTION_STRING = "<AzureIoTHubDevicePrimaryConnectionString>"
TEMPERATURE = 45.0
HUMIDITY = 60
MSG_TXT = '{{"temperature": {temperature},"humidity": {humidity}, "timesent": {timesent}}}'
def run_telemetry_sample(client):
print("IoT Hub device sending periodic messages")
client.connect()
while True:
temperature = TEMPERATURE + (random.random() * 15)
humidity = HUMIDITY + (random.random() * 20)
x = datetime.now().isoformat()
timesent = dumps(datetime.now(), default=json_serial)
msg_txt_formatted = MSG_TXT.format(
temperature=temperature, humidity=humidity, timesent=timesent)
message = Message(msg_txt_formatted, content_encoding="utf-8", content_type="application/json")
print("Sending message: {}".format(message))
client.send_message(message)
print("Message successfully sent")
time.sleep(10)
def main():
print("IoT Hub Quickstart #1 - Simulated device")
print("Press Ctrl-C to exit")
client = IoTHubDeviceClient.create_from_connection_string(CONNECTION_STRING)
try:
run_telemetry_sample(client)
except KeyboardInterrupt:
print("IoTHubClient sample stopped by user")
finally:
print("Shutting down IoTHubClient")
client.shutdown()
if __name__ == '__main__':
main()
You can edit the MSG_TXT variable in the code to match the payload format and pass the values. Note that the SDK uses Message class from Azure IoT Device library which has an overload for content type and content encoding. Here is how I have passed the overloads in the code message = Message(msg_txt_formatted, content_encoding="utf-8", content_type="application/json")
I have validated the message by routing to a Blob Storage container and could see the telemetry data in the JSON format. Please refer below image screenshot referring the data captured at end point.
Hope this helps!

I can't get binance Futures order book historical data

I'm trying to get binance Futures order history data using API. So I asked for data from binance, got the answer "Your application for historical futures order book data has been approved, please follow our Github guidance to access with your whitelisted account API key" and I have set up the API as follows.
And I have modified the Enable Symbol Whitelist like this:
The next step, I followed Github guidance: https://github.com/binance/binance-public-data/tree/master/Futures_Order_Book_Download
which has the following sample code:
"""
This example python script shows how to download the Historical Future Order Book level 2 Data via API.
The data download API is part of the Binance API (https://binance-docs.github.io/apidocs/spot/en/#general-api-information).
For how to use it, you may find info there with more examples, especially SIGNED Endpoint security as in https://binance-docs.github.io/apidocs/spot/en/#signed-trade-user_data-and-margin-endpoint-security
Before executing this file, please note:
- The API account needs to have a Futures account to access Futures data.
- The API key has been whitelisted to access the data.
- Read the comments section in this file to know where you should specify your request values.
"""
# Install the following required packages
import requests
import time
import hashlib
import hmac
from urllib.parse import urlencode
S_URL_V1 = "https://api.binance.com/sapi/v1"
# Specify the api_key and secret_key with your API Key and secret_key
api_key = "your_api_key"
secret_key = "your_secret_key "
# Specify the four input parameters below:
symbol = "ADAUSDT" # specify the symbol name
startTime = 1635561504914 # specify the starttime
endTime = 1635561604914 # specify the endtime
dataType = "T_DEPTH" # specify the dataType to be downloaded
# Function to generate the signature
def _sign(params={}):
data = params.copy()
ts = str(int(1000 * time.time()))
data.update({"timestamp": ts})
h = urlencode(data)
h = h.replace("%40", "#")
b = bytearray()
b.extend(secret_key.encode())
signature = hmac.new(b, msg=h.encode("utf-8"), digestmod=hashlib.sha256).hexdigest()
sig = {"signature": signature}
return data, sig
# Function to generate the download ID
def post(path, params={}):
sign = _sign(params)
query = urlencode(sign[0]) + "&" + urlencode(sign[1])
url = "%s?%s" % (path, query)
header = {"X-MBX-APIKEY": api_key}
resultPostFunction = requests.post(url, headers=header, timeout=30, verify=True)
return resultPostFunction
# Function to generate the download link
def get(path, params):
sign = _sign(params)
query = urlencode(sign[0]) + "&" + urlencode(sign[1])
url = "%s?%s" % (path, query)
header = {"X-MBX-APIKEY": api_key}
resultGetFunction = requests.get(url, headers=header, timeout=30, verify=True)
return resultGetFunction
"""
Beginning of the execution.
The final output will be:
- A link to download the specific data you requested with the specific parameters.
Sample output will be like the following: {'expirationTime': 1635825806, 'link': 'https://bin-prod-user-rebate-bucket.s3.amazonaws.com/future-data-download/XXX'
Copy the link to the browser and download the data. The link would expire after the expirationTime (usually 24 hours).
- A message reminding you to re-run the code and download the data hours later.
Sample output will be like the following: {'link': 'Link is preparing; please request later. Notice: when date range is very large (across months), we may need hours to generate.'}
"""
timestamp = str(
int(1000 * time.time())
) # current timestamp which serves as an input for the params variable
paramsToObtainDownloadID = {
"symbol": symbol,
"startTime": startTime,
"endTime": endTime,
"dataType": dataType,
"timestamp": timestamp,
}
# Calls the "post" function to obtain the download ID for the specified symbol, dataType and time range combination
path = "%s/futuresHistDataId" % S_URL_V1
resultDownloadID = post(path, paramsToObtainDownloadID)
print(resultDownloadID)
downloadID = resultDownloadID.json()["id"]
print(downloadID) # prints the download ID, example: {'id': 324225}
# Calls the "get" function to obtain the download link for the specified symbol, dataType and time range combination
paramsToObtainDownloadLink = {"downloadId": downloadID, "timestamp": timestamp}
pathToObtainDownloadLink = "%s/downloadLink" % S_URL_V1
resultToBeDownloaded = get(pathToObtainDownloadLink, paramsToObtainDownloadLink)
print(resultToBeDownloaded)
print(resultToBeDownloaded.json())
I have modified api_key and secret_key to my own keys and this is the result I got.
Can you tell me where I made a mistake? Thanks in advance for the answer.

Look at https://www.binance.com/en-NG/landing/data.
Futures Order Book Data Available only on Binance Futures. It requires
futures account be whitelisted first and can only be download via API.
Orderbook snapshot (S_Depth): Since January 2020, only on BTC/USDT
symbol. Tick-level orderbook (T_Depth): Since January 2020, on all
symbols
The page says you should to apply the Binance form to be whitelisted in futures section here:
https://docs.google.com/forms/d/e/1FAIpQLSexCgyvZEMI1pw1Xj6gwKtfQTYUbH5HrUQ0gwgPZtM9FaM2Hw/viewform
Second thing - you are interested in futures, not spots, so the url should be api.binance.com/fapi instead of api.binance.com/sapi
Third thing - API endpoint for order book is
GET /fapi/v1/depth

Writing code using graph APIs

I am extremely new to python , scripting and APIs, well I am just learning. I came across a very cool code which uses facebook api to reply for birthday wishes.
I will add my questions, I will number it so that it will be easier for someone else later too. I hope this question will clear lots of newbies doubts.
1) Talking about APIs, in what format are the usually in? is it a library file which we need to dowload and later import? for instance, twitter API, we need to import twitter ?
Here is the code :
import requests
import json
AFTER = 1353233754
TOKEN = ' <insert token here> '
def get_posts():
"""Returns dictionary of id, first names of people who posted on my wall
between start and end time"""
query = ("SELECT post_id, actor_id, message FROM stream WHERE "
"filter_key = 'others' AND source_id = me() AND "
"created_time > 1353233754 LIMIT 200")
payload = {'q': query, 'access_token': TOKEN}
r = requests.get('https://graph.facebook.com/fql', params=payload)
result = json.loads(r.text)
return result['data']
def commentall(wallposts):
"""Comments thank you on all posts"""
#TODO convert to batch request later
for wallpost in wallposts:
r = requests.get('https://graph.facebook.com/%s' %
wallpost['actor_id'])
url = 'https://graph.facebook.com/%s/comments' % wallpost['post_id']
user = json.loads(r.text)
message = 'Thanks %s :)' % user['first_name']
payload = {'access_token': TOKEN, 'message': message}
s = requests.post(url, data=payload)
print "Wall post %s done" % wallpost['post_id']
if __name__ == '__main__':
commentall(get_posts())`
Questions:
importing json--> why is json imported here? to give a structured reply?
What is the 'AFTER' and the empty variable 'TOKEN' here?
what is the variable 'query' and 'payload' inside get_post() function?
Precisely explain almost what each methods and functions do.
I know I am extremely naive, but this could be a good start. A little hint, I can carry on.
If not going to explain the code, which is pretty boring, I understand, please tell me how to link to APIs after a code is written, meaning how does a script written communicate with the desired API.
This is not my code, I copied it from a source.

json is needed to access the web service and interpret the data that is sent via HTTP.
The 'AFTER' variable is supposed to get used to assume all posts after this certain timestamp are birthday wishes.
To make the program work, you need a token which you can obtain from Graph API Explorer with the appropriate permissions.

Facebook Posting to a Page

I am trying to post to the wall of a facebook page that I am administrator (not profile), however no luck. How do I achieve this ? I'm stucked at the page access token retrieval part.
#!/usr/bin/python
# coding: utf-8
import facebook
import urllib
import urlparse
import subprocess
import warnings
# Hide deprecation warnings. The facebook module isn't that up-to-date (facebook.GraphAPIError).
warnings.filterwarnings('ignore', category=DeprecationWarning)
# Parameters of your app and the id of the profile you want to mess with.
FACEBOOK_APP_ID = 'XXXXXXXXXXXXXX'
FACEBOOK_APP_SECRET = 'XXXXXXXXXXXXXXXXXXXXX'
FACEBOOK_PROFILE_ID = 'XXXXXXXXXXX'
# Trying to get an access token. Very awkward.
oauth_args = dict(client_id = FACEBOOK_APP_ID,
client_secret = FACEBOOK_APP_SECRET,
scope = 'manage_pages',
response_type = 'token'
)
oauth_curl_cmd = ['curl',
'https://graph.facebook.com/oauth/access_token?' + urllib.urlencode(oauth_args)]
oauth_response = subprocess.Popen(oauth_curl_cmd,
stdout = subprocess.PIPE,
stderr = subprocess.PIPE).communicate()[0]
print urllib.urlencode(oauth_args)
try:
oauth_access_token = urlparse.parse_qs(str(oauth_response))['access_token'][0]
except KeyError:
print('Unable to grab an access token!')
exit()
print oauth_access_token
facebook_graph = facebook.GraphAPI(oauth_access_token)
# Try to post something on the wall.
try:
fb_response = facebook_graph.put_wall_post('Hello from Python', \
profile_id = FACEBOOK_PROFILE_ID)
print fb_response
except facebook.GraphAPIError as e:
print 'Something went wrong:', e.type, e.message

I would not recommend doing this through the command line with curl as it is less secure and less reliable. You can do all of this with the urllib2 and json modules
to get the access token you just want to make a call to https://graph.facebook.com/oauth/access_token?client_id=YOUR_APP_ID&client_secret=YOUR_APP_SECRET&grant_type=client_credentials
so you would do:
url='https://graph.facebook.com/oauth/access_token?client_id=YOUR_APP_ID&client_secret=YOUR_APP_SECRET&grant_type=client_credentials'
target=urllib2.urlopen(url)
token = target.read()[13:]
EDIT:
My bad, I forgot that facebook/oauth gives you the access token in plain text so you don't need the json module. I've updated the example to show what you should be doing. Note target.read() will give you the string 'access_token=ACCESS_TOKEN' and then you are just parsing it to remove the identifier.
to see what response is go to the url and put in your information you will a json dict with acess_token.
the second half of this page should have all the information you need.

OAuth with Twitter script in Python is not working

I'm writing a script of OAuth in Python.
For testing this, I use Twitter API. But it is not working well.
def test():
params = {
"oauth_consumer_key": TWITTER_OAUTH_CONSUMER_KEY,
"oauth_nonce": "".join(random.choice(string.digits + string.letters) for i in xrange(7)),
"oauth_signature_method": "HMAC-SHA1",
"oauth_timestamp": str(int(time.time())),
"oauth_token": res_dict["oauth_token"],
"oauth_version": "1.0",
}
status = {"status": u"Always_look_on_the_bright_side_of_life".encode("UTF-8")}
print status
params.update(status)
url = "http://twitter.com/statuses/update.xml"
key = "&".join([TWITTER_OAUTH_CONSUMER_SECRET, res_dict["oauth_token_secret"]])
msg = "&".join(["POST", urllib.quote(url,""),
urllib.quote("&".join([k+"="+params[k] for k in sorted(params)]), "-._~")])
print msg
signature = hmac.new(key, msg, hashlib.sha1).digest().encode("base64").strip()
params["oauth_signature"] = signature
req = urllib2.Request(url,
headers={"Authorization":"OAuth", "Content-type":"application/x-www-form-urlencoded"})
req.add_data("&".join([k+"="+urllib.quote(params[k], "-._~") for k in params]))
print req.get_data()
res = urllib2.urlopen(req).read()
print res
This script (status="Always_look_on_the_bright_side_of_life") is working.
But, in case status is "Always look on the bright side of life"(replaced underscore with space), it isn't working(is returning HTTP Error 401: Unauthorized).
I referenced this question, but failed.
Please give me some advice. Thank you.

I got the same problem in OAuth with FaceBook a while ago. The problem is that the signature validation on server side fails. See your signature generation code here:
msg = "&".join(["POST", urllib.quote(url,""),
urllib.quote("&".join([k+"="+params[k] for k in sorted(params)]), "-._~")])
print msg
signature = hmac.new(key, msg, hashlib.sha1).digest().encode("base64").strip()
It uses the raw (non-encoded) form of the string to generate the signature. However, the server side generates validates the signature against the URL quoted string:
req.add_data("&".join([k+"="+urllib.quote(params[k], "-._~") for k in params]))
To fix the code, you need to do fix this line by creating the signature from the url encoded parameter:
msg = "&".join(["POST", urllib.quote(url,""),
urllib.quote("&".join([k+"="+urllib.quote(params[k], "-._~") for k in sorted(params)]), "-._~")])

The easiest way to fix this is to add status = urllib.quote(status) after status = {"status": u"Always_look_on_the_bright_side_of_life".encode("UTF-8")}. This will escape the spaces and other special characters as required.