We Keep Coding

Returning a big object from activity function throws an error

I have a Durable Functions instance with the following functions:
Starter/Trigger function. It uses a Storage Queue trigger. It also implements Singleton Orchestrator, because I want only one instance of the Orchestrator to run at a time:
async def main(msg: func.QueueMessage, starter: str) -> None:
client = df.DurableOrchestrationClient(starter)
payload = {"day": msg.get_body().decode('utf-8')}
instance_id = msg.get_body().decode('utf-8')
await client.terminate(instance_id, "New instance to be called")
existing_instance = await client.get_status(instance_id)
status = existing_instance.runtime_status
if existing_instance.runtime_status in [df.OrchestrationRuntimeStatus.Completed, df.OrchestrationRuntimeStatus.Failed, df.OrchestrationRuntimeStatus.Terminated, None]:
instance_id = await client.start_new("Orchestrator", instance_id, payload)
logging.info(f"Started orchestration with ID = '{instance_id}'.")
return
else:
return {
'status': 409,
'body': f"An instance with ID '${existing_instance.instance_id}' already exists"
}
Orchestrator function:
def orchestrator_function(context: df.DurableOrchestrationContext):
input_context = context.get_input()
day = input_context.get('day')
day_date = datetime.datetime.strptime(day, '%y%m%d')
day_raw = day_date.strftime('%Y%m%d')
gps_data = yield context.call_activity('GetGPSData', day)
logging.info("Downloaded blobs")
Activity function:
def main(day: str) -> str:
blob_data = download_blob(day)
df_gps= pd.from_csv(blob_data)
return df_gps.to_json()
As you can see, the activity function simply downloads a blob (around 400MiB) from Azure Blob Storage, creates a dataframe out of it and returns the dataframe in JSON.
However, on execution, I get the following error:
[2022-12-12T10:53:03.677Z] 221101: Function 'Orchestrator (Orchestrator)' failed with an error. Reason: Message: Too many characters. The resulting number of bytes is larger than what can be returned as an int. (Parameter 'count')
Unhandled exception while executing task: System.ArgumentOutOfRangeException: Too many characters. The resulting number of bytes is larger than what can be returned as an int. (Parameter 'count')
[2022-12-12T10:53:03.678Z] at System.Text.UTF32Encoding.GetByteCount(Char* chars, Int32 count, EncoderNLS encoder)
[2022-12-12T10:53:03.678Z] at System.Text.UTF32Encoding.GetByteCount(String s)
[2022-12-12T10:53:03.679Z] at Microsoft.Azure.WebJobs.Extensions.DurableTask.DurableTaskExtension.GetIntputOutputTrace(String rawInputOutputData) in D:\a\_work\1\s\src\WebJobs.Extensions.DurableTask\DurableTaskExtension.cs:line 1480
I would like to split and process the dataframe concurrently using the Fan In behavior further in other activity functions; however it seems like the output of the activity function is too big for the Orchestrator. What should I do instead of it in this scenario? I would be thankful for any suggestions as I have officially entered the 'losing my mind' phase of debugging this. I've already tried compressing the DataFrame in many ways, to no avail.

Response generated while mocking is not correct in python tests

I am trying to unit test some of my functions using mocks in python. Few of them seems to be working as expected but I am trouble with the response of one the test case. Which it not returning the ideal response which it should.
###############motpenrichreceiverinfo.py##################
import requests
def getReceiverInfo(fundId):
print('Inside fetchReceiverInfo',fundId)
response = requests.get("REST URL FOR RECEIVER INFO")
if response.ok:
return response
else:
return None
def getMotpTrade(trade_id):
response = requests.get("REST URL")
if response.ok:
return response
else:
return None
def getMotpCanonical(trade_id):
response = requests.get("REST URL")
if response.ok:
return response
else:
return None
def EnrichTradeWithReceiverInfo(event, context):
#print('Lambda function started..')
trade_id = event['tradeId']
motpTrade = getMotpTrade(trade_id)
canonicalTrade = getMotpCanonical(trade_id)
fundId = motpTrade['Account']
#print(fundId)
data = getReceiverInfo(fundId)
print(data)
return data
##########################test_motptrade.py##############################
# Standard library imports
from unittest.mock import Mock, patch
# Third-party imports...
from nose.tools import assert_true
from nose.tools import assert_is_not_none, assert_list_equal, assert_dict_equal
# Local imports
from motpenrichreceiverinfo import getMotpTrade, getReceiverInfo, EnrichTradeWithReceiverInfo
#patch('motpenrichreceiverinfo.requests.get')
def test_motptraderequest_response(mock_get):
motpTrade = [{
"motpTrade":"motpTrade"
}]
# Configure the mock to return a response with an OK status code.
mock_get.return_value.ok = True
mock_get.return_value.json.return_value = motpTrade
# Send a request to the API server and store the response.
response = getMotpTrade('tradeId')
# If the request is sent successfully, then I expect a response to be returned.
assert_list_equal(response.json(), motpTrade)
#patch('motpenrichreceiverinfo.requests.get')
def test_getReceiverInfo_respose_ok(mock_get):
receiverInfo = [{
"reciever":"reciever"
}]
# Configure the mock to return a response with an OK status code.
mock_get.return_value.ok = True
mock_get.return_value.json.return_value = receiverInfo
# Send a request to the API server and store the response.
response = getReceiverInfo("1110")
# If the request is sent successfully, then I expect a response to be returned.
assert_list_equal(response.json(), receiverInfo)
#patch('motpenrichreceiverinfo.getMotpTrade')
#patch('motpenrichreceiverinfo.getMotpCanonical')
#patch('motpenrichreceiverinfo.getReceiverInfo')
def test_EnrichTradeWithReceiverInfo_ok(mock_get,mock_canonical,mock_receiverinfo):
motpTrade = [{
"motpTrade":"motpTrade"
}]
receiverInfo = [{
"reciever":"reciever"
}]
event = {"tradeId":"123456"}
# Configure the mock to return a response with an OK status code.
mock_get.return_value = Mock(ok=True)
mock_get.return_value.json.return_value = motpTrade
mock_canonical.return_value.ok = True
mock_canonical.return_value.json.return_value = [{}]
mock_receiverinfo.return_value.ok = True
mock_receiverinfo.return_value.json.return_value = receiverInfo
data = EnrichTradeWithReceiverInfo(event,"")
assert_true(mock_get.called)
assert_true(mock_receiverinfo.called)
assert_list_equal(data.json(),receiverInfo)
Here my first two cases are working as expected. But my last test case(test_EnrichTradeWithReceiverInfo_ok) is somehow not working. Ideally it should have response equal to receiverInfo object.But when I run it gives below error. It would be really helpful someone can help me understand what I am doing wrong here.
(venv) C:\Development\motplambdas>nosetests -v test_motptrade.py
test_motptrade.test_EnrichTradeWithReceiverInfo_ok ... FAIL
test_motptrade.test_getReceiverInfo_respose_ok ... ok
test_motptrade.test_motptraderequest_response ... ok
======================================================================
FAIL: test_motptrade.test_EnrichTradeWithReceiverInfo_ok
----------------------------------------------------------------------
Traceback (most recent call last):
File "c:\development\motplambdas\venv\lib\site-packages\nose\case.py", line 198, in runTest
self.test(*self.arg)
File "c:\program files\python38\lib\unittest\mock.py", line 1325, in patched
return func(*newargs, **newkeywargs)
File "C:\Development\motplambdas\test_motptrade.py", line 69, in test_EnrichTradeWithReceiverInfo_ok
assert_list_equal(data.json(),receiverInfo)
AssertionError: Lists differ: [{'motpTrade': 'motpTrade'}] != [{'reciever': 'reciever'}]
First differing element 0:
{'motpTrade': 'motpTrade'}
{'reciever': 'reciever'}
- [{'motpTrade': 'motpTrade'}]
+ [{'reciever': 'reciever'}]
----------------------------------------------------------------------
Ran 3 tests in 0.011s
FAILED (failures=1)

The issue is the order of which you are passing your Mock objects into the function call.
#patch('motpenrichreceiverinfo.getMotpTrade')
#patch('motpenrichreceiverinfo.getMotpCanonical')
#patch('motpenrichreceiverinfo.getReceiverInfo')
def test_EnrichTradeWithReceiverInfo_ok(mock_get,mock_canonical,mock_receiverinfo)
Mock objects are passed from the bottom up, meaning #patch('motpenrichreceiverinfo.getReceiverInfo') is the first Mock passed in to the function call, not the last as you have it listed. Due to this you end up setting getReceiverInfo to return the wrong value. The solution is to switch the function call to look like this:
def test_EnrichTradeWithReceiverInfo_ok(mock_receiverinfo, mock_canonical, mock_get)
You can read more about this here where it explains how nesting decorators works.
Note that the decorators are applied from the bottom upwards. This is the standard way that Python applies decorators. The order of the created mocks passed into your test function matches this order.

iot edge direct method handler in python

I have created a module for a Bacnet scan and it will respond with a list of devices and its address as a result. But I'm having trouble implementing a direct method handler in python. When i first tried implementing it myself i got this error. Which could mean I didn't successfully register the direct method callback. I have some references but it was from C# and azure docs is not helping me figure out the right method to register the callback. for IoTHubModuleClient there's a on_method_request_received and a receive_method_request. appreciate any help!
def iothub_client_scan_run():
try:
iot_client = iothub_client_init()
bacnet_scan_listener_thread = threading.Thread(target=device_method_listener, args=(iot_client,))
bacnet_scan_listener_thread.daemon = True
bacnet_scan_listener_thread.start()
while True:
time.sleep(1000)
def device_method_listener(iot_client):
while True:
# Receive the direct method request
method_request = iot_client.receive_method_request()
print (
"\nMethod callback called with:\nmethodName = {method_name}\npayload = {payload}".format(
method_name=method_request.name,
payload=method_request.payload
)
)
if method_request.name == "runBacnetScan":
response = bacnet_scan_device(method_request)
else:
response_payload = {"Response": "Direct method {} not defined".format(method_request.name)}
response_status = 404
# Send a method response indicating the method request was resolved
print('Sending method response')
iot_client.send_method_response(response)
print('Message sent!')
Edit:
Here is my route config

I was able to resolve my issue or at least find the root cause and it was my network configuration under the createOptions. It seems like there's an issue when I'm trying to do NetworkMode: host and connects to the IotModuleClient.connect_from_edge_environment via connect with connection string. I'm still trying to tweak the connection configuration but at least i know its not on the code.

async def method_request_handler(module_client):
while True:
method_request = await module_client.receive_method_request()
print (
"\nMethod callback called with:\nmethodName = {method_name}\npayload = {payload}".format(
method_name=method_request.name,
payload=method_request.payload
)
)
if method_request.name == "method1":
payload = {"result": True, "data": "some data"} # set response payload
status = 200 # set return status code
print("executed method1")
elif method_request.name == "method2":
payload = {"result": True, "data": 1234} # set response payload
status = 200 # set return status code
print("executed method2")
else:
payload = {"result": False, "data": "unknown method"} # set response payload
status = 400 # set return status code
print("executed unknown method: " + method_request.name)
# Send the response
method_response = MethodResponse.create_from_method_request(method_request, status, payload)
await module_client.send_method_response(method_response)
print('Message sent!')
def stdin_listener():
while True:
try:
selection = input("Press Q to quit\n")
if selection == "Q" or selection == "q":
print("Quitting...")
break
except:
time.sleep(10)
# Schedule task for C2D Listener
listeners = asyncio.gather(input1_listener(module_client), twin_patch_listener(module_client), method_request_handler(module_client))

SOAP call response only getting last result - Python

this is the first time i am posting . So apologizing in advance if something is not in order . So based on the below python script i am trying to call a SOAP API and recording the response. Then iterating through that response to call another method with data from the first response. But print(data) only displaying the last row,not complete result set. I am new to Python .
def lambda_handler(): #event, context--------------------
zsession = Session()
zsession.verify = False
url = 'TheURL'
local_file_name="test.wsdl"
transport = Transport(session=zsession)
transport.load_timeout=2000
client = Client(local_file_name, transport=transport)
service = client.create_service(
'{http://www.someurl.com', url)
response = service.RetrieveRegionsByCriteria('*')
content = ''
for item in response:
properties = service.RetrievePlanningSessionPropertiesByCriteria(criteria=item.regionIdentity.regionID)#item.regionIdentity.regionID))
for property in properties:
row_str ='{0}|{1}|{2}|{3}'.format(property.sessionIdentity.internalSessionID, GetStringValue(item.regionIdentity.regionID), GetStringValue(property.description), GetStringValue(property.scenario))
data = ''
data = data + row_str + "\n"
print(data)
print("Function completed successfully")

You're using custom service methods in your code, so I don't know what data you are receiving.
One issue I noticed is that you keep overwriting the same variable (row_str) in the property loop.
Change this line:
for property in properties:
row_str ='{0}|{1}|{2}|{3}'.format(property.sessionIdentity.internalSessionID, GetStringValue(item.regionIdentity.regionID), GetStringValue(property.description), GetStringValue(property.scenario))
To this:
row_str = ''
for property in properties:
row_str +='{0}|{1}|{2}|{3}\n'.format(property.sessionIdentity.internalSessionID, GetStringValue(item.regionIdentity.regionID), GetStringValue(property.description), GetStringValue(property.scenario))

Spotipy User Playlist Remove Tracks Issue

I am trying to use the Spotipy method to delete repeat occurrences of a track (so delete duplicates). But the function doesn't seem to work; the Spotify API call is returning an error that there is no authorization token.
Spotify API Return Error:
{
"error": {
"status": 401,
"message": "No token provided"
}
}
Python's Errors:
File "C:\Users\Dylan\Documents\PythonProjects\PlaylistTransfer\Spotify.py", line 87, in remove_all_duplicate_tracks
sp.user_playlist_remove_specific_occurrences_of_tracks(username, playlist_id, tracks)
File "C:\Users\Dylan\Documents\PythonProjects\PlaylistTransfer\venv\lib\site-packages\spotipy\client.py", line 539, in user_playlist_remove_specific_occurrences_of_tracks
payload=payload)
File "C:\Users\Dylan\Documents\PythonProjects\PlaylistTransfer\venv\lib\site-packages\spotipy\client.py", line 183, in _delete
return self._internal_call('DELETE', url, payload, kwargs)
File "C:\Users\Dylan\Documents\PythonProjects\PlaylistTransfer\venv\lib\site-packages\spotipy\client.py", line 124, in _internal_call
headers=r.headers)
spotipy.client.SpotifyException: http status: 400, code:-1 - https://api.spotify.com/v1/___________________________/tracks:
Could not remove tracks, please check parameters.
Here is my code:
def remove_all_duplicate_tracks(playlist_id, token):
sp = spotipy.Spotify(token)
username = get_username(token)
existing_tracks = get_track_uris_for_playlist(playlist_id, token)
duplicate_counter = Counter(existing_tracks)
tracks = []
for uri, count in duplicate_counter.items():
count = count-1
if count > 0:
# hard coded position as 1 for testing...
positions = [1]
#positions = [x for x in range(1, count+1)]
track_dict = {"uri": uri, "positions": positions}
tracks.append(track_dict)
sp.user_playlist_remove_specific_occurrences_of_tracks(username, playlist_id, tracks)
This is what "tracks" contains:
[{'uri': '6jq6rcOikCZAmjliAgAmfT', 'positions': [1]}, {'uri': '3tSmXSxaAnU1EPGKa6NytH', 'positions': [1]}, {'uri': '7jeI6EdY0elPSNz80mAKS8', 'positions': [1]}]
I tested the other methods get_username() and get_track_uris_for_playlist and they return what you'd expect and are working.

Although this answer comes quite late, it is needed because 1) the question is not solved and 2) I believe that it will be helpful to people with a similar problem.
First of all, you should restrict your question to the specific problem, which is the authorization error produced by calling the sp.user_playlist_remove_specific_occurrences_of_tracks() function. This would make the problem more clear. (In the way it is put, one has to dig up the code to find the "hot" spot! Also the details about the tracks just add to the confusion.)
So, I will limit my answer to just the problem and suggest using the following code as a basis:
# Data
username = (your username)
playlist_id = (playlist id) # The ID of the playlist containing the tracks to be deleted
track_ids = [(track_id), (track_id), ...] # List of track IDs to delete
# Authorization process
scope = "playlist-read-private"
token = spotipy.util.prompt_for_user_token(username, scope=scope)
sp = spotipy.Spotify(auth=token)
# Call the track deletion function
sp.user_playlist_remove_all_occurrences_of_tracks(username, playlist_id, track_ids)
I am using this process myself. I have just tried the above code with data of mine and it should also work for you.

You are trying to change user data:
Could not remove tracks, please check parameters.
Pass a valid scope, such as:
playlist-modify-public
playlist-modify-private
More on scopes: https://developer.spotify.com/documentation/general/guides/scopes/
import spotipy
import spotipy.util as util
scope = 'playlist-modify-public playlist-modify-private'
token = util.prompt_for_user_token(username, scope)
if token:
sp = spotipy.Spotify(auth=token)
results = sp.current_user_saved_tracks()
for item in results['items']:
track = item['track']
print track['name'] + ' - ' + track['artists'][0]['name']
else:
print "Can't get token for", username