We Keep Coding

Iterating through Azure ItemPaged object

I am calling the list operation to retrieve the metadata values of a blob storage.
My code looks like:
blob_service_list = storage_client.blob_services.list('rg-exercise1', 'sa36730')
for items in blob_service_list:
print((items.as_dict()))
What's happening in this case is that the returned output only contains the items which had a corresponding Azure object:
{'id': '/subscriptions/0601ba03-2e68-461a-a239-98cxxxxxx/resourceGroups/rg-exercise1/providers/Microsoft.Storage/storageAccounts/sa36730/blobServices/default', 'name': 'default', 'type': 'Microsoft.Storage/storageAccounts/blobServices', 'sku': {'name': 'Standard_LRS', 'tier': 'Standard'}, 'cors': {'cors_rules': [{'allowed_origins': ['www.xyz.com'], 'allowed_methods': ['GET'], 'max_age_in_seconds': 0, 'exposed_headers': [''], 'allowed_headers': ['']}]}, 'delete_retention_policy': {'enabled': False}}
Where-as, If I do a simple print of items, the output is much larger:
{'additional_properties': {}, 'id': '/subscriptions/0601ba03-2e68-461a-a239-98c1xxxxx/resourceGroups/rg-exercise1/providers/Microsoft.Storage/storageAccounts/sa36730/blobServices/default', 'name': 'default', 'type': 'Microsoft.Storage/storageAccounts/blobServices', 'sku': <azure.mgmt.storage.v2021_06_01.models._models_py3.Sku object at 0x7ff2f8f1a520>, 'cors': <azure.mgmt.storage.v2021_06_01.models._models_py3.CorsRules object at 0x7ff2f8f1a640>, 'default_service_version': None, 'delete_retention_policy': <azure.mgmt.storage.v2021_06_01.models._models_py3.DeleteRetentionPolicy object at 0x7ff2f8f1a6d0>, 'is_versioning_enabled': None, 'automatic_snapshot_policy_enabled': None, 'change_feed': None, 'restore_policy': None, 'container_delete_retention_policy': None, 'last_access_time_tracking_policy': None}
Any value which is None has been removed from my example code. How can I extend my example code to include the None fields and have the final output as a list?

I tried in my environment and got below results:
If you need to include the None values in the dictionary you can follow the below code:
Code:
from azure.mgmt.storage import StorageManagementClient
from azure.identity import DefaultAzureCredential
storage_client=StorageManagementClient(credential=DefaultAzureCredential(),subscription_id="<your sub id>")
blob_service_list = storage_client.blob_services.list('v-venkat-rg', 'venkat123')
for items in blob_service_list:
items_dict = items.as_dict()
for key, value in items.__dict__.items():
if value is None:
items_dict[key] = value
print(items_dict)
Console:
The above code executed with None value successfully.

Sending python requests and handling JSON lists

I am sending requests to a crypto network for data on accounts. You get sent back information, but I haven't yet encountered lists being sent in JSON until now. I want to parse certain information, but am having trouble because the JSON is a list and is not as easy to parse compared to normal JSON data.
import requests
import json
url = ' https://s1.ripple.com:51234/'
payload = {
"method": "account_objects",
"params": [
{
"account": "r9cZA1mLK5R5Am25ArfXFmqgNwjZgnfk59",
"ledger_index": "validated",
"type": "state",
"deletion_blockers_only": False,
"limit": 10
}
]
}
response = requests.post(url, data=json.dumps(payload))
print(response.text)
data = response.text
parsed = json.loads(data)
price = parsed['result']
price = price['account_objects']
for Balance in price:
print(Balance)
You will receive all the tokens the account holds and the value. I can not figure out how to parse this correctly and receive the correct one. This particular test account has a lot of tokens so I will only show the first tokens info.
RESULT
{'Balance': {'currency': 'ASP', 'issuer': 'rrrrrrrrrrrrrrrrrrrrBZbvji', 'value': '0'}, 'Flags': 65536, 'HighLimit': {'currency': 'ASP', 'issuer': 'r9cZA1mLK5R5Am25ArfXFmqgNwjZgnfk59', 'value': '0'}, 'HighNode': '0', 'LedgerEntryType': 'RippleState', 'LowLimit': {'currency': 'ASP', 'issuer': 'r3vi7mWxru9rJCxETCyA1CHvzL96eZWx5z', 'value': '10'}, 'LowNode': '0', 'PreviousTxnID': 'BF7555B0F018E3C5E2A3FF9437A1A5092F32903BE246202F988181B9CED0D862', 'PreviousTxnLgrSeq': 1438879, 'index': '2243B0B630EA6F7330B654EFA53E27A7609D9484E535AB11B7F946DF3D247CE9'}
I want to get the first bit of info, here. {'Balance': {'currency': 'ASP', 'issuer': 'rrrrrrrrrrrrrrrrrrrrBZbvji', 'value': '0'},
Specifically 'value' and the number
I have tried to take parse 'Balance' but since it is a list it is not as straight forward.

You're mixing up lists and dictionaries. In order to access a dictionary by key, you need to invoke the key, as such:
for Balance in price:
print(Balance['Balance'])
Yields the following results:
{'currency': 'CHF', 'issuer': 'rrrrrrrrrrrrrrrrrrrrBZbvji', 'value': '-0.3488146605801446'}
{'currency': 'BTC', 'issuer': 'rrrrrrrrrrrrrrrrrrrrBZbvji', 'value': '0'}
{'currency': 'USD', 'issuer': 'rrrrrrrrrrrrrrrrrrrrBZbvji', 'value': '-11.68225001668339'}
If you only wanted to extract the value, you simply dive one level deeper:
for Balance in price:
print(Balance['Balance']['value')
Which yields:
-0.3488146605801446
0
-11.68225001668339

I assume that under price['account_objects'] you have a list of dictionaries? And then in each dictionary you have in one of the keys: 'Balance': {'currency': 'ASP', 'issuer': 'rrrrrrrrrrrrrrrrrrrrBZbvji', 'value': '0'. If so, why don't you iterate over the list and then access each dictionary, like:
account_objects = price['account_objects']
for account_object in price:
print(account_object['Balance'])

Create a new dictionary from a nested JSON output after parsing

In python3 I need to get a JSON response from an API call,
and parse it so I will get a dictionary That only contains the data I need.
The final dictionary I ecxpt to get is as follows:
{'Severity Rules': ('cc55c459-eb1a-11e8-9db4-0669bdfa776e', ['cc637182-eb1a-11e8-9db4-0669bdfa776e']), 'auto_collector': ('57e9a4ec-21f7-4e0e-88da-f0f1fda4c9d1', ['0ab2470a-451e-11eb-8856-06364196e782'])}
the JSON response returns the following output:
{
'RuleGroups': [{
'Id': 'cc55c459-eb1a-11e8-9db4-0669bdfa776e',
'Name': 'Severity Rules',
'Order': 1,
'Enabled': True,
'Rules': [{
'Id': 'cc637182-eb1a-11e8-9db4-0669bdfa776e',
'Name': 'Severity Rule',
'Description': 'Look for default severity text',
'Enabled': False,
'RuleMatchers': None,
'Rule': '\\b(?P<severity>DEBUG|TRACE|INFO|WARN|ERROR|FATAL|EXCEPTION|[I|i]nfo|[W|w]arn|[E|e]rror|[E|e]xception)\\b',
'SourceField': 'text',
'DestinationField': 'text',
'ReplaceNewVal': '',
'Type': 'extract',
'Order': 21520,
'KeepBlockedLogs': False
}],
'Type': 'user'
}, {
'Id': '4f6fa7c6-d60f-49cd-8c3d-02dcdff6e54c',
'Name': 'auto_collector',
'Order': 4,
'Enabled': True,
'Rules': [{
'Id': '2d6bdc1d-4064-11eb-8856-06364196e782',
'Name': 'auto_collector',
'Description': 'DO NOT CHANGE!! Created via API coralogix-blocker tool',
'Enabled': False,
'RuleMatchers': None,
'Rule': 'AUTODISABLED',
'SourceField': 'subsystemName',
'DestinationField': 'subsystemName',
'ReplaceNewVal': '',
'Type': 'block',
'Order': 1,
'KeepBlockedLogs': False
}],
'Type': 'user'
}]
}
I was able to create a dictionary that contains the name and the RuleGroupsID, like that:
response = requests.get(url,headers=headers)
output = response.json()
outputlist=(output["RuleGroups"])
groupRuleName = [li['Name'] for li in outputlist]
groupRuleID = [li['Id'] for li in outputlist]
# Create a dictionary of NAME + ID
ruleDic = {}
for key in groupRuleName:
for value in groupRuleID:
ruleDic[key] = value
groupRuleID.remove(value)
break
Which gave me a simple dictionary:
{'Severity Rules': 'cc55c459-eb1a-11e8-9db4-0669bdfa776e', 'Rewrites': 'ddbaa27e-1747-11e9-9db4-0669bdfa776e', 'Extract': '0cb937b6-2354-d23a-5806-4559b1f1e540', 'auto_collector': '4f6fa7c6-d60f-49cd-8c3d-02dcdff6e54c'}
but when I tried to parse it as nested JSON things just didn't work.

In the end, I managed to create a function that returns this dictionary,
I'm doing it by breaking the JSON into 3 lists by the needed elements (which are Name, Id, and Rules from the first nest), and then create another list from the nested JSON ( which listed everything under Rule) which only create a list from the keyword "Id".
Finally creating a dictionary using a zip command on the lists and dictionaries created earlier.
def get_filtered_rules() -> List[dict]:
groupRuleName = [li['Name'] for li in outputlist]
groupRuleID = [li['Id'] for li in outputlist]
ruleIDList = [li['Rules'] for li in outputlist]
ruleIDListClean = []
ruleClean = []
for sublist in ruleIDList:
try:
lstRule = [item['Rule'] for item in sublist]
ruleClean.append(lstRule)
ruleContent=list(zip(groupRuleName, ruleClean))
ruleContentDictionary = dict(ruleContent)
lstID = [item['Id'] for item in sublist]
ruleIDListClean.append(lstID)
# Create a dictionary of NAME + ID + RuleID
ruleDic = dict(zip(groupRuleName, zip(groupRuleID, ruleIDListClean)))
except Exception as e: print(e)
return ruleDic

Pandas read the chat log log json to data frame?

How to converting the multiple list to data frame. below list contains the details about cloud containers want to extract the information like name , language , description and workspace id.
{'workspaces': [{'name': 'A_SupportAgent_dev',
'language': 'en',
'metadata': {'api_version': {'major_version': 'v1',
'minor_version': '2019-02-28'},
'digressions': True},
'description': 'Credit Card Banking Support Agent to assist with Sales And Service, created by Oliver Ivanoski and Steve Green',
'workspace_id': '',
'learning_opt_out': False},
{'name': 'Neatnik Watson Assistant Webhook Demo Skill',
'language': 'en',
'metadata': {'api_version': {'major_version': 'v1',
'minor_version': '2019-02-28'}},
'webhooks': [{'url': 'https://neatnik.net/watson/assistant/webhook/',
'name': 'main_webhook',
'headers': []}],
'description': '',
'workspace_id': '',
'system_settings': {'tooling': {'store_generic_responses': True},
'system_entities': {'enabled': True},
'spelling_auto_correct': True},
'learning_opt_out': False}]
'pagination': {'refresh_url': '/v1/workspaces?version=2019-02-28'}}
Want to convert the above list below data frame
Tried
pd.DataFrame(list(Workspace_List.items()) ,columns=['workspaces', 'pagination'])
columns = list(Workspace_List.keys())
values = list(Workspace_List.values())
arr_len = len(values)

You need to specify columns as you have another dictionary. So i think following below code will help u to organize your desire output
key = ['name','language','description','workspace_id']
output = pd.DataFrame(columns = key)
for i in range(len(df['workspaces'])):
ll = df['workspaces'][i]
output.loc[i] = [ll[x] for x in key]

how to use nested dictionary in python?

I am trying to write some code with the Hunter.io API to automate some of my b2b email scraping. It's been a long time since I've written any code and I could use some input. I have a CSV file of Urls, and I want to call a function on each URL that outputs a dictionary like this:
`{'domain': 'fromthebachrow.com', 'webmail': False, 'pattern': '{f}{last}', 'organization': None, 'emails': [{'value': 'fbach#fromthebachrow.com', 'type': 'personal', 'confidence': 91, 'sources': [{'domain': 'fromthebachrow.com', 'uri': 'http://fromthebachrow.com/contact', 'extracted_on': '2017-07-01'}], 'first_name': None, 'last_name': None, 'position': None, 'linkedin': None, 'twitter': None, 'phone_number': None}]}`
for each URL I call my function on. I want my code to return just the email address for each key labeled 'value'.
Value is a key that is contained in a list that itself is an element of the directory my function outputs. I am able to access the output dictionary to grab the list that is keyed to 'emails', but I don't know how to access the dictionary contained in the list. I want my code to return the value in that dictionary that is keyed with 'value', and I want it to do so for all of my urls.
from pyhunyrt import PyHunter
import csv
file=open('urls.csv')
reader=cvs.reader (file)
urls=list(reader)
hunter=PyHunter('API Key')
for item in urls:
output=hunter.domain_search(item)
output['emails'`
which returns a list that looks like this for each item:
[{
'value': 'fbach#fromthebachrow.com',
'type': 'personal',
'confidence': 91,
'sources': [{
'domain': 'fromthebachrow.com',
'uri': 'http://fromthebachrow.com/contact',
'extracted_on': '2017-07-01'
}],
'first_name': None,
'last_name': None,
'position': None,
'linkedin': None,
'twitter': None,
'phone_number': None
}]
How do I grab the first dictionary in that list and then access the email paired with 'value' so that my output is just an email address for each url I input initially?

To grab the first dict (or any item) in a list, use list[0], then to grab a value of a key value use ["value"]. To combine it, you should use list[0]["value"]