How do I parse this data with fabric? - python

I use fabric perform scrapyd task, the server returns the id of the task being performed, but I want to get this code below and put all id in list, but when I use r.status , there occur error: '_AttributeString' object has no attribute 'status' error, how do I get all id?the code blow:
#task
def stop_slave_machine(slave_ip = None):
jobs_id = []
with cd("/spider/distributed/wzws"):
if not None:
r = local("curl http://%s:%s/listjobs.json?project=WzwsSpider" % (slave_ip, scrapyd_port))
print(r.status)
and the server return data:
{"status": "ok", "running": [{"start_time": "2016-03-28 18:21:21.951943", "id": "d10eae6cf4ce11e5a6646cae8b23c5da", "spider": "wzws"}, {"start_time": "2016-03-28 18:21:26.945244", "id": "d11a47f4f4ce11e5a6646cae8b23c5da", "spider": "wzws"}, {"start_time": "2016-03-28 18:21:31.941162", "id": "d12320ccf4ce11e5a6646cae8b23c5da", "spider": "wzws"}, {"start_time": "2016-03-28 18:21:36.941122", "id": "d12975b2f4ce11e5a6646cae8b23c5da", "spider": "wzws"}, {"start_time": "2016-03-28 18:21:41.941010", "id": "d131096cf4ce11e5a6646cae8b23c5da", "spider": "wzws"}], "finished": [], "pending": [], "node_name": "XXXXXXX"}

That's a JSON body being returned. You can use python's json library to turn the response into a python object. From there you can iterate over the list of "running" to extract out the id for each.
Something like this:
from json import loads
# turn r into a python object as long as r is a string (hence loads not load)
returned = loads(r)
# Make a list ids from a list comprehension where we pull out the value
# id from each item in the list 'running' from the object returned
ids = [ r["id"] for r in returned["running"] ]

Related

How to get the value from particular key using python?

resp = {
"Name": "test",
"os": "windows",
"Agent": {
"id": "2",
"status": [
{
"code": "123",
"level": "Info",
"displayStatus": "Ready",
"message": "running",
"time": "2022-01-18T09:51:08+00:00"
}
]
}
I am trying to get the time value from the JSON.
I tried the below code but faced error with dict
resp1 = json.loads(resp)
resp2 = resp1.values()
creation_time = resp2.get("Agent").get("status")
val= creation_time["time"]
print(val) ## Thrwoing error as dict_values has no 'get'
Any suggestion on python how to take this time values
Few problems I noticed
You are trying to load a Dict type using the json's loads function which is supposed to get a string in json format (ex: '{ "name":"John", "age":30, "city":"New York"}')
You tried to access resp2 before declaration (I guessed you meant "resp1?")
You're using resp3 without declaration.
You are missing }
You don't need the .value() function because it will return a list.
Also creation time is a list with one object, so you need to access it too.
Considering all this, you can change it as follows:
import json
resp = '{ "Name": "test", "os": "windows","Agent": {"id": "2","status": [{"code": "123","level": "Info","displayStatus": "Ready","message": "running","time": "2022-01-18T09:51:08+00:00"}]}}'
resp1 = json.loads(resp)
creation_time = resp1.get("Agent").get("status")
val= creation_time[0]["time"]
print(val)
You just need to access the dicts using [] like so:
resp = {"Name": "test", "os": "windows", "Agent": {"id": "2","status": [{"code": "123","level": "Info","displayStatus": "Ready","message": "running","time": "2022-01-18T09:51:08+00:00"}]}}
creation_time = resp["Agent"]["status"]
val= creation_time[0]["time"]
print(val)
Output:
2022-01-18T09:51:08+00:00

How to parse nested JSON object?

I am working on a new project in HubSpot that returns nested JSON like the sample below. I am trying to access the associated contacts id, but am struggling to reference it correctly (the id I am looking for is the value '201' in the example below). I've put together this script, but this script only returns the entire associations portion of the JSON and I only want the id. How do I reference the id correctly?
Here is the output from the script:
{'contacts': {'paging': None, 'results': [{'id': '201', 'type': 'ticket_to_contact'}]}}
And here is the script I put together:
import hubspot
from pprint import pprint
client = hubspot.Client.create(api_key="API_KEY")
try:
api_response = client.crm.tickets.basic_api.get_page(limit=2, associations=["contacts"], archived=False)
for x in range(2):
pprint(api_response.results[x].associations)
except ApiException as e:
print("Exception when calling basic_api->get_page: %s\n" % e)
Here is what the full JSON looks like ('contacts' property shortened for readability):
{
"results": [
{
"id": "34018123",
"properties": {
"content": "Hi xxxxx,\r\n\r\nCan you clarify on how the blocking of script happens? Is it because of any CSP (or) the script will decide run time for every URL’s getting triggered from browser?\r\n\r\nRegards,\r\nLogan",
"createdate": "2019-07-03T04:20:12.366Z",
"hs_lastmodifieddate": "2020-12-09T01:16:12.974Z",
"hs_object_id": "34018123",
"hs_pipeline": "0",
"hs_pipeline_stage": "4",
"hs_ticket_category": null,
"hs_ticket_priority": null,
"subject": "RE: call followup"
},
"createdAt": "2019-07-03T04:20:12.366Z",
"updatedAt": "2020-12-09T01:16:12.974Z",
"archived": false
},
{
"id": "34018892",
"properties": {
"content": "Hi Guys,\r\n\r\nI see that we were placed back on the staging and then removed again.",
"createdate": "2019-07-03T07:59:10.606Z",
"hs_lastmodifieddate": "2021-12-17T09:04:46.316Z",
"hs_object_id": "34018892",
"hs_pipeline": "0",
"hs_pipeline_stage": "3",
"hs_ticket_category": null,
"hs_ticket_priority": null,
"subject": "Re: Issue due to server"
},
"createdAt": "2019-07-03T07:59:10.606Z",
"updatedAt": "2021-12-17T09:04:46.316Z",
"archived": false,
"associations": {
"contacts": {
"results": [
{
"id": "201",
"type": "ticket_to_contact"
}
]
}
}
}
],
"paging": {
"next": {
"after": "35406270",
"link": "https://api.hubapi.com/crm/v3/objects/tickets?associations=contacts&archived=false&hs_static_app=developer-docs-ui&limit=2&after=35406270&hs_static_app_version=1.3488"
}
}
}
You can do api_response.results[x].associations["contacts"]["results"][0]["id"].
Sorted this out, posting in case anyone else is struggling with the response from the HubSpot v3 Api. The response schema for this call is:
Response schema type: Object
String results[].id
Object results[].properties
String results[].createdAt
String results[].updatedAt
Boolean results[].archived
String results[].archivedAt
Object results[].associations
Object paging
Object paging.next
String paging.next.after
String paging.next.linkResponse schema type: Object
String results[].id
Object results[].properties
String results[].createdAt
String results[].updatedAt
Boolean results[].archived
String results[].archivedAt
Object results[].associations
Object paging
Object paging.next
String paging.next.after
String paging.next.link
So to access the id of the contact associated with the ticket, you need to reference it using this notation:
api_response.results[1].associations["contacts"].results[0].id
notes:
results[x] - reference the result in the index
associations["contacts"] -
associations is a dictionary object, you can access the contacts item
by it's name
associations["contacts"].results is a list - reference
by the index []
id - is a string
In my case type was ModelProperty or CollectionResponseProperty couldn't reach dict anyhow.
For the record this got me to go through the results.
for result in list(api_response.results):
ID = result.id

Why does my pandas list contain duplicates?

I have a function that takes api_response and tests to see if a condition is met if "meta" not in api_response:. If the condition is met, I extract the key/pair value percent_complete value from the response and print it to the console. This value is a percentage, and only appears once in the api_response.
My issue is, when it prints to the console, the list (which should contain 1x value e.g, 0.19) is printing the value twice.
E.g., if percent_complete == 0.19, the console will print Your data requested, associated with ID: 2219040 is (0.19, 0.19) complete!.
Is there anything wrong with my code, that might be causing this?
Function -
def api_call():
# Calling function that returns API authentication details for use in endpoint_initializer()
key, secret, url = ini_reader()
# Calling function that makes initial API POST call and returns endpoint_url to call, until data is returned.
endpoint_url = endpoint_initializer()
# saving current date in a variable, for use when printing user message
date = dt.datetime.today().strftime("%Y-%m-%d")
# Printing endpoint_url and current date.
print("-------------------------------------\n","API URL constructed for:", date, "\n-------------------------------------")
print("-------------------------------------------------------------\n","Endpoint:", endpoint_url, "\n-------------------------------------------------------------")
# Loop will continously call the end_point URL until data is returned. When data is not returned the `percent_complete' key value is extracted from api response.
# this will inform user of status of data aggregation.
while True:
response = requests.get(url = endpoint_url, auth = HTTPBasicAuth(key, secret), headers = {"Vendor-firm": "343"})
api_response = json.loads(response.text)
# Test condition to see if "meta" is in api_response. Meta only in response, when data is ready.
if "meta" not in api_response:
id_value = "id"
res1 = [val[id_value] for key, val in api_response.items() if id_value in val]
id_value = "".join(res1)
percent_value = "percent_complete"
res2 = (tuple(api_response["data"]["attributes"].get("percent_complete", '') for key, val in api_response.items()))
print(f' Your data requested, associated with ID: {id_value} is {res2} complete!')
time.sleep(5)
# Condition to allow API response to be returned, if condition is not met.
elif "meta" in api_response:
return api_response
Example API response -
{
"data": {
"id": "2219040",
"type": "jobs",
"attributes": {
"job_type": "PORTFOLIO_VIEW_RESULTS",
"started_at": "2021-12-18T17:40:17Z",
"parameters": {
"end_date": "2021-12-14",
"output_type": "json",
"view_id": 304078,
"portfolio_id": 1,
"portfolio_type": "firm",
"start_date": "2021-12-14"
},
"percent_complete": 0.19,
"status": "In Progress"
},
"relationships": {
"creator": {
"links": {
"self": "/v1/jobs/2219040/relationships/creator",
"related": "/v1/jobs/2219040/creator"
},
"data": {
"type": "users",
"id": "731221"
}
}
},
"links": {
"self": "/v1/jobs/2219040"
}
},
"included": []
}
The dictionary in your response contains two items ('data' and 'included'). Your code that creates res2 iterates over all of the items:
res2 = (tuple(api_response["data"]["attributes"].get("percent_complete", '') for key, val in api_response.items()))
so you get the information twice. Since you are just pulling data from the 'data' key, it's silly to iterate over the items. Right? Just do:
res2 = api_response["data"]["attributes"].get("percent_complete", '')

Accessing nested objects with python

I have a response that I receive from foursquare in the form of json. I have tried to access the certain parts of the object but have had no success. How would I access say the address of the object? Here is my code that I have tried.
url = 'https://api.foursquare.com/v2/venues/explore'
params = dict(client_id=foursquare_client_id,
client_secret=foursquare_client_secret,
v='20170801', ll=''+lat+','+long+'',
query=mealType, limit=100)
resp = requests.get(url=url, params=params)
data = json.loads(resp.text)
msg = '{} {}'.format("Restaurant Address: ",
data['response']['groups'][0]['items'][0]['venue']['location']['address'])
print(msg)
Here is an example of json response:
"items": [
{
"reasons": {
"count": 0,
"items": [
{
"summary": "This spot is popular",
"type": "general",
"reasonName": "globalInteractionReason"
}
]
},
"venue": {
"id": "412d2800f964a520df0c1fe3",
"name": "Central Park",
"contact": {
"phone": "2123106600",
"formattedPhone": "(212) 310-6600",
"twitter": "centralparknyc",
"instagram": "centralparknyc",
"facebook": "37965424481",
"facebookUsername": "centralparknyc",
"facebookName": "Central Park"
},
"location": {
"address": "59th St to 110th St",
"crossStreet": "5th Ave to Central Park West",
"lat": 40.78408342593807,
"lng": -73.96485328674316,
"labeledLatLngs": [
{
"label": "display",
"lat": 40.78408342593807,
"lng": -73.96485328674316
}
],
the full response can be found here
Like so
addrs=data['items'][2]['location']['address']
Your code (at least as far as loading and accessing the object) looks correct to me. I loaded the json from a file (since I don't have your foursquare id) and it worked fine. You are correctly using object/dictionary keys and array positions to navigate to what you want. However, you mispelled "address" in the line where you drill down to the data. Adding the missing 'a' made it work. I'm also correcting the typo in the URL you posted.
I answered this assuming that the example JSON you linked to is what is stored in data. If that isn't the case, a relatively easy way to see exact what python has stored in data is to import pprint and use it like so: pprint.pprint(data).
You could also start an interactive python shell by running the program with the -i switch and examine the variable yourself.
data["items"][2]["location"]["address"]
This will access the address for you.
You can go to any level of nesting by using integer index in case of an array and string index in case of a dict.
Like in your case items is an array
#items[int index]
items[0]
Now items[0] is a dictionary so we access by string indexes
item[0]['location']
Now again its an object s we use string index
item[0]['location']['address]

Can't iterate over my own object

I am new to Python and can't figure this out. I am trying to make an object from a json feed. I am trying to basically make a dictionary for each item in the json fed that has every property. The error I get is either TypeError: 'mediaObj' object is not subscriptable or not iterable
For bonus points, the array has many sub dictionaries too. What I would like is to be able to access that nested data as well.
Here is my code:
url = jsonfeedwithalotofdata.com
data = urllib.request.urlopen(url)
data = instagramData.read()
data = instagramData.decode('utf-8')
data = json.loads(data)
media = data['data']
class mediaObj:
def __init__(self, item):
for key in item:
setattr(self, key, item[key])
print(self[key])
def run(self):
return self['id']
for item in media:
mediaPiece = mediaObj(item)
This would come from a json feed that looks as follows (so data is the array that comes after "data"):
"data": [
{
"attribution": null,
"videos": {},
"tags": [],
"type": "video",
"location": null,
"comments": {},
"filter": "Normal",
"created_time": "1407423448461",
"link": "http://instagram.com/p/rabdfdIw9L7D-/",
"likes": {},
"images": {},
"users_in_photo": [],
"caption": {},
"user_has_liked": true,
"id": "782056834879232959294_1051813051",
"user": {}
}
So my hope was that I could create an object for every item in the array, and then I could, for instance, say:
print(mediaPiece['id'])
or even better
print(mediaPiece['comments'])
And see a list of comments. Thanks a million
You're having a problem because you're using attributes to store your data items, but using list/dictionary lookup syntax to try to retrieve them.
Instead of print(self[key]), use print(getattr(self, key)), and instead of return self['id'] use return self.id.

Categories