Related
I'm trying to scrape the countries with their rates from this page: https://www.bossrevolution.ca/en-ca. The webpage has a drop down list which shows all the countries. Once you click on the country you are redirected to a next page and on that webpage you can see the rate of the country in question. What I basically want to do is a run a code whereby I have all the countries with their rates. The only difficult thing is that for every country I'm redirected to a new webpage.
I started with step one and that is scraping all the countries from the drop down list but I have some difficulties with that. The code that I wrote:
from bs4 import BeautifulSoup
import requests
url = "https://www.bossrevolution.ca/en-ca/services/international-calling"
req = requests.get(url)
data = BeautifulSoup(req.text, "lxml")
data1 = data.find("ul")
for li in data1.find_all("li"):
print(li.text, end=" ")
My output:
Services
International Calling
International Mobile Top Up
International Calling
International Mobile Top Up
Support
My Account
I'm trying to get the countries seen on the picture?
All countries
Could someone please help an intern out? It would help me a lot with a achieving a good grade :)
Thanks in advance!!
The data you see on the page is loaded via JavaScript, so beautifulsoup doesn't see them. To get all rates for all countries in Json format you can use next example:
import json
import requests
country_list = "https://www.bossrevolution.com/en-us/rates/ajax/countries_list"
rates_api = "https://www.bossrevolution.com/en-us/rates/ajax/data/{}"
countries = requests.get(country_list).json()
for c in countries:
url = rates_api.format(c[0])
data = requests.get(url).json()
# pretty print the returned data:
print(json.dumps(data, indent=4))
print("-" * 80)
Prints:
{
"CountryId": "AF",
"CountryName": "Afghanistan",
"Amount": "",
"Page": 2,
"Rates": {
"CountryCode": "AF",
"name": "Afghanistan",
"dest": [
{
"name": "Afghanistan",
"Currency": null,
"rate": "0.25"
},
{
"name": "Afghanistan Cellular",
"Currency": null,
"rate": "0.117"
},
{
"name": "Afghanistan Cellular-Roshan",
"Currency": null,
"rate": "0.135"
}
],
"IsdCode": null
},
"IsRatesFound": true,
"DestinationRates": [
{
"name": "Afghanistan",
"Currency": null,
"rate": "0.25"
},
{
"name": "Afghanistan Cellular",
"Currency": null,
"rate": "0.117"
},
{
"name": "Afghanistan Cellular-Roshan",
"Currency": null,
"rate": "0.135"
}
],
"LastUpdatedAt": "/Date(1668229257000)/"
}
--------------------------------------------------------------------------------
{
"CountryId": "AL",
"CountryName": "Albania",
"Amount": "",
"Page": 2,
"Rates": {
"CountryCode": "AL",
"name": "Albania",
"dest": [
{
"name": "Albania",
"Currency": null,
"rate": "0.15"
},
{
"name": "Albania Cellular",
"Currency": null,
"rate": "0.359"
},
{
"name": "Albania,Tirana",
"Currency": null,
"rate": "0.161"
}
],
"IsdCode": null
},
"IsRatesFound": true,
"DestinationRates": [
{
"name": "Albania",
"Currency": null,
"rate": "0.15"
},
{
"name": "Albania Cellular",
"Currency": null,
"rate": "0.359"
},
{
"name": "Albania,Tirana",
"Currency": null,
"rate": "0.161"
}
],
"LastUpdatedAt": "/Date(1668229257000)/"
}
--------------------------------------------------------------------------------
{
"CountryId": "DZ",
"CountryName": "Algeria",
"Amount": "",
"Page": 2,
"Rates": {
"CountryCode": "DZ",
"name": "Algeria",
"dest": [
{
"name": "Algeria",
"Currency": null,
"rate": "0.083"
},
{
"name": "Algeria Cellular",
"Currency": null,
"rate": "0.625"
}
],
"IsdCode": null
},
"IsRatesFound": true,
"DestinationRates": [
{
"name": "Algeria",
"Currency": null,
"rate": "0.083"
},
{
"name": "Algeria Cellular",
"Currency": null,
"rate": "0.625"
}
],
"LastUpdatedAt": "/Date(1668229257000)/"
}
--------------------------------------------------------------------------------
...and so on.
I'm trying to compare the contents of a dictionary from 2 lists of dict, I have data like this
data1 = [
{
"name": "MoreDependentsInd",
"description": "Form 1040-SP Indicator",
"data_type": "CheckboxType",
"xpath": "",
},
{
"name": "DependentFirstNm",
"description": "Form 1040-SR Indicator",
"data_type": "PersonNameControlType",
"xpath": "",
},
]
data2 = [
{
"id": 29,
"kind": "IN",
"container": 1,
"content": "null",
"uid": "IRS 1040-DependentDetail",
"title": "DependentDetail",
"display_name": "null",
"description": "null",
"extra_info": {
"kind": "null",
"include": {
"fields": [
{
"kind": "PersonFirstNameType",
"name": "DependentFirstNm",
"annotation": {
"documentation": {
"description": "Dependent First Name",
"line_number": "(1)",
}
},
"restriction": "null",
},
{
"kind": "PersonLastNameType",
"name": "DependentLastNm",
"annotation": {
"documentation": {
"description": "Dependent Last Name",
"line_number": "(1)",
}
},
"restriction": "null",
},
{
"kind": "PersonNameControlType",
"name": "DependentNameControlTxt",
"annotation": {
"documentation": {
"description": "Dependent Name Control",
"line_number": "(1)",
}
},
"restriction": "null",
},
{
"kind": "null",
"name": "IdentityProtectionPIN",
"annotation": {
"documentation": {
"description": "Dependent/Qualifying Child Identity Protection PIN",
"line_number": "(2)",
}
},
"restriction": {"rule": "null"},
},
{
"kind": "null",
"name": "DependentRelationshipCd",
"annotation": {
"documentation": {
"description": "Dependent Relationship Code",
"line_number": "(3)",
}
},
"restriction": {
"rule": {
"kind": "TextType",
"enumeration": [
"SON",
"DAUGHTER",
"STEPCHILD",
"FOSTER CHILD",
"BROTHER",
"SISTER",
"STEPBROTHER",
"STEPSISTER",
"HALF BROTHER",
"HALF SISTER",
"GRANDCHILD",
"NIECE",
"NEPHEW",
"PARENT",
"GRANDPARENT",
"AUNT",
"UNCLE",
"OTHER",
"NONE",
],
}
},
},
{
"kind": "PersonFirstNameType",
"name": "DependentFirstNm",
"annotation": {
"documentation": {
"description": "Dependent First Name",
"line_number": "(1)",
}
},
"restriction": "null",
},
{
"kind": "PersonLastNameType",
"name": "DependentLastNm",
"annotation": {
"documentation": {
"description": "Dependent Last Name",
"line_number": "(1)",
}
},
"restriction": "null",
},
{
"kind": "PersonNameControlType",
"name": "DependentNameControlTxt",
"annotation": {
"documentation": {
"description": "Dependent Name Control",
"line_number": "(1)",
}
},
"restriction": "null",
},
{
"kind": "null",
"name": "IdentityProtectionPIN",
"annotation": {
"documentation": {
"description": "Dependent/Qualifying Child Identity Protection PIN",
"line_number": "(2)",
}
},
"restriction": {"rule": "null"},
},
{
"kind": "null",
"name": "DependentRelationshipCd",
"annotation": {
"documentation": {
"description": "Dependent Relationship Code",
"line_number": "(3)",
}
},
"restriction": {
"rule": {
"kind": "TextType",
"enumeration": [
"SON",
"DAUGHTER",
"STEPCHILD",
"FOSTER CHILD",
"BROTHER",
"SISTER",
"STEPBROTHER",
"STEPSISTER",
"HALF BROTHER",
"HALF SISTER",
"GRANDCHILD",
"NIECE",
"NEPHEW",
"PARENT",
"GRANDPARENT",
"AUNT",
"UNCLE",
"OTHER",
"NONE",
],
}
},
},
{
"kind": "PersonFirstNameType",
"name": "DependentFirstNm",
"annotation": {
"documentation": {
"description": "Dependent First Name",
"line_number": "(1)",
}
},
"restriction": "null",
},
{
"kind": "PersonLastNameType",
"name": "DependentLastNm",
"annotation": {
"documentation": {
"description": "Dependent Last Name",
"line_number": "(1)",
}
},
"restriction": "null",
},
{
"kind": "PersonNameControlType",
"name": "DependentNameControlTxt",
"annotation": {
"documentation": {
"description": "Dependent Name Control",
"line_number": "(1)",
}
},
"restriction": "null",
},
{
"kind": "null",
"name": "IdentityProtectionPIN",
"annotation": {
"documentation": {
"description": "Dependent/Qualifying Child Identity Protection PIN",
"line_number": "(2)",
}
},
"restriction": {"rule": "null"},
},
{
"kind": "null",
"name": "DependentRelationshipCd",
"annotation": {
"documentation": {
"description": "Dependent Relationship Code",
"line_number": "(3)",
}
},
"restriction": {
"rule": {
"kind": "TextType",
"enumeration": [
"SON",
"DAUGHTER",
"STEPCHILD",
"FOSTER CHILD",
"BROTHER",
"SISTER",
"STEPBROTHER",
"STEPSISTER",
"HALF BROTHER",
"HALF SISTER",
"GRANDCHILD",
"NIECE",
"NEPHEW",
"PARENT",
"GRANDPARENT",
"AUNT",
"UNCLE",
"OTHER",
"NONE",
],
}
},
},
],
"reference": "null",
},
"annotation": "null",
"max_occurs": "100",
"min_occurs": "0",
"restriction": "null",
},
"instruction": "null",
"created_at": "2022-04-26T09:53:36.426118Z",
"updated_at": "2022-04-26T09:53:36.426203Z",
},
{
"id": 30,
"kind": "IN",
"container": 1,
"content": "null",
"uid": "IRS 1040-MoreDependentsInd",
"title": "MoreDependentsInd",
"display_name": "null",
"description": "null",
"extra_info": {
"kind": "CheckboxType",
"include": "null",
"annotation": {
"documentation": {
"description": "More Dependents Indicator",
"line_number": "null",
}
},
"max_occurs": "null",
"min_occurs": "0",
"restriction": "null",
},
"instruction": "null",
"created_at": "2022-04-26T09:53:36.427552Z",
"updated_at": "2022-04-26T09:53:36.427647Z",
},
]
for i in data1:
for j in data2:
if i["name"] == j["title"]:
print("success")
the result is a success, but in my code, there is a deficiency, if it processes a lot of data then the performance will feel slow, how to fix it? and what if the value such as the data type is in a key that we don't know is located?
for example, I search value data_type from data1 I compare it with CheckboxType
"extra_info": {
"kind": "CheckboxType",
"include": "null",
"annotation": {
"documentation": {
"description": "More Dependents Indicator",
"line_number": "null",
}
},
from data2 on index 1 data2[1] but the value CheckboxType is also exist in another key on data2 in another index
Thanks!
You can do the test in O(n) instead of O(n^2) by building sets of name and title (which is O(n)) and then taking their intersection (which is also O(n) now that they're sets):
>>> {i["name"] for i in data1} & {i["title"] for i in data2}
{'MoreDependentsInd'}
Non-empty sets are truthy so you can use this in a conditional if you don't care about what the intersection is:
>>> if {i["name"] for i in data1} & {i["title"] for i in data2}:
... print("success")
...
success
If you want to look at all string values without having to know that name and title are the intersecting keys, use another generator expression to put all string values into the two sets:
>>> {v for i in data1 for v in i.values() if isinstance(v, str)} & {v for i in data2 for v in i.values() if isinstance(v, str)}
{'MoreDependentsInd'}
Everything with my script runs fine until I try to run it through a for loop. Specifically, when I attempt to index a specific array within the object. Before I get to the The script is intended to grab the delivery date for each tracking number in my list.
This is my script:
import requests
import json
TrackList = ['1Z3X756E0310496105','1ZX0373R0303581450','1ZX0373R0103574417']
url = 'https://onlinetools.ups.com/rest/Track'
para1 = '...beginning of JSON request string...'
para2 = '...end of JSON request string...'
for TrackNum in TrackList:
parameters = para1+TrackNum+para2
resp = requests.post(url = url, data = parameters, verify=False)
data = json.loads(resp.text)
DelDate = data['TrackResponse']['Shipment']['Package'][0]['Activity'][0]['Date']
print(DelDate)
JSON API Response (if needed):
{
"TrackResponse": {
"Response": {
"ResponseStatus": {
"Code": "1",
"Description": "Success"
},
"TransactionReference": {
"CustomerContext": "Analytics Inquiry"
}
},
"Shipment": {
"InquiryNumber": {
"Code": "01",
"Description": "ShipmentIdentificationNumber",
"Value": "1ZX0373R0103574417"
},
"Package": {
"Activity": [
{
"ActivityLocation": {
"Address": {
"City": "OKLAHOMA CITY",
"CountryCode": "US",
"PostalCode": "73128",
"StateProvinceCode": "OK"
},
"Code": "M3",
"Description": "Front Desk",
"SignedForByName": "CUMMINGS"
},
"Date": "20190520",
"Status": {
"Code": "9E",
"Description": "Delivered",
"Type": "D"
},
"Time": "091513"
},
{
"ActivityLocation": {
"Address": {
"City": "Oklahoma City",
"CountryCode": "US",
"StateProvinceCode": "OK"
},
"Description": "Front Desk"
},
"Date": "20190520",
"Status": {
"Code": "OT",
"Description": "Out For Delivery Today",
"Type": "I"
},
"Time": "085943"
},
{
"ActivityLocation": {
"Address": {
"City": "Oklahoma City",
"CountryCode": "US",
"StateProvinceCode": "OK"
},
"Description": "Front Desk"
},
"Date": "20190520",
"Status": {
"Code": "DS",
"Description": "Destination Scan",
"Type": "I"
},
"Time": "011819"
},
{
"ActivityLocation": {
"Address": {
"City": "Oklahoma City",
"CountryCode": "US",
"StateProvinceCode": "OK"
},
"Description": "Front Desk"
},
"Date": "20190519",
"Status": {
"Code": "AR",
"Description": "Arrival Scan",
"Type": "I"
},
"Time": "235100"
},
{
"ActivityLocation": {
"Address": {
"City": "DFW Airport",
"CountryCode": "US",
"StateProvinceCode": "TX"
},
"Description": "Front Desk"
},
"Date": "20190519",
"Status": {
"Code": "DP",
"Description": "Departure Scan",
"Type": "I"
},
"Time": "195500"
},
{
"ActivityLocation": {
"Address": {
"City": "DFW Airport",
"CountryCode": "US",
"StateProvinceCode": "TX"
},
"Description": "Front Desk"
},
"Date": "20190517",
"Status": {
"Code": "OR",
"Description": "Origin Scan",
"Type": "I"
},
"Time": "192938"
},
{
"ActivityLocation": {
"Address": {
"CountryCode": "US"
},
"Description": "Front Desk"
},
"Date": "20190517",
"Status": {
"Code": "MP",
"Description": "Order Processed: Ready for UPS",
"Type": "M"
},
"Time": "184621"
}
],
"PackageWeight": {
"UnitOfMeasurement": {
"Code": "LBS"
},
"Weight": "2.00"
},
"ReferenceNumber": [
{
"Code": "01",
"Value": "8472745558"
},
{
"Code": "01",
"Value": "5637807:1007379402:BN81-17077A:1"
},
{
"Code": "01",
"Value": "5637807"
}
],
"TrackingNumber": "1ZX0373R0103574417"
},
"PickupDate": "20190517",
"Service": {
"Code": "001",
"Description": "UPS Next Day Air"
},
"ShipmentAddress": [
{
"Address": {
"AddressLine": "S 600 ROYAL LN",
"City": "COPPELL",
"CountryCode": "US",
"PostalCode": "750193827",
"StateProvinceCode": "TX"
},
"Type": {
"Code": "01",
"Description": "Shipper Address"
}
},
{
"Address": {
"City": "OKLAHOMA CITY",
"CountryCode": "US",
"PostalCode": "73128",
"StateProvinceCode": "OK"
},
"Type": {
"Code": "02",
"Description": "ShipTo Address"
}
}
],
"ShipmentWeight": {
"UnitOfMeasurement": {
"Code": "LBS"
},
"Weight": "2.00"
},
"ShipperNumber": "X0373R"
}
}
}
Below is the error I receive:
Traceback (most recent call last):
File "/Users/***/Library/Preferences/PyCharmCE2019.1/scratches/UPS_API.py", line 15, in <module>
DelDate = data['TrackResponse']['Shipment']['Package'][0]['Activity'][0]['Date']
KeyError: 0
You're trying to index "Package" at index 0, but it's an object not an array. So you should be accessing ['Package']['Activity']
just take away the [0] because there is no [1] or [2]
I am trying to parse JIRA webhook comment section properly. Now, following code works well for the first comment:
data = request.json
jira_comment = data['issue']['fields']['comment']['comments'][0].get('body')
However, every time new webhook is fired, for example, new comment is added, I can only receive the first comment in the JSON-message. If I could somehow identify each comment from another or only parse always the last comment of the issue. What would be the best way to do it or is it even possible since JIRA webhook JSON looks broken (multiple 'body' keys).
See JSON data below, to make it cleaner, use this http://jsbeautifier.org/:
{
"timestamp": 1443024903340,
"webhookEvent": "jira:issue_updated",
"user": {
"self": "http://192.168.10.58:8080/rest/api/2/user?username=jhdoe",
"name": "jhdoe",
"key": "jhdoe",
"emailAddress": "john.doe#corp.com",
"avatarUrls": {
"48x48": "http://www.gravatar.com/avatar/f94b6b6c15d155f4d4b3b2c05509ab0a?d=mm&s=48",
"24x24": "http://www.gravatar.com/avatar/f94b6b6c15d155f4d4b3b2c05509ab0a?d=mm&s=24",
"16x16": "http://www.gravatar.com/avatar/f94b6b6c15d155f4d4b3b2c05509ab0a?d=mm&s=16",
"32x32": "http://www.gravatar.com/avatar/f94b6b6c15d155f4d4b3b2c05509ab0a?d=mm&s=32"
},
"displayName": "John Doe",
"active": true,
"timeZone": "Europe/Berlin"
},
"issue": {
"id": "10300",
"self": "http://192.168.10.58:8080/rest/api/2/issue/10300",
"key": "DEMO-6",
"fields": {
"issuetype": {
"self": "http://192.168.10.58:8080/rest/api/2/issuetype/3",
"id": "3",
"description": "A task that needs to be done.",
"iconUrl": "http://192.168.10.58:8080/images/icons/issuetypes/task.png",
"name": "Task",
"subtask": false
},
"components": [],
"timespent": null,
"timeoriginalestimate": null,
"description": "Hey",
"project": {
"self": "http://192.168.10.58:8080/rest/api/2/project/10000",
"id": "10000",
"key": "DEMO",
"name": "DEMO",
"avatarUrls": {
"48x48": "http://192.168.10.58:8080/secure/projectavatar?avatarId=10011",
"24x24": "http://192.168.10.58:8080/secure/projectavatar?size=small&avatarId=10011",
"16x16": "http://192.168.10.58:8080/secure/projectavatar?size=xsmall&avatarId=10011",
"32x32": "http://192.168.10.58:8080/secure/projectavatar?size=medium&avatarId=10011"
}
},
"fixVersions": [],
"aggregatetimespent": null,
"resolution": null,
"timetracking": {},
"customfield_10006": null,
"attachment": [],
"customfield_10009": "0|i0000f:",
"aggregatetimeestimate": null,
"resolutiondate": null,
"workratio": -1,
"summary": "Comment test",
"lastViewed": "2015-09-23T19:14:11.979+0300",
"watches": {
"self": "http://192.168.10.58:8080/rest/api/2/issue/DEMO-6/watchers",
"watchCount": 1,
"isWatching": true
},
"creator": {
"self": "http://192.168.10.58:8080/rest/api/2/user?username=jhdoe",
"name": "jhdoe",
"key": "jhdoe",
"emailAddress": "john.doe#corp.com",
"avatarUrls": {
"48x48": "http://www.gravatar.com/avatar/f94b6b6c15d155f4d4b3b2c05509ab0a?d=mm&s=48",
"24x24": "http://www.gravatar.com/avatar/f94b6b6c15d155f4d4b3b2c05509ab0a?d=mm&s=24",
"16x16": "http://www.gravatar.com/avatar/f94b6b6c15d155f4d4b3b2c05509ab0a?d=mm&s=16",
"32x32": "http://www.gravatar.com/avatar/f94b6b6c15d155f4d4b3b2c05509ab0a?d=mm&s=32"
},
"displayName": "John Doe",
"active": true,
"timeZone": "Europe/Berlin"
},
"subtasks": [],
"created": "2015-09-21T17:39:14.518+0300",
"reporter": {
"self": "http://192.168.10.58:8080/rest/api/2/user?username=jhdoe",
"name": "jhdoe",
"key": "jhdoe",
"emailAddress": "john.doe#corp.com",
"avatarUrls": {
"48x48": "http://www.gravatar.com/avatar/f94b6b6c15d155f4d4b3b2c05509ab0a?d=mm&s=48",
"24x24": "http://www.gravatar.com/avatar/f94b6b6c15d155f4d4b3b2c05509ab0a?d=mm&s=24",
"16x16": "http://www.gravatar.com/avatar/f94b6b6c15d155f4d4b3b2c05509ab0a?d=mm&s=16",
"32x32": "http://www.gravatar.com/avatar/f94b6b6c15d155f4d4b3b2c05509ab0a?d=mm&s=32"
},
"displayName": "John Doe",
"active": true,
"timeZone": "Europe/Berlin"
},
"customfield_10000": null,
"aggregateprogress": {
"progress": 0,
"total": 0
},
"priority": {
"self": "http://192.168.10.58:8080/rest/api/2/priority/2",
"iconUrl": "http://192.168.10.58:8080/images/icons/priorities/high.png",
"name": "High",
"id": "2"
},
"customfield_10001": null,
"customfield_10100": "1234",
"customfield_10200": null,
"labels": [],
"customfield_10004": null,
"environment": null,
"timeestimate": null,
"aggregatetimeoriginalestimate": null,
"versions": [],
"duedate": null,
"progress": {
"progress": 0,
"total": 0
},
"comment": {
"startAt": 0,
"maxResults": 3,
"total": 3,
"comments": [{
"self": "http://192.168.10.58:8080/rest/api/2/issue/10300/comment/10600",
"id": "10600",
"author": {
"self": "http://192.168.10.58:8080/rest/api/2/user?username=jhdoe",
"name": "jhdoe",
"key": "jhdoe",
"emailAddress": "john.doe#corp.com",
"avatarUrls": {
"48x48": "http://www.gravatar.com/avatar/f94b6b6c15d155f4d4b3b2c05509ab0a?d=mm&s=48",
"24x24": "http://www.gravatar.com/avatar/f94b6b6c15d155f4d4b3b2c05509ab0a?d=mm&s=24",
"16x16": "http://www.gravatar.com/avatar/f94b6b6c15d155f4d4b3b2c05509ab0a?d=mm&s=16",
"32x32": "http://www.gravatar.com/avatar/f94b6b6c15d155f4d4b3b2c05509ab0a?d=mm&s=32"
},
"displayName": "John Doe",
"active": true,
"timeZone": "Europe/Berlin"
},
"body": "Comment 1",
"updateAuthor": {
"self": "http://192.168.10.58:8080/rest/api/2/user?username=jhdoe",
"name": "jhdoe",
"key": "jhdoe",
"emailAddress": "john.doe#corp.com",
"avatarUrls": {
"48x48": "http://www.gravatar.com/avatar/f94b6b6c15d155f4d4b3b2c05509ab0a?d=mm&s=48",
"24x24": "http://www.gravatar.com/avatar/f94b6b6c15d155f4d4b3b2c05509ab0a?d=mm&s=24",
"16x16": "http://www.gravatar.com/avatar/f94b6b6c15d155f4d4b3b2c05509ab0a?d=mm&s=16",
"32x32": "http://www.gravatar.com/avatar/f94b6b6c15d155f4d4b3b2c05509ab0a?d=mm&s=32"
},
"displayName": "John Doe",
"active": true,
"timeZone": "Europe/Berlin"
},
"created": "2015-09-23T19:05:26.593+0300",
"updated": "2015-09-23T19:08:07.010+0300"
}, {
"self": "http://192.168.10.58:8080/rest/api/2/issue/10300/comment/10601",
"id": "10601",
"author": {
"self": "http://192.168.10.58:8080/rest/api/2/user?username=jhdoe",
"name": "jhdoe",
"key": "jhdoe",
"emailAddress": "john.doe#corp.com",
"avatarUrls": {
"48x48": "http://www.gravatar.com/avatar/f94b6b6c15d155f4d4b3b2c05509ab0a?d=mm&s=48",
"24x24": "http://www.gravatar.com/avatar/f94b6b6c15d155f4d4b3b2c05509ab0a?d=mm&s=24",
"16x16": "http://www.gravatar.com/avatar/f94b6b6c15d155f4d4b3b2c05509ab0a?d=mm&s=16",
"32x32": "http://www.gravatar.com/avatar/f94b6b6c15d155f4d4b3b2c05509ab0a?d=mm&s=32"
},
"displayName": "John Doe",
"active": true,
"timeZone": "Europe/Berlin"
},
"body": "Comment 2",
"updateAuthor": {
"self": "http://192.168.10.58:8080/rest/api/2/user?username=jhdoe",
"name": "jhdoe",
"key": "jhdoe",
"emailAddress": "john.doe#corp.com",
"avatarUrls": {
"48x48": "http://www.gravatar.com/avatar/f94b6b6c15d155f4d4b3b2c05509ab0a?d=mm&s=48",
"24x24": "http://www.gravatar.com/avatar/f94b6b6c15d155f4d4b3b2c05509ab0a?d=mm&s=24",
"16x16": "http://www.gravatar.com/avatar/f94b6b6c15d155f4d4b3b2c05509ab0a?d=mm&s=16",
"32x32": "http://www.gravatar.com/avatar/f94b6b6c15d155f4d4b3b2c05509ab0a?d=mm&s=32"
},
"displayName": "John Doe",
"active": true,
"timeZone": "Europe/Berlin"
},
"created": "2015-09-23T19:08:13.644+0300",
"updated": "2015-09-23T19:08:13.644+0300"
}, {
"self": "http://192.168.10.58:8080/rest/api/2/issue/10300/comment/10606",
"id": "10606",
"author": {
"self": "http://192.168.10.58:8080/rest/api/2/user?username=jhdoe",
"name": "jhdoe",
"key": "jhdoe",
"emailAddress": "john.doe#corp.com",
"avatarUrls": {
"48x48": "http://www.gravatar.com/avatar/f94b6b6c15d155f4d4b3b2c05509ab0a?d=mm&s=48",
"24x24": "http://www.gravatar.com/avatar/f94b6b6c15d155f4d4b3b2c05509ab0a?d=mm&s=24",
"16x16": "http://www.gravatar.com/avatar/f94b6b6c15d155f4d4b3b2c05509ab0a?d=mm&s=16",
"32x32": "http://www.gravatar.com/avatar/f94b6b6c15d155f4d4b3b2c05509ab0a?d=mm&s=32"
},
"displayName": "John Doe",
"active": true,
"timeZone": "Europe/Berlin"
},
"body": "Comment 3",
"updateAuthor": {
"self": "http://192.168.10.58:8080/rest/api/2/user?username=jhdoe",
"name": "jhdoe",
"key": "jhdoe",
"emailAddress": "john.doe#corp.com",
"avatarUrls": {
"48x48": "http://www.gravatar.com/avatar/f94b6b6c15d155f4d4b3b2c05509ab0a?d=mm&s=48",
"24x24": "http://www.gravatar.com/avatar/f94b6b6c15d155f4d4b3b2c05509ab0a?d=mm&s=24",
"16x16": "http://www.gravatar.com/avatar/f94b6b6c15d155f4d4b3b2c05509ab0a?d=mm&s=16",
"32x32": "http://www.gravatar.com/avatar/f94b6b6c15d155f4d4b3b2c05509ab0a?d=mm&s=32"
},
"displayName": "John Doe",
"active": true,
"timeZone": "Europe/Berlin"
},
"created": "2015-09-23T19:13:24.836+0300",
"updated": "2015-09-23T19:14:04.464+0300"
}]
},
"issuelinks": [],
"votes": {
"self": "http://192.168.10.58:8080/rest/api/2/issue/DEMO-6/votes",
"votes": 0,
"hasVoted": false
},
"worklog": {
"startAt": 0,
"maxResults": 20,
"total": 0,
"worklogs": []
},
"assignee": null,
"updated": "2015-09-23T19:15:03.338+0300",
"status": {
"self": "http://192.168.10.58:8080/rest/api/2/status/10000",
"description": "",
"iconUrl": "http://192.168.10.58:8080/images/icons/subtask.gif",
"name": "To Do",
"id": "10000",
"statusCategory": {
"self": "http://192.168.10.58:8080/rest/api/2/statuscategory/2",
"id": 2,
"key": "new",
"colorName": "blue-gray",
"name": "To Do"
}
}
}
},
"changelog": {
"id": "10611",
"items": [{
"field": "Custom ID",
"fieldtype": "custom",
"from": null,
"fromString": null,
"to": null,
"toString": "1234"
}]
}
}
Well your comment section in the json is a list of dicts so you can always get the last in the list then call .get('body')
jira_comment = data['issue']['fields']['comment']['comments'][-1].get('body')
this works because in python negative numbers in an array index are treated as going backwards in the array. so -1 gives you the last item in the array.
Does your code work when there is no comments at all?
If it doesn't consider checking that the length of the the comments array is greater than 0 or just wrap it in a try except block.
When a comment is added, the request body should contain data['comment'], which will contain the comment that triggered the issue update. You can compare data['comment']['created'] and data['comment']['updated'] to figure if the comment is new or if the comment has been edited.
see https://developer.atlassian.com/jiradev/jira-apis/webhooks
I m trying to access a post on a Facebook Page which has the content in Hindi. So the raw response that I get from Facebook API is as below.
In this response the message is actually in Hindi, how do I detect the encoding of the message and print it in Hindi?
{
"id": "182929845081087_579535732087161",
"from": {
"id": "182929845081087",
"category": "Non-profit organization",
"name": "Brahma Kumaris"
},
"message": "\u092e\u0941\u0930\u0932\u0940 \u0938\u093e\u0930:- \u092e\u0940\u0920\u0947 \u092c\u091a\u094d\u091a\u0947-\u0924\u0941\u092e\u094d\u0939\u0947\u0902 \u0905\u0928\u094d\u0924 \u0924\u0915 \u092f\u0939 \u092e\u0940\u0920\u0940 \u0928\u0949\u0932\u0947\u091c \u0938\u0941\u0928\u0924\u0947 \u0930\u0939\u0928\u093e \u0939\u0948 \u091c\u092c \u0924\u0915 \u091c\u0940\u0928\u093e \u0939\u0948-\u092a\u0922\u093c\u0928\u093e \u0914\u0930 \u092f\u094b\u0917 \u0938\u0940\u0916\u0928\u093e \u0939\u0948 \n \u092a\u094d\u0930\u0936\u094d\u0928:- \u092c\u093e\u092a \u0915\u0947 \u0938\u093e\u0925-\u0938\u093e\u0925 \u0924\u0941\u092e \u092c\u091a\u094d\u091a\u0947 \u0915\u093f\u0938 \u0938\u0947\u0935\u093e \u0915\u0947 \u0928\u093f\u092e\u093f\u0924\u094d\u0924 \u092c\u0928\u0947 \u0939\u0941\u090f \u0939\u094b? \n \u0909\u0924\u094d\u0924\u0930:- \u091c\u0948\u0938\u0947 \u092c\u093e\u092a \u0938\u093e\u0930\u0947 \u0935\u093f\u0936\u094d\u0935 \u0915\u094b \u0932\u093f\u092c\u0930\u0947\u091f \u0915\u0930\u0924\u0947 \u0939\u0948\u0902, \u0938\u092c \u092a\u0930 \u092c\u094d\u0932\u093f\u0938 \u0915\u0930\u0924\u0947 \u0939\u0948\u0902, \u092a\u0940\u0938 \u092e\u0947\u0915\u0930 \u092c\u0928 \u092a\u0940\u0938 \u0938\u094d\u0925\u093e\u092a\u0928 \u0915\u0930\u0924\u0947 \u0939\u0948\u0902 \u0910\u0938\u0947 \u0924\u0941\u092e \u092c\u091a\u094d\u091a\u0947 \u092d\u0940 \u092c\u093e\u092a \u0915\u0947 \u0938\u093e\u0925 \u0907\u0938 \u0938\u0947\u0935\u093e \u0915\u0947 \u0928\u093f\u092e\u093f\u0924\u094d\u0924 \u0939\u094b\u0964 \u0924\u0941\u092e \u0939\u094b \u0938\u0948\u0932\u0935\u0947\u0936\u0928 \u0906\u0930\u094d\u092e\u0940\u0964 \u0924\u0941\u092e\u094d\u0939\u0947\u0902 \u092d\u093e\u0930\u0924 \u0915\u0947 \u0921\u0942\u092c\u0947 \u0939\u0941\u090f \u092c\u0947\u095c\u0947 \u0915\u094b \u0938\u0948\u0932\u0935\u0947\u091c \u0915\u0930\u0928\u093e \u0939\u0948\u0964 21 \u091c\u0928\u094d\u092e\u094b\u0902 \u0915\u0947 \u0932\u093f\u090f \u0938\u092c\u0915\u094b \u0938\u092e\u094d\u092a\u0924\u094d\u0924\u093f\u0935\u093e\u0928 \u092c\u0928\u093e\u0928\u093e \u0939\u0948\u0964 \u0910\u0938\u0940 \u0938\u0947\u0935\u093e \u0924\u0941\u092e \u092c\u091a\u094d\u091a\u094b\u0902 \u0915\u0947 \u0938\u093f\u0935\u093e\u090f \u0914\u0930 \u0915\u094b\u0908 \u0915\u0930 \u0928\u0939\u0940\u0902 \u0938\u0915\u0924\u093e\u0964 \n \u0927\u093e\u0930\u0923\u093e \u0915\u0947 \u0932\u093f\u090f \u092e\u0941\u0916\u094d\u092f \u0938\u093e\u0930:- \n 1) \u0935\u093f\u0915\u0930\u094d\u092e\u093e\u091c\u0940\u0924 \u092c\u0928\u0928\u0947 \u0915\u0947 \u0932\u093f\u090f \u091a\u0932\u0924\u0947 \u092b\u093f\u0930\u0924\u0947 \u092c\u093e\u092a \u0915\u094b \u092f\u093e\u0926 \u0915\u0930\u0928\u0947 \u0915\u093e \u0905\u092d\u094d\u092f\u093e\u0938 \u0915\u0930\u0928\u093e \u0939\u0948\u0964 \u092f\u093e\u0926 \u0915\u093e \u091a\u093e\u0930\u094d\u091f \u091c\u0930\u0942\u0930 \u0930\u0916\u0928\u093e \u0939\u0948\u0964 \n 2) \u0905\u092a\u0928\u0940 \u0939\u0930 \u091a\u0932\u0928 \u0938\u0947 \u092e\u093e\u0924-\u092a\u093f\u0924\u093e \u0914\u0930 \u091f\u0940\u091a\u0930 \u0915\u093e \u0936\u094b \u0915\u0930\u0928\u093e \u0939\u0948\u0964 \u0935\u093f\u0928\u093e\u0936 \u0915\u093e\u0932 \u092e\u0947\u0902 \u092a\u094d\u0930\u0940\u0924 \u092c\u0941\u0926\u094d\u0927\u093f \u092c\u0928\u0915\u0930 \u0930\u0939\u0928\u093e \u0939\u0948\u0964 \u0930\u0942\u0939\u093e\u0928\u0940 \u0938\u0947\u0935\u093e \u0915\u0930\u0928\u0940 \u0939\u0948\u0964 \n \u0935\u0930\u0926\u093e\u0928:- \u0935\u093e\u092f\u0926\u094b\u0902 \u0915\u0940 \u0938\u094d\u092e\u0943\u0924\u093f \u0926\u094d\u0935\u093e\u0930\u093e \u095e\u093e\u092f\u0926\u093e \u0909\u0920\u093e\u0928\u0947 \u0935\u093e\u0932\u0947 \u0938\u0926\u093e \u092c\u093e\u092a \u0915\u0940 \u092c\u094d\u0932\u0948\u0938\u093f\u0902\u0917 \u0915\u0947 \u092a\u093e\u0924\u094d\u0930 \u092d\u0935 \n \u091c\u094b \u092d\u0940 \u0935\u093e\u092f\u0926\u0947 \u092e\u0928 \u0938\u0947, \u092c\u094b\u0932 \u0938\u0947 \u0905\u0925\u0935\u093e \u0932\u093f\u0916\u0915\u0930 \u0915\u0930\u0924\u0947 \u0939\u094b, \u0909\u0928\u094d\u0939\u0947\u0902 \u0938\u094d\u092e\u0943\u0924\u093f \u092e\u0947\u0902 \u0930\u0916\u094b \u0924\u094b \u0935\u093e\u092f\u0926\u0947 \u0915\u093e \u092a\u0942\u0930\u093e \u092b\u093e\u092f\u0926\u093e \u0909\u0920\u093e \u0938\u0915\u0924\u0947 \u0939\u094b\u0964 \u091a\u0947\u0915 \u0915\u0930\u094b \u0915\u093f \u0915\u093f\u0924\u0928\u0947 \u092c\u093e\u0930 \u0935\u093e\u092f\u0926\u093e \u0915\u093f\u092f\u093e \u0939\u0948 \u0914\u0930 \u0915\u093f\u0924\u0928\u093e \u0928\u093f\u092d\u093e\u092f\u093e \u0939\u0948! \u0935\u093e\u092f\u0926\u093e \u0914\u0930 \u095e\u093e\u092f\u0926\u093e - \u0907\u0928 \u0926\u094b\u0928\u094b\u0902 \u0915\u093e \u092c\u0948\u0932\u0947\u0928\u094d\u0938 \u0930\u0939\u0947 \u0924\u094b \u0935\u0930\u0926\u093e\u0924\u093e \u092c\u093e\u092a \u0926\u094d\u0935\u093e\u0930\u093e \u092c\u094d\u0932\u0948\u0938\u093f\u0902\u0917 \u092e\u093f\u0932\u0924\u0940 \u0930\u0939\u0947\u0917\u0940\u0964 \u091c\u0948\u0938\u0947 \u0938\u0902\u0915\u0932\u094d\u092a \u0936\u094d\u0930\u0947\u0937\u094d\u0920 \u0915\u0930\u0924\u0947 \u0939\u094b \u0910\u0938\u0947 \u0915\u0930\u094d\u092e \u092d\u0940 \u0936\u094d\u0930\u0947\u0937\u094d\u0920 \u0939\u094b\u0902 \u0924\u094b \u0938\u092b\u0932\u0924\u093e \u092e\u0942\u0930\u094d\u0924 \u092c\u0928 \u091c\u093e\u092f\u0947\u0902\u0917\u0947\u0964 \n \u0938\u094d\u0932\u094b\u0917\u0928:- \u0938\u094d\u0935\u092f\u0902 \u0915\u094b \u0910\u0938\u093e \u0926\u093f\u0935\u094d\u092f \u0906\u0907\u0928\u093e \u092c\u0928\u093e\u0913 \u091c\u093f\u0938\u092e\u0947\u0902 \u092c\u093e\u092a \u0939\u0940 \u0926\u093f\u0916\u093e\u0908 \u0926\u0947 \u0924\u092c \u0915\u0939\u0947\u0902\u0917\u0947 \u0938\u091a\u094d\u091a\u0940 \u0938\u0947\u0935\u093e\u0964",
"actions": [
{
"name": "Comment",
"link": "http://www.facebook.com/182929845081087/posts/579535732087161"
},
{
"name": "Like",
"link": "http://www.facebook.com/182929845081087/posts/579535732087161"
}
],
"privacy": {
"description": "Public",
"value": "EVERYONE",
"friends": "",
"networks": "",
"allow": "",
"deny": ""
},
"type": "status",
"status_type": "mobile_status_update",
"application": {
"name": "UpdateYou",
"id": "351985104836764"
},
"created_time": "2013-05-30T03:00:08+0000",
"updated_time": "2013-05-30T15:08:42+0000",
"shares": {
"count": 2
},
"likes": {
"data": [
{
"name": "Bhumika Mahant",
"id": "100002238635044"
},
{
"name": "Kumar DrVinay",
"id": "100002736938311"
},
{
"name": "Namrata Trehan Pathria",
"id": "100000281688593"
},
{
"name": "Devesh Sharma",
"id": "100001192346711"
}
],
"count": 37
},
"comments": {
"data": [
{
"id": "579535732087161_6364194",
"from": {
"name": "Namrata Trehan Pathria",
"id": "100000281688593"
},
"message": "Om shanti meet he baba",
"can_remove": true,
"created_time": "2013-05-30T15:08:42+0000",
"like_count": 0,
"user_likes": false
},
{
"id": "579535732087161_6363607",
"from": {
"name": "Cetan Patil",
"id": "100003155153074"
},
"message": "om shanti",
"can_remove": true,
"created_time": "2013-05-30T11:06:27+0000",
"like_count": 0,
"user_likes": false
},
{
"id": "579535732087161_6363549",
"from": {
"name": "Maya Ramchandani",
"id": "100003705148351"
},
"message": "Omshanti",
"can_remove": true,
"created_time": "2013-05-30T10:38:39+0000",
"like_count": 0,
"user_likes": false
},
{
"id": "579535732087161_6363525",
"from": {
"name": "Subhash Bambal",
"id": "100002808519452"
},
"message": "Om Shanti",
"can_remove": true,
"created_time": "2013-05-30T10:29:05+0000",
"like_count": 0,
"user_likes": false
},
{
"id": "579535732087161_6363354",
"from": {
"name": "Poonam Dhanuka",
"id": "100004088191006"
},
"message": "om shanti baba",
"can_remove": true,
"created_time": "2013-05-30T09:12:35+0000",
"like_count": 0,
"user_likes": false
},
{
"id": "579535732087161_6363232",
"from": {
"name": "Hemprakash Pant",
"id": "100004354350224"
},
"message": "Om Shanti Baba",
"can_remove": true,
"created_time": "2013-05-30T07:45:38+0000",
"like_count": 0,
"user_likes": false
},
{
"id": "579535732087161_6362963",
"from": {
"name": "Barun Sharma",
"id": "100005696734282"
},
"message": "om shanti....My baba beloved baba sweet baba.....",
"can_remove": true,
"created_time": "2013-05-30T05:33:26+0000",
"like_count": 0,
"user_likes": false
},
{
"id": "579535732087161_6362770",
"from": {
"name": "Arya Singh",
"id": "100001924554892"
},
"message": "om shanti...................",
"can_remove": true,
"created_time": "2013-05-30T04:41:45+0000",
"like_count": 0,
"user_likes": false
},
{
"id": "579535732087161_6362737",
"from": {
"name": "Khushi Dhurve",
"id": "100001700564503"
},
"message": "Om shanti...gm...mere pyare baapdada...awm...love lots...\u003C3:-):-*:-*",
"can_remove": true,
"created_time": "2013-05-30T04:23:53+0000",
"like_count": 0,
"user_likes": false
},
{
"id": "579535732087161_6362675",
"from": {
"name": "NNibedita Behera",
"id": "100002645048155"
},
"message": "Om shanti baba",
"can_remove": true,
"created_time": "2013-05-30T03:59:49+0000",
"like_count": 0,
"user_likes": false
},
{
"id": "579535732087161_6362654",
"from": {
"name": "Sonali Supe",
"id": "100002108817901"
},
"message": "OM SHANTI MERE PYARE BABA......................",
"can_remove": true,
"created_time": "2013-05-30T03:52:05+0000",
"like_count": 0,
"user_likes": false
},
{
"id": "579535732087161_6362647",
"from": {
"name": "Tejsingh Gurjar",
"id": "100004301563182"
},
"message": "om shanti",
"can_remove": true,
"created_time": "2013-05-30T03:47:46+0000",
"like_count": 0,
"user_likes": false
},
{
"id": "579535732087161_6362620",
"from": {
"name": "Swati Sonar",
"id": "100002927228747"
},
"message": "om shanti",
"can_remove": true,
"created_time": "2013-05-30T03:34:13+0000",
"like_count": 0,
"user_likes": false
},
{
"id": "579535732087161_6362605",
"from": {
"name": "Megha Gulati",
"id": "100004777265970"
},
"message": "gd mrng baba.om shanti",
"can_remove": true,
"created_time": "2013-05-30T03:28:23+0000",
"like_count": 0,
"user_likes": false
},
{
"id": "579535732087161_6362579",
"from": {
"name": "Jay Rathod",
"id": "100005154643627"
},
"message": "OM SHANTI",
"can_remove": true,
"created_time": "2013-05-30T03:14:18+0000",
"like_count": 0,
"user_likes": false
}
],
"paging": {
"cursors": {
"after": "MQ==",
"before": "MTU="
}
}
}
},
Correctly detecting encoding is it impossible. There are libraries that try to guess and work very well but you can't completely trust them. Normally in web environments encodings come in response headers (Content-Encoding), have you checked there?
Then, when you know what encoding is (by guessing or by looking at charset encoding header) you then will have to parse the JSON dictionary and in the proper fields where the strings are encoding (message, for example) you may do message.decode('').
That will return unicode decode string and you may work with that.
Now, it seems to me that you're already getting the response decoded in unicode. The reason why I think that is that in message you get \u092e character which is DEVANAGARI LETTER MA.
So probably you have already decode in Unicode the string and you may processing as you want in python. For instance, do something like: message.encode('utf-8') and there you'll have it encoded in utf-8.
Hope this bring some light!