We Keep Coding

Scraping images with Wikipedia API does not work all the time

I'm trying to scrape the first image of Wikipedia pages of companies. (Quite often it's a logo.)
This works sometimes:
import requests
API_ENDPOINT = "https://en.wikipedia.org/w/api.php"
title = "Microsoft"
# Set parameters for the API request
params = {
"action": "query",
"format": "json",
"formatversion": 2,
"prop": "pageimages",
"piprop": "original",
"titles": title,
}
response = requests.get(API_ENDPOINT, params=params)
data = response.json()
print(data)
But for other companies, let's say Binance or Coinbase, it does not work. I'm not able to figure out why.

I can't see it documented anywhere, but I suspect that pageimages does not include .svg which is the only image file in the Coinbase article. Using images instead works fine:
import requests
API_ENDPOINT = "https://en.wikipedia.org/w/api.php"
title = "Coinbase"
# Set parameters for the API request
params = {
"action": "query",
"format": "json",
"formatversion": 2,
"prop": "images",
"titles": title,
"imlimit":1
}
response = requests.get(API_ENDPOINT, params=params)
data = response.json()
print(data)
Returns:
{'continue': {'imcontinue': '39596725|Commons-logo.svg', 'continue': '||'}, 'query': {'pages': [{'pageid': 39596725, 'ns': 0, 'title': 'Coinbase', 'images': [{'ns': 6, 'title': 'File:Coinbase.svg'}]}]}}

Reading key values a JSON array which is a set in Python

I have the following code
import requests
import json
import sys
credentials_User=sys.argv[1]
credentials_Password=sys.argv[2]
email=sys.argv[3]
def auth_api(login_User,login_Password,):
gooddata_user=login_User
gooddata_password=login_Password
body = json.dumps({
"postUserLogin":{
"login": gooddata_user,
"password": gooddata_password,
"remember":1,
"verify_level":0
}
})
headers = {
'Content-Type': 'application/json',
'Accept': 'application/json'
}
url="https://reports.domain.com/gdc/account/login"
response = requests.request(
"POST",
url,
headers=headers,
data=body
)
sst=response.headers.get('Set-Cookie')
return sst
def query_api(cookie,email):
url="https://reports.domain.com/gdc/account/domains/domain/users?login="+email
body={}
headers={
'Content-Type': 'application/json',
'Accept': 'application/json',
'Cookie': cookie
}
response = requests.request(
"GET",
url,
headers=headers,
data=body
)
jsonContent=[]
jsonContent.append({response.text})
accountSettings=jsonContent[0]
print(accountSettings)
cookie=auth_api(credentials_User,credentials_Password)
profilehash=query_api(cookie,email)
The code itself works and sends a request to the Gooddata API.
The query_api() function returns JSON similar to below
{
"accountSettings": {
"items": [
{
"accountSetting": {
"login": "user#example.com",
"email": "user#example.com",
"firstName": "First Name",
"lastName": "Last Name",
"companyName": "Company Name",
"position": "Data Analyst",
"created": "2020-01-08 15:44:23",
"updated": "2020-01-08 15:44:23",
"timezone": null,
"country": "United States",
"phoneNumber": "(425) 555-1111",
"old_password": "secret$123",
"password": "secret$234",
"verifyPassword": "secret$234",
"authenticationModes": [
"SSO"
],
"ssoProvider": "sso-domain.com",
"language": "en-US",
"ipWhitelist": [
"127.0.0.1"
],
"links": {
"projects": "/gdc/account/profile/{profile_id}/projects",
"self": "/gdc/account/profile/{profile_id}",
"domain": "/gdc/domains/default",
"auditEvents": "/gdc/account/profile/{profile_id}/auditEvents"
},
"effectiveIpWhitelist": "[ 127.0.0.1 ]"
}
}
],
"paging": {
"offset": 20,
"count": 100,
"next": "/gdc/uri?offset=100"
}
}
}
The issue I am having is reading specific keys from this JSON Dict, I can use accountSettings=jsonContent[0] but that just returns the same JSON.
What I want to do is read the value of the project key within links
How would I do this with a dict?
Thanks

Based on your description, uyou have your value inside a list, (not a set. Foergt about set: sets are not used with JSON). Inside your list, you either your content as a single string, which then you'd have to parse with json.loads, or it is simply a well behaved nested data structure already extracted from JSON, but which is inside a single element list. This seems the most likely.
So, you should be able to do:
accountlink = jsonContent[0]["items"][0]["accountSetting"]["login"]
otherwise, if it is encoded as a a json string, you have to parse it first:
import json
accountlink = json.loads(jsonContent[0])["items"][0]["accountSetting"]["login"]
Now, given your question, I'd say your are on a begginer level as a programmer, or a casual user, just using Python to automatize something either way, I'd recommend you do try some exercising before proceeding: it will save you time (a lot of time). I am not trying to bully or mock anything here: this is the best advice I can offer you. Seek for tutorials that play around on the interactive mode, rather than trying entire programs at once that you'd just copy and paste.

Using the below code fixed the issue
jsonContent=json.loads(response.text)
print(type(jsonContent))
test=jsonContent["accountSettings"]["items"][0]
test2=test["accountSetting"]["links"]["self"]
print(test)
print(test2)
I believe this works because for some reason I didn't notice I was using .append for my jsonContent. This resulted in the data type being something other than it should have been.
Thanks to everyone who tried helping me.

How to use asynchronous Matrix Routing API v8 from HERE to get travel time response including traffic

I'm after the travel times between a set of nodes (in both directions) including traffic. In the old version of the API (7.2) I would use the following code in Python to request this:
payload = {
"apiKey": "API_KEY_HERE",
"start0": "-33.795602,151.185366",
"start1": "-33.865103,151.205627",
"destination0": "-33.795602,151.185366",
"destination1": "-33.865103,151.205627",
"mode": "fastest;car;traffic:enabled",
"departure": "2020-10-27T08:00:00+11",
"summaryattributes": "all"
}
base_url = "https://matrix.route.ls.hereapi.com/routing/7.2/calculatematrix.json?"
r = requests.get(base_url, params=payload)
print(r.json())
The new version has fewer examples, and to be honest I'm not so familiar with POSTs and Asynchronous responses.
Why change to the new version? Well, it seems that you could only feed a single set of origin nodes/locations and then a matrix will be computed (in the background) and once ready it can be pulled with a GET request. No specifying start0, start1, ..etc
New try with the version 8:
Steps:
Request matrix (POST)
Poll for status (GET)
Download result when ready (GET)
# 1. Request matrix (POST)
base_url = "https://matrix.router.hereapi.com/v8/matrix?"
params = {"apiKey": "MY_API_KEY_HERE",
"async": "true",
"departureTime": "2020-10-27T08:00:00+11"}
payload = {"origins": [{"lat": -33.759688, "lng": 151.156369}, {"lat": -33.865189, "lng": 151.208162},
{"lat": -33.677066, "lng": 151.302117}],
"regionDefinition": {"type": "autoCircle", "margin": 10000},
"matrixAttributes": ["travelTimes", "distances"]}
headers = {'Content-Type': 'application/json'}
r = requests.post(base_url, params=params, json=payload, headers=headers)
response = r.json()
# pretty print
print(json.dumps(response, indent=4))
This gives an "accepted" status:
// Example response
{
"matrixId": "eba6780c-0379-40f1-abaf-5c62d07dabb4",
"status": "accepted",
"statusUrl": "https://aws-eu-west-1.matrix.router.hereapi.com/v8/matrix/eba6780c-0379-40f1-abaf-5c62d07dabb4/status"
}
Then I use the statusUrl and my apiKey to poll the status. This is where I'm stuck. How should I authenticate? I'm not sure how the authentication should work. The authentication in step 1 worked.
# 2. Poll for status (GET)
time.sleep(3)
statusUrl = response['statusUrl']
params = {'apiKey': 'MY_API_KEY_HERE'}
headers = {'Content-Type': 'application/json'}
r = requests.get(statusUrl, params=params, headers=headers)
response = r.json()
# pretty print
print(json.dumps(response, indent=4))
Where 'MY_API_KEY_HERE' is written I input my apiKey. The response:
{
"error": "Unauthorized",
"error_description": "No credentials found"
}
Evidently, there is an error using the authentication. How should the authentication be used? Would it be possible to show how a successful request for checking the status of a submitted matrix calculation looks like and how the request for downloading such matrix looks like in Python (the next step after polling the status using gzip headers)?
Any pointers would be welcome.

It looks like based on the documentation, when you provide departure time, they no longer take live traffic into account - I'm not sure about historical though. So if you need live traffic taken into account, you will have to remove departure time.
This will calculate travel times including traffic time:
Method URL: https://matrix.router.hereapi.com/v8/matrix?apiKey={{API_KEY}}&async=false
Method: POST
Body:
{
"origins": [
{
"lat": 47.673993,
"lng": -122.356108
},
{
"lat": 47.656910,
"lng": -122.362823
},
{
"lat": 47.648015,
"lng": -122.335674
},
{
"lat": 47.653022,
"lng": -122.312461
},
{
"lat":47.675796,
"lng": -122.311520
}
],
"destinations": [
{
"lat": 47.661438,
"lng": -122.336427
}
],
"regionDefinition": {
"type": "world"
}
}

Getting Converting Value Error While Sending Data With Python Requests Module

Hello I'm trying to send a data with python requests module. I'm getting that "Converting Value Error" when I execute it. I don't know what is this error means. Thanks for any help.
Here is my error:
{'statusCode': 200, 'statusMessage': 'OK', 'result': None, 'errors': [{'field': '', 'message': 'Error converting value "{"price": 64, "stock": 11, "expiration": "2021-08-18 17:41:50.956382+00:00", "product": {"name": "Cerave Hydrating Cleanser Nemlendircili Temizleyici 236 ml", "barcode": "3337875597180"}}" to type \'System.Collections.Generic.List`1[DataTransferObjects.Listings.CreateOutSourcedListingModel]\'. Path \'\', line 1, position 207.'}]}
My code is here:
token = self.get_token()
create_listing_path = "api/v1/listings/createlistings"
create_listing_url = "https://staging.lab.xxx.com/" + create_listing_path
bearer = "Bearer "+str(token)
headers = {'Content-Type': 'application/json', 'Authorization': bearer}
fiyat = kart.urun.satis_fiyat
stok = kart.stok
son_kullanma = kart.expiration
barkod = kart.urun.barkod
urun_adi = kart.urun.urun_adi
data = {'price':int(fiyat),'stock':int(stok), 'expiration':str(son_kullanma), 'product':{'name':urun_adi,'barcode':barkod}}
data_json = json.dumps(data)
r = requests.request("POST", str(create_listing_url), json=data_json, headers=headers)
And here is the information from the documentation at server side:
Request Format:
Url: https://staging.lab.xxx.com/api/v1/listings/createlistings
HTTP METHOD: POST
Request Headers:
Content-Type: application/json
Parameters:
[
{
"price": 0, (Required)
"stock": 0, (Required)
"expiration": "2019-06-21T13:37:40.291Z",
"maxCount": 0,
"description": "string",
"isFeatured": true,
"product": {
"id": 0,
"name": "string", (Required)
"barcode": "string", (Required)
"psf": 0,
"vat": 0,
"image": "string"
}
}
]

That solved my problem:
data = [{'price':int(fiyat),'stock':int(stok),'expiration':str(son_kullanma),'product':{'name':urun_adi,'barcode':barkod}}]
r = requests.request("POST", str(create_listing_url), data=json.dumps(data), headers=headers)
data=data wasn't worked for me. Than I followed that answer:
https://stackoverflow.com/a/34051771/11027652

Python post api request

I need to make a post into an API. I'm working with python. I'm new on this and I can't create an ad tag. I tried with create a dict with the api example information but it didn't work. When I run the >>> sitios_creados, the answer is ''
and when I run sites.status_codeI reveice `415.
I don't understand why because if you see in my code, I did a right post python requests before with the token
I must to take the publisherid and with it and the token id create the ad tag.
my publisherid is: 15663
my code:
import requests
import json
headers = {
'Content-Type': 'application/x-www-form-urlencoded',
'Authorization': 'Basic ',
}
data = [
('grant_type', 'password'),
('username', ''),
('password', ''),
]
response = requests.post('http://api.site.com/v1/oauth/generateOauthToken', headers=headers, data=data)
json_data = json.loads(response.text)
token = json_data['access_token'].encode("utf-8")
headers = {
'Content-Type': 'application/x-www-form-urlencoded',
'Authorization': 'Bearer {}'.format(token)
}
sites = requests.post('http://api.site.com/v1/inventorymgmt/publisherAdTag?entityId=15663', headers=headers, data=data)
sitios_creados = sites.content
Api information example:
URL: http://api.site.com/v1/inventorymgmt/publisherAdTag?entityId=2685
Method: POST
Request Body:
{
"publisherId": 2685,
"publisherSiteurl": "http://example.org",
"adTagName": "THIS_IS_TEST_DEMAND_5",
"adCodeTypeId": 1,
"foldPlacementId": 1,
"adTypeId": 3,
"pagePlacementId": 1,
"adExpansionDirectionId": 1,
"adSize": {
"name": null,
"width": 0,
"height": 0,
"id": 9
},
"adTagPlacements": [{
"adTagPlacementId": 0,
"linkOnlyToGeo": false,
"ecpm": 1,
"adScript": "THIS IS DEMO SCRIPT",
"currency": 1
}],
"adTagCustomParamMap": [{
"name": "kadcarrier",
"macroValue": "techno.carrier"
}, {
"name": "kadcity",
"macroValue": "geo.city"
}]
}

Is it an API for a website?
When yes, you can do a network analysis in the developer tool of your browser and copy the curl command of the POST package.
Then you surf to curl.trillworks.com and convert the curl command it into a Python POST request.
Inside of your python request you can modify the values.