I'm using the Google Pagespeed API v4 to extract Pagespeed scores for a large number of URLs. I want to test for both desktop and mobile with the argument strategy within my Pagespeed call. This is possible in the general API call, but I can't find a way to pass these arguments in the batch call.
Minimal working example:
from apiclient.discovery import build
import requests
#define Google API key and API call
google_api_key = "MyKey"
#build service object to call PageSpeed, version 4, with MyKey
ps_service = build('pagespeedonline', 'v4', developerKey = google_api_key)
list_of_urls = [a list of several URLs]
#define function; make list of URLs with column for PageSpeed score
def list_websites(request_id, response, exception):
if exception is not None:
print("This is an exception: " + str(exception) + " " + request_id)
else:
score = response['ruleGroups']['SPEED']['score']
print(score)
#create URL batch
ps_batch = ps_service.new_batch_http_request(callback = list_websites)
service_list = []
for url in list_of_urls:
service_list.append(ps_service.pagespeedapi().runpagespeed(url = url))
for req in service_list:
ps_batch.add(req)
#execute API call by batch
ps_batch.execute()
Although this method works fine for doing batch requests to the API, by default it calculates a speed score based on a desktop view of the said URLs (the analysis strategy is set to desktop), whereas I would also like to receive a score for a batch request in which strategy is set to mobile, to get a score based on the mobile view of the URLs.
My question is, how do I add an extra argument to the build() function in which I can make a distinction between mobile and desktop?
Like this:
for url in list_of_urls:
service_list.append(ps_service.pagespeedapi().runpagespeed(url = url, strategy='mobile'))
Every parameter, used by google can be sent that way
Related
I'm writing a python script to generate how many changes was made within a timeframe for all projects, but when I use the Gerrit REST Api I can only get up to maximum of 500 unique users, and I want to see all of them, even if I take long timeframe (1 year Gerrit picture). This is my function for the API
def requestAPICall(url):
"""
does API stuff
"""
response = requests.get(url)
if response.status_code == 200:
JSON_response = json.loads(response.text[4:])
generateJSON(JSON_response)
return (JSON_response, True)
print("Error Occured")
return (response, False)
This is the link I used for the request in this case
https://chromium-review.googlesource.com/changes/?q=since:%222022-01-01%2011:26:25%20%2B0100%22+before:%222023-01-01%2011:31:25%20%2B0100%22
I have tried curl commands but I do not know if that works
There is a default limit on the number of returned items, and if you're making anonymous queries I don't believe you can change this. From the documentation:
The query string must be provided by the q parameter. The n parameter can be used to limit the returned results. The no-limit parameter can be used remove the default limit on queries and return all results (does not apply to anonymous requests). This might not be supported by all index backends.
However, you can return paginated resulted using the start parameter:
If the number of changes matching the query exceeds either the internal limit or a supplied n query parameter, the last change object has a _more_changes: true JSON field set.
The S or start query parameter can be supplied to skip a number of changes from the list.
So if the final result sets _more_changes: true, you can make a subsequent request using the start parameter.
That means your Python code is going to look something like:
import json
import requests
import sys
class Gerrit:
"""Wrap up Gerrit API functionality in a simple class to make
it easier to consume from our code. This limited example only
supports the `changes` endpoint.
See https://gerrit-review.googlesource.com/Documentation/rest-api.html
for complete REST API documentation.
"""
def __init__(self, baseurl):
self.baseurl = baseurl
def changes(self, query, start=None, limit=None, options=None):
"""This implements the API described in [1].
[1]: https://gerrit-review.googlesource.com/Documentation/rest-api-changes.html
"""
params = {"q": query}
if start is not None:
params["S"] = start
if limit is not None:
params["n"] = limit
if options is not None:
params["o"] = options
res = requests.get(f"{self.baseurl}/changes", params=params)
print(f"fetched [{res.status_code}]: {res.url}", file=sys.stderr)
res.raise_for_status()
return json.loads(res.text[4:])
# And here is an example in which we use the Gerrit class to perform a
# query against https://chromium-review.googlesource.com. This is similar
# to the query in your question, but using a constrained date range in order
# to limit the total number of results.
g = Gerrit("https://chromium-review.googlesource.com")
all_results = []
start = 0
while True:
res = g.changes(
'since:"2022-12-31 00:00:00" before:"2023-01-01 00:00:00"',
limit=200,
start=start,
)
if not res:
break
all_results.extend(res)
if not res[-1].get("_more_changes"):
break
start += len(res)
# Here we're just dumping all the results as a JSON document on
# stdout.
print(json.dumps(all_results))
This demonstrates how to use limit to control the number of queries returned in a "page", and the start parameter to request additional pages of results.
But look out! The example query here includes only a couple days and returns over 3000 results; I suspect that any attempt to fetch a year's worth of data, particularly with an anonymous connection, are going to run into some sort of server rate limits.
I'm fairly new to Python and I'm trying to migrate subscriptions from an older youtube account to a newer one that I'll use going forward. I pulled my subscriptions export from the old one and have around 470+ subs that I'll need to migrate over.
I found this article which absolutely works with automatically subscribing to a youtube channel via their channel_id but it seems like in the key value pair I can only run the .py script once per value.
I tried all sorts of googling to see how I can include multiple values in the key (channelId) but it always only auto subs to the last one in the dictionary.
Can someone please help show me what I'm missing? I feel like there has to be a way to add multiple channelId values in there key dictionary, right?!
Here's what my code looks like > screenshot
import os
import google.oauth2.credentials
import google_auth_oauthlib.flow
from googleapiclient.discovery import build
from googleapiclient.errors import HttpError
from google_auth_oauthlib.flow import InstalledAppFlow
# The CLIENT_SECRETS_FILE variable specifies
# the name of a file that contains
# client_id and client_secret.
CLIENT_SECRETS_FILE = "client_secret.json"
# This scope allows for full read/write access
# to the authenticated user's account and
# requires requests to use an SSL connection.
SCOPES = ['https://www.googleapis.com/auth/youtube.force-ssl']
API_SERVICE_NAME = 'youtube'
API_VERSION = 'v3'
def get_authenticated_service():
flow = InstalledAppFlow.from_client_secrets_file(CLIENT_SECRETS_FILE, SCOPES)
credentials = flow.run_console()
return build(API_SERVICE_NAME, API_VERSION, credentials = credentials)
def print_response(response):
print(response)
# Build a resource based on a list of
# properties given as key-value pairs.
# Leave properties with empty values out
# of the inserted resource.
def build_resource(properties):
resource = {}
for p in properties:
# Given a key like "snippet.title", split into
# "snippet" and "title", where "snippet" will be
# an object and "title" will be a property in that object.
prop_array = p.split('.')
ref = resource
for pa in range(0, len(prop_array)):
is_array = False
key = prop_array[pa]
# For properties that have array values, convert a name like
# "snippet.tags[]" to snippet.tags, and set a flag to handle
# the value as an array.
if key[-2:] == '[]':
key = key[0:len(key)-2:]
is_array = True
if pa == (len(prop_array) - 1):
# Leave properties without values
# out of inserted resource.
if properties[p]:
if is_array:
ref[key] = properties[p].split(', ')
else:
ref[key] = properties[p]
elif key not in ref:
# For example, the property is "snippet.title",
# but the resource does not yet have a "snippet"
# object. Create the snippet object here.
# Setting "ref = ref[key]" means that in the
# next time through the "for pa in range ..." loop,
# we will be setting a property in the
# resource's "snippet" object.
ref[key] = {}
ref = ref[key]
else:
# For example, the property is "snippet.description",
# and the resource already has a "snippet" object.
ref = ref[key]
return resource
# Remove keyword arguments that are not set
def remove_empty_kwargs(**kwargs):
good_kwargs = {}
if kwargs is not None:
for key, value in kwargs.items():
if value:
good_kwargs[key] = value
return good_kwargs
def subscriptions_insert(client, properties, **kwargs):
resource = build_resource(properties)
kwargs = remove_empty_kwargs(**kwargs)
response = client.subscriptions().insert(
body = resource,**kwargs).execute()
return print_response(response)
if __name__ == '__main__':
# When running locally, disable OAuthlib's
# HTTPs verification. When running in production
# * do not * leave this option enabled.
os.environ['OAUTHLIB_INSECURE_TRANSPORT'] = '1'
client = get_authenticated_service()
subscriptions_insert(client,
{'snippet.resourceId.kind': 'youtube# channel',
'snippet.resourceId.channelId': 'UC09fL42MpkktKZWmWxYiDhw', 'UC0Q7Hlz75NYhYAuq6O0fqHw'},
part ='snippet')```
According to YouTube Data API v3 documentation (Subscriptions: insert endpoint and Subscriptions resource), it seems that you can only subscribe a channel at a time. As you have by default 10,000 of quota per day, except if you request extended quota, because Subscriptions: insert costs 50 of quota, then for 470+ subscriptions, you would need 3 days to proceed.
Otherwise you can proceed as follows, it seems that the first time I tried with ~500 channels I have been subscribed to ~290 of them but now I mostly only receive (when removing -H 'Accept-Encoding: gzip, deflate, br' from the cURL request):
{
"error": {
"code": 429,
"message": "Resource has been exhausted (e.g. check quota).",
"errors": [
{
"message": "Resource has been exhausted (e.g. check quota).",
"domain": "global",
"reason": "rateLimitExceeded"
}
],
"status": "RESOURCE_EXHAUSTED"
}
}
So it's an unsure method that you can try to deepen.
Ever wondered how to do that in a single request without using any quota?
Go on an ad hoc YouTube channel YOUR_CHANNEL that you want to subscribed to: https://www.youtube.com/channel/YOUR_CHANNEL_ID
Open the Network tab of your web-browser by using Ctrl + Shift + E (on Firefox) and filter XHR requests.
Now click on Subscribe.
You should see a request to subscribe, copy it as cURL (by right-clicking).
Change at the end
"channelIds":["YOUR_CHANNEL_ID"]
to:
"channelIds":["YOUR_CHANNEL_ID_0, YOUR_CHANNEL_ID_1, ..., YOUR_CHANNEL_ID_499"]
Where YOUR_CHANNEL_ID_0 is your YOUR_CHANNEL_ID and YOUR_CHANNEL_ID_1 the second channel you want to subscribe to and so forth.
Execute the modified cURL request in a terminal and that's it!
Note that this webpage contains a subscriptions count and this one contains all your subscriptions.
To get more than 249 different channels, I used:
import requests, json
channelIds = set()
pageToken = ''
API_KEY = 'AIzaSy...'
i = 0
while len(channelIds) < 250:
url = f'https://www.googleapis.com/youtube/v3/search?q={i}&type=channel&maxResults=50&key={API_KEY}'
if pageToken != '':
url += f"&pageToken={pageToken}"
content = requests.get(url).text
data = json.loads(content)
for item in data['items']:
channelIds.add(item['id']['channelId'])
print(len(channelIds))
if 'nextPageToken' in data:
pageToken = data['nextPageToken']
else:
break
i += 1
print('["' + '","'.join(channelIds) + '"]', len(channelIds))
As #Benjamin Loison has mentioned, there is a quota on the limit on the usage of the API. If you'd like to raise the limit, I think there is a form you can fill out to request more. However, I don't recommend you do so since the form is applicable mainly for a large application that will be used for a long time and involve a long process of human inspection on what you're trying to build (This is based on my personal experience, might not be entirely accurate).
My suggestion would be to use the script you have to print out a list of channel links, and you can click into each of them and press the subscribe button. 470-ish channels should not take you a long time.
I'm using Grafana to show some data. I have 20 organisations and they all use the same dashboards (Dash1, Dash2, Dash3). Thus I can use the same json data for all dashboards in all organisations.
But I do not want to update it manually every time when I change something, thus I'm trying to create a python script which will do it for me.
I run the script as follows:
python update_dashboards.py Dash1
The python code is as follows:
try:
dashboard_name = sys.argv[1]
response = settings.get_request_with_token(settings.api_url + "search?query=" + dashboard_name)
dashboard = json.loads(response)
if len(dashboard) < 1:
print("There is no dashboard data.")
else:
dashboard_data = dashboard[0]
dashboard_uri = str(dashboard_data["uri"])
dashboard_data = dashboard_api.get_dashboard(dashboard_uri)
// Here I get dashboard details
except IndexError:
print("Please provide dashboard name!")
Thus I give the name of the dashboard that I want to copy to all organisations.
With my code I get successfully the dashboard that I want to copy.
My question is how can I now use this dashboard to copy it to all other organizations?
I'd think to use the API to create new dashboards. You can use requests and create a payload with what you already have in your script. You also need a key to use the API.
payload = {"authorization": "yourkey", "content": thedashboard}
url = yougrafanaaddress/api/dashboards/db
r = requests.post(url, data=payload)
I am trying to use Etsy API to add a new listing on my store. In the documents section it says (below section how to do it). First fyi I have never used HTTP Method before so I am not sure how to setup the code so that it adds a new item.
(Link to the Etsy API page https://www.etsy.com/developers/documentation/reference/listing).
Method Name createListing
Synopsis Creates a new Listing.
HTTP Method POST
URI /listings
Parameters
Name Required Default Type
quantity Y int
title Y string
description Y text
price Y float
materials N array(string)
shipping_template_id N int
shop_section_id N int
image_ids N array(int)
is_customizable N boolean
non_taxable N boolean
image N image
state N active enum(active, draft)
processing_min N int
processing_max N int
category_id N int
taxonomy_id N int
tags N array(string)
who_made Y enum(i_did, collective, someone_else)
is_supply Y boolean
when_made Y enum(made_to_order, 2010_2017, 2000_2009, 1998_1999, before_1998, 1990_1997, 1980s, 1970s, 1960s, 1950s, 1940s, 1930s, 1920s, 1910s, 1900s, 1800s, 1700s, before_1700)
recipient N enum(men, women, unisex_adults, teen_boys, teen_girls, teens, boys, girls, children, baby_boys, baby_girls, babies, birds, cats, dogs, pets, not_specified)
occasion N enum(anniversary, baptism, bar_or_bat_mitzvah, birthday, canada_day, chinese_new_year, cinco_de_mayo, confirmation, christmas, day_of_the_dead, easter, eid, engagement, fathers_day, get_well, graduation, halloween, hanukkah, housewarming, kwanzaa, prom, july_4th, mothers_day, new_baby, new_years, quinceanera, retirement, st_patricks_day, sweet_16, sympathy, thanksgiving, valentines, wedding)
style N array(string)
Requires OAuth Y
Permission Scope listings_w
Notes
A shipping_template_id is required when creating a listing.
All listings created on www.etsy.com must be actual items for sale. Please see our guidelines for testingwith live listings.
Creating a listing creates a single inventory products with the supplied price and quantity. Use updateInventory to add more products.
The code I have right know looks like this
import urllib
import requests
url = 'https://openapi.etsy.com/v2/listings/active?api_key={YOUR KEY HERE)' # I put my API key here
r = requests.get(url)
payload = {'quantity': '1', 'title': 'testdfsdfdfs0','description': 'dfsdfsdfsdfdsf','price': '2.55','who_made': 'i_did','is_supply': '0','when_made': '2010_2017'}
rrr = requests.post(url,payload)
print rrr # I get an error 404
How can I add an item for sale on Etsy through Python HTTP method?
Update
from requests_oauthlib import OAuth1Session
import requests
from requests_oauthlib import OAuth1
import json
tempory_token_url = []
oauth_response_bucket = []
client_key = '.......'
client_secret = '......'
oauth = OAuth1Session(client_key, client_secret=client_secret)
request_token_url = 'https://openapi.etsy.com/v2/oauth/request_token?scope=email_r%20listings_r'
fetch_response = oauth.fetch_request_token(request_token_url)
resource_owner_key = fetch_response.get('oauth_token') # Have it
resource_owner_secret = fetch_response.get('oauth_token_secret')
oauth_url_temp = tempory_token_url[0]['login_urI']
base_authorization_url = oauth_url_temp
authorization_url = oauth.authorization_url(base_authorization_url)
redirect_response = raw_input('Paste the full redirect URL here: ')
oauth_response = oauth.parse_authorization_response(redirect_response)
verifier = oauth_response.get('oauth_verifier')
access_token_url = redeirect_response
oauth = OAuth1Session(client_key=client_secret=client_secret,resource_owner_key=resource_owner_key,resource_owner_secret=resource_owner_secret,verifier=verifier)
oauth_tokens = oauth.fetch_access_token(access_token_url)
resource_owner_key = oauth_tokens.get('oauth_token')
resource_owner_secret = oauth_tokens.get('oauth_token_secret')
Any ideas how to make this work? There is very little info regarding Etsy API and most of the stuff is in PHP which I have no clue how to work.
Image Uploading API
Everything looks the same like above this time I just changed the payload but I am getting a 403 Error. I am not sure what is causing it. My best guess would be something with oauth1.0 i think on their website it says you need oauth 1.1.
Here is how I set it up but I am getting 403 error:
url = 'https://openapi.etsy.com/v2/listings'
payload = {'listing_id':'342434342', 'image': ("test1.jpg", open('C:\\Users\\abc\\test1.jpg'),'image/jpeg'),'type':'image/jpeg'}
result = etsy.put(url, params=payload)
print result
Comment: ... at this point I am lost I have no idea where to put the pin# that etsy gave me
etsy oauth#reference
The token credentials you receive for a account do not expire,
and can be used over and over again to make authenticated API requests.
You should keep the token secret in a secure location and never send it as a plaintext parameter
(it's only used for signing your requests, and never needs to be sent in an API request on its own.)
You will not need to step through the OAuth authorization again,
unless you decides to revoke access, or unless you add features that require additional permission scopes.
Note: Didn't find a equivalent Replacement for PHP OAUTH_AUTH_TYPE_URI.
OAuth1Session Defaults to signature_type=u'AUTH_HEADER', so this could be wrong.
If this fails, you could try:
from oauthlib.oauth1 import SIGNATURE_TYPE_QUERY, SIGNATURE_TYPE_BODY
OAuth1Session(..., signature_type=SIGNATURE_TYPE_QUERY)
Create etsy OAuth1Session to reuse for Requests:
etsy = OAuth1Session(client_key,
client_secret=client_secret,
resource_owner_key=resource_owner_key,
resource_owner_secret=resource_owner_secret)
etsy Making an Authorized Request to the API:
response = etsy.get("https://openapi.etsy.com/v2/users/__SELF__")
user_data = json.loads(response.body_as_unicode())
etsy Checking Permission Scopes After Authentication:
response = etsy.get("https://openapi.etsy.com/v2/oauth/scopes")
meta = json.loads(response.body_as_unicode())
etsy Creates a new Listing
url = 'https://openapi.etsy.com/v2/listings'
payload = {'quantity': '1', 'title':...}
result = etsy.post(url, params=payload)
Comment: for api key do I need to import oauth2
According to Reference, Yes.
For write access and for accessing private user data, an OAuth access
token is required. Your application key is required to start the OAuth
authentication process.
Requires OAuth Y
Also your url should end with
URI /listings
url = 'https://openapi.etsy.com/v2/listings'
Your url should only up to the Question mark, for example:
url = 'https://openapi.etsy.com/v2/listings/active'
payload = {'api_key':YOUR KEY HERE, 'quantity': '1', ...
rrr = requests.post(url, params=payload)
Requests Quickstart: Passing Parameters In URLs
You often want to send some sort of data in the URL's query string.
If you were constructing the URL by hand,
this data would be given as key/value pairs in the URL after a question mark, e.g. \http://bin.org/get?key=val.
Requests allows you to provide these arguments as a dictionary of strings, using the params keyword argument.
Question: I am trying to upload a picture ... getting a 403 error
Your url Endpoint and payload isn't correct.
url = 'https://openapi.etsy.com/v2/listings'
payload = {'listing_id':'342434342', 'image': ("test1.jpg", open('C:\\Users\\abc\\test1.jpg'),'image/jpeg'),'type':'image/jpeg'}
Steps to do a etsy Request(uploadListingImage):
Read the Reference for your Method Name
Method Name uploadListingImage
HTTP Method POST
URI /listings/:listing_id/images
Parameters Name Required Default Type
listing_id Y int
listing_image_id N int
image N imagefile
...
Requires OAuth Y
Respect Supported Sizes Working with Images
Note: For me, it's unclear what the image Parameter is for.
And as it's NOT required makes no sense.
I assume its a Placeholder for the Parameter at Point 4 below: {'image':...
Build the URI
uri = 'https://openapi.etsy.com/v2/listings/342434342/images'
Create the Params Dict according to the above Reference
I recommend to use a listing_image_id, as this seems the only way to delete a Image afterwards.
params = {'listing_id':'342434342', 'listing_image_id': 1}
Create Multipart-Encoded File Dict
Image uploads can be performed using a POST request with the Content-Type: multipart/form-dataheader, following RFC1867
# PHP example from Reference:
# $params = array('#image' => '#'.$source_file.';type='.$mimetype);
files = {'image': ("test1.jpg", open('C:\\Users\\abc\\test1.jpg', 'rb'), 'image/jpeg')}
Do the Request, according the Reference, you have to use OAuth and POST
result = etsy.post(uri, params=params, files=files)
Please Comment if this is working for you or why not.
I am using a Python (2.7) script to download via API Google Search Console data. I would like to get rid of the property and dates arguments when launching the script:
>python script. py ´http://www.example.com´ ´01-01-2000´ ´01-02-2000´
For the latter I managed to do it importing timedelta and commenting out the lines referring to that argument:
argparser = argparse.ArgumentParser(add_help=False)
argparser.add_argument('property_uri', type=str,
help=('Site or app URI to query data for (including '
'trailing slash).'))
# Start and end dates are commented out as timeframe is dynamically set
'''argparser.add_argument('start_date', type=str,
help=('Start date of the requested date range in '
'YYYY-MM-DD format.'))
argparser.add_argument('end_date', type=str,
help=('End date of the requested date range in '
'YYYY-MM-DD format.'))'''
now = datetime.datetime.now()
StartDate = datetime.datetime.now()- timedelta(days=14)
EndDate = datetime.datetime.now()- timedelta(days=7)
From = StartDate.strftime('%Y-%m-%d' )
To = EndDate.strftime('%Y-%m-%d' )
request = {
'startDate': StartDate.strftime('%Y-%m-%d' ),
'endDate': EndDate.strftime('%Y-%m-%d' ),
'dimensions': ['query'],
Now I would like get rid also of the property argument, so that I can simply launch the script and have the property specified in the script itself. My final goal is to get data from several properties using only one script.
I tried to repeat the same procedure used for the dates but no luck. Needless to say I am a total beginner at coding.
I think I can help as I had the same problem when working from the sample script given by google as guidance. Which is what I think you gotten your code from?
The problem is that the script uses the sample_tools.py script in the googleapiclient library, which is meant to abstract away all the authentication bits so you can make a quick query easily. If you want to modify the code, I would recommend writing it from scratch.
These are my functions that I've pieced together from various bits of documentation that you might find useful.
Stage 1: Authentication
def authenticate_http():
"""Executes a searchAnalytics.query request.
Args:
service: The webmasters service to use when executing the query.
property_uri: The site or app URI to request data for.
request: The request to be executed.
Returns:
An array of response rows.
"""
# create flow object
flow = flow_from_clientsecrets('path to client_secrets.json',
scope='https://www.googleapis.com/auth/webmasters.readonly',
redirect_uri='urn:ietf:wg:oauth:2.0:oob')
storage = Storage('credentials_file')
credentials = storage.get()
if credentials:
# print "have auth code"
http_auth = credentials.authorize(Http())
else:
print "need auth code"
# get authorization server uri
auth_uri = flow.step1_get_authorize_url()
print auth_uri
# get credentials object
code_input = raw_input("Code: ")
credentials = flow.step2_exchange(code_input)
storage.put(credentials)
# apply credential headers to all requests
http_auth = credentials.authorize(Http())
return http_auth
Stage 2: Build the Service Object
def build_service(api_name, version):
# use authenticate_http to return the http object
http_auth = authenticate_http()
# build gsc service object
service = build(api_name, version, http=http_auth)
return service
Stage 3: Execute Request
def execute_request(service, property_uri, request):
"""Executes a searchAnalytics.query request.
Args:
service: The webmasters service to use when executing the query.
property_uri: The site or app URI to request data for.
request: The request to be executed.
Returns:
An array of response rows.
"""
return service.searchanalytics().query(
siteUrl=property_uri, body=request).execute()
Stage 4: Main()
def main():
# define service object for the api service you want to use
gsc_service = build_service('webmasters', 'v3')
# define request
request = {'request goes here'}
# set your property set string you want to query
url = 'url or property set string goes here'
# response from executing request
response = execute_request(gsc_service, url, request)
print response
For multiple property sets you can just create a list of property sets, then create a loop and pass each property set into the 'url' argument of the 'execute_request' function
Hope this helps!