We Keep Coding

Several error working with blogger/google API using python and service account auth

I'm trying to access blogger API using the official Google python library and a service account with a JSON key file made at the Google Developer console.
I'm using Linux Debian.
google-api-python-client=2.65.0
The thing is that I'm getting this traceback:
Traceback (most recent call last):
File "/path/to/myproject/lib/python3.9/site-packages/googleapiclient/discovery.py", line 287, in build
content = _retrieve_discovery_doc(
File "/path/to/myproject/lib/python3.9/site-packages/googleapiclient/discovery.py", line 422, in _retrieve_discovery_doc
resp, content = req.execute(num_retries=num_retries)
File "/path/to/myproject/lib/python3.9/site-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
return wrapped(*args, **kwargs)
File "/path/to/myproject/lib/python3.9/site-packages/googleapiclient/http.py", line 923, in execute
resp, content = _retry_request(
File "/path/to/myproject/lib/python3.9/site-packages/googleapiclient/http.py", line 191, in _retry_request
resp, content = http.request(uri, method, *args, **kwargs)
File "/path/to/myproject/lib/python3.9/site-packages/httplib2/__init__.py", line 1322, in request
(response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
File "/path/to/myproject/lib/python3.9/site-packages/httplib2/__init__.py", line 1072, in _request
(response, content) = self._conn_request(conn, request_uri, method, body, headers)
File "/path/to/myproject/lib/python3.9/site-packages/httplib2/__init__.py", line 996, in _conn_request
conn.request(method, request_uri, body, headers)
File "/usr/lib/python3.9/http/client.py", line 1279, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/lib/python3.9/http/client.py", line 1325, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/lib/python3.9/http/client.py", line 1274, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/lib/python3.9/http/client.py", line 1034, in _send_output
self.send(msg)
File "/usr/lib/python3.9/http/client.py", line 995, in send
self.sock.sendall(data)
OSError: [Errno 9] Bad file descriptor
Which point to some sort of error at the OS when opening some file, but I can't imagine which is the file that is causing trouble. The credentials seem like are doing all right when recovered from the json file and at the moment of the exception is just sending this:
b'GET /discovery/v1/apis/blogger/v3/rest HTTP/1.1\r\nHost: www.googleapis.com\r\ncontent-length: 0\r\nuser-agent: Python-httplib2/0.10.3 (gzip)\r\naccept-encoding: gzip, deflate\r\n\r\n'
through the internet. This also seems to do fine from my browser.
The code is here below:
import json
import google.auth
from google.oauth2 import service_account
import googleapiclient.discovery
from oauth2client import client
from oauth2client.service_account import ServiceAccountCredentials
SCOPES = ['https://www.googleapis.com/auth/blogger']
def get_key_dict():
SERVICE_ACCOUNT_FILE = '/path/to/myproject/apis/blogger/private_secrets.json'
with open(SERVICE_ACCOUNT_FILE, 'r') as keyfile:
return json.load(keyfile)
key_dict = get_key_dict()
credentials = ServiceAccountCredentials.from_json_keyfile_dict(key_dict, scopes=SCOPES)
# Here it breaks!
service = googleapiclient.discovery.build('blogger', 'v3', credentials=credentials, static_discovery=False,)
try:
users = service.users()
# Retrieve this user's profile information
thisuser = users.get(userId="self").execute()
print("This user's display name is: %s" % thisuser["displayName"])
blogs = service.blogs()
# Retrieve the list of Blogs this user has write privileges on
thisusersblogs = blogs.listByUser(userId="self").execute()
for blog in thisusersblogs["items"]:
print("The blog named '%s' is at: %s" % (blog["name"], blog["url"]))
posts = service.posts()
# List the posts for each blog this user has
for blog in thisusersblogs["items"]:
print("The posts for %s:" % blog["name"])
request = posts.list(blogId=blog["id"])
while request != None:
posts_doc = request.execute()
if "items" in posts_doc and not (posts_doc["items"] is None):
for post in posts_doc["items"]:
print(" %s (%s)" % (post["title"], post["url"]))
request = posts.list_next(request, posts_doc)
except client.AccessTokenRefreshError:
print(
"The credentials have been revoked or expired, please re-run"
"the application to re-authorize"
)

if you check the documentation Authorizing requests and identifying your application you will find that it states that you can use Oauth2 for authorizing a user to the blogger api and api keys. There is no mention of using service account authorization with this api.
To my knowledge as i tried about five years ago this api does not support service account authentication. You will need to use Oauth2.

How should I configure my headers to make an HTTP/2 POST to APNs to avoid "Received duplicate pseudo-header field" error?

I'm pretty new to HTTP stuff, primarily stick to iOS so please bear with me.
I'm using the httpx python library to try and send a notification to an iPhone because I have to make an HTTP/2 POST to do so. Apple's Documentation says it requires ":method" and ":path" headers but I when I try to make the POST with these headers included,
headers = {
':method' : 'POST',
':path' : '/3/device/{}'.format(deviceToken),
...
}
I get the error
h2.exceptions.ProtocolError: Received duplicate pseudo-header field b':path
It's pretty apparent there's a problem with having the ":path" header included but I'm also required to send it so I'm not sure what I'm doing wrong. Apple's Documentation also says to
Encode the :path and authorization values as literal header fields without indexing.
Encode all other fields as literal header fields with incremental indexing.
I really don't know what that means or how to implement that or if it's related. I would think httpx would merge my ":path" header with the default one to eliminate the duplicate but I'm just spitballing here.
Full Traceback
File "/Users/User/.pyenv/versions/3.9.0/lib/python3.9/site-packages/httpx/_client.py", line 992, in post
return self.request(
File "/Users/User/.pyenv/versions/3.9.0/lib/python3.9/site-packages/httpx/_client.py", line 733, in request
return self.send(
File "/Users/User/.pyenv/versions/3.9.0/lib/python3.9/site-packages/httpx/_client.py", line 767, in send
response = self._send_handling_auth(
File "/Users/User/.pyenv/versions/3.9.0/lib/python3.9/site-packages/httpx/_client.py", line 805, in _send_handling_auth
response = self._send_handling_redirects(
File "/Users/User/.pyenv/versions/3.9.0/lib/python3.9/site-packages/httpx/_client.py", line 837, in _send_handling_redirects
response = self._send_single_request(request, timeout)
File "/Users/User/.pyenv/versions/3.9.0/lib/python3.9/site-packages/httpx/_client.py", line 861, in _send_single_request
(status_code, headers, stream, ext) = transport.request(
File "/Users/User/.pyenv/versions/3.9.0/lib/python3.9/site-packages/httpcore/_sync/connection_pool.py", line 218, in request
response = connection.request(
File "/Users/User/.pyenv/versions/3.9.0/lib/python3.9/site-packages/httpcore/_sync/connection.py", line 106, in request
return self.connection.request(method, url, headers, stream, ext)
File "/Users/User/.pyenv/versions/3.9.0/lib/python3.9/site-packages/httpcore/_sync/http2.py", line 119, in request
return h2_stream.request(method, url, headers, stream, ext)
File "/Users/User/.pyenv/versions/3.9.0/lib/python3.9/site-packages/httpcore/_sync/http2.py", line 292, in request
self.send_headers(method, url, headers, has_body, timeout)
File "/Users/User/.pyenv/versions/3.9.0/lib/python3.9/site-packages/httpcore/_sync/http2.py", line 330, in send_headers
self.connection.send_headers(self.stream_id, headers, end_stream, timeout)
File "/Users/User/.pyenv/versions/3.9.0/lib/python3.9/site-packages/httpcore/_sync/http2.py", line 227, in send_headers
self.h2_state.send_headers(stream_id, headers, end_stream=end_stream)
File "/Users/User/.pyenv/versions/3.9.0/lib/python3.9/site-packages/h2/connection.py", line 770, in send_headers
frames = stream.send_headers(
File "/Users/User/.pyenv/versions/3.9.0/lib/python3.9/site-packages/h2/stream.py", line 865, in send_headers
frames = self._build_headers_frames(
File "/Users/User/.pyenv/versions/3.9.0/lib/python3.9/site-packages/h2/stream.py", line 1252, in _build_headers_frames
encoded_headers = encoder.encode(headers)
File "/Users/User/.pyenv/versions/3.9.0/lib/python3.9/site-packages/hpack/hpack.py", line 249, in encode
for header in headers:
File "/Users/User/.pyenv/versions/3.9.0/lib/python3.9/site-packages/h2/utilities.py", line 496, in inner
for header in headers:
File "/Users/User/.pyenv/versions/3.9.0/lib/python3.9/site-packages/h2/utilities.py", line 441, in _validate_host_authority_header
for header in headers:
File "/Users/User/.pyenv/versions/3.9.0/lib/python3.9/site-packages/h2/utilities.py", line 338, in _reject_pseudo_header_fields
raise ProtocolError(
h2.exceptions.ProtocolError: Received duplicate pseudo-header field b':method'
Request:
devServer = "https://api.sandbox.push.apple.com:443"
title = "some title"
notification = { "aps": { "alert": title, "sound": "someSound.caf" } }
client = httpx.Client(http2=True)
try:
r = client.post(devServer, json=notification, headers=headers)
finally:
client.close()

Just need to append '/3/device/{}'.format(deviceToken) to the devServer url as the path, and the ":path" pseudo-header will be automatically set to it.
that is,
devServer = 'https://api.sandbox.push.apple.com:443/3/device/{}'.format(deviceToken)
Explanation:
The ":path", ":method" and ":scheme" pseudo-headers generally wouldn't need to be added manually in http2
Reference: Hypertext Transfer Protocol Version 2 (HTTP/2)

IOError for oauth2.request when used with mod_wsgi + Apache

For a third party API, I'm using oauth2 module, I have following code:
Which works well when I use python run.py but throws IOError permission denied when using apache +mod_wsgi configuration.
Here is the error:
ERROR: Traceback (most recent call last):
File "/home/trex/workspace/scripts/flaskapp.py", line 32, in getTwitter_acc_audiences
_, content = client.request(uri=uri, method="GET")
File "build/bdist.linux-x86_64/egg/oauth2/__init__.py", line 687, in request
connection_type=connection_type)
File "/home/trex/workspace/myenv/lib/python2.7/site-packages/httplib2-0.9.2-py2.7.egg/httplib2/__init__.py", line 1609, in request
(response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
File "/home/trex/workspace/myenv/lib/python2.7/site-packages/httplib2-0.9.2-py2.7.egg/httplib2/__init__.py", line 1351, in _request
(response, content) = self._conn_request(conn, request_uri, method, body, headers)
File "/home/trex/workspace/myenv/lib/python2.7/site-packages/httplib2-0.9.2-py2.7.egg/httplib2/__init__.py", line 1272, in _conn_request
conn.connect()
File "/home/trex/workspace/myenv/lib/python2.7/site-packages/httplib2-0.9.2-py2.7.egg/httplib2/__init__.py", line 1036, in connect
self.disable_ssl_certificate_validation, self.ca_certs)
File "/home/trex/workspace/myenv/lib/python2.7/site-packages/httplib2-0.9.2-py2.7.egg/httplib2/__init__.py", line 80, in _ssl_wrap_socket
cert_reqs=cert_reqs, ca_certs=ca_certs)
File "/usr/lib/python2.7/ssl.py", line 911, in wrap_socket
ciphers=ciphers)
File "/usr/lib/python2.7/ssl.py", line 520, in __init__
self._context.load_verify_locations(ca_certs)
IOError: [Errno 13] Permission denied
api_manager.py
import oauth2 as oauth
from settings import * #-- All global params
class APIManager(object):
""" """
#init function goes here
def get_api_content(self, consumer_key, consumer_secret, access_token, access_secret):
""" Fetch API response using Oauth2 """
try:
consumer = oauth.Consumer(key=consumer_key, secret=consumer_secret)
token = oauth.Token(key=access_token, secret=access_secret)
client = oauth.Client(consumer, token)
uri = APIURL + APIURI.format(account_id=SOME_ACC_ID)
_, content = client.request(uri=uri, method="GET")
except Exception as e:
logerror("ERROR:", e)
content = {}
return content
flaskapp.py
from flask import Flask, jsonify
from api_manager import APIManager
from settings import *
app = Flask(__name__)
#app.route("/")
def callapi():
try:
apiobj = APIManager()
response = apiobj.get_api_content(CONSUMER_KEY, CONSUMER_SECRET, ACCESS_TOKEN, ACCESS_SECRET)
except Exception, e:
logerror("ERROR:", e)
return jsonify({})
return jsonify(response)
Apache WSGI Configuration:
<VirtualHost _default_:443>
...
# Default other config goes here...
WSGIScriptAlias / /home/trex/workspace/scripts/appwsgi.py
WSGIScriptReloading On
<Directory "/home/trex/workspace/scripts">
<Files "appwsgi.py">
Require all granted
</Files>
</Directory>
</VirtualHost>
Let me know if anything else I can give you the details.

There was an issue with permissions to my virtualenv. Following file had permission issues.
myenv/lib/python2.7/site-packages/httplib2-0.9.2-py2.7.egg/httplib2/cacerts.txt
And this simple fix fixed my issue:
chmod -R 755 /home/trex/workspace/myenv

How to print HTTP responses in python?

So I am coding this little simple script that I know will not work, but the script itself is running without errors and making a call to www.googleapis.com, so something is happening there. Without seeing the HTTP response I can't know what is wrong, though. The evidence of it actually doing something is because I am in china, and when I run the interpreter there is a time out until I break the script manually :
OUTPUT WITHOUT PROXY :
File "youtubeconnect.py", line 30, in <module>
execute()
File "youtubeconnect.py", line 27, in execute
youtube = authenticate()
File "youtubeconnect.py", line 15, in authenticate
service = build('youtube', 'v3',http=http)
File "/home/xavier/Code/autotube/venv/lib/python3.5/site-packages/oauth2client-1.5.2-py3.5.egg/oauth2client/util.py", line 140, in positional_wrapper
File "/home/xavier/Code/autotube/venv/lib/python3.5/site-packages/google_api_python_client-1.4.2-py3.5.egg/googleapiclient/discovery.py", line 196, in build
File "/home/xavier/Code/autotube/venv/lib/python3.5/site-packages/google_api_python_client-1.4.2-py3.5.egg/googleapiclient/discovery.py", line 242, in _retrieve_discovery_doc
File "/home/xavier/Code/autotube/venv/lib/python3.5/site-packages/oauth2client-1.5.2-py3.5.egg/oauth2client/client.py", line 596, in new_request
File "/home/xavier/Code/autotube/venv/lib/python3.5/site-packages/httplib2-0.9.2-py3.5.egg/httplib2/__init__.py", line 1314, in request
(response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
File "/home/xavier/Code/autotube/venv/lib/python3.5/site-packages/httplib2-0.9.2-py3.5.egg/httplib2/__init__.py", line 1064, in _request
(response, content) = self._conn_request(conn, request_uri, method, body, headers)
File "/home/xavier/Code/autotube/venv/lib/python3.5/site-packages/httplib2-0.9.2-py3.5.egg/httplib2/__init__.py", line 987, in _conn_request
conn.connect()
File "/usr/lib/python3.5/http/client.py", line 1229, in connect
super().connect()
File "/usr/lib/python3.5/http/client.py", line 826, in connect
(self.host,self.port), self.timeout, self.source_address)
File "/usr/lib/python3.5/socket.py", line 702, in create_connection
sock.connect(sa)
Now, when I run the interpreter through proxychains and tunnel it it works :
ProxyChains-3.1 (http://proxychains.sf.net)
|DNS-request| www.googleapis.com
|S-chain|-<>-127.0.0.1:1080-<><>-4.2.2.2:53-<><>-OK
|DNS-response| www.googleapis.com is 173.194.208.95
|S-chain|-<>-127.0.0.1:1080-<><>-173.194.208.95:443-<><>-OK
I am new to python, how can get the output of the HTTP error that google is sending me back when I try to connect erroneously to the API ? My code :
import httplib2
import os
from oauth2client import tools
from oauth2client.client import AccessTokenCredentials
#from oauth2client.client import AccessTokenRefreshError
from googleapiclient.discovery import build
from googleapiclient.http import MediaFileUpload
import urllib
def authenticate():
#scope = "https://www.googleapis.com/auth/youtube.upload"
acc_token = "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
flow = AccessTokenCredentials(
acc_token,'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36')
http = flow.authorize(httplib2.Http())
service = build('youtube', 'v3',http=http)
return(service)
def initialize_upload(youtube):
tags = None
vid = youtube.videos().insert
media_body = MediaFileUpload('1977.mp4', mimetype='video/mp4',
chunksize=1024 * 1024, resumable=True)
def execute():
youtube = authenticate()
initialize_upload(youtube)
execute()

Use logging: https://developers.google.com/api-client-library/python/guide/logging
Also, you can use Fiddler or Wireshark to analyze the packets being sent to you.

Python + BigQuery + ResponseNotReady()

I am using "bigquery_service = build('bigquery', 'v2', http=http)" and it is executing correctly in my laptop, but when I am executing this query on the server I am getting the following error.
File "/usr/lib64/python2.6/threading.py", line 532, in __bootstrap_inner
self.run()
File "/usr/lib64/python2.6/threading.py", line 484, in run
self.__target(*self.__args, **self.__kwargs)
File "QueryAPI.py", line 117, in localMain
exportDataToGCS(canonicalDate);
File "QueryAPI.py", line 177, in exportDataToGCS
bigquery_service = build('bigquery', 'v2', http=http)
File "/homeBigQuery/src/oauth2client/util.py", line 132, in positional_wrapper
return wrapped(*args, **kwargs)
File "/homeBigQuery/src/googleapiclient/discovery.py", line 198, in build
resp, content = http.request(requested_url)
File "/homeBigQuery/src/oauth2client/util.py", line 132, in positional_wrapper
return wrapped(*args, **kwargs)
File "/homeBigQuery/src/oauth2client/client.py", line 538, in new_request
redirections, connection_type)
File "/homeBigQuery/src/httplib2/__init__.py", line 1570, in request
(response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
File "/homeBigQuery/src/httplib2/__init__.py", line 1317, in _request
(response, content) = self._conn_request(conn, request_uri, method, body, headers)
File "/homeBigQuery/src/httplib2/__init__.py", line 1286, in _conn_request
response = conn.getresponse()
File "/usr/lib64/python2.6/httplib.py", line 980, in getresponse
raise ResponseNotReady()

You could try this 3rd party library for BigQuery. It's a thin wrapper around the Google BigQuery API.
To login to BigQuery:
import bigquery as bq
with open('my_key.pem', 'r') as key:
key_content = key.read()
bq_client = bq.get_client(project_id='foo',
service_account='bar',
private_key=key_content)
To submit a load job:
schema = [{'name':'foo', 'type':'STRING'}]
job = bq_client.import_data_from_uris(source_uris=['gs://foo/bar'],
dataset='foo',
table='bar',
schema=schema,
source_format=bq.client.JOB_FORMAT_CSV,
write_disposition=bq.client.JOB_WRITE_TRUNCATE,
field_delimiter=',',
skip_leading_rows=1)
try:
job_resource = bq_client.wait_for_job(job, timeout=60)
except bq.errors.BigQueryTimeoutException:
logging.critical('BigQuery loading timeout.')

Have you tried removing the "http=http" parameter ?
here is an example of the use of the BQ API:
import httplib2
from apiclient.discovery import build
from oauth2client.client import SignedJwtAssertionCredentials
# REPLACE WITH YOUR Project ID
PROJECT_NUMBER = 'XXXXXXXXXXX'
# REPLACE WITH THE SERVICE ACCOUNT EMAIL FROM GOOGLE DEV CONSOLE
SERVICE_ACCOUNT_EMAIL = 'XXXXX#developer.gserviceaccount.com'
# OBTAIN THE KEY FROM THE GOOGLE APIs CONSOLE
# More instructions here: http://goo.gl/w0YA0
f = file('key.p12', 'rb')
key = f.read()
f.close()
credentials = SignedJwtAssertionCredentials(
SERVICE_ACCOUNT_EMAIL,
key,
scope='https://www.googleapis.com/auth/bigquery')
http = httplib2.Http()
http = credentials.authorize(http)
service = build('bigquery', 'v2')
datasets = service.datasets()
response = datasets.list(projectId=PROJECT_NUMBER).execute(http)
print 'Dataset list:'
for dataset in response['datasets']:
print '%s' % dataset['datasetReference']['datasetId']

Solved this by specifying the proxy server host and port for the server that I was running this on. While initiating an authorized http service object with the help of credentials, I modified the code as follows:
BEFORE:
http = httplib2.Http()
http = credentials.authorize(http)
service = build('bigquery', 'v2', http=http)
FIX:
http = httplib2.Http(proxy_info = httplib2.ProxyInfo(httplib2.socks.PROXY_TYPE_HTTP, 'your-proxy-host-name', your-proxy-port))
http = credentials.authorize(http)
service = build('bigquery', 'v2', http=http)