We Keep Coding

Google Cloud Secret Manager prints unnecessary debug information when running in compute engine

I am using Google Cloud Secret Manager in Jupyterlab notebooks in GCP AI Platform notebooks. I am able to access secrets as excepted but my code is printing out a lot of DEBUG lines related to the authentication.
Here is my code:
from google.cloud import secretmanager
PROJECT_ID = "<PROJECT_ID >"
def access_secret_version(secret_id, version_id="latest"):
# Create the Secret Manager client.
client = secretmanager.SecretManagerServiceClient()
# Build the resource name of the secret version.
name = f"projects/{PROJECT_ID}/secrets/{secret_id}/versions/{version_id}"
# Access the secret version.
response = client.access_secret_version(name=name)
# Return the decoded payload.
return response.payload.data.decode('UTF-8')
# Get my secret
mySecret= access_secret_version('mySecret')
Here is the log when running my code (IP address and service account has been changed):
DEBUG:google.auth._default:Checking None for explicit credentials as part of auth process...
DEBUG:google.auth._default:Checking Cloud SDK credentials as part of auth process...
DEBUG:google.auth._default:Cloud SDK credentials not found on disk; not using them
DEBUG:google.auth._default:Checking for App Engine runtime as part of auth process...
DEBUG:google.auth._default:No App Engine library was found so cannot authentication via App Engine Identity Credentials.
DEBUG:google.auth.transport._http_client:Making request: GET http://111.222.333.444
DEBUG:google.auth.transport._http_client:Making request: GET http://metadata.google.internal/computeMetadata/v1/project/project-id
DEBUG:google.auth.transport.requests:Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): metadata.google.internal:80
DEBUG:urllib3.connectionpool:http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/default/?recursive=true HTTP/1.1" 200 193
DEBUG:google.auth.transport.requests:Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/my-compute#developer.gserviceaccount.com/token
DEBUG:urllib3.connectionpool:http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/my-compute#developer.gserviceaccount.com/token HTTP/1.1" 200 260
I have tried both to set compute engine service account explicitly in the terminal and as an environment variable in the script using the service accounts JSON file.
What is the reason the Debug code is being printed, and how do I get it to go away?

Found the reason:
I had set the debugging level to debug for my script:
logging.basicConfig(level=logging.debug)

Getting 401 Client Error when using firebase

I'm new to firebase and I'm trying to update some data in an existing project but I'm getting the following error: 401 Client Error: Unauthorized for url. So how to fix this problem and thank you! I just want to mention also that I have been added to the project, so I'm not the owner.
this is my code:
from firebase import firebase
firebase = firebase.FirebaseApplication("the http path", None) # None bcz we are testing
firebase.put("/esco-lebanon/device-configs/atest-dev", "brightness", 50)
print("Updated")

According to the documentation, a 401 error means one of the following:
The auth token has expired.
The auth token used in the request is invalid.
Authenticating with an access_token failed.
The request violates your Firebase Realtime Database Rules.
The understanding here is that your client code needs to correctly identify a Firebase Authentication user with a token in access_token, and that user account must have access to read the data in the database according to its security rules. Since you haven't provided a token, your access is coming in anonymously, so you could only query data where security rules allow universal access.
If you haven't investigated using authentication in your request, you should read the documentation about that.

What permissions are required for validating Google Store In-App Purchases?

I am trying to validate user purchase with the token received from the client (internal release).
For that I am using Python script with Google Python API Client (https://github.com/googleapis/google-api-python-client).
import httplib2
from oauth2client.service_account import ServiceAccountCredentials
token = "token received from the client"
http = httplib2.Http(timeout=self.http_timeout)
credentials = ServiceAccountCredentials.from_json_keyfile_name(
"./service_account.json", "https://www.googleapis.com/auth/androidpublisher"
)
http = credentials.authorize(http)
result = build("androidpublisher", "v3", http=http)\
.purchases()\
.products()\
.get(packageName="<package name>", productId="<subscription id>", token=token)\
.execute(http=http)
The response I am getting from this call is:
HttpError 401 when requesting https://www.googleapis.com/androidpublisher/v3/applications/<package name>/purchases/products/<subscription id>/tokens/<token>?alt=json returned "The current user has insufficient permissions to perform the requested operation."
The service user being used is give admin permissions to the account (for the sake of the test) in Google Play Console, and set to be "Project Owner" in "Google Cloud Console" console (for the sake of the test again).
What seems to be wrong here?

The 'Owner' permissions are sufficient to receipt validation [Source].
The error you're getting is most likely a propagation issue where it can actually take ~24hrs for the service credentials to go into effect throughout the system.

Google Cloud Endpoints: Authentication issues (error 403)

I'm trying to connect to my Google Cloud Endpoints API that is running as an Appengine app:
#endpoints.api(name='helloworldendpoints', allowed_client_ids=["1234", "12345"], version='v1', auth_level=endpoints.AUTH_LEVEL.REQUIRED)
class HelloWorldApi(remote.Service):
...
The API request is as follows:
scopes = ["https://www.googleapis.com/auth/userinfo.email"]
credentials = ServiceAccountCredentials.from_json_keyfile_name("CloudEndpointsClient.json", scopes)
from httplib2 import Http
http_auth = credentials.authorize(Http())
from apiclient.discovery import build
api_root = 'https://myapp.appspot.com/_ah/api'
api = 'helloworldendpoints'
version = 'v1'
discovery_url = '%s/discovery/v1/apis/%s/%s/rest' % (api_root, api, version)
service = build(api, version, discoveryServiceUrl=discovery_url)
response = service.myFunction(myparameter = "123456").execute(http=http_auth)#
print response
The requests work well if I remove authentication requirements.
I know that authentication works since the error changes if after authenticating.
The error message I'm getting is:
googleapiclient.errors.HttpError: https://my-app.appspot.com/_ah/api/helloworldendpoints/v1/obtainScoreFromEmail?myparameter=1234&alt=json returned "Access Not Configured. has not been used in project 123456789 before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/helloworldendpoints/overview?project=123456789 then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.">
I cannot enable the API in my Google Cloud Project, since the API does not exist.

What I found to work is a hack around. I used a user authentication (instead of server) on the same project for the same API which worked (https://cloud.google.com/endpoints/docs/frameworks/python/access_from_python).
After I switched back to my initial server auth. method it started working.

Google Drive Python API resumable upload error 401 after 2 hours

First of all, I'm sorry if this is a too silly question... this is the first time I'm trying to use any of the technologies involved in this script (Python, the drive api, oauth 2.0, etc). I swear I've been searching and trying this for about a week before posting the question. hehehe
I'm trying to use the google-api-python-client to upload a big file (3.5GiB) that is on a terminal only Linux Debian. I've had some success uploading small files, but when I try to upload the big file, the upload stops about 1~2 hours after it started with HTTP 401 error (unauthorized). I've been looking on how to get a new access token but have had little success.
This is my (updated) code so far:
#!/usr/bin/python
import httplib2
import pprint
import time
from apiclient.discovery import build
from apiclient.http import MediaFileUpload
from apiclient import errors
from oauth2client.client import OAuth2WebServerFlow
# Copy your credentials from the APIs Console
CLIENT_ID = 'myclientid'
CLIENT_SECRET = 'myclientsecret'
# Check https://developers.google.com/drive/scopes for all available scopes
OAUTH_SCOPE = 'https://www.googleapis.com/auth/drive'
# Redirect URI for installed apps
REDIRECT_URI = 'urn:ietf:wg:oauth:2.0:oob'
# Run through the OAuth flow and retrieve credentials
flow = OAuth2WebServerFlow(CLIENT_ID, CLIENT_SECRET, OAUTH_SCOPE, REDIRECT_URI)
authorize_url = flow.step1_get_authorize_url()
print 'Go to the following link in your browser: ' + authorize_url
code = raw_input('Enter verification code: ').strip()
credentials = flow.step2_exchange(code)
# Create an httplib2.Http object and authorize it with our credentials
http = httplib2.Http()
http = credentials.authorize(http)
drive_service = build('drive', 'v2', http=http)
# Insert a file
media_body = MediaFileUpload('bigfile.zip', mimetype='application/octet-stream', chunksize=1024*256, resumable=True)
body = {
'title': 'bigfile.zip',
'description': 'Big File',
'mimeType': 'application/octet-stream'
}
retries = 0
request = drive_service.files().insert(body=body, media_body=media_body)
response = None
while response is None:
try:
print http.request.credentials.access_token
status, response = request.next_chunk()
if status:
print "Uploaded %.2f%%" % (status.progress() * 100)
retries = 0
except errors.HttpError, e:
if e.resp.status == 404:
print "Error 404! Aborting."
exit()
else:
if retries > 10:
print "Retries limit exceeded! Aborting."
exit()
else:
retries += 1
time.sleep(2**retries)
print "Error (%d)... retrying." % e.resp.status
continue
print "Upload Complete!"
After some digging, I found out that the authorized http object automatically refreshes the access token after receiving 401. Although it's really changing the access token, it's still not continuing the upload as expected... see the output below:
ya29.AHES6ZTo_-0oDqwn3JnU2uCR2bRjpRGP0CSQSMHGr6KvgEE
Uploaded 2.28%
ya29.AHES6ZTo_-0oDqwn3JnU2uCR2bRjpRGP0CSQSMHGr6KvgEE
Uploaded 2.29%
ya29.AHES6ZTo_-0oDqwn3JnU2uCR2bRjpRGP0CSQSMHGr6KvgEE
Uploaded 2.29%
ya29.AHES6ZTo_-0oDqwn3JnU2uCR2bRjpRGP0CSQSMHGr6KvgEE
Uploaded 2.30%
ya29.AHES6ZTo_-0oDqwn3JnU2uCR2bRjpRGP0CSQSMHGr6KvgEE
Error (401)... retrying.
ya29.AHES6ZQqp3_qbWsTk4yVDdHnlwc_7GvPZiFIReDnhIIiHao
Error (401)... retrying.
ya29.AHES6ZSqx90ZOUKqDEP4AAfWCVgXZYT2vJAiLwKDRu87JOs
Error (401)... retrying.
ya29.AHES6ZTp0RZ6U5K5UdDom0gq3XHnyVS-2sVU9hILOrG4o3Y
Error (401)... retrying.
ya29.AHES6ZSR-IOiwJ_p_Dm-OnCanVIVhCZLs7H_pYLMGIap8W0
Error (401)... retrying.
ya29.AHES6ZRnmM-YIZj4S8gvYBgC1M8oYy4Hv5VlcwRqgnZCOCE
Error (401)... retrying.
ya29.AHES6ZSF7Q7C3WQYuPAWrxvqbTRsipaVKhv_TfrD_gef1DE
Error (401)... retrying.
ya29.AHES6ZTsGzwIIprpPhCrqmoS3UkPsRzst5YHqL-zXJmz6Ak
Error (401)... retrying.
ya29.AHES6ZSS_1ZBiQJvZG_7t5uW3alsy1piGe4-u2YDnwycVrI
Error (401)... retrying.
ya29.AHES6ZTLFbBS8mSFWQ9zK8cgbX8RPeLghPxkfiKY54hBB-0
Error (401)... retrying.
ya29.AHES6ZQBeMWY50z6fWXvaCcd5_AJr_AYOuL2aiNKpK-mmyU
Error (401)... retrying.
ya29.AHES6ZTs2mYYSEyOqI_Ms4itKDx36t39Oc5RNZHkV4Dq49c
Retries limit exceeded! Aborting.
I'm using debian lenny with Python 2.5.2 installed, and installed the ssl and google-api-python-client through pip install about a week ago.
Thanks in advance for any help.
EDIT: Apparently, the problem isn't with the api. I tried the same code above, but with two small files, with 1h between them (system.sleep()). The output was:
ya29.AHES6ZRUssiLfuhqCP9Cu7C7LuhRV2rYzPldU27wiMJZWb8
Uploaded 66.89%
ya29.AHES6ZRUssiLfuhqCP9Cu7C7LuhRV2rYzPldU27wiMJZWb8
Upload 1 Complete!
ya29.AHES6ZRUssiLfuhqCP9Cu7C7LuhRV2rYzPldU27wiMJZWb8
Uploaded 57.62%
ya29.AHES6ZQd3o1ciwXpNFImH3CK0-dJAtQba_oeIO9DDbIq154
Upload 2 Complete!
For the second upload, a new access token was used successfully. So, perhaps the resumable session is expiring after some time or is only valid for that specific access token?

I filed an issue on the google-api-python-client project, and according to Joe Gregorio from google, the problem is in the backend:
"This is an issue with the backend and not with the API or with your code. As you deduced, if the upload goes too long the access_token expires and at that point the resumable upload can't be continued. There is work on progress to fix this issue right now, I will update this bug once the issue is fixed on the server side."

I assume the problem is that after the 1-2 hour limit your access token to your remote database expires; cutting off your connection with the remote server. I think what you could do is look at your hosts API manual... They should have something in there about 'refresh tokens'(They get you another Access Token, note some hosts only allow you to use one refresh token per session), if they are allowed an unlimited amount you can use a combination of a timer and AJAX to keep asking for more access tokens.
If not then you would have a make an AJAX request for another Authorization Token and exchange that for another Access token every hour. That sounds like a very rigorous process but I think that is the only way if your token keeps expiring.
Also just on another note have you tried other methods of uploading? If you said the above script ran for 1-2 hours and it only uploaded 1.44% of the file that could take 100+ hours to fully upload (Way too long for only 3 Gigs).