When I try to upload a sample csv data to my GAE app through appcfg.py, it shows the below 401 error.
2015-11-04 10:44:41,820 INFO client.py:571 Refreshing due to a 401 (attempt 2/2)
2015-11-04 10:44:41,821 INFO client.py:797 Refreshing access_token
Error 401: --- begin server output ---
You must be logged in as an administrator to access this.
--- end server output ---
Here is the command I tried,
appcfg.py upload_data --application=dev~app --url=http://localhost:8080/_ah/remote_api --filename=data/sample.csv
This is how we do it in order to use custom authentication.
Custom handler in app.yaml
- url: /remoteapi.*
script: remote_api.app
Custom wsgi app in remote_api.py to override CheckIsAdmin
from google.appengine.ext.remote_api import handler
from google.appengine.ext import webapp
import re
MY_SECRET_KEY = 'MAKE UP PASSWORD HERE' # make one up, use the same one in the shell command
cookie_re = re.compile('^"?([^:]+):.*"?$')
class ApiCallHandler(handler.ApiCallHandler):
def CheckIsAdmin(self):
"""Determine if admin access should be granted based on the
auth cookie passed with the request."""
login_cookie = self.request.cookies.get('dev_appserver_login', '')
match = cookie_re.search(login_cookie)
if (match and match.group(1) == MY_SECRET_KEY
and 'X-appcfg-api-version' in self.request.headers):
return True
else:
self.redirect('/_ah/login')
return False
app = webapp.WSGIApplication([('.*', ApiCallHandler)])
From here we script the uploading of data that was exported from our live app. Use the same password that you made up in the python script above.
echo "MAKE UP PASSWORD HERE" | appcfg.py upload_data --email=some#example.org --passin --url=http://localhost:8080/remoteapi --num_threads=4 --kind=WebHook --filename=webhook.data --db_filename=bulkloader-progress-webhook.sql3
WebHook and webhook.data are specific to the Kind that we exported from production.
I had a similar issue, where appcfg.py was not giving me any credentials dialog, so I could not authenticate. I downgraded from GAELauncher 1.27 to 1.26, and the authentication started working again.
Temporary solution: go to https://console.developers.google.com/storage/browser/appengine-sdks/featured/ to get version 1.9.26
Submitted bug report: https://code.google.com/p/google-cloud-sdk/issues/detail?id=340
You cannot use the appcfg.py upload_data command with the development server [edit: as is; see Josh J's answer]. It only works with the remote_api endpoint running on App Engine and authenticated with OAuth2.
An easy way to load data into the dev server's datastore is to create an endpoint that reads a CSV file and creates the appropriate datastore entities, then hit it with the browser. (Be sure to remove the endpoint before deploying the app, or restrict access to the URL with login: admin.)
You must have an oauth token for a google account that is not an admin of that project. Try passing the --no_cookies flag so that it prompts for authentication again.
Maybe this has something to do with it? From the docs
Connecting your app to the local development server
To use the local development server for your app running locally, you
need to do the following:
Set environment variables. Add or modify your app's Datastore
connection code. Setting environment variables
Create an environment variable DATASTORE_HOST and set it to the host
and port on which the local development server is listening. The
default host and port is http://localhost:8080. (Note: If you use the
port and/or host command line arguments to change these defaults, be
sure to adjust DATASTORE_HOST accordingly.) The following bash shell
example shows how to set this variable:
export DATASTORE_HOST=http://localhost:8080 Create an environment
variable named DATASTORE_DATASET and set it to your dataset ID, as
shown in the following bash shell example:
export DATASTORE_DATASET= Note: Both the Python and Java
client libraries look for the environment variables DATASTORE_HOST and
DATASTORE_DATASET.
Link to Docs
https://cloud.google.com/datastore/docs/tools/devserver
Related
Background:
I'm trying to deploy a Django app to the Google App Engine (GAE) standard environment in the python39 runtime
The database configuration is stored in a Secret Manager secret version, similar to Google's GAE Django tutorial (link)
The app is run as a user-managed service account server#myproject.iam.gserviceaccount.com, which has the appropriate permissions to access the secret, as can be confirmed using gcloud secret versions access
Problem:
In the Django settings.py module, when I try to access the secret using google.cloud.secretmanager.SecretManagerServiceClient.access_secret_version(...), I get the following CONSUMER_INVALID error:
google.api_core.exceptions.PermissionDenied: 403 Permission denied on resource project myproject. [links {
description: "Google developer console API key"
url: "https://console.developers.google.com/project/myproject/apiui/credential"
}
, reason: "CONSUMER_INVALID"
domain: "googleapis.com"
metadata {
key: "service"
value: "secretmanager.googleapis.com"
}
metadata {
key: "consumer"
value: "projects/myproject"
}
My Debugging
I cannot reproduce the error above outside of GAE;
I can confirm that the SA can access the secret:
gcloud secrets versions access latest --secret=server_env --project myproject \
--impersonate-service-account=server#myproject.iam.gserviceaccount.com
WARNING: This command is using service account impersonation. All API calls will be executed as [server#myproject.iam.gserviceaccount.com].
DATABASE_URL='postgres://django:...'
SECRET_KEY='...'
I've also confirmed I run the django app locally with service account impersonation and make the above access_secret_version(...) calls
In desperation I even created an API key for the project and hardcoded it into my settings.py file, and this also raises the same error
I've confirmed the following settings in the project:
the app is running with using the correct user-managed SA
the call to access_secret_version is being made with the correct SA (ie that the credentials are being pulled from the GAE environment correctly)
the project has the secretmanager.googleapis.com service enabled, and has billing enabled and the billing account is active
If you have any suggestions for a configuration or method to help debug this, I'd much appreciate it!
Relevant Code Snippets
app.yaml
service_account: server#myproject.iam.gserviceaccount.com
runtime: python39
handlers:
# This configures Google App Engine to serve the files in the app's static
# directory.
- url: /_static
static_dir: _static/
# This handler routes all requests not caught above to your main app. It is
# required when static routes are defined, but can be omitted (along with
# the entire handlers section) when there are no static files defined.
- url: /.*
script: auto
env_variables:
...
inbound_services:
- mail
- mail_bounce
app_engine_apis: true
Service Account Creation & Permissions
The SA is created with Terraform as below
(The SA doesn't have the role roles/secretmanager.secretAccessor, but has an IAM binding directly on the secret itself)
resource "google_service_account" "frontend_server" {
project = google_project.project.project_id
account_id = "server"
display_name = "Frontend Server Service Account"
}
resource "google_project_iam_member" "frontend_server" {
depends_on = [
google_service_account.frontend_server,
]
for_each = toset([
"roles/appengine.serviceAgent",
"roles/cloudsql.client",
"roles/cloudsql.instanceUser",
"roles/secretmanager.viewer",
"roles/storage.objectViewer",
])
project = google_project.project.project_id
role = each.key
member = "serviceAccount:${google_service_account.frontend_server.email}"
}
Django settings.py
The relevant sections of the app settings.py are shown below; the access_secret_version raises the
import logging
import environ
from google.cloud import secretmanager
import google.auth
# Load secrets from secret manager; the client is auth'd by SA IAM policies
credentials, project = google.auth.default(
scopes=['https://www.googleapis.com/auth/cloud-platform']
)
secretmanager_client = secretmanager.SecretManagerServiceClient(credentials=credentials)
# Load the database connection string into the environment
secrets = [
f"projects/{GOOGLE_CLOUD_PROJECT}/secrets/server_env/versions/latest",
]
for name in secrets:
try:
logging.info(f"Reading secret {name} into django settings module...")
payload = secretmanager_client.access_secret_version(name=name).payload.data.decode("UTF-8")
env.read_env(io.StringIO(payload))
except Exception as e:
logging.error(f"Encountered error when accessing secret {name}: {e}")
logging.error(f"Client credentials during error: {secretmanager_client._transport._credentials.__dict__}")
raise e from None
You have granted the incorrect role. Have a look to that documentation page.
Secret Viewer role allows you to view the secret and versions but NOT the content.
Secret Accessor role allows you to access to secret version content.
Sigh. This was a terrible case of a hard-to-read typo in the app.yaml file. The project had a mnemonic substring with very similar letters that I had mistyped and just couldn't see.
FWIW, if anyone is running into a similar flub, you can at least avoid this one source of error:
I was passing in a project prefix string and an environment string through the app.yaml file, and then settings.py file I concatenated these strings to make the project
When running gcloud app deploy the (correct) concatenated project string also existed in my shell $GOOGLE_CLOUD_PROJECT variable, so the deployment happened correctly with the right project id
However I removed the concatenation code in settings.py in favour of the GOOGLE_CLOUD_PROJECT env variable that is always present in GAE (docs)
TLDR: DRY is good..
I am using CKAN 2.7.2 .
I have added the following configurations in my development.ini file of ckan
ckan.oauth2.authorization_endpoint = https://account.lab.fiware.org/oauth2/authorize
ckan.oauth2.token_endpoint = https://account.lab.fiware.org/oauth2/token
ckan.oauth2.profile_api_url = https://account.lab.fiware.org/user
ckan.oauth2.client_id = xyz
ckan.oauth2.client_secret = xyz
ckan.oauth2.profile_api_user_field = abc
ckan.oauth2.profile_api_mail_field = abc#gmail.com
Also, have exported the following while running ckan using paster serve :
export OAUTHLIB_INSECURE_TRANSPORT=True
Also, I have added an application in fiware.lab also with callback URL where the CKAN instance is running (i.e a private IP of 172.30.66.XX type running on port 5000)
And when I click on Login,i get redirect to fiware lab login page and after logging in i get the following error
{"state": "eyJjYW1lX2Zyb20iOiAiL2Rhc2hib2FyZCJ9", "error": "mismatching_redirect_uri"} (HTTP 400)
If anyone could please help me in this. It would be of great help.
That error means that the redirect URL attached by CKAN is not the same as the one you have registered as callback URL when registering the application in the IDM.
Ensure that the callback URL you have included in the IDM is:
http://YOUR_CKAN_INSTANCE/oauth2/callback
The URL must match exactly (so no trailing backslash)
I'm trying to get Flask-Mail setup on in Flexible ENV on Google App Engine. Flask-Mail works on my localhost using the credentials for a domain I am trying to use to send the mail. However, when using it on GAE through my API it returns a 502 error, however it shows no error messages in the logs or console. Going through the documentation for GAE Flexible it doesn't mention anything about NOT being able to use it, however it doesn't show how one would setup Flask-Mail either.
I have this..
mail = Mail()
print('1') // We Get here
msg = Message("Hello",
sender="me#mydomain.com",
recipients=["me#mydomain.com"])
print('2') // We get here
msg.body = 'Testing'
print('3') // We get here
mail.send(msg)
print('4') // This never gets call because I timeout on a 502 before this
I can tell I am not getting any fatal errors because the app stays working. However this fails with the 502. I have tried adding my email to the list of authorized senders but it doesn't seem to have helped.
I would appreciate any feedback. If I forced to use a 3rd party service to send mail it may cause me to move the project off of GAE.
As Ivan posted on his comment, to send email from a GAE app you need to use a mail service. Right now there are 3 options for apps on a flexible environment: Mailgun, MailJet and SendGrid. Choose the one you see better for your app.
After setting up an account on the mail service you have chosen, you have to prepare your code by integrating the parts related to the mail service.
These tutorials should help you establish the mail service for your app:
Mailgun
MailJet
SendGrid
I've had the same error but on a virtual machine on the internet ( linode service ) and it turned out that it has some thing to do with rDNS and some domain name config that you have to set up for your Ip address to get things working correctly , check this
https://www.linode.com/community/questions/19082/i-just-created-my-first-linode-and-i-cant-send-emails-why
I have a django application running using gunicorn and frontended by nginx. I am trying to get the full hostname of the application because I need to redirect out of the application and need to pass the external application my url so it can pass control back to me. How do I find my hostname/application name. I tried request.get_host(), but it gave me 127.0.0.1:8000
You can access it from the request object's META attribute.
request.META['HTTP_HOST']
I am trying to use the Docker Python API to login to a Docker cloud:
https://docker-py.readthedocs.io/en/stable/client.html#creating-a-client1
What is the URL? What is the Port?
I have tried to get it to work with cloud.docker.com, but I am fine with any registry server, so long as it is free to use and I can use it to upload Docker images from one computer and run them on another.
I have already got everything running using my own locally hosted registry, but I can’t seem to figure out how to connect to a server. It’s kind of ridiculous that hosting my own registry is easier than using an existing registry server.
My code looks like this, but I am unsure what the args.* parameters should be:
client = docker.DockerClient(base_url=args.docker_registry)
client.login(username=args.docker_user, password=args.docker_password)
I’m not sure what the base_url needs to be so that I can log in, and the error messages are not helpful at all.
Can you give me an example that works?
The base_url parameter is the URL of the Docker server, not the Docker Registry.
Try something like:
from docker.errors import APIError, TLSParameterError
try:
client = docker.from_env()
client.login(username=args.docker_user, password=args.docker_password, registry=args.docker_registry)
except (APIError, TLSParameterError) as err:
# ...
Here's how I have logged in to Docker using Python:
import docker
client = docker.from_env()
client.login(username='USERNAME', password='PASSWORD', email='EMAIL',
registry='https://index.docker.io/v1/')
and here's what it returns:
{'IdentityToken': '', 'Status': 'Login Succeeded'}
So, that means it has been logged in successfully.
I still haven't figured out what the registry of cloud.docker.com is called, but I got it to work by switching to quay.io as my registry server, which works with the intuitive registry='quay.io'