Related
I'm trying to run a DAG that calls a docker container and executes a command inside it, but Airflow cannot execute the task. Follows the error launched:
*** Reading local file: /opt/airflow/logs/docker_operator_dag/docker_command_hello/2021-05-26T02:40:13.171571+00:00/2.log
[2021-05-26 02:45:26,001] {taskinstance.py:877} INFO - Dependencies all met for <TaskInstance: docker_operator_dag.docker_command_hello 2021-05-26T02:40:13.171571+00:00 [queued]>
[2021-05-26 02:45:26,030] {taskinstance.py:877} INFO - Dependencies all met for <TaskInstance: docker_operator_dag.docker_command_hello 2021-05-26T02:40:13.171571+00:00 [queued]>
[2021-05-26 02:45:26,031] {taskinstance.py:1068} INFO -
--------------------------------------------------------------------------------
[2021-05-26 02:45:26,031] {taskinstance.py:1069} INFO - Starting attempt 2 of 2
[2021-05-26 02:45:26,032] {taskinstance.py:1070} INFO -
--------------------------------------------------------------------------------
[2021-05-26 02:45:26,060] {taskinstance.py:1089} INFO - Executing <Task(DockerOperator): docker_command_hello> on 2021-05-26T02:40:13.171571+00:00
[2021-05-26 02:45:26,080] {standard_task_runner.py:52} INFO - Started process 67 to run task
[2021-05-26 02:45:26,083] {standard_task_runner.py:76} INFO - Running: ['airflow', 'tasks', 'run', 'docker_operator_dag', 'docker_command_hello', '2021-05-26T02:40:13.171571+00:00', '--job-id', '11', '--pool', 'default_pool', '--raw', '--subdir', 'DAGS_FOLDER/docker_job/docker-job.py', '--cfg-path', '/tmp/tmp3vl5dv7x', '--error-file', '/tmp/tmptazwx_tc']
[2021-05-26 02:45:26,084] {standard_task_runner.py:77} INFO - Job 11: Subtask docker_command_hello
[2021-05-26 02:45:26,181] {logging_mixin.py:104} INFO - Running <TaskInstance: docker_operator_dag.docker_command_hello 2021-05-26T02:40:13.171571+00:00 [running]> on host f1e5cfe4a07f
[2021-05-26 02:45:26,219] {taskinstance.py:1283} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_OWNER=airflow
AIRFLOW_CTX_DAG_ID=docker_operator_dag
AIRFLOW_CTX_TASK_ID=docker_command_hello
AIRFLOW_CTX_EXECUTION_DATE=2021-05-26T02:40:13.171571+00:00
AIRFLOW_CTX_DAG_RUN_ID=manual__2021-05-26T02:40:13.171571+00:00
[2021-05-26 02:45:26,227] {taskinstance.py:1482} ERROR - Task failed with exception
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 677, in urlopen
chunked=chunked,
File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 392, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/local/lib/python3.6/http/client.py", line 1287, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/local/lib/python3.6/http/client.py", line 1333, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/local/lib/python3.6/http/client.py", line 1282, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/local/lib/python3.6/http/client.py", line 1042, in _send_output
self.send(msg)
File "/usr/local/lib/python3.6/http/client.py", line 980, in send
self.connect()
File "/home/airflow/.local/lib/python3.6/site-packages/docker/transport/unixconn.py", line 43, in connect
sock.connect(self.unix_socket)
ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 727, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/util/retry.py", line 410, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/packages/six.py", line 734, in reraise
raise value.with_traceback(tb)
File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 677, in urlopen
chunked=chunked,
File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 392, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/local/lib/python3.6/http/client.py", line 1287, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/local/lib/python3.6/http/client.py", line 1333, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/local/lib/python3.6/http/client.py", line 1282, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/local/lib/python3.6/http/client.py", line 1042, in _send_output
self.send(msg)
File "/usr/local/lib/python3.6/http/client.py", line 980, in send
self.connect()
File "/home/airflow/.local/lib/python3.6/site-packages/docker/transport/unixconn.py", line 43, in connect
sock.connect(self.unix_socket)
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionRefusedError(111, 'Connection refused'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.6/site-packages/docker/api/client.py", line 207, in _retrieve_server_version
return self.version(api_version=False)["ApiVersion"]
File "/home/airflow/.local/lib/python3.6/site-packages/docker/api/daemon.py", line 181, in version
return self._result(self._get(url), json=True)
File "/home/airflow/.local/lib/python3.6/site-packages/docker/utils/decorators.py", line 46, in inner
return f(self, *args, **kwargs)
File "/home/airflow/.local/lib/python3.6/site-packages/docker/api/client.py", line 230, in _get
return self.get(url, **self._set_request_timeout(kwargs))
File "/home/airflow/.local/lib/python3.6/site-packages/requests/sessions.py", line 555, in get
return self.request('GET', url, **kwargs)
File "/home/airflow/.local/lib/python3.6/site-packages/requests/sessions.py", line 542, in request
resp = self.send(prep, **send_kwargs)
File "/home/airflow/.local/lib/python3.6/site-packages/requests/sessions.py", line 655, in send
r = adapter.send(request, **kwargs)
File "/home/airflow/.local/lib/python3.6/site-packages/requests/adapters.py", line 498, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionRefusedError(111, 'Connection refused'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/models/taskinstance.py", line 1138, in _run_raw_task
self._prepare_and_execute_task_with_callbacks(context, task)
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/models/taskinstance.py", line 1311, in _prepare_and_execute_task_with_callbacks
result = self._execute_task(context, task_copy)
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/models/taskinstance.py", line 1341, in _execute_task
result = task_copy.execute(context=context)
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/providers/docker/operators/docker.py", line 287, in execute
self.cli = self._get_cli()
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/providers/docker/operators/docker.py", line 319, in _get_cli
return APIClient(base_url=self.docker_url, version=self.api_version, tls=tls_config)
File "/home/airflow/.local/lib/python3.6/site-packages/docker/api/client.py", line 190, in __init__
self._version = self._retrieve_server_version()
File "/home/airflow/.local/lib/python3.6/site-packages/docker/api/client.py", line 215, in _retrieve_server_version
'Error while fetching server API version: {0}'.format(e)
docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', ConnectionRefusedError(111, 'Connection refused'))
[2021-05-26 02:45:26,230] {taskinstance.py:1532} INFO - Marking task as FAILED. dag_id=docker_operator_dag, task_id=docker_command_hello, execution_date=20210526T024013, start_date=20210526T024526, end_date=20210526T024526
[2021-05-26 02:45:26,261] {local_task_job.py:146} INFO - Task exited with return code 1
I'm using the Airflow docker-compose found here https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html and trying to run the following DAG:
from datetime import datetime, timedelta
from airflow import DAG
from airflow.operators.bash_operator import BashOperator
from airflow.operators.docker_operator import DockerOperator
from airflow.operators.python_operator import BranchPythonOperator
from airflow.operators.dummy_operator import DummyOperator
default_args = {
'owner' : 'airflow',
'description' : 'Use of the DockerOperator',
'depend_on_past' : False,
'start_date' : datetime(2021, 5, 1),
'email_on_failure' : False,
'email_on_retry' : False,
'retries' : 1,
'retry_delay' : timedelta(minutes=5)
}
with DAG('docker_operator_dag', default_args=default_args, schedule_interval="5 * * * *", catchup=False) as dag:
start_dag = DummyOperator(
task_id='start_dag'
)
end_dag = DummyOperator(
task_id='end_dag'
)
t1 = BashOperator(
task_id='print_current_date',
bash_command='date'
)
t2 = DockerOperator(
task_id='docker_command_sleep',
image='docker_image_task',
container_name='task___command_sleep',
api_version='auto',
auto_remove=True,
command="/bin/sleep 30",
docker_url="unix://var/run/docker.sock",
network_mode="bridge"
)
t3 = DockerOperator(
task_id='docker_command_hello',
image='docker_image_task',
container_name='task___command_hello',
api_version='auto',
auto_remove=True,
command="/bin/sleep 40",
docker_url="unix://var/run/docker.sock",
network_mode="bridge"
)
t4 = BashOperator(
task_id='print_hello',
bash_command='echo "hello world"'
)
start_dag >> t1
t1 >> t2 >> t4
t1 >> t3 >> t4
t4 >> end_dag
Also, I'm using Windows 10, I've tried adding the following in volumes: - "/var/run/docker.sock:/var/run/docker.sock" and I've succeeded in Ubuntu, but in Windows doesn't.
As requested, follow the docker-compose file:
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
#
# Basic Airflow cluster configuration for CeleryExecutor with Redis and PostgreSQL.
#
# WARNING: This configuration is for local development. Do not use it in a production deployment.
#
# This configuration supports basic configuration using environment variables or an .env file
# The following variables are supported:
#
# AIRFLOW_IMAGE_NAME - Docker image name used to run Airflow.
# Default: apache/airflow:master-python3.8
# AIRFLOW_UID - User ID in Airflow containers
# Default: 50000
# AIRFLOW_GID - Group ID in Airflow containers
# Default: 50000
# _AIRFLOW_WWW_USER_USERNAME - Username for the administrator account.
# Default: airflow
# _AIRFLOW_WWW_USER_PASSWORD - Password for the administrator account.
# Default: airflow
#
# Feel free to modify this file to suit your needs.
---
version: '3'
x-airflow-common:
&airflow-common
image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:2.0.2}
environment:
&airflow-common-env
AIRFLOW__CORE__EXECUTOR: CeleryExecutor
AIRFLOW__CORE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow#postgres/airflow
AIRFLOW__CELERY__RESULT_BACKEND: db+postgresql://airflow:airflow#postgres/airflow
AIRFLOW__CELERY__BROKER_URL: redis://:#redis:6379/0
AIRFLOW__CORE__FERNET_KEY: ''
AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION: 'true'
AIRFLOW__CORE__LOAD_EXAMPLES: 'false'
AIRFLOW__SCHEDULER__DAG_DIR_LIST_INTERVAL: 5 # Just to have a fast load in the front-end. Do not use it in production with those configurations.
AIRFLOW__API__AUTH_BACKEND: 'airflow.api.auth.backend.basic_auth'
AIRFLOW__CORE__ENABLE_XCOM_PICKLING: 'true' # "_run_image of the DockerOperator returns now a python string, not a byte string" Ref: https://github.com/apache/airflow/issues/13487
volumes:
- ./dags:/opt/airflow/dags
- ./logs:/opt/airflow/logs
- ./plugins:/opt/airflow/plugins
- "/var/run/docker.sock:/var/run/docker.sock" # We will pass the Docker Deamon as a volume to allow the webserver containers start docker images. Ref: https://stackoverflow.com/q/51342810/7024760
user: "${AIRFLOW_UID:-50000}:${AIRFLOW_GID:-50000}"
depends_on:
redis:
condition: service_healthy
postgres:
condition: service_healthy
services:
postgres:
image: postgres:13
environment:
POSTGRES_USER: airflow
POSTGRES_PASSWORD: airflow
POSTGRES_DB: airflow
volumes:
- postgres-db-volume:/var/lib/postgresql/data
healthcheck:
test: ["CMD", "pg_isready", "-U", "airflow"]
interval: 5s
retries: 5
restart: always
redis:
image: redis:latest
ports:
- 6379:6379
healthcheck:
test: ["CMD", "redis-cli", "ping"]
interval: 5s
timeout: 30s
retries: 50
restart: always
airflow-webserver:
<<: *airflow-common
command: webserver
ports:
- 8080:8080
healthcheck:
test: ["CMD", "curl", "--fail", "http://localhost:8080/health"]
interval: 10s
timeout: 10s
retries: 5
restart: always
airflow-scheduler:
<<: *airflow-common
command: scheduler
restart: always
airflow-worker:
<<: *airflow-common
command: celery worker
restart: always
airflow-init:
<<: *airflow-common
command: version
environment:
<<: *airflow-common-env
_AIRFLOW_DB_UPGRADE: 'true'
_AIRFLOW_WWW_USER_CREATE: 'true'
_AIRFLOW_WWW_USER_USERNAME: ${_AIRFLOW_WWW_USER_USERNAME:-airflow}
_AIRFLOW_WWW_USER_PASSWORD: ${_AIRFLOW_WWW_USER_PASSWORD:-airflow}
flower:
<<: *airflow-common
command: celery flower
ports:
- 5555:5555
healthcheck:
test: ["CMD", "curl", "--fail", "http://localhost:5555/"]
interval: 10s
timeout: 10s
retries: 5
restart: always
volumes:
postgres-db-volume:
I am running the Python and React MyUni Education sample app and have it loading in my localhost3000. When I click on the logins though it is giving me redirect error.
I created a new Docusign integration key in case there was something incorrectly configured in my previous set up, but the error remains the same:
"The redirect URI is not registered properly with DocuSign" error
Docusign sandbox configured with:
Redirect URIs "http://localhost:5001/api"
.env file I believe has the right environment variables set but please let me know if I am missing anything.
# Configuration file for the example
DS_CLIENT_ID=blank
DS_CLIENT_SECRET=blank
DS_IMPERSONATED_USER_GUID=blank
DS_TARGET_ACCOUNT_ID=false
DS_PAYMENT_GATEWAY_ID=blank
DS_PAYMENT_GATEWAY_NAME=Stripe
DS_PAYMENT_GATEWAY_DISPLAY_NAME=Stripe
DS_PRIVATE_KEY=blank
# React environment variables
**REACT_APP_DS_RETURN_URL=http://localhost:3000
REACT_APP_API_BASE_URL=http://localhost:5001/api**
REACT_APP_DS_AUTH_SERVER=https://account-d.docusign.com
# Demo Docusign API URL
REACT_APP_DS_DEMO_SERVER=https://demo.docusign.net
REACT_APP_DS_CLICKWRAP_URL=//demo.docusign.net/clickapi/sdk/latest/docusign-click.js
I believe it is affecting the JWT, as when I selected the "continue with a preconfigured login" I received a "Request failed with status code 500" error as well. Logs from that below.
PS C:\Users\windowsuser\OneDrive\Documents\GitHub\sample-app-myuni> flask run --port 5001
* Environment: production
WARNING: This is a development server. Do not use it in a production deployment.
Use a production WSGI server instead.
* Debug mode: off
* Running on http://127.0.0.1:5001/ (Press CTRL+C to quit)
127.0.0.1 - - [27/Jul/2020 16:13:20] "GET /api/get_status HTTP/1.1" 200 -
127.0.0.1 - - [27/Jul/2020 16:13:51] "GET /api/code_grant_auth HTTP/1.1" 401 -
127.0.0.1 - - [27/Jul/2020 16:14:03] "OPTIONS /api/jwt_auth HTTP/1.1" 200 -
[2020-07-27 16:14:04,237] ERROR in app: Exception on /api/jwt_auth [POST]
Traceback (most recent call last):
File "c:\python38\lib\site-packages\flask\app.py", line 2446, in wsgi_app
response = self.full_dispatch_request()
File "c:\python38\lib\site-packages\flask\app.py", line 1951, in full_dispatch_request
rv = self.handle_user_exception(e)
File "c:\python38\lib\site-packages\flask_cors\extension.py", line 161, in wrapped_function
return cors_after_request(app.make_response(f(*args, **kwargs)))
File "c:\python38\lib\site-packages\flask\app.py", line 1820, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "c:\python38\lib\site-packages\flask\_compat.py", line 39, in reraise
raise value
File "c:\python38\lib\site-packages\flask\app.py", line 1949, in full_dispatch_request
rv = self.dispatch_request()
File "c:\python38\lib\site-packages\flask\app.py", line 1935, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "c:\python38\lib\site-packages\flask_cors\decorator.py", line 128, in wrapped_function
resp = make_response(f(*args, **kwargs))
File "C:\Users\windowsuser\OneDrive\Documents\GitHub\sample-app-myuni\app\api\auth.py", line 43, in jwt_auth
DsClient.update_token()
File "C:\Users\windowsuser\OneDrive\Documents\GitHub\sample-app-myuni\app\ds_client.py", line 36, in update_token
client.request_jwt_user_token(DsConfig.client_id(),
File "c:\python38\lib\site-packages\docusign_esign\client\api_client.py", line 679, in request_jwt_user_token
raise ArgumentException("Private key not supplied or is invalid!")
File "c:\python38\lib\site-packages\docusign_esign\client\api_exception.py", line 51, in __init__
super(Exception).__init__(*args, **kwargs)
TypeError: super() argument 1 must be type, not str
127.0.0.1 - - [27/Jul/2020 16:14:04] "POST /api/jwt_auth HTTP/1.1" 500 -
127.0.0.1 - - [27/Jul/2020 16:18:14] "GET /api/code_grant_auth HTTP/1.1" 401 -
The integration key settings must match exactly
I think what you can check the URL where you get this error, it has first the IK which you created (GUID value starting with 61C9) then it has the redirectUri. It is URI encoded, but you can figure it out and make sure the same exact matching URI is set in the IK settings in the DocuSign Developer account.
I think it's the localhost:3000 for the front-end and not the localhost:5001 for the back-end and maybe there's an endpoint under there, just check and confirm and add it, wait 30 seconds and try again.
Okay just tested this in MySure app. Adding the following will allow the log in function to work as expected.
http://localhost:5001/api/callback
http://localhost:3000/callback
http://localhost:3000/
I am using Google AppEngine (GAE) Flexible environment with a custom runtime (python2.7). I am using custom so I can just manage a Dockerfile and have easy parity between my laptop and when deployed in GAE.
I have a python 2.7 Flask app that is doing a query against DataStore. I get the following (when starting python main.py in my local docker):
root#24dcf9b8712d:/home/vmagent/app# curl localhost:5000/things
'/home/vmagent/app/clientid-searchapp.json'
[2018-09-04 14:54:19,969] ERROR in app: Exception on /things [GET]
Traceback (most recent call last):
File "/env/local/lib/python2.7/site-packages/flask/app.py", line 2292, in wsgi_app
response = self.full_dispatch_request()
File "/env/local/lib/python2.7/site-packages/flask/app.py", line 1815, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/env/local/lib/python2.7/site-packages/flask/app.py", line 1718, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/env/local/lib/python2.7/site-packages/flask/app.py", line 1813, in full_dispatch_request
rv = self.dispatch_request()
File "/env/local/lib/python2.7/site-packages/flask/app.py", line 1799, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "main.py", line 55, in products
for rec in query.fetch(limit=100):
File "/env/lib/python2.7/site-packages/google/api_core/page_iterator.py", line 199, in _items_iter
for page in self._page_iter(increment=False):
File "/env/lib/python2.7/site-packages/google/api_core/page_iterator.py", line 230, in _page_iter
page = self._next_page()
File "/env/local/lib/python2.7/site-packages/google/cloud/datastore/query.py", line 517, in _next_page
query=query_pb,
File "/env/local/lib/python2.7/site-packages/google/cloud/datastore_v1/gapic/datastore_client.py", line 294, in run_query
request, retry=retry, timeout=timeout, metadata=metadata)
File "/env/lib/python2.7/site-packages/google/api_core/gapic_v1/method.py", line 139, in __call__
return wrapped_func(*args, **kwargs)
File "/env/lib/python2.7/site-packages/google/api_core/retry.py", line 260, in retry_wrapped_func
on_error=on_error,
File "/env/lib/python2.7/site-packages/google/api_core/retry.py", line 177, in retry_target
return target()
File "/env/lib/python2.7/site-packages/google/api_core/timeout.py", line 206, in func_with_timeout
return func(*args, **kwargs)
File "/env/lib/python2.7/site-packages/google/api_core/grpc_helpers.py", line 56, in error_remapped_callable
six.raise_from(exceptions.from_grpc_error(exc), exc)
File "/env/local/lib/python2.7/site-packages/six.py", line 737, in raise_from
raise value
PermissionDenied: 403 Missing or insufficient permissions.
127.0.0.1 - - [04/Sep/2018 14:54:19] "GET /things HTTP/1.1" 500 -
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>500 Internal Server Error</title>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.</p>
Snippets from main.py
app = Flask(__name__)
ds = datastore.Client()
<snip>
#app.route("/things")
def things():
global ds
pprint(os.environ['GOOGLE_APPLICATION_CREDENTIALS'])
query = ds.query(kind='Things', order=('thing_name',))
results = list()
for rec in query.fetch(limit=100):
results.append(rec)
return jsonify(results)
Output of gcloud auth list:
root#24dcf9b8712d:/home/vmagent/app# gcloud auth list
Credentialed Accounts
ACTIVE ACCOUNT
* <myapp>#<project-name>.iam.gserviceaccount.com
This is the configured service account. I have over-privileged it with excessive Role membership; I'm pretty certain the service account isn't lacking in permissions. In fact, it has 'DataStore Owner'
In main.py I am outputting:
pprint(os.environ['GOOGLE_APPLICATION_CREDENTIALS'])
And this outputs the path to my client-<myapp>.json file for the service account. It is the same file supplied in the Dockerfile as:
RUN gcloud auth activate-service-account --key-file=clientid-<myapp>.json --project <project-name> --quiet
I get the same results when running the Flask app in my local Docker environment and when I deploy to GAE Flexible. The Flask app has other endpoints that work as they don't call any other services. Only /things uses an API to call DataStore and it receives the above error.
I've been through all the docs. I'm sure it's something basic and obvious but afaict, everything looks right.
UPDATE:
I've tried creating the DataStore client with an explicit pass of project name with no change in result. Also, I've run this command from within the Docker container and this seems like unusual output to me but I don't know that it's incorrect for a Service Account.
root#e02b74cd269d:/home/vmagent/app# gcloud projects list
API [cloudresourcemanager.googleapis.com] not enabled on project
[<project id>]. Would you like to enable and retry (this will take a
few minutes)? (y/N)? y
When I respond w/ Y,:
ERROR: (gcloud.projects.list) PERMISSION_DENIED: Not allowed to get project settings for project <project id>
Output of gcloud config list:
root#d18c83cae166:/home/vmagent/app# gcloud config list
[core]
account = <myapp>#<myproject>.iam.gserviceaccount.com
disable_usage_reporting = False
project = <myproject>
Your active configuration is: [default]
trying to deploy the simple guestbook
https://cloud.google.com/appengine/docs/python/gettingstartedpython27/deploying-the-application
I got an error trying to deploy the app like so :
appcfg.py --verbose -A glossy-apex-130310 -V v1 update ./
07:56 PM Application: glossy-apex-130310 (was: None); version: v1 (was: None)
07:56 PM Host: appengine.google.com
2016-05-06 19:56:11,019 INFO sdk_update_checker.py:229 Checking for updates to the SDK.
2016-05-06 19:56:11,607 INFO appcfg.py:2378 Reading app configuration.
07:56 PM
Starting update of app: glossy-apex-130310, version: v1
07:56 PM Getting current resource limits.
2016-05-06 19:56:11,608 INFO appcfg.py:1693 Send: /api/appversion/getresourcelimits, params={'version': 'v1', 'app_id': 'glossy-apex-130310'}
2016-05-06 19:56:12,350 ERROR appcfg.py:2396 An error occurred processing file '': HTTP Error 403: Forbidden Unexpected HTTP status 403. Aborting.
Error 403: --- begin server output ---
You do not have permission to modify this app (app_id=u's~glossy-apex-130310').
--- end server output ---
So I deployed it like so :
gcloud preview app deploy --project glossy-apex-130310
http://glossy-apex-130310.appspot.com/
getting this
Internal Server Error
The server has either erred or is incapable of performing the requested operation.
Traceback (most recent call last):
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1535, in __call__
rv = self.handle_exception(request, response, e)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1529, in __call__
rv = self.router.dispatch(request, response)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1278, in default_dispatcher
return route.handler_adapter(request, response)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1102, in __call__
return handler.dispatch()
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 572, in dispatch
return self.handle_exception(e, self.app.debug)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 570, in dispatch
return method(*args, **kwargs)
File "/base/data/home/apps/s~glossy-apex-130310/20160506t132909.392595029440587911/guestbook.py", line 72, in get
greetings = greetings_query.fetch(10)
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/utils.py", line 160, in positional_wrapper
return wrapped(*args, **kwds)
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/query.py", line 1203, in fetch
return self.fetch_async(limit, **q_options).get_result()
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py", line 378, in get_result
self.check_success()
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py", line 422, in _help_tasklet_along
value = gen.throw(exc.__class__, exc, tb)
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/query.py", line 980, in _run_to_list
batch = yield rpc
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py", line 508, in _on_rpc_completion
result = rpc.get_result()
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 613, in get_result
return self.__get_result_hook(self)
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/datastore/datastore_query.py", line 2921, in __query_result_hook
yaml_index=yaml, xml_index=xml)
NeedIndexError: no matching index found. recommended index is:
- kind: Greeting
ancestor: yes
properties:
- name: date
direction: desc
The suggested index for this query is:
- kind: Greeting
ancestor: yes
properties:
- name: date
direction: desc
index.yaml file is
indexes:
- kind: Greeting
ancestor: yes
properties:
- name: date
direction: desc
I read that I'm supposed to wait a while. it's been 4 hours.
There are No indexes created here
https://console.cloud.google.com/datastore/indexes?project=glossy-apex-130310
The gcloud sdk can upload indexes to the Cloud Datastore with the create-indexes command, like this (docs):
$ gcloud preview datastore create-indexes ~/myapp/index.yaml
The App Engine SDK will upload indexes along with application code when appcfg.py is executed with the update command:
$ appcfg.py -A <YOUR_PROJECT_ID> update myapp/
or indexes can be uploaded separately when appcfg.py is executed with the update_indexes command:
appcfg.py update_indexes myapp/
I am competing in the Google App Challenge but I am unable to get the app working at the appspot domain website.I am using Python to deploy the app.Need Help
The app.yaml file details are given below
application: abhicorp111
version: 1
runtime: python27
api_version: 1
threadsafe: yes
handlers:
- url: /WebPlayer\.unity3d
static_files: WebPlayer/WebPlayer.unity3d
upload: WebPlayer/WebPlayer\.unity3d
- url: .*
script: main.app
libraries:
- name: webapp2
version: "2.5.2"
The main.py content
#!/usr/bin/env python
#
# Copyright 2007 Google Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
from google.appengine.ext.webapp import template
import webapp2
import os
class MainHandler(webapp2.RequestHandler):
def get(self):
template_values = {
'greetings': 'greetings',
}
path = os.path.join('WebPlayer', 'WebPlayer.html')
self.response.out.write(template.render(path, template_values))
app = webapp2.WSGIApplication([
('/', MainHandler)
], debug=True)
After updating from command line getting the 500 server error
The Folder structure for the app
The error that i am getting
The error log i got while uploading it through GAE log Console
> 2013-11-20 09:27:34 Running command: "['C:\\Python27\\python.exe',
> 'C:\\Program Files (x86)\\Google\\google_appengine\\dev_appserver.py',
> '--skip_sdk_update_check=yes', '--port=15080', '--admin_port=8007',
> u'E:\\GAEGame\\WebPlayer']" INFO 2013-11-20 09:27:35,315
> devappserver2.py:660] Skipping SDK update check. WARNING 2013-11-20
> 09:27:35,331 api_server.py:327] Could not initialize images API; you
> are likely missing the Python "PIL" module. WARNING 2013-11-20
> 09:27:35,588 simple_search_stub.py:1009] Could not read search indexes
> from
> c:\users\santu\appdata\local\temp\appengine.abhicorp111\search_indexes
> INFO 2013-11-20 09:27:35,648 api_server.py:138] Starting API
> server at: http://localhost:4755 INFO 2013-11-20 09:27:35,674
> dispatcher.py:168] Starting module "default" running at:
> http://localhost:15080 INFO 2013-11-20 09:27:35,700
> admin_server.py:117] Starting admin server at: http://localhost:8007
> ERROR 2013-11-20 03:57:44,651 wsgi.py:262]
>
> Traceback (most recent call last):
>
> File "C:\Program Files
> (x86)\Google\google_appengine\google\appengine\runtime\wsgi.py", line
> 239, in Handle
>
> handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
>
> File "C:\Program Files
> (x86)\Google\google_appengine\google\appengine\runtime\wsgi.py", line
> 298, in _LoadHandler
>
> handler, path, err = LoadObject(self._handler)
>
> File "C:\Program Files
> (x86)\Google\google_appengine\google\appengine\runtime\wsgi.py", line
> 84, in LoadObject
>
> obj = __import__(path[0])
>
> File "E:\GAEGame\WebPlayer\main.py", line 22
>
> def get(self):
>
> ^
>
> IndentationError: expected an indented block
>
> INFO 2013-11-20 09:27:44,663 module.py:599] default: "GET /
> HTTP/1.1" 500 - ERROR 2013-11-20 03:57:44,697 wsgi.py:262]
>
> Traceback (most recent call last):
>
> File "C:\Program Files
> (x86)\Google\google_appengine\google\appengine\runtime\wsgi.py", line
> 239, in Handle
>
> handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
>
> File "C:\Program Files
> (x86)\Google\google_appengine\google\appengine\runtime\wsgi.py", line
> 298, in _LoadHandler
>
> handler, path, err = LoadObject(self._handler)
>
> File "C:\Program Files
> (x86)\Google\google_appengine\google\appengine\runtime\wsgi.py", line
> 84, in LoadObject
>
> obj = __import__(path[0])
>
> File "E:\GAEGame\WebPlayer\main.py", line 22
>
> def get(self):
>
> ^
>
> IndentationError: expected an indented block
>
> INFO 2013-11-20 09:27:44,710 module.py:599] default: "GET
> /favicon.ico HTTP/1.1" 500 -
After updating and going through the errors and then starting the server i am getting this message
This is done from another folder and then i called the appserver start command of python
C:\Windows\system32>dev_appserver.py E:\ed23\WebPlayer1
INFO 2013-11-20 13:25:04,336 sdk_update_checker.py:245] Checking for updates
to the SDK.
WARNING 2013-11-20 13:25:06,148 api_server.py:327] Could not initialize images
API; you are likely missing the Python "PIL" module.
WARNING 2013-11-20 13:25:06,641 simple_search_stub.py:1009] Could not read sear
ch indexes from c:\users\santu\appdata\local\temp\appengine.enginerite11\search_
indexes
INFO 2013-11-20 13:25:06,688 api_server.py:138] Starting API server at: http
://localhost:20091
INFO 2013-11-20 13:25:06,713 dispatcher.py:168] Starting module "default" ru
nning at: http://localhost:8080
INFO 2013-11-20 13:25:06,742 admin_server.py:117] Starting admin server at:
http://localhost:8000
ERROR 2013-11-20 07:58:37,220 webapp2.py:1552] WebPlayer1.html
Traceback (most recent call last):
File "C:\Program Files (x86)\Google\google_appengine\lib\webapp2-2.5.2\webapp2
.py", line 1535, in __call__
rv = self.handle_exception(request, response, e)
File "C:\Program Files (x86)\Google\google_appengine\lib\webapp2-2.5.2\webapp2
.py", line 1529, in __call__
rv = self.router.dispatch(request, response)
File "C:\Program Files (x86)\Google\google_appengine\lib\webapp2-2.5.2\webapp2
.py", line 1278, in default_dispatcher
return route.handler_adapter(request, response)
File "C:\Program Files (x86)\Google\google_appengine\lib\webapp2-2.5.2\webapp2
.py", line 1102, in __call__
return handler.dispatch()
File "C:\Program Files (x86)\Google\google_appengine\lib\webapp2-2.5.2\webapp2
.py", line 572, in dispatch
return self.handle_exception(e, self.app.debug)
File "C:\Program Files (x86)\Google\google_appengine\lib\webapp2-2.5.2\webapp2
.py", line 570, in dispatch
return method(*args, **kwargs)
File "E:\ed23\WebPlayer1\main.py", line 28, in get
self.response.out.write(template.render(path, template_values))
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\ext\weba
pp\template.py", line 89, in render
t = _load_internal_django(template_path, debug)
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\ext\weba
pp\template.py", line 163, in _load_internal_django
template = django.template.loader.get_template(file_name)
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\_interna
l\django\template\loader.py", line 157, in get_template
template, origin = find_template(template_name)
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\_interna
l\django\template\loader.py", line 138, in find_template
raise TemplateDoesNotExist(name)
TemplateDoesNotExist: WebPlayer1.html
INFO 2013-11-20 13:28:37,285 module.py:599] default: "GET / HTTP/1.1" 500 24
09
INFO 2013-11-20 13:28:37,497 module.py:599] default: "GET /favicon.ico HTTP/
1.1" 200 1150
ERROR 2013-11-20 08:03:19,309 webapp2.py:1552] WebPlayer1.html
Traceback (most recent call last):
File "C:\Program Files (x86)\Google\google_appengine\lib\webapp2-2.5.2\webapp2
.py", line 1535, in __call__
rv = self.handle_exception(request, response, e)
File "C:\Program Files (x86)\Google\google_appengine\lib\webapp2-2.5.2\webapp2
.py", line 1529, in __call__
rv = self.router.dispatch(request, response)
File "C:\Program Files (x86)\Google\google_appengine\lib\webapp2-2.5.2\webapp2
.py", line 1278, in default_dispatcher
return route.handler_adapter(request, response)
File "C:\Program Files (x86)\Google\google_appengine\lib\webapp2-2.5.2\webapp2
.py", line 1102, in __call__
return handler.dispatch()
File "C:\Program Files (x86)\Google\google_appengine\lib\webapp2-2.5.2\webapp2
.py", line 572, in dispatch
return self.handle_exception(e, self.app.debug)
File "C:\Program Files (x86)\Google\google_appengine\lib\webapp2-2.5.2\webapp2
.py", line 570, in dispatch
return method(*args, **kwargs)
File "E:\ed23\WebPlayer1\main.py", line 28, in get
self.response.out.write(template.render(path, template_values))
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\ext\weba
pp\template.py", line 89, in render
t = _load_internal_django(template_path, debug)
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\ext\weba
pp\template.py", line 163, in _load_internal_django
template = django.template.loader.get_template(file_name)
INFO 2013-11-20 13:33:19,326 module.py:599] default: "GET / HTTP/1.1" 500 24
09
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\_interna
l\django\template\loader.py", line 157, in get_template
template, origin = find_template(template_name)
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\_interna
l\django\template\loader.py", line 138, in find_template
raise TemplateDoesNotExist(name)
TemplateDoesNotExist: WebPlayer1.html
INFO 2013-11-20 13:33:19,427 module.py:599] default: "GET /favicon.ico HTTP/
1.1" 304 -
main.py has syntax errors (incorrect indentation). You need to indent def get(self): and, further indent the code the belongs in the get method. That's what
> IndentationError: expected an indented block
is trying to tell you.
I suggest getting your app running locally in dev_appserver before uploading it. That'll cut the development cycle time way down.
From the error log:
Could not initialize images API; you are likely missing the Python "PIL" module.
You need to configure PIL for your application. From the configuring third party libraries section of the documentation, add the following to your app.yaml:
libraries:
- name: PIL
version: latest
Following that you have some basic problems with your code itself. Python is sensitive to whitepace and any statement that go inside a block need to be indented, which is why you are getting this error:
File "E:\GAEGame\WebPlayer\main.py", line 22
def get(self):
^
IndentationError: expected an indented block
All statements inside a block need to be indented; the common practice is to indent by four spaces (not tabs). Here is your code, properly indented:
from google.appengine.ext.webapp import template
import webapp2
import os
class MainHandler(webapp2.RequestHandler):
def get(self):
template_values = {
'greetings': 'greetings',
}
path = os.path.join('WebPlayer', 'WebPlayer.html')
self.response.out.write(template.render(path, template_values))
app = webapp2.WSGIApplication([
('/', MainHandler)
], debug=True)