I have a problem configuring Endpoints API. Any code i use, from my own, to google's examples on site fail with the same traceback
WARNING 2016-11-01 06:16:48,279 client.py:229] no scheduler thread, scheduler.run() will be invoked by report(...)
Traceback (most recent call last):
File "/home/vladimir/projects/sb_fork/sb/lib/vendor/google/api/control/client.py", line 225, in start
self._thread.start()
File "/home/vladimir/sdk/google-cloud-sdk/platform/google_appengine/google/appengine/api/background_thread/background_thread.py", line 108, in start
start_new_background_thread(self.__bootstrap, ())
File "/home/vladimir/sdk/google-cloud-sdk/platform/google_appengine/google/appengine/api/background_thread/background_thread.py", line 87, in start_new_background_thread
raise ERROR_MAP[error.application_error](error.error_detail)
FrontendsNotSupported
INFO 2016-11-01 06:16:48,280 client.py:327] created a scheduler to control flushing
INFO 2016-11-01 06:16:48,280 client.py:330] scheduling initial check and flush
INFO 2016-11-01 06:16:48,288 client.py:804] Refreshing access_token
/home/vladimir/projects/sb_fork/sb/lib/vendor/urllib3/contrib/appengine.py:113: AppEnginePlatformWarning: urllib3 is using URLFetch on Google App Engine sandbox instead of sockets. To use sockets directly instead of URLFetch see https://urllib3.readthedocs.io/en/latest/contrib.html.
AppEnginePlatformWarning)
ERROR 2016-11-01 06:16:49,895 service_config.py:125] Fetching service config failed (status code 403)
ERROR 2016-11-01 06:16:49,896 wsgi.py:263]
Traceback (most recent call last):
File "/home/vladimir/sdk/google-cloud-sdk/platform/google_appengine/google/appengine/runtime/wsgi.py", line 240, in Handle
handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
File "/home/vladimir/sdk/google-cloud-sdk/platform/google_appengine/google/appengine/runtime/wsgi.py", line 299, in _LoadHandler
handler, path, err = LoadObject(self._handler)
File "/home/vladimir/sdk/google-cloud-sdk/platform/google_appengine/google/appengine/runtime/wsgi.py", line 85, in LoadObject
obj = __import__(path[0])
File "/home/vladimir/projects/sb_fork/sb/main.py", line 27, in <module>
api_app = endpoints.api_server([SolarisAPI,], restricted=False)
File "/home/vladimir/projects/sb_fork/sb/lib/vendor/endpoints/apiserving.py", line 497, in api_server
controller)
File "/home/vladimir/projects/sb_fork/sb/lib/vendor/google/api/control/wsgi.py", line 77, in add_all
a_service = loader.load()
File "/home/vladimir/projects/sb_fork/sb/lib/vendor/google/api/control/service.py", line 110, in load
return self._load_func(**kw)
File "/home/vladimir/projects/sb_fork/sb/lib/vendor/google/api/config/service_config.py", line 78, in fetch_service_config
_log_and_raise(Exception, message_template.format(status_code))
File "/home/vladimir/projects/sb_fork/sb/lib/vendor/google/api/config/service_config.py", line 126, in _log_and_raise
raise exception_class(message)
Exception: Fetching service config failed (status code 403)
INFO 2016-11-01 06:16:49,913 module.py:788] default: "GET / HTTP/1.1" 500 -
My app.yaml is configured like the new Endpoints Migrating to 2.0 document states:
- url: /_ah/api/.*
script: api.solaris.api_app
And main.py imports the API into the app:
api_app = endpoints.api_server([SolarisAPI,], restricted=False)
I use Google Cloud SDK with these versions:
Google Cloud SDK 132.0.0
app-engine-python 1.9.40
bq 2.0.24
bq-nix 2.0.24
core 2016.10.24
core-nix 2016.10.24
gcloud
gsutil 4.22
gsutil-nix 4.22
Have you tried generating and uploading the OpenAPI configuration for the service? See the sections named "Generating the OpenAPI configuration file" and "Deploying the OpenAPI configuration file" in the python library documentation.
Note that in step 2 of the generation process, you may need to prepend python to the command (e.g python lib/endpoints/endpointscfg.py get_swagger_spec ...), since the PyPi package doesn't preserve executable file permissions right now.
To get rid of the "FrontendsNotSupported" you need to use a "B*" instance class.
The error "Exception: Fetching service config failed" should be gone if you follow the steps in https://cloud.google.com/endpoints/docs/frameworks/python/quickstart-frameworks-python. As already pointed out by Brad, the section "OpenAPI configuration" and the resulting environment variables are required to make the service configuration work.
Related
I'm trying to set up an Airflow DAG that is able to send emails through the EmailOperator in Composer 2, Airflow 2.3.4. I've followed this guide. I tried running the example DAG that is provided in the guide, but I get an HTTP 400 error.
The log looks like this:
[2023-01-20, 10:46:45 UTC] {taskinstance.py:1904} ERROR - Task failed with exception
Traceback (most recent call last):
File "/opt/python3.8/lib/python3.8/site-packages/airflow/operators/email.py", line 75, in execute
send_email(
File "/opt/python3.8/lib/python3.8/site-packages/airflow/utils/email.py", line 58, in send_email
return backend(
File "/opt/python3.8/lib/python3.8/site-packages/airflow/providers/sendgrid/utils/emailer.py", line 123, in send_email
_post_sendgrid_mail(mail.get(), conn_id)
File "/opt/python3.8/lib/python3.8/site-packages/airflow/providers/sendgrid/utils/emailer.py", line 142, in _post_sendgrid_mail
response = sendgrid_client.client.mail.send.post(request_body=mail_data)
File "/opt/python3.8/lib/python3.8/site-packages/python_http_client/client.py", line 277, in http_request
self._make_request(opener, request, timeout=timeout)
File "/opt/python3.8/lib/python3.8/site-packages/python_http_client/client.py", line 184, in _make_request
raise exc
python_http_client.exceptions.BadRequestsError: HTTP Error 400: Bad Request
I've looked at similar threads on Stackoverflow but none of those suggestions worked for me.
I have set up and verified the from email address in Sendgrid and it
uses a whole email address including the domain.
I also set this email address up in Secret Manager (as well as the API key).
I haven't changed the test DAG from the guide, except for the 'to' address.
In another DAG I've tried enabling 'email_on_retry' and that also didn't trigger any mail.
I'm at a loss here, can someone provide me with suggestions on things to try?
after reading the offical google translate api document, it provide us with the following sample code:
from google.cloud import translate
def translate_text(text="Hello, world!", project_id="weighty-site-333613"):
client = translate.TranslationServiceClient().from_service_account_json('key.json')
location = "global"
parent = f"projects/{project_id}/locations/{location}"
response = client.translate_text(
request={
"parent": parent,
"contents": [text],
"mime_type": "text/plain",
"source_language_code": "en-US",
"target_language_code": "zh-CN",
}
)
for translation in response.translations:
print("Translated text: {}".format(translation.translated_text))
translate_text()
These code worked properly on google cloud terminal.
However, even if i put the "key.json" file in the same folder, an error like this is shown:
/usr/local/bin/python3.6 /Users/jiajunmao/Documents/GitHub/translator_of_excel/google_trans.py
Traceback (most recent call last):
File "/Users/jiajunmao/Documents/GitHub/translator_of_excel/google_trans.py", line 37, in <module>
translate_text()
File "/Users/jiajunmao/Documents/GitHub/translator_of_excel/google_trans.py", line 22, in translate_text
client = translate.TranslationServiceClient().from_service_account_json('key.json')
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/google/cloud/translate_v3/services/translation_service/client.py", line 354, in __init__
always_use_jwt_access=True,
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/google/cloud/translate_v3/services/translation_service/transports/grpc.py", line 158, in __init__
always_use_jwt_access=always_use_jwt_access,
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/google/cloud/translate_v3/services/translation_service/transports/base.py", line 110, in __init__
**scopes_kwargs, quota_project_id=quota_project_id
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/google/auth/_default.py", line 488, in default
raise exceptions.DefaultCredentialsError(_HELP_MESSAGE)
google.auth.exceptions.DefaultCredentialsError: Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or explicitly create credentials and re-run the application. For more information, please see https://cloud.google.com/docs/authentication/getting-started
Process finished with exit code 1
can someone tell me what should i do at this step? thank you so much
You need service account json file with correct permissions from GCP under IAM & Service Accounts.
Then you need to implement command,
GOOGLE_APPLICATION_CREDENTIALS = "/path/to/your/service_account.json"
I am new to GCP. I created a cloud function and tried to deploy it. I encountered the following error while deploying. Can anyone help me solve this issue? Thank you!
Command:
gcloud functions deploy first_cloud_function_http --runtime python37 --trigger-http --allow-unauthenticated --verbosity debug
Error Logs:
DEBUG: Running [gcloud.functions.deploy] with arguments: [--allow-unauthenticated: "True", --runtime: "python37", --trigger-http: "True", --verbosity: "debug", NAME: "first_cloud_function_http"]
INFO: Not using ignore file.
INFO: Not using ignore file.
Deploying function (may take a while - up to 2 minutes)...failed.
DEBUG: (gcloud.functions.deploy) OperationError: code=13, message=Failed to initialize region (action ID: 78ed38913711b6cd)
Traceback (most recent call last):
File "/home/hasher/GN/google-cloud-sdk/lib/googlecloudsdk/calliope/cli.py", line 983, in Execute
resources = calliope_command.Run(cli=self, args=args)
File "/home/hasher/GN/google-cloud-sdk/lib/googlecloudsdk/calliope/backend.py", line 808, in Run
resources = command_instance.Run(args)
File "/home/hasher/GN/google-cloud-sdk/lib/surface/functions/deploy.py", line 351, in Run
return _Run(args, track=self.ReleaseTrack())
File "/home/hasher/GN/google-cloud-sdk/lib/surface/functions/deploy.py", line 305, in _Run
on_every_poll=[TryToLogStackdriverURL])
File "/home/hasher/GN/google-cloud-sdk/lib/googlecloudsdk/api_lib/functions/util.py", line 318, in CatchHTTPErrorRaiseHTTPExceptionFn
return func(*args, **kwargs)
File "/home/hasher/GN/google-cloud-sdk/lib/googlecloudsdk/api_lib/functions/util.py", line 369, in WaitForFunctionUpdateOperation
on_every_poll=on_every_poll)
File "/home/hasher/GN/google-cloud-sdk/lib/googlecloudsdk/api_lib/functions/operations.py", line 151, in Wait
on_every_poll)
File "/home/hasher/GN/google-cloud-sdk/lib/googlecloudsdk/api_lib/functions/operations.py", line 121, in _WaitForOperation
sleep_ms=SLEEP_MS)
File "/home/hasher/GN/google-cloud-sdk/lib/googlecloudsdk/core/util/retry.py", line 219, in RetryOnResult
result = func(*args, **kwargs)
File "/home/hasher/GN/google-cloud-sdk/lib/googlecloudsdk/api_lib/functions/operations.py", line 73, in _GetOperationStatus
raise exceptions.FunctionsError(OperationErrorToString(op.error))
googlecloudsdk.api_lib.functions.exceptions.FunctionsError: OperationError: code=13, message=Failed to initialize region (action ID: 78ed38913711b6cd)
ERROR: (gcloud.functions.deploy) OperationError: code=13, message=Failed to initialize region (action ID: 78ed38913711b6cd)
Notice that as stated on the documentation of the gcloud functions deploy command you'd necessarily need to set the --region flag.
To check the available regions where Cloud Functions is available refer to the following sections of the documentation.
For example, if you'd like to deploy the function in the europe-west1 region running the following command would suffice:
gcloud functions deploy first_cloud_function_http --region europe-west1 --runtime python37 --trigger-http --allow-unauthenticated --verbosity debug
Additionally, if you'd like to avoid using the --region flag you can set a default region for Cloud Functions by running:
gcloud config set functions/region REGION
where you could change the REGION field to any of the locations mentioned above.
I know just enough devops to be dangerous. I've successfully deployed a VERY simple python flask app to App Engine that basically publishes received post data as a message to PubSub. It is almost identical to Google's sample code to do so. Only difference is it uses a service account I push with the app repository to access PubSub to circumvent this issue.
Works very well so far, but I've started seeing a very small number of errors around starting a new thread in threading.py:
1)
Traceback (most recent call last):
File "src/python/grpcio/grpc/_cython/_cygrpc/credentials.pyx.pxi", line 33, in grpc._cython.cygrpc._spawn_callback_async
File "src/python/grpcio/grpc/_cython/_cygrpc/credentials.pyx.pxi", line 24, in grpc._cython.cygrpc._spawn_callback_in_thread
File "/usr/lib/python2.7/threading.py", line 736, in start
_start_new_thread(self.__bootstrap, ())
thread.error: can't start new thread
2)
Traceback (most recent call last):
File "src/python/grpcio/grpc/_cython/_cygrpc/credentials.pyx.pxi", line 33, in grpc._cython.cygrpc._spawn_callback_async
File "src/python/grpcio/grpc/_cython/_cygrpc/credentials.pyx.pxi", line 24, in grpc._cython.cygrpc._spawn_callback_in_thread
3)
Traceback (most recent call last):
File "src/python/grpcio/grpc/_cython/_cygrpc/credentials.pyx.pxi", line 33, in grpc._cython.cygrpc._spawn_callback_async
File "src/python/grpcio/grpc/_cython/_cygrpc/credentials.pyx.pxi", line 33, in grpc._cython.cygrpc._spawn_callback_async
File "src/python/grpcio/grpc/_cython/_cygrpc/credentials.pyx.pxi", line 24, in grpc._cython.cygrpc._spawn_callback_in_thread
File "src/python/grpcio/grpc/_cython/_cygrpc/credentials.pyx.pxi", line 24, in grpc._cython.cygrpc._spawn_callback_in_thread
File "/usr/lib/python2.7/threading.py", line 736, in start
File "/usr/lib/python2.7/threading.py", line 736, in start
I have 2 questions, in order of importance:
This is an app that basically needs 100% uptime in order to not lose data (not confident the clients attempt retries if there is an error on my server side). Are these errors internal to how App Engine is managing my app's resources, and not resulting in errors handling actual requests? How can I determine if I ever responded with an HTTP error/didn't successfully handle a request? I don't see any errors in my nginx logs...is that the place I need to look to see if anything failed?
Is there a way I can fix this error?
https://github.com/GoogleCloudPlatform/google-cloud-python/blob/master/pubsub/google/cloud/pubsub_v1/publisher/client.py#L143
it looks like publisher.publish(topic_path, data=data) is an async operation, returning a concurrent.futures.Future object
Have you trying calling the Future's result()? https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.Future.result
This will block until the future object is successful, fails, or timesout.
You could then forward that result as your HTTP response.
Hopefully, the result object will give you more information about the error.
Ended up changing the methodology a bit. Instead of posting a pubsub message then having dataflow ingest through GCS to BigQuery I decided to stream directly into BQ using the BigQuery python client. Updated dependencies for the python flask app to:
Flask==1.0.2
google-cloud-pubsub==0.39.1
gunicorn==19.9.0
google-cloud-bigquery==1.11.2
and I am no longer seeing any of those exceptions. It's worth noting that I'm still using a service account .json credentials file in the same directory as the app source, and I'm creating the BigQuery client with
bq_client = bigquery.Client.from_service_account_json(BQ_SVC_ACCT_FILE).
For anyone else with similar issues I'd recommend updating your dependencies (esp any Google Cloud client libraries) and create the Client you need from a local service account credentials file. I attempted to use the inherited Compute engine environment credentials (basically the default project compute engine service account) but that was less stable than pushing up an actual credential file and using that locally. However...assess your own security needs before doing the same.
I'm trying to make backup of files on my computer in onedrive with duplicity.
I have installed all dependencies, when running duplicity there is the auth link generated which I must open in browser and than in duplicity after giving permissions for app paste the return url.
I do all this steps but duplicity is returning me
Traceback (most recent call last):
File "/usr/bin/duplicity", line 1532, in <module>
with_tempdir(main)
File "/usr/bin/duplicity", line 1526, in with_tempdir
fn()
File "/usr/bin/duplicity", line 1364, in main
action = commandline.ProcessCommandLine(sys.argv[1:])
File "/usr/lib/python2.7/site-packages/duplicity/commandline.py", line 1116, in ProcessCommandLine
backup, local_pathname = set_backend(args[0], args[1])
File "/usr/lib/python2.7/site-packages/duplicity/commandline.py", line 1005, in set_backend
globals.backend = backend.get_backend(bend)
File "/usr/lib/python2.7/site-packages/duplicity/backend.py", line 223, in get_backend
obj = get_backend_object(url_string)
File "/usr/lib/python2.7/site-packages/duplicity/backend.py", line 209, in get_backend_object
return factory(pu)
File "/usr/lib/python2.7/site-packages/duplicity/backends/onedrivebackend.py", line 90, in __init__
self.initialize_oauth2_session()
File "/usr/lib/python2.7/site-packages/duplicity/backends/onedrivebackend.py", line 153, in initialize_oauth2_session
authorization_response=redirected_to)
File "/usr/lib/python2.7/site-packages/requests_oauthlib/oauth2_session.py", line 232, in fetch_token
self._client.parse_request_body_response(r.text, scope=self.scope)
File "/usr/lib/python2.7/site-packages/oauthlib/oauth2/rfc6749/clients/base.py", line 409, in parse_request_body_response
self.token = parse_token_response(body, scope=scope)
File "/usr/lib/python2.7/site-packages/oauthlib/oauth2/rfc6749/parameters.py", line 376, in parse_token_response
validate_token_parameters(params)
File "/usr/lib/python2.7/site-packages/oauthlib/oauth2/rfc6749/parameters.py", line 383, in validate_token_parameters
raise_from_error(params.get('error'), params)
File "/usr/lib/python2.7/site-packages/oauthlib/oauth2/rfc6749/errors.py", line 271, in raise_from_error
raise cls(**kwargs)
InvalidClientError: (invalid_client) The client does not exist. If you are the application developer, configure a new application through the application management site at https://manage.dev.live.com/.
It looks like there is no app with ID which duplicity generate auth link with.
But when I go to the link provided by duplicity I see that "Duplicity is asking for permissions".
So should I add my own app and in some way provide its id to duplicity? (I was searching how to do it but without result) or is it a duplicity bug?
All programmatic interaction with Windows Live requires a client ID,
which uniquely identifies your application to Windows Live. Your
application must include the client ID in every request that it sends
to the Messenger Connect API Service.
You have to register your application as shown in this official Windows Live tutorial:
https://msdn.microsoft.com/en-us/library/ff751474.aspx
And then pass your ID to the application to be able to authentificate in Windows Live in execution time when asking to the API.
You can use the code in
https://github.com/fkalis/bash-onedrive-upload
which also provide full support for upload files which size is bigger then 100MB