I have searched in multiple places and have tried multiple ways to find a solution to this. But nothing was helpful, can you please help me provide the solution. It is getting extremely difficult to host a python flask application in google app engine flexible environment. Python code: "getresource.py"
Here is the link from google to deploy Python Flask application
Here is the simple code:
from flask import Flask, jsonify
from pymysql import connections, ProgrammingError, DatabaseError, MySQLError, DataError
from os import environ
DB_HOST = environ.get("DB_HOST")
USER = environ.get("USER")
PWD = environ.get("PASSWORD")
DATABASE = environ.get("DATABASE")
app = Flask(__name__)
dbconn = connections.Connection(
host=DB_HOST,
port=3306,
user=USER,
password=PWD,
db=DATABASE
)
#app.route("/getResource/<id>", methods=['GET'])
def getresource(id):
result = ()
select_sql = "SELECT `id`, `firstName`, `middleName`, `lastName`, `listOfTechWorkedOn`, `certifications`, `projects`, `applicationWorkLoadTypes` FROM `resource` WHERE `id`=%s"
cursor = dbconn.cursor()
try:
cursor.execute(select_sql, (id))
result = cursor.fetchone()
response = {}
response['id'] = result[0]
response['firstName'] = result[1]
response['middleName'] = result[2]
response['lastName'] = result[3]
response['listOfTechWorkedOn'] = result[4]
response['certifications'] = result[5]
response['projects'] = result[6]
response['applicationWorkLoadTypes'] = result[7]
return jsonify(response)
except ProgrammingError as p:
return
except DatabaseError as d:
return d
except MySQLError as m:
return m
except DataError as de:
return de
dbconn.close()
if __name__ == '__main__':
app.run(host='127.0.0.1',port=8080, debug=True)
Here is the app.yaml file
runtime: python
env: flex
entrypoint: gunicorn -b :$PORT getresource:app
service: getresource
runtime_config:
python_version: 3
manual_scaling:
instances: 1
env_variables:
DB_HOST: "1.2.3.4"
USER: "root"
PASSWORD: "abcdefg"
DATABASE: "abcd"
requirements.txt file
Flask==1.0.2
gunicorn==19.9.0
PyMySQL==0.9.2
I am getting the following error: "gcloud app deploy"
ERROR: (gcloud.app.deploy) Error Response: [13] An internal error occurred during deployment.
I have tried multiple ways, like changing the python code, putting google cloud libraries in the requirements.txt, chaning port, service name etc. But nothing seems to be working.
And surprisingly, I am not getting the error trace in stackdriver logs as well. And Google does not have proper answer it seems (at least from all the googling that i have done)
gcloud version
Google Cloud SDK 222.0.0
bq 2.0.36
core 2018.10.19
gsutil 4.34
**
EDIT:
** Thanks for all the response. I tried all the options, but getting the same error with following debug message. I cannot make anything out of it
Thanks for the response. I tried with all the suggested options, but getting the same error. Following is the debug log, and I cannot make anything out of it.
Updating service [getresource] (this may take several minutes)...failed.
DEBUG: (gcloud.app.deploy) Error Response: [13] An internal error occurred during deployment.
Traceback (most recent call last):
File "C:\Users\M1044921\AppData\Local\Google\Cloud SDK\google-cloud-sdk\lib\googlecloudsdk\calliope\cli.py", line 841, in Execute
resources = calliope_command.Run(cli=self, args=args)
File "C:\Users\M1044921\AppData\Local\Google\Cloud SDK\google-cloud-sdk\lib\googlecloudsdk\calliope\backend.py", line 770, in Run
resources = command_instance.Run(args)
File "C:\Users\M1044921\AppData\Local\Google\Cloud SDK\google-cloud-sdk\lib\surface\app\deploy.py", line 90, in Run
parallel_build=False)
File "C:\Users\M1044921\AppData\Local\Google\Cloud SDK\google-cloud-sdk\lib\googlecloudsdk\command_lib\app\deploy_util.py", line 620, in RunDeploy
flex_image_build_option=flex_image_build_option)
File "C:\Users\M1044921\AppData\Local\Google\Cloud SDK\google-cloud-sdk\lib\googlecloudsdk\command_lib\app\deploy_util.py", line 422, in Deploy
extra_config_settings)
File "C:\Users\M1044921\AppData\Local\Google\Cloud SDK\google-cloud-sdk\lib\googlecloudsdk\api_lib\app\appengine_api_client.py", line 207, in DeployService
poller=done_poller)
File "C:\Users\M1044921\AppData\Local\Google\Cloud SDK\google-cloud-sdk\lib\googlecloudsdk\api_lib\app\operations_util.py", line 315, in WaitForOperation
sleep_ms=retry_interval)
File "C:\Users\M1044921\AppData\Local\Google\Cloud SDK\google-cloud-sdk\lib\googlecloudsdk\api_lib\util\waiter.py", line 254, in WaitFor
sleep_ms, _StatusUpdate)
File "C:\Users\M1044921\AppData\Local\Google\Cloud SDK\google-cloud-sdk\lib\googlecloudsdk\api_lib\util\waiter.py", line 316, in PollUntilDone
sleep_ms=sleep_ms)
File "C:\Users\M1044921\AppData\Local\Google\Cloud SDK\google-cloud-sdk\lib\googlecloudsdk\core\util\retry.py", line 229, in RetryOnResult
if not should_retry(result, state):
File "C:\Users\M1044921\AppData\Local\Google\Cloud SDK\google-cloud-sdk\lib\googlecloudsdk\api_lib\util\waiter.py", line 310, in _IsNotDone
return not poller.IsDone(operation)
File "C:\Users\M1044921\AppData\Local\Google\Cloud SDK\google-cloud-sdk\lib\googlecloudsdk\api_lib\app\operations_util.py", line 184, in IsDone
encoding.MessageToPyValue(operation.error)))
OperationError: Error Response: [13] An internal error occurred during deployment.
ERROR: (gcloud.app.deploy) Error Response: [13] An internal error occurred during deployment.
Related
I am trying to deploy my JiraAPI flask app on gunicorn in centos VM.
i have used JIRA module for it.
Here is my class
class Jira:
#initilizing the class
def __init__(self):
# private variables
self.logger = logging.getLogger('jiraUtil')
self.__email = environ.get('email')
self.__secret = environ.get('secret')
self.logger.debug("Connecting to Jira API")
jira_options = {
'server': environ.get('baseUrl'),
}
self.jira_con = JIRA(jira_options, basic_auth=(self.__email, self.__secret),max_retries=0)
if not self.jira_con:
self.logger.error("Error connecting to Jira")
raise ValueError("Error connecting to Jia")
def createIssue(self,issue_dict) -> dict:
self.logger.info("inside jira ticket creation")
new_issue_body = {}
try:
new_issue = self.jira_con.create_issue(fields=issue_dict)
print('new_issue')
new_issue_body['id'] = new_issue.id
new_issue_body['key'] = new_issue.key
new_issue_body['self'] = new_issue.self
return new_issue_body
except Exception as e:
self.logger.error("Error creating SSD ticket: ",e)
return new_issue_body
In my app.py
#app.route('/api/createTicket/', methods=['POST'])
def createTicket():
app.logger.info('Creating Issue Ticket')
request_data = request.get_json()
fields_dict = request_data.get('fields',None)
if fields_dict == None :
return jsonify({"result":"failure","status":"400", "message":"Bad Request"}),400
jiraObj = JiraUtil.Jira()
newIssueKey = jiraObj.createIssue(fields_dict)
if newIssueKey == None:
return jsonify({"result":"failure","status":"503", "message":"service Unavailable"}),500
return newIssueKey,200
if __name__ == '__main__':
app.run()
When i am running the app using flask run -h 0.0.0.0 -p 5000it is working fine and I am able to connect with my local postman
the logs got generated :
INFO:werkzeug:_[31m_[1mWARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead._[0m
* Running on all addresses (0.0.0.0)
* Running on http://127.0.0.1:5000
* Running on http://192.168.113.24:5000
INFO:werkzeug:_[33mPress CTRL+C to quit_[0m
INFO:app:Creating Issue Ticket
DEBUG:jiraUtil:Connecting to Jira API
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): codecharlee.atlassian.net:443
DEBUG:urllib3.connectionpool:https://codecharlee.atlassian.net:443 "GET /rest/api/2/serverInfo HTTP/1.1" 200 None
INFO:jiraUtil:inside jira ticket creation
but when i am running
gunicorn --bind 0.0.0.0:5000 app:app
it starts, but gets an error when hitting the api
INFO:app:Creating Issue Ticket
DEBUG:jiraUtil:Connecting to Jira API
ERROR:app:Exception on /api/createTicket/ [POST]
Traceback (most recent call last):
File "/opt/ctier/JenkinsPythonAPI/venv/lib/python3.8/site-packages/flask/app.py", line 2525, in wsgi_app
response = self.full_dispatch_request()
File "/opt/ctier/JenkinsPythonAPI/venv/lib/python3.8/site-packages/flask/app.py", line 1822, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/opt/ctier/JenkinsPythonAPI/venv/lib/python3.8/site-packages/flask/app.py", line 1820, in full_dispatch_request
rv = self.dispatch_request()
File "/opt/ctier/JenkinsPythonAPI/venv/lib/python3.8/site-packages/flask/app.py", line 1796, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
File "/opt/ctier/JenkinsPythonAPI/app.py", line 25, in createTicket
jiraObj = JiraUtil.Jira()
File "/opt/ctier/JenkinsPythonAPI/JiraUtil.py", line 21, in __init__
self.jira_con = JIRA(jira_options, basic_auth=(self.__email, self.__secret),max_retries=0)
File "/opt/ctier/JenkinsPythonAPI/venv/lib/python3.8/site-packages/jira/client.py", line 516, in __init__
assert isinstance(self._options["server"], str) # to help mypy
AssertionError
Please help
The problem is that gunicorn is not able to read the environment variables that you have declared in the .env file.
So, just configure a service file app.service in /etc/systemd/system and add EnvironmentFile=/path_to_env_file under [service] and start the service.
For more info on configuring the env file in gunicorn service, you can see
here.
I have a problem configuring Endpoints API. Any code i use, from my own, to google's examples on site fail with the same traceback
WARNING 2016-11-01 06:16:48,279 client.py:229] no scheduler thread, scheduler.run() will be invoked by report(...)
Traceback (most recent call last):
File "/home/vladimir/projects/sb_fork/sb/lib/vendor/google/api/control/client.py", line 225, in start
self._thread.start()
File "/home/vladimir/sdk/google-cloud-sdk/platform/google_appengine/google/appengine/api/background_thread/background_thread.py", line 108, in start
start_new_background_thread(self.__bootstrap, ())
File "/home/vladimir/sdk/google-cloud-sdk/platform/google_appengine/google/appengine/api/background_thread/background_thread.py", line 87, in start_new_background_thread
raise ERROR_MAP[error.application_error](error.error_detail)
FrontendsNotSupported
INFO 2016-11-01 06:16:48,280 client.py:327] created a scheduler to control flushing
INFO 2016-11-01 06:16:48,280 client.py:330] scheduling initial check and flush
INFO 2016-11-01 06:16:48,288 client.py:804] Refreshing access_token
/home/vladimir/projects/sb_fork/sb/lib/vendor/urllib3/contrib/appengine.py:113: AppEnginePlatformWarning: urllib3 is using URLFetch on Google App Engine sandbox instead of sockets. To use sockets directly instead of URLFetch see https://urllib3.readthedocs.io/en/latest/contrib.html.
AppEnginePlatformWarning)
ERROR 2016-11-01 06:16:49,895 service_config.py:125] Fetching service config failed (status code 403)
ERROR 2016-11-01 06:16:49,896 wsgi.py:263]
Traceback (most recent call last):
File "/home/vladimir/sdk/google-cloud-sdk/platform/google_appengine/google/appengine/runtime/wsgi.py", line 240, in Handle
handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
File "/home/vladimir/sdk/google-cloud-sdk/platform/google_appengine/google/appengine/runtime/wsgi.py", line 299, in _LoadHandler
handler, path, err = LoadObject(self._handler)
File "/home/vladimir/sdk/google-cloud-sdk/platform/google_appengine/google/appengine/runtime/wsgi.py", line 85, in LoadObject
obj = __import__(path[0])
File "/home/vladimir/projects/sb_fork/sb/main.py", line 27, in <module>
api_app = endpoints.api_server([SolarisAPI,], restricted=False)
File "/home/vladimir/projects/sb_fork/sb/lib/vendor/endpoints/apiserving.py", line 497, in api_server
controller)
File "/home/vladimir/projects/sb_fork/sb/lib/vendor/google/api/control/wsgi.py", line 77, in add_all
a_service = loader.load()
File "/home/vladimir/projects/sb_fork/sb/lib/vendor/google/api/control/service.py", line 110, in load
return self._load_func(**kw)
File "/home/vladimir/projects/sb_fork/sb/lib/vendor/google/api/config/service_config.py", line 78, in fetch_service_config
_log_and_raise(Exception, message_template.format(status_code))
File "/home/vladimir/projects/sb_fork/sb/lib/vendor/google/api/config/service_config.py", line 126, in _log_and_raise
raise exception_class(message)
Exception: Fetching service config failed (status code 403)
INFO 2016-11-01 06:16:49,913 module.py:788] default: "GET / HTTP/1.1" 500 -
My app.yaml is configured like the new Endpoints Migrating to 2.0 document states:
- url: /_ah/api/.*
script: api.solaris.api_app
And main.py imports the API into the app:
api_app = endpoints.api_server([SolarisAPI,], restricted=False)
I use Google Cloud SDK with these versions:
Google Cloud SDK 132.0.0
app-engine-python 1.9.40
bq 2.0.24
bq-nix 2.0.24
core 2016.10.24
core-nix 2016.10.24
gcloud
gsutil 4.22
gsutil-nix 4.22
Have you tried generating and uploading the OpenAPI configuration for the service? See the sections named "Generating the OpenAPI configuration file" and "Deploying the OpenAPI configuration file" in the python library documentation.
Note that in step 2 of the generation process, you may need to prepend python to the command (e.g python lib/endpoints/endpointscfg.py get_swagger_spec ...), since the PyPi package doesn't preserve executable file permissions right now.
To get rid of the "FrontendsNotSupported" you need to use a "B*" instance class.
The error "Exception: Fetching service config failed" should be gone if you follow the steps in https://cloud.google.com/endpoints/docs/frameworks/python/quickstart-frameworks-python. As already pointed out by Brad, the section "OpenAPI configuration" and the resulting environment variables are required to make the service configuration work.
I have tried to get a group settings with group settings API using Google App Engine 1.7.5 with python 2.5 following a simple example here.
this is my app.yaml:
application: {{APP_ID}}
version: 1
runtime: python
api_version: 1
handlers:
- url: /.*
script: main.py
and my main.py is like following:
def main():
application = webapp.WSGIApplication([('/', TestHandler)])
run_wsgi_app(application)
class TestHandler(webapp.RequestHandler):
"""Handles a request for the query page."""
def get(self):
self.response.out.write(Test())
def Test():
credentials = AppAssertionCredentials(scope='https://www.googleapis.com/auth/apps.groups.settings')
http = credentials.authorize(httplib2.Http(memcache))
service = build('groupssettings', 'v1', http=http)
group = service.groups().get(groupUniqueId='{{GROUP_ID}}').execute()
logging.error(group)
if __name__=="__main__":
main()
and this is the stacktrace telling me invalid credentials!
Traceback (most recent call last):
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\ext\webapp\_webapp25.py", line 714, in __call__
handler.get(*groups)
File "D:\Projets\GS\main.py", line 26, in get
self.response.out.write(Test())
File "D:\Projets\GS\main.py", line 41, in Test
group = service.groups().get(groupUniqueId='{{GROUP_ID}}').execute()
File "D:\Projets\GS\oauth2client\util.py", line 132, in positional_wrapper
return wrapped(*args, **kwargs)
File "D:\Projets\GS\apiclient\http.py", line 723, in execute
raise HttpError(resp, content, uri=self.uri)
HttpError: <HttpError 401 when requesting https://www.googleapis.com/groups/v1/groups/{{GROUP_ID}}?alt=json returned "Invalid Credentials">
INFO 2014-11-21 15:51:45,423 dev_appserver.py:3104] "GET / HTTP/1.1" 500 -
INFO 2014-11-21 15:51:45,611 dev_appserver.py:3104] "GET /favicon.ico HTTP/1.1" 404 -
Have anyone got this error before or have an idea of the root cause?
Have you turned on access to the groups API through the developer console for the project? This is a simple step that is often missed by people trying to get APIs going.
Also, I'd recommend using the decorator patterns for OAuth as described here to simplify the reasoning around authorization flows.
In addition, your app.yaml should specify python27 rather than just python. App Engine runs python27.
The following code works in the Python interactive shell:
import urllib2
result = urllib2.urlopen("http://www.google.com/")
and gives a 200 result.
If I run the same code in an AppEngine app running locally with the development server, it fails with the following error:
URLError: <urlopen error An error occured while connecting to the server:
Unable to fetch URL: http://www.google.com/
Error: [Errno 11004] getaddrinfo failed>`
I've tried using the urlfetch library directly:
from google.appengine.api import urlfetch
result = urlfetch.fetch("http://www.google.com")
and this also fails (which makes sense, as I believe urllib2 within AppEngine calls URLFetch internally?)
I can clearly access the URL from my local machine - so what's happening?
UPDATE: the relevant stack trace:
File "c:\dev\repos\stackoverflow\main.py", line 40, in get_latest_comments
result = urlfetch.fetch("http://www.google.com")
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\api\urlfetch.py", line 266, in fetch
return rpc.get_result()
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\api\apiproxy_stub_map.py", line 604, in get_result
return self.__get_result_hook(self)
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\api\urlfetch.py", line 397, in _get_fetch_result
raise DownloadError("Unable to fetch URL: " + url + error_detail)
DownloadError: Unable to fetch URL: http://www.google.com Error: [Errno 11004] getaddrinfo failed
DO you have a proxy configured with environment variables? The dev_appserver clears all env vars.
I ran into an error that was painful to track down, so I thought I'd add the cause + "solution" here.
The setup:
Devbox - Running Google App Engine listening on all ports ("--address=0.0.0.0") serving a URL that launches a task.
Client - Client (Python requests library) which queries the callback URL
App Engine code:
class StartTaskCallback(webapp.RequestHandler):
def post(self):
param = self.request.get('param')
logging.info('STARTTASK: %s' % param)
# launch a task
taskqueue.add(url='/tasks/mytask',
queue_name='myqueue',
params={'param': param})
class MyTask(webapp.RequestHandler):
def post(self):
param = self.request.get('param')
logging.info('MYTASK: param = %s' % param)
When I queried the callback with my browser, everything worked, but the same query from the remote client gave me the following error:
ERROR 2012-03-23 21:18:27,351 taskqueue_stub.py:1858] An error occured while sending the task "task1" (Url: "/tasks/mytask") in queue "myqueue". Treating as a task error.
Traceback (most recent call last):
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/taskqueue/taskqueue_stub.py", line 1846, in ExecuteTask
connection.endheaders()
File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/httplib.py", line 868, in endheaders
self._send_output()
File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/httplib.py", line 740, in _send_output
self.send(msg)
File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/httplib.py", line 699, in send
self.connect()
File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/httplib.py", line 683, in connect
self.timeout)
File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/socket.py", line 498, in create_connection
for res in getaddrinfo(host, port, 0, SOCK_STREAM):
gaierror: [Errno 8] nodename nor servname provided, or not known
This error would just spin in a loop as the task retried. Though oddly, I could go to Admin -> Task Queues and click 'Run' to get the task to complete successfully.
At first I thought this was an error with the binding. I would not get an error if I queried the StartTaskCallback via the browser or if I ran the client locally.
Finally I noticed that App Engine is using the 'host' field of the request in order to build an absolute URL for the task. In /Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/taskqueue/taskqueue_stub.py (1829):
connection_host, = header_dict.get('host', [self._default_host])
if connection_host is None:
logging.error('Could not determine where to send the task "%s" '
'(Url: "%s") in queue "%s". Treating as an error.',
task.task_name(), task.url(), queue.queue_name)
return False
connection = httplib.HTTPConnection(connection_host)
In my case, I was using a special name + hosts file on the remote client to access the server.
192.168.1.208 devbox
So the 'host' for the remote client looked like 'devbox:8085' which the local server could not resolve.
To fix the issue, I simply added devbox to my AppEngine server's hosts file, but it sure would have been nice if the gaierror exception had printed the name it failed to resolve, or if App Engine didn't use the 'host' of the incoming request to build a URL for task creation.