Importing python modules in azure webjob - python

I have been attempting to deploy a web scraper written in python for the past few weeks on azure. I initially tried to do this in an azure app service by building pushing a docker image to the service. I have had success previously with this method when deploying a flask rest api. Unfortunately the time out limit of the azure app service meant that the web scraper container is terminated as it is does not give a proper response when azure attempts to get a response - this option won't work.
I have since tried setting up a windows based app service in order to create an azure WebJob however, this has the problem of being capped at python 3.6 which i believe is causing import problems. I cannot import the "requests" module which is essential for the scraper to work correctly. I have a requirements.txt inside the zip i upload for the webjob but this doesnt seem to allow the importing of this module either. Is there a way to import modules from inside the webjob?
Below is the error i am receiving in the webjob logs :
[12/19/2020 16:36:48 > 658fd3: SYS INFO] Run script 'run.py' with script host - 'PythonScriptHost'
[12/19/2020 16:36:48 > 658fd3: SYS INFO] Status changed to Running
[12/19/2020 16:36:48 > 658fd3: ERR ] Traceback (most recent call last):
[12/19/2020 16:36:48 > 658fd3: ERR ] File "run.py", line 12, in <module>
[12/19/2020 16:36:48 > 658fd3: ERR ] import requests
[12/19/2020 16:36:48 > 658fd3: ERR ] ModuleNotFoundError: No module named 'requests'

Search Advanced Tools in Filter. Click on the GO Link. Then choose Debug Console and click on CMD.
Find your Source Code directory path. Then type in the path:
python -m pip install - requirements.txt
d:\ |-- home
|-- site
|-- wwwroot
|-- App_Data
|-- jobs
|-- triggered
| -- continuous

Related

Failed to find Flask application or factory in module 'app'. Use 'FLASK_APP=app:name' to specify one

>>> flask run
* Serving Flask app 'app' (lazy loading)
* Environment: development
* Debug mode: on
Usage: flask run [OPTIONS]
Try 'flask run --help' for help.
Error: Failed to find Flask application or factory in module 'app'. Use 'FLASK_APP=app:name' to specify one.
Getting this error when trying to make a package from my simple flask app
Also it can be another error:
Error: While importing 'app', an ImportError was raised.
And maybe it is
>>> flask run
* Serving Flask app 'app' (lazy loading)
* Environment: development
* Debug mode: on
Usage: flask run [OPTIONS]
Try 'flask run --help' for help.
Error: Could not import 'app.app'.
But I'm doing everything by the docs "Large Applications as Packages":
https://flask.palletsprojects.com/en/2.0.x/patterns/packages/
In my case renaming app name from app to another one helped.
Maybe it was some kind of namespace error, because my package was ./app and Flask app object variable in __init__.py was named app (Despite Miguel in his tutorial did it the same way)
In general, while developing web applications you have to be careful with names like wsgi - they can be reserved.
I am executing set FLASK_APP=module, set FLASK_ENV=development, pip install -e . and flask run commands in the first (upper) module directory.
And maybe you only need to revert changes from flask doc tutorial returning to the file structure without __init__.py and with normal app.py. After that delete all __pycache__ folders and repeat this tutorial again

Choosing an appropriate azure service to deploy repetitive python task

I have been attempting to deploy a web scraper written in python for the past few weeks on azure. I initially tried to do this in an azure app service by building pushing a docker image to the service. I have had success previously with this method when deploying a flask rest api. Unfortunately the time out limit of the azure app service meant that the web scraper container is terminated as it is does not give a proper response when azure attempts to get a response.
I have since tried setting up a windows based app service in order to create an azure WebJob however, this has the problem of being capped at python 3.6 which i believe is causing import problems. I cannot import the "requests" module which is essential for the scraper to work correctly. I have a requirements.txt inside the zip i upload for the webjob but this doesnt seem to allow the importing of this module either. Is there a way to import modules from inside the webjob or is there another/better way to deploy a repetitive task on azure that doesnt have to give an api response like a normal azure app service deployment.
Below is the error i am receiving in the webjob logs :
[12/19/2020 16:36:48 > 658fd3: SYS INFO] Run script 'run.py' with script host - 'PythonScriptHost'
[12/19/2020 16:36:48 > 658fd3: SYS INFO] Status changed to Running
[12/19/2020 16:36:48 > 658fd3: ERR ] Traceback (most recent call last):
[12/19/2020 16:36:48 > 658fd3: ERR ] File "run.py", line 12, in <module>
[12/19/2020 16:36:48 > 658fd3: ERR ] import requests
[12/19/2020 16:36:48 > 658fd3: ERR ] ModuleNotFoundError: No module named 'requests'
Any help is greatly appreciated thank you.
You can use Azure's Serverless feature called Functions. If you want to make more use of it, then make a docker container with any image you want and upload it as your function, but beware of cold starts

Issue running Python37 on Google App Engine

I know this question has been asked in some facet or another, but I have gone through readings as shown here and I am still not seeing where my issue is as I am still unable to publish my Django 2.1.1 app in the Python37 environment in Google App Engine:
Python 3 Django on App Engine Standard: App Fails to Start
Overall what I am attempting to do is publish a simple app engine app using:
gcloud app deploy
My application works locally but when I publish, it goes through without issue, but I get the annoying:
500 Server Error message
When I look at the logs in Google I get the same error as many others have gotten:
ModuleNotFoundError: No module named 'main'
here is my relevant directory structure
project_portal
project_portal
init.py
settings.py
urls.py
wsgi.py
main.py
app.yaml
requirements.txt
my app.yaml file
runtime: python37
entrypoint: gunicorn -b :$PORT project_portal.wsgi
env: standard
handlers:
- url: .*
secure: always
redirect_http_response_code: 301
script: project_portal.wsgi.application
my project_portal/wsgi.py file
import os
from django.core.wsgi import get_wsgi_application
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'project_portal.settings')
application = get_wsgi_application()
from main.py in project root
from project_portal.wsgi import application
from requirements.txt
django == 2.1.1
Without an entrypoint defined, it will try to start from a file called main.py. Create one, at the same level as app.yamlwith some simple logic in it:
from project_portal.wsgi import application
Edit:
Since you have an entrypoint defined, it seems that is the issue. So, make sure you have gunicorn in your 'requirements.txt':
gunicorn==19.9.0
This did not help:
Try changing
entrypoint: gunicorn -b :$PORT project_portal.wsgi
to
entrypoint: gunicorn -b :$PORT project_portal.wsgi:application

Cannot import subprocess.call when running google app engine locally

I am beginning to test google app engine for running my flask app. I can run the app directly using flask run without a problem. My app.yaml looks like this
runtime: python27
api_version: 1
threadsafe: true
# [START handlers]
handlers:
- url: /static
static_dir: CameraMeerat/static
- url: /.*
script: CameraMeerkat.app
# [END handlers]
When running dev_appserver.py I get
File "C:\Users\Ben\Documents\CameraMeerkat\Frontend\CameraMeerkat\commands.py", line 5, in <module>
from subprocess import call
ImportError: cannot import name call
INFO 2017-07-21 21:26:37,585 module.py:813] default: "GET / HTTP/1.1" 500 -
From my python shell I can run that command
from subprocess import call
help(call)
Help on function call in module subprocess:
call(*popenargs, **kwargs)
Run command with arguments. Wait for command to complete, then
return the returncode attribute.
The arguments are the same as for the Popen constructor. Example:
retcode = call(["ls", "-l"])
What might be going on here? Subprocess is not a module that can be installed, or really messed with. Similar to unanswered here
ImportError: cannot import name Popen google cloud compute delpoyment error ?
This is one of the limitation imposed by the GAE standard environment Python sandbox. From The sandbox (emphasis mine):
An App Engine application cannot:
write to the filesystem. Applications must use Cloud Datastore for storing persistent data. Reading from the filesystem is allowed, and
all application files uploaded with the application are available.
respond slowly. A web request to an application must be handled within a few seconds. Processes that take a very long time to respond
are terminated to avoid overloading the web server.
make other kinds of system calls.
One of the disallowed system calls is subprocess.call. The development server will raise an exception as it contains a modified version of subprocess.
If your app requires such call you may need to switch to the flexible enviroment. See also Choosing your App Engine environment

Error using Google App Engine cloudshell to upload Python app

I'm trying to use appcfg to upload my helloworld app on GCloud using GAE. According to Google's own quickstarter site, the syntax is:
appcfg.py -A YOUR_PROJECT_ID_ -V v1 update helloworld/
So I did:
appcfg.py -A project_name_123456 -V v1 update c:/users/user/helloworld/
Which makes this error:
"appcfg.py: error: Not a directory: C:/users/user/helloworld"
What am I doing wrong? I have the app.yaml file in C:/users/user/helloworld, and I've tried lots of different things. Thanks

Categories