We Keep Coding

Why Apache server throws syntax error with snowflake ingest module in python

I have a flask application running on apache server in the Linux box(ec2 instance), which calls the python script that does the work of moving the files from s3 to snowflake. The script works fine if executed directly. It throws the below error for the snowflake's ingest python module when trying to host it on apache using virtualhost and mod_wsgi. The module is installed in the default site packages.
Apache server version: 2.4.41
Python Version: 3.7
snowflake: snowflake_connector_python-2.1.3-py3.7-nspkg.pth
OS: AMAZON linux 2
 
Below is the apache error log, when i try to restart:
[Fri Jan 10 15:47:53.703083 2020] [:error] [pid 19755] [remote 67.79.202.36:20] mod_wsgi (pid=19755): Target WSGI script '/var/www/FLASKAPPS/snowflakeingestapp/snowflakeingestapp.wsgi' cannot be loaded as Python module.
[Fri Jan 10 15:47:53.703113 2020] [:error] [pid 19755] [remote 67.79.202.36:20] mod_wsgi (pid=19755): Exception occurred processing WSGI script '/var/www/FLASKAPPS/snowflakeingestapp/snowflakeingestapp.wsgi'.
[Fri Jan 10 15:47:53.703129 2020] [:error] [pid 19755] [remote 67.79.202.36:20] Traceback (most recent call last):
[Fri Jan 10 15:47:53.703142 2020] [:error] [pid 19755] [remote 67.79.202.36:20]  File "/var/www/FLASKAPPS/snowflakeingestapp/snowflakeingestapp.wsgi", line 4, in
[Fri Jan 10 15:47:53.703176 2020] [:error] [pid 19755] [remote 67.79.202.36:20]    from snowflakeingestapp import app as application
[Fri Jan 10 15:47:53.703181 2020] [:error] [pid 19755] [remote 67.79.202.36:20]  File "/var/www/FLASKAPPS/snowflakeingestapp/init.py", line 6, in
[Fri Jan 10 15:47:53.703206 2020] [:error] [pid 19755] [remote 67.79.202.36:20]    import snowflake_ingest
[Fri Jan 10 15:47:53.703210 2020] [:error] [pid 19755] [remote 67.79.202.36:20]  File "/var/www/FLASKAPPS/snowflakeingestapp/snowflake_ingest/init.py", line 1, in
[Fri Jan 10 15:47:53.703225 2020] [:error] [pid 19755] [remote 67.79.202.36:20]    import ingest_ecomm_json_snowpipe
[Fri Jan 10 15:47:53.703230 2020] [:error] [pid 19755] [remote 67.79.202.36:20]  File "/var/www/FLASKAPPS/snowflakeingestapp/snowflake_ingest/ingest_ecomm_json_snowpipe.py", line 20, in
[Fri Jan 10 15:47:53.703296 2020] [:error] [pid 19755] [remote 67.79.202.36:20]    from snowflake.ingest import SimpleIngestManager
[Fri Jan 10 15:47:53.703300 2020] [:error] [pid 19755] [remote 67.79.202.36:20]  File "/usr/local/lib/python3.7/site-packages/snowflake/ingest/init.py", line 1, in
[Fri Jan 10 15:47:53.703315 2020] [:error] [pid 19755] [remote 67.79.202.36:20]    from .simple_ingest_manager import SimpleIngestManager, StagedFile
[Fri Jan 10 15:47:53.703342 2020] [:error] [pid 19755] [remote 67.79.202.36:20]  File "/usr/local/lib/python3.7/site-packages/snowflake/ingest/simple_ingest_manager.py", line 65
[Fri Jan 10 15:47:53.703345 2020] [:error] [pid 19755] [remote 67.79.202.36:20]    def init(self, account: Text, user: Text, pipe: Text, private_key: Text,
[Fri Jan 10 15:47:53.703347 2020] [:error] [pid 19755] [remote 67.79.202.36:20]                              ^
[Fri Jan 10 15:47:53.703350 2020] [:error] [pid 19755] [remote 67.79.202.36:20] SyntaxError: invalid syntax

It looks like when you're executing this from mod_wsgi, you're using Python 2 instead of Python 3. The error message you see is happening because Snowflake's simple ingest manager uses type hints (via the typing module), which are not available in Python 2.
Here's a simple fragment that fails with the same error in Python 2 but succeeds in Python 3:
from typing import Text
def foo(bar: Text):
print(bar)
foo('huh')

Importing Flask application to Elastic Beanstalk error: can't open file 'mod_wsgi': [Errno 2] No such file or directory

I am attempting to put my Flask application on Elastic Beanstalk (Amazon Web Services, EC2), but after uploading the files I receive a 500 error. Any thoughts on what could be the issue? The error log:
/opt/python/run/venv/bin/python3: can't open file 'mod_wsgi': [Errno 2] No such file or directory
[Fri Sep 08 03:23:05.754341 2017] [:error] [pid 13619] [remote 128.12.244.5:31824] mod_wsgi (pid=13619): Target WSGI script '/opt/python/current/app/application.py' cannot be loaded as Python module.
[Fri Sep 08 03:23:05.754577 2017] [:error] [pid 13619] [remote 128.12.244.5:31824] mod_wsgi (pid=13619): SystemExit exception raised by WSGI script '/opt/python/current/app/application.py' ignored.
[Fri Sep 08 03:23:05.754729 2017] [:error] [pid 13619] [remote 128.12.244.5:31824] Traceback (most recent call last):
[Fri Sep 08 03:23:05.754992 2017] [:error] [pid 13619] [remote 128.12.244.5:31824] File "/opt/python/current/app/application.py", line 11, in <module>
[Fri Sep 08 03:23:05.755007 2017] [:error] [pid 13619] [remote 128.12.244.5:31824] application.run(debug=True)
[Fri Sep 08 03:23:05.755116 2017] [:error] [pid 13619] [remote 128.12.244.5:31824] File "/opt/python/run/venv/lib/python3.4/site-packages/flask/app.py", line 772, in run
[Fri Sep 08 03:23:05.755131 2017] [:error] [pid 13619] [remote 128.12.244.5:31824] run_simple(host, port, self, **options)
[Fri Sep 08 03:23:05.755242 2017] [:error] [pid 13619] [remote 128.12.244.5:31824] File "/opt/python/run/venv/lib/python3.4/site-packages/werkzeug/serving.py", line 737, in run_simple
[Fri Sep 08 03:23:05.755257 2017] [:error] [pid 13619] [remote 128.12.244.5:31824] reloader_type)
[Fri Sep 08 03:23:05.755344 2017] [:error] [pid 13619] [remote 128.12.244.5:31824] File "/opt/python/run/venv/lib/python3.4/site-packages/werkzeug/_reloader.py", line 265, in run_with_reloader
[Fri Sep 08 03:23:05.755358 2017] [:error] [pid 13619] [remote 128.12.244.5:31824] sys.exit(reloader.restart_with_reloader())
[Fri Sep 08 03:23:05.755429 2017] [:error] [pid 13619] [remote 128.12.244.5:31824] SystemExit: 2
My file structure is as follows:
file hierarchy
My WSGI path is
config

Turns out I needed to wrap application.run() within a if __name__ == '__main__': block. This ensures that a local WSGI server is not loaded by default.
Source: http://flask.pocoo.org/docs/0.11/deploying/mod_wsgi/

Issue with WSGI and Django 1.8

Please help to understand what is going on.
I have a project in Django. It's working correctly without WSGI (I mean via manage.py), but when I move to WSGI I got the problems.
There is httpd error logs :
[Mon May 02 16:58:23.316772 2016] [ssl:info] [pid 16850] [client IP:29158] AH01964: Connection to child 1 established (server server:443)
[Mon May 02 16:58:23.352418 2016] [socache_shmcb:debug] [pid 16850] mod_socache_shmcb.c(485): AH00831: socache_shmcb_store (0x81 -> subcache 1)
[Mon May 02 16:58:23.352488 2016] [socache_shmcb:debug] [pid 16850] mod_socache_shmcb.c(810): AH00847: insert happened at idx=0, data=(0:32)
[Mon May 02 16:58:23.352500 2016] [socache_shmcb:debug] [pid 16850] mod_socache_shmcb.c(815): AH00848: finished insert, subcache: idx_pos/idx_used=0/1, data_pos/data_used=0/180
[Mon May 02 16:58:23.352542 2016] [socache_shmcb:debug] [pid 16850] mod_socache_shmcb.c(506): AH00834: leaving socache_shmcb_store successfully
[Mon May 02 16:58:23.352600 2016] [ssl:debug] [pid 16850] ssl_engine_kernel.c(1812): [client IP:29158] AH02041: Protocol: TLSv1, Cipher: RC4-SHA (128/128 bits)
[Mon May 02 16:58:23.370771 2016] [ssl:debug] [pid 16850] ssl_engine_kernel.c(224): [client IP:29158] AH02034: Initial (No.1) HTTPS request received for child 1 (server server:443)
[Mon May 02 16:58:23.371004 2016] [authz_core:debug] [pid 16850] mod_authz_core.c(809): [client IP:29158] AH01626: authorization result of Require all granted: granted
[Mon May 02 16:58:23.371015 2016] [authz_core:debug] [pid 16850] mod_authz_core.c(809): [client IP:29158] AH01626: authorization result of <RequireAny>: granted
[Mon May 02 16:58:23.371202 2016] [authz_core:debug] [pid 16850] mod_authz_core.c(809): [client IP:29158] AH01626: authorization result of Require all granted: granted
[Mon May 02 16:58:23.371211 2016] [authz_core:debug] [pid 16850] mod_authz_core.c(809): [client IP:29158] AH01626: authorization result of <RequireAny>: granted
[Mon May 02 16:58:23.371650 2016] [:info] [pid 16839] [remote IP] mod_wsgi (pid=16839, process='project', application=''): Loading WSGI script '/usr/local/env/project/apache/project.wsgi'.
[Mon May 02 16:58:23.849532 2016] [:error] [pid 16839] [remote IP] mod_wsgi (pid=16839): Target WSGI script '/usr/local/env/project/apache/project.wsgi' cannot be loaded as Python module.
[Mon May 02 16:58:23.849596 2016] [:error] [pid 16839] [remote IP] mod_wsgi (pid=16839): Exception occurred processing WSGI script '/usr/local/env/project/apache/project.wsgi'.
[Mon May 02 16:58:23.849636 2016] [:error] [pid 16839] [remote IP] Traceback (most recent call last):
[Mon May 02 16:58:23.849671 2016] [:error] [pid 16839] [remote IP] File "/usr/local/env/project/apache/project.wsgi", line 13, in <module>
[Mon May 02 16:58:23.849742 2016] [:error] [pid 16839] [remote IP] application = get_wsgi_application()
[Mon May 02 16:58:23.849775 2016] [:error] [pid 16839] [remote IP] File "/usr/local/env/project_django18/lib/python2.7/site-packages/django/core/wsgi.py", line 14, in get_wsgi_application
[Mon May 02 16:58:23.849832 2016] [:error] [pid 16839] [remote IP] django.setup()
[Mon May 02 16:58:23.849850 2016] [:error] [pid 16839] [remote IP] File "/usr/local/env/project_django18/lib/python2.7/site-packages/django/__init__.py", line 18, in setup
[Mon May 02 16:58:23.849900 2016] [:error] [pid 16839] [remote IP] apps.populate(settings.INSTALLED_APPS)
[Mon May 02 16:58:23.849920 2016] [:error] [pid 16839] [remote IP] File "/usr/local/env/project_django18/lib/python2.7/site-packages/django/apps/registry.py", line 108, in populate
[Mon May 02 16:58:23.850096 2016] [:error] [pid 16839] [remote IP] app_config.import_models(all_models)
[Mon May 02 16:58:23.850125 2016] [:error] [pid 16839] [remote IP] File "/usr/local/env/project_django18/lib/python2.7/site-packages/django/apps/config.py", line 198, in import_models
[Mon May 02 16:58:23.850224 2016] [:error] [pid 16839] [remote IP] self.models_module = import_module(models_module_name)
[Mon May 02 16:58:23.850246 2016] [:error] [pid 16839] [remote IP] File "/usr/lib64/python2.7/importlib/__init__.py", line 37, in import_module
[Mon May 02 16:58:23.850303 2016] [:error] [pid 16839] [remote IP] __import__(name)
[Mon May 02 16:58:23.850324 2016] [:error] [pid 16839] [remote IP] File "/usr/local/env/project/config/models.py", line 10, in <module>
[Mon May 02 16:58:23.851120 2016] [:error] [pid 16839] [remote IP] from webui.models import Change
[Mon May 02 16:58:23.851181 2016] [:error] [pid 16839] [remote IP] ImportError: No module named models
[Mon May 02 16:58:23.851589 2016] [ssl:debug] [pid 16850] ssl_engine_io.c(992): [client IP:29158] AH02001: Connection closed to child 1 with standard shutdown (server server:443)
Why WSGI can't import it?
There is my wsgi.conf
#!/usr/local/env/project_django18/bin/python
import os
import sys
sys.path.append('/usr/local/env/project/')
import project
os.environ['DJANGO_SETTINGS_MODULE'] = 'project.settings'
activate_this = os.path.expanduser("/usr/local/env/project_django18/bin/activate_this.py")
execfile(activate_this, dict(__file__=activate_this))
from django.core.wsgi import get_wsgi_application
application = get_wsgi_application()
P.S. webui it's just application of Django. And Change is one of models of it.
P.S.S. If need additional information please give me know

Configuration file for Flask application

I'm new to Python, so please bear with me.
I am attempting to create a file in which to store my configuration settings in a Flask project. However, I seem to be getting errors when I attempt to import the file.
Here's my configuration file (location: app/config.py):
database_uri = 'something here'
secret_key = something here"
And here's where I'm using it (location: app/models.py):
from app import config
...
app.config['SQLALCHEMY_DATABASE_URI'] = config.database_uri
However, I seem to be getting this error when launching the application:
[Sat Aug 08 19:00:15.539773 2015] [:error] [pid 29784] [client 188.183.57.54:64122] mod_wsgi (pid=29784): Target WSGI script '/var/www/pwforum/pwforum.wsgi' cannot be loaded as Python module.
[Sat Aug 08 19:00:15.540014 2015] [:error] [pid 29784] [client 188.183.57.54:64122] mod_wsgi (pid=29784): Exception occurred processing WSGI script '/var/www/pwforum/pwforum.wsgi'.
[Sat Aug 08 19:00:15.540146 2015] [:error] [pid 29784] [client 188.183.57.54:64122] Traceback (most recent call last):
[Sat Aug 08 19:00:15.540250 2015] [:error] [pid 29784] [client 188.183.57.54:64122] File "/var/www/pwforum/pwforum.wsgi", line 7, in <module>
[Sat Aug 08 19:00:15.540448 2015] [:error] [pid 29784] [client 188.183.57.54:64122] from app import app as application
[Sat Aug 08 19:00:15.540537 2015] [:error] [pid 29784] [client 188.183.57.54:64122] File "/var/www/pwforum/app/__init__.py", line 12, in <module>
[Sat Aug 08 19:00:15.540685 2015] [:error] [pid 29784] [client 188.183.57.54:64122] from app import views, models
[Sat Aug 08 19:00:15.540773 2015] [:error] [pid 29784] [client 188.183.57.54:64122] File "/var/www/pwforum/app/views.py", line 3, in <module>
[Sat Aug 08 19:00:15.541061 2015] [:error] [pid 29784] [client 188.183.57.54:64122] from app.models import db, User, Category, Topic, Post
[Sat Aug 08 19:00:15.541154 2015] [:error] [pid 29784] [client 188.183.57.54:64122] File "/var/www/pwforum/app/models.py", line 11, in <module>
[Sat Aug 08 19:00:15.541333 2015] [:error] [pid 29784] [client 188.183.57.54:64122] app.config['SQLALCHEMY_DATABASE_URI'] = config.database_uri
[Sat Aug 08 19:00:15.541413 2015] [:error] [pid 29784] [client 188.183.57.54:64122] AttributeError: 'module' object has no attribute 'database_uri'

Your config file should look like this:
SQLALCHEMY_DATABASE_URI = '<your-db-driver>://<user>:<pw>#<db-url>'
SECRET_KEY = '<your-very-secret-key>'
Then you can do:
app = Flask(__name__)
app.config.from_object('config')

How to set environ variables in local development with virtualenv

I am currently trying to publish via ZIP my django app to an AWS elasticbeanstalk.
As I have a local and online development enviroment I would like to make use of the environ variables, that are used anyways by AWS.
For my development environments I am already using virtualenv. Apache runs with wsgi enabled and is supposed to use such environments. Unfortunately, it does not know the environ that I have set in the bin/activate.
Django throws a 500 error, since it cannot connect to the database as it does not have access to the environ vars:
1:53.862779 2015] [wsgi:error] [pid 20639] [remote 127.0.0.1:51896] KeyError: 'RDS_DB_NAME'
If I activate my env and look into the env vars:
(myenv)[bs#debian-gohan:]/var/www/vhosts/mysite $ env | grep RDS
RDS_HOSTNAME=localhost
... and I get the others as well...
My Apache Vhost is:
<VirtualHost *:80>
WSGIDaemonProcess mysite python-path=/home/bs/envs/myenv/bin/python2.7:/home/bs/envs/myenv/lib/python2.7/site-packages
WSGIProcessGroup mysite
WSGIScriptAlias / /var/www/vhosts/main-page/mysite/wsgi.py
Does anyone have an idea why the environ vars are not seen by the apache ?
EDIT 1:
I have tried to use SetEnv in the Apache config, like this:
SetEnv RDS_HOSTNAME "localhost"
Unfortunately, this did not work. I then removed the ProcessGroup and only used a simple WSGIPythonPath. That did not work either.
The error in the apache log remains the same:
[Thu Mar 26 11:30:34.046807 2015] [wsgi:info] [pid 23012] [client 127.0.0.1:55231] mod_wsgi (pid=23012, process='', application='mysite.dbz.dev|'): Loading WSGI script '/var/www/vhosts/main-page/mysite/wsgi.py'.
[Thu Mar 26 11:30:34.178717 2015] [wsgi:error] [pid 23012] [client 127.0.0.1:55231] mod_wsgi (pid=23012): Target WSGI script '/var/www/vhosts/main-page/mysite/wsgi.py' cannot be loaded as Python module.
[Thu Mar 26 11:30:34.178812 2015] [wsgi:error] [pid 23012] [client 127.0.0.1:55231] mod_wsgi (pid=23012): Exception occurred processing WSGI script '/var/www/vhosts/main-page/mysite/wsgi.py'.
[Thu Mar 26 11:30:34.178924 2015] [wsgi:error] [pid 23012] [client 127.0.0.1:55231] Traceback (most recent call last):
[Thu Mar 26 11:30:34.178947 2015] [wsgi:error] [pid 23012] [client 127.0.0.1:55231] File "/var/www/vhosts/main-page/mysite/wsgi.py", line 17, in <module>
[Thu Mar 26 11:30:34.179111 2015] [wsgi:error] [pid 23012] [client 127.0.0.1:55231] application = get_wsgi_application()
[Thu Mar 26 11:30:34.179157 2015] [wsgi:error] [pid 23012] [client 127.0.0.1:55231] File "/usr/local/lib/python2.7/dist-packages/django/core/wsgi.py", line 14, in get_wsgi_application
[Thu Mar 26 11:30:34.179319 2015] [wsgi:error] [pid 23012] [client 127.0.0.1:55231] django.setup()
[Thu Mar 26 11:30:34.179363 2015] [wsgi:error] [pid 23012] [client 127.0.0.1:55231] File "/usr/local/lib/python2.7/dist-packages/django/__init__.py", line 20, in setup
[Thu Mar 26 11:30:34.179606 2015] [wsgi:error] [pid 23012] [client 127.0.0.1:55231] configure_logging(settings.LOGGING_CONFIG, settings.LOGGING)
[Thu Mar 26 11:30:34.179651 2015] [wsgi:error] [pid 23012] [client 127.0.0.1:55231] File "/usr/local/lib/python2.7/dist-packages/django/conf/__init__.py", line 46, in __getattr__
[Thu Mar 26 11:30:34.179774 2015] [wsgi:error] [pid 23012] [client 127.0.0.1:55231] self._setup(name)
[Thu Mar 26 11:30:34.179789 2015] [wsgi:error] [pid 23012] [client 127.0.0.1:55231] File "/usr/local/lib/python2.7/dist-packages/django/conf/__init__.py", line 42, in _setup
[Thu Mar 26 11:30:34.179918 2015] [wsgi:error] [pid 23012] [client 127.0.0.1:55231] self._wrapped = Settings(settings_module)
[Thu Mar 26 11:30:34.179959 2015] [wsgi:error] [pid 23012] [client 127.0.0.1:55231] File "/usr/local/lib/python2.7/dist-packages/django/conf/__init__.py", line 94, in __init__
[Thu Mar 26 11:30:34.180072 2015] [wsgi:error] [pid 23012] [client 127.0.0.1:55231] mod = importlib.import_module(self.SETTINGS_MODULE)
[Thu Mar 26 11:30:34.180112 2015] [wsgi:error] [pid 23012] [client 127.0.0.1:55231] File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
[Thu Mar 26 11:30:34.180220 2015] [wsgi:error] [pid 23012] [client 127.0.0.1:55231] __import__(name)
[Thu Mar 26 11:30:34.180246 2015] [wsgi:error] [pid 23012] [client 127.0.0.1:55231] File "/var/www/vhosts/main-page/mysite/../mysite/settings.py", line 94, in <module>
[Thu Mar 26 11:30:34.180400 2015] [wsgi:error] [pid 23012] [client 127.0.0.1:55231] 'NAME': os.environ['RDS_DB_NAME'],
[Thu Mar 26 11:30:34.180443 2015] [wsgi:error] [pid 23012] [client 127.0.0.1:55231] File "/usr/lib/python2.7/UserDict.py", line 23, in __getitem__
[Thu Mar 26 11:30:34.180613 2015] [wsgi:error] [pid 23012] [client 127.0.0.1:55231] raise KeyError(key)
[Thu Mar 26 11:30:34.180665 2015] [wsgi:error] [pid 23012] [client 127.0.0.1:55231] KeyError: 'RDS_DB_NAME'

Set like this
<VirtualHost hostname:80>
...
SetEnv VARIABLE_NAME variable_value
...
</VirtualHost>
In wsgi script.
os.environ.setdefault("variable", "value")
To override variable,
os.environ['var']="val"