We Keep Coding

Superset with Keycloak - Logged out users are not redirected to login

I am trying to implement Superset using Keycloak for authentication. Following the post here: Using KeyCloak(OpenID Connect) with Apache SuperSet, the login part works fine.
I also have a timeout set on the session (security requirement) using the Superset Docs: https://superset.apache.org/docs/installation/configuring-superset#flask-app-configuration-hook
The part that doesn't work, when a user is logged out, they are not redirected to the login page. It's just a bunch of errors thrown on the screen, and the user can't see anything. Anyone have a hint as to how I get the user redirected to the login page?
Worth noting, the whole thing is behind an nginx reverse proxy.
Here's the full superset_config.py, in case it's helpful...
from flask_appbuilder.security.manager import AUTH_OID
from superset.security import SupersetSecurityManager
from flask_oidc import OpenIDConnect
from flask_appbuilder.security.views import AuthOIDView
from flask_login import login_user
from urllib.parse import quote
from flask_appbuilder.views import ModelView, SimpleFormView, expose
import logging
class AuthOIDCView(AuthOIDView):
#expose('/login/', methods=['GET', 'POST'])
def login(self, flag=True):
sm = self.appbuilder.sm
oidc = sm.oid
#self.appbuilder.sm.oid.require_login
def handle_login():
user = sm.auth_user_oid(oidc.user_getfield('email'))
if user is None:
info = oidc.user_getinfo(['preferred_username', 'given_name', 'family_name', 'email'])
user = sm.add_user(info.get('preferred_username'), info.get('given_name'), info.get('family_name'), info.get('email'), sm.find_role('Gamma'))
login_user(user, remember=False)
return redirect(self.appbuilder.get_url_for_index)
return handle_login()
#expose('/logout/', methods=['GET', 'POST'])
def logout(self):
oidc = self.appbuilder.sm.oid
oidc.logout()
super(AuthOIDCView, self).logout()
redirect_url = request.url_root.strip('/') + self.appbuilder.get_url_for_login
return redirect(oidc.client_secrets.get('issuer') + '/protocol/openid-connect/logout?redirect_uri=' + quote(redirect_url))
class OIDCSecurityManager(SupersetSecurityManager):
authoidview = AuthOIDCView
def __init__(self,appbuilder):
super(OIDCSecurityManager, self).__init__(appbuilder)
if self.auth_type == AUTH_OID:
self.oid = OpenIDConnect(self.appbuilder.get_app)
SQLALCHEMY_DATABASE_URI = 'a sting'
MENU_HIDE_USER_INFO = True
FEATURE_FLAGS = {
"ROW_LEVEL_SECURITY": True,
"DASHBOARD_RBAC": True,
}
ENABLE_PROXY_FIX = True
PROXY_FIX_CONFIG = {"x_for": 1, "x_proto": 0, "x_host": 1, "x_port": 0, "x_prefix": 0}
class ReverseProxied(object):
def __init__(self, app):
self.app = app
def __call__(self, environ, start_response):
script_name = environ.get('HTTP_X_SCRIPT_NAME', '')
# print(environ)
if script_name:
environ['SCRIPT_NAME'] = script_name
path_info = environ['PATH_INFO']
if path_info.startswith(script_name):
environ['PATH_INFO'] = path_info[len(script_name):]
scheme = environ.get('HTTP_X_SCHEME', '')
print(scheme)
if scheme:
environ['wsgi.url_scheme'] = scheme
return self.app(environ, start_response)
ADDITIONAL_MIDDLEWARE = [ReverseProxied, ]
def role_mapper(role_list):
# not exposing our roles
# Auth Settings
AUTH_TYPE = AUTH_OID
OIDC_CLIENT_SECRETS = '/a/path' #real config contains correct path
OIDC_ID_TOKEN_COOKIE_SECURE = False
OIDC_REQUIRE_VERIFIED_EMAIL = False
AUTH_USER_REGISTRATION = True
AUTH_USER_REGISTRATION_ROLE = 'Gamma'
CUSTOM_SECURITY_MANAGER = OIDCSecurityManager
# Webserver Setting
SUPERSET_WEBSERVER_PROTOCOL = "http"
SUPERSET_WEBSERVER_ADDRESS = "127.0.0.1"
SUPERSET_WEBSERVER_PORT = 8088
# Flask Application Builder Settings
SILENCE_FAB = False
FAB_ADD_SECURITY_VIEWS = True
FAB_ADD_SECURITY_PERMISSION_VIEW = True
FAB_ADD_SECURITY_VIEW_MENU_VIEW = True
FAB_ADD_SECURITY_PERMISSION_VIEWS_VIEW = True
# Session Timeout
from flask import session
from flask import Flask
from datetime import timedelta
def make_session_permanent():
session.permanent = True
# Set up max age of session to 1 minute for testing
PERMANENT_SESSION_LIFETIME = timedelta(minutes=1)
def FLASK_APP_MUTATOR(app: Flask) -> None:
app.before_request_funcs.setdefault(None, []).append(make_session_permanent)```

I am also facing same issue, when user is trying to login agin after logout once its throwing 502 error, Because when user made logout only from superset he is getting logout but in keykloak status kept as login , How to logout from keykloak also when user logout from superset dashboard

Duplicate cloud function log on GCP stackdriver

I try to add a logging feature to my cloud function. And integrate the logging module with google-cloud-logging. Below is the code I deploy on cloud function:
import logging
import platform
import sys
from datetime import datetime
from google.cloud import logging as gcp_logging
from google.cloud.logging.handlers import CloudLoggingHandler
from google.cloud.logging.resource import Resource
class Singleton(type):
_instances = {}
def __call__(cls, *args, **kwargs):
if cls not in cls._instances:
cls._instances[cls] = super(Singleton, cls).__call__(*args, **kwargs)
else:
# uncommnet this if you need initialzation everytime
cls._instances[cls].__init__(*args, **kwargs)
return cls._instances[cls]
class LogManager(object, metaclass=Singleton):
#TODO duplicate log issue
gcp_client = None
gcp_handler = None
log_file = ''
log_options = {
'console': {
'log_level': logging.DEBUG
}
}
default_fmt = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(filename)s - %(lineno)s - %(message)s')
def __new__(cls, options=None):
print('new LOG manager')
return object.__new__(cls)
def __init__(self, options=None):
print('init LOG manager')
if options is not None:
self.log_options = options
if 'gcp' in self.log_options:
if platform.node() and platform.node() != 'localhost':
print('use auth_token when executed on local')
self.gcp_client = gcp_logging.Client.from_service_account_json('MY_SECRET.json')
else:
print('get environment default, use it when executed on cloud')
self.gcp_client = gcp_logging.Client()
def get_logger(self, logger_name, fmt=None, log_tags={}):
# there is a manager keeping all loggers for reusing
logger = logging.getLogger(logger_name)
logger.setLevel(logging.DEBUG)
if logger.hasHandlers():
logger.handlers.clear()
if fmt is not None:
self.formatter = fmt
else:
self.formatter = self.default_fmt
if 'console' in self.log_options:
ch = logging.StreamHandler(sys.stdout)
ch.setLevel(self.log_options['console']['log_level'])
ch.setFormatter(self.formatter)
logger.addHandler(ch) # add the handlers to the logger
if 'gcp' in self.log_options:
gcp_handler = CloudLoggingHandler(
client=self.gcp_client,
resource=Resource(
type=self.log_options['gcp']['log_type'],
labels=self.log_options['gcp']['labels']
)
)
gcp_handler.setFormatter(self.formatter)
gcp_handler.setLevel(self.log_options['gcp']['log_level'])
logger.addHandler(gcp_handler)
return logger
def stop_logging(self):
logging.shutdown()
def hello_world(request):
"""Responds to any HTTP request.
Args:
request (flask.Request): HTTP request object.
Returns:
The response text or any set of values that can be turned into a
Response object using
`make_response <http://flask.pocoo.org/docs/1.0/api/#flask.Flask.make_response>`.
"""
mode = request.args.get('mode', default = 'gcp')
if mode=='console':
print('using default logger setup')
log_manager = LogManager() # use this for default setup
else:
print('using gcp logger setup')
log_manager = LogManager(
options={
'gcp': {
'log_level': logging.INFO,
'log_type': 'cloud_function',
'labels': {
'function_name': 'test_logging',
},
},
}
)
logger = log_manager.get_logger(__name__)
logger.debug('debug msg1')
logger.info('info msg1')
logger.warning('warning msg1')
logger.error('error msg1')
logger.critical('critical msg1')
logger = LogManager().get_logger(__name__)
logger.debug('debug msg2')
logger.info('info msg2')
logger.warning('warning msg2')
logger.error('error msg2')
logger.critical('critical msg2')
log_manager.stop_logging()
return 'ok'
With dependency google-cloud-logging==1.11.0 and google-api-core==1.22.2.
Here is the result when I use StreamHandler:
As you can see, there are 2 types of logging messages:
formatted log output from logger
pure log message
And both of these messages are automatically tagged with a random ID by GCP.
Here is another result when I use CloudLoggingHandler:
As you can see, there are 2 types of logging messages:
formatted log output from logger without random ID tagged by GCP
pure log message tagged with a random ID by GCP
My questions are:
How to resolve the duplicates issue? (either CloudLoggingHandler or StreamHandler)
How is the auto-tagging mechanism work? I am trying to keep these tags since they could be useful for searching (e.g. test_logging & r2b4rhft6z1i in the screenshot)
The last one is a little bit beyond this topic. Should I use singleton in this case? Am I use it right? Is there any improvement for this code snippet?
Sorry for asking so many questions at once. Any suggestion will be appreciated!

adding user ip and user id to logfile in flask application

I am using the standard Python logging package to write logfiles in a Flask application. I would like to add the user ip address and if the user is authenticated I would also like to log the user id. Does anybody know how to do this?
Right now my formatter looks like this
fmt = ('%(asctime)s - %(name)s - %(levelname)s - '
'%(filename)s:%(lineno)s - %(funcName)20s() - %(message)s')
formatter = logging.Formatter(fmt)

Ok I manager to do it by adding a costum filter
class ContextualFilter(logging.Filter):
def filter(self, log_record):
''' Provide some extra variables to give our logs some better info '''
log_record.utcnow = datetime.datetime.utcnow().strftime('%Y-%m-%d %H:%M:%S,%f %Z')
log_record.url = request.path
log_record.method = request.method
# Try to get the IP address of the user through reverse proxy
log_record.ip = request.environ.get('HTTP_X_REAL_IP', request.remote_addr)
if current_user.is_anonymous():
log_record.user_id = 'guest'
else:
log_record.user_id = current_user.get_id()
return True
which can be added to the logger as
std_logger.addFilter(ContextualFilter())
the formatter than should be something like
fmt = ('%(utcnow)s - %(levelname)s - '
'user_id:%(user_id)s - ip:%(ip)s - %(method)s - %(url)s - '
'%(filename)s:%(lineno)s - %(funcName)20s() - %(message)s')

How can I resolve these flask 500 server errors? [duplicate]

This question already has an answer here:
Why not generate the secret key every time Flask starts?
(1 answer)
Closed 4 years ago.
I'm currently super stumped. I have a large flask application that I recently deployed to my production ec2 server that's not functioning properly. It works fine in dev/test. It looks like, maybe, apache is hanging. When I first deployed it, it worked a few times. Then, after that I started getting 500 server errors. When I restart apache, it works fine again for a session but the 500 server errors come back. Each time, on a different page of the application. This is reoccurring. How would I go about fixing this? Here's my routes/app.py:
from flask import Flask, render_template,redirect,request,url_for, flash, session
from Index_generator import index_generator
from Yelp_api import request_yelp
from plaid import auth
from search_results import search,search1
from Options import return_data
from datetime import timedelta
import os
import urllib2
import time
from path import path_data
#from writer import create_config_file
import logging
from logging_path import log_dir
logger = logging.getLogger("Routing")
logger.setLevel(logging.INFO)
foodie_log = log_dir()
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - % (message)s')
foodie_log.setFormatter(formatter)
logger.addHandler(foodie_log)
app = Flask(__name__)
app.secret_key = os.urandom(24)
go = path_data()
#app.before_request
def make_session_permanent():
session.permanent = False
app.permanent_session_lifetime = timedelta(seconds=300)
##app.route('/setup',methods=['GET','POST'])
#def setup():
# if request.method == 'POST':
# Username=request.form['username']
# Password=request.form['password']
# Port=request.form['port']
# Host=request.form['host']
# C_Key=request.form['CONSUMER_KEY']
# C_Sec=request.form['CONSUMER_SECRET']
# Tok=request.form['TOKEN']
# Tok_Sec=request.form['TOKEN_SECRET']
# clientid=request.form['client_id']
# Sec=request.form['secret']
# create_config_file(Username,Password,Port,Host,C_Key,C_Sec,Tok,Tok_Sec,clientid,Sec)
# return 'complete'
# elif request.method == 'GET':
#
# return render_template('setup.html')
#app.route('/')
def home():
store = index_generator()
session['store'] = store
return render_template('home.html')
#app.route('/about')
def about():
return render_template('about.html')
#app.route('/home_city',methods = ['POST'])
def home_city():
try:
CITY=request.form['city']
store = session.get('store')
request_yelp(DEFAULT_LOCATION=CITY,data_store=store)
return render_template('bank.html')
except Exception as e:
logger.error(e)
error = 'Sorry, no results. Is' + ' ' +CITY + ' '+ 'your hometown? If not, try again and if so, we have been made aware of the issue and is working to resolve it'
return render_template('home.html',error=error)
#app.route('/traveling',methods = ['POST'])
def travel():
store = session.get('store')
answer=request.form['Travel']
if answer == 'yes':
#time.sleep(2)
return render_template('destination.html')
else:
results_home = search(index=store)
time.sleep(2)
return return_data(results_home)
#app.route('/dest_city',methods = ['POST'])
def dest_city():
store = session.get('store')
try:
DESTINATION=request.form['dest_city']
request_yelp(DEFAULT_LOCATION=DESTINATION, data_store=store,sourcetype='dest_city')
results_dest = search1(index=store)
time.sleep(2)
return return_data(results_dest)
except urllib2.HTTPError:
error = 'Sorry, no results. Is your destination city? If not, try again and if so, we have been made aware of the issue and is working to resolve it'
return render_template('destination.html',error=error)
#app.route('/bank',methods = ['POST'])
def bank():
try:
store = session.get('store')
print store
Bank=request.form['Fin_Bank']
Username=request.form['username']
Password=request.form['password']
Test = auth(account=Bank,username=Username,password=Password,data_store=store)
if Test == 402 or Test ==401:
error = 'Invalid credentials'
return render_template('bank.html',error=error)
else :
return render_template('travel.html')
except:
logger.error(e)
if __name__ == '__main__':
app.run(debug=True)
app.secret_key=os.urandom(24)

Not completely sure if this causes your errors but you should definitely change this:
You have to set app.secret_key to a static value rather than os.urandom(24), e.g.:
app.secret_key = "/\xfa-\x84\xfeW\xc3\xda\x11%/\x0c\xa0\xbaY\xa3\x89\x93$\xf5\x92\x9eW}"
With your current code, each worker thread of your application will have a different secret key, which can result in errors or session inconsistency depending on how you use the session object in your application.

python logging to database

I'm seeking a way to let the python logger module to log to database and falls back to file system when the db is down.
So basically 2 things: How to let the logger log to database and how to make it fall to file logging when the db is down.

I recently managed to write my own database logger in Python. Since I couldn't find any example I thought I post mine here. Works with MS SQL.
Database table could look like this:
CREATE TABLE [db_name].[log](
[id] [bigint] IDENTITY(1,1) NOT NULL,
[log_level] [int] NULL,
[log_levelname] [char](32) NULL,
[log] [char](2048) NOT NULL,
[created_at] [datetime2](7) NOT NULL,
[created_by] [char](32) NOT NULL,
) ON [PRIMARY]
The class itself:
class LogDBHandler(logging.Handler):
'''
Customized logging handler that puts logs to the database.
pymssql required
'''
def __init__(self, sql_conn, sql_cursor, db_tbl_log):
logging.Handler.__init__(self)
self.sql_cursor = sql_cursor
self.sql_conn = sql_conn
self.db_tbl_log = db_tbl_log
def emit(self, record):
# Set current time
tm = time.strftime("%Y-%m-%d %H:%M:%S", time.localtime(record.created))
# Clear the log message so it can be put to db via sql (escape quotes)
self.log_msg = record.msg
self.log_msg = self.log_msg.strip()
self.log_msg = self.log_msg.replace('\'', '\'\'')
# Make the SQL insert
sql = 'INSERT INTO ' + self.db_tbl_log + ' (log_level, ' + \
'log_levelname, log, created_at, created_by) ' + \
'VALUES (' + \
'' + str(record.levelno) + ', ' + \
'\'' + str(record.levelname) + '\', ' + \
'\'' + str(self.log_msg) + '\', ' + \
'(convert(datetime2(7), \'' + tm + '\')), ' + \
'\'' + str(record.name) + '\')'
try:
self.sql_cursor.execute(sql)
self.sql_conn.commit()
# If error - print it out on screen. Since DB is not working - there's
# no point making a log about it to the database :)
except pymssql.Error as e:
print sql
print 'CRITICAL DB ERROR! Logging to database not possible!'
And usage example:
import pymssql
import time
import logging
db_server = 'servername'
db_user = 'db_user'
db_password = 'db_pass'
db_dbname = 'db_name'
db_tbl_log = 'log'
log_file_path = 'C:\\Users\\Yourname\\Desktop\\test_log.txt'
log_error_level = 'DEBUG' # LOG error level (file)
log_to_db = True # LOG to database?
class LogDBHandler(logging.Handler):
[...]
# Main settings for the database logging use
if (log_to_db):
# Make the connection to database for the logger
log_conn = pymssql.connect(db_server, db_user, db_password, db_dbname, 30)
log_cursor = log_conn.cursor()
logdb = LogDBHandler(log_conn, log_cursor, db_tbl_log)
# Set logger
logging.basicConfig(filename=log_file_path)
# Set db handler for root logger
if (log_to_db):
logging.getLogger('').addHandler(logdb)
# Register MY_LOGGER
log = logging.getLogger('MY_LOGGER')
log.setLevel(log_error_level)
# Example variable
test_var = 'This is test message'
# Log the variable contents as an error
log.error('This error occurred: %s' % test_var)
Above will log both to the database and to the file. If file is not needed - skip the 'logging.basicConfig(filename=log_file_path)' line. Everything logged using 'log' - will be logged as MY_LOGGER. If some external error appears (i.e. in the module imported or something) - error will appear as 'root', since 'root' logger is also active, and is using the database handler.

Write yourself a handler that directs the logs to the database in question. When it fails, you can remove it from the handler list of the logger. There are many ways to deal with the failure-modes.

Python logging to a database with a backup logger
Problem
I had the same problem when I ran a Django project inside the server since sometimes you need to check the logs remotely.
Solution
First, there is a need for a handler for the logger to insert logs in to the database. Before that and since my SQL is not good, an ORM is needed that I choose SQLAlchemy.
model:
# models.py
from sqlalchemy import Column, Integer, String, DateTime, Text
from sqlalchemy.ext.declarative import declarative_base
import datetime
base = declarative_base()
class Log(base):
__tablename__ = "log"
id = Column(Integer, primary_key=True, autoincrement=True)
time = Column(DateTime, nullable=False, default=datetime.datetime.now)
level_name = Column(String(10), nullable=True)
module = Column(String(200), nullable=True)
thread_name = Column(String(200), nullable=True)
file_name = Column(String(200), nullable=True)
func_name = Column(String(200), nullable=True)
line_no = Column(Integer, nullable=True)
process_name = Column(String(200), nullable=True)
message = Column(Text)
last_line = Column(Text)
This is the crud for insertion into the database:
#crud.py
import sqlalchemy
from .models import base
from traceback import print_exc
class Crud:
def __init__(self, connection_string=f'sqlite:///log_db.sqlite3',
encoding='utf-8',
pool_size=10,
max_overflow=20,
pool_recycle=3600):
self.connection_string = connection_string
self.encoding = encoding
self.pool_size = pool_size
self.max_overflow = max_overflow
self.pool_recycle = pool_recycle
self.engine = None
self.session = None
def initiate(self):
self.create_engine()
self.create_session()
self.create_tables()
def create_engine(self):
self.engine = sqlalchemy.create_engine(self.connection_string)
def create_session(self):
self.session = sqlalchemy.orm.Session(bind=self.engine)
def create_tables(self):
base.metadata.create_all(self.engine)
def insert(self, instances):
try:
self.session.add(instances)
self.session.commit()
self.session.flush()
except:
self.session.rollback()
raise
def __del__(self):
self.close_session()
self.close_all_connections()
def close_session(self):
try:
self.session.close()
except:
print_exc()
else:
self.session = None
def close_all_connections(self):
try:
self.engine.dispose()
except:
print_exc()
else:
self.engine = None
The handler:
# handler.py
from logging import Handler, getLogger
from traceback import print_exc
from .crud import Crud
from .models import Log
my_crud = Crud(
connection_string=<connection string to reach your db>,
encoding='utf-8',
pool_size=10,
max_overflow=20,
pool_recycle=3600)
my_crud.initiate()
class DBHandler(Handler):
backup_logger = None
def __init__(self, level=0, backup_logger_name=None):
super().__init__(level)
if backup_logger_name:
self.backup_logger = getLogger(backup_logger_name)
def emit(self, record):
try:
message = self.format(record)
try:
last_line = message.rsplit('\n', 1)[-1]
except:
last_line = None
try:
new_log = Log(module=record.module,
thread_name=record.threadName,
file_name=record.filename,
func_name=record.funcName,
level_name=record.levelname,
line_no=record.lineno,
process_name=record.processName,
message=message,
last_line=last_line)
# raise
my_crud.insert(instances=new_log)
except:
if self.backup_logger:
try:
getattr(self.backup_logger, record.levelname.lower())(record.message)
except:
print_exc()
else:
print_exc()
except:
print_exc()
Test to check the logger:
# test.py
from logging import basicConfig, getLogger, DEBUG, FileHandler, Formatter
from .handlers import DBHandler
basicConfig(format='%(asctime)s - %(name)s - %(levelname)s - %(message)s',
datefmt='%d-%b-%y %H:%M:%S',
level=DEBUG)
format = Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
backup_logger = getLogger('backup_logger')
file_handler = FileHandler('file.log')
file_handler.setLevel(DEBUG)
file_handler.setFormatter(format)
backup_logger.addHandler(file_handler)
db_logger = getLogger('logger')
db_handler = DBHandler(backup_logger_name='backup_logger')
db_handler.setLevel(DEBUG)
db_handler.setFormatter(format)
db_logger.addHandler(db_handler)
if __name__ == "__main__":
db_logger.debug('debug: hello world!')
db_logger.info('info: hello world!')
db_logger.warning('warning: hello world!')
db_logger.error('error: hello world!')
db_logger.critical('critical: hello world!!!!')
You can see the handler accepts a backup logger that can use it when the database insertion fails.
A good improvement can be logging into the database by threading.

I am digging this out again.
There is a solution with SqlAlchemy (Pyramid is NOT required for this recipe):
https://docs.pylonsproject.org/projects/pyramid-cookbook/en/latest/logging/sqlalchemy_logger.html
And you could improve logging by adding extra fields, here is a guide: https://stackoverflow.com/a/17558764/1115187
Fallback to FS
Not sure that this is 100% correct, but you could have 2 handlers:
database handler (write to DB)
file handler (write to file or stream)
Just wrap the DB-commit with a try-except. But be aware: the file will contain ALL log entries, but not only entries for which DB saving was failed.

Old question, but dropping this for others. If you want to use python logging, you can add two handlers. One for writing to file, a rotating file handler. This is robust, and can be done regardless if the dB is up or not.
The other one can write to another service/module, like a pymongo integration.
Look up logging.config on how to setup your handlers from code or json.