i am implementing logging across a python web flask application. The below is an example structure
controller - com.org.python.MainController
has routes say /login, /doSomething, /logout
each of these routes call separate methods in separate modules say,
com.org.python.login.begin()
com.org.python.dosomething.execute()
com.org.python.logout.end()
my maincontroller routes have access to user id who has invoked the transaction and a unique transaction id which is generated. i can log this information easily using loggers.
usecase: i need to log user id and transaction id in the lower level modules as well , without passing the details as parameters
How can i achieve the same?
solution so far
class CustomFilter(logger.Filter):
def __init__(self, username, trid):
self.username = username
self.trid = trid
def filter(record):
record.username = self.username
record.trid = self.trid
i have added the filter to logger as given above
maincontroller.py
def login(username):
trid = ....
log = logger.getLogger("something")
log.addFilter(CustomFilter(username, trid))
#handlers, formatters added as used
log.info("inside login method")
#import login module
login.begin()
login.py
def begin():
log = logger.getLogger("something")
log.info("im inside begin method in login module")
### something something
the above works fine, username and trid is printed everywhere even across modules. however the same trid and username is being used everywhere. even transactions not initiated by the same user. do we have to clear the filters? how can i achieve my usecase m the desired username and diff trid for diff transactions
P.S: for some reason loggerAdaptors are not working. the info in adaptor is not getting transferred across module calls
please help.
Related
I'm currently building an application that allows users to collaborate together and create things, as I require a sort of discord like group chatfeed. I need to be able to subscribe logged in users to a project for notifications.
I have a method open_project that retrieves details from a project that has been selected by the user, which I use to subscribe him to any updates for that project.
So I can think of 2 ways of doing this. I have created a instance variable in my connect function, like this:
def connect(self):
print("connected to projectconsumer...")
self.accept()
self.projectSessions = {}
And here is the open_project method:
def open_project(self, message):
p = Project.objects.values("projectname").get(id=message)
if len(self.projectSessions) == 0:
self.projectSessions[message] = []
pass
self.projectSessions[message] = self.projectSessions[message].append(self)
print(self.projectSessions[message])
self.userjoinedMessage(self.projectSessions[message])
message = {}
message["command"] = "STC-openproject"
message["message"] = p
self.send_message(json.dumps(message))
Then when the user opens a project, he is added to the projectSessions list, this however doesn't work (I think) whenever a new user connects to the websocket, he gets his own projectconsumer.
The second way I thought of doing this is to create a managing class that only has 1 instance and keeps track of all the users connected to a project. I have not tried this yet as I would like some feedback on if I'm even swinging in the right ball park. Any and all feedback is appreciated.
EDIT 1:
I forgot to add the userjoinedMessage method to the question, this method is simply there to mimic future mechanics and for feedback to see if my solution actually works, but here it is:
def userjoinedMessage(self, pointer):
message = {}
message["command"] = "STC-userjoinedtest"
message["message"] = ""
pointer.send_message(json.dumps(message))
note that i attempt to reference the instance of the consumer.
I will also attempt to implement a consumer manager that has the role of keeping track of what consumers are browsing what projects and sending updates to the relevant channels.
From the question, the issue is how to save projectSessions and have it accessible across multiple instances of the consumer. Instead of trying to save it in memory, you can save in a database. It is a dictionary with project as key. You can make it a table with ForeignKey to the Project model.
In that way, it is persisted and there would be no issue retrieving it even across multiple channels server instances if you ever decide to scale your channels across multiple servers.
Also, if you feel that a traditional database will slow down the retrieval of the sessions, then you can use faster storage systems like redis
Right, this is probably a horrible way of doing things and i should be taken out back and shot for doing it but i have a fix for my problem. I have made a ProjectManager class that handles subscriptions and updates to the users of a project:
import json
class ProjectManager():
def __init__(self):
if(hasattr(self, 'projectSessions')):
pass
else:
self.projectSessions = {}
def subscribe(self, projectid, consumer):
print(projectid not in self.projectSessions)
if(projectid not in self.projectSessions):
self.projectSessions[projectid] = []
self.projectSessions[projectid].append(consumer)
self.update(projectid)
def unsubscribe(self, projectid, consumer):
pass
def update(self, projectid):
if projectid in self.projectSessions:
print(self.projectSessions[projectid])
for consumer in self.projectSessions[projectid]:
message = {}
message["command"] = "STC-userjoinedtest"
message["message"] = ""
consumer.send_message(json.dumps(message))
pass
in my apps.py file i initialize the above ProjectManager class and assign it to a variable.
from django.apps import AppConfig
from .manager import ProjectManager
class ProjectConfig(AppConfig):
name = 'project'
manager = ProjectManager()
Which i then use in my consumers.py file. I import the manager from the projectconfig class and assign it to a instance variable inside the created consumer whenever its connected:
def connect(self):
print("connected to projectconsumer...")
self.accept()
self.manager = ProjectConfig.manager
and whenever i call open_project i subscribe to that project with the given project id recieved from the front-end:
def open_project(self, message):
p = Project.objects.values("projectname").get(id=message)
self.manager.subscribe(message, self)
message = {}
message["command"] = "STC-openproject"
message["message"] = p
self.send_message(json.dumps(message))
as i said i in no way claim that this is the correct way of doing it and i am also aware that channel_layers supposedly does this for you in a neat way. i however don't really have the time to get into channel_layers and will therefore be using this.
I am still open to suggestions ofcourse and am always happy to learn more.
I have made a flask application which gets and receives messages from the user and generates a reply from a Chatbot backend I have created. When I run my my app.py file and I go to my localhost it runs fine if I only open one instance, however if I try to open multiple instances then they all try to use the same bot. How do I create a unique bot for each session. I tried using g.bot = mybot() but the problem was it still kept creating a new bot each time the user replied to the bot. I am relatively new to this so links to a detailed explanation would be appreciated. Note some pieces of code are unrelated junk from previous versions.
app = Flask(__name__)
items = ["CommonName","Title",
"Department","Address","City","PhoneNum"]
app.config.from_object(__name__)
bot2 = Bot2()
#app.before_request
def before_request():
session['uid'] = uuid.uuid4()
print(session['uid'])
g.bot2 = Bot2()
#app.route("/", methods=['GET'])
def home():
return render_template("index.html")
#app.route("/tables")
def show_tables():
data = bot2.df
if data.size == 0:
return render_template('sad.html')
return render_template('view.html',tables=[data.to_html(classes='df')], titles = items)
#app.route("/get")
def get_bot_response():
userText = request.args.get('msg')
bot2.input(str(userText))
print(bot2.message)
g.bot2.input(str(userText))
print(g.bot2.message)
show_tables()
if (bot2.export):
return (str(bot2.message) + "<br/>\nWould you like to narrow your results?")
#return (str(bot.message) + "click here" + "</span></p><p class=\"botText\"><span> Would you like to narrow your results?")
else:
return (str(bot2.message))
if __name__ == "__main__":
app.secret_key = 'super secret key'
app.run(threaded=True)
Problem: A new bot is created each time the user replies to the bot
Reason: app.before_request runs before each request that your flask server receives. Hence each reply will create a new Bot2 instance. You can read more about it here.
Problem: Creating a bot per instance
Possible Solution: I'm not really sure what you mean by opening multiple instances (are you trying to run multiple instances of the same server, or there are multiple ppl accessing the single server). I would say to read up on Sessions in Flask and store a Bot2 instance inside sessions as a server side variable. You can read more about it here and here.
You could try assigning the session id from flask-login to the bot instead of creating a unique id at before request. Right now, like #Imma said, a new bot is created for every request.
You will have to store an array of classes. When a session is created, i.e., a user or an anonymous user logs in, a bot/class instance gets created and is pushed onto the array.
You can keep the array in the session object... do note that the session object will get transferred in the form of a cookie to the front end... so you may be potentially exposing all the chat sessions...Also, a lot of users will unnecessarily slow down the response.
Another alternative is to create a separate container and run the bot as a separate microservice rather than integrate it with your existing flask application (this is what we ended up doing)
Delete the line bot2 = Bot2(), and change all the reference to bot2 to g.bot2.
I use tornado 4.5.2 with routing implementation.
My server have two versions of API, let them call base and fancy. So a client is able to use both of them:
GET /base/foo
GET /base/baz
GET /fancy/foo
GET /fancy/baz
However, some fancy handlers may not be implemented; In this case a base one should be used.
In example:
application = web.Application([
(r"/base/foo", handlers.BaseFooHandler, {"some": "settings"}),
(r"/base/baz", handlers.BaseBazHandler, {"some": "settings"}),
(r"/fancy/foo", handlers.FancyFooHandler, {"some": "settings"}),
])
when cilent requests GET /fancy/baz the BaseBazHandler should do the job.
How can I achieve that with tornado routing?
Since you're registering your routes using a decorator, you can create a custom router that will respond to all the unmatched/unregistered /fancy/.* routes. For this to work correctly, you'll have to register your router at the end.
That way your custom router will be matched only if there isn't already a /fancy/... route registered. So, that means the custom router class will need to do these things:
Check if a fallback BaseBazHandler exists or not.
If exists, forward the request to it.
Else, return a 404 error.
Before proceeding any further, you'll have to create a custom class to handle 404 requests. This is necessary because if not handler is found, then this is the easiest way to return a 404 error.
class Handle404(RequestHandler):
def get(self):
self.set_status(404)
self.write('404 Not Found')
Okay, now let's write the custom router:
from tornado.routing import Router
class MyRouter(Router):
def __init__(self, app):
self.app = app
def find_handler(self, request, **kwargs):
endpoint = request.path.split('/')[2] # last part of the path
fallback_handler = 'Base%sHandler' % endpoint.title()
# fallback_handler will look like this - 'BaseBazHandler'
# now check if the handler exists in the current file
try:
handler = globals()[fallback_handler]
except KeyError:
handler = Handle404
return self.app.get_handler_delegate(request, handler)
Finally, after you've added all other routes, you can register your custom router:
from tornado.routing import PathMatches
application.add_handlers(r'.*', # listen for all hosts
[
(PathMatches(r"/fancy/.*"), MyRouter(application)),
]
)
I should point out that MyRouter.find_handler, only check checks for handlers in the current module (file). Modify the code to search for handlers in different modules, if you want.
I'm working on a Flask app that needs user management. However, I have multiple parts of my site, let's call them /site/<name>/ with each of them having their own users (linked in the DB to name).
So, what I want to achieve is:
Any unauthorised access to anything /site/<name>/<path> gets redirected to /site/<name>/login (so, I cannot simply redefine login_view_function).
When the user logs in, the argument name is also used (so, a user from /site/name1/ cannot log in at /site/name2/).
A login from /site/name1/login works inside /site/name1/<path>, but not in /site/name2/<path> (the user is unauthorised there; can also get logged out of name1... I don't particularly care what exactly happens).
Whatever happens outside of /site/<name>/ (or for invalid name) is irrelevant for now.
I tried redefining URL rules in a before_request decorated function:
site_manager = UserManager(db_adapter, app)
...
#app.before_request
def config_flask_user():
m = re.match(r'https?://[^/]+/site/(?P<name>[^:/]*)', request.url)
if m:
name = m.group('name')
flask.g.app_theme = name
if not name_ok(name):
flask.abort(404)
# Make a no-arguments node for the current site
app.debug = False
app.add_url_rule(
'/site/%s/' % name,
'site.current',
site,
methods=['GET', 'POST'],
defaults={'name': name},
)
app.debug = DEBUG
# Use it as the post-logout endpoint, so that it leads to a page
# where re-login works properly
site_manager.after_logout_endpoint = 'site.current'
# Override existing `login_url` and `logout_url` routes, to fit
# the current site path
for action in ('login', 'logout'):
rule = '/site/%s/%s/' % (name, action)
endpoint = getattr(site_manager, action + "_url")
view_func = getattr(site_manager, action + "_view_function")
for r in app.url_map._rules:
if r.rule == endpoint:
print("Redefining rule", endpoint, "as", rule)
r.rule = rule
r.refresh()
This seemed to work fine, but it actually breaks if the server is restarted between loading the Login page and actually submitting user's login and password, seemingly due to that spot being too late to redefine these paths, and I'm worried that it's the wrong approach (maybe it can cause collisions between different requests to different "sites"?), especially for the intended multi-worker environment.
TL;DR: How to get /site/<name>/ paths to have Flask-User login dependent on the name argument?
Is there a way to group logs of a python web application which belong to one web request?
Example:
2015-02-11 13:06:32 myapp.middleware.MYAPPMiddleware: INFO Login of user foo was successful
2015-02-11 13:06:32 myapp.middleware.MYAPPMiddleware: INFO Login of user bar failed
2015-02-11 13:06:32 myapp.send_mails: INFO failed to send mail to someone#example.com
The above log lines are unrelated to each other.
How can you solve this the pythonic way?
Log entries in their essence are designed to be independent from each other.
The correct way to connect them together is to include some contextual information into the entries to filter by when looking through the logs later.
Here's a example of a Sharepoint log record with such information:
Timestamp Process TID Area Category EventID Level Message Correlation
02/26/2015 17:49:19.65 w3wp.exe (0x1F40) 0x2358 SharePoint Foundation Logging Correlation Data xmnv Medium Name=Request (POST:http://reserver2:80/pest/_vti_bin/sitedata.asmx) d1e2b688-e0b2-481e-98ce-497a11acab44
In Python logging docs, Adding contextual information to your logging output recommends either of two methods: using a LoggerAdapter or a Filter.
LoggerAdapter is used like this (examples are based on those in the docs):
class AddConnIdAdapter(logging.LoggerAdapter):
def process(self, msg, kwargs):
return <augment_message(msg,arbitrary_info)>, kwargs
la = AddConnIdAdapter(<logger>,extra=<parameters, saved in self.extra>)
<...>
la.info(<message>)
Filter is used like this:
#Either all messages should have custom fields
# or the Formatter used should support messages
# both with and without custom fields
logging.basicConfig(<...>,format='%(asctime)-15s %(name)-5s %(levelname)-8s IP: %(ip)-15s User: %(user)-8s %(message)s')
class AddClientInfo(logging.Filter):
#override __init__ or set attributes to specify parameters
def filter(self, record):
record.ip = <get_client_ip()>
record.user = <get_client_name()>
return True #do not filter out anything
l=<logger()>
l.addFilter(AddClientInfo()) #can attach to either loggers or handlers
<...>
l.info('message')
As you can see, the difference is LoggerAdapter is non-transparent while Filter is transparent. In the examples, the former modifies the message text while the latter sets custom attributes (and actually writing them requires cooperation of the Formatter used) but in fact, both can do both.
So, the former is more useful if you only need to add the context to some messages while the latter is more fit to augment all, or a large portion of, the messages being logged.
You can assign random UUID to each request in init method, and add it to all log messages.
For example, in Tornado:
class MainRequestHandler(RequestHandler):
def __init__(self, application, request):
super(MainRequestHandler, self).__init__(application, request)
self.uuid = uuid.uuid4()
logging.info("%s | %s %s %s",
self.uuid,
request.method,
request.full_url(),
request.remote_ip)
As result, you will be able to grep log by this UUID to find all messages which belong to separate request.