I'm currently managing my API using Celery tasks and a kubernetes cluster on Google Cloud Platform.
Celery is automatically logging input and output of each task. This is something I want but I would like to use the possibility of google-cloud-logging to log input and output as jsonPayload.
I use for all other log the following:
from google.cloud.logging.handlers import CloudLoggingHandler
from google.cloud.logging_v2.handlers import setup_logging
# Imports the Cloud Logging client library
import google.cloud.logging
# Instantiates a client
client = google.cloud.logging.Client()
handler = CloudLoggingHandler(client)
setup_logging(handler)
import logging
logger = logging.getLogger(__name__)
data_dict = {"my": "data"}
logger.info("this is an example", extra={"json_fields": data_dict})
And I use Celery with the following template:
app = Celery(**my_params)
#app.task
def task_test(data):
# Update dictonary with new data
data["key1"] = "value1"
return data
...
detection_task = celery.signature('tasks.task_test', args=([[{"hello": "world"}]]))
r = detection_task.apply_async()
data = r.get()
Here's an example of log I receive from Celery:
The blurred part correspond to the dict/json I would like to have in a jsonPayload instead of a textPayload.
(Also note that this log is marked as error on GCP but INFO from celery)
Any idea how I could connect python built-in logging, celery logger and gcp logger ?
To connect your Python logger with GCP Logger:
import logging
import google.cloud.logging
from google.cloud.logging.handlers import CloudLoggingHandler, setup_logging
client = google.cloud.logging.Client()
handler = CloudLoggingHandler(client, name="your_log_name")
cloud_logger = logging.getLogger('cloudLogger')
# configure cloud_logger
cloud_logger.addHandler(handler)
To connect this logger to Celery's logger:
def initialize_log(logger=None,loglevel=logging.DEBUG, **kwargs):
logger = logging.getLogger('celery')
handler = # use GCP handler defined above
handler.setLevel(loglevel)
logger.addHandler(handler)
return logger
from celery.signals import after_setup_task_logger
after_setup_task_logger.connect(initialize_log)
from celery.signals import after_setup_logger
after_setup_logger.connect(initialize_log)
Related
I would like to send LogRecord attributes (like processName, threadName, exc_info etc) as part of logging information on GCP in json format. How to do that? I am using below python code to send logs to GCP (with jsonpayload)
(thanks #ianyoung code)
import logging
import google.cloud.logging
import json
client = google.cloud.logging.Client()
client.setup_logging()
logger = logging.getLogger('test')
data_dict = {"hello": "world"}
logging.info(json.dumps(data_dict))
I had tried the code below for getting the exception in https://learn.microsoft.com/en-us/azure/azure-monitor/app/opencensus-python
from opencensus.ext.azure.log_exporter import AzureLogHandler
logger = logging.getLogger(__name__)
# TODO: replace the all-zero GUID with your instrumentation key.
logger.addHandler(AzureLogHandler(
connection_string='InstrumentationKey=00000000-0000-0000-0000-000000000000')
)
properties = {'custom_dimensions': {'key_1': 'value_1', 'key_2': 'value_2'}}
# Use properties in exception logs
try:
result = 1 / 0 # generate a ZeroDivisionError
except Exception:
logger.exception('Captured an exception.', extra=properties)
It is working. I can catch the exception.
However, I want to ask if there is an easy way to catch the exception automatically in the python flask?
Since I try the below code, it just gives me a request record, not the exception.
app = Flask(__name__)
app.logger.addHandler(file_handler)
handler = AzureEventHandler(
connection_string="InstrumentationKey={}".format(app.config['APPINSIGHTS_INSTRUMENTATIONKEY']))
handler.setFormatter(logging.Formatter(
'%(asctime)s %(levelname)s: %(message)s '
'[in %(pathname)s:%(lineno)d]'))
handler.setLevel(logging.ERROR)
app.logger.addHandler(handler)
Thank you for helping
Your code should like below. For more details, you can check offical sample code.
Flask "To-Do" Sample Application
import logging
import sys
from flask import Flask
sys.path.append('..')
from config import Config
from flask_sqlalchemy import SQLAlchemy
from opencensus.ext.azure import metrics_exporter
from opencensus.ext.azure.log_exporter import AzureLogHandler
from opencensus.ext.flask.flask_middleware import FlaskMiddleware
from opencensus.trace import config_integration
logger = logging.getLogger(__name__)
app = Flask(__name__)
app.config.from_object(Config)
db = SQLAlchemy(app)
# Import here to avoid circular imports
from app import routes # noqa isort:skip
# Trace integrations for sqlalchemy library
config_integration.trace_integrations(['sqlalchemy'])
# Trace integrations for requests library
config_integration.trace_integrations(['requests'])
# FlaskMiddleware will track requests for the Flask application and send
# request/dependency telemetry to Azure Monitor
middleware = FlaskMiddleware(app)
# Processor function for changing the role name of the app
def callback_function(envelope):
envelope.tags['ai.cloud.role'] = "To-Do App"
return True
# Adds the telemetry processor to the trace exporter
middleware.exporter.add_telemetry_processor(callback_function)
# Exporter for metrics, will send metrics data
exporter = metrics_exporter.new_metrics_exporter(
enable_standard_metrics=False,
connection_string='InstrumentationKey=' + Config.INSTRUMENTATION_KEY)
# Exporter for logs, will send logging data
logger.addHandler(
AzureLogHandler(
connection_string='InstrumentationKey=' + Config.INSTRUMENTATION_KEY
)
)
if __name__ == '__main__':
app.run(host='localhost', port=5000, threaded=True, debug=True)
I'm attempting to use structlog in an AWS Lambda function and directly stream log events to our Logstash server. To this end, I'm using the logstash-async library using the beats transport layer.
The code:
from constants import LOGSTASH_HOST, LOGSTASH_PORT
from logstash_async.handler import AsynchronousLogstashHandler
from logstash_async.transport import BeatsTransport
from s4utils.exceptions import CatConfigNotFoundException
import structlog
from structlog.stdlib import LoggerFactory, BoundLogger
def get_log():
logstash_handler = AsynchronousLogstashHandler(
LOGSTASH_HOST,
LOGSTASH_PORT,
database_path=None,
transport=BeatsTransport)
logging.basicConfig(
level=logging.INFO,
format='%(message)s',
handlers=[logstash_handler, logging.StreamHandler(sys.stdout)])
structlog.configure(
wrapper_class=BoundLogger,
logger_factory=LoggerFactory(),
processors=[structlog.processors.TimeStamper(fmt='iso', utc=True),
structlog.processors.JSONRenderer()])
return structlog.get_logger().new(fields={'type': 'aws-lambda'})
I find that while I can see the stdout logging in CloudWatch, the Logstash server doesn't appear to be receiving events.
My Stack:
Google App Engine Standard
Python (2.7)
Goal:
To create named logs in Google Stackdriver Logging, https://console.cloud.google.com/logs/viewer
Docs - Stackdriver Logging:
https://google-cloud-python.readthedocs.io/en/latest/logging/usage.html
Code:
from google.cloud import logging as stack_logging
from google.cloud.logging.resource import Resource
import threading
class StackdriverLogging:
def __init__(self, resource=Resource(type='project', labels={'project_id': 'project_id'}), project_id='project_id'):
self.resource = resource
self.client = stack_logging.Client(project=project_id)
def delete_logger(self, logger_name):
logger = self.client.logger(logger_name)
logger.delete()
def async_log(self, logger_name, sev, msg):
t = threading.Thread(target=self.log, args=(logger_name, sev, msg,))
t.start()
def log(self, logger_name, sev, msg):
logger = self.client.logger(logger_name)
if isinstance(msg, str):
logger.log_text(msg, severity=sev, resource=self.resource)
elif isinstance(msg, dict):
logger.log_struct(msg, severity=sev, resource=self.resource)
class hLog(webapp2.RequestHandler):
def get(self):
stackdriver_logger = StackdriverLogging()
stackdriver_logger.async_log("my_new_log", "WARNING", msg="Hello")
stackdriver_logger.async_log("my_new_log", "INFO", msg="world")
ERROR:
Found 1 RPC request(s) without matching response
If this is not possible in Google App Engine Standard (Python) any way to get this code to work:
from google.cloud import logging
client = logging.Client()
# client = logging.Client.from_service_account_json('credentials.json')
logger = client.logger("my_new_log")
logger.log_text("hello world")
If credentials are required, I like to use the project service account.
Any help would be appreciated. Thank you.
I usually tie the Python logging module directly into Google Stackdriver Logging.
To do this, I create a log_helper module:
from google.cloud import logging as gc_logging
import logging
logging_client = gc_logging.Client()
logging_client.setup_logging(logging.INFO)
from logging import *
Which I then import into other files like this:
import log_helper as logging
After which you can use the module like you would the default python logging module.
To create named logs with the default python logging module, use different loggers for different namespaces:
import log_helper as logging
test_logger = logging.getLogger('test')
test_logger.setLevel(logging.INFO)
test_logger.info('is the name of this logger')
Output:
INFO:test:is the name of this logger
In my Celery application I am getting 2 types of logs on the console i.e celery application logs and task level logs (inside task I am using logger.INFO(str) syntax for logging)
I wanted to send both of them to a custom handler (in my case python-logstash handler )
For django logs I was successfull, by setting handler and logger in settings.py but I am helpless with celery
def initialize_logstash(logger=None,loglevel=logging.DEBUG, **kwargs):
# logger = logging.getLogger('celery')
handler = logstash.TCPLogstashHandler('localhost', 5959,tags=['worker'])
handler.setLevel(loglevel)
logger.addHandler(handler)
# logger.setLevel(logging.DEBUG)
return logger
from celery.signals import after_setup_task_logger
after_setup_task_logger.connect(initialize_logstash)
from celery.signals import after_setup_logger
after_setup_logger.connect(initialize_logstash)
using both after_setup_task_logger and after_setup_logger signals solved the problem
Celery provides a after_setup_logger signal that is triggered after Celery has set up the logger. Among other few arguments, the signal passes the logger object which you can add your custom logging handlers to.
from celery import signals
import logstash
import logging
#signals.after_setup_logger.connect
def setup_logstash_logger(logger, *args, **kwargs):
handler = logstash.TCPLogstashHandler('localhost', 5959)
# More logger/handler configuration
# handler.setLevel(logging.ERROR)
# ...
logger.addHandler(handler)
After fine-tuning Celery's logger you can simply rely on it to send the app messages to Logstash, even if you need to send your own messages:
logger = logging.getLogger(__name__)
logger.info('My message') # This message will also be sent to Logstash