We Keep Coding

Syslog not sending logs to Papertrail

This code snippet, linked here, should be sending a log message to Papertrail, unfortunately, there's nothing happening at all (no error message either). I did replace the "N" and the "XXXXX" with my own values. The environment is Python 3.10.6 and I'm inside of a virtual environment created by Poetry.
I would expect calling an undefined function to log an exception to Papertrail.
Telnet works fine and my other applications work fine as well.
import logging
import socket
from logging.handlers import SysLogHandler
syslog = SysLogHandler(address=('logsN.papertrailapp.com', XXXXX))
format = '%(asctime)s YOUR_APP: %(message)s'
formatter = logging.Formatter(format, datefmt='%b %d %H:%M:%S')
syslog.setFormatter(formatter)
logger = logging.getLogger()
logger.addHandler(syslog)
logger.setLevel(logging.INFO)
def my_handler(type, value, tb):
logger.exception('Uncaught exception: {0}'.format(str(value)))
# Install exception handler
sys.excepthook = my_handler
logger.info('This is a message')
nofunction() #log an uncaught exception

AWS Glue Python ETL: logger messages appear in the error cloudwatch stream

I'm writing a Glue ETL, and I'm trying to log using Python default's logger.
The problem is that all the log messages I'm printing using the logger appear in the error stream of the job.
If I print directly to stdout (using print), I see the printed messages in the regular cloudwatch log stream.
I tried to redirect my logger to stdout, but I still got the same result: messages appear in error stream.
Does anyone know how I can use logger, and still see my messages in the log cloudwatch stream? (and not in the error cloudwatch stream)
This is the code sample I'm using to test:
import logging
import sys
MSG_FORMAT = '%(asctime)s %(levelname)s %(name)s: %(message)s'
DATETIME_FORMAT = '%Y-%m-%d %H:%M:%S'
logging.basicConfig(format=MSG_FORMAT, datefmt=DATETIME_FORMAT, stream=sys.stdout)
logger = logging.getLogger()
logger.setLevel(logging.INFO)
logger.info("Test log message. This appear on the job error cloudwatch stream")
print("This is a print. This appear on the job log cloudwatch stream")

I ended up adding
logging.getLogger().addHandler(logging.StreamHandler(sys.stdout))
to my logger definition.
That resolved the problem

SMTPSenderRefused when sending email with Python logging

I'm using SMTPHandler to log my caught exceptions in my python script which requests new data to my backend.
Here's how I initialized my SMTPHandler:
# write errors to email
error_mail_subject = "ERROR: Script error in %s" % sys.argv[0]
error_mail_handler = logging.handlers.SMTPHandler(mailhost=("smtp.gmail.com", 587),
fromaddr="test#gmail.com",
toaddrs="test#gmail.com",
subject=error_mail_subject)
error_mail_handler.setLevel(logging.ERROR)
error_mail_handler.setFormatter(debug_format)
# Build the logger
logger = logging.getLogger()
logger.addHandler(error_mail_handler)
But unfortunately, I get this following error during handling of a script exception
smtplib.SMTPSenderRefused: (530, b'5.7.0 Must issue a STARTTLS command first. w15sm3562054qta.16 - gsmtp', 'test#gmail.com')
When the script calls:
logger.exception(e)

This successfully sent an email from my gmail, tested just now:
import logging.handlers
error_mail_subject = "hello, it's me"
error_mail_handler = logging.handlers.SMTPHandler(mailhost=("smtp.gmail.com", 587),
fromaddr="email",
toaddrs="email",
subject=error_mail_subject,
credentials=('email', 'password'),
secure=())
error_mail_handler.setLevel(logging.ERROR)
logger = logging.getLogger()
logger.addHandler(error_mail_handler)
logger.exception(Exception("hey"))
I also had to sign into gmail and then go here to allow access to less secure apps: https://www.google.com/settings/security/lesssecureapps

Application logging in Python Eve

I'd like to add kind of application logging in a REST API I'm developing with the fantastic Python EVE framework. The goal is to provide auditing features to the application, so that application logging is my first choice for it.
I've followed steps detailed in this section from the official documentation, but I haven't been lucky. Probably I'm missing something:
from eve import Eve
import logging
def log_every_get(resource, request, payload):
# custom INFO-level message is sent to the log file
app.logger.info('We just answered to a GET request!')
app = Eve()
app.on_post_GET += log_every_get
if __name__ == "__main__":
# enable logging to 'app.log' file
handler = logging.FileHandler('<chrootedfolder>/app.log')
# set a custom log format, and add request
# metadata to each log line
handler.setFormatter(logging.Formatter(
'%(asctime)s %(levelname)s: %(message)s '
'[in %(filename)s:%(lineno)d] -- ip: %(clientip)s, '
'url: %(url)s, method:%(method)s'))
# the default log level is set to WARNING, so
# we have to explictly set the logging level
# to INFO to get our custom message logged.
app.logger.setLevel(logging.INFO)
# append the handler to the default application logger
app.logger.addHandler(handler)
app.run()
To give you some more info about the environment where I'm running my REST API: it's running under UWSGI (chrooted to a specific folder) which is behind NGINX (acting as a proxy).
Hope you can help somehow guys.
Thanks in advance.

How to set source host address in Python Logging?

There is a script, written in Python, which parse sensors data and events from number of servers by IPMI. Then it sends graph data to one server and error logs to the other. Logging server is Syslog-ng+Mysql
So the task is to store logs by owner, but not by script host.
Some code example:
import logging
import logging.handlers
loggerCent = logging.getLogger(prodName + 'Remote')
ce = logging.handlers.SysLogHandler(address=('192.168.1.11', 514), facility='daemon')
formatter = logging.Formatter('%(name)s: %(levelname)s: %(message)s')
loggerCent.setLevel(logging.INFO)
loggerCent.addHandler(ce)
loggerCent.warning('TEST MSG')
So I need to extend the code so I could tell to syslog-ng, that the log belongs to the other host. Or some other desigion.
Any ideas?
UPD:
So it looks like there is the way to use LogAdapter. But how can use it:
loggerCent = logging.getLogger(prodName + 'Remote')
ce = logging.handlers.SysLogHandler(address=('192.168.1.11', 514), facility='daemon')
logging.basicConfig(level=logging.DEBUG,
format='%(asctime)-15s %(name)-5s %(levelname)-8s host: %(host)-15s %(message)s')
loggerCent.addHandler(ce)
loggerCent2 = logging.LoggerAdapter(loggerCent,
{'host': '123.231.231.123'})
loggerCent2.warning('TEST MSG')
Looking for the message through TcpDump I see nothing about host in LoggerAdapter
What am I doing wrong?
UPD2:
Well, I did't find the way to send host to syslog-ng. Though it is possible to send first host in chain, but I really can't find the way to sent it through Python Logging.
Anyway, I made parser in syslog-ng:
CSV Parser
parser ipmimon_log {
csv-parser(
columns("LEVEL", "UNIT", "MESSAGE")
flags(escape-double-char,strip-whitespace)
delimiters(";")
quote-pairs('""[](){}')
);
};
log {
source(s_net);
parser(ipmimon_log);
destination(d_mysql_ipmimon);
};
log {
source(s_net);
destination(d_mysql_norm);
flags(fallback);
};
Then I send logs to the syslog-ng delimited by ;

edit-rewrite
You are missing the critical step of actually adding your Formatter to your Handler. Try:
loggerCent = logging.getLogger(prodName + 'Remote')
loggerCent.setLevel(logging.DEBUG)
ce = logging.handlers.SysLogHandler(address=('192.168.1.11', 514), facility='daemon')
formatter = logging.Formatter('%(host)s;%(message)s')
ce.setFormatter(formatter)
loggerCent.addHandler(ce)
loggerCent2 = logging.LoggerAdapter(loggerCent, {'host': '123.231.231.123'})
loggerCent2.warning('TEST MSG')
Note that you won't be running basicConfig any more, so you will be responsible for attaching a Handler and setting a log level for the root logger yourself (or you will get 'no handler' errors).