I have problem sending loggging stream over udp to targeted third part machine that collects log streams.
I've got just couple of lines of init to appear on remote console. The encoding over the udp should be UTF-8, too.
## setup_loggin function body
handler=logging.StreamHandler()
handler.setFormatter(formatter)
socketHandler=logging.handlers.SysLogHandler(address(SYSLOG_IP, 514))
socketHandler.setFormatter(formatter)
logger = logging.getLogger(name)
if not logger.handlers:
logger.setLevel(log_level)
logger.addHandler(handler)
logger.addHandler(socketHandler)
return logger
The remote server gets just begining of the logging at end results. Probably because it expects UDP to receive "UTF-8", and than parses it through.
Is there a way to change logging encoding to "UTF-8" using SysLogHandler or any other loggin handler.?
Problem solved. Third party syslog was parsing based on RFC3164
The MSG part has two fields known as the TAG field and the CONTENT field.
So, formatter would be something like
formatter_syslog = logging.Formatter(fmt='foo: %(levelname)s - %(module)s.%(funcName)s - %(message)s')
Related
I'm writing a Glue ETL, and I'm trying to log using Python default's logger.
The problem is that all the log messages I'm printing using the logger appear in the error stream of the job.
If I print directly to stdout (using print), I see the printed messages in the regular cloudwatch log stream.
I tried to redirect my logger to stdout, but I still got the same result: messages appear in error stream.
Does anyone know how I can use logger, and still see my messages in the log cloudwatch stream? (and not in the error cloudwatch stream)
This is the code sample I'm using to test:
import logging
import sys
MSG_FORMAT = '%(asctime)s %(levelname)s %(name)s: %(message)s'
DATETIME_FORMAT = '%Y-%m-%d %H:%M:%S'
logging.basicConfig(format=MSG_FORMAT, datefmt=DATETIME_FORMAT, stream=sys.stdout)
logger = logging.getLogger()
logger.setLevel(logging.INFO)
logger.info("Test log message. This appear on the job error cloudwatch stream")
print("This is a print. This appear on the job log cloudwatch stream")
I ended up adding
logging.getLogger().addHandler(logging.StreamHandler(sys.stdout))
to my logger definition.
That resolved the problem
I am writing a script that connects to N hosts via SSH ... queries the 3rd party system and extracts data and then displays all the collected data in a certain format.
I want to log all the actions the script is executing as well as any exceptions encountered on to the console and to a log file, so that the user can see what is happening while the script is running (If someone used Ansible - then just like the output we get on the console and logs when running the playbooks)
Expected output
[timestamp]: connecting machine 1
[timestamp]: connection established
[timestamp]: querying database xyz
[timestamp]: ERR: invalid credentials
[timestamp]: aborting data extraction
[timestamp]: connection closed
[timestamp]: ---------------------------
[timestamp]: connecting machine 2
[timestamp]: connection established
[timestamp]: querying database xyz
[timestamp]: extraction complete
[timestamp]: closing the connection
I hope I am able to explain it correctly - Logging actions and exceptions with timestamp for the whole script and all the data iterations.
Please advice and if possible with an example script that uses the technique. Thanks
You can have a look here for some more detailed guidance. Here's how I usually set up logging on my stuff:
import logging
...
logger = logging.getLogger()
log_handler = logging.StreamHandler(sys.stdout)
log_handler.setFormatter(logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s - %(funcName)s - line %(lineno)d"))
log_handler.setLevel(logging.DEBUG)
logger.addHandler(log_handler)
logger.setLevel(logging.DEBUG)
This will produce output like this, for any event from DEBUG upwards:
2017-05-16 13:30:03,193 - root - INFO - Starting execution - main - line 35
2017-05-16 13:30:03,206 - root - DEBUG - Config file grabbed successfully - readConfig - line 71
...
2017-05-15 13:30:26,792 - root - WARNING - Reached maximum number of attempts (3) for this request; skipping request. - main - line 79
2017-05-15 13:30:26,797 - root - ERROR - Failed to grab item. Unfortunately, this is a showstopper :( - main - line 79
The above is produced by a line in the main function of my app, that reads:
logger.info("Starting execution")
Another line in my readConfig function:
logging.debug("Config file grabbed successfully")
And another two lines in main again:
logging.warning("Reached maximum number of attempts ({max_attempts}) for this request; skipping request.".format(max_attempts = max_tries))
...
logging.error("Failed to grab item. Unfortunately, this is a showstopper :(")
Then it's a matter of how much information and context you need on each log entry. Have a look here at formatting the entries, and here at formatters. I'll have these sent to me via email, anytime the app runs triggered by crontab, by adding MAILTO = root to the top of my crontab file, and making sure my system email is properly set.
If you want to set it to go to the console and a file, you'll just need to set two different handlers. This answer provides a good example, where you'd set a StreamHandler to log to the console, and a FileHandler to log to a file. So instead of setting it up as I mentioned above I usually do, you could try:
import logging
...
# Set up logging and formatting
logger = logging.getLogger()
logFormatter = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s - %(funcName)s - line %(lineno)d")
# Set up the console handler
consoleHandler = logging.StreamHandler()
consoleHandler.setFormatter(logFormatter)
logger.addHandler(consoleHandler)
# Set up the file handler
fileHandler = logging.FileHandler("{0}/{1}.log".format(logPath, fileName))
fileHandler.setFormatter(logFormatter)
logger.addHandler(fileHandler)
# Set up logging levels
consoleHandler.setLevel(logging.DEBUG)
fileHandler.setLevel(logging.DEBUG)
logger.setLevel(logging.DEBUG)
Check out the logging module here, there's a nice example section with both basic and advanced applications. Doing stuff in the format you've described appears to be included in the tutorial.
I am using logging for generating logs on my python (2.7) project. I create one instance of logger class on each python file (LOGGER = logging.getLogger(__name__) like this).
I want to send all the logs to syslog. Ways I have found using SysLogHandler needs to specify the address (/dev/log, /var/run/syslog etc.) of syslogs which can be different on different platforms.
Is there any generic way to do it (send the logs to syslog)?
As you found, you configure logging to use the SysLogHandler. As you also noted, the "address" will be different on different systems and configurations. You will need to make that a configurable part of your application.
If you are configuring the logging programmatically, you could implement some heuristics to search for the various UNIX domain socket paths you expect to find.
Note the syslog can use UDP or TCP sockets to log to a remote syslog server instead of using a UNIX domain socket on the local machine.
import sys, logging
class streamToLogger(object):
def __init__(self, logger, log_level = logging.INFO):
self.logger = logger
self.log_level = log_level
self.linebuf = ''
def write(self, buf):
for line in buf.rstrip().splitlines():
self.logger.log(self.log_level, line.rstrip())
logging.basicConfig(
level = logging.DEBUG,
format = '%(asctime)s:%(levelname)s:%(name)s:%(message)s',
filename = os.path.join('/var/log', 'logfile.log'),
filemode = 'a'
)
sys.stdout = streamToLogger(logging.getLogger('STDOUT'), logging.INFO)
sys.stderr = streamToLogger(logging.getLogger('STDERR'), logging.ERROR)
Having a new problem where the logger is writing about every other line without the format, just the message.
My code:
import logging
from logging.handlers import RotatingFileHandler
# Set up logging
LOG_FILE = argv[0][:-3] + '.log'
logging.basicConfig(
filename=LOG_FILE,
filemode='a',
level=logging.INFO,
format='%(asctime)s - %(levelname)s - %(message)s')
logger = logging.getLogger(__name__)
handler = RotatingFileHandler(LOG_FILE,maxBytes=1000000)
logger.addHandler(handler)
def main():
target, command, notify_address, wait = get_args(argv)
logger.info('Checking status of %s every %d minutes.' % (target, wait))
logger.info('Running %s and sending output to %s when online.' % (command, notify_address))
The vars returned from get_args() are all strings, even though wait is a number.
Note that I am not receiving any errors in my IDE or when running.
The output I am getting in my log file:
2015-02-12 16:26:27,483 - INFO - Checking status of <ip address> every 30 minutes.
Running <arbitrary bash command string> and sending output to <my email address> when online.
2015-02-12 16:26:27,483 - INFO - Running <arbitrary bash command string> and sending output to <my email address> when online.
What is causing the second logger.info() to print twice, and only once formatted properly?
I have another script that logs perfectly, no idea what I've done here. (Copy/pasted the logging setup section to be safe)
Are you using loggers at different levels of your code? It sounds like the log messages could be propagating upwards. Try adding
logger.propagate = False
after you add the handler. You can check out the python docs for a more detailed explanation here, but the relevant text below sounds exactly like what you're seeing.
Note If you attach a handler to a logger and one or more of its ancestors, it may emit the same record multiple times. In general, you should not need to attach a handler to more than one logger - if you just attach it to the appropriate logger which is highest in the logger hierarchy, then it will see all events logged by all descendant loggers, provided that their propagate setting is left set to True. A common scenario is to attach handlers only to the root logger, and to let propagation take care of the rest.
There is a script, written in Python, which parse sensors data and events from number of servers by IPMI. Then it sends graph data to one server and error logs to the other. Logging server is Syslog-ng+Mysql
So the task is to store logs by owner, but not by script host.
Some code example:
import logging
import logging.handlers
loggerCent = logging.getLogger(prodName + 'Remote')
ce = logging.handlers.SysLogHandler(address=('192.168.1.11', 514), facility='daemon')
formatter = logging.Formatter('%(name)s: %(levelname)s: %(message)s')
loggerCent.setLevel(logging.INFO)
loggerCent.addHandler(ce)
loggerCent.warning('TEST MSG')
So I need to extend the code so I could tell to syslog-ng, that the log belongs to the other host. Or some other desigion.
Any ideas?
UPD:
So it looks like there is the way to use LogAdapter. But how can use it:
loggerCent = logging.getLogger(prodName + 'Remote')
ce = logging.handlers.SysLogHandler(address=('192.168.1.11', 514), facility='daemon')
logging.basicConfig(level=logging.DEBUG,
format='%(asctime)-15s %(name)-5s %(levelname)-8s host: %(host)-15s %(message)s')
loggerCent.addHandler(ce)
loggerCent2 = logging.LoggerAdapter(loggerCent,
{'host': '123.231.231.123'})
loggerCent2.warning('TEST MSG')
Looking for the message through TcpDump I see nothing about host in LoggerAdapter
What am I doing wrong?
UPD2:
Well, I did't find the way to send host to syslog-ng. Though it is possible to send first host in chain, but I really can't find the way to sent it through Python Logging.
Anyway, I made parser in syslog-ng:
CSV Parser
parser ipmimon_log {
csv-parser(
columns("LEVEL", "UNIT", "MESSAGE")
flags(escape-double-char,strip-whitespace)
delimiters(";")
quote-pairs('""[](){}')
);
};
log {
source(s_net);
parser(ipmimon_log);
destination(d_mysql_ipmimon);
};
log {
source(s_net);
destination(d_mysql_norm);
flags(fallback);
};
Then I send logs to the syslog-ng delimited by ;
edit-rewrite
You are missing the critical step of actually adding your Formatter to your Handler. Try:
loggerCent = logging.getLogger(prodName + 'Remote')
loggerCent.setLevel(logging.DEBUG)
ce = logging.handlers.SysLogHandler(address=('192.168.1.11', 514), facility='daemon')
formatter = logging.Formatter('%(host)s;%(message)s')
ce.setFormatter(formatter)
loggerCent.addHandler(ce)
loggerCent2 = logging.LoggerAdapter(loggerCent, {'host': '123.231.231.123'})
loggerCent2.warning('TEST MSG')
Note that you won't be running basicConfig any more, so you will be responsible for attaching a Handler and setting a log level for the root logger yourself (or you will get 'no handler' errors).