I am trying to use python logging module to log messages to a remote rsyslog server. The messages are received, but its concatenating the messages together on one line for each message. Here is an example of my code:
to_syslog_priority: dict = {
Level.EMERGENCY: 'emerg',
Level.ALERT: 'alert',
Level.CRITICAL: 'crit',
Level.ERROR: 'err',
Level.NOTICE: 'notice',
Level.WARNING: 'warning',
Level.INFO: 'info',
Level.DEBUG: 'debug',
Level.PERF: 'info',
Level.AUDIT: 'info'
}
#staticmethod
def make_logger(*, name: str, log_level: Level, rsyslog_address: Tuple[str, int], syslog_facility: int) -> Logger:
"""Initialize the logger with the given attributes"""
logger = logging.getLogger(name)
num_handlers = len(logger.handlers)
for i in range(0, num_handlers):
logger.removeHandler(logger.handlers[0])
logger.setLevel(log_level.value)
syslog_priority = Log.to_syslog_priority[log_level]
with Timeout(seconds=RSYSLOG_TIMEOUT, timeout_message="Cannot reach {}".format(rsyslog_address)):
sys_log_handler = handlers.SysLogHandler(rsyslog_address, syslog_facility, socket.SOCK_STREAM)
# There is a bug in the python implementation that prevents custom log levels
# See /usr/lib/python3.6/logging/handlers.SysLogHandler.priority_map on line 789. It can only map
# to 5 log levels instead of the 8 we've implemented.
sys_log_handler.mapPriority = lambda *args: syslog_priority
logger.addHandler(sys_log_handler)
stdout_handler = logging.StreamHandler(sys.stdout)
logger.addHandler(stdout_handler)
return logger
if __name__ == '__main__':
app_logger = Log.make_logger(name='APP', log_level=Log.Level.INFO, rsyslog_address=('localhost', 514),
syslog_facility=SysLogHandler.LOG_USER)
audit_logger = Log.make_logger(name='PERF', log_level=Log.Level.INFO, rsyslog_address=('localhost', 514),
syslog_facility=SysLogHandler.LOG_LOCAL0)
perf_logger = Log.make_logger(name='AUDIT', log_level=Log.Level.INFO, rsyslog_address=('localhost', 514),
syslog_facility=SysLogHandler.LOG_LOCAL1)
log = Log(log_level=Log.Level.WARNING, component='testing', worker='tester', version='1.0', rsyslog_srv='localhost',
rsyslog_port=30514)
app_logger.warning("Testing warning logging")
perf_logger.info("Testing performance logging1")
audit_logger.info("Testing aduit logging1")
audit_logger.info("Testing audit logging2")
app_logger.critical("Testing critical logging")
perf_logger.info("Testing performance logging2")
audit_logger.info("Testing audit logging3")
app_logger.error("Testing error logging")
On the server side, I'm added the following the following line to the /etc/rsyslog.d/50-default.conf to disable /var/log/syslog logging for USER, LOCAL0 and LOCAL1 facilities (which I use for app, perf, and audit logging).
*.*;user,local0,local1,auth,authpriv.none -/var/log/syslog
And I update the to the /etc/rsyslog.config:
# /etc/rsyslog.conf Configuration file for rsyslog.
#
# For more information see
# /usr/share/doc/rsyslog-doc/html/rsyslog_conf.html
#
# Default logging rules can be found in /etc/rsyslog.d/50-default.conf
#################
#### MODULES ####
#################
module(load="imuxsock") # provides support for local system logging
#module(load="immark") # provides --MARK-- message capability
# provides UDP syslog reception
#module(load="imudp")
#input(type="imudp" port="514")
# provides TCP syslog reception
module(load="imtcp")
input(type="imtcp" port="514")
# provides kernel logging support and enable non-kernel klog messages
module(load="imklog" permitnonkernelfacility="on")
###########################
#### GLOBAL DIRECTIVES ####
###########################
#
# Use traditional timestamp format.
# To enable high precision timestamps, comment out the following line.
#
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
# Filter duplicated messages
$RepeatedMsgReduction on
#
# Set the default permissions for all log files.
#
$FileOwner syslog
$FileGroup adm
$FileCreateMode 0640
$DirCreateMode 0755
$Umask 0022
$PrivDropToUser syslog
$PrivDropToGroup syslog
#
# Where to place spool and state files
#
$WorkDirectory /var/spool/rsyslog
#
# Include all config files in /etc/rsyslog.d/
#
$IncludeConfig /etc/rsyslog.d/*.conf
user.* -/log/app.log
local0.* -/log/audit.log
local1.* -/log/perf.log
So after doing all that when I run the python code (listed above) these are these are the messages I'm seeing on the remote server:
for log in /log/*.log; do echo "${log} >>>"; cat ${log}; echo "<<< ${log}"; echo; done
/log/app.log >>>
Oct 23 13:00:23 de4bba6ac1dd rsyslogd: imklog: cannot open kernel log (/proc/kmsg): Operation not permitted.
Oct 23 13:01:34 Testing warning logging#000<14>Testing critical logging#000<14>Testing error logging
<<< /log/app.log
/log/audit.log >>>
Oct 23 13:01:34 Testing aduit logging1#000<134>Testing audit logging2#000<134>Testing audit logging3
<<< /log/audit.log
/log/perf.log >>>
Oct 23 13:01:34 Testing performance logging1#000<142>Testing performance logging2
<<< /log/perf.log
As you can see the messages are being filtered to the proper log file, but they're being concatenated onto one line. I'm guessing that its doing it because they arrive at the same time, but I'd like the messages to be split onto separate lines.
In addition, I've tried adding a formatter to the SysLogHandler so that it inserts a line break to the message like this:
sys_log_handler.setFormatter(logging.Formatter('%(message)s\n'))
However, this really screws it up:
for log in /log/*.log; do echo "${log} >>>"; cat ${log}; echo "<<< ${log}"; echo; done
/log/app.log >>>
Oct 23 13:00:23 de4bba6ac1dd rsyslogd: imklog: cannot open kernel log (/proc/kmsg): Operation not permitted.
Oct 23 13:01:34 Testing warning logging#000<14>Testing critical logging#000<14>Testing error logging
Oct 23 13:12:00 Testing warning logging
Oct 23 13:12:00 172.17.0.1 #000<134>Testing audit logging2
Oct 23 13:12:00 172.17.0.1 #000<14>Testing critical logging
Oct 23 13:12:00 172.17.0.1 #000<142>Testing performance logging2
Oct 23 13:12:00 172.17.0.1 #000<134>Testing audit logging3
Oct 23 13:12:00 172.17.0.1 #000<14>Testing error logging
Oct 23 13:12:00 172.17.0.1
<<< /log/app.log
/log/audit.log >>>
Oct 23 13:01:34 Testing aduit logging1#000<134>Testing audit logging2#000<134>Testing audit logging3
Oct 23 13:12:00 Testing aduit logging1
<<< /log/audit.log
/log/perf.log >>>
Oct 23 13:01:34 Testing performance logging1#000<142>Testing performance logging2
Oct 23 13:12:00 Testing performance logging1
<<< /log/perf.log
As you can see the first message gets put into the right file for the audit and performance logs, but then all the other messages get put into the application log file. However, there is definitely a line break now.
My question is, I want to filter the messages based on facility, but with each message on a seperate line. How can I do this using the python logging library? I guess I could take a look at the syslog library?
So I came across this python bug:
https://bugs.python.org/issue28404
So I took a look at the source code (nice thing about python), specifically the SysLogHander.emit() method:
def emit(self, record):
"""
Emit a record.
The record is formatted, and then sent to the syslog server. If
exception information is present, it is NOT sent to the server.
"""
try:
msg = self.format(record)
if self.ident:
msg = self.ident + msg
if self.append_nul:
# Next line is always added by default
msg += '\000'
# We need to convert record level to lowercase, maybe this will
# change in the future.
prio = '<%d>' % self.encodePriority(self.facility,
self.mapPriority(record.levelname))
prio = prio.encode('utf-8')
# Message is a string. Convert to bytes as required by RFC 5424
msg = msg.encode('utf-8')
msg = prio + msg
if self.unixsocket:
try:
self.socket.send(msg)
except OSError:
self.socket.close()
self._connect_unixsocket(self.address)
self.socket.send(msg)
elif self.socktype == socket.SOCK_DGRAM:
self.socket.sendto(msg, self.address)
else:
self.socket.sendall(msg)
except Exception:
self.handleError(record)
As you can see it adds a '\000' to the end of the message by default, so if I set this to False and then set a Formatter that adds a line break, then things work the way I expect. Like this:
sys_log_handler.mapPriority = lambda *args: syslog_priority
# This will add a line break to the message before it is 'emitted' which ensures that the messages are
# split up over multiple lines, see https://bugs.python.org/issue28404
sys_log_handler.setFormatter(logging.Formatter('%(message)s\n'))
# In order for the above to work, then we need to ensure that the null terminator is not included
sys_log_handler.append_nul = False
Thanks for your help #Sraw, I tried to use UDP, but never got the message. After applying these changes this is what I see in my log files:
$ for log in /tmp/logging_test/*.log; do echo "${log} >>>"; cat ${log}; echo "<<< ${log}"; echo; done
/tmp/logging_test/app.log >>>
Oct 23 21:06:40 083c9501574d rsyslogd: imklog: cannot open kernel log (/proc/kmsg): Operation not permitted.
Oct 23 21:06:45 Testing warning logging
Oct 23 21:06:45 Testing critical logging
Oct 23 21:06:45 Testing error logging
<<< /tmp/logging_test/app.log
/tmp/logging_test/audit.log >>>
Oct 23 21:06:45 Testing audit logging1
Oct 23 21:06:45 Testing audit logging2
Oct 23 21:06:45 Testing audit logging3
<<< /tmp/logging_test/audit.log
/tmp/logging_test/perf.log >>>
Oct 23 21:06:45 Testing performance logging1
Oct 23 21:06:45 Testing performance logging2
<<< /tmp/logging_test/perf.log
I believe tcp stream makes it more complicated. When you are using tcp stream, rsyslog won't help you to split the message, it is all on your own.
Why not use udp protocol? In this case, every single message will be treated as a single log. So you don't need to add \n manually. And manually adding \n will makes you impossible to log multiple line logs correctly.
So my suggestion is changing to udp protocol and:
# Disable escaping to accept multiple line log
$EscapeControlCharactersOnReceive off
I came accross the same issue.
capture multiline events with rsyslog and storing them to file
The cause is related to the question above.
NULL character is not handled as delimilter in imtcp.
My solution was setting AddtlFrameDelimiter="0", something like below.
module(load="imtcp" AddtlFrameDelimiter="0")
input(type="imtcp" port="514")
reference:
AddtlFrameDelimiter
Syslog Message Format
related rsyslog code
EDITED 2022/10/04
The previous example was using input parameter.
And AddtlFrameDelimiter's input paramter has been newly supported since v8.2106.0.
Therefore, I changed the input paramter's example into old style module golbal parameter's one.
# module golbal parameter case, supported since v6.XXXX
module(load="imtcp" AddtlFrameDelimiter="0")
input(type="imtcp" port="514")
# input paramter case, supported since v8.2106.0
module(load="imtcp")
input(type="imtcp" port="514" AddtlFrameDelimiter="0")
Related
I'm trying to send a zip file to a remote server via sftp but it keeps holding up the script. On windows with a decent connection the file is sent without a problem but on a linux it doesn't work. I've tried raising an exception when the upload finishes but with no luck.
These are the log messages:
Jul 28 10:24:40 igt22-dev mqtt.sh[1353]: 46.21875 46.258296966552734
Jul 28 10:24:40 igt22-dev mqtt.sh[1353]: 46.25 46.258296966552734
Jul 28 10:24:40 igt22-dev mqtt.sh[1353]: 46.258296966552734 46.258296966552734
Jul 28 10:24:40 igt22-dev mqtt.sh[1353]: finished
and this is the code:
ftp_client = server_client.open_sftp()
source = "/usr/local/bin/log_history.zip"
destination = "/home/ubuntu/log_history.zip"
def callback_fun(current, maximum):
print(current / 1048576, maximum / 1048576)
sys.stdout.flush()
if current == maximum:
print("finished")
sys.stdout.flush()
raise Exception("finished")
ftp_client.put(source, destination, callback=callback_fun)
ftp_client.close()
print("exited")
sys.stdout.flush()
Not a fix but I managed to find a workaround. I run the code in a separate thread and when it finishes i just kill it
In order to improve readability I want to change level names for my logging system. My current approach is to use logging.addLevelName(). Also I want the logger write to stderr and syslog. I can achieve both with the following code:
import logging
import logging.handlers
logging.basicConfig(level=logging.DEBUG)
logging.addLevelName(logging.CRITICAL, "(CC)")
logging.addLevelName(logging.ERROR, "(EE)")
logging.addLevelName(logging.WARNING, "(WW)")
logging.addLevelName(logging.INFO, "(II)")
logging.addLevelName(logging.DEBUG, "(DD)")
logging.addLevelName(logging.NOTSET, "(--)")
logger = logging.getLogger('mylogger')
logger.addHandler(logging.handlers.SysLogHandler(address="/dev/log"))
handler = logging.StreamHandler()
handler.setFormatter(logging.Formatter(
'[%(asctime)s %(levelname)s] %(message)s', datefmt="%y%m%d-%H%M%S"))
logger.addHandler(handler)
logger.propagate = False
logger.debug("debug")
logger.info("info")
logger.warning("warning")
logger.error("error")
logger.critical("critical")
The output on the terminal now looks like this:
[180303-224014 (DD)] debug
[180303-224014 (II)] info
[180303-224014 (WW)] warning
[180303-224014 (EE)] error
[180303-224014 (CC)] critical
But unfortunately the syslog-handler now writes all messages to the WARNING level:
Mar 3 22:40:14 user.warning debug
Mar 3 22:40:14 user.warning info
Mar 3 22:40:14 user.warning warning
Mar 3 22:40:14 user.warning error
Mar 3 22:40:14 user.warning critical
Is this a bug in Pythons logging module? Is there a better way to just set the strings used for '%(levelname)s'?
It might qualify as a bug. The SysLogHandler uses self.priority_map.get(levelName, "warning") to determine the priority to send to syslog, and it appears addLevelName doesn't (or possibly cannot) update the map. As a workaround, you could update it manually:
SysLogHandler.priority_names.update({
'(CC)': logging.LOG_CRIT,
'(EE)': logging.LOG_ERR,
# etc
})
I'm logging messages to syslog with Python's SysLogHandler. The problem is that startswith combined with a template seems to "eat" the beginning of the logged string.
Rsyslogd is version 8.4.2, Python 2.7.9 (same behaviour on 2.7.11). It does not seem to happen on rsyslogd 7.x with Python 2.7.4 however.
Example:
#!/usr/bin/env python
import logging
from logging.handlers import SysLogHandler
my_fmt = logging.Formatter('%(name)s:%(message)s', '%Y-%m-%d %H:%M:%S')
foo_handler = SysLogHandler(address='/dev/log', facility=SysLogHandler.LOG_LOCAL5)
foo_handler.setLevel(logging.INFO)
foo_handler.setFormatter(my_fmt)
foo = logging.getLogger('foo')
foo.setLevel(logging.INFO)
foo.addHandler(foo_handler)
foo.propagate = False
foo.info("This is foo")
With this rsyslog configuration:
$template myt,"%TIMESTAMP:::date-rfc3339%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n"
if $syslogfacility-text == "local5" then {
if $msg startswith "foo" then {
action(type="omfile" file="/var/log/foo.log" template="myt")
} else {
action(type="omfile" file="/var/log/bar.log" template="myt")
}
stop
}
Produces the following:
=> /var/log/bar.log <==
2016-06-29T17:29:55.330941+01:00 is foo
Notice the missing 'This' in the message.
Conversely, removing the use the template in rsyslog config file results in:
==> /var/log/bar.log <==
Jun 29 18:19:40 localhost foo:This is foo
Removing %msg:::sp-if-no-1st-sp% from the template does not seem to help either.
The solution appears to be:
Use $syslogtag startswith instead of $msg startswith
In the python source, separate the name from the rest of the string with an empty space: logging.Formatter('%(name)s: %(message)s', '%Y-%m-%d %H:%M:%S')
I am unsure why this wasn't a problem on 2.7.4 and if anyone finds the reason please post a comment below.
In order to write log messages of "myapp" into /var/log/local5.log, I use SysLogHandler.
problem
"myapp" runs well, no error, but nothing gets logged, /var/log/local5.log remains empty.
logging configuration
Relevant parts of the logging configuration file:
handlers:
mainHandler:
class: logging.handlers.SysLogHandler
level: INFO
formatter: defaultFormatter
address: '/dev/log'
facility: 'local5'
loggers:
__main__:
level: INFO
handlers: [mainHandler]
logging test
Here is how I try to write a log in the main script of "myapp":
with open('myconfig.yml') as f:
logging.config.dictConfig(yaml.load(f))
log = logging.getLogger(__name__)
log.info("Starting")
I have added some sys.stderr.write() to /usr/lib/python3.4/logging/handlers.py to see what's happening and I get:
$ myapp
[SysLogHandler._connect_unixsocket()] Sucessfully connected to socket: /dev/log
[SysLogHandler.emit()] called
[SysLogHandler.emit()] msg=b'<174>2016/04/23 07:17:00.453 myapp: main: Starting\x00'
[SysLogHandler.emit()] msg sent to unix socket (no OSError)
rsyslog configuration
/etc/rsyslog.conf (relevant parts; TCP and UDP syslog receptions are disabled):
$ModLoad imuxsock # provides support for local system logging
$ModLoad imklog # provides kernel logging support
[...]
$IncludeConfig /etc/rsyslog.d/*.conf
/etc/rsyslog.d/40-local.conf:
local5.* /var/log/local5.log
rsyslog test
According to lsof output, it looks like rsyslogd is listening to /dev/log (or am I wrong?):
# lsof | grep "/dev/log"
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/1000/gvfs
Output information may be incomplete.
rsyslogd 28044 syslog 0u unix 0xffff8800b4b9b100 0t0 3088160 /dev/log
in:imuxso 28044 28045 syslog 0u unix 0xffff8800b4b9b100 0t0 3088160 /dev/log
in:imklog 28044 28046 syslog 0u unix 0xffff8800b4b9b100 0t0 3088160 /dev/log
rs:main 28044 28047 syslog 0u unix 0xffff8800b4b9b100 0t0 3088160 /dev/log
I don't put the whole rsyslogd -N1 output since it's a bit long, but the mentionning"local" lines:
# rsyslogd -N1 | grep local
rsyslogd: version 7.4.4, config validation run (level 1), master config /etc/rsyslog.conf
3119.943361369:7f39080fc780: cnf:global:cfsysline: $ModLoad imuxsock # provides support for local system logging
3119.944034769:7f39080fc780: rsyslog/glbl: using '127.0.0.1' as localhost IP
3119.946084095:7f39080fc780: requested to include config file '/etc/rsyslog.d/40-local.conf'
3119.946135638:7f39080fc780: config parser: pushed file /etc/rsyslog.d/40-local.conf on top of stack
3119.946432390:7f39080fc780: config parser: resume parsing of file /etc/rsyslog.d/40-local.conf at line 1
3119.946678298:7f39080fc780: config parser: reached end of file /etc/rsyslog.d/40-local.conf
3119.946697644:7f39080fc780: Decoding traditional PRI filter 'local5.*'
3119.946723904:7f39080fc780: symbolic name: local5 ==> 168
3119.949560475:7f39080fc780: PRIFILT 'local5.*'
3119.949675782:7f39080fc780: ACTION 0x224cda0 [builtin:omfile:/var/log/local5.log]
3119.953397587:7f39080fc780: PRIFILT 'local5.*'
3119.953806713:7f39080fc780: ACTION 0x224cda0 [builtin:omfile:/var/log/local5.log]
rsyslogd: End of config validation run. Bye.
I don't understand what I am missing. rsyslog's documentation matching the version I use (7.4.4) seems outdated and I can't find my way in it. Not sure that's the place to find how to fix my problem.
EDITS:
It's not possible to define a "personal" facility, like "myapp" (even if it's defined in rsyslog.conf, so I changed to use the 'local5' one.
Cause of the problem
I finally found out that I previously created /var/log/local5.log with inappropriate owner and group (root:root). They were inappropriate because /etc/rsyslog.conf tells explicitely owner and group should be syslog:syslog:
#
# Set the default permissions for all log files.
#
$FileOwner syslog
$FileGroup adm
$FileCreateMode 0640
$DirCreateMode 0755
$Umask 0022
$PrivDropToUser syslog
$PrivDropToGroup syslog
Unfortunately, the other log files rsyslog should take care of (like auth.log) were also root:root, so, seen from ls -lah, mine was not different from others... (what are also empty, I wonder why such a non-functional configuration is installed by default).
Unfortunately, rsyslog does not log any error (or at least I haven't found where).
Some more details that could be useful to finish rsyslog configuration
As a side note, rsyslog expects a special format for the messages it gets, and if it doesn't, it adds some informations, by default (timestamp hostname). It's possible to modify them. Anyway, from my python script, I decided to only send the message to log and let rsyslog format the output. So finally, the relevant parts of my logging configuration file are:
formatters:
rsyslogdFormatter:
format: '%(filename)s: %(funcName)s: %(message)s'
handlers:
mainHandler:
class: logging.handlers.SysLogHandler
level: INFO
formatter: rsyslogdFormatter
address: '/dev/log'
facility: 'local5'
loggers:
__main__:
level: INFO
handlers: [mainHandler]
And I added a customized template in /etc/rsyslog.conf:
$template MyappTpl,"%$now% %timegenerated:12:23:date-rfc3339% %syslogtag%%msg%\n"
and accordingly modified /etc/rsyslog.d/40-local.conf:
local5.* /var/log/local5.log;MyappTpl
I also want to mention that the documentation provided by the matching package (rsyslog-doc for ubuntu) matches the installed version, of course, and provides hints I hadn't found in the online documentation.
I am trying to log data to stderr and into a file. The file should contain all log messages, and to stderr should go only the log level configured on the command line. This is described several times in the logging howto - but it does not seem to work for me. I have created a small test script which illustrates my problem:
#!/usr/bin/env python
import logging as l
l.basicConfig(level=100)
logger = l.getLogger("me")
# ... --- === SEE THIS LINE === --- ...
logger.setLevel(l.CRITICAL)
sh = l.StreamHandler()
sh.setLevel(l.ERROR)
sh.setFormatter(l.Formatter('%(levelname)-8s CONSOLE %(message)s'))
logger.addHandler(sh)
fh = l.FileHandler("test.dat", "w")
fh.setLevel(l.DEBUG)
fh.setFormatter(l.Formatter('%(levelname)-8s FILE %(message)s'))
logger.addHandler(fh)
logger.info("hi this is INFO")
logger.error("well this is ERROR")
In line 5th code line I can go for logger.setLevel(l.CRITICAL) or logger.setLevel(l.DEBUG). Both results are unsatisfying.
With logger.setLevel(l.CRITICAL) I get ...
$ python test.py
$ cat test.dat
$
Now with logger.setLevel(l.DEBUG) I get ...
$ python test.py
INFO:me:hi this is INFO
ERROR CONSOLE well this is ERROR
ERROR:me:well this is ERROR
$ cat test.dat
INFO FILE hi this is INFO
ERROR FILE well this is ERROR
$
In one case I see nothing nowhere, in the other I see everything everywhere, and one message is being displayed even twice on the console.
Now I get where the ERROR CONSOLE and ERROR FILE outputs come from, those I expect. I don't get where the INFO:me... or ERROR:me... outputs are coming from, and I would like to get rid of them.
Things I already tried:
Creating a filter as described here: https://stackoverflow.com/a/7447596/902327 (does not work)
Emptying handlers from the logger with logger.handlers = [] (also does not work)
Can somebody help me out here? It seems like a straightforward requirement and I really don't seem to get it.
You can set the root level to DEBUG, set propagate to False and then set the appropriate level for the other handlers.
import logging as l
l.basicConfig()
logger = l.getLogger("me")
# ... --- === SEE THIS LINE === --- ...
logger.setLevel(l.DEBUG)
logger.propagate = False
sh = l.StreamHandler()
sh.setLevel(l.ERROR)
sh.setFormatter(l.Formatter('%(levelname)-8s CONSOLE %(message)s'))
logger.addHandler(sh)
fh = l.FileHandler("test.dat", "w")
fh.setLevel(l.INFO)
fh.setFormatter(l.Formatter('%(levelname)-8s FILE %(message)s'))
logger.addHandler(fh)
logger.info("hi this is INFO")
logger.error("well this is ERROR")
Output:
~$ python test.py
ERROR CONSOLE well this is ERROR
~$ cat test.dat
INFO FILE hi this is INFO
ERROR FILE well this is ERROR