i making a script in python for reading the last 5 minutes of a log file, this is my code so far
from datetime import datetime, timedelta
now = datetime.now()
before = timedelta(minutes=5)
now = now.replace(microsecond=0)
before = (now-before)
now = (now.strftime("%b %d %X"))
before = (before.strftime("%b %d %X"))
print(before)
print(now)
with open('user.log','r') as f:
for line in f:
if before in line:
break
for line in f:
if now in line:
break
print (line.strip())
the output is Sep 03 11:47:25 Sep 03 11:52:25 which is the print to check if the time is correct, nearly 100 lines in the log that has it but dont bring me nothing, if i take the ifs out then print all the lines which proves the problem is on the if...
any ideas?
here is a exemple of my log file content:
Sep 03 10:18:47 bni..........teagagfaesa.....
Sep 03 10:18:48 bni..........teagagfaesa.....2
I managed to find a Python even older than yours.
#!/usr/bin/env python
from __future__ import with_statement
from datetime import datetime, timedelta
before = timedelta(minutes=5)
now = datetime.now().replace(microsecond=0, year=1900)
before = (now-before)
with open('user.log','r') as f:
for line in f:
if datetime.strptime(line[0:15], '%b %d %X') < before:
continue
print line.strip()
The change compared to your code is that we convert each time stamp from the file into a datetime object; then we can trivially compare these properly machine-readable representations the way you'd expect (whereas without parsing the dates, it can't work, except by chance -- "Sep" comes after "Aug" but "Sep" comes after "Oct", too; so it seems to work if you run it in a suitable month, but then breaks the next month!)
The year=1900 hack is because strptime() defaults to year 1900 for inputs which don't have a year.
Related
I'm trying to parse dates from a textfile, but executing the scripts throws incorrect data format, when the format is correct.
The file is a .txt file with the following structure
2018/02/15 05:00:13 - somestring - anotherstring
2018/02/15 05:00:14 - somestring - anotherstring
2018/02/15 05:00:15 - somestring - anotherstring
... etc
The script gets the file divided in lines, and each line is divided on fields, of which one field is a date and time. I divided the date and the time in two separate fields, the time gets converted ok so the problem is in the date.
This is what I get on execution:
ValueError: time data "b'2018/02/15" does not match format '%Y/%m/%d'
I noticed it prints the string with a "b" in front of it, which if I'm not mistaken it means it's a byte literal. I've tried using "decode("utf-8")" on it, but it throw's exception as "string" has no method decode.
#the file is in one long string as I get it from a 'cat' bash command via ssh
file = str(stdout.read()) #reads the cat into a long string
strings = file.split("\\n") #splits the string into lines
for string in strings:
fields = string.split(" - ")
if len(fields) >= 3:
#dates.append(datetime.strptime(campos[0],"%Y/%m/%d %H:%M:%S")) #Wrong format
datentime = fields[0].split()
dates.append(datetime.strptime(datentime[0],"%Y/%m/%d")) #Wrong format
print(datentime[1])
dates.append(datetime.strptime(datentime[1],"%H:%M:%S")) #WORKS
I can't figure out why that is happening to you with the code you gave so I can't offer a fix for that but I tried testing on it and this worked for me:
datetime.strptime(str(datentime[0])[2,:-1], "%Y/%m/%d")
It removes the B and ' from the string, if you still have problems with that, please post how you got that string, maybe there was some error on the way.
use try and except:
import datetime
def convertDate(d):
strptime = datetime.datetime.strptime
try:
return strptime(d, "%Y/%m/%d")
except TypeError:
return strptime(d.decode("utf-8"), "%Y/%m/%d")
print(convertDate(b'2018/02/15'))
print(convertDate('2018/02/15'))
I need some tool to read latest 10 minutes entry in my log file, and if some words are logged then print some text.
log file:
23.07.2014 09:22:11 INFO Logging.LogEvent 0 Failed login test#test.com
23.07.2014 09:29:02 INFO Logging.LogEvent 0 login test#test.com
23.07.2014 09:31:55 INFO Logging.LogEvent 0 login test#test.com
23.07.2014 09:44:14 INFO Logging.LogEvent 0 Failed login test#test.com
if during last 10min some entry = Failed -print ALARM.
All what i did is find 'Failed' match but i have no idea how to check last 10min in my log file ;/ -any idea??
from sys import argv
from datetime import datetime, timedelta
with open('log_test.log', 'r') as f:
for line in f:
try:
e = line.index("Failed")
except:
pass
else:
print(line)
Your format %d.%m.%Y is worse than %Y:%m:%d which can be used in string comparison.
We also do not know if log is big and if it is sorted. If it is not sorted (it is common for multithreaded applications) you will have to analyze each line and convert it into datetime:
def get_dt_from_line(s):
return datetime.datetime.strptime(s[:20], '%d.%m.%Y %H:%M:%S')
Then use it as filter (for small files):
MAX_CHECK_TIMEDELTA = datetime.timedelta(minutes=10)
LOG_START_ANALYZE_DATETIME = (datetime.datetime.today() - MAX_CHECK_TIMEDELTA)
lines = [s for s in TXT.split('\n') if 'Failed' in s and get_dt_from_line(s) >= LOG_START_ANALYZE_DATETIME]
print('\n'.join(lines))
For big files you can read file line by line.
If your log file is just for one day you can use string comparison instead of datetime comparison:
LOG_START_ANALYZE_DATETIME = (datetime.datetime.today() - datetime.timedelta(minutes=10)).strftime('%d.%m.%Y %H:%M:%S')
lines = [s for s in TXT.split('\n') if 'Failed' in s and s >= LOG_START_ANALYZE_DATETIME]
If I were you, I would lookup line by line, get the timestamp of the first line and then iterate until the difference between the first date and the current one is more than 10 minutes, while counting occurences of the word "Failed".
I think that you'll sort something out with splitting your line following spaces. But be careful as if someday, your log format changes, your script is likely not gonna be working too.
i am writing a script in python that searches for strings and suposedly does different things when encounters strings.
import re, datetime
from datetime import *
f = open(raw_input('Name of file to search: ')
strToSearch = ''
for line in f:
strToSearch += line
patFinder = re.compile('\d{2}\/\d{2}\/\d{4}\sA\d{3}\sB\d{3}')
findPat1 = re.findall(patFinder, strToSearch)
# search only dates
datFinder = re.compile('\d{2}\/\d{2}\/\d{4}')
findDat = re.findall(datFinder, strToSearch)
nowDate = date.today()
fileLst = open('cels.txt', 'w')
ntrdLst = open('not_ready.txt', 'w')
for i in findPat1:
for Date in findDat:
Date = datetime.strptime(Date, '%d/%m/%Y')
Date = Date.date()
endDate = Date + timedelta(days=731)
if endDate < nowDate:
fileLst.write(i)
else:
ntrdLst.write(i)
f.close()
fileLst.close()
ntrdLst.close()
toClose = raw_input('File was modified, press enter to close: ')
so basically it searches for a string with dates and numbers and then same list but only dates, converts the dates, adds 2 years to each and compares, if the date surpass today's date, goes to the ntrdLst, if not, to fileLst.
My problem is that it writes the same list (i) multiple times and doesn't do the sorting.
i am fearly new to python and programming so i am asking for your help. thanks in advance
edit: -----------------
the normal output was this (without the date and if statement)
27/01/2009 A448 B448
22/10/2001 A434 B434
06/09/2007 A825 B825
06/09/2007 A434 B434
06/05/2010 A826 B826
what i would like is if i had a date that is after date.today() say like 27/01/2016 to write to another file and what i keep getting is the script printing this list 30x times or doesn't take to account the if statement.
(sorry, the if was indeed indented the last loop, i went wrong while putting it in here)
You're computing endDate in a loop, once for each date... but not doing anything with it in the loop. So, after the loop is over, you have the very last endDate, and you use only that one to decide which file to write to.
I'm not sure what your logic is supposed to be, but I'm pretty sure you want to put the if statement with the writes inside the inner loop.
If you do that, then if you have, say, 100 pattern matches and 25 dates, you'll end up writing 2500 strings--some to one file, some to the other. Is that what you wanted?
SOLVED
i gave it a little (A LOT) of thought about it and just got all together in one piece. i knew that there too many for loops but now i got it. Thanks anyway to you whom have reached a helping hand to me. I leave the code for anyone having a similar problem.
nowDate = date.today()
for line in sourceFile:
s = re.compile('(\d{2}\/\d{2}\/\d{4})\s(C\d{3}\sS\d{3})')
s1 = re.search(s, line)
if s1:
date = s1.group(1)
date = datetime.strptime(date, '%d/%m/%Y')
date = date.date()
endDate = date + timedelta(days=731)
if endDate <= nowDate:
fileLst.write(s1.group())
fileLst.write('\n')
else:
print ('not ready: ', date.strftime('%d-%m-%Y'))
ntrdLst.write(s1.group(1))
ntrdLst.write('\n')
In Python v2, is there a way to get a date/time stamp and put it into creating a new text file?
IE: When I want to create a new text file and write the contents of my program to it, it will create a new text file with the time/date in it.
Thanks for any help.
import datetime
def timeStamped(fname, fmt='%Y-%m-%d-%H-%M-%S_{fname}'):
return datetime.datetime.now().strftime(fmt).format(fname=fname)
with open(timeStamped('myfile.txt'),'w') as outf:
outf.write('data!')
This will prepend a timestamp to the front of the filename:
from datetime import datetime
# define a timestamp format you like
FORMAT = '%Y%m%d%H%M%S'
path = 'foo.txt'
data = 'data to be written to the file\n'
new_path = '%s_%s' % (datetime.now().strftime(FORMAT), path)
open(new_path, 'w').write(data)
import datetime
f=open("/home/rohitsai/Documents/acs.txt",'a')
f.write ("heloo"+'\t')
f.write(datetime.datetime.now().ctime())
print datetime.datetime.now()
this code will add helo as well as current date on same file. 'a' is for append mode, \t for tab space.
import datetime
open("file", "w").write(datetime.datetime.now().ctime())
open(datetime.datetime.now().ctime(), "w").write("foo")
I like just having the date in my file handles:
from datetime import date
def timeIzNow():
'''
returns current date as a string
'''
now = date.today()
full = "-" + str(now.month) + "-" + str(now.day) + "-" + str(now.year)
return full
fileN = "findGenes"
with open(fileN + timeIzNow() + ".txt", 'w') as f:
#DO STUFF
Your new file name will look like
findGenes-6-5-2013.txt
A lot of these answers are unnecessarily complicated if you just want to spit out a timestamp to a file and don't care about tweaking the format of said stamp
You can literally do it in one line:
f.write(datetime.now().__str__() + "\n")
Here I assume f is a file - so we print the string representation of datetime.now() and add a new line, because write() doesn't do that for you. (You probably already knew that though.)
Note: checked working with Python 3, I didn't test with Python 2, I'm not sure which version of the language spec you are using.
i have a loop where i try to process set of data where one action is to convert ordinary string to datetime. everything works fine except sometimes happend a weird thing ... here is what i know
there are exactly the same parameters entering the function always
those parameters are the same type always
first time i run it, it always get trought
when it gets to second element in the loop in appx 80% throw and value error (time data did not match format)
but after i run it again, everything is ok, and it gets stuck on next emelement ...
because my function is pretty big and there are many things happing i decide to provide you with some saple code whitch i wrote right here, just for clarification:
data = ['January 20 1999', 'March 4 2010', 'June 11 1819']
dformat = '%B %d %Y'
for item in data:
out = datetime.datetime.strptime(item, dformat)
print out
although this clearly works in my program it doesnt ... i have try everything i have came up with but havent succeeded yet therefore i would be glad with any idea you provide thanks
btw: the error i always get looks like this
ValueError: time data did not match format: data=March 4 2010 fmt=%B %d %Y
You probably have a different locale set up. %B is March in locales that use English, but in other locales it will fail.
For example:
>>> import locale
>>> locale.setlocale(locale.LC_ALL, 'sv_SE.utf8')
'sv_SE.utf8'
>>> import datetime
>>>
>>> data = ['January 20 1999', 'March 4 2010', 'June 11 1819']
>>> for item in data:
... print datetime.datetime.strptime(item, '%B %d %Y')
...
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
File "/usr/lib/python2.6/_strptime.py", line 325, in _strptime
(data_string, format))
ValueError: time data 'January 20 1999' does not match format '%B %d %Y'
Here you see that even though the format does match, it claims it doesn't. And that's because the month names doesn't match. Change it to Swedish locale names, and it works again:
>>> import locale
>>> locale.setlocale(locale.LC_ALL, 'sv_SE.utf8')
'sv_SE.utf8'
>>> import datetime
>>>
>>> data = ['Januari 20 1999', 'Mars 4 2010', 'Juni 11 1819']
>>> for item in data:
... print datetime.datetime.strptime(item, '%B %d %Y')
...
1999-01-20 00:00:00
2010-03-04 00:00:00
1819-06-11 00:00:00
(Note that the above locale 'sv_SE.utf8' might not work for you, because you have to have that specific locale installed. To see which ones that are installed on a Unix machine, run this command from the command line:
$ locale -a
C
en_AG
en_AU.utf8
en_BW.utf8
en_CA.utf8
en_DK.utf8
en_GB.utf8
en_HK.utf8
en_IE.utf8
en_IN
en_NG
en_NZ.utf8
en_PH.utf8
en_SG.utf8
en_US.utf8
en_ZA.utf8
en_ZW.utf8
POSIX
sv_FI.utf8
sv_SE.utf8
)
Pretty weird though... In the same run locale usually doesn't change. However, if your program keeps doing this, you might want to call 'setlocale' everytime the code enters into the loop (ugly solution, I know).