I have a log file above
m = '''Jan 23 2010 10:30:08AM - abc def Server-1 connected
Jan 23 2010 11:04:56AM - abc def Server-2 connected
Jan 23 2010 1:18:32PM - abc def Server-2 disconnected from server
Jan 23 2010 4:16:09PM - abc def Aws activity from 10.0.0.45
Jan 23 2010 9:43:44PM - abc def Server-1 disconnected from server
Feb 1 2010 12:40:28AM - abc def Server-1 connected
Feb 1 2010 1:21:52AM - abc def Aws activity from 192.168.123.4
Mar 29 2010 1:13:07PM - def def Server-1 disconnected from server'''
My expected out
Server-1 - connected at 23/01/2016 10:30:08 and disconnected at 23/01/2016 21:43:44
Server-2 - connected at 23/01/2016 11:04:56 and disconnected at 23/01/2016 13:18:32
Aws - connected at 01/02/2016 12:40:28 and disconnected at 29/03/2016 13:13:07
My approach and code convert the file into dictionary and format
Pseudo code
import re
x = re.findall(r'(.*) - abc def (Server-1|Server-2) (connected|disconnected)',m)
y = re.findall(r'Aws activity from (\d.+)', m)
for i in x:
if i[2] == 'connected':
print (f'{i[1]} - connected at {i[0]} ' )
if i[2] == 'disconnected':
print (f'{i[1]} - disconnected at {i[0]} ' )
Related
I worked on calculating churn using the mix of pandas dataframe of git logs and git show command for a particular commit to see where exactly the changes has been done based on loc. However, I could not able to calculate churn based on the days i.e. I mean calculate churn when an engineer rewrites or deletes their own code that is less than 3 weeks old.
This is how I have done for such dataframe for each commit based
git logs dataframe
sha timestamp date author message body age insertion deletion filepath churn merges
1 1 cae635054 Sat Jun 26 14:51:23 2021 -0400 2021-06-26 18:51:23+00:00 Andrew Clark `act`: Resolve to return value of scope function (#21759) When migrating some internal tests I found it annoying that I couldn't -24 days +12:21:32.839997
2 21 cae635054 Sat Jun 26 14:51:23 2021 -0400 2021-06-26 18:51:23+00:00 Andrew Clark `act`: Resolve to return value of scope function (#21759) When migrating some internal tests I found it annoying that I couldn't -24 days +12:21:32.839997 31.0 0.0 packages/react-reconciler/src/__tests__/ReactIsomorphicAct-test.js 31.0
3 22 cae635054 Sat Jun 26 14:51:23 2021 -0400 2021-06-26 18:51:23+00:00 Andrew Clark `act`: Resolve to return value of scope function (#21759) When migrating some internal tests I found it annoying that I couldn't -24 days +12:21:32.839997 1.0 1.0 packages/react-test-renderer/src/ReactTestRenderer.js 0.0
4 23 cae635054 Sat Jun 26 14:51:23 2021 -0400 2021-06-26 18:51:23+00:00 Andrew Clark `act`: Resolve to return value of scope function (#21759) When migrating some internal tests I found it annoying that I couldn't -24 days +12:21:32.839997 24.0 14.0 packages/react/src/ReactAct.js 10.0
5 25 e2453e200 Fri Jun 25 15:39:46 2021 -0400 2021-06-25 19:39:46+00:00 Andrew Clark act: Add test for bypassing queueMicrotask (#21743) Test for fix added in #21740 -25 days +13:09:55.839997 50.0 0.0 packages/react-reconciler/src/__tests__/ReactIsomorphicAct-test.js 50.0
6 27 73ffce1b6 Thu Jun 24 22:42:44 2021 -0400 2021-06-25 02:42:44+00:00 Brian Vaughn DevTools: Update tests to fix warnings/errors (#21748) Some new ones had slipped in (e.g. deprecated ReactDOM.render message from 18) -26 days +20:12:53.839997 4.0 5.0 packages/react-devtools-shared/src/__tests__/FastRefreshDevToolsIntegration-test.js -1.0
7 28 73ffce1b6 Thu Jun 24 22:42:44 2021 -0400 2021-06-25 02:42:44+00:00 Brian Vaughn DevTools: Update tests to fix warnings/errors (#21748) Some new ones had slipped in (e.g. deprecated ReactDOM.render message from 18) -26 days +20:12:53.839997 4.0 4.0 packages/react-devtools-shared/src/__tests__/componentStacks-test.js 0.0
8 29 73ffce1b6 Thu Jun 24 22:42:44 2021 -0400 2021-06-25 02:42:44+00:00 Brian Vaughn DevTools: Update tests to fix warnings/errors (#21748) Some new ones had slipped in (e.g. deprecated ReactDOM.render message from 18) -26 days +20:12:53.839997 12.0 12.0 packages/react-devtools-shared/src/__tests__/console-test.js 0.0
9 30 73ffce1b6 Thu Jun 24 22:42:44 2021 -0400 2021-06-25 02:42:44+00:00 Brian Vaughn DevTools: Update tests to fix warnings/errors (#21748) Some new ones had slipped in (e.g. deprecated ReactDOM.render message from 18) -26 days +20:12:53.839997 7.0 6.0 packages/react-devtools-shared/src/__tests__/editing-test.js 1.0
10 31 73ffce1b6 Thu Jun 24 22:42:44 2021 -0400 2021-06-25 02:42:44+00:00 Brian Vaughn DevTools: Update tests to fix warnings/errors (#21748) Some new ones had slipped in (e.g. deprecated ReactDOM.render message from 18) -26 days +20:12:53.839997 47.0 42.0 packages/react-devtools-shared/src/__tests__/inspectedElement-test.js 5.0
11 32 73ffce1b6 Thu Jun 24 22:42:44 2021 -0400 2021-06-25 02:42:44+00:00 Brian Vaughn DevTools: Update tests to fix warnings/errors (#21748) Some new ones had slipped in (e.g. deprecated ReactDOM.render message from 18) -26 days +20:12:53.839997 7.0 6.0 packages/react-devtools-shared/src/__tests__/ownersListContext-test.js 1.0
12 33 73ffce1b6 Thu Jun 24 22:42:44 2021 -0400 2021-06-25 02:42:44+00:00 Brian Vaughn DevTools: Update tests to fix warnings/errors (#21748) Some new ones had slipped in (e.g. deprecated ReactDOM.render message from 18) -26 days +20:12:53.839997 22.0 21.0 packages/react-devtools-shared/src/__tests__/profilerContext-test.js 1.0
churn calculation
commits = df["sha"].unique().tolist()
for commit in commits:
contribution, churn = await self.calculate_churn(commit)
async def calculate_churn(self, stream):
PREVIOUS_BASE_DIR = os.path.abspath("")
try:
GIT_DIR = os.path.join(PREVIOUS_BASE_DIR, "app/git/react.git")
os.chdir(GIT_DIR)
except FileNotFoundError as e:
raise ValueError(e)
cmd = f"git show --format= --unified=0 --no-prefix {stream}"
cmds = [f"{cmd}"]
results = get_proc_out(cmds)
[files, contribution, churn] = get_loc(results)
# need to circle back to previous path
os.chdir(PREVIOUS_BASE_DIR)
return contribution, churn
def is_new_file(result, file):
# search for destination file (+++ ) and update file variable
if result.startswith("+++"):
return result[result.rfind(" ") + 1 :]
else:
return file
def is_loc_change(result, loc_changes):
# search for loc changes (## ) and update loc_changes variable
# ## -1,5 +1,4 ##
# ## -l,s +l,s ##
if result.startswith("##"):
# loc_change = result[2+1: ] -> -1,5 +1,4 ##
loc_change = result[result.find(" ") + 1 :]
# loc_change = loc_change[:9] -> -1,5 +1,4
loc_change = loc_change[: loc_change.find(" ##")]
return loc_change
else:
return loc_changes
def get_loc_change(loc_changes):
# removals
# -1,5 +1,4 = -1,5
left = loc_changes[: loc_changes.find(" ")]
left_dec = 0
# 2
if left.find(",") > 0:
# 2
comma = left.find(",")
# 5
left_dec = int(left[comma + 1 :])
# 1
left = int(left[1:comma])
else:
left = int(left[1:])
left_dec = 1
# additions
# +1,4
right = loc_changes[loc_changes.find(" ") + 1 :]
right_dec = 0
if right.find(",") > 0:
comma = right.find(",")
right_dec = int(right[comma + 1 :])
right = int(right[1:comma])
else:
right = int(right[1:])
right_dec = 1
if left == right:
return {left: (right_dec - left_dec)}
else:
return {left: left_dec, right: right_dec}
def get_loc(results):
files = {}
contribution = 0
churn = 0
file = ""
loc_changes = ""
for result in results:
new_file = is_new_file(result, file)
if file != new_file:
file = new_file
if file not in files:
files[file] = {}
else:
new_loc_changes = is_loc_change(
result, loc_changes
) # returns either empmty or -6 +6 or -13, 0 +14, 2 format
if loc_changes != new_loc_changes:
loc_changes = new_loc_changes
locc = get_loc_change(loc_changes) # {2: 0} or {8: 0, 9: 1}
for loc in locc:
# files[file] = {2: 0, 8: 0, 9: 1}
# print("loc", loc, files[file], locc[loc])
if loc in files[file]:
# change of lines triggered
files[file][loc] += locc[loc]
churn += abs(locc[loc])
else:
files[file][loc] = locc[loc]
contribution += abs(locc[loc])
else:
continue
return [files, contribution, churn]
How can I utilize this same code but check churn only if there is changes in code that is only 3 weeks old?
The only practical way to do this is to iterate through the DataDrame, and because that sucks with pandas, it almost always means you have the wrong data structure. If you're not doing numerical analysis, and it looks like you aren't, then just keep a simple list of dicts. Pandas has its shining points, but it's not a universal database.
Here's the rough code you'd need, although I'm glossing over details:
# Go through the df row by row.
lastdate = {}
for index,row in df.iterrows():
if row['filepath'] in lastdate:
if lastdate[row['filepath']] - row['date'] < timedelta(days=21):
print( "Last change to", row['filepath'], "was within three weeks" )
lastdate[row['filepath']] = row['date']
I am trying to create java jar with the files in AWS S3 by Lambda function using Python
I have tried in both ways
import os
def lambda_handler(event, context):
os.system('jar cf abc.jar, file1.txt, file2.txt')
return "Success"
and
import subprocess
def lambda_handler(event, context):
subprocess.call('jar cf abc.jar, file1.txt, file2.txt')
return "Success"
Remove commas
import subprocess
subprocess.call('jar cf abc.jar *.py')
# import os
# os.system('jar cf abc.jar *.py')
print("Done")
Output :
Done
Process finished with exit code 0
Verifying :
import subprocess
subprocess.call('jar tvf abc.jar')
# import os
# os.system('jar tvf abc.jar')
Output :
0 Wed Feb 28 18:22:32 IST 2018 META-INF/
68 Wed Feb 28 18:22:32 IST 2018 META-INF/MANIFEST.MF
196 Wed Jan 24 20:27:32 IST 2018 dirTut.py
438 Tue Jan 09 20:44:00 IST 2018 epochtodatetime.py
1540 Sun Jan 07 15:30:42 IST 2018 firstServer.py
164 Sun Jan 14 19:34:20 IST 2018 flaskTut.py
295 Tue Jan 09 17:59:20 IST 2018 funcOverriding.py
390 Tue Jan 09 16:55:24 IST 2018 underscore.py
I have a text file that is output from a command that I ran with Netmiko to retrieve data from a Cisco WLC of things that are causing interference on our WiFi network. I stripped out just what I needed from the original 600k lines of code down to a couple thousand lines like this:
AP Name.......................................... 010-HIGH-FL4-AP04
Microwave Oven 11 10 -59 Mon Dec 18 08:21:23 2017
WiMax Mobile 11 0 -84 Fri Dec 15 17:09:45 2017
WiMax Fixed 11 0 -68 Tue Dec 12 09:29:30 2017
AP Name.......................................... 010-2nd-AP04
Microwave Oven 11 10 -61 Sat Dec 16 11:20:36 2017
WiMax Fixed 11 0 -78 Mon Dec 11 12:33:10 2017
AP Name.......................................... 139-FL1-AP03
Microwave Oven 6 18 -51 Fri Dec 15 12:26:56 2017
AP Name.......................................... 010-HIGH-FL3-AP04
Microwave Oven 11 10 -55 Mon Dec 18 07:51:23 2017
WiMax Mobile 11 0 -83 Wed Dec 13 16:16:26 2017
The goal is to end up with a csv file that strips out the 'AP Name ...' and puts what left on the same line as the rest of the information in the next line. The problem is some have two lines below the AP name and some have 1 or none. I have been at it for 8 hours and cannot find the best way to make this happen.
This is the latest version of code that I was trying to use, any suggestions for making this work? I just want something I can load up in excel and create a report with:
with open(outfile_name, 'w') as out_file:
with open('wlc-interference_raw.txt', 'r')as in_file:
#Variables
_ap_name = ''
_temp = ''
_flag = False
for i in in_file:
if 'AP Name' in i:
#write whatever was put in the temp file to disk because new ap now
#add another temp variable in case an ap has more than 1 interferer and check if new AP name
out_file.write(_temp)
out_file.write('\n')
#print(_temp)
_ap_name = i.lstrip('AP Name.......................................... ')
_ap_name = _ap_name.rstrip('\n')
_temp = _ap_name
#print(_temp)
elif '----' in i:
pass
elif 'Class Type' in i:
pass
else:
line_split = i.split()
for x in line_split:
_temp += ','
_temp += x
_temp += '\n'
I think your best option is to read all lines of the file, then split into sections starting with AP Name. Then you can work on parsing each section.
Example
s = """AP Name.......................................... 010-HIGH-FL4-AP04
Microwave Oven 11 10 -59 Mon Dec 18 08:21:23 2017
WiMax Mobile 11 0 -84 Fri Dec 15 17:09:45 2017
WiMax Fixed 11 0 -68 Tue Dec 12 09:29:30 2017
AP Name.......................................... 010-2nd-AP04
Microwave Oven 11 10 -61 Sat Dec 16 11:20:36 2017
WiMax Fixed 11 0 -78 Mon Dec 11 12:33:10 2017
AP Name.......................................... 139-FL1-AP03
Microwave Oven 6 18 -51 Fri Dec 15 12:26:56 2017
AP Name.......................................... 010-HIGH-FL3-AP04
Microwave Oven 11 10 -55 Mon Dec 18 07:51:23 2017
WiMax Mobile 11 0 -83 Wed Dec 13 16:16:26 2017"""
import re
class AP:
"""
A class holding each section of the parsed file
"""
def __init__(self):
self.header = ""
self.content = []
sections = []
section = None
for line in s.split('\n'): # Or 'for line in file:'
# Starting new section
if line.startswith('AP Name'):
# If previously had a section, add to list
if section is not None:
sections.append(section)
section = AP()
section.header = line
else:
if section is not None:
section.content.append(line)
sections.append(section) # Add last section outside of loop
for section in sections:
ap_name = section.header.lstrip("AP Name.") # lstrip takes all the characters given, not a literal string
for line in section.content:
print(ap_name + ",", end="")
# You can extract the date separately, if needed
# Splitting on more than one space using a regex
line = ",".join(re.split(r'\s\s+', line))
print(line.rstrip(',')) # Remove trailing comma from imperfect split
Output
010-HIGH-FL4-AP04,Microwave Oven,11,10,-59,Mon Dec 18 08:21:23 2017
010-HIGH-FL4-AP04,WiMax Mobile,11,0,-84,Fri Dec 15 17:09:45 2017
010-HIGH-FL4-AP04,WiMax Fixed,11,0,-68,Tue Dec 12 09:29:30 2017
010-2nd-AP04,Microwave Oven,11,10,-61,Sat Dec 16 11:20:36 2017
010-2nd-AP04,WiMax Fixed,11,0,-78,Mon Dec 11 12:33:10 2017
139-FL1-AP03,Microwave Oven,6,18,-51,Fri Dec 15 12:26:56 2017
010-HIGH-FL3-AP04,Microwave Oven,11,10,-55,Mon Dec 18 07:51:23 2017
010-HIGH-FL3-AP04,WiMax Mobile,11,0,-83,Wed Dec 13 16:16:26 2017
Tip:
You don't need Python to write the CSV, you can output to a file using the command line
python script.py > output.csv
I have class A which inheritance the class B
i need to send main_b to thread and continue with the program (main_a)
import threading
import time
class B(object):
def main_b(self):
i = 0
while i < 5:
print "main_b: %s" %time.ctime(time.time())
time.sleep(1)
i += 1
class A(B):
def main_a(self):
b = threading.Thread(target=self.main_b())
b.start()
i = 0
while i < 5:
print "main_a: %s" %time.ctime(time.time())
time.sleep(1)
i += 1
b.join()
aa = A()
aa.main_a()
Expected result main_b and main_a print at same time
Actual:
main_b: Thu Sep 22 09:57:44 2016
main_b: Thu Sep 22 09:57:45 2016
main_b: Thu Sep 22 09:57:46 2016
main_a: Thu Sep 22 09:57:47 2016
main_a: Thu Sep 22 09:57:48 2016
main_a: Thu Sep 22 09:57:49 2016
Just pass method as a target for thread:
b = threading.Thread(target=self.main_b)
i have a list which goes as follows:
-------------------------------------------------------------------------------------------
www.mydomain.de UP Thu May 8 09:10:57 2014
HTTPS OK Thu May 8 09:10:08 2014
HTTPS-Cert OK Thu May 8 09:10:55 2014
-------------------------------------------------------------------------------------------
www.someotherdomain.de UP Thu May 8 09:09:17 2014
HTTPS OK Thu May 8 09:09:30 2014
HTTPS-Cert OK Thu May 8 09:11:10 2014
-------------------------------------------------------------------------------------------
www.somedifferentdomain.at UP Thu May 8 09:08:47 2014
HTTPS OK Thu May 8 09:10:26 2014
HTTPS-Cert OK Thu May 8 09:11:13 2014
-------------------------------------------------------------------------------------------
www.foobladomain.de UP Thu May 8 09:09:17 2014
HTTPS OK Thu May 8 09:09:30 2014
HTTPS-Cert OK Thu May 8 09:11:08 2014
-------------------------------------------------------------------------------------------
www.snafudomain.at UP Thu May 8 09:09:17 2014
HTTP OK Thu May 8 09:09:42 2014
HTTPS OK Thu May 8 09:10:10 2014
HTTPS-Cert OK Thu May 8 09:10:09 2014
-------------------------------------------------------------------------------------------
www.lolnotanotherdomain.de UP Thu May 8 09:06:57 2014
HTTP OK Thu May 8 09:11:10 2014
HTTPS OK Thu May 8 09:11:16 2014
HTTPS-Cert OK Thu May 8 09:11:10 2014
and i have a function which takes the hostname as parameter and prints it out:
please enter hostname to search for: www.snafudomain.at
www.snafudomain.at UP Thu May 8 09:09:17 2014
but what i want to archive is that the following lines after the hostname are printed out until the delimiter line "-----" the function i right now looks like this:
def getChecks(self,hostname):
re0 = "%s" % hostname
mylist = open('myhostlist', 'r')
for i in mylist:
if re.findall("^%s" % re0, str(i)):
print i
else:
continue
is there some easy way to do this? If something is unclear please comment. Thanks in advance
edit
to clarify the output should look like this:
www.mydomain.de UP Thu May 8 09:10:57 2014
HTTPS OK Thu May 8 09:10:08 2014
HTTPS-Cert OK Thu May 8 09:10:55 2014
-------------------------------------------------------------------------------------
just want to print out the lines from the searched domain name till the line with only minuses.
How about not using regex at all?
def get_checks(self, hostname):
record = False
with open('myhostlist', 'r') as file_h:
for line in file_h:
if line.startswith(hostname):
record = True
print(line)
elif line.startswith("---"):
record = False
print(line)
elif record:
print(line)
import re
def get_checks(hostname):
pattern = re.compile(r"{}.*?(?=---)".format(re.escape(hostname)), re.S)
with open("Input.txt") as in_file:
return re.search(pattern, in_file.read())
print get_checks("www.snafudomain.at").group()
This will returns all the lines starting with www.snafudomain.at till it finds ---. The pattern generated will be like this
www\.snafudomain\.at.*?(?=---)
Online Demo
We use re.escape because your hostname has . in it. Since . has special meaning in the Regular Expressions, we just want the RegEx engine to treat . as literal dot.