We Keep Coding

Verify data on a file

I'm trying to make my life easier on my work, and writing down errors and solutions for that same errors. The program itself works fine when it's about adding new errors, but then I added a function to verify if the error exists in the file and then do something to it (not added yet).
The function doesn't work and I don't know why. I tried to debug it, but still not able to find the error, maybe a conceptual error?
Anyway, here's my entire code.
import sys
import os
err = {}
PATH = 'C:/users/userdefault/desktop/errordb.txt'
#def open_file(): #Not yet used
#file_read = open(PATH, 'r')
#return file_read
def verify_error(error_number, loglist): #Verify if error exists in file
for error in loglist:
if error_number in loglist:
return True
def dict_error(error_number, solution): #Puts input errors in dict
err = {error_number: solution}
return err
def verify_file(): #Verify if file exists. Return True if it does
archive = os.path.isfile(PATH)
return archive
def new_error():
file = open(PATH, 'r') #Opens file in read mode
loglist = file.readlines()
file.close()
found = False
error_number = input("Error number: ")
if verify_error(error_number, loglist) == True:
found = True
# Add new solution, or another solution.
pass
solution = str(input("Solution: "))
file = open(PATH, 'a')
error = dict_error(error_number, solution)
#Writes dict on file
file.write(str(error))
file.write("\n")
file.close()
def main():
verify = verify_file() #Verify if file exists
if verify == True:
new = str.lower(input("New job Y/N: "))
if new == 'n':
sys.exit()
while new == 'y':
new_error()
new = str.lower(input("New job Y/N: "))
else:
sys.exit()
else:
file = open(PATH, "x")
file.close()
main()
main()
To clarify, the program executes fine, it don't return an error code. It just won't execute the way I'm intended, I mean, it supposed to verify if certain error number already exists.
Thanks in advance :)

The issue I believe you're having is the fact that you're not actually creating a dictionary object in the file and modifying it but instead creating additional dictionaries every time an error is added then reading them back as a list of strings by using the .readlines() method.
An easier way of doing it would be to create a dictionary if one doesn't exist and append errors to it. I've made a few modifications to your code which should help.
import sys
import os
import json # Import in json and use is as the format to store out data in
err = {}
PATH = 'C:/users/userdefault/desktop/errordb.txt'
# You can achieve this by using a context manager
#def open_file(): #Not yet used
#file_read = open(PATH, 'r')
#return file_read
def verify_error(error_number, loglist): #Verify if error exists in file
# Notice how we're looping over keys of your dictionary to check if
# an error already exists.
# To access values use loglist[k]
for k in loglist.keys():
if error_number == k:
return True
return False
def dict_error(loglist, error_number, solution): #Puts input errors in dict
# Instead of returning a new dictionary, return the existing one
# with the new error appended to it
loglist[error_number] = solution
return loglist
def verify_file(): #Verify if file exists. Return True if it does
archive = os.path.isfile(PATH)
return archive
def new_error():
# Let's move all the variables to the top, makes it easier to read the function
# Changes made:
# 1. Changed the way we open and read files, now using a context manager (aka with open() as f:
# 2. Added a json parser to store in and read from file in a json format. If data doesn't exist (new file?) create a new dictionary object instead
# 3. Added an exception to signify that an error has been found in the database (this can be removed to add additional logic if you'd like to do more stuff to the error, etc)
# 4. Changed the way we write to file, instead of appending a new line we now override the contents with a new updated dictionary that has been serialized into a json format
found = False
loglist = None
# Open file as read-only using a context manager, now we don't have to worry about closing it manually
with open(PATH, 'r') as f:
# Lets read the file and run it through a json parser to get a python dictionary
try:
loglist = json.loads(f.read())
except json.decoder.JSONDecodeError:
loglist = {}
error_number = input("Error number: ")
if verify_error(error_number, loglist) is True:
found = True
raise Exception('Error exists in the database') # Raise exception if you want to stop loop execution
# Add new solution, or another solution.
solution = str(input("Solution: "))
# This time open in write only and replace the dictionary
with open(PATH, 'w') as f:
loglist = dict_error(loglist, error_number, solution)
# Writes dict on file in json format
f.write(json.dumps(loglist))
def main():
verify = verify_file() #Verify if file exists
if verify == True:
new = str.lower(input("New job Y/N: "))
if new == 'n':
sys.exit()
while new == 'y':
new_error()
new = str.lower(input("New job Y/N: "))
else:
sys.exit()
else:
with open(PATH, "x") as f:
pass
main()
main()
Note that you will have to create a new errordb file for this snippet to work.
Hope this has helped somehow. If you have any further questions hit me up in the comments!
References:
Reading and Writing files in Python
JSON encoder and decoder in Python

I think that there may be a couple of problems with your code, but the first thing that I noticed was that you are saving Error Numbers and Solutions as a dictionary in errorsdb.txt and when you read them back in you are reading them back in as a list of strings:
The line:
loglist = file.readlines()
in new_error returns a list of strings. This means that verify_error will always return False.
So you have a couple of choices:
You could modify verify_error to the following:
def verify_error(error_number, loglist): #Verify if error exists in file
for error in loglist:
if error_number in error:
return True
Although, I think that a better solution would be to load errorsdb.txt as a JSON file and then you'll have a dictionary. That would look something like:
import json
errordb = {}
with open(PATH) as handle:
errordb = json.load(handle)
So here are the full set of changes I would make:
import json
def verify_error(error_number, loglist): #Verify if error exists in file
for error in loglist:
if error_number in error:
return True
def new_error():
errordb = list()
exitsting = list()
with open(PATH) as handle:
existing = json.load(handle)
errordb += existing
error_number = input("Error number: ")
if verify_error(error_number, errordb) == True:
# Add new solution, or another solution.
print("I might do something here.")
else:
solution = str(input("Solution: "))
errordb.append({error_number, solution})
#Writes dict on file
with open(PATH, "w") as handle:
json.dump(errordb, handle)

I need a shortcut

So im just trying to make a simple script that can filter emails with different domains its working great but i need a shortcut, cause i dont wana write if and elif statements many time , Can anyone tell my how to write my script with function so that will become shorter and easier.. thanks in advance ,Script is below:
f_location = 'C:/Users/Jack The Reaper/Desktop/mix.txt'
text = open(f_location)
good = open('C:/Users/Jack The Reaper/Desktop/good.txt','w')
for line in text:
if '#yahoo' in line:
yahoo = None
elif '#gmail' in line:
gmail = None
elif '#yahoo' in line:
yahoo = None
elif '#live' in line:
live = None
elif '#outlook' in line:
outlook = None
elif '#hotmail' in line:
hotmail = None
elif '#aol' in line:
aol = None
else:
if ' ' in line:
good.write(line.strip(' '))
elif '' in line:
good.write(line.strip(''))
else:
good.write(line)
text.close()
good.close()

I would suggest you to use dict for this instead of having separate variables for all the cases.
my_dict = {}
...
if '#yahoo' in line:
my_dict['yahoo'] = None
But if you want to do the way you described in the question, you can do as done below,
email_domains = ['#yahoo', '#gmail', '#live', '#outlook', '#hotmail', '#aol']
for e in email_domains:
if e in line:
locals()[e[1:]] = None
#if you use dict, use the below line
#my_dict[e[1:]] = None
locals() returns a dictionary of the current namespace. The keys in this dict are the variable names and value is the value of the variable.
So locals()['gmail'] = None creates a local variable named gmail(if it doesn't exist) and assigns it None.

As you stated the problem and provided the sample file :
So i have two solution : One line solution and other is detailed solution.
First let's define regex pattern and import re module
import re
pattern=r'.+#(?!gmail|yahoo|aol|hotmail|live|outlook).+'
Now detailed version code:
emails=[]
with open('emails.txt','r') as f:
for line in f:
match=re.finditer(pattern,line)
for find in match:
emails.append(find.group())
with open('result.txt','w') as f:
f.write('\n'.join(emails))
output in result.txt file :
nic-os9#gmx.de
angelique.charuel#sfr.fr
nannik#interia.pl
l.andrioli#freenet.de
kamil_sieminski8#o2.pl
hugo.lebrun.basket#orange.fr
One line solution if you want too short:
with open('results.txt','w') as file:
file.write('\n'.join([find.group() for line in open('emails.txt','r') for find in re.finditer(pattern,line)]))
output:
nic-os9#gmx.de
angelique.charuel#sfr.fr
nannik#interia.pl
l.andrioli#freenet.de
kamil_sieminski8#o2.pl
hugo.lebrun.basket#orange.fr
P.S : with one line solution file will not close automatically but python clear that stuff its not a big issue (but not always) but still if you want you can use.

program to read a file contents

I have a snort log file named "logs" and want to extract IP addresses from it and store them to another file named "blacklist". it can extract unique IP Addresses but if I run the program again, it adds the previous IPs as well. I want the program to first check whether IP is already in blacklist file? if so, just ignore it otherwise add unique IPs from logs file to blacklist. code:
#!/usr/bin/python
import re
mylist1 = []
mylist2 = []
mylist3 = []
mylist4 = []
logfile = open('/var/log/snort/logs', 'r')
blklist = open('blacklist', 'ab+')
for line in open ('blacklist', 'r').readlines():
mylist4.append(line)
for l in logfile.readlines():
l = l.rstrip()
ip = re.findall(r'[0-9]+(?:\.[0-9]+){3}',l)
if ip is not None and ip not in mylist1:
mylist1.append(ip)
for ip in mylist1:
addr = ",".join(ip)
if ',' in addr:
a = addr.split(',')
for ip in a:
addr = "".join(ip)
if addr is not '':
mylist2.append(addr)
else:
mylist3.append(addr)
for x in blklist:
mylist2.append(x.strip())
for x in mylist2:
if x not in mylist3 and x not in mylist4:
blklist.write(x+'\n')
mylist3.append(x)
Logs file is:
12/16-10:34:27.070967 [**] [1:10000001:1] snort alert [1:0000001] [**][classification ID: 0] [Priority ID: 0] {ICMP} 192.168.40.19 -> 192.168.50.29
12/16-10:34:27.070967 [**] [1:10000001:1] snort alert [1:0000001] [**][classification ID: 0] [Priority ID: 0] {ICMP} 192.168.50.29 -> 192.168.30.20
Output of blacklist file after first program run:
192.168.30.20
192.168.50.29
192.168.40.19
Output of blacklist file after second program run:
192.168.30.20
192.168.50.29
192.168.40.19
192.168.30.20
192.168.50.29
192.168.40.19
any help please?

You can read everything in from your blacklist file and log into lists. Join those list and then ouput a set back to the blacklist file (sets are unique values) since the read empties the file your will have a unique list of all new and old IPs. If the order matters (doubt it does) then a set will cause issues. Let me know and I can revamp the below.
if __name__ == '__main__':
import re
blacklist = list(open("blacklist", 'r').read().split('\n'))
logfile = list(open("/var/log/snort/logs", 'r').read().split('\n'))
newentry = []
for entry in logfile:
ips = re.findall( r'[0-9]+(?:\.[0-9]+){3}', entry)
for ip in ips:
newentry.append(ip)
newblacklist = blacklist + newentry
with open("blacklist", 'w+') as f:
f.write('\n' .join(set(newblacklist)))
f.close()

You could utilize the Python container type set which stores only unique elements. The procedure below should work for you:
create a 'current' blacklist set
read the blacklist file IP's into the current set
create a 'delta' blacklist set
for each IP address in the log file
if not already in current blacklist
add the IP into the delta set
append (by writing) the delta set into the black list file

How can I parse a formatted file into variables using Python?

I have a pre-formatted text file with some variables in it, like this:
header one
name = "this is my name"
last_name = "this is my last name"
addr = "somewhere"
addr_no = 35
header
header two
first_var = 1.002E-3
second_var = -2.002E-8
header
As you can see, each score starts with the string header followed by the name of the scope (one, two, etc.).
I can't figure out how to programmatically parse those options using Python so that they would be accesible to my script in this manner:
one.name = "this is my name"
one.last_name = "this is my last name"
two.first_var = 1.002E-3
Can anyone point me to a tutorial or a library or to a specific part of the docs that would help me achieve my goal?

I'd parse that with a generator, yielding sections as you parse the file. ast.literal_eval() takes care of interpreting the value as a Python literal:
import ast
def load_sections(filename):
with open(filename, 'r') as infile:
for line in infile:
if not line.startswith('header'):
continue # skip to the next line until we find a header
sectionname = line.split(None, 1)[-1].strip()
section = {}
for line in infile:
if line.startswith('header'):
break # end of section
line = line.strip()
key, value = line.split(' = ', 1)
section[key] = ast.literal_eval(value)
yield sectionname, section
Loop over the above function to receive (name, section_dict) tuples:
for name, section in load_sections(somefilename):
print name, section
For your sample input data, that results in:
>>> for name, section in load_sections('/tmp/example'):
... print name, section
...
one {'last_name': 'this is my last name', 'name': 'this is my name', 'addr_no': 35, 'addr': 'somewhere'}
two {'first_var': 0.001002, 'second_var': -2.002e-08}

Martijn Pieters is correct in his answer given your preformatted file, but if you can format the file in a different way in the first place, you will avoid a lot of potential bugs. If I were you, I would look into getting the file formatted as JSON (or XML), because then you would be able to use python's json (or XML) libraries to do the work for you. http://docs.python.org/2/library/json.html . Unless you're working with really bad legacy code or a system that you don't have access to, you should be able to go into the code that spits out the file in the first place and make it give you a better file.

def get_section(f):
section=[]
for line in f:
section += [ line.strip("\n ") ]
if section[-1] == 'header': break
return section
sections = dict()
with open('input') as f:
while True:
section = get_section(f)
if not section: break
section_dict = dict()
section_dict['sname'] = section[0].split()[1]
for param in section[1:-2]:
k,v = [ x.strip() for x in param.split('=')]
section_dict[k] = v
sections[section_dict['sname']] = section_dict
print sections['one']['name']
You can also access these sections as attributes:
class Section:
def __init__(self, d):
self.__dict__ = d
one = Section(sections['one'])
print one.name

Test if string exists in file with Python

I'd like to add only servers currently doesn't exist at file
My current code :
f = open(filename,'a')
for server in cmo.getServers() :
print >>f, server.getListenAddress()
Thanks in advance

try this:
data = set( [i.strip() for i in open( filename, 'r' ).readlines()] )
for server in cmo.getServers() :
data.add( server.getListenAddress() )
open( filename, 'w' ).write('\n'.join(data))

Build a list of servers already present in the file:
present = [l.strip() for l in open(filename)]
(assuming the file format is just one server per line, no other symbols).
Then check if an address is in the list:
for server in cmo.getServers():
address = server.getListenAddress()
if address not in present:
print >>f, address
This assumes that the addresses you get from getServers() will not repeat.
If that's also possible, then build a set of them first:
new = set(server.getListenAddress() for server in cmo.getServers())
for address in new:
if address not in present:
print >>f, address