We Keep Coding

Finding a string from a list made from a file in Python 3

I am trying to find a name that is in two separate lists I created and have a function check to see if it is there. I know it is checking the list and I have printed out the list to make sure it is stored correctly but it keeps giving me my error statement that the name is not found in the list. Here is the code I have for it.
def readBoyFiles():
boyfile = 'BoyNames.txt'
boyList = []
with open(boyfile, 'r') as lis:
for line in lis:
boyList.append(line)
return boyList
def readGirlFiles():
girlfile = 'GirlNames.txt'
girlList = []
with open(girlfile, 'r') as names:
for line in names:
girlList.append(line)
return girlList
def nameInput():
name = input('Please enter the name you would like to search: ')
list1 = readBoyFiles()
list2 = readGirlFiles()
findName(name, list1)
findName(name, list2)
def findName(name, list):
if name in list:
print('This name is among the most popular!')
else:
print('This name is not among the most popular.')
nameInput()
When I throw in a print statement like print(list1), it gives me the names in this format, ['Jacob\n', ....] and when I test it it prints out my else statement regardless of what I type in for the input. I have also tried checking it with the index function and it tells me that 'Jacob' is not in list if I try that. I feel like I have to be overlooking something because I've written similar code that works properly and this is almost a mirror image of it except with different data types.

Remember to strip your strings! It removes leading and trailing whitespace. Technically, "Jacob" isn't in the list because "Jacob\n" is.
def readBoyFiles():
boyfile = 'BoyNames.txt'
boyList = []
with open(boyfile, 'r') as lis:
for line in lis:
boyList.append(line.strip())
return boyList
def readGirlFiles():
girlfile = 'GirlNames.txt'
girlList = []
with open(girlfile, 'r') as names:
for line in names:
girlList.append(line.strip())
return girlList

A more pythonic version of your code
def load_list(file_name):
with open(file_name, 'r') as f:
return [name.strip() for name in f.readlines()]
def get_lists_and_user_input():
name = raw_input('Please enter the name you would like to search: ')
boys_list = load_list('popular_boys.txt')
girls_list = load_list('popular_girls.txt')
return boys_list, girls_list, name
def check_name(name, lst, _type):
if name in lst:
print('The name {} is a popular {} name'.format(name, _type))
else:
print('The name {} is NOT a popular {} name'.format(name, _type))
boys, girls, _name = get_lists_and_user_input()
check_name(_name, boys, 'boys')
check_name(_name, girls, 'girls')

Python, class attribute error

I have a set of code that looks like this:
class DataFilter:
def __init__(self, csvData):
# converts csv string data to float lists, if possible
data = []
for line in csvData:
try:
line = line.split(',')
except:
print(line)
return
for i in range( len(line) ):
try:
line[i] = float(line[i])
except ValueError:
pass
data.append(line)
self.data = data
def find_depth_index(self, depth):
for index, line in enumerate( self.data ):
if line[1] > depth:
return index
def remove_above_depth(self, depth):
index = self.find_depth_index( depth )
return self.data[ index: ]
def remove_beyond_depth(self, depth):
index = self.find_depth_index(depth)
return self.data[ :index ]
data = DataFilter(data).remove_above_depth(SURF_CASING_DEPTH)
print('-----------------------')
data = DataFilter(data).remove_beyond_depth(VERTICAL_SEC_DEPTH)
Then it give me an error:
File "C:/Users/Eric Soobin Kim/PycharmProjects/untitled/RAPID_use_filtered_data.py", line 35, in remove_beyond_depth
def remove_beyond_depth(self, depth):
File "C:/Users/Eric Soobin Kim/PycharmProjects/untitled/RAPID_use_filtered_data.py", line 26, in find_depth_index
def find_depth_index(self, depth):
AttributeError: 'DataFilter' object has no attribute 'data'
The thing that I don't understand is that, it ran without a problem for the line:
data = DataFilter(data).remove_above_depth(SURF_CASING_DEPTH)
but its not working for,
data = DataFilter(data).remove_beyond_depth(VERTICAL_SEC_DEPTH)
I think my first filtering somehow alters elements in __ini__(), but i don't know what's going on. Why is this happening, and how can i fix it?

You've reassigned data to be equal to something other than what you want.
data = DataFilter(data).remove_above_depth(SURF_CASING_DEPTH)
This means that now you've lost the pointer to data that you once had. Might I suggest making a copy like
new_data = DataFilter(data).remove_above_depth(SURF_CASING_DEPTH)
new_data2 = DataFilter(data).remove_beyond_depth(VERTICAL_SEC_DEPTH)
This way you still have the reference to the old data variable

Object has no attribute data because you didnt give it the attribute.
atributes are defined by making writing:
self.object = []
instead of:
object = []

Class duplicates input objects

I have two python class objects that I use to extract data from Oracle to ArcMap. The various activities in the process cause me to start with a list of 'column' objects and build a pyTable object. The pyTable object has a list of insert Fields by name. During __init__ I use the getSelect() function to populate the list of insert Fields.
I have added a bunch of statements to make sure that each time I call pyTable I get a newly created object, but I'm still seeing a strange result. The first time I use the class, all is fine. The second time I issue the same statement, the colList is new, but the field list is duplicated. My apologies for not cleaning out the extraneous code sections.
Where am I messing up my object references?
Here are the execution results. myList has 8 column objects.
>>> arcTable = pyTable(myList)
>>> len(arcTable.getTuple())
8
>>> arcTable = pyTable(myList)
>>> len(arcTable.getTuple())
16
>>> arcTable = pyTable(myList)
>>> len(arcTable.getTuple())
8
>>> arcTable = pyTable(myList)
>>> len(arcTable.getTuple())
8
>>> newTable = pyTable(myList)
>>> len(newTable.getTuple())
8
>>> thirdTable = pyTable(myList)
>>> len(thirdTable.getTuple())
16
>>> thirdTable = pyTable(myList)
>>> len(thirdTable.getTuple())
24
>>> thirdTable = pyTable(myList)
>>> len(thirdTable.getTuple())
8
>>>
Here are the two classes:
import arcpy, cx_Oracle
class column:
# Add to the arcType and cxType functions to support more Oracle data types.
# BLOB and CLOB fields will need additional support in Read and Write fx's.
name = ''
dataType = ''
dataLen = 1
dataPrecision = 0
dataScale = 0
query = ''
isShape = False
isLOB = False
def __init__(self, Name, DataType, DataLen, DataPrecision, DataScale):
self.name = Name
self.dataType = DataType
self.dataLen = DataLen
self.dataPrecision = DataPrecision
self.dataScale = DataScale
if DataType == 'WKT':
self.query = 'sdo_util.to_wktgeometry(t.' + Name + ') wkb, '
else:
self.query = 't.' + Name
if DataType == 'SDO_GEOMETRY':
self.isShape = True
if DataType == 'BLOB' or DataType == 'CLOB' or DataType == 'WKT':
self.isLOB = True
def getArcType(self, *args): # Data type translation 'Oracle_type':'ESRI_type'
return {
# 'BINARY_DOUBLE':'DOUBLE',
# 'BINARY_FLOAT':'FLOAT',
# 'BLOB':'BLOB',
'CHAR':'STRING',
'CLOB':'CLOB',
'DATE':'DATE',
# 'FLOAT':'FLOAT',
# 'LONG':'LONG',
# 'LONG RAW':'BLOB',
'NUMBER':'DOUBLE',
# 'RAW':'BLOB',
# 'ROWID':'SHORT',
'SDO_GEOMETRY':'GEOMETRY',
'VARCHAR2':'STRING',
'WKT':'WKT',
}.get(self.dataType,"undefined")
def getCxType(self, *args): # Data type translation 'Oracle_type':'cx_Oracle.type'
return {
'BLOB':cx_Oracle.BLOB,
'CHAR':cx_Oracle.STRING,
'CLOB':cx_Oracle.CLOB,
'DATE':cx_Oracle.DATETIME,
'NUMBER':cx_Oracle.NUMBER,
'SDO_GEOMETRY':cx_Oracle.CLOB,
'VARCHAR2':cx_Oracle.STRING,
}.get(self.dataType,"undefined")
class pyTable:
# Create an object to track columns for read and write operations.
# BLOB, CLOB and SDO_GEOMETRY types will need additional support in Read and Write fx's.
length = 0
# colList = [] # The original list of columns is coming from an Oracle query.
# These two lists are different because of the way I treat shape.
# I create a FC and then add attribute columns. This puts the Shape column first in the list.
__insCols = [] # I use insCols as a list of column type objects to write to ArcMap.
__insertFields = []
__colTuple = None
__myData = []
__pKey = 'P_KEY' # The name of the primary key field should be <table>_CN
__insBlobCols = [] # A list of column positions that contain BLOB data types.
__insKeyCol = -1 # The position of the primary key column.
def __init__(self, ColList):
self.colList = ColList[:]
self.length = len(ColList)
self.isFC = self.__getShape()
self.__select = self.getSelect()
arcpy.AddMessage('New table class created with ' + str(self.length) + ' columns.')
def __del__(self):
self.colList = []
del self.__insCols [:]
del self.__insertFields [:]
del self.__myData [:]
del self.__insBlobCols [:]
def addDataRow(self, inDataRow):
self.__myData.append(inDataRow)
def getInsCols(self):
return self.__insCols
def getTuple(self):
return self.__colTuple
def getPK(self):
return self.__pKey
def getInsBlobCols(self):
return self.__insBlobCols
def clearData(self):
self.__myData = []
def getData(self):
return self.__myData
def getKeyCol(self):
return self.__insKeyCol
def __getShape(self):
isFeature = False
featureName = ''
for col in self.colList:
if col.isShape:
isFeature = True
featureName = col.name
if isFeature:
wktShape = column(featureName, 'WKT', 0, 0, 0)
self.__insCols.append(wktShape)
for col in self.colList:
if not col.isShape:
self.__insCols.append(col)
return isFeature
def getSelect(self):
# Build the select statement
# Build the list of insert Field names
# Build the Tuple of insert Field names
# Identify the LOB columns by index number
statement = 'select '
del self.__insertFields[:]
indx = 0
# print ('Table has ', len(self.__insCols), ' insert columns.')
for col in self.__insCols:
if col.dataType == 'WKT':
statement += 'sdo_util.to_wktgeometry(t.shape) wkb, '
self.__insertFields.append('SHAPE#WKT')
else:
statement += 't.' + col.name + ', '
self.__insertFields.append(col.name)
if col.dataType == 'BLOB':
self.__insBlobCols.append(indx)
#
# ToDo: The key column should be <table>_CN
# But, the logic needs to work for views with different names.
#
if col.name == self.__pKey:
self.__insKeyCol = indx
indx += 1
statement = statement[:statement.rfind(',')] # Trim off the trailing comma
# print ('Insert is composed of ', len(self.__insertFields), ' fields.' )
self.__colTuple = tuple(self.__insertFields)
return statement
def createTemp(self, WorkSpace, tempFC):
success = False
insertCols = self.__insCols
arcpy.AddMessage('Adding ' + tempFC + ' with ' + str(len(insertCols)) + ' columns.')
try:
if self.isFC:
arcpy.CreateFeatureclass_management(WorkSpace, tempFC, 'POINT')
arcpy.AddMessage(tempFC + ' feature class was successfully created.')
else:
arcpy.CreateTable_management(WorkSpace, tempFC)
arcpy.AddMessage(tempFC + ' table was successfully created.')
for col in insertCols:
esriType = col.getArcType()
if esriType == "undefined":
arcpy.AddError('Data type not currently supported, ' + col.dataType)
return success
if col.dataType <> 'WKT':
arcpy.AddField_management(tempFC, col.name, esriType, col.dataPrecision, col.dataScale, col.dataLen)
arcpy.AddMessage('Created column: ' + col.name)
success = True
except:
e = sys.exc_info()[1]
arcpy.AddError('Create of ' + tempFC + ' failed with ' + str(e.args[0]))
return success

You are making a shallow copy of the list passed to your class in the init function.
See Shallow and deep copy operations in the Python documentation for some basic information.
self.colList = ColList[:] makes a new LIST, but inside that new list are references to the same objects that were in the original list (a shallow copy).
You need a deep copy:
import copy
...
self.colList = copy.deepcopy(ColList)
A deep copy has a new list, as well as new objects initialized to match the objects in the original list. So if the objects in one class' list change, they don't change in every class.

Not getting a sorted list in python

I have a list of process objects(user-defined) that I want to sort to get the most memory-intensive processes at a time.
But reverse-sorting through the sorted is not yielding the required result.
My code:
import psutil as pu
import time
class proc:
def __init__(self,pid,pname,pmem):
self.pid = pid
self.pname = pname
self.pmem = int(pmem)
# def __lt__(self,other):
# return self.pmem<other.pmem
# def __repr__(self):
# return str(self.pmem)+"\t"+self.pname
if __name__ == "__main__":
meg = 1024*1024
gig = meg*1024
while True:
print(pu.cpu_count())
print(pu.cpu_percent())
print("{:.3f} GB".format(pu.virtual_memory().used/gig))
x = []
for p in pu.pids():
pro = pu.Process(pid=p)
# print(pro.memory_info()[0])
# print(pro.memory_info()[1])
x.append(proc(pid=p,pname=pro.name(),pmem=pro.memory_info()[0]))
sorted(x,key=lambda x:x.pmem,reverse=True)
for i in x:
print(str(i.pmem)+'\t'+i.pname)
time.sleep(5)
Output:
http://pastebin.com/7Pz5Yn7A

You should use sort instead of sorted:
x.sort(key=lambda item: item.pmem, reverse=True)
sort sorts existing list; sorted creates a new one.

Reading a fortigate configuration file with Python

Appologies for the really long drawn out question.
I am trying to read in a config file and get a list of rules out.
I have tried to use ConfigParser to do this but it is not a standard config file.
The file contains no section header and no token.
i.e.
config section a
set something to something else
config subsection a
set this to that
next
end
config firewall policy
edit 76
set srcintf "There"
set dstintf "Here"
set srcaddr "all"
set dstaddr "all"
set action accept
set schedule "always"
set service "TCP_5600"
next
edit 77
set srcintf "here"
set dstintf "there"
set srcaddr "all"
set dstaddr "all"
set action accept
set schedule "always"
set service "PING"
next
end
As I couldn't work out how to get ConfigParser to work I thought I would try to iterate through the file, unfortunately I don't have much programming skill so I have got stuck.
I really think I am making this more complicated than it should be.
Here's the code I have written;
class Parser(object):
def __init__(self):
self.config_section = ""
self.config_header = ""
self.section_list = []
self.header_list = []
def parse_config(self, fields): # Create a new section
new_list = []
self.config_section = " ".join(fields)
new_list.append(self.config_section)
if self.section_list: # Create a sub section
self.section_list[-1].append(new_list)
else: self.section_list.append(new_list)
def parse_edit(self, line): # Create a new header
self.config_header = line[0]
self.header_list.append(self.config_header)
self.section_list[-1].append(self.header_list)
def parse_set(self, line): # Key and values
key_value = {}
key = line[0]
values = line[1:]
key_value[key] = values
if self.header_list:
self.header_list.append(key_value)
else: self.section_list[-1].append(key_value)
def parse_next(self, line): # Close the header
self.config_header = []
def parse_end(self, line): # Close the section
self.config_section = []
def parse_file(self, path):
with open(path) as f:
for line in f:
# Clean up the fields and remove unused lines.
fields = line.replace('"', '').strip().split(" ")
if fields[0] == "set":
pass
elif fields[0] == "end":
pass
elif fields[0] == "edit":
pass
elif fields[0] == "config":
pass
elif fields[0] == "next":
pass
else: continue
# fetch and call method.
method = fields[0]
parse_method = "parse_" + method
getattr(Parser, parse_method)(self, fields[1:])
return self.section_list
config = Parser().parse_file('test_config.txt')
print config
The output I am looking for is something like the following;
[['section a', {'something': 'to something else'}, ['subsection a', {'this': 'to that'}]],['firewall policy',['76',{'srcintf':'There'}, {'dstintf':'Here'}{etc.}{etc.}]]]
and this is what I get
[['section a']]
EDIT
I have changed the above to reflect where I am currently at.
I am still having issues getting the output I expect. I just can't seem to get the list right.

class Parser(object):
def __init__(self):
self.my_section = 0
self.flag_section = False
# ...
def parse_config(self, fields):
self.my_section += 1
# go on with fields
# ...
self.flag_section = True
def parse_edit(self, line):
...
def parse_set(self, line):
...
def parse_end(self, line):
...
def parse_file(self, path):
with open(path) as f:
for line in f:
fields = f.strip().split(" ")
method = fields[0]
# fetch and call method
getattr(Parser, "parse_" + method)(self, fields[1:])

I post my answer for people who first come here from Google when trying to parse Fortigate configuration file !
I rewrote what I found here based on my own needs and it works great.
from collections import defaultdict
from pprint import pprint
import sys
f = lambda: defaultdict(f)
def getFromDict(dataDict, mapList):
return reduce(lambda d, k: d[k], mapList, dataDict)
def setInDict(dataDict, mapList, value):
getFromDict(dataDict, mapList[:-1])[mapList[-1]] = value
class Parser(object):
def __init__(self):
self.config_header = []
self.section_dict = defaultdict(f)
def parse_config(self, fields): # Create a new section
self.config_header.append(" ".join(fields))
def parse_edit(self, line): # Create a new header
self.config_header.append(line[0])
def parse_set(self, line): # Key and values
key = line[0]
values = " ".join(line[1:])
headers= self.config_header+[key]
setInDict(self.section_dict,headers,values)
def parse_next(self, line): # Close the header
self.config_header.pop()
def parse_end(self, line): # Close the section
self.config_header.pop()
def parse_file(self, path):
with open(path) as f:
gen_lines = (line.rstrip() for line in f if line.strip())
for line in gen_lines:
# pprint(dict(self.section_dict))
# Clean up the fields and remove unused lines.
fields = line.replace('"', '').strip().split(" ")
valid_fields= ["set","end","edit","config","next"]
if fields[0] in valid_fields:
method = fields[0]
# fetch and call method
getattr(Parser, "parse_" + method)(self, fields[1:])
return self.section_dict
config = Parser().parse_file('FGT02_20130308.conf')
print config["system admin"]["admin"]["dashboard-tabs"]["1"]["name"]
print config["firewall address"]["ftp.fr.debian.org"]["type"]

I do not know if this can help you too, but it did for me : http://wiki.python.org/moin/ConfigParserExamples
Have fun !

I would do it in a simpler way:
flagSection = False
flagSub = False
mySection = 0
mySubsection = 0
myItem = 0
with open('d:/config.txt', 'r') as f:
gen_lines = (line.rstrip() for line in f if line.strip())
for line in gen_lines:
if line[0:7]=='config ':
mySection = mySection + 1
newLine = line[7:]
# Create a new section
# Mark section as open
flagSection == True
elif line[0:5]=='edit '):
mySubsection = mySubsection + 1
newLine = line[5:]
# Create a new sub-section
# Mark subsection as open
flagSub == true
elif line[0:4]=='set '):
myItem = myItem + 1
name, value = x.split(' ',2)[1:]
# Add to whatever is open
elif line=='end':
# If subsection = open then close and goto end
if flagSub:
# Or if section = open then close and goto end
elif flagSection:
# :End
continue
The instruction gen_lines = (line.rstrip() for line in f if line.strip())
creates a generator of not empty lines (thanks to the test if line.strip()) without newline and without blanks at the right (thanks to line.rstrip())
.
If I would know more about the operations you want to perform with name,value and in the section opened with if line=='end' , I could propose a code using regexes.
Edit
from time import clock
n = 1000000
print 'Measuring times with clock()'
te = clock()
for i in xrange(n):
x = ('abcdfafdf'[:3] == 'end')
print clock()-te,
print "\tx = ('abcdfafdf'[:3] == 'end')"
te = clock()
for i in xrange(n):
x = 'abcdfafdf'.startswith('end')
print clock()-te,
print "\tx = 'abcdfafdf'.startswith('end')"
print '\nMeasuring times with timeit module'
import timeit
ti = timeit.repeat("x = ('abcdfafdf'[:3] == 'end')",repeat=10,number = n)
print min(ti),
print "\tx = ('abcdfafdf'[:3] == 'end')"
to = timeit.repeat("x = 'abcdfafdf'.startswith('end')",repeat=10,number = n)
print min(to),
print "\tx = 'abcdfafdf'.startswith('end')"
result:
Measuring times with clock()
0.543445605517 x = ('abcdfafdf'[:3] == 'end')
1.08590449345 x = 'abcdfafdf'.startswith('end')
Measuring times with timeit module
0.294152748464 x = ('abcdfafdf'[:3] == 'end')
0.901923289133 x = 'abcdfafdf'.startswith('end')
Is the fact the times are smaller with timieit than with clock() due to the fact that the GC is unplugged when the program is run ? Anyway, with either clock() or timeit module , executing startswith() takes more time than slicing.