I have written a simple script that prints out and adds the name of a table and it's associated column headings to a python list:
for table in arcpy.ListTables():
for field in arcpy.ListFields(table):
b.append(field.name + "," + fc)
print b
In each table there are a number of column headings. There are many instances where one or more tables contain the same column headings. I want to do a bit of a reverse python dictionary instead of a list, where keys are the column headings and the values are the table names. My idea is, to find the all the tables that each column heading lies within.
I've been playing around all afternoon and I think I am over thinking this so I came here for some help. If anyone can suggest how I can accomplish this, i would appreciate it.
Thanks,
Mike
Try this:
result = {}
for table in arcpy.ListTables():
for field in arcpy.ListFields(table):
result.setdefault(field.name, []).append(table)
If I understand correctly, you want to map from a column name to a list of tables that contain that have columns with that name. That should be easy enough to do with a defaultdict:
from collections import defaultdict
header_to_table_dict = defaultdict(list)
for table in arcpy.ListTables():
for field in arcpy.ListFields(table):
header_to_table_dict[field.name].append(table.name)
I'm not sure if table.name is what you want to save, exactly, but this should get you on the right track.
You want to create a dictionary in which each key is a field name, and each value is a list of table names:
# initialize the dictionary
col_index = {}
for table in arcpy.ListTables():
for field in arcpy.ListFields(table):
if field.name not in col_index:
# this is a field name we haven't seen before,
# so initialize a dictionary entry with an empty list
# as the corresponding value
col_index[field.name] = []
# add the table name to the list of tables for this field name
col_index[field.name].append(table.name)
And then, if you want want a list of tables that contain the field LastName:
list_of_tables = col_index['LastName']
If you're using a database that is case-insensitive with respect to column names, you might want to convert field.name to upper case before testing the dictionary.
Related
I have a data frame like
I have a dictionary with the ec2 instance details
Now, I want to add a new column 'Instance Name' and populate it based on a condition that the instance ID in the dictionary is in the column 'ResourceId' and further, depending on what is there in the Name field in dictionary for that instance Id, I want to populate the new column value for each matching entry
Finally I want to create separate data frames for my specific use-cases e.g. to get only Box-Usage results. Something like this
box_usage = df[df['lineItem/UsageType'].str.contains('BoxUsage')]
print(box_usage.groupby('Instance Name')['lineItem/BlendedCost'].sum())
The new column value is not coming up against the respective Resource Id as I desire. It is rather coming up sequentially.
I have tried bunch of things including what I mentioned in above code, but no result yet. Any help?
After struggling through several options, I used the .apply() way and it did the trick
df.insert(loc=17, column='Instance_Name', value='Other')
instance_id = []
def update_col(x):
for key, val in ec2info.items():
if x == key:
if ('MyAgg' in val['Name']) | ('MyAgg-AutoScalingGroup' in val['Name']):
return 'SharkAggregator'
if ('MyColl AS Group' in val['Name']) | ('MyCollector-AutoScalingGroup' in val['Name']):
return 'SharkCollector'
if ('MyMetric AS Group' in val['Name']) | ('MyMetric-AutoScalingGroup' in val['Name']):
return 'Metric'
df['Instance_Name'] = df.ResourceId.apply(update_col)
df.Instance_Name.fillna(value='Other', inplace=True)
Say I have 100 different integers I want to store like a row with 100 columns.
I am trying it like this:
db = sqlite3.connect("test.db")
c = db.cursor()
c.execute('''
CREATE TABLE IF NOT EXISTS nums(
id INTEGER PRIMARY KEY,
''')
for i in range(100):
c.execute('''
ALTER TABLE nums
ADD ''' + 'column_' + i + '''INTEGER''')
db.commit()
Someone told me that when you are using numbers as column names you could probably do it a better way. But if I for example have a list with strings in python, and I want to loop through them and store every individual string in its own column, the approach would be the same, right?
However, this code runs without errors for me, but no new table is created, how come?
Your ALTER statement is incorrect as it's missing the COLUMN after ADD. You can use the following:
for i in range(100):
c.execute(f'ALTER TABLE nums ADD COLUMN column_{i} INTEGER')
I am new to sqlalchemy. So any help is appreciated.
I have a function that constructs my queries for my application. I pass it a list of tables to join.
Here are the relevant code snippets.
class Scope(Base):
entry = Column(String(512))
location_id = Column(Integer, ForeignKey('location_id'))
type = Column(String(128))
class Location(Base):
id = Column(Integer, primary_key=True)
name = Column(String(512)
modified_by = Column(String(128))
instances = [Scope, Location]
join_classes = [Location]
queryset = session.query(*instances).join(*join_classes).all()
Here is the SQL query that runs (when I print queryset to screen before the .all()):
queryset: SELECT scope.id AS scope_id, scope.location_id AS scope_location_id, scope.entry AS scope_entry, scope.type AS scope_type, location.name AS location_name, location.modified_by AS location_modified_by,
FROM scope JOIN location ON location.id = scope.location_id
My end result I want is: a list of dictionaries for all columns (from both tables - like regular inner join gives a single table).
However, I am getting the a list when I type(queryset) and when I just try to do [u._asdict() for u in queryset] which is how I return a list of dictionaries in queries that don't have a join, it only returns a list of dictionaries for 1 column from each table (the column in the __repr__.
I need all columns from both tables to be returned.
Right now this is how what is what I get:
[{'Scope': 192.168.0.0/24, 'Location': main}, ...]
I need something like, where all columns from the join are returned in a list of dictionaries:
[{'Scope.entry': 192.168.0.0/24, 'Scope.type': 'virtual', 'Location.name': main, 'Location.modified_by': 'jim'}, ...]
In my code the instances & join_classes are dynamically passed and not hard coded as different functions pass the table models to join on (with the 1st model table being the table that all proceeding join on). I need this to work with a join on multiple tables (but all tables will be joined to the 1st model table, Scope in this example.)
Edit: I finally realized I was getting a list of sqlalchemy table objects back. That is why I was getting the __repr__ values when displaying.
records = DBSession.query(GeneralLedger, ConsolidatedLedger).join(ConsolidatedLedger, GeneralLedger.invoiceId == ConsolidatedLedger.invoiceId).all()
this sort of query return both table data
Well, writing something done definitely helps you figure out the answer.
For anyone that might benefit this is what I did. I am sure there is a more eloquent way to do this so please let me know if so.
I finally read my output correctly and realized that it was giving me 2 table model objects (1 per table joined). I then iterated over each, converted each iteration to a list of dictionaries and then merged those dictionaries appropriately so that i had one list of dictionaries like a inner join table would give me.
Here is some of my code:
for obj in queryset:
result.append(queryset_to_dict(obj))
for r in result:
new_dict = {}
for inner in r:
new_dict = {**new_dict, **inner}
new_result.append(new_dict)
Note the queryset_to_dict is a function I created for converting sqlalchemy table model objects to list of dictionaries.
I am trying to get a specific document from a Domino view.
The view has 3 columns: Name, Surname, Age.
The problem is, that Name is not unique, so I need to get the document that matches 'John' in the Name column (1st column) as well as 'Doe' in the second column (Surname).
So obviously the following won't work: doc = view.GetDocumentByKey('John')
There is a NotesView COM class which contains the .GetDocumentByKey() method, which allows one to enter a key array. But I am not able to enter a key array in Python.
I have tried the following:
doc = view.GetDocumentByKey('John Doe')
doc = view.GetDocumentByKey('John, Doe')
doc = view.GetDocumentByKey(('John', 'Doe'))
doc = view.GetDocumentByKey(['John', 'Doe'])
But none of them are able to get the needed document.
What is the correct way to pass a key array?
EDIT:
Solution found. There was a sorted hidden column with unique values that I ended up using.
Solution found. There was a sorted hidden column with unique values that I ended up using.
I am writing a python script that is converting a CSV file into an sqlite3 database. There is an id column that i have set up to be "primary key unique" and i know in the CSV file there is repeating information. How to i tell it to only store non-repeating information into the database?
Here is what i have so far.
for row in reader:
counter += 1
#this gets rid of the header in the CSV file
if counter == 1:
continue
s = (row[0],row[2],row[1],row[4],row[3],row[7],row[8],row[9])
course = row[5].split(" ")
c = (row[0],course[0],course[1],row[6])
#when it hits here and sees that two ids are the same, it crashes because it will not allow non-unique values.
curs.execute('''insert into students (id,lastname,firstname,major,email,city,state,zip)
values (?,?,?,?,?,?,?,?)''', s)
curs.execute('''insert into classes (id,subjcode,coursenumber,termcode)
values (?,?,?,?)''', c)
I would really appreciate the help.
You could use INSERT OR IGNORE:
curs.execute('''INSERT OR IGNORE INTO students (id,lastname,firstname,major,email,city,state,zip) VALUES (?,?,?,?,?,?,?,?)''', s)
This will insert the first row with a duplicate id, but ignore all successive duplicates.
you can do that by using a UNIQUE constraint on your tables' id, and then use a INSERT OR IGNORE