I have a for loop that cycles through and creates 3 data frames and stores them in a dictionary. From each of these data frames, I would like to be able to create another dictionary, but I cant figure out how to do this.
Here is the repetitive code without the loop:
Trad = allreports2[allreports2['Trad'].notna()]
Alti = allreports2[allreports2['Alti'].notna()]
Alto = allreports2[allreports2['Alto'].notna()]
Trad_dict = dict(zip(Trad.State, Trad.Position))
Alti_dict = dict(zip(Alti.State, Alti.Position))
Alto_dict = dict(zip(Alto.State, Alto.Position))
As stated earlier, I understand how to make the 3 dataframes by storing them in a dictionary and I understand what needs to go on the right side of the equal sign in the second statement in the for loop, but not what goes on the left side (denoted below as XXXXXXXXX).
Routes = ['Trad', 'Alti', 'Alto']
dfd = {}
for route in Routes:
dfd[route] = allreports2[allreports2[route].notna()]
XXXXXXXXX = dict(zip(dfd[route].State, dfd[route].Position))
(Please note: I am very new to Python and teaching myself so apologies in advance!)
This compromises readability, but this should work.
Routes = ['Trad', 'Alti', 'Alto']
dfd, output = [{},{}] # Unpack List
for route in Routes:
dfd[route] = allreports2[allreprots2[route].notna()]
output[route] = dict(zip(dfd[route].State, dfd[route].Position))
Trad_dict, Alti_dict, Alto_dict = list(output.values()) # Unpack List
Reference
How can I get list of values from dict?
I am trying to write a orientation routine for a 3-axis accelerometer. The part I am getting stuck on is, I have one dict with all my axis' listed, after taking the 'z-axis' reading, I want to remove that axis from the Availiable_axis list. Here is a portion of my code that demonstrates what I am trying to do.
import operator
Readings1 = { 0:{'x':0.1, 'y':-1, 'z':-0.1}, 1:{'x':.4, 'y':-.1, 'z':-0.1},
2:{'x':-0.4, 'y':-.8, 'z':-0.1}, 3:{'x':0.1, 'y':-.1, 'z':-0.6},
4:{'x':0.1, 'y':-.2, 'z':0.4}}
SetupValue = {'Axis':{'x-axis':'x','y-axis':'y','z-axis':'z'}}
Available_axis = [SetupValue['Axis']['x-axis'], SetupValue['Axis']['y-axis'], SetupValue['Axis']['z-axis']]
axes = Readings1[0]
print axes
for key in axes:
axes[key] = abs(axes[key])
print axes
print (max(axes.iteritems(), key = operator.itemgetter(1))[0])
Available_axis.pop(max(axes.iteritems(), key = operator.itemgetter(1))[0],0)
Any help would be appreciated.
Available_axis is a list. When popping from a list, you must specify the integer location.
You can also have a short list comprehension that removes the target variable.
Available_axis = [x for x in Available_axis
if x != max(axes.iteritems(), key = operator.itemgetter(1))[0]]
I am having a hard time trying to figure out how to address leaking memory. I think this might be an issue with pywin32 but I am not completely sure.
My code to do reading/writing individual items seems to work just fine, but when using group functions it slowly leaks memory. I suspect this is from the 1-based array that must be passed in the server_handles.
Does anyone know of a work around?
def read_group(self, group, mode=OPC_SYNC, source=OPC_DS_CACHE):
""" Read a group returning a list of Result tuples or None. """
if group not in self.groups:
raise OPCError("Group does not exist, add first.")
items = []
server_handles = []
for name, item in self.items.items():
if item.group == group:
items.append(name)
server_handles.append(item.server_handle)
if not server_handles:
return None
# Note that arrays are 1 based so must add an element to beginning
server_handles.insert(0, 0)
if mode == OPC_SYNC:
val, err, qual, ts = self.groups[group].SyncRead(
Source=source,
NumItems=len(server_handles) - 1,
ServerHandles=server_handles)
# Map the return arrays into a list of result tuples
result = []
for i in range(len(items)):
result.append(Result(
items[i],
val[i],
OPC_QUALITY[qual[i]],
str(ts[i]),
err[i],
))
return result
Looking for a simple example of retrieving 500 items from dynamodb minimizing the number of queries. I know there's a "multiget" function that would let me break this up into chunks of 50 queries, but not sure how to do this.
I'm starting with a list of 500 keys. I'm then thinking of writing a function that takes this list of keys, breaks it up into "chunks," retrieves the values, stitches them back together, and returns a dict of 500 key-value pairs.
Or is there a better way to do this?
As a corollary, how would I "sort" the items afterwards?
Depending on you scheme, There are 2 ways of efficiently retrieving your 500 items.
1 Items are under the same hash_key, using a range_key
Use the query method with the hash_key
you may ask to sort the range_keys A-Z or Z-A
2 Items are on "random" keys
You said it: use the BatchGetItem method
Good news: the limit is actually 100/request or 1MB max
you will have to sort the results on the Python side.
On the practical side, since you use Python, I highly recommend the Boto library for low-level access or dynamodb-mapper library for higher level access (Disclaimer: I am one of the core dev of dynamodb-mapper).
Sadly, neither of these library provides an easy way to wrap the batch_get operation. On the contrary, there is a generator for scan and for query which 'pretends' you get all in a single query.
In order to get optimal results with the batch query, I recommend this workflow:
submit a batch with all of your 500 items.
store the results in your dicts
re-submit with the UnprocessedKeys as many times as needed
sort the results on the python side
Quick example
I assume you have created a table "MyTable" with a single hash_key
import boto
# Helper function. This is more or less the code
# I added to devolop branch
def resubmit(batch, prev):
# Empty (re-use) the batch
del batch[:]
# The batch answer contains the list of
# unprocessed keys grouped by tables
if 'UnprocessedKeys' in prev:
unprocessed = res['UnprocessedKeys']
else:
return None
# Load the unprocessed keys
for table_name, table_req in unprocessed.iteritems():
table_keys = table_req['Keys']
table = batch.layer2.get_table(table_name)
keys = []
for key in table_keys:
h = key['HashKeyElement']
r = None
if 'RangeKeyElement' in key:
r = key['RangeKeyElement']
keys.append((h, r))
attributes_to_get = None
if 'AttributesToGet' in table_req:
attributes_to_get = table_req['AttributesToGet']
batch.add_batch(table, keys, attributes_to_get=attributes_to_get)
return batch.submit()
# Main
db = boto.connect_dynamodb()
table = db.get_table('MyTable')
batch = db.new_batch_list()
keys = range (100) # Get items from 0 to 99
batch.add_batch(table, keys)
res = batch.submit()
while res:
print res # Do some usefull work here
res = resubmit(batch, res)
# The END
EDIT:
I've added a resubmit() function to BatchList in Boto develop branch. It greatly simplifies the worklow:
add all of your requested keys to BatchList
submit()
resubmit() as long as it does not return None.
this should be available in next release.
Started my first Python pet project using PyGTK. Though it is a really powerful GUI toolkit and looks excellent, I have some pet peeves. So I thought about transitioning to something else, as it's not yet too extensive. Had a look around on SO and python documentation, but didn't get a good overview.
What's nice about PyGTK:
Glade files
self.signal_autoconnect({...})
self.get_widget() as __getattr__
This is bugging me however:
manual gobject.idle_add(lambda: ... and False)
no standard functionality to save application/window states
TreeView needs array building
widget.get_selection().get_selected(), model.get_value(iter, liststore_index)
TreeView: Because this is the main interface element, it's the most distracting. Basically my application builds a list of dictionaries to be displayed name=column+row=>value. To display it using GTK there needs to be a manual conversion process, ordering, typecasts. This seems a lot of overhead, and I wished for something more object-oriented here. PyGtk has many abstractions atop gtk+ but still seems rather low-levelish. I'd prefer to pass my dict as-is and have columns pre-defined somehow. (GtkBuilder can predefine TreeView columns, but this doesn't solve the data representation overhead.)
When I get a mousclick on my TreeView list, I also have to convert everything back into my application data structures. And it's also irksome that PyGTK doesn't wrap gtk+ calls with gobject.idle itself, if run from a non-main thread. Right now there is a lot of GUI code that I believe shouldn't be necessary, or could be rationalized away.
? So, are there maybe additional wrappers on top of PyGTK. Or which other toolkit supports simpler interfaces for displaying a Grid / TreeView. I've read a lot about wxPython being everyones favourite, but it's less mature on Linux. And PyQT seems to be mostly the same abstraction level as PyGTK. Haven't used TkInter much so don't know about if it has simpler interfaces, but it anyway looks unattractive. As does PyFLTK. PyJamas sounds fascinating, but is already too far out (Desktop application).
.
So, GUI toolkit with dict -> Grid display. Which would you pick?
.
Just as exhibit, this is my current TreeView mapping function. Sort of works, but I would rather have something standard:
#-- fill a treeview
#
# Adds treeviewcolumns/cellrenderers and liststore from a data dictionary.
# Its datamap and the table contents can be supplied in one or two steps.
# When new data gets applied, the columns aren't recreated.
#
# The columns are created according to the datamap, which describes cell
# mapping and layout. Columns can have multiple cellrenderers, but usually
# there is a direct mapping to a data source key from entries.
#
# datamap = [ # title width dict-key type, renderer, attrs
# ["Name", 150, ["titlerow", str, "text", {} ] ],
# [False, 0, ["interndat", int, None, {} ] ],
# ["Desc", 200, ["descriptn", str, "text", {} ], ["icon",str,"pixbuf",{}] ],
#
# An according entries list then would contain a dictionary for each row:
# entries = [ {"titlerow":"first", "interndat":123}, {"titlerow":"..."}, ]
# Keys not mentioned in the datamap get ignored, and defaults are applied
# for missing cols. All values must already be in the correct type however.
#
#staticmethod
def columns(widget, datamap=[], entries=[], pix_entry=False):
# create treeviewcolumns?
if (not widget.get_column(0)):
# loop through titles
datapos = 0
for n_col,desc in enumerate(datamap):
# check for title
if (type(desc[0]) != str):
datapos += 1 # if there is none, this is just an undisplayed data column
continue
# new tvcolumn
col = gtk.TreeViewColumn(desc[0]) # title
col.set_resizable(True)
# width
if (desc[1] > 0):
col.set_sizing(gtk.TREE_VIEW_COLUMN_FIXED)
col.set_fixed_width(desc[1])
# loop through cells
for var in xrange(2, len(desc)):
cell = desc[var]
# cell renderer
if (cell[2] == "pixbuf"):
rend = gtk.CellRendererPixbuf() # img cell
if (cell[1] == str):
cell[3]["stock_id"] = datapos # for stock icons
expand = False
else:
pix_entry = datapos
cell[3]["pixbuf"] = datapos
else:
rend = gtk.CellRendererText() # text cell
cell[3]["text"] = datapos
col.set_sort_column_id(datapos) # only on textual cells
# attach cell to column
col.pack_end(rend, expand=cell[3].get("expand",True))
# apply attributes
for attr,val in cell[3].iteritems():
col.add_attribute(rend, attr, val)
# next
datapos += 1
# add column to treeview
widget.append_column(col)
# finalize widget
widget.set_search_column(2) #??
widget.set_reorderable(True)
# add data?
if (entries):
#- expand datamap
vartypes = [] #(str, str, bool, str, int, int, gtk.gdk.Pixbuf, str, int)
rowmap = [] #["title", "desc", "bookmarked", "name", "count", "max", "img", ...]
if (not rowmap):
for desc in datamap:
for var in xrange(2, len(desc)):
vartypes.append(desc[var][3]) # content types
rowmap.append(desc[var][0]) # dict{} column keys in entries[] list
# create gtk array storage
ls = gtk.ListStore(*vartypes) # could be a TreeStore, too
# prepare for missing values, and special variable types
defaults = {
str: "",
unicode: u"",
bool: False,
int: 0,
gtk.gdk.Pixbuf: gtk.gdk.pixbuf_new_from_data("\0\0\0\0",gtk.gdk.COLORSPACE_RGB,True,8,1,1,4)
}
if gtk.gdk.Pixbuf in vartypes:
pix_entry = vartypes.index(gtk.gdk.Pixbuf)
# sort data into gtk liststore array
for row in entries:
# generate ordered list from dictionary, using rowmap association
row = [ row.get( skey , defaults[vartypes[i]] ) for i,skey in enumerate(rowmap) ]
# autotransform string -> gtk image object
if (pix_entry and type(row[pix_entry]) == str):
row[pix_entry] = gtk.gdk.pixbuf_new_from_file(row[pix_entry])
# add
ls.append(row) # had to be adapted for real TreeStore (would require additional input for grouping/level/parents)
# apply array to widget
widget.set_model(ls)
return ls
pass
Try Kiwi, maybe? Especially with its ObjectList.
Update: I think Kiwi development has moved to PyGTKHelpers.
I hadn't come across Kiwi before. Thanks, Johannes Sasongko.
Here are some more tooklits that I keep bookmarked. Some of these are wrappers around other toolkits (GTK, wxWidgets) while others stand alone:
AVC
Dabo
pyFLTK
pyglet
PyGTK
PyGUI
uxPython
Wax
wxPython
wxpita
WxWrappers
(I've included a few that were already mentioned for the sake of others who come across this post. I would have posted this as a comment, but it's a bit too long.)
See also pygtkhelpers' ObjectList
I would suggest taking a look at wxPython. I found it really easy to pick up and very powerful too although I'd have to admit I've not done a lot with Treeviews myself.
wxWidgets calls the equivalent control a wxTreeCtrl
[Edit] The wxDataViewTreeCtrl might actually be of more use in your case.