I have started using web2py for a web application and try to use SQLFORM.grid(...) to display a paginated listing of one of my db-table's data like in the following minimal example.
grid=SQLFORM.grid(query,
links=links,
fields=[db.example.date,db.example.foo, db.example.bar])
The db.example.date field contains a Python datetime.datetime object in UTC. At the moment it is displayed just plainly like that. However, I want to have more control about the actual output in a way that I can set the local timezone and modify the output string to have something like "2 hours ago".
As seen in another question[0] I can use the links to insert new columns. Unfortunately I can't seem to sort the rows by a field I have inserted in such way. Also, they are inserted on the right instead of actually replacing my first column. So that does not seem to be a solution.
To sum it up: How do I gain control about the way db.example.date is printed out in the end?
[0] Calculated Fields in web2py sqlgrid
You can achieve your goal when you define the table in your model. The represent parameter in the Field constructor that you used in define_table will be recognized by the SQLFORM.grid. For example, if you wanted to just print the date with the month name you could put the following in your model.
Field('a_date', type='date', represent=lambda x, row: x.strftime("%B %d, %Y")),
your function could also convert to local time.
You need to use prettydate to change the datetime arid format in a humanized string, and call it in the represent parameter of your Field() descriptor. For example :
from gluon.tools import prettydate
db.example.date.represent = lambda v,r: prettydate(r.date)
That way, any display of the db.example.date would be displayed humanized, including through SQLFORM.grid
If you don't want to have the date always represented in this way as per David Nehme's answer. Just before your grid creation, you can set the db.table.field.represent in the controller.
db.example.date.represent = lambda value, row: value.strftime("%B %d, %Y")
followed by.
grid = SQLFORM.grid(query,....
I use this often when I join tables. If there is a row.field in the represent from the model file it breaks because it then must be more specific, row.table.field.
Related
I want to show some data in table form. I took QTableWidget for it having multiple columns. One column of it will contain time(hh:mm format).
I also want user to edit any item of table but with corresponding format.
I was able to add data in QTableWidget but i couldn't set text format of time column.
This i want to achieve so that user can edit time only in hh:mm format.
If possible please write your answer code in python.
Since you want the user enter time date, I would suggest to reuse the already existing QDateTimeEdit class in the following way:
dateTime = QDateTimeEdit();
dateTime.setDisplayFormat("hh:mm");
dateTime.setFrame(False);
myTableWidget.setCellWidget(row, column, dateTime);
The user will be able to edit the "time data" this way in your table widget. Moreover, it will be also convenient due to the steps that can be applied.
If you really insist on reinventing this yourself, you can use a QLineEdit with custom validator againt the desired hh::mm format.
dateTime = QLineEdit();
dateTime.setValidator(QRegExpValidator(QRegExp("^([0-9]|0[0-9]|1[0-9]|2[0-3]):[0-5][0-9]$")));
myTableWidget.setCellWidget(row, column, dateTime);
I plan to make a 'table' class that I can use throughout my data-analyzis program to store gathered data to. Objective is to make simple tables like this:
ID Mean size Stdv Date measured Relative flatness
----------------------------------------------------------------
1 133.4242 34.43 Oct 20, 2013 32093
2 239.244 34.43 Oct 21, 2012 3434
I will follow the sqlite3 suggestion from this post: python-data-structure-for-maintaing-tabular-data-in-memory, but I will still need to save it as a csv file (not as a dbase) and I want it to eat my data as we go: add columns on the fly whenever new measures become available and are deemed to be interesting. For that the class will need to be able to determine the data type of the data thrown at it.
Sqlite3 has limited datatypes, float, int, date and string. Python and numpy together have many types. Is there an easy was to quickly decide what the datatype is of the variable? So my table class can automatically add a column when new data is entered containing new fields.
I am not too concerned about performance, the table should be fairly small.
I want to use my class like so:
dt = Table()
dt.add_record({'ID':5, 'Mean size':39.4334'})
dt.add_record({'ID':5, 'Goodness of fit': 12})
In the last line, there is new data. the Table class needs to figure out what kind of data that is and then add a column to the sqlite3 table. Making it all string seems a bit to floppy, I still want to keep my high precision floats correct....
Also: If something like this already exists, I'd like to know about it.
It seems that your question is: "Is there an easy was to quickly decide what the datatype is of the variable?". This is a simple question, and the answer is:
type(variable).
But the context you provide requires a more careful answer.
Since SQLite3 only provides only a few data types (slightly different ones than what you said), you need to map your input variables to the types provided by SQLite3.
But you may encounter further problems: You may need to change the types of columns as you receive new records, if you do not want to require that the column type be fixed in advance.
For example, for the Goodness of fit column in your example, you get an int (12) first. But you may get a float (e.g. 10.1) the second time, which shows that both values must be interpreted as floats. And if next time you receive a string, then all of them must be strings, right? But then the exact formatting of the numbers also counts: whereas 12 and 12.0 are the same when you interpret them as floats, they are not when you interpret them as strings; and the first value may become "12.0" when you convert all of them to strings.
So either you throw an exception when the type of consecutive values for the same column do not match, or you try to convert the previous values according to the new ones; but occasionally you may need to re-read the input.
Nevertheless, once you make those decision regarding the expected behavior, it should not be a very difficult problem to implement.
Regarding your last question: I personally do not know of an existing implementation to this problem.
I have chapter times in the form of HH:MM:SS. I am parsing them from a document, and I will have times as a string in the format of '00:12:14'. How would I store this in a mysql column, and then retrieve it in the required format to be able to:
1) order by time;
2) convert to a string in the above format.
I suggest you look at the MySQL time type. It will allow you to sort and format as you wish.
http://dev.mysql.com/doc/refman/5.0/en/time.html
Use the TIME type.
It allows "time values to be represented in several formats, such as quoted strings or as numbers, depending on the exact type of the value and other factors." In addition, you can perform various functions to manipulate the time.
If I have such a simple task, I choose a simple solution: I would choose the python datetime.time module (see: datetime.time) and store a TIME object using strftime.
Loading it back in is a little painful as you would have to split your string at : and then pass the values to the time constructor. Example:
def load(timestr):
hours,minutes,seconds = timestr.split(":")
return datetime.time(hours,minutes,seconds)
Hope this helps.
I have a scientific model which I am running in Python which produces a lookup table as output. That is, it produces a many-dimensional 'table' where each dimension is a parameter in the model and the value in each cell is the output of the model.
My question is how best to store this lookup table in Python. I am running the model in a loop over every possible parameter combination (using the fantastic itertools.product function), but I can't work out how best to store the outputs.
It would seem sensible to simply store the output as a ndarray, but I'd really like to be able to access the outputs based on the parameter values not just indices. For example, rather than accessing the values as table[16][5][17][14] I'd prefer to access them somehow using variable names/values, for example:
table[solar_z=45, solar_a=170, type=17, reflectance=0.37]
or something similar to that. It'd be brilliant if I were able to iterate over the values and get their parameter values back - that is, being able to find out that table[16]... corresponds to the outputs for solar_z = 45.
Is there a sensible way to do this in Python?
Why don't you use a database? I have found MongoDB (and the official Python driver, Pymongo) to be a wonderful tool for scientific computing. Here are some advantages:
Easy to install - simply download the executables for your platform (2 minutes tops, seriously).
Schema-less data model
Blazing fast
Provides map/reduce functionality
Very good querying functionalities
So, you could store each entry as a MongoDB entry, for example:
{"_id":"run_unique_identifier",
"param1":"val1",
"param2":"val2" # etcetera
}
Then you could query the entries as you will:
import pymongo
data = pymongo.Connection("localhost", 27017)["mydb"]["mycollection"]
for entry in data.find(): # this will yield all results
yield entry["param1"] # do something with param1
Whether or not MongoDB/pymongo are the answer to your specific question, I don't know. However, you could really benefit from checking them out if you are into data-intensive scientific computing.
If you want to access the results by name, then you could use a python nested dictionary instead of ndarray, and serialize it in a .JSON text file using json module.
One option is to use a numpy ndarray for the data (as you do now), and write a parser function to convert the query values into row/column indices.
For example:
solar_z_dict = {...}
solar_a_dict = {...}
...
def lookup(dataArray, solar_z, solar_a, type, reflectance):
return dataArray[solar_z_dict[solar_z] ], solar_a_dict[solar_a], ...]
You could also convert to string and eval, if you want to have some of the fields to be given as "None" and be translated to ":" (to give the full table for that variable).
For example, rather than accessing the values as table[16][5][17][14]
I'd prefer to access them somehow using variable names/values
That's what numpy's dtypes are for:
dt = [('L','float64'),('T','float64'),('NMSF','float64'),('err','float64')]
data = plb.loadtxt(argv[1],dtype=dt)
Now you can access the data elements using date['T']['L']['NMSF']
More info on dtypes:
http://docs.scipy.org/doc/numpy/reference/generated/numpy.dtype.html
In one of my Django projects that use MySQL as the database, I need to have a date fields that accept also "partial" dates like only year (YYYY) and year and month (YYYY-MM) plus normal date (YYYY-MM-DD).
The date field in MySQL can deal with that by accepting 00 for the month and the day. So 2010-00-00 is valid in MySQL and it represent 2010. Same thing for 2010-05-00 that represent May 2010.
So I started to create a PartialDateField to support this feature. But I hit a wall because, by default, and Django use the default, MySQLdb, the python driver to MySQL, return a datetime.date object for a date field AND datetime.date() support only real date. So it's possible to modify the converter for the date field used by MySQLdb and return only a string in this format 'YYYY-MM-DD'. Unfortunately the converter use by MySQLdb is set at the connection level so it's use for all MySQL date fields. But Django DateField rely on the fact that the database return a datetime.date object, so if I change the converter to return a string, Django is not happy at all.
Someone have an idea or advice to solve this problem? How to create a PartialDateField in Django ?
EDIT
Also I should add that I already thought of 2 solutions, create 3 integer fields for year, month and day (as mention by Alison R.) or use a varchar field to keep date as string in this format YYYY-MM-DD.
But in both solutions, if I'm not wrong, I will loose the special properties of a date field like doing query of this kind on them: Get all entries after this date. I can probably re-implement this functionality on the client side but that will not be a valid solution in my case because the database can be query from other systems (mysql client, MS Access, etc.)
First, thanks for all your answers. None of them, as is, was a good solution for my problem, but, for your defense, I should add that I didn't give all the requirements. But each one help me think about my problem and some of your ideas are part of my final solution.
So my final solution, on the DB side, is to use a varchar field (limited to 10 chars) and storing the date in it, as a string, in the ISO format (YYYY-MM-DD) with 00 for month and day when there's no month and/or day (like a date field in MySQL). This way, this field can work with any databases, the data can be read, understand and edited directly and easily by a human using a simple client (like mysql client, phpmyadmin, etc.). That was a requirement. It can also be exported to Excel/CSV without any conversion, etc. The disadvantage is that the format is not enforce (except in Django). Someone could write 'not a date' or do a mistake in the format and the DB will accept it (if you have an idea about this problem...).
This way it's also possible to do all of the special queries of a date field relatively easily. For queries with WHERE: <, >, <=, >= and = work directly. The IN and BETWEEN queries work directly also. For querying by day or month you just have to do it with EXTRACT (DAY|MONTH ...). Ordering work also directly. So I think it covers all the query needs and with mostly no complication.
On the Django side, I did 2 things. First, I have created a PartialDate object that look mostly like datetime.date but supporting date without month and/or day. Inside this object I use a datetime.datetime object to keep the date. I'm using the hours and minutes as flag that tell if the month and day are valid when they are set to 1. It's the same idea that steveha propose but with a different implementation (and only on the client side). Using a datetime.datetime object gives me a lot of nice features for working with dates (validation, comparaison, etc.).
Secondly, I have created a PartialDateField that mostly deal with the conversion between the PartialDate object and the database.
So far, it works pretty well (I have mostly finish my extensive unit tests).
You could store the partial date as an integer (preferably in a field named for the portion of the date you are storing, such as year, month or day) and do validation and conversion to a date object in the model.
EDIT
If you need real date functionality, you probably need real, not partial, dates. For instance, does "get everything after 2010-0-0" return dates inclusive of 2010 or only dates in 2011 and beyond? The same goes for your other example of May 2010. The ways in which different languages/clients deal with partial dates (if they support them at all) are likely to be highly idiosyncratic, and they are unlikely to match MySQL's implementation.
On the other hand, if you store a year integer such as 2010, it is easy to ask the database for "all records with year > 2010" and understand exactly what the result should be, from any client, on any platform. You can even combine this approach for more complicated dates/queries, such as "all records with year > 2010 AND month > 5".
SECOND EDIT
Your only other (and perhaps best) option is to store truly valid dates and come up with a convention in your application for what they mean. A DATETIME field named like date_month could have a value of 2010-05-01, but you would treat that as representing all dates in May, 2010. You would need to accommodate this when programming. If you had date_month in Python as a datetime object, you would need to call a function like date_month.end_of_month() to query dates following that month. (That is pseudocode, but could be easily implemented with something like the calendar module.)
It sounds like you want to store a date interval. In Python this would (to my still-somewhat-noob understanding) most readily be implemented by storing two datetime.datetime objects, one specifying the start of the date range and the other specifying the end. In a manner similar to that used to specify list slices, the endpoint would not itself be included in the date range.
For example, this code would implement a date range as a named tuple:
>>> from datetime import datetime
>>> from collections import namedtuple
>>> DateRange = namedtuple('DateRange', 'start end')
>>> the_year_2010 = DateRange(datetime(2010, 1, 1), datetime(2011, 1, 1))
>>> the_year_2010.start <= datetime(2010, 4, 20) < the_year_2010.end
True
>>> the_year_2010.start <= datetime(2009, 12, 31) < the_year_2010.end
False
>>> the_year_2010.start <= datetime(2011, 1, 1) < the_year_2010.end
False
Or even add some magic:
>>> DateRange.__contains__ = lambda self, x: self.start <= x < self.end
>>> datetime(2010, 4, 20) in the_year_2010
True
>>> datetime(2011, 4, 20) in the_year_2010
False
This is such a useful concept that I'm pretty sure that somebody has already made an implementation available. For example, a quick glance suggests that the relativedate class from the dateutil package will do this, and more expressively, by allowing a 'years' keyword argument to be passed to the constructor.
However, mapping such an object into database fields is somewhat more complicated, so you might be better off implementing it simply by just pulling both fields separately and then combining them. I guess this depends on the DB framework; I'm not very familiar with that aspect of Python yet.
In any case, I think the key is to think of a "partial date" as a range rather than as a simple value.
edit
It's tempting, but I think inappropriate, to add more magic methods that will handle uses of the > and < operators. There's a bit of ambiguity there: does a date that's "greater than" a given range occur after the range's end, or after its beginning? It initially seems appropriate to use <= to indicate that the date on the right-hand side of the equation is after the start of the range, and < to indicate that it's after the end.
However, this implies equality between the range and a date within the range, which is incorrect, since it implies that the month of May, 2010 is equal to the year 2010, because May the 4th, 2010 equates to the both of them. IE you would end up with falsisms like 2010-04-20 == 2010 == 2010-05-04 being true.
So probably it would be better to implement a method like isafterstart to explicitly check if a date is after the beginning of the range. But again, somebody's probably already done it, so it's probably worth a look on pypi to see what's considered production-ready. This is indicated by the presence of "Development Status :: 5 - Production/Stable" in the "Categories" section of a given module's pypi page. Note that not all modules have been given a development status.
Or you could just keep it simple, and using the basic namedtuple implementation, explicitly check
>>> datetime(2012, 12, 21) >= the_year_2010.start
True
Can you store the date together with a flag that tells how much of the date is valid?
Something like this:
YEAR_VALID = 0x04
MONTH_VALID = 0x02
DAY_VALID = 0x01
Y_VALID = YEAR_VALID
YM_VALID = YEAR_VALID | MONTH_VALID
YMD_VALID = YEAR_VALID | MONTH_VALID | DAY_VALID
Then, if you have a date like 2010-00-00, convert that to 2010-01-01 and set the flag to Y_VALID. If you have a date like 2010-06-00, convert that to 2010-06-01 and set the flag to YM_VALID.
So, then, PartialDateField would be a class that bundles together a date and the date-valid flag described above.
P.S. You don't actually need to use the flags the way I showed it; that's the old C programmer in me coming to the surface. You could use Y_VALID, YM_VALID, YMD_VALID = range(3) and it would work about as well. The key is to have some kind of flag that tells you how much of the date to trust.
Although not in Python - here's an example of how the same problem was solved in Ruby - using a single Integer value - and bitwise operators to store year, month and day - with month and day optional.
https://github.com/58bits/partial-date
Look at the source in lib for date.rb and bits.rb.
I'm sure a similar solution could be written in Python.
To persist the date (sortable) you just save the Integer to the database.