We Keep Coding

Django - regex optional function parameters in URL

I am starting off with Django now (already have quite a bit of Python knowledge, as well as with other languages). I am wondering whether it is possible to pass optional parameters through the url to a view (function that is called when a certain url is entered). What I have:
url(regex=r'^bydate/year=(?P<year>[0-9]+)_month=(?P<month>[0-9]+)_day=(?P<day>[0-9]+)/$', view=views.question_by_date, name='question_by_date')
So, in other words, if the end of the url looks like this, for example:
...bydate/year=2001_month=11_day=2/
then it calls the question_by_date function, whose signature looks as follows:
question_by_date(request, **kwargs)
So with the above url, question_by_date will be called as
question_by_date(request, year=2001, month=11, day=2)
But I also want the user to be able to type in the url specifying just the year, e.g.
...bydate/year=2005/
which will call
question_by_date(request, year=2005)
Or for that matter, any combination of year, month, day (like just the year and the month, or just the year and day even, etc.)
So, is this possible? I am not so experienced in regex, and I understand that you can have optional string matches (zero or more) in regex, which will match the above just fine in normal circumstances, but here we are also passing (optional) parameters to a function.
NOTE:
A very similar question to this has already been asked here. I realize that I could make a different URL for each combination, but that would entail making 8 different URLs. Also, that question was asked 6 years ago. Hopefully some enhancement has been made in the meantime?

I think what you need is GET parameters for this:
Define you url without any parameters:
url(r'^bydate/$', views.question_by_date, name='question-by-date')
In your views, extract GET parameters:
from datetime import date
def question_by_date(request):
year = request.GET.get('year', 2005)
month = request.GET.get('month', 1)
day = request.GET.get('day', 1)
# use the parameters however you want afterwards
Call your url like:
http://localhost:8000/bydate/?year=2016&month=1&day=1
Check django doc for more details about http GET.

How to change the date dynamically using a command in linux

I am writing a python code to change the date in linux system to today-1 (dynamically). I tried various combinations but, yet I am not able to succeed. I searched and I found a close proximity to my scenario in this question .
I am able to change the date if I execute the command with static value say:
date --set="$(date +'2013%m%d %H:%M')"
However, I don't want to specify hardcoded value for year i.e., 2013. Instead i want to specify something like "%y-1" i.e.,
date --set="$(date +'%y-1%m%d %H:%M')"
If I run the above command I get the following error
[root#ramesh ~]$ date --set="$(date +'%y-1%m%d %H:%M')"
date: invalid date `14-11016 13:05'

Thanks for your answer. I did not try your approach though, reason being it has to be once again dealt with formatting issues when working with arithmetic operations incase if you want to.
So, I figured out a much simpler and generalized approach
Fetch the previous_year value with this command
date --date='1 years ago'
This gives the previous year date. Now this can be used in the python program to update the system in the following way
"date --set=$(date +'%%y%%m%s %%H:%%M') % previous_year"
This method has few advantages like
I can apply this method for day and month as well like "1 days ago", "1 month ago" along with +%d, +%m, +%y values.
e.g., date --date='1 years ago' +%y
I don't have to worry about the date and month arithmetic calculation logics

date will interpret the %y-1 literally has you showed. What you need is to retrieve the current year, subtract 1 and use this value as the new year.
To get the current_year - 1 you can do:
previous_year=$((`date +'%y'`-1))
echo $previous_year
>>> 13
Now you just need to use this variable to set the new date.

Specifying limit and offset in Django QuerySet wont work

I'm using Django 1.6.5 and have MySQL's general-query-log on, so I can see the sql hitting MySQL.
And I noticed that Specifying a bigger limit in Django's QuerySet would not work:
>>> from blog.models import Author
>>> len(Author.objects.filter(pk__gt=0)[0:999])
>>> len(Author.objects.all()[0:999])
And MySQL's general log showed that both query had LIMIT 21.
But a limit smaller than 21 would work, e.g. len(Author.objects.all()[0:10]) would make a sql with LIMIT 10.
Why is that? Is there something I need to configure?

It happens when you make queries from the shell - the LIMIT clause is added to stop your terminal filling up with thousands of records when debugging:
You were printing (or, at least, trying to print) the repr() of the
queryset. To avoid people accidentally trying to retrieve and print a
million results, we (well, I) changed that to only retrieve and print
the first 20 results and print "remainder truncated" if there were more.
This is achieved by limiting the query to 21 results (if there are 21
results there are more than 20, so we print the "truncated" message).
That only happens in the repr() -- i.e. it's only for diagnostic
printing. No normal user code has this limit included automatically, so
you happily create a queryset that iterates over a million results.
(Source)

Django implements OFFSET using Python’s array-slicing syntax. If you want to offset the first 10 elements and then show the next 5 elements then use it
MyModel.objects.all()[OFFSET:OFFSET+LIMIT]
For example if you wanted to check 5 authors after an offset of 10 then your code would look something like this:
Author.objects.all()[10:15]
You can read more about it here in the official Django doc
I have also written a blog around this concept, you can here more here

The LIMIT and OFFSET doesn't work in the same way in Django, the way we expect it to work.
For example.
If we have to read next 10 rows starting from 10th row and if we specify :
Author.objects.all()[10:10]
It will return the empty record list. In order to fetch the next 10 rows, we have to add the offset to the limit.
Author.objects.all()[10:10+10]
And it will return the record list of next 10 rows starting from the 10th row.

for offset and limit i used and worked for me :)
MyModel.objects.all()[offset:limit]
for exapmle:-
Post.objects.filter(Post_type=typeId)[1:1]

I does work, but django uses an iterator. It does not load all objects at once.

How to define multiple optional variables in the URL?

In django doc proposed instead GET method use urlpatterns, and made ​​convenient way to handle these variables. But if at least one of the variables is not necessary I'll have to write more lines in url.py. I like that I can avoid this?
Example:
If I want to take a sample of posts in a given year, in urlpatterns I should add something like this:
url(r'^articles/(?P<year>\d{4})/$', 'news.views.show_archive'),
url: .../articles/1994/
If I want to make the sample positions for a particular month a specific year, in urlpatterns I should add something like this:
url(r'^articles/(?P<year>\d{4})/(?P<month>\d{2})/$', 'news.views.show_archive'),
url: .../articles/2003/03/
But if I want to see the records of all the years created particular month of year I have to add also this line:
url(r'^articles/(?P<month>\d{2})/$', 'news.views.show_archive'),
url: .../articles/03/
But I would like to do only one line that specifies the maximum set of variables, but that would process any of these URL.
To be honest I'm not sure that this is possible.

regexps can have optional parts, and view functions can have optional arguments. Also, you can still use querystrings (through request.GET) for what has no business being part of the URL (like query terms for a "search" view, ordering and filtering for a listing view, etc).
The point of using urlpatterns instead of querystrings is to build clean "semantic" urls, ie /blog/posts/<post_id>/ instead of /blog/posts/?post_id=<post_id>.

you could try like this
url(r'^articles/(?P<year>\d{4})/(?P<month>\d{2})/$', 'news.views.show_archive'),
def show_archive(request,year=None,month=None):
if year and month:
.....................
elif year:
.....................
elif month:
....................

Irregular String Parsing on Python

I'm new to python/django and I am trying to suss out more effective information from my scraper. Currently, the scraper takes a list of comic book titles and correctly divides them into a CSV list in three parts (Published Date, Original Date, and Title). I then pass the current date and title through to different parts of my databse, which I do in my Loader script (convert mm/dd/yy into yyyy-mm-dd, save to "pub_date" column, title goes to "title" column).
A common string can look like this:
10/12/11|10/12/11|Stan Lee's Traveler #12 (10 Copy Incentive Cover)
I am successfully grabbing the date, but the title is trickier. In this instance, I'd ideally like to fill three different columns with the information after the second "|". The Title should go to "title", a charfield. the number 12 (after the '#') should go into the DecimalField "issue_num", and everything between the '()' 's should go into the "Special" charfield. I am not sure how to do this kind of rigorous parsing.
Sometimes, there are multiple #'s (one comic in particular is described as a bundle, "Containing issues #90-#95") and several have multiple '()' groups (such as, "Betrayal Of The Planet Of The Apes #1 (Of 4)(25 Copy Incentive Cover)
)
What would be a good road to start onto crack this problem? My knowledge of If/else statements quickly fell apart for the more complicated lines. How can I efficiently and (if possible) pythonic-ly parse through these lines and subdivide them so I can later slot them into the correct place in my database?

Use the regular expression module re. For example, if you have the third |-delimited field of your sample record in a variable s, then you can do
match = re.match(r"^(?P<title>[^#]*) #(?P<num>[0-9]+) \((?P<special>.*)\)$", s)
title = match.groups('title')
issue = match.groups('num')
special = match.groups('special')
You'll get an IndexError in the last three lines for a missing field. Adapt the RE until it parses everything your want.

Parsing the title is the hard part, it sounds like you can handle the dates etc yourself. The problem is that there is not one rule that can parse every title but there are many rules and you can only guess which one works on a particular title.
I usually handle this by creating a list of rules, from most specific to general and try them out one by one until one matches.
To write such rules you can use the re module or even pyparsing.
The general idea goes like this:
class CantParse(Exception):
pass
# one rule to parse one kind of title
import re
def title_with_special( title ):
""" accepts only a title of the form
<text> #<issue> (<special>) """
m = re.match(r"[^#]*#(\d+) \(([^)]+)\)", title)
if m:
return m.group(1), m.group(2)
else:
raise CantParse(title)
def parse_extra(title, rules):
""" tries to parse extra information from a title using the rules """
for rule in rules:
try:
return rule(title)
except CantParse:
pass
# nothing matched
raise CantParse(title)
# lets try this out
rules = [title_with_special] # list of rules to apply, add more functions here
titles = ["Stan Lee's Traveler #12 (10 Copy Incentive Cover)",
"Betrayal Of The Planet Of The Apes #1 (Of 4)(25 Copy Incentive Cover) )"]
for title in titles:
try:
issue, special = parse_extra(title, rules)
print "Parsed", title, "to issue=%s special='%s'" % (issue, special)
except CantParse:
print "No matching rule for", title
As you can see the first title is parsed correctly, but not the 2nd. You'll have to write a bunch of rules that account for every possible title format in your data.

Regular expression is the way to go. But if you fill uncomfortably writing them, you can try a small parser that I wrote (https://github.com/hgrecco/stringparser). It translates a string format (PEP 3101) to a regular expression. In your case, you will do the following:
>>> from stringparser import Parser
>>> p = Parser(r"{date:s}\|{date2:s}\|{title:s}#{issue:d} \({special:s}\)")
>>> x = p("10/12/11|10/12/11|Stan Lee's Traveler #12 (10 Copy Incentive Cover)")
OrderedDict([('date', '10/12/11'), ('date2', '10/12/11'), ('title', "Stan Lee's Traveler "), ('issue', 12), ('special', '10 Copy Incentive Cover')])
>>> x.issue
12
The output in this case is an (ordered) dictionary. This will work for any simple cases and you might tweak it to catch multiple issues or multiple ()
One more thing: notice that in the current version you need to manually escape regex characters (i.e. if you want to find |, you need to type \|). I am planning to change this soon.