We Keep Coding

How to get readable unicode string from single bibtex entry field in python script

Suppose you have a .bib file containing bibtex-formatted entries. I want to extract the "title" field from an entry, and then format it to a readable unicode string.
For example, if the entry was:
#article{mypaper,
author = {myself},
title = {A very nice {title} with annoying {symbols} like {\^{a}}}
}
what I want to extract is the string:
A very nice title with annoying symbols like â
I am currently trying to use the pybtex package, but I cannot figure out how to do it. The command-line utility pybtex-format does a good job in converting full .bib files, but I need to do this inside a script and for single title entries.

Figured it out:
def load_bib(filename):
from pybtex.database.input.bibtex import Parser
parser = Parser()
DB = parser.parse_file(filename)
return DB
def get_title(entry):
from pybtex.plugin import find_plugin
style = find_plugin('pybtex.style.formatting', 'plain')()
backend = find_plugin('pybtex.backends', 'plaintext')()
sentence = style.format_title(entry, 'title')
data = {'entry': entry,
'style': style,
'bib_data': None}
T = sentence.f(sentence.children, data)
title = T.render(backend)
return title
DB = load_bib("bibliography.bib")
print ( get_title(DB.entries["entry_label"]) )
where entry_label must match the label you use in latex to cite the bibliography entry.

Building upon the answer by Daniele, I wrote this function that lets one render fields without having to use a file.
from io import StringIO
from pybtex.database.input.bibtex import Parser
from pybtex.plugin import find_plugin
def render_fields(author="", title=""):
"""The arguments are in bibtex format. For example, they may contain
things like \'{i}. The output is a dictionary with these fields
rendered in plain text.
If you run tests by defining a string in Python, use r'''string''' to
avoid issues with escape characters.
"""
parser = Parser()
istr = r'''
#article{foo,
Author = {''' + author + r'''},
Title = {''' + title + '''},
}
'''
bib_data = parser.parse_stream(StringIO(istr))
style = find_plugin('pybtex.style.formatting', 'plain')()
backend = find_plugin('pybtex.backends', 'plaintext')()
entry = bib_data.entries["foo"]
data = {'entry': entry, 'style': style, 'bib_data': None}
sentence = style.format_author_or_editor(entry)
T = sentence.f(sentence.children, data)
rendered_author = T.render(backend)[0:-1] # exclude period
sentence = style.format_title(entry, 'title')
T = sentence.f(sentence.children, data)
rendered_title = T.render(backend)[0:-1] # exclude period
return {'title': rendered_title, 'author': rendered_author}

Python Praw ways to store data for calling later?

Is a dictionary the correct way to be doing this? Ideally this will be more then 5+ deep. Sorry my only language experience is powershell there I would just make an array of object. Im not looking for someone to write the code I just wanna know if there is a better way?
Thanks
Cody
My Powershell way:
[$title1,$title2,$title3]
$titleX.comment = "comment here"
$titleX.comment.author = "bob"
$titleX.comment.author.karma = "200"
$titleX.comment.reply = "Hey Bob love your comment."
$titleX.comment.reply.author = "Alex"
$titleX.comment.reply.reply = "I disagree"
#
Python code Borken:
import praw
d = {}
reddit = praw.Reddit(client_id='XXXX',
client_secret='XXXX',
user_agent='android:com.example.myredditapp:'
'v1.2.3 (by /u/XXX)')
for submission in reddit.subreddit('redditdev').hot(limit=2):
d[submission.id] = {}
d[submission.id]['comment'] = {}
d[submission.id]['title']= {}
d[submission.id]['comment']['author']={}
d[submission.id]['title'] = submission.title
mySubmission = reddit.submission(id=submission.id)
mySubmission.comments.replace_more(limit=0)
for comment in mySubmission.comments.list():
d[submission.id]['comment'] = comment.body
d[submission.id]['comment']['author'] = comment.author.name
print(submission.title)
print(comment.body)
print(comment.author.name)
print(d)
File "C:/git/tensorflow/Reddit/pull.py", line 23, in <module>
d[submission.id]['comment']['author'] = comment.author.name
TypeError: 'str' object does not support item assignment
#
{'6xg24v': {'comment': 'Locking this version. Please comment on the [original post](https://www.reddit.com/r/changelog/comments/6xfyfg/an_update_on_the_state_of_the_redditreddit_and/)!', 'title': 'An update on the state of the reddit/reddit and reddit/reddit-mobile repositories'}}

I think your approach using a dictionary is okay, but you might also solve this by using a data structure for your posts: Instead of writing
d[submission.id] = {}
d[submission.id]['comment'] = {}
d[submission.id]['title']= {}
d[submission.id]['comment']['author']={}
d[submission.id]['title'] = submission.title
you could create a class Submission like this:
class Submission(object):
def __init__(self, id, author, title, content):
self.id = id
self.author = author
self.title = title
self.content = content
self.subSubmissions = {}
def addSubSubmission(self,submission):
self.subSubmission[submission,id] = submission
def getSubSubmission(self,id):
return self.subSubmission[id]
by using you could change your code to this
submissions = {}
for sm in reddit.subreddit('redditdev').hot(limit=2):
submissions[sm.id] = Submission(sm.id, sm.author, sm.title, sm.content)
# I am not quite sure what these lines are supposed to do, so you might be able to improve these, too
mySubmission = reddit.submission(id=sm.id)
mySubmission.comments.replace_more(limit=0)
for cmt in mySubmission.comments.list():
submissions[sm.id].addSubSubmission(Submission(cmt.id, cmt.title, cmt.author, cmt.body))
By using this apporach you are also able to export the code to readout the comments/subSubmissions into an extra function which can call itself recursively, so that you can read infitive depths of the comments.

Django ORM, how to use values() and still work with choicefield?

I am using django v1.10.2
I am trying to create dynamic reports whereby I store fields and conditions and the main ORM model information into database.
My code for the generation of the dynamic report is
class_object = class_for_name("app.models", main_model_name)
results = (class_object.objects.filter(**conditions_dict)
.values(*display_columns)
.order_by(*sort_columns)
[:50])
So main_model_name can be anything.
This works great except that sometimes associated models of the main_model have choicefield.
So for one of the reports main_model is Pallet.
Pallet has many PalletMovement.
My display columns are :serial_number, created_at, pallet_movement__location
The first two columns are fields that belong to Pallet model.
The last one is from PalletMovement
What happens is that PalletMovement model looks like this:
class PalletMovement(models.Model):
pallet = models.ForeignKey(Pallet, related_name='pallet_movements',
verbose_name=_('Pallet'))
WAREHOUSE_CHOICES = (
('AB', 'AB-Delaware'),
('CD', 'CD-Delaware'),
)
location = models.CharField(choices=WAREHOUSE_CHOICES,
max_length=2,
default='AB',
verbose_name=_('Warehouse Location'))
Since the queryset will return me the raw values, how can I make use of the choicefield in PalletMovement model to ensure that the pallet_movement__location gives me the display of AB-Delaware or CD-Delaware?
Bear in mind that the main_model can be anything depending on what I store in the database.
Presumably, I can store more information in the database to help me do the filtering and presentation of data even better.

The values() method returns a dictionary of key-value pairs representing your field name and a corresponding value.
For example:
Model:
class MyModel(models.Model):
name = models.CharField()
surname = models.CharField()
age = models.IntegerField()
...
Query:
result = MyModel.objects.filter(surname='moutafis').values('name', 'surname')
Result:
< Queryset [{'name': 'moutafis', 'surname': 'john'}] >
You can now manipulate this result as you would a normal dictionary:
if main_model_name is 'PalletMovement':
# Make life easier
choices = dict(PalletMovement.WAREHOUSE_CHOICES)
for item in result:
item.update({
pallet_movement__location: verbal_choice.get(
pallet_movement__location, pallet_movement__location)
})
You can even make this into a function for better re-usability:
def verbalize_choices(choices_dict, queryset, search_key):
result = queryset
for item in result:
item.update({ search_key: choices_dict.get(search_key, search_key) })
return result
verbal_result = verbalize_choices(
dict(PalletMovement.WAREHOUSE_CHOICES),
result,
'pallet_movement__location'
)
I suggest the use of the update() and get() methods because they will save you from potential errors, like:
The search_key does not exist in the choice_dict then get() will return the value of the search_key
update() will try to update the given key-value pair if exists, else it will add it to the dictionary.
If the usage of the above will be in the template representation of your data, you can create a custom template filter instead:
#register.filter(name='verbalize_choice')
def choice_to_verbal(choice):
return dict(PalletMovement.WAREHOUSE_CHOICES)[choice]
Have an extra look here: Django: How to access the display value of a ChoiceField in template given the actual value and the choices?

You would use get_foo_display
In your template:
{{ obj.get_location_display }}
or
{{ obj.pallet_movement.get_location_display }}
[Edit:] As pointed out in the comments this will not work when calling values()

an alternative to create a templatetag is :
{{form.choicefield.1}}
This shows the value of the initial data of the foreign key field instead the id.

The universal solution for any main_model_name is by Django Model _meta API introspection: class_object._meta.get_field(field_name).choices
That is:
choice_dicts = {}
for field_name in display_columns:
choice_dicts[field_name] = {
k: v for k, v in class_object._meta.get_field(field_name).choices
}
out = []
for row in results:
out.append({name: choice_dicts[name].get(value, value)
for name, value in row.items()
})
The rest is a trivial example, mostly copied code from the question
>>> pallet = app.models.Pallet.objects.create()
>>> palletm = app.models.PalletMovement.objects.create(pallet=pallet, location='AB')
>>>
>>> main_model_name = 'PalletMovement'
>>> conditions_dict = {}
>>> display_columns = ['pallet_id', 'location']
>>> sort_columns = []
>>>
>>> class_object = class_for_name("app.models", main_model_name)
>>> results = (class_object.objects.filter(**conditions_dict)
... .values(*display_columns)
... .order_by(*sort_columns)
... )[:50]
>>>
>>> # *** INSERT HERE ALL CODE THAT WAS ABOVE ***
>>>
>>> print(out)
[{'location': 'AB-Delaware', 'pallet_id': 1}]
It works equally with 'pallet_id' or with 'pallet' in display_columns. Even that "_meta" starts with underscore, it is a documented API.

Jinja2 Extension Tag within another Tag

I'm tring to do something like this
{% mytag country "italy" year "2014" %}
workday
{% holyday %}
not workday
{% endmytag %}
But the holyday tag is optional.
This must work too:
{% mytag country "italy" year "2014" %}
workday
{% endmytag %}
I wrote the code
class MytagExtension(Extension):
tags = set(['mytag'])
def __init__(self, environment):
super(TranslateExtension, self).__init__(environment)
def parse(self, parser):
lineno = parser.stream.next().lineno
if parser.stream.skip_if('name:country'):
country= parser.parse_primary()
else:
country= nodes.Const(None)
if parser.stream.skip_if('name:year'):
year = parser.parse_primary()
else:
year = nodes.Const(None)
args = [country, year]
# body1 = parser.parse_statements(['name:holyday']) # not working :)
body = parser.parse_statements(['name:endmytag'], drop_needle=True)
return nodes.CallBlock(self.call_method('_helper', args),
[], [], body).set_lineno(lineno)
def _helper(self, country, year, caller):
etc ....
Is similar to a if else endif, but i didn't find the source code of the if tag (if it exists as Extension)
How can i do this?
Obviously in my _helper I need both the first and second branch because is there that I choose which one to show.

ok, here is an answer, it's not perfect and probably not the best, but it's something.
...
body = parser.parse_statements(
['name:endmytag', 'name:holyday'],
drop_needle=True
)
if not parser.stream.current.test('block_end'):
body.extend ( parser.parse_statements(['name:endmytag'], drop_needle=True) )
args = [
country,
year,
nodes.Const([y.data for x in body for y in x.nodes]), #I don't like this line a lot :)
]
return nodes.CallBlock(self.call_method('_helper', args),
[], [], body).set_lineno(lineno)
...
In this way in the _helper you will receive the third parameter (a list) and choose if return the first or the second element of this list.

archive list with django templatetags

I have a django application, a blog. The entries for the blog can be accessed through a /year/month/day/slug url pattern, it works fine.
My problem is, I want to have an archive list accessible to any template on my website. So i thought the best solution would be to create a templatetag that would create and return the info i needed.
I wanted the format of the archive to be as such:
August 2011
July 2011
etc..
2010
2009
2008
etc..
So only show months for the current year.
This is the tag i came up with:
from django.template import Library, Node, TemplateSyntaxError
from core.blog.models import Entry
import datetime, calendar
register = Library()
class ArchiveNode(Node):
def __init__(self, varname):
self.varname = varname
def render(self, context):
temp = list()
#Get Info about the first post
first = Entry.objects.order_by("pub_date")[0]
first_year = first.pub_date.year
first_month = first.pub_date.month
#Loop over years and months since first post was created
today = datetime.datetime.today()
this_year = today.year
this_month = today.month
for year in range(this_year - first_year):
if year != this_year:
temp += (year,'/blog/'+year+'/')
else:
for month in range(this_month - first_month):
month_name = calendar.month_name[month]
temp += (month_name+" "+year,'/blog/'+year+'/'+month+'/')
context[self.varname] = temp.reverse()
return ''
#register.tag
def get_archive(parser, token):
bits = token.contents.split()
if len(bits) != 3:
raise TemplateSyntaxError, "get_archive tag takes exactly 1 argument"
if bits[1] != 'as':
raise TemplateSyntaxError, "second argument to get_archive tag must be 'as'"
return ArchiveNode(bits[2])
As you can see im returning a list of tuples, containing a name and a url.
Would this be valid in django? or do i need to pack the information in some django container? (It's doen't seem to return anything)
This is the site im working on ctrl-dev.com/blog.
The archive will be in the green box on the lower right.

There is no need to return something special. Django is just Python, so it is your choice what you want to return. In this case I would recommend to return dictionary like this (just inventing) {{'title':'some title if you want','year': 'year if you want', 'url': url}, {...}, ...}. Then in template you just run through like:
{% for entry in returned_dict %}
{{ entry.title }}
{% endfor %}
Also I would recommend to not hardcode the link in to the code. Read https://docs.djangoproject.com/en/dev/topics/http/urls/ about url resolvers, then https://docs.djangoproject.com/en/dev/ref/templates/builtins/?from=olddocs#url about {% url %} template tag. You can name the urls and later you can get the urls to the stuff you want without hardcoding it in the code. This will help in future ;)
Hope that helped.