How to cycle through a defined list of elements - python

I have a large dictionary of lists which is generated dynamically as part of a larger python program. This is an example of what the dictionary looks like
params = {
'output': ['quantity_tot_1m', 'quantity_tot_3m', 'quantity_tot_6m',
'quantity_tot_12m', 'quantity_tot_full', 'cost_tot_1m', 'cost_tot_3m', 'cost_tot_6m', 'cost_tot_12m', 'cost_tot_full', 'selling_tot_1m',
'selling_tot_3m', 'selling_tot_6m', 'selling_tot_12m', 'selling_tot_full', 'profit_tot_1m', 'profit_tot_3m', 'profit_tot_6m', 'profit_tot_12m', 'profit_tot_full'],
'agg_types': ['By SKU - 01 Month','By SKU - 03 Months','By SKU - 06 Months','By SKU - 12 Months']
}
I need assistance being able to cycle over these lists
I have tried the below
from jinjasql import JinjaSql
j = JinjaSql(param_style='pyformat')
QUERY_TEMPLATE = """
{% set mix_agg = agg_types|replace("[","")|replace("]","") %}
{% set mix_agg2 = cycler(mix_agg) %}
{% for o in output %}
ROUND(CAST(SUM(CASE WHEN agg_type = '{{ mix_agg2.next() | sqlsafe }}' THEN profit_tot ELSE 0 END ) AS NUMERIC),2) AS {{o | sqlsafe}} {% endfor%}
"""
query, bind_params = j.prepare_query(QUERY_TEMPLATE, params)
print(query)
This doesn't cycle over the list, it merely outputs the list as is. Of course I cannot explicitly type the list into the cycler because this has to be dynamic.
Perhaps a cycler() isn't the best approach but I need an output like the below:
ROUND(CAST(SUM(CASE WHEN agg_type = 'By SKU - 01 Month' THEN profit_tot ELSE 0 END ) AS NUMERIC),2) AS profit_tot_1m,
ROUND(CAST(SUM(CASE WHEN agg_type = 'By SKU - 03 Months' THEN profit_tot ELSE 0 END ) AS NUMERIC),2) AS profit_tot_3m,
ROUND(CAST(SUM(CASE WHEN agg_type = 'By SKU - 06 Months' THEN profit_tot ELSE 0 END ) AS NUMERIC),2) AS profit_tot_6m,
ROUND(CAST(SUM(CASE WHEN agg_type = 'By SKU - 12 Months' THEN profit_tot ELSE 0 END ) AS NUMERIC),2) AS profit_tot_12m
not like this:
ROUND(CAST(SUM(CASE WHEN agg_type = 'By SKU - 01 Month', 'By SKU - 03 Months', 'By SKU - 06 Months', 'By SKU - 12 Months' THEN profit_tot ELSE 0 END ) AS NUMERIC),2) AS profit_tot_1m,
ROUND(CAST(SUM(CASE WHEN agg_type = 'By SKU - 01 Month', 'By SKU - 03 Months', 'By SKU - 06 Months', 'By SKU - 12 Months' THEN profit_tot ELSE 0 END ) AS NUMERIC),2) AS profit_tot_3m,
ROUND(CAST(SUM(CASE WHEN agg_type = 'By SKU - 01 Month', 'By SKU - 03 Months', 'By SKU - 06 Months', 'By SKU - 12 Months' THEN profit_tot ELSE 0 END ) AS NUMERIC),2) AS profit_tot_6m,
ROUND(CAST(SUM(CASE WHEN agg_type = 'By SKU - 01 Month', 'By SKU - 03 Months', 'By SKU - 06 Months', 'By SKU - 12 Months' THEN profit_tot ELSE 0 END ) AS NUMERIC),2) AS profit_tot_12m
There are a total of 20 output columns {{o | sqlsafe}}


The line
{% set mix_agg = agg_types|replace("[","")|replace("]","") %}
turns what was an array type into a string which just makes it harder to use later, and is probably not what you want.
The line
{% set mix_agg2 = cycler(mix_agg) %}
then creates a cycler with a single sting to cycle through - so it will repeatadly return this single string.
To cycle through an array use the python * operator - which turns an array into function arguments:
{% set mix_agg2 = cycler(*agg_types) %}
Putting this together we have:
QUERY_TEMPLATE = """
{% set mix_agg2 = cycler(*agg_types) %}
{% for o in output %}
ROUND(CAST(SUM(CASE WHEN agg_type = '{{ mix_agg2.next() | sqlsafe }}' THEN profit_tot ELSE 0 END ) AS NUMERIC),2) AS {{o | sqlsafe}}
{% endfor %}
"""
Furthermore: output includes _tot_full which is missing from agg_types. You can fix this by adding it to agg_types or removing it from output when they are defined, or adding a if to skip it in the loop.

Related

python pulp not matching value of variable dict

I am struggling to simplify the constraint for a few days now. still learning python. please help and thank you in advance.
Employees=['Paul', 'Ben', 'Nasim', 'Ceci', 'Victoria',]
Days = ['Monday', 'Tuesday', 'Wednesday', 'Thursday', 'Friday',
'Saturday', 'Sunday']
avail = pulp.LpVariable.dicts("on_off", ((employee, day) for
employee in Employees for day in Days), cat="Binary")
requests={"Paul": {"Monday":1, "Tuesday":1, "Wednesday":1,
"Thursday":1, "Friday":1, "Saturday":1, "Sunday":1},
"Ben": {"Monday":1, "Tuesday":1, "Wednesday":1, "Thursday":1,
"Friday":1, "Saturday":1, "Sunday":0},
"Nasim": {"Monday":1, "Tuesday":1, "Wednesday":1, "Thursday":1,
"Friday":1, "Saturday":1, "Sunday":1},
"Ceci": {"Monday":1, "Tuesday":1, "Wednesday":1, "Thursday":1,
"Friday":1,"Saturday":1, "Sunday":0},
"Victoria": {"Monday":1, "Tuesday":1, "Wednesday":0, "Thursday":0,
"Friday":0, "Saturday":0, "Sunday":0}}
for employee, day in avail:
prob += avail[employee, day] == [requests[i][j] for i in
requests for j in requests[i]]
as an example, the first few constraints:
"_C13: on_off_('Paul',_'Monday') = 28
_C14: on_off_('Paul',_'Tuesday') = 28
_C15: on_off_('Paul',_'Wednesday') = 28
_C16: on_off_('Paul',_'Thursday') = 28
_C17: on_off_('Paul',_'Friday') = 28
_C18: on_off_('Paul',_'Saturday') = 28"
("28" is just so happened to be the SUM of all the 1's and the 0's in my nested dict.)
Instead i like to have each variable matching 1's and 0's from my nested dict.
"_C13: on_off_('Paul',_'Monday') = 1
_C14: on_off_('Paul',_'Tuesday') = 1
_C15: on_off_('Paul',_'Wednesday') = 1
_C16: on_off_('Paul',_'Thursday') = 1
_C17: on_off_('Paul',_'Friday') = 1
_C18: on_off_('Paul',_'Saturday') = 1"

After the == sign you want the value 1 or 0 from the dictionary, therefore you should use:
for employee, day in avail:
prob += avail[employee, day] == requests[employee][day]
With requests[employee] you select the dictionary for that employee
requests['Paul'] = {"Monday":1, "Tuesday":1, "Wednesday":1,
"Thursday":1, "Friday":1, "Saturday":1, "Sunday":1}
Then by adding [day] you select the day from that dictionary:
requests['Paul']['Monday'] = 1

Exception/Error Handling in Python

I have a class that I created, at the end of the task I have to create two lists: one for tweets and the other for tweet labels.
After initiation, I want to load the tweets from a file and from another file their labels. After loading I want to check that each tweet is a json object and that it has no error, if it does, then I want to remove it and remove the associated label if provided. Labels are either 'pos' or 'neg'.
class flu_tweets:
def __init__(self):
self.tweets = [] #init create empty tweet list
self.labels = [] #init create empty label list
def load(self, tweets_filename, labels_filename = ''):
open_tweet_file = open(tweets_filename, 'r')
for tweet in open_tweet_file:
if tweet !='\n' and tweet != '\r\n':
self.tweets.extend([tweet])
if labels_filename != '':
open_label_file = open(labels_filename, 'r')
for label in open_label_file:
if label != '\n' and label != '\r\n':
if label[3:] != '\n':
self.labels.extend([label[:3]])
else:
self.labels.extend([label])
open_label_file.close()
index_tweet = 0
for tweet in self.tweets:
try:
json.loads(tweet)
index_tweet += 1
break
except:
print(index_tweet-1)
self.tweets.pop(index_tweet-1)
if self.labels != []:
self.labels.pop(index_tweet-1)
open_tweet_file.close()
Right now the method doesn't do that, and upon checking the list it does contain non-json objects.
Below is a copy of text file used that has tweets in it:
{"created_at":"Fri Oct 20 14:35:19 +0000 2017","id":921384339421745153,"id_str":"921384339421745153","text":"RT #alvindchipmunk: Dont let the DNC slide with no handcuffs. https://t.co/h72q7lGAHF","source":"\u003ca href=\"http://twitter.com/download/iphone\" rel=\"nofollow\"\u003eTwitter for iPhone\u003c/a\u003e","truncated":false,"in_reply_to_status_id":null,"in_reply_to_status_id_str":null,"in_reply_to_user_id":null,"in_reply_to_user_id_str":null,"in_reply_to_screen_name":null,"user":{"id":3435638633,"id_str":"3435638633","name":"alwaystrump","screen_name":"rodilosso_patty","location":"New Jersey, USA","url":null,"description":"Let's not give the media the Race war they want. POTUS we have your back! MAGA","translator_type":"none","protected":false,"verified":false,"followers_count":2770,"friends_count":2048,"listed_count":183,"favourites_count":85745,"statuses_count":152520,"created_at":"Sat Aug 22 16:27:18 +0000 2015","utc_offset":null,"time_zone":null,"geo_enabled":true,"lang":"en","contributors_enabled":false,"is_translator":false,"profile_background_color":"C0DEED","profile_background_image_url":"http://abs.twimg.com/images/themes/theme1/bg.png","profile_background_image_url_https":"https://abs.twimg.com/images/themes/theme1/bg.png","profile_background_tile":false,"profile_link_color":"1DA1F2","profile_sidebar_border_color":"C0DEED","profile_sidebar_fill_color":"DDEEF6","profile_text_color":"333333","profile_use_background_image":true,"profile_image_url":"http://pbs.twimg.com/profile_images/773719268026257410/AuXU_l-D_normal.jpg","profile_image_url_https":"https://pbs.twimg.com/profile_images/773719268026257410/AuXU_l-D_normal.jpg","profile_banner_url":"https://pbs.twimg.com/profile_banners/3435638633/1461499638","default_profile":true,"default_profile_image":false,"following":null,"follow_request_sent":null,"notifications":null},"geo":null,"coordinates":null,"place":null,"contributors":null,"retweeted_status":{"created_at":"Fri Oct 20 12:27:04 +0000 2017","id":921352064160034816,"id_str":"921352064160034816","text":"Dont let the DNC slide with no handcuffs. https://t.co/h72q7lGAHF","display_text_range":[0,41],"source":"\u003ca href=\"http://twitter.com\" rel=\"nofollow\"\u003eTwitter Web Client\u003c/a\u003e","truncated":false,"in_reply_to_status_id":null,"in_reply_to_status_id_str":null,"in_reply_to_user_id":null,"in_reply_to_user_id_str":null,"in_reply_to_screen_name":null,"user":{"id":35962023,"id_str":"35962023","name":"alvin maldonado","screen_name":"alvindchipmunk","location":"Bunnell, FL","url":"http://alvindchipmunk-theconservativecomet.blogspot.com/","description":"Artist-musician-Medical Professional-Patriot-Guns-God-Country & 2Unite with others of like mind 2 re-elect Trump, redecorate DC making d USA gr8 as it still is","translator_type":"none","protected":false,"verified":false,"followers_count":1660,"friends_count":2051,"listed_count":43,"favourites_count":2001,"statuses_count":16294,"created_at":"Tue Apr 28 02:43:18 +0000 2009","utc_offset":-14400,"time_zone":"Eastern Time (US & Canada)","geo_enabled":true,"lang":"en","contributors_enabled":false,"is_translator":false,"profile_background_color":"9818A1","profile_background_image_url":"http://pbs.twimg.com/profile_background_images/627772771741732864/MHLgViA4.jpg","profile_background_image_url_https":"https://pbs.twimg.com/profile_background_images/627772771741732864/MHLgViA4.jpg","profile_background_tile":true,"profile_link_color":"981CEB","profile_sidebar_border_color":"DE3C88","profile_sidebar_fill_color":"E887E8","profile_text_color":"333333","profile_use_background_image":true,"profile_image_url":"http://pbs.twimg.com/profile_images/542852260422639616/75bqMWY3_normal.jpeg","profile_image_url_https":"https://pbs.twimg.com/profile_images/542852260422639616/75bqMWY3_normal.jpeg","profile_banner_url":"https://pbs.twimg.com/profile_banners/35962023/1481464103","default_profile":false,"default_profile_image":false,"following":null,"follow_request_sent":null,"notifications":null},"geo":null,"coordinates":null,"place":null,"contributors":null,"quoted_status_id":920847040132792320,"quoted_status_id_str":"920847040132792320","quoted_status":{"created_at":"Thu Oct 19 03:00:17 +0000 2017","id":920847040132792320,"id_str":"920847040132792320","text":"I'm sick of all the evidence against the Democrats and no handcuffs. Retweet-\nif you agree!\n\n#realDonaldTrump \ud83c\uddfa\ud83c\uddf8","source":"\u003ca href=\"http://twitter.com/download/android\" rel=\"nofollow\"\u003eTwitter for Android\u003c/a\u003e","truncated":false,"in_reply_to_status_id":null,"in_reply_to_status_id_str":null,"in_reply_to_user_id":null,"in_reply_to_user_id_str":null,"in_reply_to_screen_name":null,"user":{"id":889982846428782592,"id_str":"889982846428782592","name":"c\u2113\u03b9\u03b7\u0442\u03c3\u03b7 \u043c\u03b9c\u043d\u03b1\u03b5\u2113","screen_name":"crusher614","location":"*not of this world","url":"https://www.youtube.com/channel/UCthNh_qChVqAh_zamo0IQxQ","description":"FMR U.S. Border Patrol | New Mexico SWAT Operator | Legend Who Lives Rent Free In The Minds of Liberals World Wide. #MAGA \u03bc\u03bf\u03bb\u1f7c\u03bd \u03bb\u03b1\u03b2\u03ad III","translator_type":"none","protected":false,"verified":false,"followers_count":19821,"friends_count":135,"listed_count":73,"favourites_count":16032,"statuses_count":7708,"created_at":"Tue Jul 25 22:57:00 +0000 2017","utc_offset":-25200,"time_zone":"America/Phoenix","geo_enabled":false,"lang":"en","contributors_enabled":false,"is_translator":false,"profile_background_color":"000000","profile_background_image_url":"http://abs.twimg.com/images/themes/theme1/bg.png","profile_background_image_url_https":"https://abs.twimg.com/images/themes/theme1/bg.png","profile_background_tile":false,"profile_link_color":"19CF86","profile_sidebar_border_color":"000000","profile_sidebar_fill_color":"000000","profile_text_color":"000000","profile_use_background_image":false,"profile_image_url":"http://pbs.twimg.com/profile_images/916414292882219008/fvSIJCC6_normal.jpg","profile_image_url_https":"https://pbs.twimg.com/profile_images/916414292882219008/fvSIJCC6_normal.jpg","profile_banner_url":"https://pbs.twimg.com/profile_banners/889982846428782592/1506312325","default_profile":false,"default_profile_image":false,"following":null,"follow_request_sent":null,"notifications":null},"geo":null,"coordinates":null,"place":null,"contributors":null,"is_quote_status":false,"quote_count":551,"reply_count":703,"retweet_count":15439,"favorite_count":13676,"entities":{"hashtags":[],"urls":[],"user_mentions":[{"screen_name":"realDonaldTrump","name":"Donald J. Trump","id":25073877,"id_str":"25073877","indices":[93,109]}],"symbols":[]},"favorited":false,"retweeted":false,"filter_level":"low","lang":"en"},"is_quote_status":true,"quote_count":0,"reply_count":0,"retweet_count":1,"favorite_count":0,"entities":{"hashtags":[],"urls":[{"url":"https://t.co/h72q7lGAHF","expanded_url":"https://twitter.com/crusher614/status/920847040132792320","display_url":"twitter.com/crusher614/sta\u2026","indices":[42,65]}],"user_mentions":[],"symbols":[]},"favorited":false,"retweeted":false,"possibly_sensitive":false,"filter_level":"low","lang":"en"},"quoted_status_id":920847040132792320,"quoted_status_id_str":"920847040132792320","quoted_status":{"created_at":"Thu Oct 19 03:00:17 +0000 2017","id":920847040132792320,"id_str":"920847040132792320","text":"I'm sick of all the evidence against the Democrats and no handcuffs. Retweet-\nif you agree!\n\n#realDonaldTrump \ud83c\uddfa\ud83c\uddf8","source":"\u003ca href=\"http://twitter.com/download/android\" rel=\"nofollow\"\u003eTwitter for Android\u003c/a\u003e","truncated":false,"in_reply_to_status_id":null,"in_reply_to_status_id_str":null,"in_reply_to_user_id":null,"in_reply_to_user_id_str":null,"in_reply_to_screen_name":null,"user":{"id":889982846428782592,"id_str":"889982846428782592","name":"c\u2113\u03b9\u03b7\u0442\u03c3\u03b7 \u043c\u03b9c\u043d\u03b1\u03b5\u2113","screen_name":"crusher614","location":"*not of this world","url":"https://www.youtube.com/channel/UCthNh_qChVqAh_zamo0IQxQ","description":"FMR U.S. Border Patrol | New Mexico SWAT Operator | Legend Who Lives Rent Free In The Minds of Liberals World Wide. #MAGA \u03bc\u03bf\u03bb\u1f7c\u03bd \u03bb\u03b1\u03b2\u03ad III","translator_type":"none","protected":false,"verified":false,"followers_count":19821,"friends_count":135,"listed_count":73,"favourites_count":16032,"statuses_count":7708,"created_at":"Tue Jul 25 22:57:00 +0000 2017","utc_offset":-25200,"time_zone":"America/Phoenix","geo_enabled":false,"lang":"en","contributors_enabled":false,"is_translator":false,"profile_background_color":"000000","profile_background_image_url":"http://abs.twimg.com/images/themes/theme1/bg.png","profile_background_image_url_https":"https://abs.twimg.com/images/themes/theme1/bg.png","profile_background_tile":false,"profile_link_color":"19CF86","profile_sidebar_border_color":"000000","profile_sidebar_fill_color":"000000","profile_text_color":"000000","profile_use_background_image":false,"profile_image_url":"http://pbs.twimg.com/profile_images/916414292882219008/fvSIJCC6_normal.jpg","profile_image_url_https":"https://pbs.twimg.com/profile_images/916414292882219008/fvSIJCC6_normal.jpg","profile_banner_url":"https://pbs.twimg.com/profile_banners/889982846428782592/1506312325","default_profile":false,"default_profile_image":false,"following":null,"follow_request_sent":null,"notifications":null},"geo":null,"coordinates":null,"place":null,"contributors":null,"is_quote_status":false,"quote_count":551,"reply_count":703,"retweet_count":15439,"favorite_count":13676,"entities":{"hashtags":[],"urls":[],"user_mentions":[{"screen_name":"realDonaldTrump","name":"Donald J. Trump","id":25073877,"id_str":"25073877","indices":[93,109]}],"symbols":[]},"favorited":false,"retweeted":false,"filter_level":"low","lang":"en"},"is_quote_status":true,"quote_count":0,"reply_count":0,"retweet_count":0,"favorite_count":0,"entities":{"hashtags":[],"urls":[{"url":"https://t.co/h72q7lGAHF","expanded_url":"https://twitter.com/crusher614/status/920847040132792320","display_url":"twitter.com/crusher614/sta\u2026","indices":[62,85]}],"user_mentions":[{"screen_name":"alvindchipmunk","name":"alvin maldonado","id":35962023,"id_str":"35962023","indices":[3,18]}],"symbols":[]},"favorited":false,"retweeted":false,"possibly_sensitive":false,"filter_level":"low","lang":"en","timestamp_ms":"1508510119668"}
{"created_at":"Fri Oct 20 14:35:19 +0000 2017","id":921384340113670149,"id_str":"921384340113670149","text":"RT #mitchelmusso: My girl is so fine! but gaw}
Now that last tweet is incomplete,I expect that my function should raise an error and eliminate it.

python, django, sending data to a template

EDIT: Updated code, im trying to create a new nested dictionary from the orginal results.
however the dictionary is currently not updating, its only adding/editing the last value
so my current context just has greg on it and no one else
my current code is as below
# Create your views here.
def index(request):
### Get all the Polices ###
context = {}
for objPolicy in objPolicyData['escalation_policies']:
strPolicyName = objPolicy['name']
if strPolicyName.lower().find('test') == -1:
context['strPolicyName'] = strPolicyName
obj = {}
for objOnCall in objPolicy['on_call']:
obj['strLevel'] = objOnCall['level']
obj['strStartDate'] = getDate(objOnCall['start'])
obj['strStartTime'] = getTime(objOnCall['start'])
obj['strEndDate'] = getDate(objOnCall['end'])
obj['strEndTime'] = getTime(objOnCall['end'])
objUser = objOnCall['user']
obj['strUsername'] = objUser['name']
obj['strUserMobile'] = getUserMobile(objUser['id'])
context['objUsers'] = obj
return render(request, 'oncall/rota.html', context)
sample data would be
Network Policy
Level 1: John Smith
Start date: 27 April
Start time: 8am
end Date: 05 May
end time: 8am
Level 2: Bob Smith
Start date: 27 April
Start time: 8am
end Date: 05 May
end time: 8am
Server Policy
Level 1: Jane Doe
Start date: 23 April
Start time: 8am
end Date: 02 May
end time: 8am
Level 2: Greg Brad
Start date: 23 April
Start time: 8am
end Date: 02 May
end time: 8am
and so on...
Update:
#Alix, your current solution gives me the below, i think i need nested lists? as the level 2 engineer gets posted twice instead of level 1 and level 2, also missing the policy names for each one
#!/usr/bin/python
# -*- coding: utf-8 -*-
{'policies': [{
'strStartTime': '09:00AM',
'strEndTime': '09:00AM',
'strLevel': 2,
'strUserMobile': u'01234 5678',
'strEndDate': 'Monday 02 May',
'strUsername': u'John Smith',
'strStartDate': 'Monday 25 April',
}, {
'strStartTime': '09:00AM',
'strEndTime': '09:00AM',
'strLevel': 2,
'strUserMobile': u'01234 5678'',
'strEndDate': 'Monday 02 May',
'strUsername': u'John Smith',
'strStartDate': 'Monday 25 April',
}, {
'strStartTime': '09:00AM',
'strEndTime': '05:00PM',
'strLevel': 1,
'strUserMobile': u'011151588',
'strEndDate': 'Thursday 28 April',
'strUsername': u'Jane Doe',
'strStartDate': 'Thursday 28 April',
}, {
'strStartTime': '05:00PM',
'strEndTime': '03:30PM',
'strLevel': 1,
'strUserMobile': 'User does not have a company phone no',
'strEndDate': 'Thursday 28 April',
'strUsername': u'Fred Perry',
'strStartDate': 'Wednesday 27 April',
}, {
'strStartTime': '09:00AM',
'strEndTime': '07:00AM',
'strLevel': 1,
'strUserMobile': 'User does not have a company phone no',
'strEndDate': 'Tuesday 03 May',
'strUsername': u'Sally Cinomon',
'strStartDate': 'Monday 25 April',
}]}

Just expanding my comment above with how to use the data in template:
You can send your data within render:
return render(request, "oncall/rota.html", {"policies": objPolicyData['escalation_policies'])
Then, in your template file, you can do something like this:
{% for policy in policies %}
{% for objOnCall in policy.on_call %}
<p> Level: {{ objOnCall.level }} </p>
<p> Start Time: {{ objOnCall.start }} </p>
{% endfor %}
{% endfor %}
UPDATE
According to the your last update to the question;
You said,
however the dictionary is currently not updating, its only
adding/editing the last value
This is right, because you don't have an array contains your policy objects. You only set the last value in the loop to the dictionary. This is why you are getting only last object.
This should do the work;
# Create your views here.
def index(request):
### Get all the Polices ###
policies = []
for objPolicy in objPolicyData['escalation_policies']:
strPolicyName = objPolicy['name']
policy = {}
policy['name'] = strPolicyName
if strPolicyName.lower().find('test') == -1:
policy = {}
policy['strPolicyName'] = strPolicyName # add policy name here
policy['objUsers'] = [] # define an empty array for users
for objOnCall in objPolicy['on_call']:
obj['strLevel'] = objOnCall['level']
obj['strStartDate'] = getDate(objOnCall['start'])
obj['strStartTime'] = getTime(objOnCall['start'])
obj['strEndDate'] = getDate(objOnCall['end'])
obj['strEndTime'] = getTime(objOnCall['end'])
objUser = objOnCall['user']
obj['strUsername'] = objUser['name']
obj['strUserMobile'] = getUserMobile(objUser['id'])
policy['objUsers'].append(obj) # add each user to the users array belongs to this policy object
policies.append(policy) # and finally append final and prepared policy object to our main policies array.
context = {"policies": policies}
return render(request, 'oncall/rota.html', context)
Now you can do anything you want with this array inside a for loop in template. (see my above example)

I think, this question is not good.
there are many kind of solution for your goal.
even, django documents.
this is just sample.
context = dict()
for objOnCall in objPolicy['on_call']:
obj = dict()
obj['strLevel'] = objOnCall['level']
obj['strStartDate'] = getDate(objOnCall['start'])
obj['strStartTime'] = getTime(objOnCall['start'])
obj['strEndDate'] = getDate(objOnCall['end'])
obj['strEndTime'] = getTime(objOnCall['end'])
obj['objUser'] = objOnCall['user']
obj['strUsername'] = objUser['name']
obj['strUserMobile'] = getUserMobile(objUser['id'])
context[objUser['name']] = obj
return render(request, 'oncall/rota.html', context)

Organize by Twitter unique identifier using python

I have a CSV file with each line containing information pertaining to a particular tweet (i.e. each line contains Lat, Long, User_ID, tweet and so on). I need to read the file and organize the tweets by the User_ID. I am trying to end up with a given User_ID attached to all of the tweets with that specific ID.
Here is what I want:
user_id: 'lat', 'long', 'tweet'
: 'lat', 'long', 'tweet'
user_id2: 'lat', 'long', 'tweet'
: 'lat', 'long', 'tweet'
: 'lat', 'long', 'tweet'
and so on...
This is a snip of my code that reads in the CSV file and creates a list:
UID = []
myID = []
ID = []
f = None
with open(csv_in,'rU') as f:
myreader = csv.reader(f, delimiter=',')
for row in myreader:
# Assign columns in csv to variables.
latitude = row[0]
longitude = row[1]
user_id = row[2]
user_name = row[3]
date = row[4]
time = row[5]
tweet = row[6]
flag = row[7]
compound = row[8]
Vote = row[9]
# Read variables into separate lists.
UID.append(user_id + ', ' + latitude + ', ' + longitude + ', ' + user_name + ', ' + date + ', ' + time + ', ' + tweet + ', ' + flag + ', ' + compound)
myID = ', '.join(UID)
ID = myID.split(', ')

I'd suggest you use pandas for this. It will allow you not only to list your tweets by user_id, as in your question, but also to do many other manipulations quite easily.
As an example, take a look at this python notebook from NLTK. At the end of it, you see an operation very closed to yours, reading a csv file containing tweets,
In [25]:
import pandas as pd
​
tweets = pd.read_csv('tweets.20150430-223406.tweet.csv', index_col=2, header=0, encoding="utf8")
You can also find a simple operation: looking for the tweets of a certain user,
In [26]:
tweets.loc[tweets['user.id'] == 557422508]['text']
Out[26]:
id
593891099548094465 VIDEO: Sturgeon on post-election deals http://...
593891101766918144 SNP leader faces audience questions http://t.c...
Name: text, dtype: object
For listing the tweets by user_id, you would simply do something like the following (this is not in the original notebook),
In [9]:
tweets.set_index('user.id')[0:4]
Out[9]:
created_at favorite_count in_reply_to_status_id in_reply_to_user_id retweet_count retweeted text truncated
user.id
107794703 Thu Apr 30 21:34:06 +0000 2015 0 NaN NaN 0 False RT #KirkKus: Indirect cost of the UK being in ... False
557422508 Thu Apr 30 21:34:06 +0000 2015 0 NaN NaN 0 False VIDEO: Sturgeon on post-election deals http://... False
3006692193 Thu Apr 30 21:34:06 +0000 2015 0 NaN NaN 0 False RT #LabourEoin: The economy was growing 3 time... False
455154030 Thu Apr 30 21:34:06 +0000 2015 0 NaN NaN 0 False RT #GregLauder: the UKIP east lothian candidat... False
Hope it helps.

Parsing unstructured text in Python

I wanted to parse a text file that contains unstructured text. I need to get the address, date of birth, name, sex, and ID.
. 55 MORILLO ZONE VIII,
BARANGAY ZONE VIII
(POB.), LUISIANA, LAGROS
F
01/16/1952
ALOMO, TERESITA CABALLES
3412-00000-A1652TCA2
12
. 22 FABRICANTE ST. ZONE
VIII LUISIANA LAGROS,
BARANGAY ZONE VIII
(POB.), LUISIANA, LAGROS
M
10/14/1967
AMURAO, CALIXTO MANALO13
In the example above, the first 3 lines is the address, the line with just an "F" is the sex, the DOB would be the line after "F", name after the DOB, the ID after the name, and the no. 12 under the ID is the index/record no.
However, the format is not consistent. In the second group, the address is 4 lines instead of 3 and the index/record no. is appended after the name (if the person doesn't have an ID field).
I wanted to rewrite the text into the following format:
name, ID, address, sex, DOB

Here is a first stab at a pyparsing solution (easy-to-copy code at the pyparsing pastebin). Walk through the separate parts, according to the interleaved comments.
data = """\
. 55 MORILLO ZONE VIII,
BARANGAY ZONE VIII
(POB.), LUISIANA, LAGROS
F
01/16/1952
ALOMO, TERESITA CABALLES
3412-00000-A1652TCA2
12
. 22 FABRICANTE ST. ZONE
VIII LUISIANA LAGROS,
BARANGAY ZONE VIII
(POB.), LUISIANA, LAGROS
M
10/14/1967
AMURAO, CALIXTO MANALO13
"""
from pyparsing import LineEnd, oneOf, Word, nums, Combine, restOfLine, \
alphanums, Suppress, empty, originalTextFor, OneOrMore, alphas, \
Group, ZeroOrMore
NL = LineEnd().suppress()
gender = oneOf("M F")
integer = Word(nums)
date = Combine(integer + '/' + integer + '/' + integer)
# define the simple line definitions
gender_line = gender("sex") + NL
dob_line = date("DOB") + NL
name_line = restOfLine("name") + NL
id_line = Word(alphanums+"-")("ID") + NL
recnum_line = integer("recnum") + NL
# define forms of address lines
first_addr_line = Suppress('.') + empty + restOfLine + NL
# a subsequent address line is any line that is not a gender definition
subsq_addr_line = ~(gender_line) + restOfLine + NL
# a line with a name and a recnum combined, if there is no ID
name_recnum_line = originalTextFor(OneOrMore(Word(alphas+',')))("name") + \
integer("recnum") + NL
# defining the form of an overall record, either with or without an ID
record = Group((first_addr_line + ZeroOrMore(subsq_addr_line))("address") +
gender_line +
dob_line +
((name_line +
id_line +
recnum_line) |
name_recnum_line))
# parse data
records = OneOrMore(record).parseString(data)
# output the desired results (note that address is actually a list of lines)
for rec in records:
if rec.ID:
print "%(name)s, %(ID)s, %(address)s, %(sex)s, %(DOB)s" % rec
else:
print "%(name)s, , %(address)s, %(sex)s, %(DOB)s" % rec
print
# how to access the individual fields of the parsed record
for rec in records:
print rec.dump()
print rec.name, 'is', rec.sex
print
Prints:
ALOMO, TERESITA CABALLES, 3412-00000-A1652TCA2, ['55 MORILLO ZONE VIII,', 'BARANGAY ZONE VIII', '(POB.), LUISIANA, LAGROS'], F, 01/16/1952
AMURAO, CALIXTO MANALO, , ['22 FABRICANTE ST. ZONE', 'VIII LUISIANA LAGROS,', 'BARANGAY ZONE VIII', '(POB.), LUISIANA, LAGROS'], M, 10/14/1967
['55 MORILLO ZONE VIII,', 'BARANGAY ZONE VIII', '(POB.), LUISIANA, LAGROS', 'F', '01/16/1952', 'ALOMO, TERESITA CABALLES', '3412-00000-A1652TCA2', '12']
- DOB: 01/16/1952
- ID: 3412-00000-A1652TCA2
- address: ['55 MORILLO ZONE VIII,', 'BARANGAY ZONE VIII', '(POB.), LUISIANA, LAGROS']
- name: ALOMO, TERESITA CABALLES
- recnum: 12
- sex: F
ALOMO, TERESITA CABALLES is F
['22 FABRICANTE ST. ZONE', 'VIII LUISIANA LAGROS,', 'BARANGAY ZONE VIII', '(POB.), LUISIANA, LAGROS', 'M', '10/14/1967', 'AMURAO, CALIXTO MANALO', '13']
- DOB: 10/14/1967
- address: ['22 FABRICANTE ST. ZONE', 'VIII LUISIANA LAGROS,', 'BARANGAY ZONE VIII', '(POB.), LUISIANA, LAGROS']
- name: AMURAO, CALIXTO MANALO
- recnum: 13
- sex: M
AMURAO, CALIXTO MANALO is M

you have to exploit whatever regularity and structure the text does have.
I suggest you read one line at a time and match it to a regular expression to determine its type, fill in the appropriate field in a person object. writing out that object and starting a new one whenever you get a field that you already have filled in.

It may be overkill, but the leading edge machine learning algorithms for this type of problem are based on conditional random fields. For example, Accurate Information Extraction from Research Papers
using Conditional Random Fields.
There is software out there that makes training these models relatively easy. See Mallet or CRF++.

You can probably do this with regular expressions without too much difficulty. If you have never used them before, check out the python documentation, then fire up redemo.py (on my computer, it's in c:\python26\Tools\scripts).
The first task is to split the flat file into a list of entities (one chunk of text per record). From the snippet of text you gave, you could split the file with a pattern matching the beginning of a line, where the first character is a dot:
import re
re_entity_splitter = re.compile(r'^\.')
entities = re_entity_splitter.split(open(textfile).read())
Note that the dot must be escaped (it's a wildcard character by default). Note also the r before the pattern. The r denotes 'raw string' format, which excuses you from having to escape the escape characters, resulting in so-called 'backslash plague.'
Once you have the file split into individual people, picking out the gender and birthdate is a snap. Use these:
re_gender = re.compile(r'^[MF]')
re_birth_Date = re.compile(r'\d\d/\d\d/\d\d')
And away you go. You can paste the flat file into re demo GUI and experiment with creating patterns to match what you need. You'll have it parsed in no time. Once you get good at this, you can use symbolic group names (see docs) to pick out individual elements quickly and cleanly.

Here's a quick hack job.
f = open('data.txt')
def process(file):
address = ""
for line in file:
if line == '': raise StopIteration
line = line.rstrip() # to ignore \n
if line in ('M','F'):
sex = line
break
else:
address += line
DOB = file.readline().rstrip() # to ignore \n
name = file.readline().rstrip()
if name[-1].isdigit():
name = re.match(r'^([^\d]+)\d+', name).group(1)
ID = None
else:
ID = file.readline().rstrip()
file.readline() # ignore the record #
print (name, ID, address, sex, DOB)
while True:
process(f)

Categories