I have a python script where I'm trying to fetch data from meraki dashboard through its API. Now the data is stored in a dataframe which needs to be pushed to a Smartsheet using the Smartsheet API integration. I've tried searching the Smartsheet API documentation but couldn't find any solution to the problem. Has anyone worked on this kind of use case before or know a script to push a simple data frame to the smartsheet?
The code is something like this:
for device in list_of_devices:
try:
dict1 = {'Name': [device['name']],
"Serial_No": [device['serial']],
'MAC': [device['mac']],
'Network_Id': [device['networkId']],
'Product_Type': [device['productType']],
'Model': [device['model']],
'Tags': [device['tags']],
'Lan_Ip': [device['lanIp']],
'Configuration_Updated_At': [device['configurationUpdatedAt']],
'Firmware': [device['firmware']],
'URL': [device['url']]
}
except KeyError:
dict1['Lan_Ip'] = "NA"
temp = pd.DataFrame.from_dict(dict1)
alldata = alldata.append(temp)
alldata.reset_index(drop=True, inplace=True)
The dataframe("alldata") looks something like this:
Name Serial_No MAC \
0 xxxxxxxxxxxxxxxx xxxxxxxxxxxxxx xxxxxxxxxxxxxxxxx
1 xxxxxxxxxxxxxxxx xxxxxxxxxxxxxx xxxxxxxxxxxxxxxxx
2 xxxxxxxxxxxxxxxx xxxxxxxxxxxxxx xxxxxxxxxxxxxxxxx
the dataframe has somewhere around 1000 rows and 11 columns
I've tried pushing this dataframe similar to the code mentioned in the comments but I'm getting a "Bad Request" error.
smart = smartsheet.Smartsheet(access_token='xxxxxxxx')
sheet_id = xxxxxxxxxxxxx
sheet = smart.Sheets.get_sheet(sheet_id)
column_map = {}
for column in sheet.columns:
column_map[column.title] = column.id
data_dict = alldata.to_dict('index')
rowsToAdd = []
for i,i in data_dict.items():
new_row = smart.models.Row()
new_row.to_top = True
for k,v in i.items():
new_cell = smart.models.Cell()
new_cell.column_id = column_map[k]
new_cell.value = v
new_row.cells.append(new_cell)
rowsToAdd.append(new_row)
result = smart.Sheets.add_rows(sheet_id, rowsToAdd)
{"response": {"statusCode": 400, "reason": "Bad Request", "content": {"detail": {"index": 0}, "errorCode": 1012, "message": "Required object attribute(s) are missing from your request: cell.value.", "refId": "1ob56acvz5nzv"}}}
Smartsheet photo where the data must be pushed
The following code adds data from a dataframe to a sheet in Smartsheet -- this should be enough to at least get you started. If you still can't get the desired result using this code, please update your original post to include the code you're using, the outcome you're wanting, and a detailed description of the issue you encountered. (Add a comment to this answer if you update your original post, so I'll be notified and will know to look.)
# target sheet
sheet_id = 3932034054809476
sheet = smartsheet_client.Sheets.get_sheet(sheet_id)
# translate column names to column id
column_map = {}
for column in sheet.columns:
column_map[column.title] = column.id
df = pd.DataFrame({'item_id': [111111, 222222],
'item_color': ['red', 'yellow'],
'item_location': ['office', 'kitchen']})
data_dict = df.to_dict('index')
rowsToAdd = []
# each object in data_dict represents 1 row of data
for i, i in data_dict.items():
# create a new row object
new_row = smartsheet_client.models.Row()
new_row.to_top = True
# for each key value pair, create & add a cell to the row object
for k, v in i.items():
# create the cell object and populate with value
new_cell = smartsheet_client.models.Cell()
new_cell.column_id = column_map[k]
new_cell.value = v
# add the cell object to the row object
new_row.cells.append(new_cell)
# add the row object to the collection of rows
rowsToAdd.append(new_row)
# add the collection of rows to the sheet in Smartsheet
result = smartsheet_client.Sheets.add_rows(sheet_id, rowsToAdd)
UPDATE #1 - re Bad Request error
Seems like the error you've described in your first comment below is perhaps being caused by the fact that some of the cells in your dataframe don't have a value. When you add a new row using the Smartsheet API, each cell that's specified for the row must specify a value for the cell -- otherwise you'll get the Bad Request error you've described. Maybe try adding an if statement inside the for loop to skip adding the cell if the value of v is None?
for k,v in i.items():
# skip adding this cell if there's no value
if v is None:
continue
...
UPDATE #2 - re further troubleshooting
In response to your second comment below: you'll need to debug further using the data in your dataframe, as I'm unable to repro the issue you describe using other data.
To simplify things -- I'd suggest that you start by trying to debug with just one item in the dataframe. You can do so by adding the line (statement) break at the end of the for loop that's building the dict -- that way, only the first device will be added.
for device in list_of_devices:
try:
...
except KeyError:
dict1['Lan_Ip'] = "NA"
temp = pd.DataFrame.from_dict(dict1)
alldata = alldata.append(temp)
# break out of loop after one item is added
break
alldata.reset_index(drop=True, inplace=True)
# print dataframe contents
print (alldata)
If you get the same error when testing with just one item, and can't recognize what it is about that data (or the way it's stored in your dataframe) that's causing the Smartsheet error, then feel free to add a print (alldata) statement after the for loop (as I show in the code snippet above) to your code and update your original post again to include the output of that statement (changing any sensitive data values, of course) -- and then I can try to repro/troubleshoot using that data.
UPDATE #3 - repro'd issue
Okay, so I've reproduced the error you've described -- by specifying None as the value of a field in the dict.
The following code successfully inserts two new rows into Smartsheet -- because every field in each dict it builds contains a (non-None) value. (For simplicity, I'm manually constructing two dicts in the same manner as you do in your for loop.)
# target sheet
sheet_id = 37558492129156
sheet = smartsheet_client.Sheets.get_sheet(sheet_id)
# translate column names to column id
column_map = {}
for column in sheet.columns:
column_map[column.title] = column.id
#----
# start: repro SO question's building of dataframe
#----
alldata = pd.DataFrame()
dict1 = {'Name': ['name1'],
"Serial_No": ['serial_no1'],
'MAC': ['mac1'],
'Network_Id': ['networkId1'],
'Product_Type': ['productType1'],
'Model': ['model1'],
'Tags': ['tags1'],
'Lan_Ip': ['lanIp1'],
'Configuration_Updated_At': ['configurationUpdatedAt1'],
'Firmware': ['firmware1'],
'URL': ['url1']
}
temp = pd.DataFrame.from_dict(dict1)
alldata = alldata.append(temp)
dict2 = {'Name': ['name2'],
"Serial_No": ['serial_no2'],
'MAC': ['mac2'],
'Network_Id': ['networkId2'],
'Product_Type': ['productType2'],
'Model': ['model2'],
'Tags': ['tags2'],
'Lan_Ip': ['lanIp2'],
'Configuration_Updated_At': ['configurationUpdatedAt2'],
'Firmware': ['firmware2'],
'URL': ['URL2']
}
temp = pd.DataFrame.from_dict(dict2)
alldata = alldata.append(temp)
alldata.reset_index(drop=True, inplace=True)
#----
# end: repro SO question's building of dataframe
#----
data_dict = alldata.to_dict('index')
rowsToAdd = []
# each object in data_dict represents 1 row of data
for i, i in data_dict.items():
# create a new row object
new_row = smartsheet_client.models.Row()
new_row.to_top = True
# for each key value pair, create & add a cell to the row object
for k, v in i.items():
# create the cell object and populate with value
new_cell = smartsheet_client.models.Cell()
new_cell.column_id = column_map[k]
new_cell.value = v
# add the cell object to the row object
new_row.cells.append(new_cell)
# add the row object to the collection of rows
rowsToAdd.append(new_row)
result = smartsheet_client.Sheets.add_rows(sheet_id, rowsToAdd)
However, running the following code (where the value of the URL field in the second dict is set to None) results in the same error you've described:
{"response": {"statusCode": 400, "reason": "Bad Request", "content": {"detail": {"index": 1}, "errorCode": 1012, "message": "Required object attribute(s) are missing from your request: cell.value.", "refId": "dw1id3oj1bv0"}}}
Code that causes this error (identical to the successful code above except that the value of the URL field in the second dict is None):
# target sheet
sheet_id = 37558492129156
sheet = smartsheet_client.Sheets.get_sheet(sheet_id)
# translate column names to column id
column_map = {}
for column in sheet.columns:
column_map[column.title] = column.id
#----
# start: repro SO question's building of dataframe
#----
alldata = pd.DataFrame()
dict1 = {'Name': ['name1'],
"Serial_No": ['serial_no1'],
'MAC': ['mac1'],
'Network_Id': ['networkId1'],
'Product_Type': ['productType1'],
'Model': ['model1'],
'Tags': ['tags1'],
'Lan_Ip': ['lanIp1'],
'Configuration_Updated_At': ['configurationUpdatedAt1'],
'Firmware': ['firmware1'],
'URL': ['url1']
}
temp = pd.DataFrame.from_dict(dict1)
alldata = alldata.append(temp)
dict2 = {'Name': ['name2'],
"Serial_No": ['serial_no2'],
'MAC': ['mac2'],
'Network_Id': ['networkId2'],
'Product_Type': ['productType2'],
'Model': ['model2'],
'Tags': ['tags2'],
'Lan_Ip': ['lanIp2'],
'Configuration_Updated_At': ['configurationUpdatedAt2'],
'Firmware': ['firmware2'],
'URL': [None]
}
temp = pd.DataFrame.from_dict(dict2)
alldata = alldata.append(temp)
alldata.reset_index(drop=True, inplace=True)
#----
# end: repro SO question's building of dataframe
#----
data_dict = alldata.to_dict('index')
rowsToAdd = []
# each object in data_dict represents 1 row of data
for i, i in data_dict.items():
# create a new row object
new_row = smartsheet_client.models.Row()
new_row.to_top = True
# for each key value pair, create & add a cell to the row object
for k, v in i.items():
# create the cell object and populate with value
new_cell = smartsheet_client.models.Cell()
new_cell.column_id = column_map[k]
new_cell.value = v
# add the cell object to the row object
new_row.cells.append(new_cell)
# add the row object to the collection of rows
rowsToAdd.append(new_row)
result = smartsheet_client.Sheets.add_rows(sheet_id, rowsToAdd)
Finally, note that the error message I received contains {"index": 1} -- this implies that the value of index in this error message indicates the (zero-based) index of the problematic row. The fact that your error message contains {"index": 0} implies that there's a problem with the data in the first row you're trying to add to Smartsheet (i.e., the first item in the dataframe). Therefore, following the troubleshooting guidance I posted in my previous update (Update #2 above) should allow you to closely examine the data for the first item/row and hopefully spot the problematic data (i.e., where the value is missing).
Related
So this is a somewhat of a continuation from a previous post of mine except now I have API data to work with. I am trying to get keys Type and Email as columns in a data frame to come up with a final number. My code:
jsp_full=[]
for p in payloads:
payload = {"payload": {"segmentId":p}}
r = requests.post(url,headers = header, json = payload)
#print(r, r.reason)
time.sleep(r.elapsed.total_seconds())
json_data = r.json() if r and r.status_code == 200 else None
json_keys = json_data['payload']['supporters']
json_package = []
jsp_full.append(json_package)
for row in json_keys:
SID = row['supporterId']
Handle = row['contacts']
a_key = 'value'
list_values = [a_list[a_key] for a_list in Handle]
string = str(list_values).split(",")
data = {
'SupporterID' : SID,
'Email' : strip_characters(string[-1]),
'Type' : labels(p)
}
json_package.append(data)
t2 = round(time.perf_counter(),2)
b_key = "Email"
e = len([b_list[b_key] for b_list in json_package])
t = str(labels(p))
#print(json_package)
print(f'There are {e} emails in the {t} segment')
print(f'Finished in {t2 - t1} seconds')
excel = pd.DataFrame(json_package)
excel.to_excel(r'C:\Users\am\Desktop\email parsing\{0} segment {1}.xlsx'.format(t, str(today)), sheet_name=t)
This part works all well and good. Each payload in the API represents a different segment of people so I split them out into different files. However, I am at a point where I need to combine all records into a single data frame hence why I append out to jsp_full. This is a list of a list of dictionaries.
Once I have that I would run the balance of my code which is like this:
S= pd.DataFrame(jsp_full[0], index = {0})
Advocacy_Supporters = S.sort_values("Type").groupby("Type", as_index=False)["Email"].first()
print(Advocacy_Supporters['Email'].count())
print("The number of Unique Advocacy Supporters is :")
Advocacy_Supporters_Group = Advocacy_Supporters.groupby("Type")["Email"].nunique()
print(Advocacy_Supporters_Group)
Some sample data:
[{'SupporterID': '565f6a2f-c7fd-4f1b-bac2-e33976ef4306', 'Email': 'somebody#somewhere.edu', 'Type': 'd_Student Ambassadors'}, {'SupporterID': '7508dc12-7647-4e95-a8b8-bcb067861faf', 'Email': 'someoneelse#email.somewhere.edu', 'Type': 'd_Student Ambassadors'},...`
My desired output is a dataframe that looks like so:
SupporterID Email Type
565f6a2f-c7fd-4f1b-bac2-e33976ef4306 somebody#somewhere.edu d_Student Ambassadors
7508dc12-7647-4e95-a8b8-bcb067861faf someoneelse#email.somewhere.edu d_Student Ambassadors
Any help is greatly appreciated!!
So because this code creates an excel file for each segment, all I did was read back in the excels via a for loop like so:
filesnames = ['e_S Donors', 'b_Contributors', 'c_Activists', 'd_Student Ambassadors', 'a_Volunteers', 'f_Offline Action Takers']
S= pd.DataFrame()
for i in filesnames:
data = pd.read_excel(r'C:\Users\am\Desktop\email parsing\{0} segment {1}.xlsx'.format(i, str(today)),sheet_name= i, engine = 'openpyxl')
S= S.append(data)
This did the trick since it was in a format I already wanted.
I'm trying to insert a 2D array in order to get two columns inserted into a sheet via gspread. I'm able to insert the individual lists fine, but inserting the array causes an error. Here's my code.
def megaDepotScrape():
listings = 0
priceList = []
skuList = []
# Iterate through the listings on the page, printing the price per entry
for listings in range(0, 12):
# Connect to the site to be scraped
siteURL = "https://megadepot.com/catalog/lab-equipment/multiwell-plates/brand:brandtech/"
response = requests.get(siteURL, headers=headers)
# with open('brandtech.html', 'wb') as fp:
# fp.write(response.content)
# Cook the soup
html_soup = BeautifulSoup(response.text, 'html.parser')
# Find all containers with the appropriate class name
# The 'strong' class 'hot' contains the price information
price_containers = html_soup.find_all("strong", class_="hot")
price = price_containers[listings]
priceStr = list(price)
priceList.append(priceStr)
# Find all containers for the appropriate class name
# The 'div' class 'product-wrapper' contains the SKU
sku_containers = html_soup.find_all("div", class_="product-wrapper")
sku = sku_containers[listings]
# The sku is stored in the 'data-variant' of the 'article' tag
for data in sku.find_all("article"):
skuData = data["data-variant"]
skuList.append(skuData)
# Iterate through the loop
listings += 1
# Write both lists to the sheets document
# Reference update() in docs
rows = [priceList, skuList]
print(rows)
#sheet.update('A1', [list(e) for e in zip(*rows)])
sheet.insert_row(skuList)
Here's the value of rows
[[['$81.57'], ['$80.91'], ['$91.63'], ['$91.63'], ['$455.20'], ['$196.90'], ['$282.60'], ['$146.10'], ['$97.22'], ['$166.70'], ['$287.30'], ['$237.50']], ['781411', '781415', '781412', '781416', '701355', '701330', '701346', '701352', '782153', '701354', '781347', '781345']]
And here's the error I get
sheet.update('A1', [list(e) for e in zip(*rows)])
File "C:\Users\Jacob\PythonTestProject\venv\lib\site-packages\gspread\utils.py", line 592, in wrapper
return f(*args, **kwargs)
File "C:\Users\Jacob\PythonTestProject\venv\lib\site-packages\gspread\models.py", line 1127, in update
{'values': values, 'majorDimension': kwargs['major_dimension']}
File "C:\Users\Jacob\PythonTestProject\venv\lib\site-packages\gspread\models.py", line 236, in values_update
r = self.client.request('put', url, params=params, json=body)
File "C:\Users\Jacob\PythonTestProject\venv\lib\site-packages\gspread\client.py", line 76, in request
raise APIError(response)
gspread.exceptions.APIError: {'code': 400, 'message': 'Invalid values[0][0]: list_value {\n values {\n string_value: "$81.57"\n }\n}\n', 'status': 'INVALID_ARGUMENT'}```
I'm not sure if there's some kind of limit that I'm hitting from uploading so much, or if there's some kind of error. Please let me know. Thank you.
I believe your goal as follows.
From the following sample value.
Here's the value of rows
[[['$81.57'], ['$80.91'], ['$91.63'], ['$91.63'], ['$455.20'], ['$196.90'], ['$282.60'], ['$146.10'], ['$97.22'], ['$166.70'], ['$287.30'], ['$237.50']], ['781411', '781415', '781412', '781416', '701355', '701330', '701346', '701352', '782153', '701354', '781347', '781345']]
I understood that the values of priceList and skuList might be the following values.
priceList = [['$81.57'], ['$80.91'], ['$91.63'], ['$91.63'], ['$455.20'], ['$196.90'], ['$282.60'], ['$146.10'], ['$97.22'], ['$166.70'], ['$287.30'], ['$237.50']]
skuList = ['781411', '781415', '781412', '781416', '701355', '701330', '701346', '701352', '782153', '701354', '781347', '781345']
You want to put the values of priceList and skuList to 2 columns.
Modification points:
In this case, the array is required to be as follows.
[["a1", "b1"], ["a2", "b2"],,,]
When you want to insert the several rows with 2 columns, you can use insert_rows().
When this is reflected to your script, it becomes as follows.
Sample script:
client = gspread.authorize(credentials)
spreadsheetId = "###" # Please set the Spreadsheet ID.
sheetName = "Sheet1" # Please set the sheet name you want to put the values.
spreadsheet = client.open_by_key(spreadsheetId)
sheet = spreadsheet.worksheet(sheetName)
# These values are from your question.
priceList = [['$81.57'], ['$80.91'], ['$91.63'], ['$91.63'], ['$455.20'], ['$196.90'], ['$282.60'], ['$146.10'], ['$97.22'], ['$166.70'], ['$287.30'], ['$237.50']]
skuList = ['781411', '781415', '781412', '781416', '701355', '701330', '701346', '701352', '782153', '701354', '781347', '781345']
# I modified below script.
row = [[e1[0], e2] for e1, e2 in zip(priceList, skuList)]
print(row) # You can confirm the value of "row".
sheet.insert_rows(row)
When you run this script, the values of priceList and skuList are put to the columns "A" and "B" of "Sheet1".
References:
insert_rows(values, row=1, value_input_option='RAW')
Why tf you use insert_row as insert_cols ??? :D
If you want to insert values into one row just do
wks.insert_row(priceList, index=10, value_input_option='RAW')
If you want to insert values into one column just do
wks.insert_cols(values=[priceList], col=2, value_input_option='RAW')
Note that
priceList=['781411', '781415', '781412', '781416']
is like
priceList=['row1', 'row2', 'row3', 'row4']
and
priceList=[['781411'], ['781415'], ['781412'], ['781416']]
is like
priceList=[['col1'], ['col2'], ['col3'], ['col4']]
I had the column in the smartsheet as 'Allowing multiple contacts to be selected'.
I am using simple_smartsheet package (https://pypi.org/project/simple-smartsheet/) but I cannot seems to find in the internet that anyone adding multiple contact using this package.
Below is the piece of code that i tried:
from simple_smartsheet import Smartsheet
from simple_smartsheet.models import Sheet, Column, Row, Cell, ColumnType
#%%
access_token='XXX'
smartsheet = Smartsheet(access_token)
sheet_name = 'test'
sh = smartsheet.sheets.get(sheet_name)
new_rows = [
Row(
to_top=True,
cells=[
Cell(column_id=released_by.id, value=[{'objectType': 'CONTACT',
'email': 'xxx.yyy#westrac.com.au',
'name': 'xxx yyy'},
{'objectType': 'CONTACT',
'email': 'aaa.bbb#westrac.com.au',
'name': 'aaa bbb'}])
],
),
]
#new_rows.append(Row(to_top=True,cells=sh.make_cells(row_value)))
smartsheet.sheets.add_rows(sh.id, new_rows)
But I got this error:
SmartsheetHTTPClientError: HTTP response code 400 - Error code 1008 - Unable to parse request. The following error occurred: Field "value" was not parsable. value must be a primitive type
at [Source: java.io.PushbackInputStream#786472ed; line: 1, column: 241].
I am not quite sure where did i do wrong. Any thoughts?
From the python doc to the original doc you can see that the Cell class only accept as value either a string a boolean or a number
So this should work:
new_rows = [
Row(
to_top=True,
cells=[
Cell(column_id=released_by.id, value="your value")
],
),
]
You were close and this was a pain. This is working for me...example for multi contact and single...
#build to update a cell
new_cell = smartsheet_client.models.Cell()
new_cell.column_id = pasteCol
new_cell.object_value = {"objectType":"MULTI_CONTACT", "values":[{"name":"rOB","email":"test#test.com"}, {"name":"rob","email":"rob#test.com"}]}
#this will work on single select
#new_cell.object_value = {'objectType': 'CONTACT','email': 'test#test.com', 'name': 'test'}
new_cell.strict = True
print(new_cell)
#append to update
get_row = smartsheet.models.Row()
get_row.id = rowID
get_row.cells.append(new_cell)
updated_row = smartsheet_client.Sheets.update_rows(sheet_id,[get_row])
Create the list of values as text delimited (I created a function to convert it to a correct formatted string because SS is very strict)
Initialize the row as usual.
Create the cell adding the "objectType" option as "MULTI_PICKLIST" and add to the row (this should be inside "object_value")
new_row_add.cells.append({'column_id': col_id, 'object_value': {'objectType': 'MULTI_PICKLIST', 'values': new_list}, 'strict': True})
Add the row as usual
This also works for row updates
I am working with the Twitter Search API which returns a dictionary of dictionaries. My goal is to create a dataframe from a list of keys in the response dictionary.
Example of API response here: Example Response
I have a list of keys within the Statuses dictionary
keys = ["created_at", "text", "in_reply_to_screen_name", "source"]
I would like to loop through each key value returned in the Statuses dictionary and put them in a dataframe with the keys as the columns.
Currently have code to loop through a single key individually and assign to list then append to dataframe but want a way to do more than one key at a time. Current code below:
#w is the word to be queired
w = 'keyword'
#count of tweets to return
count = 1000
#API call
query = twitter.search.tweets(q= w, count = count)
def data_l2 (q, k1, k2):
data = []
for results in q[k1]:
data.append(results[k2])
return(data)
screen_names = data_l3(query, "statuses", "user", "screen_name")
data = {'screen_names':screen_names,
'tweets':tweets}
frame=pd.DataFrame(data)
frame
I will share a more generic solution that I came up with, as I was working with the Twitter API. Let's say you have the ID's of tweets that you want to fetch in a list called my_ids :
# Fetch tweets from the twitter API using the following loop:
list_of_tweets = []
# Tweets that can't be found are saved in the list below:
cant_find_tweets_for_those_ids = []
for each_id in my_ids:
try:
list_of_tweets.append(api.get_status(each_id))
except Exception as e:
cant_find_tweets_for_those_ids.append(each_id)
Then in this code block we isolate the json part of each tweepy status object that we have downloaded and we add them all into a list....
my_list_of_dicts = []
for each_json_tweet in list_of_tweets:
my_list_of_dicts.append(each_json_tweet._json)
...and we write this list into a txt file:
with open('tweet_json.txt', 'w') as file:
file.write(json.dumps(my_list_of_dicts, indent=4))
Now we are going to create a DataFrame from the tweet_json.txt file (I have added some keys that were relevant to my use case that I was working on, but you can add your specific keys instead):
my_demo_list = []
with open('tweet_json.txt', encoding='utf-8') as json_file:
all_data = json.load(json_file)
for each_dictionary in all_data:
tweet_id = each_dictionary['id']
whole_tweet = each_dictionary['text']
only_url = whole_tweet[whole_tweet.find('https'):]
favorite_count = each_dictionary['favorite_count']
retweet_count = each_dictionary['retweet_count']
created_at = each_dictionary['created_at']
whole_source = each_dictionary['source']
only_device = whole_source[whole_source.find('rel="nofollow">') + 15:-4]
source = only_device
retweeted_status = each_dictionary['retweeted_status'] = each_dictionary.get('retweeted_status', 'Original tweet')
if retweeted_status == 'Original tweet':
url = only_url
else:
retweeted_status = 'This is a retweet'
url = 'This is a retweet'
my_demo_list.append({'tweet_id': str(tweet_id),
'favorite_count': int(favorite_count),
'retweet_count': int(retweet_count),
'url': url,
'created_at': created_at,
'source': source,
'retweeted_status': retweeted_status,
})
tweet_json = pd.DataFrame(my_demo_list, columns = ['tweet_id', 'favorite_count',
'retweet_count', 'created_at',
'source', 'retweeted_status', 'url'])
Want to prompt browser to save csv
^^working off above question, file is exporting correctly but the data is not displaying correctly.
#view_config(route_name='csvfile', renderer='csv')
def csv(self):
name = DBSession.query(table).join(othertable).filter(othertable.id == 9701).all()
header = ['name']
rows = []
for item in name:
rows = [item.id]
return {
'header': header,
'rows': rows
}
Getting _csv.Error
Error: sequence expected but if I change in my renderer writer.writerows(value['rows']) to writer.writerow(value['rows']) the file will download via the browser just fine. Problem is, it's not displaying data in each row. The entire result/dataset is in one row, so each entry is in it's own column rather than it's own row.
First, I wonder if having a return statement inside your for loop isn't also causing problems; from the linked example it looks like their loop was in the prior statement.
I think what it looks like it's doing is it's building a collection of rows based on "table" having columns with the same name as the headers. What are the fields in your table table?
name = DBSession.query(table).join(othertable).filter(othertable.id == 9701).all()
This is going to give you back essentially a collection of rows from table, as if you did a SELECT query on it.
Something like
name = DBSession.query(table).join(othertable).filter(othertable.id == 9701).all()
header = ['name']
rows = []
for item in name:
rows.append(item.name)
return {
'header': header,
'rows': r
}
Figured it out. kept getting Error: sequence expected so I was looking at the output. Decided to try putting the result inside another list.
#view_config(route_name='csv', renderer='csv')
def csv(self):
d = datetime.now()
query = DBSession.query(table, othertable).join(othertable).join(thirdtable).filter(
thirdtable.sid == 9701)
header = ['First Name', 'Last Name']
rows = []
filename = "csvreport" + d.strftime(" %m/%d").replace(' 0', '')
for i in query:
items = [i.table.first_name, i.table.last_name, i.othertable.login_time.strftime("%m/%d/%Y"),
]
rows.append(items)
return {
'header': header,
'rows': rows,
'filename': filename
}
This accomplishes 3 things. Fills out the header, fills the rows, and passes through a filename.
Renderer should look like this:
class CSVRenderer(object):
def __init__(self, info):
pass
def __call__(self, value, system):
fout = StringIO.StringIO()
writer = csv.writer(fout, delimiter=',',quotechar =',',quoting=csv.QUOTE_MINIMAL)
writer.writerow(value['header'])
writer.writerows(value['rows'])
resp = system['request'].response
resp.content_type = 'text/csv'
resp.content_disposition = 'attachment;filename='+value['filename']+'.csv'
return fout.getvalue()
This way, you can use the same csv renderer anywhere else and be able to pass through your own filename. It's also the only way I could figure out how to get the data from one column in the database to iterate through one column in the renderer. It feels a bit hacky but it works and works well.