We Keep Coding

Python: Create a list in which each element is a list with a name

before you all tell me to search for stuff, let me tell you "I HAVE".
I have searched quite a bit. Forums, educational/training sites, stackexchange and all the regulars. I haven't been able to get what I need. I am not running to SE at the first sign of trouble either. I have registered on SE quite a while ago and this is my first question.
I have found some replies(suggestions, not solutions) and some ANSWERS(sort of) too. However, the most useful thing I have found is https://stackoverflow.com/a/6181978/15065496
I do know what dictionaries are, I know how to use them.
I have tried the answer above too, but it isn't exactly what I want.
I am a mechanical engineer learning python to solve some engineering problems that I am having. Please read till the end to get a better idea of what I am trying to achieve.
I need to create a list of lists with names. I have been able to create a list of lists, which looks like this:
list_xyz = []
for i in range(1,5,1):
temp_array_name = "list_" + str(i)
list_xyz.append(temp_array_name)
>>> list_xyz
['list_1', 'list_2', 'list_3', 'list_4']
Now, I want to create a new list out of each of the elements.
Basically, what I want is to create lists called list_1, list_2, list_3, list_4.
I want to automate the task of doing
list_1 = []
list_2 = []
list_3 = []
list_4 = []
what I have already tried are:
for i in list_xyz:
i = list()
*this gives:*
>>> list_xyz
['list_1', 'list_2', 'list_3', 'list_4']
*I thought good*. Then I typed:
>>> list_1
*I was expecting the following *
[]
*Instead, I got:*
Traceback (most recent call last):
File "<pyshell#93>", line 1, in <module>
list_1
NameError: name 'list_1' is not defined
*Then I tried the following and understood what had happened.*
>>> i
[]
*So then, I tried:*
for i in range(0,len(list_xyz),1):
list_xyz[i] = list()
*this gave:*
>>> list_xyz
[[], [], [], []]
Now, technically I could use the elements of list_xyz, but calling them would be a pain.
If I need the third element inside the second element of list_xyz, then I would have to say
list_xyz[1][2]
Instead, if they were named, I could just say list_2[2] and could get the element I was looking for.
I have also tried using dictionaries, as suggested by https://stackoverflow.com/users/455276/the-wolf in https://stackoverflow.com/a/6181978/15065496.
What I have tried is:
>>> dict1 = {}
>>> list_pqr = ['list_1', 'list_2', 'list_3', 'list_4']
>>> for i in range(0,len(list_pqr),1):
dict1[list_pqr[i]] = list()
>>> dict1
{'list_3': [], 'list_2': [], 'list_1': [], 'list_4': []}
This is close to what I want but isn't exactly what I want. Please read the following to understand exactly what I want.
The reason I need this particular list this exact way is:
I am trying to calculate some temperatures at different times at different positions.
I want to create a dictionary with times as keys and a list of temperatures at various locations as the values.
and another dictionary with locations as keys and list temperatures at different times as the values.
What I want is:
dict1 = {time1: temps_at_time_1
time2: temps_at_time_2
time3: temps_at_time_3
time4: temps_at_time_4
time5: temps_at_time_5
}
dict2 = {loc1: temps_at_times_L1
loc2: temps_at_times_L2
loc3: temps_at_times_L3
loc4: temps_at_times_L4
loc5: temps_at_times_L5
}
where temps_at_times_L1 is the list of temperatures at location 1 at various times.
where temp_at_time_1 is the list of temperatures at Time 1 at various locations.
let's say there are 25 locations and 100 timesteps, then:
len(temps_at_times_L1) = 100
len(temp_at_time_1) = 25
now, if temps_at_times_L1 and temp_at_time_1 are variables that I could call, It would make it very easy for me to plot the temperatures etc.
Basically, I want the same thing that I have achieved with dictionaries after using the wolf's suggestion, but the lists created there should have names that I could call so that when calculating temperatures, I could just update the lists by calling the append() method.
If I am calculating timestep1, I wanna be able to just say:
while(timestep == 1):
for i in temperature_location_array:
temps_at_time_1.append(i)
Even better would be if i was able to do the following:
for a in range(startTime, endTime, stepSize):
while(timestep == a):
for i in temperature_location_array:
temps_at_time_a.append(i) or vars("temps_at_time_" + str(a)).append(i)
I'm pretty sure the last line will not work and I will have to rack my brains again for that. I have tried reading about how vars() works but it is giving me a headache.
If you guys can't help me, and/or tell me that the last line of my code is stooooopid, then I'll just have to make do with the basic dictionary and ridiculously complicated(for my level anyway) indexing and calling and assigning directly to the values dictionaries.
I have previously faced the same need whilst editing a spreadsheet using openpyxl, but was unable to create lists from variable names inside another list. I gave up and used another method that got the job done. However, this time I would like to know how to do it: if at all it can be done.
Thank you all for the trouble of reading through my long winded explanation. I did this so people who reply will understand exactly what I need and will not give suggestions that would not serve my purpose. I will read all suggestions and take all useful ones into account though. I am not trying to offend/insult anyone.

Sorry for the negativity but appending numbers to variables is the exact opposite what any programmer should be doing - nested_lists[1][2] is the correct way of handling nested lists, nested_list_1[2] is not. Do not write data into the names of variables, just don't, it won't end well and globals() is terrible misuse here.
What you need is some proper structure that will give meaning to your data. I would recommend pandas library. It's like Excel for Python.
Let's take the following example of 5 measurements in two locations foo and bar at various times.
# Assuming the raw data is in format:
# (time, temperature, location)
import pandas as pd
data=pd.DataFrame({
(1,12, 'foo'),
(2,12,'bar'),
(10,23,'foo'),
(3,12,'foo'),
(3,15,'bar')
},columns=['time','temperature','location'])
print(data)
yields:
time temperature location
0 1 12 foo
1 2 12 bar
2 3 15 bar
3 10 23 foo
4 3 12 foo
I cannot enumerate all the things you can do with a dataframe but it can be sorted, filtered, mapped, indexed... see the documentation or search for answers here.
You might be interested in e.g. filtering data by location data[data['location']=='foo']
time temperature location
0 1 12 foo
3 10 23 foo
4 3 12 foo
or setting the dataframe's index to time column and indexing using it:
new_data = data.set_index('time')
print(new_data.loc[3])
temperature location
time
3 15 bar
3 12 foo

As several commenters are mentioned before me you actually can create variables programatically, but is a very ugly way of solving a problem.
Anyway, you can add variables to the global namespace with globals() as follows:
>>> list_xyz
['list_1', 'list_2', 'list_3', 'list_4']
>>> for n in list_xyz:
... globals()[n] = list()
...
>>> list_1
[]
>>> list_2
[]
>>> list_3
[]
>>> list_4
[]
>>>

For Loop 60 items 10 per 10

I'm working with an api that gives me 61 items that I include in a discord embed in a for loop.
As all of this is planned to be included into a discord bot using pagination from DiscordUtils, I need to make it so it male an embed for each 10 entry to avoid a too long message / 2000 character message.
Currently what I use to do my loop is here: https://api.nepmia.fr/spc/ (I recomend the usage of a parsing extention for your browser or it will be a bit hard to read it)
But what I want to create is something that will look like that : https://api.nepmia.fr/spc/formated/
So I can iterate each range in a different embed and then use pagination.
I use TinyDB to generate the JSON files I shown before with this script:
import urllib.request, json
from shutil import copyfile
from termcolor import colored
from tinydb import TinyDB, Query
db = TinyDB("/home/nepmia/Myazu/db/db.json")
def api_get():
print(colored("[Myazu]","cyan"), colored("Fetching WynncraftAPI...", "white"))
try:
with urllib.request.urlopen("https://api.wynncraft.com/public_api.php?action=guildStats&command=Spectral%20Cabbage") as u1:
api_1 = json.loads(u1.read().decode())
count = 0
if members := api_1.get("members"):
print(colored("[Myazu]","cyan"),
colored("Got expecteded answer, starting saving process.", "white"))
for member in members:
nick = member.get("name")
ur2 = f"https://api.wynncraft.com/v2/player/{nick}/stats"
u2 = urllib.request.urlopen(ur2)
api_2 = json.loads(u2.read().decode())
data = api_2.get("data")
for item in data:
meta = item.get("meta")
playtime = meta.get("playtime")
print(colored("[Myazu]","cyan"),
colored("Saving playtime for player", "white"),
colored(f"{nick}...","green"))
db.insert({"username": nick, "playtime": playtime})
count += 1
else:
print(colored("[Myazu]","cyan"),
colored("Unexpected answer from WynncraftAPI [ERROR 1]", "white"))
except:
print(colored("[Myazu]","cyan"),
colored("Unhandled error in saving process [ERROR 2]", "white"))
finally:
print(colored("[Myazu]","cyan"),
colored(f"Finished saving data for", "white"),
colored(f"{count}", "green"),
colored("players.", "white"))
but this will only create a range like this : https://api.nepmia.fr/spc/
what I would like is something like this : https://api.nepmia.fr/spc/formated/
Thanks for your help!
PS: Sorry for your eyes I'm still new to Python so I know I don't do stuff really properly :s

To follow up from the comments, you shouldn't store items in your database in a format that is specific to how you want to return results from the database to a different API, as it will make it more difficult to query in other contexts, among other reasons.
If you want to paginate items from a database it's better to do that when you query it.
According to the docs, you can iterate over all documents in a TinyDB database just by iterating directly over the DB like:
for doc in db:
...
For any iterable you can use the enumerate function to associate an index to each item like:
for idx, doc in enumerate(db):
...
If you want the indices to start with 1 as in your examples you would just use idx + 1.
Finally, to paginate the results, you need some function that can return items from an iterable in fixed-sized batches, such as one of the many solutions on this question or elsewhere. E.g. given a function chunked(iter, size) you could do:
pages = enumerate(chunked(enumerate(db), 10))
Then list(pages) gives a list of lists of tuples like [(page_num, [(player_num, player), ...].
The only difference between a list of lists and what you want is you seem to want a dictionary structure like
{'range1': {'1': {...}, '2': {...}, ...}, 'range2': {'11': {...}, ...}}
This is no different from a list of lists; the only difference is you're using dictionary keys to give numerical indices to each item in a collection, rather than the indices being implict in the list structure. There's many ways you can go from a list of lists to this. The easiest I think is using a (nested) dict comprehension:
{f'range{page_num + 1}': {str(player_num + 1): player for player_num, player in page}
for page_num, page in pages}
This will give output in exactly the format you want.

Thanks #Iguananaut for your precious help.
In the end I made something similar from your solution using a generator.
def chunker(seq, size):
for i in range(0, len(seq), size):
yield seq[i:i+size]
def embed_creator(embeds):
pages = []
current_page = None
for i, chunk in enumerate(chunker(embeds, 10)):
current_page = discord.Embed(
title=f'**SPC** Last week online time',
color=3903947)
for elt in chunk:
current_page.add_field(
name=elt.get("username"),
value=elt.get("play_output"),
inline=False)
current_page.set_footer(
icon_url="https://cdn.discordapp.com/icons/513160124219523086/a_3dc65aae06b2cf7bddcb3c33d7a5ecef.gif?size=128",
text=f"{i + 1} / {ceil(len(embeds) / 10)}"
)
pages.append(current_page)
current_page = None
return pages
Using embed_creator I generate a list named pages that I can simply use with DiscordUtils paginator.

Problem when deleting records from mongodb using pymongo

So I have some 50 document ID's. My python veno list contains document ID's as shown below.
5ddfc565bd293f3dbf502789
5ddfc558bd293f3dbf50263b
5ddfc558bd293f3dbf50264f
5ddfc558bd293f3dbf50264d
5ddfc565bd293f3dbf502792
But when I am trying to delete those 50 document ID's then I am finding a hard time. Let me explain - I need to run my python script over and over again in order to delete all the 50 documents. The first time I run my script it will delete some 10, the next time I run then it deletes 18 and so on. My for loop is pretty simple as shown below
for i in veno:
vv = i[0]
db.Products2.delete_many({'_id': ObjectId(vv)})

If your list is just the ids, then you want:
for i in veno:
db.Products2.delete_many({'_id': ObjectId(i)})
full example:
from pymongo import MongoClient
from bson import ObjectId
db = MongoClient()['testdatabase']
# Test data setup
veno = [str(db.testcollection.insert_one({'a': 1}).inserted_id) for _ in range(50)]
# Quick peek to see we have the data correct
for x in range(3): print(veno[x])
print(f'Document count before delete: {db.testcollection.count_documents({})}')
for i in veno:
db.testcollection.delete_many({'_id': ObjectId(i)})
print(f'Document count after delete: {db.testcollection.count_documents({})}')
gives:
5ddffc5ac9a13622dbf3d88e
5ddffc5ac9a13622dbf3d88f
5ddffc5ac9a13622dbf3d890
Document count before delete: 50
Document count after delete: 0

I dont have any mongo instance to test but what about
veno = [
'5ddfc565bd293f3dbf502789',
'5ddfc558bd293f3dbf50263b',
'5ddfc558bd293f3dbf50264f',
'5ddfc558bd293f3dbf50264d',
'5ddfc565bd293f3dbf502792',
]
# Or for your case (Whatever you have in **veno**)
veno = [vv[0] for vv in veno]
####
db.Products2.delete_many({'_id': {'$in':[ObjectId(vv) for vv in veno]}})
If this doesnt work, then maybe this
db.Products2.remove({'_id': {'$in':[ObjectId(vv) for vv in veno]}})
From what I understand, delete_many's first argument is filter, so
its designed in such a way, that you dont delete particular documents
but instead documents that satisfies particular condition.
In above case, best is delete all documents at once by saying -> delete all documents whose _id is in ($in) the list [ObjectId(vv) for vv in veno]

Create a summary Pandas DataFrame using concat/append via a for loop

Can't get my mind around this...
I read a bunch of spreadsheets, do a bunch of calculations and then want to create a summary DF from each set of calculations. I can create the initial df but don't know how to control my loops so that I
create the initial DF (1st time though the loop)
If it has been created append the next DF (last two rows) for each additional tab.
I just can't wrap my head how to create the right nested loop so that once the 1st one is done the subsequent ones get appended?
My current code looks like: (which just dumbly prints each tab's results separately rather than create a new consolidated sumdf with just the last 2 rows of each tabs' results..
#make summary
area_tabs=['5','12']
for area_tabs in area_tabs:
actdf,aname = get_data(area_tabs)
lastq,fcast_yr,projections,yrahead,aname,actdf,merged2,mergederrs,montdist,ols_test,mergedfcst=do_projections(actdf)
sumdf=merged2[-2:]
sumdf['name']= aname #<<< I'll be doing a few more calculations here as well
print sumdf
Still a newb learning basic python loop techniques :-(

Often a neater way than writing for loops, especially if you are planning on using the result, is to use a list comprehension over a function:
def get_sumdf(area_tab): # perhaps you can name better?
actdf,aname = get_data(area_tab)
lastq,fcast_yr,projections,yrahead,aname,actdf,merged2,mergederrs,montdist,ols_test,mergedfcst=do_projections(actdf)
sumdf=merged2[-2:]
sumdf['name']= aname #<<< I'll be doing a few more calculations here as well
return sumdf
[get_sumdf(area_tab) for area_tab in areas_tabs]
and concat:
pd.concat([get_sumdf(area_tab) for area_tab in areas_tabs])
or you can also use a generator expression:
pd.concat(get_sumdf(area_tab) for area_tab in areas_tabs)
.
To explain my comment re named tuples and dictionaries, I think this line is difficult to read and ripe for bugs:
lastq,fcast_yr,projections,yrahead,aname,actdf,merged2,mergederrs,montdist,ols_test,mergedfcst=do_projections(actdf)
A trick is to have do_projections return a named tuple, rather than a tuple:
from collections import namedtuple
Projection = namedtuple('Projection', ['lastq', 'fcast_yr', 'projections', 'yrahead', 'aname', 'actdf', 'merged2', 'mergederrs', 'montdist', 'ols_test', 'mergedfcst'])
then inside do_projections:
return (1, 2, 3, 4, ...) # don't do this
return Projection(1, 2, 3, 4, ...) # do this
return Projection(last_q=last_q, fcast_yr=f_cast_yr, ...) # or this
I think this avoids bugs and is a lot cleaner, especially to access the results later.
projections = do_projections(actdf)
projections.aname

Do the initialisation outside the for loop. Something like this:
#make summary
area_tabs=['5','12']
if not area_tabs:
return # nothing to do
# init the first frame
actdf,aname = get_data(area_tabs[0])
lastq,fcast_yr,projections,yrahead,aname,actdf,merged2,mergederrs,montdist,ols_test,mergedfcst =do_projections(actdf)
sumdf=merged2[-2:]
sumdf['name']= aname
for area_tabs in area_tabs[1:]:
actdf,aname = get_data(area_tabs)
lastq,fcast_yr,projections,yrahead,aname,actdf,merged2,mergederrs,montdist,ols_test,mergedfcst =do_projections(actdf)
sumdf=merged2[-2:]
sumdf['name']= aname #<<< I'll be doing a few more calculations here as well
print sumdf
You can further improve the code by putting the common steps into a function.

urllib.urlencode problems

I’m trying to write a python script that sends a query to TweetSentiments.com API.
The idea is that it will perform like this –
Reads CSV tweet file > construct query > Interrogates API > format JSON response > writes to CSV file.
So far I’ve come up with this –
import csv
import urllib
import os
count = 0
TweetList=[] ## Creates empty list to store tweets.
TweetWriter = csv.writer(open('test.csv', 'w'), dialect='excel', delimiter=' ',quotechar='|')
TweetReader = csv.reader(open("C:\StoredTweets.csv", "r"))
for rows in TweetReader:
TweetList.append(rows)
#print TweetList [0]
for rows in TweetList:
data = urllib.urlencode(TweetList[rows])
connect = httplib.HTTPConnection("http://data.tweetsentiments.com:8080/api/analyze.json?q=")
connect.result = json.load(urllib.request("POST", "", data))
TweetWriter.write(result)
But when its run I get “line 20, data = urllib.urlencode(TweetList[rows]) Type Error: list indices must be integers, not list”
I know my list “TweetList” is storing the tweets just as I’d like but I don’t think I’m using urllib.urlencode correct. The API requires that queries are sent like –
http://data.tweetsentiments.com:8080/api/analyze.json?q= (text to analyze)
So the idea was that urllib.urlencode would simply add the tweets to the end of the address to allow a query.
The last four lines of code have become a mess after looking at so many examples. Your help would be much appreciated.

I'm not 100% sure what it is you're trying to do since I don't know what's the format of the files you are reading, but this part looks suspicious:
for rows in TweetList:
data = urllib.urlencode(TweetList[rows])
since TweetList is a list, the for loop puts in the rows one single value from the list in each iteration, and so this for example:
list = [1, 2, 3, 4]
for num in list:
print num
will print 1 2 3 4. But if this:
list = [1, 2, 3, 4]
for num in list:
print list[num]
Will end up with this error: IndexError: list index out of range.
Can you please elaborate a bit more about the format of the files you are reading?
Edit
If I understand you correctly, you need something like this:
tweets = []
tweetReader = csv.reader(open("C:\StoredTweets.csv", "r"))
for row in tweetReader:
tweets.append({ 'tweet': row[0], 'date': row[1] })
for row in tweets:
data = urllib.urlencode(row)
.....