We Keep Coding

automatically remove items of a list every few second in redis

I'm trying to see how many users used my bot in the last 5 minute.
I've got an idea to every time a user used my bot I add his/hers id into a redis list with a timer. (reset the timer if user is already in the list)
And every time I want to check how many users are using the bot, I get the length of the list.
But i have no idea how to do that.
something like below code that expiers five minute later:
redis.setex('foo_var', 60 * 5, 'foo_value')
I've managed to add items to a list with :
redis.zadd('foo', {'item1': 0, 'item2': 1})
And get the length of the list like this (I don't know how to get full length of the list. (without using min and max)):
min = 0.0
max = 1000.0
redis.zcount('foo', min, max)
Right now the problem is how to expire items of a list on specific time.

Items within Lists, Sets, Hashes, and their ilk cannot be expired automatically by Redis. That said, it might be worth looking at Streams.
If you're not familiar, Streams are essentially a list of events with associated times and associated data. Think of it like a log file. The nice thing is you can add extra data to the event, like maybe the type of bot interaction.
So, just log an event every time the bot is used:
XADD bot_events * eventType login
The * here means to auto generate the ID of the event based on the server time. You can also provided one manually, but you almost never want to. An event ID is just a UNIX Epoch time in milliseconds and a sequence number separated by a dash like this: 1651232477183-0.
XADD can automatically trim the Stream for your time period so old record don't hang around. Do this by providing an event ID before which events will be deleted:
XADD bot_events MINID ~ 1651232477183-0 * eventType login
Note that ~ instructs Redis to trim Streams performantly. This means that it might not delete all the events. However, it will never delete more than you expect, only less. It can be replaced with = if you want exactness over performance.
Now that you have a Stream of events, you can then query that Stream for events over a specific time period, based on event IDs:
XRANGE bot_events 1651232477183-0 +
+ here, means until the end of the stream. The initial event ID could be replaces with - if you want all the stream events regardless of the time.
From here, you just count the number of results.
Note, all the examples here are presented as raw Redis commands, but it should be easy enough to translate them to Python.

Using a variable as a dict key

I have JSON objects coming in:
{"Name":"server1","NodeID":1063,"RowID":"3","Vendor":"HP","Load":"41"},
{"Name":"server2","NodeID":1064,"RowID":"7","Vendor":"HP","Load":"12"},
{"Name":"server82","NodeID":1064,"RowID":12","Vendor":"HP","Load":"2"},
{"Name":"server242","NodeID":1064,"RowID":"4","Vendor":"HP","Load":"1"},
{"Name":"server572","NodeID":1064,"RowID":"4","Vendor":"HP","Load":"44"},
{"Name":"server8","NodeID":1064,"RowID":"2","Vendor":"HP","Load":"23"},
{"Name":"server8","NodeID":1064,"RowID":"7","Vendor":"HP","Load":"1"},
...
And I am outputting a different format, here is the dict before I feed it to urllib.request.
import json
import urllib.request
machine = {}
machine['_type'] = "general_data"
machine['_system'] = "LinuxDataCpu"
machine['_device'] = i['Name']
machine['CPUload1'] = i['Load']
If this was a bash script, CPUload$RowID would likely form the basis of the solution.
I have some machines with 1 CPU, some with 8, and some with 12.
I have a JSON object machine() that will be created and pushed out in a post.
Each JSON object will contain only one CPU value, so the key needs to reflect which cpu is being reported (1 to 12).
How do I use a variable for a key such that I can indicate a different key name for each CPU?
So the first line for server1 would get:
machine['CPUload3'] = i['Load']
And the second line gets:
machine['CPUload7'] = i['Load']
For the respective output key: value pair. Each line only reports one row at a time, so a 12 cpu machine would randomly send reports listing individual cpu's. On the back end nothing is random, but the raw sql generating the data is not ordering the data in any way. The data is actually a large block of data, but the API I am sending the json to can only take one key value pair for the data. it should let me pass the whole json object, but decisions were made out of my control, and one payload variable is allowd per post.
Michael Butscher offers: "['CPUload' + i['RowID']]"... this was what I was looking for. (yeah I'm a rookie hack with python).
Type it up Michael and I'll mark it to give you the credit.
Thanks everyone!!

Just create a string variable, pass the incremental server number via a loop and pass the string variable into the key.
Something like:
incrementalCPUnumber = 1
key = 'CPUload{0}'.format(incrementalCPUnumber)
machine[x] = i['Load']
I think you'll need to clarify more, but on face value, this is what it seems like you are trying to do.

How to check if a key already exists in the database using Firebase and Python?

Essentially, I'll be using a database of this structure:
to keep track of the users' xp. Under the xp_data section, there will be multiple timestamps and xp numbers for each timestamp. A function will run every 24 hours, that will log the users' XP. I want to have some way to check if the player is already in the database (and if so, add to their existing xp count) and if not, create a new node for them. Here is my code for writing to the server:
db_ref = db.reference('/')
for i in range(100):
tom = await mee6API.levels.get_leaderboard_page(i)
if xp_trigger:
break
this_lb_list = {}
for l in tom['players']:
if l['xp'] < 300:
xp_trigger = True
break
this_lb_list.update({l['id']: {'name': l['username'], 'xp_data': {time.strftime(time_format_str, time.gmtime()): l['xp']}}})
details += [{ int(l['id']) : l['xp']}]
print(i)
db_ref.update(this_lb_list)
Basically, this code loops through each page in the leaderboard, obtains the XP for each user, and appends it to a dict, which is then used to update the database. there are two problems with this code, one is that it does not check if the user already exists, meaning that, and this is the second problem, that it overwrites the user's existing data. I've also attempted to write the data for each player individually, but problem 1 was still an issue, and it was painfully slow. What can I do to rectify this?

When you pass a value for a property in update(), that value replaces the entire existing value of the property in the database. So while update() leaves the properties you don't specify in the call unmodified, it does completely replace any property you do specify.
To add a value to an existing property, you'll want to specify the entire path as the key, separating the various child nodes with /.
So something like:
this_lb_list.update({'xp_data/13-Auth-2021': l['xp']})
This will write only the 13-Auth-2021 of xp_data, leaving all other child nodes of xp_data unmodified.
You'll of course want to use a variable for the date/time, but the important thing is that you specify it in the key, and not in the value of the dictionary.

Get code from module in Excel

I have a number of workbooks that have Macros which point to a particular SQL server using a connection string embedded in the code. We've migrated to a new SQL server so I need to go through these and alter the connection string to look at the new server in each of the Macros that explicitly mentions it.
Currently I'm able to list all of the modules in the workbook, however I'm unable to get the code from each module, just the name and type number.
for vbc in wb.VBProject.VBComponents:
print(vbc.Name + ": " + str(vbc.Type) + "\n" + str(vbc.CodeModule))
What property stores the code so that I can find and replace the server name? I've had a look through the VBA and pywin32 docs but can't find anything.

Got it- there's a Lines method in the CodeModule object that allows you to take a selection based on a starting and ending line. Using this in conjunction with the CountOfLines property allows you to get the whole thing.
for vbc in wb.VBProject.VBComponents:
print(vbc.Name + ":\n" + vbc.CodeModule.Lines(1, vbc.CodeModule.CountOfLines))
It's worth noting as well that the first line is line 1, not line 0 as that caught me out. The following will error vbc.CodeModule.Lines(0, vbc.CodeModule.CountOfLines - 1) because the index 0 is out of range.

Method Lines of property CodeModule has the signature
Function Lines(ByVal first as Integer, ByVal count as Integer) as String
first is in range 1...CodeModule.CountOfLines,
index of the first row of the code section, you want to
retrieve
count is in range 1...CodeModule.CountOfLines-first+1,
number of lines of the section
The return value is a concatenation of the code lines of the section with separator vbNewLine.

Algorithm for A/B testing

I need to develop an A/B testing method for my users. Basically I need to split my users into a number of groups - for example 40% and 60%.
I have around 1,000,00 users and I need to know what would be my best approach. Random numbers are not an option because the users will get different results each time. My second option is to alter my database so each user will have a predefined number (randomly generated). The negative side is that if I get 50 for example, I will always have that number unless I create a new user. I don't mind but I'm not sure that altering the database is a good idea for that purpose.
Are there any other solutions so I can avoid that?

Run a simple algorithm against the primary key. For instance, if you have an integer for user id, separate by even and odd numbers.
Use a mod function if you need more than 2 groups.

Well you are using MySQL so whether it's a good idea or not, it's hard to tell. Altering databases could be costly. Also it could affect performance in the long run if it starts getting bigger. Also you would have to modify your system to include that number in the database for every new user. You have tagged this as a python question. So here is another way of doing it without making any changes to the database. Since you are talking about users you probably have a unique identifier for all of them, let's say e-mail. Instead of email I'll be using uuid's.
import hashlib
def calculateab(email):
maxhash = 16**40
emailhash = int(hashlib.sha1(email).hexdigest(), 16)
div = (maxhash/100)-1
return int(float(emailhash/div))
#A small demo
if __name__ == '__main__':
import uuid, time, json
emails = []
verify = {}
for i in range(1000000):
emails.append(str(uuid.uuid4()))
starttime = time.time()
for i in emails:
ab = calculateab(i)
if ab not in verify:
verify[ab] = 1
else:
verify[ab] += 1
#json for your eye's pleasure
print json.dumps(verify, indent = 4)
#if you look at the numbers, you'll see that they are well distributed so
#unless you are going to do that every second for all users, it should work fine
print "total calculation time {0} seconds".format((time.time() - starttime))
Not that much to do with python, more of a math solution. You could use md5, sha1 or anything along those lines, as long as it has a fixed length and it's a hex number. The -1 on the 6-th line is optional - it sets the range from 0 to 99 instead of 1 to 100. You could also modify that to use floats which will give you a greater flexibility.

I would add an auxiliary table with just userId and A/B. You do not change existent table and it is easy to change the percentage per class if you ever need to. It is very little invasive.

Here is the JS one liner:
const AB = (str) => parseInt(sha1(str).slice(0, 1), 16) % 2 === 0 ? 'A': 'B';
and the result for 10 million random emails:
{ A: 5003530, B: 4996470 }