Related
I developed a python script to send JSON messages to an IoT infrastructure. The message incl. a lot of simulated sensor values that are based on a "master" device. There is one valuepair to measure the filling degree of a container [0 - 100 %].
The valuepair should increment starting with 0 to 100 over a few days. The script runs every 20 secs.
Currently I pass a "static" value but I would like to simulate an increase of the value with the result that every simulated device has a different filllevel over 24 hours. I would like to use something as a "trend" to define how much time the device should take to reach 100%.
If each device has a time t (in seconds) that it would take to fill, then lets say T is a tuple of all the devices T = (t1, t2, .... tn,)
On average, the number of seconds that each device would need to increment (if increment in integer percentages) would be t/100.
So for each iteration, there should be a probability 20/t/100 of an increment.
So for N devices, to return a list of (device #, do I increment?) pairs you could do this.
from random import random
class devices():
def __init__(self, T, interval=20):
self.devices = N * [0]
self.T = T
def iteration(self):
for i in range(len(self.devices)):
self.devices[i] = min(100, self.devices[i] + \
(1 if random() < (interval * 100 / self.T[i])) else 0)
return [[i, self.devices[i] for i in range(len(self.T))]
To use it
devs = devices([t1, t2, t3, ..., tn])
then every 20 seconds
state = devs.iteration()
There may be other factors you want like not having a purely random selection. I don't know what kind of devices you are simulating but perhaps the process that fills them has time-of-day, weather, or other factors that might be incorporated into the simulation
I have a question .
I'm making a platformer game in tkinter and I have an issue :
I have for now : player , blocks and coins .
I'm updating the player's move and it's animation and the coin's animation and for some reason when I'm putting too much coins , the player's movement starts lagging.
Note: I'm using the after function of tkinter for animations for player's movement + animation and same goes for the coins .
For other things like gravity and etc I'm using just threads .
code of coins updating :
def coinsCheckCollision(self):
cRemove = None
indexRemove = -1
count = 0
for c in self.frame.coins:
x, y , width , height = c.getRectangle()
xP = self.player.getX; yP = self.player.getY; wP = self.player.getWidth; hP = self.player.getHeight
if collisionDetect(xP , x, yP , y, wP , width, hP , height) or collisionDetect(x , xP , y , yP , width , wP , height , hP):
if count not in coinsRemoved:
indexRemove = count
if indexRemove != -1:
if indexRemove not in coinsRemoved:
coinsRemoved.append(indexRemove)
count +=1
def coinsUpdateAnimations(self):
count = 0
for c in self.frame.coins:
if count not in coinsRemoved:
self.img = c.getAnimation()
self.img = ImageTk.PhotoImage(self.img)
self.frame.coinsImages[count] = self.img
else:
if self.frame.coinsImages[count] is not '' :
self.frame.coinsImages[count] = ''
self.frame.canvas.delete('coinB'+str(count))
what = self.frame.canvas.itemconfig('coin' + str(count), image=self.frame.coinsImages[count])
count += 1
self.coinsCheckCollision()
self.frame.frame.after(40 , self.coinsUpdateAnimations)
Anyway , the question in short is : why when I'm updating multiple things that aren't really "related" to each other , the gui starts lagging ?
Your design seems to expect your functions to run every 40ms. Maybe +/- a few ms, but averaging 25 times per second.
But that's not what happens.
First, how many coins do you have, and how complicated is that collisionDetect function? If it only takes a tiny fraction of 1ms to run through that loop, it's no big deal, but think about what happens if it takes, say, 15ms: You wait 40ms, then do 15ms of work, then wait another 40ms, then do 15ms of work, etc. So your work is running only 15 times per second, instead of 25.
Now imagine each coin takes, say, 0.2ms. At 3 coins, there's a lag of 0.6ms, which is barely noticeably. But at 100 coins, there's a lag of 20ms. That slows the coins down by 50%, which is pretty obviously noticeable.
Second, as the docs say:
Tkinter only guarantees that the callback will not be called earlier than that; if the system is busy, the actual delay may be much longer.
Being off a few ms randomly in either direction might be fine; it would all average out in the end. But after is always a few ms late, never a few ms early, so instead of averaging out, it just builds up and you get further and further behind.
And, worse, if one of your functions gets behind, it will tend to make the delay in each after a bit longer—so it won't just be your coin animation slowing down 50%, but the whole game slowing down by some unpredictable amount arbitrarily between 0-50%, but probably enough to be noticeable.
To solve both problems, you need to carry around something like the time you expected to run at, then, instead of doing after(40), you do something like this:
expected_time += 40
delay = expected_time - current_time
after(max(0, delay), func)
To put it in concrete (although untested) terms, using the datetime module:
def __init__(self):
self.next_frame_time = datetime.datetime.now()
self.schedule()
def schedule(self):
self.next_frame_time += datetime.timedelta(seconds=0.040)
now = datetime.datetime.now()
delta = max(datetime.timedelta(), now - self.next_frame_time)
self.frame.frame.after(delta.total_seconds * 1000, self.coinsUpdateAnimations)
def coinsUpdateAnimations(self):
# all the existing code before the last two lines
self.coinsCheckCollision()
self.schedule()
This still won't solve things if the total work you do takes more than 40ms, of course. Imagine that you spend 50ms, then do an after(0, func), which triggers at least 10ms late, and then spend another 50ms, then the next after(0, func) triggers at least 20ms late, and so on. If you can't do all of your work in something that's usually significantly less than 40ms, you won't be able to keep up. You have to either:
Find a way to optimize your code (e.g., maybe you can use a better algorithm, or use numpy instead of a for loop),
Redesign your game to do less work, or
Slow down your frame rate to something you actually can keep up with.
A possibly better solution is to stop trying to bend Tkinter into a gaming framework. It's not designed for that, doesn't help you get all the fiddly details right, and doesn't work all that well even once you do get them right.
By contrast, something like Pygame Zero is, as the name implies, designed for creating games. And designed to make it easy enough that people with a lot less Python experience than you seem to have can use it.
For example, instead of an event loop that runs at whatever speed your OS wants to run it, making it your responsibility to get everything timed right, Pygame Zero runs a frame loop that calls your update function N times per second, as close to evenly as possible. And it has built-in functions for things like collision detection, drawing animated sprites, etc.
I am running an algorithm which reads an excel document by rows, and pushes the rows to a SQL Server, using Python. I would like to print some sort of progression through the loop. I can think of two very simple options and I would like to know which is more lightweight and why.
Option A:
for x in xrange(1, sheet.nrows):
print x
cur.execute() # pushes to sql
Option B:
for x in xrange(1, sheet.nrows):
if x % some_check_progress_value == 0:
print x
cur.execute() # pushes to sql
I have a feeling that the second one would be more efficient but only for larger scale programs. Is there any way to calculate/determine this?
I'm a newbie, so I can't comment. An "answer" might be overkill, but it's all I can do for now.
My favorite thing for this is tqdm. It's minimally invasive, both code-wise and output-wise, and it gets the job done.
I am one of the developers of tqdm, a Python progress bar that tries to be as efficient as possible while providing as many automated features as possible.
The biggest performance sink we had was indeed I/O: printing to the console/file/whatever.
But if your loop is tight (more than 100 iterations/second), then it's useless to print every update, you'd just as well print just 1/10 of the updates and the user would see no difference, while your bar would be 10 times less overhead (faster).
To fix that, at first we added a mininterval parameter which updated the display only every x seconds (which is by default 0.1 seconds, the human eye cannot really see anything faster than that). Something like that:
import time
def my_bar(iterator, mininterval=0.1)
counter = 0
last_print_t = 0
for item in iterator:
if (time.time() - last_print_t) >= mininterval:
last_print_t = time.time()
print_your_bar_update(counter)
counter += 1
This will mostly fix your issue as your bar will always have a constant display overhead which will be more and more negligible as you have bigger iterators.
If you want to go further in the optimization, time.time() is also an I/O operation and thus has a cost greater than simple Python statements. To avoid that, you want to minimize the calls you do to time.time() by introducing another variable: miniters, which is the minimum number of iterations you want to skip before even checking the time:
import time
def my_bar(iterator, mininterval=0.1, miniters=10)
counter = 0
last_print_t = 0
last_print_counter = 0
for item in iterator:
if (counter - last_print_counter) >= miniters:
if (time.time() - last_print_t) >= mininterval:
last_print_t = time.time()
last_print_counter = counter
print_your_bar_update(counter)
counter += 1
You can see that miniters is similar to your Option B modulus solution, but it's better fitted as an added layer over time because time is more easily configured.
With these two parameters, you can manually finetune your progress bar to make it the most efficient possible for your loop.
However, miniters (or modulus) is tricky to get to work generally for everyone without manual finetuning, you need to make good assumptions and clever tricks to automate this finetuning. This is one of the major ongoing work we are doing on tqdm. Basically, what we do is that we try to calculate miniters to equal mininterval, so that time checking isn't even needed anymore. This automagic setting kicks in after mininterval gets triggered, something like that:
from __future__ import division
import time
def my_bar(iterator, mininterval=0.1, miniters=10, dynamic_miniters=True)
counter = 0
last_print_t = 0
last_print_counter = 0
for item in iterator:
if (counter - last_print_counter) >= miniters:
cur_time = time.time()
if (cur_time - last_print_t) >= mininterval:
if dynamic_miniters:
# Simple rule of three
delta_it = counter - last_print_counter
delta_t = cur_time - last_print_t
miniters = delta_it * mininterval / delta_t
last_print_t = cur_time
last_print_counter = counter
print_your_bar_update(counter)
counter += 1
There are various ways to compute miniters automatically, but usually you want to update it to match mininterval.
If you are interested in digging more, you can check the dynamic_miniters internal parameters, maxinterval and an experimental monitoring thread of the tqdm project.
Using the modulus check (counter % N == 0) is almost free compared print and a great solution if you run a high frequency iteration (log a lot).
Specially if you does not need to print for each iteration but want some feedback along the way.
I'm writing a disease simulation in Python, using SQLalchemy, but I'm hitting some performance issues when running queries on a SQLite file I create earlier in the simulation.
The code is below. There are more queries in the outer for loop, but what I've posted is what slowed it down to a crawl. There are 365 days, about 76,200 mosquitos, and each mosquito makes 5 contacts per day, bringing it to about 381,000 queries per simulated day, and 27,813,000 through the entire simulation (and that's just for the mosquitos). It goes along at about 2 days / hour which, if I'm calculating correctly, is about 212 queries per second.
Do you see any issues that could be fixed that could speed things up? I've experimented with indexing the fields which are used in selection but that didn't seem to change anything. If you need to see the full code, it's available here on GitHub. The function begins on line 399.
Thanks so much, in advance.
Run mosquito-human interactions
for d in range(days_to_run):
... much more code before this, but it ran reasonably fast
vectors = session.query(Vectors).yield_per(1000) #grab each vector..
for m in vectors:
i = 0
while i < biting_rate:
pid = random.randint(1, number_humans) # Pick a human to bite
contact = session.query(Humans).filter(Humans.id == pid).first() #Select the randomly-chosen human from SQLite table
if contact: # If the random id equals an ID in the table
if contact.susceptible == 'True' and m.infected == 'True' and random.uniform(0, 1) < beta: # if the human is susceptible and mosquito is infected, infect the human
contact.susceptible = 'False'
contact.exposed = 'True'
elif contact.infected == 'True' and m.susceptible == 'True': # otherwise, if the mosquito is susceptible and the human is infected, infect the mosquito
m.susceptible = 'False'
m.infected = 'True'
nInfectedVectors += 1
nSuscVectors += 1
i += 1
session.commit()
I'd like to create a revenue counter for the sales team at work and would love to use Python. E.g. Joe Bloggs shifts his target from 22.1 to 23.1 (difference of 1.0.) I'd like the counter to tick evenly from 22.1 to 23.1 over an hour.
I've created this script, which works fine for counting a minute (runs 2 seconds over the minute); however, when it's supposed to run for an hour, it runs for 47 minutes.
Question: Does anyone know why it runs faster when I set it to an hour? Is sleep.time inaccurate?
import time
def rev_counter(time_length):
time_start = (time.strftime("%H:%M:%S"))
prev_pp = 22.1
new_pp = 23.1
difference = new_pp - prev_pp
iter_difference = (difference / 100000.) # Divide by 100,000 to show 10 decimal places
time_difference = ((time_length / difference) / 100000.)
i = prev_pp
while i < new_pp:
print("%.10f" % i)
i = i + iter_difference
time.sleep(time_difference)
time_end = (time.strftime("%H:%M:%S"))
print "Time started at", time_start
print "Time ended at", time_end
rev_counter(60) # 60 seconds. Returns 62 seconds
rev_counter(600) # 10 minutes. Returns 10 minutes, 20 secs
rev_counter(3600) # 1 hour. Returns 47 minutes
Please note this quote from the Python documentation for time.sleep()
The actual suspension time may be less than that requested because any
caught signal will terminate the sleep() following execution of that
signal's catching routine. Also, the suspension time may be longer
than requested by an arbitrary amount because of the scheduling of
other activity in the system.
As a suggestion, if faced with this problem, I would use a variable to track the time that the interval starts. When sleep wakes up, check to see if the expected time has elapsed. If not, restart a sleep for the difference, etc.
First of all, your loop doesn't only contain sleep statements -- the things you do between calling time.sleep take time, too, so if you do 10 repetions, you'll spent only 10% of the time doing these compared to when you have 100 iterations through your loop.
Is sleep.time inaccurate?
Yes. Or well. Quite.
I come from a real-time signal processing background. PC clocks are only somewhat accurate, and the time you spend in your OS, your standard libraries, your scripting language run time and your scripting logic between the point in time when a piece of hardware notifies you that your time has elapsed and the point in time your software notices is significant.
I just noticed time.sleep taking way too long (5-30000 times longer for input values between .0001 to 1 second), and searching for an answer, found this thread. I ran some tests and it is consistently doing this (see code and results below). The weird thing is, I restarted, then it was back to normal, working very accurately. When code started to hang it was time.sleep taking 10000 times too long?!
So a restart is a temporary solution, but not sure what the cause is/ permanent solution is.
import numpy as np
import time
def test_sleep(N,w):
data = []
for i in xrange(N):
t0 = time.time()
time.sleep(w)
t1 = time.time()
data.append(t1-t0)
print "ave = %s, min = %s, max = %s" %(np.average(data), np.min(data), np.max(data))
return data
data1 = test_sleep(20,.0001)
Out: ave = 2.95489487648, min = 1.11787080765, max = 3.23506307602
print data1
Out: [3.1929759979248047,
3.121081829071045,
3.1982388496398926,
3.1221959590911865,
3.098078966140747,
3.131525993347168,
3.12644100189209,
3.1535091400146484,
3.2167508602142334,
3.1277999877929688,
3.1103289127349854,
3.125699996948242,
3.1129801273345947,
3.1223208904266357,
3.1313750743865967,
3.1280829906463623,
1.117870807647705,
1.3357980251312256,
3.235063076019287,
3.189779043197632]
data2 = test_sleep(20, 1)
Out: ave = 9.44276217222, min = 1.00008392334, max = 10.9998381138
print data2
Out: [10.999573945999146,
10.999622106552124,
3.8115758895874023,
1.0000839233398438,
3.3502109050750732,
10.999613046646118,
10.99983811378479,
10.999617099761963,
10.999662160873413,
10.999619960784912,
10.999650955200195,
10.99962306022644,
10.999721050262451,
10.999620914459229,
10.999532222747803,
10.99965500831604,
10.999596118927002,
10.999563932418823,
10.999600887298584,
4.6992621421813965]