I'm running some experiments in a virtual machine, which has its system time updated if its suspended. I want to know if I can suspend the virtual machine and not affect the timer. That is, does Timer use system time or wall time?
I've tried looking through the source code and got to _thread.lock.acquire before the code dips into C.
Below is my code. I delegate to a subprocess that outputs 'plans'. I keep collecting these plans until the optimal plan is found or the maximum allowed time has elapsed. The timer is used to terminate the process if time runs out (which is the expected state of affairs). I need to know that the Timer will not be effected by system time being updated as it will invalidate the experiment I'm running.
p = Popen(args, stdout=PIPE, cwd=self.working_directory)
timer = Timer(float(self.planning_time), p.terminate)
timer.start()
plan = None
while True:
try:
plan = list(decode_plan_from_optic(self.decode(p.stdout),
report_incomplete_plan=True))
except IncompletePlanException:
break
timer.cancel()
Upon examination of the python source code for *unix it systems, I found that python eventually delegates to sem_timedwait from semaphore.h or pthread_cond_timedwait from pthread.h depending on support. Either way, both functions take a struct timespec from time.h as an absolute time to wait till -- timespec is the number of seconds and nanoseconds since the epoch.
So on the face of it seems that waiting in python is dependent on system time -- meaning my program would be effected by a change in system time. However, time.h specifies the constant CLOCK_MONOTONIC_RAW and the function clock_gettime, if a monotonically increasing clock is required. Showing there is an ability to wait independently of system time. However, sadly python uses gettimeofday (marked as obsolete since 2008) which is affected by changes to system time.
In short, waiting in python on *unix systems is effected by changes to system time.
Related
I am working with a groundwater modeling executable (HYDRUS1D) which I call with a Python script. I want to do some Monte Carlo runs but sometimes the program gets hung up and does not converge for extended periods of time.
Is there a way to give the executable a certain amount of time to run, cancel it if it goes over this time, and then start a new simulation all without interrupting the Python script? The simulation should take no more than 3-5 seconds, so I am hoping to give it a maximum of 10 seconds to finish.
I first run a function that changes some input parameters to the model, then execute Hydrus via the 'run_single_sim' function:
for value in n_variations_21:
for value2 in n_variations_23:
write_hydraulic_params('foo',layers,value,value2)
run_single_sim()
Where run_single_sim() executes Hydrus via os.system:
def run_single_sim():
os.system('./hydrus LEVEL_01.DIR')
I have tried a few solutions involving threading such as this, and this; but it seems like my script gets stuck on the os.system call and therefore cannot check to see how long the thread has been running or kill the thread after sleeping the script for some specified amount of time.
You asked "how to stop an executable called via Python ...", but I feel
this question is simply about "how to stop an executable".
What's interesting is that we have a child that might misbehave.
The parent is uninteresting, could be rust, ruby, random other language.
The timeout issue you pose is a sensible question,
and there's a stock answer for it, in the GNU coreutils package.
Instead of
os.system('./hydrus LEVEL_01.DIR')
you want
os.system('timeout 10 ./hydrus LEVEL_01.DIR')
Here is a quick demo, using a simpler command than hydrus.
$ timeout 2 sleep 1; echo $?
0
$
$ timeout 2 sleep 3; echo $?
124
As an entirely separate matter, prefer check_output()
over the old os.system().
You quoted a pair of answer articles that deal with threading.
But you're spawning a separate child process,
with no shared memory, so threading's not relevant here.
We wish to eventually send a SIGTERM signal to an ill behaved process,
and we hope it obeys the signal by quickly dropping out.
Timing out a child that explicitly ignores such signals would
be a slightly stickier problem.
An uncatchable SIGKILL can be sent
by using the --kill-after=duration flag.
I'd like to know how'd you measure the amount of clock cycles per instruction say copy int from one place to another?
I know you can time it down to nano seconds but with today's cpu's that resolution is too low to get a correct reading for the oprations that take just a few clock cycles?
It there a way to confirm how many clock cycles per instructions like adding and subing it takes in python? if so how?
This is a very interesting question that can easily throw you into the rabbit's hole. Basically any CPU cycle measurements depends on your processors and compilers RDTSC implementation.
For python there is a package called hwcounter that can be used as follows:
# pip install hwcounter
from hwcounter import Timer, count, count_end
from time import sleep
# Method-1
start = count()
# Do something here:
sleep(1)
elapsed = count_end() - start
print(f'Elapsed cycles: {elapsed:,}')
# Method-2
with Timer() as t:
# Do something here:
sleep(1)
print(f'Elapsed cycles: {t.cycles:,}')
NOTE:
It seem that the hwcounter implementation is currently broken for Windows python builds. A working alternative is to build the pip package using the mingw compiler, instead of MS VS.
Caveats
Using this method, always depend on how your computer is scheduling tasks and threads among its processors. Ideally you'd need to:
bind the test code to one unused processor (aka. processor affinity)
Run the tests over 1k - 1M times to get a good average.
Need a good understanding of not only compilers, but also how python optimize its code internally. Many things are not at all obvious, especially if you come from C/C++/C# background.
Rabbit Hole:
http://en.wikipedia.org/wiki/Time_Stamp_Counter
https://github.com/MicrosoftDocs/cpp-docs/blob/main/docs/intrinsics/rdtsc.md
How to get the CPU cycle count in x86_64 from C++?
__asm
__rdtsc
__cpuid, __cpuidex
Defining __asm Blocks as C Macros
My script has to run over a day and its core cycle runs 2-3 times per a minute. I used multiprocessing to give a command simultaneously and each of them will be terminated/join within one cycle.
But in reality I found the software end up out of swap memory or computer freezing situation, I guess this is caused by accumulated processes. I can see on another session while running program, python PID abnormally increasing by time. So I just assume this must be something process thing. What I don't understand is how it happens though I made sure each cycle's process has to be finished on that cycle before proceed the next one.
so I am guessing, actual computing needs more time to progress 'terminate()/join()' job, so I should not "reuse" same object name. Is this proper guessing or is there other possibility?
def function(a,b):
try:
#do stuff # audio / serial things
except:
return
flag_for_2nd_cycle=0
for i in range (1500): # main for running long time
#do something
if flag_for_2nd_cycle==1:
while my_process.is_alive():
if (timecondition) < 30: # kill process if it still alive
my_process.terminate()
my_process.join()
flag_for_2nd_cycle=1
my_process=multiprocessing.process(target=function, args=[c,d])
my_process.start()
#do something and other process jobs going on, for example
my_process2 = multiprocessing.process() ##*stuff
my_process2.terminate()
my_process2.join()
Based on your comment, you are controlling three projectors over serial ports.
The simplest way to do that would be to simply open three serial connections (using pySerial). Then run a loop where you check for available data each of the connections and if so, read and process it. Then you send commands to each of the projectors in turn.
Depending on the speed of the serial link you might not need more than this.
I am using python's sched module to run a task periodically, and I think I have come across a bug.
I find that it relies on the time of the system on which the python script is run. For example, let's say that I want to run a task every 5 seconds. If I forward the system time, the scheduled task will run as expected. However, if I rewind the system time to, say 1 day, then the next scheduled task will run in 5 seconds + 1 day.
If you run the script below and then change your system time by a few days back, then you can reproduce the issue. The problem can be reproduced on Linux and Windows.
import sched
import time
import threading
period = 5
scheduler = sched.scheduler(time.time, time.sleep)
def check_scheduler():
print time.time()
scheduler.enter(period, 1, check_scheduler, ())
if __name__ == '__main__':
print time.time()
scheduler.enter(period, 1, check_scheduler, ())
thread = threading.Thread(target=scheduler.run)
thread.start()
thread.join()
exit(0)
Anyone has any python solution around this problem?
From the sched documentation:
class sched.scheduler(timefunc, delayfunc)
The scheduler class defines a generic interface to scheduling events. It needs two functions to actually deal with the “outside
world” — timefunc should be callable without arguments, and return a
number (the “time”, in any units whatsoever). The delayfunc function
should be callable with one argument, compatible with the output of
timefunc, and should delay that many time units. delayfunc will also
be called with the argument 0 after each event is run to allow other
threads an opportunity to run in multi-threaded applications.
The problem you have is that your code uses time.time() as timefunc, whose return value (when called without arguments) is the current system time and is thus affected by re-winding the system clock.
To make your code immune to system time changes you'd need to provide a timefunc which doesn't depend on the system time, start/current timestamps, etc.
You can write your own function, for example one returning the number of seconds since your process is started, which you'd have to actually count in your code (i.e. don't compute it based on timestamp deltas). The time.clock() function might help, if it's based on CPU time counters, but I'm not sure if that's true or not.
I am working on a Opencv based Python project. I am working on program development which takes less time to execute. For that i have tested my small program print hello world on python to test the time taken to run the program. I had run many time and every time it run it gives me a different run time.
Can you explain me why a simple program is taking different time to execute?
I need my program to be independent of system processes ?
Python gets different amounts of system resources depending upon what else the CPU is doing at the time. If you're playing Skyrim with the highest graphics levels at the time, then your script will run slower than if no other programs were open. But even if your task bar is empty, there may be invisible background processes confounding things.
If you're not already using it, consider using timeit. It performs multiple runs of your program in order to smooth out bad runs caused by a busy OS.
If you absolutely insist on requiring your program to run in the same amount of time every time, you'll need to use an OS that doesn't support multitasking. For example, DOS.