I am trying to build a browser-based debugging tool for Python but I am having trouble combining the Python inner workings with user interactions. My setup looks like this (jsfiddle link). Essentially I am using sys.settrace to inspect a function one opcode at a time. My goal is for the user to be able to manually step forward each instruction by pressing the Step button.
The problems I am having is that the tracer function cannot be asynchronous because Python does not like that, and if it is synchronous there is no real way to stop and wait for input/interaction. If I make just an infinite while loop it freezes the browsers main thread so the page becomes unresponsive.
Does anyone have advice on how I can structure this to allow interaction between the UI and the tracer function?
I managed to get a work-around working based on a service worker. I used an example implementation from here. It utilizes the fact that you can make a synchronous thread wait for an HTTP request, so we intercept that request using a service worker and make it last for as long as we need, and when we're done we can even send data back with the request.
Related
What I'm trying to accomplish:
I have a sensor that is constantly reading in data. I need to print this data to a UI whenever data appears. While the aforementioned task is taking place, the user should be able to write data to the sensor. Ideally, both these tasks would / could happen at the same time. Currently, I have the program written using flask; but if django would be better suited (or a third party) I would be willing to make the switch. Note: this website will never be deployed so no need to worry about that. Only user will be me, running program from my laptop.
I have spent a lot of time researching flask async functions and coroutines; however I have not seen any clear indications if something like this would be possible.
Not looking for a line by line solution. Rather, a way (async, threading etc) to set up the code such that the aforementioned tasks are possible. All help is appreciated, thanks.
I'm a Django guy, so I'll throw out what I think could be possible
Django has a decorator #start_new_thread which can be put on any function and it will run in a thread.
You could make a view, POST to it with Javascript/Ajax and start a thread for communication with the sensor using the data POSTed.
You could also make a threading function that will read from the sensor
Could be a management command or a 'start' btn that POSTs to a view that then starts the thread
Note: You need to do Locks or some other logic so the two threads don't conflict when reading/writing
Maybe it's a single thread that reads/writes to the sensor and each loop it checks if there's anything to write (existence + contents of a file? Maybe db entry?
Per the UI, lets say a webpage. You're best best would be Websockets, but because you're the only one that will ever use it you could just write up some Javascript/Ajax that would Ping a view every x seconds and display the new data on the webpage
Note: that's essentially what websockets do, ping every x seconds
Now the common thread is Javascript/Ajax, this is so the page doesn't need to refresh and you can constantly see the data coming in without the page being refreshed.
You can probably do all of this in Flask if you find a similar threading ability and just add some javascript to the frontend
Hopefully you find some of this useful, and idk why stackoverflow hates these types of questions... They're literally fine
so I'm currently writing a Flask application and I am new to flask.
There is some processing going on, which I outsourced to a separate function.
As this processing takes some time, I wanted to give the user a progress update on how many iterations have passed. No problem so far.
However, as soon I call the render template, the function ends and I cannot update that variable anymore.
I was imagining an if loop. if that variable changes, render template with the new variable as input.
But after the first iteration, the if loop will brake.
Currently, the render template renders an html function, which just displays the variable it receives. I want to update that variable as soon as it changes.
Do you guys have any suggestion, on how I could achieve this "background update"?
Cheers and thanks!
You need some kind of ongoing request/response cycle.
As soon as your app sends the response with the rendered template back to the browser, this connection is closed and there's no way to send any more data.
There's a few things that need to happen in order to accomplish what you want:
The long running function needs to run in the background so it doesn't block execution of the rest of the application
There has to be a way to get a status update from the long running function
The client (ie browser) needs a way to receive the status updates
1 and 2 can be solved using celery. It allows you to run tasks in the background and the task to send information via a side channel to be consumed elsewhere.
The easiest way to achieve 3 would be to set up a route in your flask application that returns information about the task, and request it periodically from the browser using some JavaScript. The more favorable method in my opinion would be to use websockets to actively send out the information to the client, but this is a bit more complicated.
This is just a rough outline, but there's a tutorial by Miguel Grinberg about how to set this up using celery and polling from JS.
I am implementing a Python plugin that is part of a larger C++ program. The goal of this program is to allow the user to input a command's actions in Python. It currently receives a string from the C++ function and runs it via the exec() function. The user can then use an API to affect changes on the larger C++ program.
The current feature I am working on is a pause execution feature. It needs to remember where it is in the code execution as well as the state of any local variables, and resume execution once a condition has been met. I am not very familiar with Python, and I would like some advice how to implement this feature. My first design ideas:
1) Using the yield command.
This seemed to be a good idea at the start since when you use the next command it remembers everything I needed it to, but the problem is that yield only returns to the previous level in the call stack as far as I can tell. So if the user calls a function that yields it will simply return to the user's code, and not the larger C++ program. As far as I can tell there isn't a way to propagate the yield command up the stack???
2) Threading
Create a main python thread that creates a thread for each command. This main thread would spawn a thread for each command executed and kill it when it is done. If it needs to be suspended and restarted it could do so through a queue of locks.
Those were the only two options I came up with. I am not sure the yield function would work or is what it was designed to do. I think the Threading approach would work but might be overkill, and take a long time to develop. I also was looking for some sort of Task Module in Python, but couldn't find exactly what I was looking for. I was wondering if anyone has any other suggestions as I am not very familiar with Python.
EDIT: As mentioned in the comments I did not explain what needs to happen when the script "Pauses". The python plugin needs to allow the C++ program to continue execution. In my mind this means A) returning if we are talking about a single threaded approach, or B) Sending a message(Function call?) to C++
EDIT EDIT: As stated I didn't fully explain the problem description. I will make another post that has a better statement of what currently exists, and what needs to happen as well as providing some sudo code. I am new to Stack Overflow, so if this is not the appropriate response please let me know.
Whenever a signal is sent in Python, execution is immediately paused until whatever signal handler function is being used is finished executing; at that point, the execution continues right where it left off. My suggestion would be to use one of the user-defined signals (signal.SIGUSR1 and signal.SIGUSR2). Take a look at the signal documentation here:
https://docs.python.org/2/library/signal.html
At the beginning of the program, you'd define a signal handler function like so:
def signal_pause(signum, frame):
if signum == signal.SIGUSR1:
# Do your pause here - function processing, etc
else:
pass
Then in the main program somewhere, you'll switch out the default signal handler for the one you just created:
signal.signal(signal.SIGUSR1, signal_pause)
And finally, whenever you want to pause, you'll send the SIGUSR1 signal like so:
os.kill(os.getpid(),signal.SIGUSR1)
Your code will immediately pause, saving its state, and head to the signal_pause function to do whatever you need to do. Once that function exits, normal program execution will resume.
EDIT: this assumes you want to do something sophisticated while you're pausing the program. If all you want to do is wait a few seconds or ask for some user input, there are some much easier ways (time.sleep or input respectively).
EDIT EDIT: this assumes you're on a Unix system.
If you need to communicate with a C program, then sockets are probably the way to go.
https://docs.python.org/2/library/socket.html
One of your two programs acts as the socket server, and the other connects to it as the socket client. When you want the C++ program to continue, you use socket.send() to transmit a continue message. Then your Python program would use socket.recv(), which will cause it to wait around until it receives a message back from the C++ program.
If you need two programs to send signals to each other, this is probably the safest way to go about it.
I have a python code which starts running on a button click. This function takes some time to complete. In my django code, the result page is displayed after this function has completed execution.
But what I want is some sort of queue based mechanism using which my currently running function can pass the partial results (list of links) as it computes it, to the views and the result page contents are updated dynamically according to data in the queue so that the user won't have to wait for the whole function to finish exception and can see partial results on the result page.
I am new to django and web development. How can this be done in django? Is there any library for doing this?
Thank you in advance
You can't do the partial computation in a function synchronous way. You can try to consider this library celery to do the asynchronous execution.
OR ELSE
You have to go with intermediate middleware where Python and Javascript can be communicated in thread safe way like redis queue, zmq etc..
Ajax would be more appropriate to do this kind of "real time" updates. It allows you to send asynchronous (the A of AJAX) requests/datas.
Put a script in your web page that will update the render everytime your server send new datas.
You can find more on this here : http://api.jquery.com/jquery.ajax/
My apologies beforehand for the length of the question, I didn't want to leave anything out.
Some background information
I'm trying to automate a data entry process by writing a Python application that uses the Windows API to simulate keystrokes, mouse movement and window/control manipulation. I have to resort to this method because I do not (yet) have the security clearance required to access the datastore/database directly (e.g. using SQL) or indirectly through a better suited API. Bureaucracy, it's a pain ;-)
The data entry process involves the correction of sales orders due to changes in article availability. The unavailable articles are either removed from the order or replaced by another suitable article.
Initially I want a human to be able to monitor the automatic data entry process to make sure everything goes right. To achieve this I slow down the actions on the one hand but also inform the user of what is currently going on through a pinned window.
The actual question
To allow the user to halt the automation process I'm registering the Pause/Break key as a hotkey and in the handler I want to pause the automation functionality. However, I'm currently struggling to figure out a way to properly pause the execution of the automation functionality. When the pause function is invoked I want the automation process to stop dead in its tracks, no matter what it is doing. I don't want it to even execute another keystroke.
UPDATE [23/01]: I actually want to do more than just pause, I want to be able to communicate with the automation process while it is running and request it to pause, skip the current sales order, give up completely and perhaps even more.
Can anybody show me The Right Way (TM) to achieve what I want?
Some more information
Here's an example of how the automation works (I'm using the pywinauto library):
from pywinauto import application
app = application.Application()
app.start_("notepad")
app.Notepad.TypeKeys("abcdef")
UPDATE [25/01]: After a few days of working on my application I've noticed I don't really use pywinauto that much, right now I'm only using it for finding window and then I directly use SendKeysCtypes.SendKeys to simulate keyboard input and win32api functions to simulate mouse input.
What I've found out so far
Here are a few methods I've come across so far in my search for an answer:
I could separate the automation functionality and the interface + hotkey listener in two separate processes. Let's refer to the former as "automator" and the latter as "manager". The manager can then pause the execution of the automator by sending the process a SIGSTOP signal and unpause it using the SIGCONT signal (or the Windows equivalents through SuspendThread/ResumeThread).
To be able to update the user interface the automator will need to inform the manager of its progression through some sort of an IPC mechanism.
Cons:
Would using SIGSTOP not be a little harsh? Would it even work properly? Lots of people seem to be advising against it and even calling it "dangerous".
I am worried that implementing the IPC mechanism is going to be a bit complicated. On the other hand, I have worked with DBus which wouldn't be too hard to implement.
The second method and one that lots of people seem to be suggesting involves using threads and essentially boils down to the following (simplified):
while True:
if self.pause: # pause
# Do the work...
However, doing it this way it seems it will only pause after there is no more work to do. The only way I see this method would work would be to divide the work (the entire automation process) into smaller work segments (i.e. tasks). Before starting on a new task the worker thread would check if it should pause and wait.
Cons:
Seems like an implementation to divide the work into smaller segments, such as the one above, would be very ugly code wise (aesthetically).
The way I imagine it, all statements would be transformed to look something like: queue.put((function, args)) (e.g. queue.put((app.Notepad.TypeKeys, "abcdef"))) and you'd have the automating process thread running through the tasks and continuously checking for the pause state before starting a task. That just can't be right...
The program would not actually stop dead in its tracks, but would first finish a task (however small) before actually pausing.
Progress made
UPDATE [23/01]: I've implemented a version of my application using the first method through the mentioned SuspendThread/ResumeThread functionality. So far this seems to work very nicely and also allows me to write the automation stuff just like you'd write any other script. The only quirk I've come across is that keyboard modifiers (CTRL, ALT, SHIFT) get "stuck" while paused. Something I can probably easily work around.
I've also written a test using the second method (threads and signals/message passing) and implemented the pause functionality. However, it looks really ugly (both checking for the pause flag and everything related to the "doing the work"). So if anybody can show me a proper example of something similar to the second method I'd appreciate it.
Related questions
Pausing a process?
Pausing a thread using threading class
Alex Martelli posted an answer saying:
There is no method for other threads to forcibly pause a thread (any more than there is for other threads to kill that thread) -- the target thread must cooperate by occasionally checking appropriate "flags" (a threading.Condition might be appropriate for the pause/unpause case).
He then referred to the multiprocessing module and SIGSTOP/SIGCONT.
Is there a way to indefinitely pause a thread?
Pausing a process in Windows
An answer to this question quotes the MSDN documentation regarding SuspendThread:
This function is primarily designed for use by debuggers. It is not intended to be used for thread synchronization. Calling SuspendThread on a thread that owns a synchronization object, such as a mutex or critical section, can lead to a deadlock if the calling thread tries to obtain a synchronization object owned by a suspended thread. To avoid this situation, a thread within an application that is not a debugger should signal the other thread to suspend itself. The target thread must be designed to watch for this signal and respond appropriately.
Is there any way to kill a Thread in Python?
How do I pass an exception between threads in python
Keep in mind that although in your level of abstraction, "executing a keystroke" is a single atomic operation, it's implemented on the machine as a rather complicated sequence of machine instructions. So, pausing a thread at arbitrary points could lead to things being in an indeterminate state. Sending SIGSTOP is the same level of dangerous as pausing a thread at an arbitrary point. Depending on where you are in a particular step, though, your automation could potentially be broken. For example, if you pause in the middle of a timing-dependent step.
It seems to me that this problem would be best solved at the level of the automation library. I'm not very familiar with the automation library that you're using. It might be worth contacting the developers of the library to see if they have any suggestions for pausing the execution of automation steps at safe sub-step levels.
I don't know pywinauto. But I'll assume that you have something like an Application class which you obtain and have methods like SendKeys/SendMouseEvent/etc to do things.
Create your own MyApplication class which holds a reference to pywinauto's application class. Provide the same methods but before each method check whether a pause event has occurred. If it has, you can jump into code which handles the pause event. That way you are checking for a pause every time you cause an event, but this all is handled by the one class without putting pause all over your code.
Once you've detected the pause you can handle it any way you like. For example, you can throw an exception to force giving up on the current task.
Separating the functionality and the interface thread/process is definately the best option imho, the second solution is quicker and easier but definately not better.
Perhaps using multiple threads and an exception would be a better idea than using multiple processes. But if you're using multiple processes than SIGSTOP might be your only way to get it to work.
Is there anything against using 2 threads for this?
1 thread for actually executing
1 thread for reading the user input
I use Python but not pywinauto; for this sort of tasks I use AutoHotKey . One way to implement a simple pause in an AutoHotkey script may be using a "toggle" key like ScrollLock and testing the key state in the script. Also, the script can restore the key state after switching the internal pause setting on / off.