I'm using python scripts to execute simple but long measurements. I as wondering if (and how) it's possible to edit a running script.
An example:
Let's assume I made an error in the last lines of a running script.These lines have not yet been executed. Now I'd like to fix it without restarting the script. What should I do?
Edit:
One Idea I had was loading each line of the script in a list. Then pop the first one. Feed it to an interpreter instance. Wait for it to complete and pop the next one. This way I could modify the list.
I guess I can't be the first one thinking about it. Someone must have implemented something like this before and I don't wan't to reinvent the weel. I one of you knows about a project please let me know.
I am afraid there's no easy way to arbitrarily modify a running Python script.
One approach is to test the script on a small amount of data first. This way you'll reduce the likelihood of discovering bugs when running on the actual, large, dataset.
Another possibility is to make the script periodically save its state to disk, so that it can be restarted from where it left off, rather than from the beginning.
Related
I am working with multiprocessing and I want to ask if there is some option to create new process with new terminal window in Ubuntu
I have 3 processes starting simultaneously, but I want results from them in separated terminals for each
Thanks
No. Sadly there is no* way to do what you want within python, as python has no control over the terminal it is running in.
What I think you want though is to separate the messages from your different processes, so you can see what's going on. What I sometimes do for this (in testing only!) is to have each process log to a different file, and then watch those three files in three terminal windows. You can do this with watch or even a simple while loop in bash:
watch -n 3 "cat /my/output/file" # or:
while true; do cat /my/output/file; sleep 3; done
of course you can replace cat with something more useful, perhaps tail. Or you can just open the output files in a text editor which has an auto-revert facility (e.g. Emacs with M-x auto-revert-mode). This does exactly the same thing internally---poll the file for changes and update if need be.
I also really suggest you use logging inside your code, and give each paralleled function it's own logger (with the name derived from the parameters of the function). (This can be easier with a small class rather than a function). That way later on you can send all your output to a file, and if something goes wrong you can easily find out which run failed and extract information from only that run (with grep!). I use this approach in paralleled fuzzy matching code (actually for matching music libraries) and it's invaluable when you need to dig into how some strange result occurred.
*Okay, I'm sure there's some horrible way to control some particular terminal and output to it, but that's not what you meant.
I am trying to find a way to edit and run a sequence of python scripts in a less manual way.
For context, I am running a series of simulations, which consist in running three codes in order 10 times, making minor changes to each code every time. The problem I am encountering is that this process leads to easy mistakes and chaotic work.
These are the type of edits I have to make to each code.
- Modify input/output file name
- Change value of a parameter
What is the best practice to deal with this? I imagine that the best idea would be to write another python script that does all this. Is there a way to edit other python codes, from within a code, and run them?
I don't intend or want anyone to write a code for me. I just need to be pointed in a general direction. I have searched for ways to 'automatize' codes, but haven't yet been successful in finding a solution to my query (mainly the editing part).
Thanks!
The thing that can change (files or parameter values) should be able to be either passed in or injected. Could be from a command line parameter, configuration file, or method argument. This is the "general direction" I offer.
I have a couple of functions written in a single Python file. They perform a sequence of steps on a file-based dataset.
My workflow:
After I finished coding a part of the function's body, I run the function to see how it goes.
It may break at a certain point.
I fix the code and re-run the function.
The problem is that when I re-run the function, it will execute the lines that were already completed successfully. Yet I want to be able to start not from the beginning but rather from an arbitrary point. This is because the whole function runs for several minutes and it would be a waste time to wait for it to complete.
I could implement "checks" to see whether this operation is required (e.g., don't create a file if it already exists), but this would imply adding a lot of new validation code (e.g., make sure that the existing file does contain the content needed); in reality, my function will be run on a dataset in known format and the whole function should be executed.
The most obvious solution is to comment out the parts that were executed successfully, but it's a hustle and I got tired of commenting and uncommenting parts as I move forwards and the function gets larger.
Are there are any better approaches than commenting out lines for ignoring certain part of the function's body when executing?
I am on Wing IDE if this has something to do with the debugging tricks in an IDE itself.
Wing can move the program counter to a different line in the function via the right-click popup menu, but you'd need to do this every time you run the function. I think a better approach is to refactor the function into smaller functions -- then you can comment out or conditionalize only the function calls. You could also write tests that call some of the functions and not others.
I'm working on a python project that spans across multiple files and directories. Here's my workflow:
Run main python script
Main script calls some functions in other files
Functions in other files/directories execute
In the middle of execution, there is a bug in one of the functions, but I find the bug only after the main script finishes. Sometimes, there may not be a bug, but rather some parameter that needs tweaking.
I go back and fix the bug/make the necessary tweaks and re-run the main program and this time it executes fine.
Obviously, this workflow is terribly inefficient as considerable amount of code (that runs prior to the buggy function) gets re-executed. What would be ideal is to run the program in ipython and after discovering the issue and making the necessary changes, restart from the place where the buggy function executions starts and not from the beginning. I'm not sure how to achieve this and any help would be much appreciated.
I know how to rerun lines from ipython history (%rerun) and how to ensure autoreload of changed files in ipython, but in this case, I can't really type out the lines of code into ipython. Writing unit tests may not always be feasible, so I need an alternate solution. My use case is something similar to setting a "breakpoint" and then re-executing code past the breakpoint multiple times so as to avoid re-executing the code prior to the breakpoint more than once, while ensuring that all the necessary variables (until that stage) are correctly populated. One final condition is that I may not be able to use an IDE and vim is the only editor available across all the environments I work with.
You could start writing test cases for every function and try to debug each function separately instead of the whole program/script.
There is a python unittest-module: https://docs.python.org/3.4/library/unittest.html and a lot of tutorials like (just an example): http://docs.python-guide.org/en/latest/writing/tests/
It seems annoying writing tests but thinking about test cases gives deeper understanding of "How should the function behave if...".
You could use the code module to include "breakpoints" in your code
import code
# ... later in your program add
code.interact(local=locals()) # enter python interpreter at this point (ctrl+D to continue execution)
Python is pretty clean, and I can code neat apps quickly.
But I notice I have some minor error someplace and I dont find the error at compile but at run time. Then I need to change and run the script again. Is there a way to have it break and let me modify and run?
Also, I dislike how python has no enums. If I were to write code that needs a lot of enums and types, should I be doing it in C++? It feels like I can do it quicker in C++.
"I don't find the error at compile but at run time"
Correct. True for all non-compiled interpreted languages.
"I need to change and run the script again"
Also correct. True for all non-compiled interpreted languages.
"Is there a way to have it break and let me modify and run?"
What?
If it's a run-time error, the script breaks, you fix it and run again.
If it's not a proper error, but a logic problem of some kind, then the program finishes, but doesn't work correctly. No language can anticipate what you hoped for and break for you.
Or perhaps you mean something else.
"...code that needs a lot of enums"
You'll need to provide examples of code that needs a lot of enums. I've been writing Python for years, and have no use for enums. Indeed, I've been writing C++ with no use for enums either.
You'll have to provide code that needs a lot of enums as a specific example. Perhaps in another question along the lines of "What's a Pythonic replacement for all these enums."
It's usually polymorphic class definitions, but without an example, it's hard to be sure.
With interpreted languages you have a lot of freedom. Freedom isn't free here either. While the interpreter won't torture you into dotting every i and crossing every T before it deems your code worthy of a run, it also won't try to statically analyze your code for all those problems. So you have a few choices.
1) {Pyflakes, pychecker, pylint} will do static analysis on your code. That settles the syntax issue mostly.
2) Test-driven development with nosetests or the like will help you. If you make a code change that breaks your existing code, the tests will fail and you will know about it. This is actually better than static analysis and can be as fast. If you test-first, then you will have all your code checked at test runtime instead of program runtime.
Note that with 1 & 2 in place you are a bit better off than if you had just a static-typing compiler on your side. Even so, it will not create a proof of correctness.
It is possible that your tests may miss some plumbing you need for the app to actually run. If that happens, you fix it by writing more tests usually. But you still need to fire up the app and bang on it to see what tests you should have written and didn't.
You might want to look into something like nosey, which runs your unit tests periodically when you've saved changes to a file. You could also set up a save-event trigger to run your unit tests in the background whenever you save a file (possible e.g. with Komodo Edit).
That said, what I do is bind the F7 key to run unit tests in the current directory and subdirectories, and the F6 key to run pylint on the current file. Frequent use of these allows me to spot errors pretty quickly.
Python is an interpreted language, there is no compile stage, at least not that is visible to the user. If you get an error, go back, modify the script, and try again. If your script has long execution time, and you don't want to stop-restart, you can try a debugger like pdb, using which you can fix some of your errors during runtime.
There are a large number of ways in which you can implement enums, a quick google search for "python enums" gives everything you're likely to need. However, you should look into whether or not you really need them, and if there's a better, more 'pythonic' way of doing the same thing.