I'm relatively new to working with server instances so please excuse my limited knowledge. I have a python script that I run for 5 different people. Each person who I run it for, has a slightly different file (Ie: some names in the script are changed to work with their branding and name), and right now when I get a new client, I have to copy the script, and change each part that needs to be changed by hand, and then run that in my server (I'm running all 5 scripts on the same server). If I make any changes to my codebase, I need to change the script for each of my clients, go to the server and swap out the old versions by hands.
I'm expecting to have this grow to a couple hundred clients so this is just not a feasible method. Are there any products or methods I can use to fix this product, or atleast make it easier to update all the files at once?
Without looking at the specifics of the script it's really hard to say, but here are a couple of solutions that might work:
Pass the values that change as arguments to the script - Use something like argparse to capture the arguments into variables
Modify the script to run on a list of arguments - This is pretty easy to do - literally just add a for-loop around your entire script and iterate over the values for each of your clients.
Related
I am trying to find a way to edit and run a sequence of python scripts in a less manual way.
For context, I am running a series of simulations, which consist in running three codes in order 10 times, making minor changes to each code every time. The problem I am encountering is that this process leads to easy mistakes and chaotic work.
These are the type of edits I have to make to each code.
- Modify input/output file name
- Change value of a parameter
What is the best practice to deal with this? I imagine that the best idea would be to write another python script that does all this. Is there a way to edit other python codes, from within a code, and run them?
I don't intend or want anyone to write a code for me. I just need to be pointed in a general direction. I have searched for ways to 'automatize' codes, but haven't yet been successful in finding a solution to my query (mainly the editing part).
Thanks!
The thing that can change (files or parameter values) should be able to be either passed in or injected. Could be from a command line parameter, configuration file, or method argument. This is the "general direction" I offer.
I currently have two files: control.py and list.py. list.py contains a list of objects that I want to access and edit in control.py. In the project I'm working on, I should be able to open multiple instances of control.py (essentially, open x number of terminals each running its own instance of control.py). I want to able to access the list in list.py from each instance of control.py. Each instance should be able to edit the list such that after editing, all the other instances should be able to see the edited list.
I've looked into the shelve module but it turns out that shelve doesn't support concurrent modification, so that won't work because i need each instance to simultaneously be able to read and edit the list.
The usual solution to this problem is known as "pubsub"; i.e., publish/subscribe. See https://en.wikipedia.org/wiki/Publish%E2%80%93subscribe_pattern for a primer on the topic. There is no way to do this using just the capabilities provided by the core Python libraries; at least not without writing a pubsub implementation using those capabilities.
However, I get the sense you are doing the programming equivalent of trying to use a hammer to drive a screw. That is, because a hammer is the only tool you know how to use every problem looks like a nail. Reframing the problem is likely to help you, and us, provide better solutions.
I am implementing a workflow management system, where the workflow developer overloads a little process function and inherits from a Workflow class. The class offers a method named add_component in order to add a component to the workflow (a component is the execution of a software or can be more complex).
My Workflow class in order to display status needs to know what components have been added to the workflow. To do so I tried 2 things:
execute the process function 2 times, the first time allow to gather all components required, the second one is for the real execution. The problem is, if the workflow developer do something else than adding components (add element in a databases, create a file) this will be done twice!
parse the python code of the function to extract only the add_component lines, this works but if some components are in a if / else statement and the component should not be executed, the component apears in the monitoring!
I'm wondering if there is other solution (I thought about making my workflow being an XML or something to parse easier but this is less flexible).
You cannot know what a program does without "executing" it (could be in some context where you mock things you don't want to be modified but it look like shooting at a moving target).
If you do a handmade parsing there will always be some issues you miss.
You should break the code in two functions :
a first one where the code can only add_component(s) without any side
effects, but with the possibility to run real code to check the
environment etc. to know which components to add.
a second one that
can have side effects and rely on the added components.
Using an XML (or any static format) is similar except :
you are certain there are no side effects (don't need to rely on the programmer respecting the documentation)
much less flexibility but be sure you need it.
I'm using python scripts to execute simple but long measurements. I as wondering if (and how) it's possible to edit a running script.
An example:
Let's assume I made an error in the last lines of a running script.These lines have not yet been executed. Now I'd like to fix it without restarting the script. What should I do?
Edit:
One Idea I had was loading each line of the script in a list. Then pop the first one. Feed it to an interpreter instance. Wait for it to complete and pop the next one. This way I could modify the list.
I guess I can't be the first one thinking about it. Someone must have implemented something like this before and I don't wan't to reinvent the weel. I one of you knows about a project please let me know.
I am afraid there's no easy way to arbitrarily modify a running Python script.
One approach is to test the script on a small amount of data first. This way you'll reduce the likelihood of discovering bugs when running on the actual, large, dataset.
Another possibility is to make the script periodically save its state to disk, so that it can be restarted from where it left off, rather than from the beginning.
I'm currently using Python's function,
os.walk('/')
to loop through my whole filesystem on OS X. My aim is to make a personal daemon that keeps track of:
Newly made files/dirs
Adjusted/touched files/dirs
Deleted files (maybe)
The idea
This is more of a precautious function I wanted to add to my Macs to be able to see if weird stuff gets placed in my directories unwanted, so if my Macs ever get infected by some (yet unknown) trojan I can maybe detect it myself already.
Also I'm looking into adding features later to maybe shut down my internet connections etc. when something off is detected. This is maybe an irrational function, but as it's just a personal script I think it's not that bad :P.
What I want to achieve
So my main question is. After the first run I will save an array of the whole filesystem and their metadata (creation data, modification date). After that the I want the daemon to run in the background in a "watching" mode, practically mirroring the last known stored array of the filesystem with a newly looped one.
The problem now is that when I run the script to test it litteraly starts to burn my CPU, making my MacBook starting to get hiccups after a while. I want to add sleeps between each directory step "os.walk()" makes in my for loop.
My question is: What is a right sleep time? My MacBook's Disk Utility says I have 183.867 folders and 1.013.320 files making a total of 1.197.187 entries (as Folders are practically files too). So setting my code to:
time.sleep(0.001)
..would approximately take those 1.2Million entries to be dealt in about 2 minutes. I have no clue if this is good and I prefer to make it more dynamically based on the total files/folders count.
An extra feature
Based on my question I noticed that the OS X Disk Utility already knows my total Files and Folders. Can Python get this data too without doing an extreme loop-through? Maybe calling a Terminal function built-in to OS X.
This way I could also keep an indicator if using a little GUI for my Daemon's status.
Thanks in advance!
This is not a direct answer but directed to the requirement to track:
Newly made files/dirs
Adjusted/touched files/dirs
Deleted files (maybe)
You can use : http://pyinotify.sourceforge.net/ which ties with inotify and will send events on file changes, deletion and creation. This will avoid the walk through of large directories.
This is wrapper over Inotify. This is available for Linux. I also see that there are libraries and modules for OSX in fink and macports. So this should be more elegant solution.
Not a complete answer, but a few pointers:
For OSX, fseventsd implements a mechanism similar to inotify. fslogger is an example of how it can be used. pymacadmin seems to allow you to interface it from Python.
What you want to implement is similar to Tripwire.