I am tyring to find a solution for running a python script of mine 24/7.
Research led me to the conclusion that I probably need a CGI hosting.
But suppose that I have one (and which type of hosts should I look for?) :
how will I make the script run all the time?
In pseudo-code it should be something like this:
if (time_since_last_run(script.py)>100):
run(script.py)
Please give me a direction.
CGI has nothing to do with running a script continuously. Also your problem seems to be to run your script on a regular schedule. Depending on your operating system, you can look into Scheduled Tasks on Windows (http://support.microsoft.com/kb/814596) or cron on other systems (https://en.wikipedia.org/wiki/Cron).
You still need a computer that runs continuously, either at home or with some hosting enterprise.
You can also try to use supervisord (http://supervisord.org/)
CGI is used for creating websites with e.g. forms, but not for scripts which run in 24/7 and do tasks A,B, etc.
You need a physical PC, dedicated server or at least a virtual machine.
Unlike Linux, which is widely used today, Unix is a less popular operating system. Servers operating 24 x 7 primarily use Unix operating systems.
A unix server is one of the cheapest way for your solution.
Related
I have some python scripts that I look to run daily form a Windows PC.
My current workflow is:
The desktop PC stays all every day except for a weekly restart over the weekend
After the restart I open VS Code and run a little bash script ./start.sh that kicks off the tasks.
The above works reasonably fine, but it is also fairly painful. I need to re-run start.sh if I ever close VS Code (eg. for an update). Also the processes use some local python libraries so I need to stop them if I'm going to update them.
With regards to how to do this properly, 4 tools came to mind:
Windows Scheduler
Airflow
Prefect (https://www.prefect.io/)
Rocketry (https://rocketry.readthedocs.io/en/stable/)
However, I can't quite get my head around the fundamental issue that Prefect/Airflow/Rocketry run on my PC then there is nothing that will restart them after the PC reboots. I'm also not sure they will give me the isolation I'd prefer on these tools.
Docker came to mind, I could put each task into a docker image and run them via some form of docker swarm or something like that. But not sure if I'm re-inventing the wheel.
I'm 100% sure I'm not the first person in this situation. Could anyone point me to a guide on how this could be done well?
Note:
I am not considering running the python scripts in the cloud. They interact with local tools that are only licenced for my PC.
You can definitely use Prefect for that - it's very lightweight and seems to be matching what you're looking for. You install it with pip install prefect, start Orion API server: prefect orion start and once you create a Deployment, and start an agent prefect agent start -q default you can even configure schedule from the UI
For more information about Deployments, check our FAQ section.
It sounds Rocketry could also be suitable. Rocketry can shut down itself using a task. You could do a task that:
Runs on the main thread and process (blocking starting new tasks)
Waits or terminates all the currently running tasks (use the session)
Calls session.shut_down() which sets a flag to the scheduler.
There is also a app configuration shut_cond which is simply a condition. If this condition is True, the scheduler exits so alternatively you can use this.
Then after the line app.run() you simply have a line that runs shutdown -r (restart) command on shell using a subprocess library, for example. Then you need something that starts Rocketry again when the restart is completed. For this, perhaps this could be an answer: https://superuser.com/a/954957, or use Windows scheduler to have a simple startup task that starts Rocketry.
Especially if you had Linux machines (Raspberry Pis for example), you could integrate Rocketry with FastAPI and make a small cluster in which Rocketry apps communicate with each other, just put script with Rocketry as a startup service. One machine could be a backup that calls another machine's API which runs Linux restart command. Then the backup executes tasks until the primary machine answers to requests again (is up and running).
But as the author of the library, I'm possibly biased toward my own projects. But Rocketry very capable on complex scheduling problems, that's the purpose of the project.
You can use schtasks for windows to schedule the tasks like running bash script or python script and it's pretty reliable too.
I've written a lot of python scripts. Now I want to run it on another computer which running non-stop to crawling, analyzing data and update to an sql database.
Normally I open a command prompt and run the scripts:
python [script directory]
But with many scripts I have to open many cmd and every script call an python interpreter, so It end up with huge mess using a lot of memory.
What should I do to manage these scripts.
You haven't specified what OS your server is, but assuming that it's a Linux server you should probably research a process management tool such as Supervisord or Systemd. These are tools designed to run and monitor your program automatically, and even restart it if it crashes.
If you're using Ubuntu 16.04 then it comes with Systemd out of the box, however I personally find Supervisord easier to configure and use for simple tasks.
These programs won't necessarily help with your memory consumption issues however. Sure you can place caps on memory use for a process, but that's not really going to help you if it stops your program from working. You're probably best to re-evaluate your code and look for ways to reduce its memory footprint or use a server with more ram.
EDIT:
You've just added that the OS is Windows 10, which makes the above irrelevant. You can use the Windows Task Scheduler to automatically execute long running tasks.
you can use pythonw *.py , and it will run in background.
I have python script for ssh which help to run various Linux commands on remote server using paramiko module. All the outputs are saved in text file, script is running properly. Now I wanted to run these script twice a day automatically at 11am and 5pm everyday.
How can I run these script automatically every day at given time without compiling every time manually. Is there any software or module.
Thanks for your help.
If you're running Windows, your best bet would be to create a Scheduled Task to execute Python itself, passing the path to your script as an argument.
If you're using OSX or Linux, CRON is your friend. There are references abound for how to create scheduled events in crontab. This is a good start for setting up CRON tasks.
One thing to mention is permissions. If you're running this from a Linux machine, you'll want to ensure you set up the CRON job to run under the right account (best practice not to use your own).
Assuming you are running on a *nix system, cron is definitely a good option. If you are running a Linux system that uses systemd, you could try creating a timer unit. It is probably more work than cron, but it has some advantages.
I won't go though all the details here, but basically:
Create a service unit that runs your program.
Create a timer unit that activates the server unit at the prescribed times.
Start and enable the timer unit.
We run many Python scripts for data processing tasks. We have a modeling computer that has been upgraded to provide the best performance for these tasks, but it is shared by many people that all need to run different scripts on it at the same time.
Is it possible for me to run a Python script remotely on that machine from my laptop while others are either directly logged into it or also remotely running a script?
Is SSH a possibility? I haven't ever run any scripts remotely aside from logging in via remote desktop. Ideally, I could start the Python script on that remote machine, but all the messages would be visible to me on my laptop. Does this sound doable?
EDIT:
I forgot to mention all machines are running Windows 7.
SSH is definitely the way to go and also have a look at Fabric.
Regarding your edit. You can use Fabric on Windows. And I think that using SSH on Windows will be a bit easier than dancing with their Powershell's remoting capabilities.
SSH does seem like it should meet your needs.
You could also consider setting up an iPython notebook server that everyone could use.
Its got nice parallel processing capabilities if you are doing some serious number crunching.
I'm writing a Python script to access all computers on the network, log in to them and read some log files. I don't want to use something as low-level as socket, but I can if I must. I realize that my problem is similar to this question, but not the same.
Are there any modules for accessing external Windows machines?
Has anyone done anything like this before?
I'm specifically looking to log into Windows 7 machines, not unix.
Let's also assume that each computer I want to log into has Remote Desktop installed and enabled. I'm also not worried about network security or encryption because these files are not confidential. Windows machines don't have SSH installed on the by default do they?
There has to be something on the other side for you to talk to. This limits you to either setting up a "server" on each machine, installing a real server (i.e. sshd), building a "server" yourself and installing it, or using a built in and active feature of the OS.
Based upon this, what kind of system do you want to set up on these machines? What does it need to do? Just read the contents of a prespecified file list? Will that list change?
One solution is to turn on telnet, and use paramiko or twisted to
talk across it. This isn't very secure of course
Next up, set up a samba share, and access the folder remotely. This
is also insecure, though less so than telnet
You could find a ssh daemon port and run that, if you are so inclined
Psexec from sysinternals might work
Use twisted to build a server app with the features you need
Use ncat to listen on a port and spawn a cmd prompt
Be aware that most of the solutions for accessing windows remotely are... poor. The best solution is probably to roll your own, but that is hard work and you will probably make mistakes.
Also, Windows 7 is not exactly multi-user friendly. Individual processes can run as separate users, but the OS does not support having multiple users logged in at the same time. Someone is going to be the "user" and everyone else is just a process with a different credential set.
This is more an artificial limitation on M$'s part than anything technical. To see this in action, try to log in with RDP while a user is logged in locally. Fun times.
Per your edit, the easiest thing to do is just set up a samba share on the box.
After this share is set up:
with open(r'\\myCompNameOrIP\C\windows\logs\logfile.txt','rb') as logfile:
loglines = logfile.readlines()
Or you can use the gencat sample found here. Just give it r'\\myCompNameOrIP\C\windows\logs\*.txt' as the search path and watch the magic.
From Ubuntu I use samba:
In Bash:
gvfs-mount smb://them/folder
Here I give name, domain and password
Then in python:
folder = '/home/me/.gvfs/folder on them'
using the os module I read folders and files inside.
I am working in a small business environment.
Why not have each of the computers send the log file to the central computer?