I had a program that Scraped certain data from certain Web-Pages, and when the Web-Pages changed, acted accordingly.
How would one set up the program so it continues to run in the background?
I don't need any specifics
I'm just really confused on this concept and would appreciate whatever help anybody has to offer.
start path-to-pythonw.exe your-code.py
pythonw means without console.
start means start on background.
if your python is installed system-wide, you can probably start your-code.pyw
.pyw is associated with pythonw.exe
remember you cannot use print (to stdout) in this case.
If you want to be able to just start your process and have it background itself and do a few more typical things that "daemon" processes do in Unix, look here: How do you create a daemon in Python?
There is no concept of "background" in Windows. But the UNIX shell concept of a background process can be reasonably emulated by running your Python script as a Windows service. There are a couple of suggestions in this question: Is it possible to run a Python script as a service in Windows? If possible, how?
For casual use, I suggest that you learn how to use srvany from the second answer.
You simply need to leave your program running! Please google "python daemon" and see how to implement a persistent background process in Python.
Now, you cannot know when a website changes unless you poll it. If the website is well designed, the page you are trying to poll will have a "Last-Modified" header, you can make a "HEAD" request every so often (be nice: don't poll like crazy) and act when Last-Modified is >= than the one on record. If the site is not well designed, it will not have a reliable Last-Modified or ETAG header, in that case you will have to parse manually and check for changes yourself.
Cheers.
Related
in my vps i have run 4 Python Script and its been 60 days that i don't reboot my vps and now i have to, but if i reboot vps my python Variables & data will be removed because i don't store them in file and they are store in variables in python script.
my OS is Ubuntu Server 16.04 LTS and i was run my python codes with nohup command until they can run in background.
Now i need a way to stop my scripts without removing they variables and start them with same variables and data after i reboot my vps.
Is There Any Way That I Can Do This?
In Addition, I'm sorry for writing mistakes in my question.
Python doesn't provide any way of doing this.
But you might be able to use CRIU, or a similar tool, to freeze and snapshot the interpreter process. Then, after restart, you can resume the snapshot into a new process that just picks up exactly where you left off.
It may not work.1 But there's a good chance it will. This is essentially the same thing as a Live Migration in the CRIU docs, except that you're not migrating to a new computer/container/etc., just to the future of the same computer. So, start reading with that page, and follow the links from there.
You should probably test before you commit to it.
* Try it (obviously don't include the system restart, just kill -9 the executable) on a Python script that doesn't do anything important (maybe increments a counter, print it out, sleep for a second, repeat.
* Maybe try it on a script that does similar kinds of stuff to what yours are doing.
* If it's safe to have two copies of one of your programs running at the same time (they're not going to stomp all over each other writing to the same file, or fight over the same socket, or whatever), start a second copy and test dump/kill/resume that.
* Try it on one of your real processes, still without restart.
* Try it on all four.
* Cross your fingers, sacrifice a chicken, and do it for real.
If that doesn't pan out, the only option I can think of is to go through your scripts, manually figure out everything that needs to be saved and how it could be accessed from the top-level global, and do that in the debugger.
Ideally, you'll write a script that will automate accessing and saving all that stuff—plus another one to feed it into a new instance at restart. Then you just pdb the live interpreters and start dumping everything.
This is guaranteed to be a whole lot of work, and not much fun. On the plus side, it is guaranteed to work if you do it right. On the third hand, it's pretty easy to not do it right.
1. If you rely on open files, pipes, sockets, etc., CRIU does about as much as you could do, which is more than you might expect at first, but still not everything you could possibly want… Also, if you're using almost all of your RAM, it can be hard to wedge things back into exactly the same state. And there are probably other possible issues.
At the begining i would like to state that i did look for an answer before posting my question, but if i missed anything I'm really sorry.
Ok to the point.
I'm trying to create a tool that will monitor behaviour of my 2 external devices comunicating over BT(communication over BT i have pretty much solved). but what i'm strugling with is monitoring them.
So Manually i open cmdline 2 times and from there i use putty to connect to devices and do stuff.
Now I want(and pretty much need) to do the manual part in python. So i tried using subprocess.Popen to connect to cmdline(and from there to putty) but the problem is that this only works as request/response. what i need is to open (and keep) cmdline streamlike connection and pass and receive commands/response without closing.
P.S. I'm using windows enviroment and python 2.7.
Thank You for any response.
Kind Regards.
Is there any way to watch for a new process with name 'X' starting in python (ideally) or bash? I know that I can look at running processes, but that is not fast enough for my needs. The only think that I can think of is some how hooking into the new process, and registering that, but how?
More background: I am part of a CCDC team (http://www.nationalccdc.org/) and am on the blue team. The premise of the competition is to give students a network to defend against professional pen testers to help the next generation of security experts be better. What I want to do is load this python script on the linux boxs and watch for certain commands that are being run, that likely would only be used by the red team, for example the 'chattr' command. Ideally I would like to be able to provide the script a list of processes to watch. I can figure out that part but do not know how to watch for a process spawning.
Any direction is appreciated. Thank you.
I know of no way for a process which does not have root privileges to be notified when a process is started via any means on a fully-running Linux system. If polling isn't fast enough, you're going to have to do some serious hackery.
If you've got root, this is possible. If not, I can't see it.
With root, you could set a system-wide replacement of the fork and exec system calls which provides you with your desired notification. This could be in the kernel, or it could be an LD_PRELOAD hack.
This applies not just to Python; even with a C program, I don't know of an "inotify for process creation".
I have not tested this idea, but on Linux each process is given a directory under /proc/<it's process id>/ If you opened an inotify on directory creation in /proc you might be able to track creation of process directories and then see if /proc/<dir>/cmdline matches the process your looking for. This is just a thought, hope it helps!
Now I am working in a project where the testscript has to connect many (3-10) remote computers (SSH and do some stuff).
I started to use the pexpect and it is simple as a button. It works fine.
I want to see the communication during test. I know it is possible to redirect the log to the screen. But in this case the logs (from different computer) are mixed.
What I would like is to open new terminal window (or consol or whatever) for every new spawn object. In this case I could see all communication in different windows. Additionally I would like to keep the possibility of spawn.interact() in every window.
I feel that it is possible somehow but I don't know how. I think some file pointer (or pipe) should pass to the new window somehow(?)
(SecureCRT knows sometihng like this, it has tabbed consol windows and can access them separately, but it is a commercial product)
Or let me make the problem more simple.
If I do this, I can open a new shell in a new window:
p=Popen(["cygstart", "bash"])
How can I read and write into this shell from my script (parent) to see it in this new window?
I would really appreciate it, if one of you could point me in the right direction.
It is enough if you tell me what to read or find for (on Google) because I did not find anybody such kind of problem.
The environment is cygwin.
Thanks in advance
br:drv
Have you tried using the logfile parameter?
child = pexpect.spawn('some_command')
mylog = open('/tmp/mylog','w')
child.logfile = mylog
This will automatically log all communication to the file, including commands you enter after calling spawn.interact()
More info available on the website: http://pexpect.sourceforge.net/pexpect.html
Search for 'logfile' to find the relevant documentation.
I would like to daemonize a python process, and now want to ask if it is good practice to have a daemon running, like a parent process and call another class which opens 10-30 threads.
I'm planning on writing a monitoring script for group of servers and would like to check every server every 5 mins, that each server is checked exactly 5minutes.
I would like to have it this way ( sort of speak, ps auxf style output ):
|monitor-daemon.py
\-check-server.py
\-check-server.py
....
Thank you!
Maybe you should use http://pypi.python.org/pypi/python-daemon
You can use supervisord for this. You can configure tasks to respond to events. The events can be manually created or automatically by monitoring processes or based on regular intervals.
It is fully customizable and written in Python.
Example:
[program:your_daemon_name]
command=your_daemon_process
# Add extra options here according to the manual...
[eventlistener:your_monitor_name]
command=your_monitor_process
events=PROCESS_STATE_RUNNING # Will be triggered after a program changes from starting to running
# Add extra options here according to the manual...
Or if you want the eventlistener to respond to the process output use the event PROCESS_COMMUNICATION_STDOUT or TICK_60 for a check every minute. The logs can be redirected to files and such so you can always view the state.
There's really not much to creating your own daemonize function: The source for Advanced Programming in the Unix Environment (2nd edition) is freely available: http://www.apuebook.com/src.tar.gz -- you're looking for the apue.2e/daemons/init.c file.
There is a small helper program that does all the work of creating a proper daemon, it can be used to wrap arbitrary programs; this might save some hassle.