We Keep Coding

Checking for dead links locally in a static website (using wget?)

A very nice tool to check for dead links (e.g. links pointing to 404 errors) is wget --spider. However, I have a slightly different use-case where I generate a static website, and want to check for broken links before uploading. More precisely, I want to check both:
Relative links like file.pdf
Absolute links, most likely to external sites like example.
I tried wget --spyder --force-html -i file-to-check.html, which reads the local file, considers it as HTML and follows each links. Unfortunately, it can't deal with relative links within the local HTML file (errors out with Cannot resolve incomplete link some/file.pdf). I tried using file:// but wget does not support it.
Currently, I have a hack based on running a local webserver throught python3 http.serve and checking the local files through HTTP:
python3 -m http.server &
pid=$!
sleep .5
error=0
wget --spider -nd -nv -H -r -l 1 http://localhost:8000/index.html || error=$?
kill $pid
wait $pid
exit $error
I'm not really happy with this for several reasons:
I need this sleep .5 to wait for the webserver to be ready. Without it, the script fails, but I can't guarantee that 0.5 seconds will be enough. I'd prefer having a way to start the wget command when the server is ready.
Conversely, this kill $pid feels ugly.
Ideally, python3 -m http.server would have an option to run a command when the server is ready and would shutdown itself after the command is completed. That sounds doable by writing a bit of Python, but I was wondering whether a cleaner solution exists.
Did I miss anything? Is there a better solution? I'm mentioning wget in my question because it does almost what I want, but using wget is not a requirement for me (nor is python -m http.server). I just need to have something easy to run and automate on Linux.

So I think you are running in the right direction. I would use wget and python as they are two readily available options on many systems. And the good part is that it gets the job done for you. Now what you want is to listen for Serving HTTP on 0.0.0.0 from the stdout of that process.
So I would start the process using something like below
python3 -u -m http.server > ./myserver.log &
Note the -u I have used here for unbuffered output, this is really important
Now next is waiting for this text to appear in myserver.log
timeout 10 awk '/Serving HTTP on 0.0.0.0/{print; exit}' <(tail -f ./myserver.log)
So 10 seconds is your maximum wait time here. And rest is self-explanatory. Next about your kill $pid. I don't think it is a problem, but if you want it to be more like the way a user does it then I would change it to
kill -s SIGINT $pid
This will be equivalent to you processing CTRL+C after launching the program. Also I would handle the SIGINT my bash script as well using something like below
https://unix.stackexchange.com/questions/313644/execute-command-or-function-when-sigint-or-sigterm-is-send-to-the-parent-script/313648
The above basically adds below to top of the bash script to handle you killing the script using CTRL+C or external kill signal
#!/bin/bash
exit_script() {
echo "Printing something special!"
echo "Maybe executing other commands!"
trap - SIGINT SIGTERM # clear the trap
kill -- -$$ # Sends SIGTERM to child/sub processes
}
trap exit_script SIGINT SIGTERM

Tarun Lalwani's answer is correct, and following the advices given there one can write a clean and short shell script (relying on Python and awk). Another solution is to write the script completely in Python, giving a slightly more verbose but arguably cleaner script. The server can be launched in a thread, then the command to check the website is executed, and finally the server is shut down. We don't need to parse the textual output nor to send a signal to an external process anymore. The key parts of the script are therefore:
def start_server(port,
server_class=HTTPServer,
handler_class=SimpleHTTPRequestHandler):
server_address = ('', port)
httpd = server_class(server_address, handler_class)
thread = threading.Thread(target=httpd.serve_forever)
thread.start()
return httpd
def main(cmd, port):
httpd = start_server(port)
status = subprocess.call(cmd)
httpd.shutdown()
sys.exit(status)
I wrote a slightly more advanced script (with a bit of command-line option parsing on top of this) and published it as: https://gitlab.com/moy/check-links

Stopping Django's Server [duplicate]

So im using ruby on rails in windows (i hear you all spitting your coffee onto the screen), its only a short term thing. (using ubuntu at home) So i tried to fire up webrick this afternoon and i get the error message
TCPServer Error, only one usage of each socket address is normally permitted
So it seems as if port 3000 is still running from last week? My question is how do i kill the process from the Windows command line. normally i have to press ctrl and pause/break in windows as ctrl c is not working which is only killing the batch process it seems..
Any solutions welcomed
Edit
So it seems as if
tasklist
will give me the list of processes, but where do i find the process for running the webrick server?
ruby.exe is not listed as a running process

Try using netstat -a -o -n to determine the pid of the process running on port 3000. Then you should be able to use taskkill /pid #### to kill whatever process is running on that port.
Probably not the most graceful way to do it, but I think it should work.
EDIT
You'll probably have to also use the /F flag to force-kill the process. I just tried it on my local machine, and that worked fine.

Go into rails_project\tmp\pids and delete the .pid file in there.
run:
rails server

Run Python script forever, logging errors and restarting when crashes

I have a python script that continuously process new data and writes to a mongodb. In the script, its a while loop and a sleep that runs the code continuously.
What is the recommended way to run the Python script forever, logging errors when they occur, and restarting when it crashes?
Will node.js's forever be suitable? I'm also running node/meteor on the same Ubuntu server.

supervisord is perfect for this sort of thing. While I used to check that programs were still running every couple of minutes with a cron job, supervisord runs all programs in an in-process thread, so in the event your program terminates, supervisord will automatically restart the process. I no longer need to parse the output of ps to see if a program crashed.
It has a simple declaritive config file and configurable logging. By default it creates a log file for your-program-name-stderr.log your-program-name-stdout.log which are automatically handled by logrotate when supervisord is installed from an OS package manager (Debian for me).
If you don't want to configure supervisord's logging, you should look at logging in python so you can control what goes into those files.
if you're on a debian derivative you should be able to install and start the daemon simply by executing apt-get install supervisord as root.
The config file is very straightforward too:
[program:myprogram]
command=/path/to/my/program/script
directory=/path/to/my/program/base
user=myuser
autostart=true
autorestart=true
redirect_stderr=True
supervisorctl also allows you to see what your program is doing interactively and can start and stop multiple programs with supervisorctl start myprogram etc

Recently wrote something similar. The basic pattern I follow is
while True:
try:
#functionality
except SpecificError:
#log exception
except: #catch everything else
finally:
time.sleep(600)
to handle reboots you can use init.d or cron jobs.

If you are writing a daemon, you should probably do it with this command:
http://manpages.ubuntu.com/manpages/lucid/man8/start-stop-daemon.8.html
You can spawn this from a System V /etc/init.d/ script, or use Upstart which is slowly replacing it.
Upstart: http://upstart.ubuntu.com/getting-started.html
System V: http://www.cyberciti.biz/tips/linux-write-sys-v-init-script-to-start-stop-service.html
I find System V easier to write, but if this will ever be packaged and distributed in a debian file, I recommend writing an Upstart conf.
Definitely keep the sleep so it won't keep a grip on CPU load.

I don't know if this is still relevant to you, but I have been reading forever about how to do this and want to share somewhere what I did.
For me, the goal was to have a python script running always (on my Linux computer). The python script also has a "while True " loop in it which should theoretically run forever, but if it for any reason I cannot think of would crash, I want the script to restart. Also, when I restart the computer it should run the script.
I am not an expert but for me the best and most understandable was to use systemd (assuming you use Linux).
There are two nice examples of how to do this given here and here, showing how to write your .service files in either /etc/systemd/system or /lib/systemd/system. If you want to be completely correct you should take the former:
" /etc/systemd/system/: units installed by the system administrator" 1
The documentation of systemd here is actually nice to read, even if you are not an expert.
Hope this helps someone!

GUI not opening

I have written a python script which scans my gmail INBOX for a particular mail, and if that mail is present it opens up a GUI. I have tested this script and works correctly.
I want to run this script whenever the network connection is established. So, I have added a script in the dispatch.d directory of the NetworkManager. My bash script is shown below.
#!/bin/bash
#/etc/NetworkManager/dispatcher.d/90filename.sh
IF=$1
STATUS=$2
if [ "$IF" == "wlan0" ]; # for wireless internet
then
case "$2" in
up)
logger -s "NM Script up triggered"
python /home/rahul/python/expensesheet/emailReader.py
logger -s "emailReader completed"
exitValue=$?
python3.2 /home/rahul/python/expensesheet/GUI.py &
logger -s "GUI completed with exit status $exitValue"
;;
down)
logger -s "NM Script down triggered"
#place custom here
;;
pre-up)
logger -s "NM Script pre-up triggered"
#place custom here
;;
post-down)
logger -s "NM Script post-down triggered"
#place custom here
;;
*)
;;
esac
fi
I have used tkinter to design my GUI.
My problem is that, emailReader(which has no GUI) gets executed correctly, but GUI.py doesn't get executed. It exits with the exit status 1.
Can somebody throw some light on this matter and explain what I'm doing wrong?

NetworkManager is a process that is running on a virtual terminal, outside of your X-server.
(e.g. NetworkManager get's started on bootup before your window manager gets started; they are totally unrelated).
therefore, any script started by NetworkManager will not (directly) be able to access the GUI. (it is very similar to what you get when you change from your desktop to a virtual terminal (e.g. Ctrl-Alt-1), and then try to run your GUI from there: you will most likely get an error like "Can't open display".
if you want to start a GUI-program, you have 2 possibilities
tell a notification daemon (a sub-process of your window-manager) to start your GUI
tell your GUI to start on the correct display (the one, where your desktop is running)
i'd go for the first solution (notification daemons are designed for that very purpose), but how to do it, heavily depends on the window-manager you use.
the 2nd solution is a bit more dirty and involves potential security breaches but basically try something like starting DISPLAY=:0.0 myguiapp.py instead of starting myguiapp.py (this assumes you are running an X-server on localhost:0.0).
you can check whether this works by simply launching the command with the DISPLAY-line from a virtualterminal.
to get the display you are actually using, simply run echo $DISPLAY in a terminal within your X-server.
usually, remote connections are disabled to your running Xserver (as it allows non-proviliged users to take over your desktop - everything from starting new GUI-programs (which is what you want) to installing keyloggers); if that's the case check man xhost (or go for solution #1)
UPDATE
for the 1st solution, you probably want to check out libraries like libnotify (there's python bindings in python-notify and python-notify2).
if you want more than simple "notification popups", you probably have to dig into D-BUS.
a simple example (haven't tested it personally, though), can be found here.

Python Daemon Packaging Best Practices

I have a tool which I have written in python and generally should be run as a daemon. What are the best practices for packaging this tool for distribution, particularly how should settings files and the daemon executable/script be handled?
Relatedly are there any common tools for setting up the daemon for running on boot as appropriate for the given platform (i.e. init scripts on linux, services on windows, launchd on os x)?

The best tool I found for helping with init.d scripts is "start-stop-daemon". It will run any application, monitor run/pid files, create them when necessary, provide ways to stop the daemon, set process user/group ids, and can even background your process.
For example, this is a script which can start/stop a wsgi server:
#! /bin/bash
case "$1" in
start)
echo "Starting server"
# Activate the virtual environment
. /home/ali/wer-gcms/g-env/bin/activate
# Run start-stop-daemon, the $DAEMON variable contains the path to the
# application to run
start-stop-daemon --start --pidfile $WSGI_PIDFILE \
--user www-data --group www-data \
--chuid www-data \
--exec "$DAEMON"
;;
stop)
echo "Stopping WSGI Application"
# Start-stop daemon can also stop the application by sending sig 15
# (configurable) to the process id contained in the run/pid file
start-stop-daemon --stop --pidfile $WSGI_PIDFILE --verbose
;;
*)
# Refuse to do other stuff
echo "Usage: /etc/init.d/wsgi-application.sh {start|stop}"
exit 1
;;
esac
exit 0
You can also see there an example of how to use it with a virtualenv, which I would always recommend.

To answer one part of your question, there are no tools I know of that will do daemon setup portably even across Linux systems let alone Windows or Mac OS X.
Most Linux distributions seem to be using start-stop-daemon within init scripts now, but you're still going to have minor difference in filesystem layout and big differences in packaging. Using autotools/configure, or distutils/easy_install if your project is all Python, will go a long way to making it easier to build packages for different Linux/BSD distributions.
Windows is a whole different game and will require Mark Hammond's win32 extensions and maybe Tim Golden's WMI extensions.
I don't know Launchd except that "none of the above" are relevant.
For tips on daemonizing Python scripts, I would look to Python apps that are actually doing it in the real world, for example inside Twisted.

There are many snippets on the internet offering to write a daemon in pure python (no bash scripts)
http://www.jejik.com/articles/2007/02/a_simple_unix_linux_daemon_in_python/
looks clean...
If you want to write your own,
the principle is the same as with the bash daemon function.
Basically:
On start:
you fork to another process
open a logfile to redirect your
stdout and stderr
Save the pid somewhere.
On stop:
You send SIGTERM to the process with pid stored in your pidfile.
With signal.signal(signal.SIGTERM, sigtermhandler) you can bind a stopping
procedure to the SIGTERM signal.
I don't know any widely used package doing this though.

Check the Ben Finney's daemon module. He has started to write a PEP targeting python 3.X:
http://www.python.org/dev/peps/pep-3143/
But an implementation is already available here :
http://pypi.python.org/pypi/python-daemon/

Not a silver bullet for what you're asking, but check out supervisord. It handles all the fun bits of managing processes. I use it heavily in a large production environment. Also, it's written in Python!

I can't remember where I downloaded it... but this is the best daemonizing script that I've found. It works beautifully (on Mac and Linux.) (save it as daemonize.py)
import sys, os
def daemonize (stdin='/dev/null', stdout='/dev/null', stderr='/dev/null'):
# Perform first fork.
try:
pid = os.fork( )
if pid > 0:
sys.exit(0) # Exit first parent.
except OSError, e:
sys.stderr.write("fork #1 failed: (%d) %sn" % (e.errno, e.strerror))
sys.exit(1)
# Decouple from parent environment.
os.chdir("/")
os.umask(0)
os.setsid( )
# Perform second fork.
try:
pid = os.fork( )
if pid > 0:
sys.exit(0) # Exit second parent.
except OSError, e:
sys.stderr.write("fork #2 failed: (%d) %sn" % (e.errno, e.strerror))
sys.exit(1)
# The process is now daemonized, redirect standard file descriptors.
for f in sys.stdout, sys.stderr: f.flush( )
si = file(stdin, 'r')
so = file(stdout, 'a+')
se = file(stderr, 'a+', 0)
os.dup2(si.fileno( ), sys.stdin.fileno( ))
os.dup2(so.fileno( ), sys.stdout.fileno( ))
os.dup2(se.fileno( ), sys.stderr.fileno( ))
In your script, you would simply:
from daemonize import daemonize
daemonize()
And you can also specify places to redirect the stdio, err, etc...

On Linux systems, the system's package manager (Portage for Gentoo, Aptitude for Ubuntu/Debian, yum for Fedora, etc.) usually takes care of installing the program including placing init scripts in the right places. If you want to distribute your program for Linux, you might want to look into bundling it up into the proper format for various distributions' package managers.
This advice is obviously irrelevant on systems which don't have package managers (Windows, and Mac I think).

This blog entry made it clear for me that there are actually two common ways to have your Python program run as a deamon (I hadn't figured that out so clearly from the existing answers):
There are two approaches to writing daemon applications like servers
in Python.
The first is to handle all the tasks of sarting and
stopping daemons in Python code itself. The easiest way to do this is
with the python-daemon package which might eventually make its way
into the Python distribution.
Poeljapon's answer is an example of this 1st approach, although it doesn't use the python-daemon package, but links to a custom but very clean python script.
The other approach is to use the tools
supplied by the operating system. In the case of Debain, this means
writing an init script which makes use of the start-stop-daemon
program.
Ali Afshar's answer is a shell script example of the 2nd approach, using the start-stop-daemon.
The blog entry I quoted has a shell script example, and some additional details on things such as starting your daemon at system startup and restarting your daemon automatically when it stopped for any reason.

correct me if wrong, but I believe the question is how to DEPLOY the daemon. Set your app to install via pip and then make the entry_point a cli(daemon()). Then create an init script that simply runs $app_name &

"generally should be run as a daemon?"
Doesn't -- on surface -- make a lot of sense. "Generally" isn't sensible. It's either a a daemon or not. You might want to update your question.
For examples of daemons, read up on daemons like Apache's httpd or any database server (they're daemons) or the SMTPD mail daemon.
Or, perhaps, read up on something simpler, like the FTP daemon, SSH daemon, Telnet daemon.
In Linux world, you'll have your application installation directory, some working directory, plus the configuration file directories.
We use /opt/ourapp for the application (it's Python, but we don't install in Python's lib/site-packages)
We use /var/ourapp for working files and our configuration files.
We could use /etc/ourapp for configuration files -- it would be consistent -- but we don't.
We don't -- yet -- use the init.d scripts for startup. But that's the final piece, automated startup. For now, we have sys admins start the daemons.
This is based, partly, on http://www.pathname.com/fhs/ and http://tldp.org/LDP/Linux-Filesystem-Hierarchy/html/Linux-Filesystem-Hierarchy.html.