We Keep Coding

How to keep AWS Lightsail virtual server running [duplicate]

I have set up Flask on my Rapsberry Pi and I am using it for the sole purpose of acting as a server for an xml file which I created with a Python script to pass data to an iPad app (iRule).
My RPI is set up as headless and my access is with Windows 10 using PuTTY, WinSCP and TightVNC Viewer.
I run the server by opening a terminal window and the following command:
sudo python app1c.py
This sets up the server and I can access my xml file quite well. However, when I turn off the Windows machine and the PuTTY session, the Flask server shuts down!
How can I set it up so that the Flask server continues even when the Windows machine is turned off?
I read in the Flask documentation:
While lightweight and easy to use, Flask’s built-in server is not suitable for production as it doesn’t scale well and by default serves only one request at a time.
Then they go on to give examples of how to deploy your Flask application to a WSGI server! Is this necessary given the simple application I am dealing with?

Use:
$ sudo nohup python app1c.py > log.txt 2>&1 &
nohup allows to run command/process or shell script that can continue running in the background after you log out from a shell.
> log.txt: it forword the output to this file.
2>&1: move all the stderr to stdout.
The final & allows you to run a command/process in background on the current shell.

Install Node package forever at here https://www.npmjs.com/package/forever
Then use
forever start -c python your_script.py
to start your script in the background. Later you can use
forever stop your_script.py
to stop the script

You have multiple options:
Easy: deattach the process with &, for example:
$ sudo python app1c.py &
Medium: install tmux with apt-get install tmux
launch tmux and start your app as before and detach with CTRL+B.
Complexer:
Read run your flask script with a wsgi server - uwsgi, gunicorn, nginx.

Been stressing lately so I decide to go deep.
pm2 start app.py --interpreter python3
Use PM2 for things like this. I also use it for NodeJs app and a Python app on a single server.

Use:
$sudo python app1c.py >> log.txt 2>&1 &
">> log.txt" pushes all your stdout inside the log.txt file (You may check the application logs in it)
"2>&1" pushes all the stderr inside the log.txt file (This would push all the error logs inside log.txt)
"&" at the end makes it run in the background.
You would get the process id immediately after executing this command with which you can monitor or verify it.
$sudo ps -ef | grep <process-id>
Hope it helps..!!

You can always use nohup to run any scripts as background process.
nohup python script.py
This will run your script in background and also have its logs appended in nohup.out file which will be located in the directory script.py is store.
Make sure, you close the terminal and not press Ctrl + C. This will allow it to run in background even when you log out.
To stop it from running , ssh in to the pi again and run ps -ef |grep nohup and kill -9 XXXXX
where XXXX is the pid you will get ps command.

I've always found a detached screen process to be best for use cases such as these.
Run:
screen -m -d sudo python app1c.py

I was trying to run my flask app for testing in my GitHub CI and the step where I was running the app was getting stuck for ever. The reason was that it was never releasing the command line
Best solution I found was a combination of two other responses in here:
nohup python script.py &

Checking for dead links locally in a static website (using wget?)

A very nice tool to check for dead links (e.g. links pointing to 404 errors) is wget --spider. However, I have a slightly different use-case where I generate a static website, and want to check for broken links before uploading. More precisely, I want to check both:
Relative links like file.pdf
Absolute links, most likely to external sites like example.
I tried wget --spyder --force-html -i file-to-check.html, which reads the local file, considers it as HTML and follows each links. Unfortunately, it can't deal with relative links within the local HTML file (errors out with Cannot resolve incomplete link some/file.pdf). I tried using file:// but wget does not support it.
Currently, I have a hack based on running a local webserver throught python3 http.serve and checking the local files through HTTP:
python3 -m http.server &
pid=$!
sleep .5
error=0
wget --spider -nd -nv -H -r -l 1 http://localhost:8000/index.html || error=$?
kill $pid
wait $pid
exit $error
I'm not really happy with this for several reasons:
I need this sleep .5 to wait for the webserver to be ready. Without it, the script fails, but I can't guarantee that 0.5 seconds will be enough. I'd prefer having a way to start the wget command when the server is ready.
Conversely, this kill $pid feels ugly.
Ideally, python3 -m http.server would have an option to run a command when the server is ready and would shutdown itself after the command is completed. That sounds doable by writing a bit of Python, but I was wondering whether a cleaner solution exists.
Did I miss anything? Is there a better solution? I'm mentioning wget in my question because it does almost what I want, but using wget is not a requirement for me (nor is python -m http.server). I just need to have something easy to run and automate on Linux.

So I think you are running in the right direction. I would use wget and python as they are two readily available options on many systems. And the good part is that it gets the job done for you. Now what you want is to listen for Serving HTTP on 0.0.0.0 from the stdout of that process.
So I would start the process using something like below
python3 -u -m http.server > ./myserver.log &
Note the -u I have used here for unbuffered output, this is really important
Now next is waiting for this text to appear in myserver.log
timeout 10 awk '/Serving HTTP on 0.0.0.0/{print; exit}' <(tail -f ./myserver.log)
So 10 seconds is your maximum wait time here. And rest is self-explanatory. Next about your kill $pid. I don't think it is a problem, but if you want it to be more like the way a user does it then I would change it to
kill -s SIGINT $pid
This will be equivalent to you processing CTRL+C after launching the program. Also I would handle the SIGINT my bash script as well using something like below
https://unix.stackexchange.com/questions/313644/execute-command-or-function-when-sigint-or-sigterm-is-send-to-the-parent-script/313648
The above basically adds below to top of the bash script to handle you killing the script using CTRL+C or external kill signal
#!/bin/bash
exit_script() {
echo "Printing something special!"
echo "Maybe executing other commands!"
trap - SIGINT SIGTERM # clear the trap
kill -- -$$ # Sends SIGTERM to child/sub processes
}
trap exit_script SIGINT SIGTERM

Tarun Lalwani's answer is correct, and following the advices given there one can write a clean and short shell script (relying on Python and awk). Another solution is to write the script completely in Python, giving a slightly more verbose but arguably cleaner script. The server can be launched in a thread, then the command to check the website is executed, and finally the server is shut down. We don't need to parse the textual output nor to send a signal to an external process anymore. The key parts of the script are therefore:
def start_server(port,
server_class=HTTPServer,
handler_class=SimpleHTTPRequestHandler):
server_address = ('', port)
httpd = server_class(server_address, handler_class)
thread = threading.Thread(target=httpd.serve_forever)
thread.start()
return httpd
def main(cmd, port):
httpd = start_server(port)
status = subprocess.call(cmd)
httpd.shutdown()
sys.exit(status)
I wrote a slightly more advanced script (with a bit of command-line option parsing on top of this) and published it as: https://gitlab.com/moy/check-links

How to run Flask Server in the background

I have set up Flask on my Rapsberry Pi and I am using it for the sole purpose of acting as a server for an xml file which I created with a Python script to pass data to an iPad app (iRule).
My RPI is set up as headless and my access is with Windows 10 using PuTTY, WinSCP and TightVNC Viewer.
I run the server by opening a terminal window and the following command:
sudo python app1c.py
This sets up the server and I can access my xml file quite well. However, when I turn off the Windows machine and the PuTTY session, the Flask server shuts down!
How can I set it up so that the Flask server continues even when the Windows machine is turned off?
I read in the Flask documentation:
While lightweight and easy to use, Flask’s built-in server is not suitable for production as it doesn’t scale well and by default serves only one request at a time.
Then they go on to give examples of how to deploy your Flask application to a WSGI server! Is this necessary given the simple application I am dealing with?

Use:
$ sudo nohup python app1c.py > log.txt 2>&1 &
nohup allows to run command/process or shell script that can continue running in the background after you log out from a shell.
> log.txt: it forword the output to this file.
2>&1: move all the stderr to stdout.
The final & allows you to run a command/process in background on the current shell.

Install Node package forever at here https://www.npmjs.com/package/forever
Then use
forever start -c python your_script.py
to start your script in the background. Later you can use
forever stop your_script.py
to stop the script

You have multiple options:
Easy: deattach the process with &, for example:
$ sudo python app1c.py &
Medium: install tmux with apt-get install tmux
launch tmux and start your app as before and detach with CTRL+B.
Complexer:
Read run your flask script with a wsgi server - uwsgi, gunicorn, nginx.

Been stressing lately so I decide to go deep.
pm2 start app.py --interpreter python3
Use PM2 for things like this. I also use it for NodeJs app and a Python app on a single server.

Use:
$sudo python app1c.py >> log.txt 2>&1 &
">> log.txt" pushes all your stdout inside the log.txt file (You may check the application logs in it)
"2>&1" pushes all the stderr inside the log.txt file (This would push all the error logs inside log.txt)
"&" at the end makes it run in the background.
You would get the process id immediately after executing this command with which you can monitor or verify it.
$sudo ps -ef | grep <process-id>
Hope it helps..!!

You can always use nohup to run any scripts as background process.
nohup python script.py
This will run your script in background and also have its logs appended in nohup.out file which will be located in the directory script.py is store.
Make sure, you close the terminal and not press Ctrl + C. This will allow it to run in background even when you log out.
To stop it from running , ssh in to the pi again and run ps -ef |grep nohup and kill -9 XXXXX
where XXXX is the pid you will get ps command.

I've always found a detached screen process to be best for use cases such as these.
Run:
screen -m -d sudo python app1c.py

I was trying to run my flask app for testing in my GitHub CI and the step where I was running the app was getting stuck for ever. The reason was that it was never releasing the command line
Best solution I found was a combination of two other responses in here:
nohup python script.py &

Launching SimpleHTTPServer in background

I am trying to write a shell script that launches a server and then fires up a browser to view the served content
(cd app && python -m SimpleHTTPServer) &
open http://localhost:8000
Disclaimer: I'm new to bash scripting and I don't really know python.
It seems that when adding the & operator at the end of the line, the process doesn't stay alive long enough to remain functional.
What am I doing wrong?

The problem is (probably) not that the server doesn't stay alive long enough; it's that it doesn't start up fast enough. When the open command executes, the webserver hasn't started yet, so you get an error.
Try sleeping for a second or two before the open

SQLite error when loading two python scripts simultaneously

I have two python scripts that have to run simultaneously because they interact with each other. One script is a 'server' script running locally and the other is client script that connects to it via a socket. Normally I just open a couple terminal tabs and run the server script in one and the client in the other. After starting and stopping each script over and over, I wanted to make a bash alias to run both scripts with just one command and came up with this:
gnome-terminal --tab -e "python server.py" --tab -e "python client.py"
However, now the server script is raising an sqlite OperationalError saying that one of my data tables doesn't exist. But when I run the scripts manually everything works fine. I have no clue what is going on, but I thought that maybe running the scripts together wasn't giving the server script enough time to initialize and make its connection to the database. So I put a time.sleep(5) in the client script, but as soon as it starts I get the same error.
Anyone have an idea what could be happening? Or does anyone know of any alternatives for starting two python scripts with one command?

Try combining the two commands into one:
gnome-terminal --tab -x bash -c "python server.py & sleep 5; python client.py"
I think it is better to put the sleep command (if needed) outside client since there may be situations where the server is already started and the client does not have to sleep.
The -x flag means
-x, --execute
Execute the remainder of the command line inside the terminal.
The command calls bash:
bash -c "python server.py & sleep 5; python client.py"
bash in turn, has a -c flag which means
-c string If the -c option is present, then commands are read from string. If
there are arguments after the string, they are assigned to the posi‐
tional parameters, starting with $0.
You might want to experiment with
gnome-terminal --tab -e "python server.py & sleep 5; python client.py"
That might work too. When you run bash first, then your ~/.bashrc is read. Without calling bash, I think by default, /bin/sh is called instead.
If you get
"socket.error: [Errno 98] Address already in use",
it probably means that your server has already been started, and running the server a second time fails.