We Keep Coding

Going back to running script in virtual environment after connection to remote server closed

I am running a Python script that collects data and is running inside a Virtual Environment hosted remotely on a VPS (Debian based).
My PC crashed and I am trying to get back into the visual logs of the python script.
I know that the script is still running because it saves its data into a CSV file. That CSV is still being written.
If I activate the source again, then I can rerun the script. It sounds to me that I will have 2 instances of the same script running in this case...
I am not familiar with the virtual environment and I cannot find the right way to do it without deactivating and reactivating it. I am running my script on the cheapest OVH VPS I could buy because my computer is clearly not reliable for running 24/7.

You might use screen to run your script in a separate terminal session. This will avoid losing logging if the ssh connection gets dropped.
The workflow would be something in the lines of (on your host):
# Install screen
$ sudo apt udpate
$ sudo apt install screen
# Start a screen session
$ screen
# Run your script
$ python myscript.py
In case of dropping your ssh connections, it'll be enough to:
# ssh back into the host from your client
# reattach previous screen session
$ screen -r
For advanced use the official docs are quite comprehensive.
Note: As a more general note, what explained above is pretty much the basic logic of a terminal mulitplexer. You'll be able to achieve the same using tmux.

How to let the python script (abc.py) keep executing on AWS even after Connection lost or SSH connection is terminated?

I was using Jupyter notebook on AWS AMI and using the port forwarding on Windows using PuTTY. I got my connection terminated and all the work of 24 hours is lost now and I'm unable to get that. So I used a script instead of Notebook and the same thing happened. I used to think that the process would keep going on even if the Shell connection is lost. But next time I login, I don't see anything.
I used top, htop aux to find if my processes are still running but they don't show my process. Please help how can I stop this from happening.
I am using Windows on 10 with Putty on local and Ubuntu 18 on the AWS AMI.

You can try
nohup python3 myscript.py &

Try the tmux
apt install tmux
I often use it to keep my web runing on server(not professional but convenient）

You can use screen.
You can install it wit:
sudo apt install screen
You can find more information right here: https://linuxize.com/post/how-to-use-linux-screen/

: cannot connect to X server python cv2

When I am using cv2.imshow() while running SLIC, I get the below error.
: cannot connect to X server
I am running this program on a remote machine with GPU access. I have seen this answer and a few other related answers but none of them solve my issue. Can you suggest on how to proceed with this.

This helped me :
In your python script add these lines :
import os
os.environ['DISPLAY'] = ':0'
This problem means that your code is trying (and failing) to connect to an X server -- a GUI environment -- presumably being forwarded over your SSH session.

You have to allow the root user access to the X Server which some remote machines wont allow. Try this
xhost local:root
sudo DISPLAY=$DISPLAY gedit /etc/profile

How to use opencv in a remote jupyter notebook execution?

In my local machine with macOS Mojave, I installed a virtual environment with Python 3.6, opencv package and some additional unrelated packages.
I started a very simple Jupyter Notebook that is executed with no problem, here it's the code:
import cv2
print(cv2.__version__)
my_img = cv2.imread("colibri_763_460.jpeg",1)
cv2.imshow("Original", my_img)
cv2.waitKey(0)
cv2.destroyAllWindows()
Now, I'm trying to execute the same code in a Jupyter Notebook in my remote Ubuntu machine but I can't make it work. These are the steps that I took:
I copied the image file and the Notebook ypnb file to the remote ubuntu machine.
I opened an ssh session, selected a virtualenv with the same packages than my local machine and executed the "jupyter notebook"
I opened another terminal with ssh -L 8000:localhost:8888 mpastorg#mpgubu18 for the ssh tunneling
I opened my local browser in the localhost:8000 to execute the Notebooks that I have in my remote Ubuntu machine, it works with any Notebook not involving opencv.
I tried to solve my problem using two different ways:
Installing in the ubuntu remote machine the package opencv-contrib-python-headless: when I try to use cv2.imshow, I got the error method not found
Installing in the ubuntu remote machine the package opencv-contrib-python: I got the error that kernel is dead.
Does anyone know if there's any way to execute the remote jupyter notebook and see the picture locally? it maybe would be possible a workaround to embed the picture in the browser window?
Thank you very much
Marcos Pastor

I wonder if the image is being rendered using X - which you have not explicitly enabled, in both your host machine, nor in your ssh tunnel.
On the Host machine. (Where you want to see the Image).
Enter this comment
xhost+
This now allows remote X-Hosts to send you data. As you are on a Mac - you will need to have the XQuartz package installed.
Now modify your ssh command. - so that you allow X - this is typically done using a -X flag (Or it can be placed in your .ssh/config file like this
Host *
ForwardX11 yes
ForwardX11Trusted no
XAuthLocation /opt/X11/bin/xauth
ForwardAgent yes
Compression yes
KeepAlive yes
Compression yes
If you are not using a config try the -X to start with.
Hope that helps

Python plotting on remote server using PyCharm

I have started to learn Python and so far my setup has been following - Python 3.5 installation on Win10 64bit local machine with PyCharm as a great IDE. Everything works, matplotlib charts and other visual outputs display fine, debugging works, etc.
Now, I have came across some libraries which works only on Linux. I have set up Ubuntu 16.4 64bit VPS on Digital Ocean, installed Python 3.5. In PyCharm I have set up SFTP connection to remote host. Code running works, debugging works, however, I am not able to bring display output (matplotlib plots,...) to local (Win10) machine. As I am not at all familiar with Linux GUI environments (X11?), after googling I have following questions:
1) Should anything be installed on remote Linux machine? (e.g. x11 client/server/smth?)
2) Should anything be installed on local Win machine? (e.g. Xming?)
3) Should anything be configured on remote Linx machine? (e.g. X11 forwarding)
4) Should anything be configured on local Win machine PyCharm?
5) There are X11 forwarding settings in Putty and some have suggested to use those but I am not sure, should Putty session run in paraller with PyCharm and can that be avoided.
Thanks a lot!
PS - I have installed Jupyter Notebook (and latest Jupyter Lab) on remote machine and it works excellent, however I am still prefering PyCharm as primary IDE with better code completion, debugger and other perks.

Ok, after some more googling I finally managed to get this process working, hope it helps somebody:
1) on remote host (VPS, Ubuntu 16.04) I had to install X11 server, which I did by:
sudo apt-get install xorg
sudo apt-get install openbox
2) On remote host I had to make sure that X11Forwarding is enabled in /etc/ssh/sshd_config
3) On local Win10 machine I had to install Xming server and launch it with default settings.
4) On local Win10 machine I had to configure Putty to use X11 forwarding (Connection-> SSH -> X11 Forwarding) with default settings and keep connection open while running PyCharm (it seems there is no option in PyCharm to enable x11 forwarding, so putty must be running in the background)
5) On remote machine I had to check Display number (echo $DISPLAY) - this can be different for everyone. For me it was localhost:10.0
6) In PyCharm Run configuration -> Environment variables I had to add DISPLAY=localhost:10.0
After all these steps and Putty+Xming running in backgroud, I was able to execute remote code and bring graphic back to my Windows 10 PC!
PS - process is actually slow, I have to wait around 10 seconds before image is brought back to me. I am not sure why or how to speed it up. Might be another question. (reducing chipher strength and enabling compression does not help. It seems some sort of initialization problem with x11 remote and local)

Mac user should install XQuartz instead of Xming.
And another important thing: if you install xquartz via homebrew, you should relogin your macos or reboot.
As the rackpas's answer saying.