I want to create a micro-VPN like environment where users can run and manage HPC applications from strictly python. The servers running the commands on the HPC systems should only ever talk to the user's host python script, and should be firewalled from communicating with anything else. Basically like a python level TailScale.
My original idea was to use Wireguard + pyroute2, but that requires a kernel level install which might not always be possible from the user's host machine. Is there any solution to this that is near pure python (or python + pip install) that runs only in users space?
Related
I and other team members develop, test, and debug our compute-intensive Python code on a cloud-based Linux server using large datasets and many CPUs/GPUs. During the day there can be one or more users with interactive sessions on this machine (e.g. SSH console or PyCharm over SSH) specifically so we can debug.
The cloud instance we run on costs $10-$20 per hour, which is fine as long as people are using it. We try to remember to shut it down manually when nobody is using it (which requires checking that others aren't logged in). Sometimes we forget, which can cost ~$300 overnight or $1,000 if left idle over a weekend. Note that user sessions can be set to timeout on the client side by configuring OpenSSH, but that leaves the server running.
How to set up scripts and configurations that either:
detect that all interactive users have been idle for X hours ("ideal" condition); or
detect that there have been no interactive sessions for X >= 0 hours ("good-enough" condition); and
sudo shutdown now when the condition is detected?
I'm aware that (for example) on AWS there are some hacky/complex/proprietary/unreliable ways to sort of do this by setting up external monitor services, and I assume there are similar kludges for GCP and Azure. We may want to do similar things on different cloud platforms (AWS, GCP, Azure), but on all of them we'd likely use Ubuntu 20.04+ as the common environment, so I'm looking for implementations that can be coded at the Ubuntu/Linux level.
I would prefer that solutions are based on bash or python. Assume all users are sudoers.
I've already tried proprietary services that are unreliable and not portable.
I'm familiar with python within the 3D application I use (OSX platform), but am struggling with its usage in a client/server relationship. I've written a simple distributed rendering script which breaks my 3D render script into smaller OSX bash shell scripts and saves them to a directory on my machine. The remote machines in the room then look at my local folder with these smaller bash shell scripts and execute them one by one until they are all gone. It is a rudimentary solution to distributed rendering, but it works. What I would like to do is have the remote machines listen for a command from my local machine (the local machine would need to send an OSX bash command to the remote machines). I have been looking and this site: http://www.tutorialspoint.com/python/python_networking.htm
This seems to be what I'm looking for, but not knowing much about how python works with the network, I'm not sure whether its secure, and I am not sure how to send a Bash command rather than a message.
If anyone has any suggestions it'd be much appreciated.
In our company we using vagrant VM's to have the save env. for all. Is it possible to configure VisualStudio + PTVS (python tools for VS) to use vagrant based python interpreter via ssh for example?
There's no special support for remote interpreters in PTVS, like what PyCharm has. It's probably possible to hack something based on the existing constraints, but it would be some work...
To register an interpreter that can actually run, it would have to have a local (well, CreateProcess'able - so e.g. SMB shares are okay) binary that accepts the same command line options as python.exe. It might be possible to use ssh directly by adding the corresponding command line options to project settings. Otherwise, a proxy binary that just turns around and invokes the remote process would definitely work.
Running under debugger is much trickier. For that to work, the invoked Python binary would also have to be able to load the PTVS debugging bits (which is a bunch of .py files in PTVS install directory), and to connect to VS over TCP to establish a debugger connection. I don't see how this could be done without writing significant amounts of code to correctly proxy everything.
Attaching to a remotely running process using ptvsd, on the other hand, would be trivial.
For code editing experience, you'd need a local copy (or a share etc) of the standard library for that interpreter, so that it can be analyzed by the type inference engine.
We run many Python scripts for data processing tasks. We have a modeling computer that has been upgraded to provide the best performance for these tasks, but it is shared by many people that all need to run different scripts on it at the same time.
Is it possible for me to run a Python script remotely on that machine from my laptop while others are either directly logged into it or also remotely running a script?
Is SSH a possibility? I haven't ever run any scripts remotely aside from logging in via remote desktop. Ideally, I could start the Python script on that remote machine, but all the messages would be visible to me on my laptop. Does this sound doable?
EDIT:
I forgot to mention all machines are running Windows 7.
SSH is definitely the way to go and also have a look at Fabric.
Regarding your edit. You can use Fabric on Windows. And I think that using SSH on Windows will be a bit easier than dancing with their Powershell's remoting capabilities.
SSH does seem like it should meet your needs.
You could also consider setting up an iPython notebook server that everyone could use.
Its got nice parallel processing capabilities if you are doing some serious number crunching.
I'm writing a Python script to access all computers on the network, log in to them and read some log files. I don't want to use something as low-level as socket, but I can if I must. I realize that my problem is similar to this question, but not the same.
Are there any modules for accessing external Windows machines?
Has anyone done anything like this before?
I'm specifically looking to log into Windows 7 machines, not unix.
Let's also assume that each computer I want to log into has Remote Desktop installed and enabled. I'm also not worried about network security or encryption because these files are not confidential. Windows machines don't have SSH installed on the by default do they?
There has to be something on the other side for you to talk to. This limits you to either setting up a "server" on each machine, installing a real server (i.e. sshd), building a "server" yourself and installing it, or using a built in and active feature of the OS.
Based upon this, what kind of system do you want to set up on these machines? What does it need to do? Just read the contents of a prespecified file list? Will that list change?
One solution is to turn on telnet, and use paramiko or twisted to
talk across it. This isn't very secure of course
Next up, set up a samba share, and access the folder remotely. This
is also insecure, though less so than telnet
You could find a ssh daemon port and run that, if you are so inclined
Psexec from sysinternals might work
Use twisted to build a server app with the features you need
Use ncat to listen on a port and spawn a cmd prompt
Be aware that most of the solutions for accessing windows remotely are... poor. The best solution is probably to roll your own, but that is hard work and you will probably make mistakes.
Also, Windows 7 is not exactly multi-user friendly. Individual processes can run as separate users, but the OS does not support having multiple users logged in at the same time. Someone is going to be the "user" and everyone else is just a process with a different credential set.
This is more an artificial limitation on M$'s part than anything technical. To see this in action, try to log in with RDP while a user is logged in locally. Fun times.
Per your edit, the easiest thing to do is just set up a samba share on the box.
After this share is set up:
with open(r'\\myCompNameOrIP\C\windows\logs\logfile.txt','rb') as logfile:
loglines = logfile.readlines()
Or you can use the gencat sample found here. Just give it r'\\myCompNameOrIP\C\windows\logs\*.txt' as the search path and watch the magic.
From Ubuntu I use samba:
In Bash:
gvfs-mount smb://them/folder
Here I give name, domain and password
Then in python:
folder = '/home/me/.gvfs/folder on them'
using the os module I read folders and files inside.
I am working in a small business environment.
Why not have each of the computers send the log file to the central computer?