So I have a RedHat System in AWS running Spark on top of HDFS. Now I want to access PySpark from my local machine i.e. Interactive Python.
So, I installed Spyder-Py2 to connect to the remote AWS machine so I can access Spyder Python.
Route:
Using Cookbook: Connecting to a remote kernel via ssh, I started Kernel on AWS machine i.e. Server and copied the json file over to local machine.
Changed the ip in kernel1234.json to point to the public ip.
Went to Consoles in Spyder-Py2 and select "Connect to existing kernel".
Passed the kernel1234.json file, added username and host in the hostname section.
Passed my AWS pem key in SSH Key section and entered the password
But it fails with the below error:
Unable to connect to IPython kernel-1234.json
Can anybody please tell me what am I missing here ?
Note: The server on the AWS VM is still running.
Related
I have a Jupyter notebook server running on my local machine, with a gateway server running on a more powerful/remote machine.
I can write to a file on the remote machine with normal code in a normal notebook, but I can't figure out how to write to a file on my local machine. Is there some %magic command for being able to execute code locally, or any mechanism to access a local file, when using a gateway server?
I have this Jenkins server running on my Windows machine. What the jobs do is it runs some python code that is also on the same machine, to package .ipa file (for iOS) using UE4 engine and ships it.
Everything works just fine if I run python code locally. But when I run it through Jenkins, it will fail at some point where UE4 is trying to sign the package. This is partial error message I'm getting.
Using remote server 'XXX.YYY.ZZZ.56' on port 22 (user '***********')
[Remote] Using private key at ...\RemoteToolChainPrivate.key
ERROR: Unable to determine home directory for remote user. SSH output:
Host key verification failed.
I think for some reason Jenkins doesn't know where ssh related stuffs are located in windows.
How can I fix this issue?
Assuming you installed Jenkins as a Windows Service, then Jenkins is probably running as another user (not the user you sign in as).
In the Services app, you can change the user the service is running as if necessary. You can find the Services app here:
C:\ProgramData\Microsoft\Windows\Start Menu\Programs\Administrative Tools\services.lnk
See About Services for more info.
That explains why the remote cannot find the home directory (because the remote user has a different name than the service account user) and also why the host key verification failed (since you never accepted the host key as the service account user).
I wonder if there is any way to coding python with remote server data and run the python script on remote server with pycharm only installed on my local machine?
The remote server can be access by two ip address (ssh and ftp each), id and password.
But when I try to use pycharm professional project interpreter, there always an error like this (the ip address I used is for ssh):
I know the easiest way is install jupyter notebbok on my remote server, but I don't have permission to do that.
I have a compute engine instance running on Google cloud platform.
I would like to use the Python interpreter of the compute engine as a remote interpreter with Pycharm. This means that I would be using Pycharm on my local machine and running computations remotely.
Any clue on how to achieve this?
The following requires, as James Hirschhorn pointed out, the Professional verison of PyCharm.
Assign a public IP to the remote machine on GCP.
Run gcloud compute config-ssh to automatically add the VMs of your project to your ~/.ssh/config or manually add the public IP of your VM to it. If you skipped step 1. then you have to run gcloud compute config-ssh every time you re-start the remote VM, because it always gets a new IP assigned. The ~/.ssh/config gets populated with many entries in the following format:
Host the-vm-host-name # use this in PyCharm's Host field
HostName 123.456.789.00 # the VM's IP address
Use the Host name of the remote you want to connect in your Deployment configuration in PyCharm
Add a remote interpreter: select the remote server from the drop-down (the one previously created) and point PyCharm to the executable python of your Python installation.
Done
My understanding is that you need the Pycharm Ultimate Edition to support remote servers. If you have Ultimate, then you can follow these instructions.
It's fairly easy to accomplish.
You need:
PyCharm Pro
Create and format SSH keys
Config your Compute Engine instance with the SSH keys
Configure PyCharm
You can follow this tutorial that I wrote.
Here is what I am trying to do. I have a Windows VM and another Linux VM which is used as server. I have Spyder installed on my Windows VM and would like to run my Python code in Spyder on remote Linux server.
I did try using option in Spyder called "Connect to remote kernel" but it did not work and I am getting error "Could not open ssh tunnel ; Paramiko not available". I was using username#servername:22 for making ssh connection. Needless to say, I am able to ssh the machine using putty but not using Spyder. Any ideas how should I fix this?
I found another way to make a connection to an external server, here is the link explaining step by step.
Basically, you have to connect your client PC to the server through a PuTTY SSH tunnel, it will allow to redirect the client ports to the correct ipython kernel server ports.