Remove the "-u" parameter when executing code remotely from PyCharm - python

I have set up a Remote Interpreter to a Server from PyCharm. The connection is working fine and i have also configured the Deployment from my local machine to this server.
The interpreter i configured is from Spark (/usr/bin/spark-submit). But, when i run my code, it outputs the following error:
Error: Unrecognized option: -u
Below is the command that is being sent to the remote server:
ssh://user#MYHOST:22/usr/bin/spark-submit -u /home/user/pycharm/main.py
I have tried with the Python Interpreter, using the Python from the remote server and it worked just fine. The problem is that spark-submit does not accept the option -u but i cannot remove that option from PyCharm. Also, i've removed the PYTHONUNBUFFERED=1 environment variable from the Run Configurations in PyCharm, but it did not work.
Field which i have removed the PYTHONUNBUFFERED=1 option
Parameters Field in Run Configurations:
I have also searched the JetBrains forum, but didn't find anything that refers to this problem.

Related

Modules appear to not be installed when running python through ssh

I am connecting to a remote server through
ssh user#server.com
and run
python script.py
in the appropriate directory. However, I get the error
ImportError: No module named numpy
even though I know the module is installed and the script runs with no problems when I am physically logged in to that server.
None of the answers I was able to find worked (for example this, and this). Do have any ideas as to how I can run the script using ssh?
The remote server has Python 2.6.6 installed, and
which python
returns
/usr/bin/python
The remote serves runs CentOS.
See similar problem describe here: Why does an SSH remote command get fewer environment variables then when run manually?.
Compare your environment variables in the local (physical) mode to the remote mode by running env in both cases. Move missing variables from your local profile to /etc/profile. Then log out from ssh session and connect again.
Another approach: If you don't want to change anything, then after ssh switch to your user via su - <your user>. This may look weird because you already logged it with this user. The difference is, that after su all your env. variables will set like in a local (physical) mode. Advantage: it is quick. Disadvantage: You will have to do it each time you want to run your Python script. So the first approach with configuring /etc/profile may be better on the long run.

WLST disconnect command issue

I had ran wlst.cmd in my local system after I ran my Weblogic Admin Instance. But as WLST is stateful, I am getting connected to my IT env which is my Integration Testing environment (some UNIX machine for my project). I tried disconnect() to goto offline mode, however it is failed.
wls:/beaProjDir/serverConfig> disconnect()
You will need to be connected to a running server to execute this command
Please help to go offline mode in WLST. As I need to get some work done in my local system.
You can try to getting out of WLST Shell ctrl+D. Which is similar to exiting from any other shell.

How to debug remote python script?

I am looking for a way to debug remote python script without any set up like in the answer here
I set remote interpreter, set up a debug configuration, click a debug button and got error Event not found:
ssh://myuser#my_ip:my_port/usr/local/bin/python -u /myuser/.pycharm_helpers/pydev/pydevd.py --multiproc --qt-support --client '0.0.0.0' --port 46994 --file /opt/my_work_dir/my_script.py
/: Event not found.
Seems like pyCharm takes care about all debug setting --multiproc --qt-support --client '0.0.0.0' --port 46994 but I have the wrong configuration.
Is there a way how to debug python script without putting pydevd.settrace('host', port=21000, stdoutToServer=True, stderrToServer=True) and other actions?
I was able to fix this problem by reinstalling PyCharm.
(# 1 and 5 are optional)
Export minimum settings you want to restore such as Keymap (File > Export Settings)
Uninstall your current PyCharm
Install new PyCharm
Launch new PyCharm and don't import your pre-existing setting at the first start up
Import your settings from the old PyCharm (File > Import Settings)
I guess this happens when some environment related settings are messed up.
Notice that you need to remember your settings for the remote interpreter as it will be gone when reinstalling.
I use vagrant quite often with pycharm and debugging has always worked great for me as long as i setup the interpreter to the one inside the vagrant box running.
If you go to settings->project->project interpreter and then setup a connection to your remote interpreter it will usually install the pycharm-debug.egg etc and you can debug run your project against this interpreter and set breakpoints in pycharm and it will break on those.
Here is their docs on setting this up https://www.jetbrains.com/help/pycharm/2016.1/configuring-remote-python-interpreters.html
I set up remote debugging as described under the section
Remote debug with a remote interpreter
(not the Server!!) and it works very well.
Here the link: https://www.jetbrains.com/help/pycharm/2016.1/remote-debugging.html
You just have to set up the remote interpreter (I used ssh) and then set up your run configurations. Let me know if you need further help.
Here is my run configuration in PyCharm:
It might be the same issue as this. Use bash as your login shell if you use another one like csh or tcsh.

pycharm can't complete remote interpreter setup for Docker

I'm new to Docker. I'm using Docker & docker-compose, going through a flask tutorial. The base docker image is python 2.7 slim.
It's running on Linux. docker 1.11.2
The application is working fine.
I want to get pycharm pro connecting to the remote interpreter, something I have never done before.
I followed the instructions for docker-compose. Initially it was failing because it could not connect to port 2376. I added this port to docker-compose.yml and the error went away.
However, trying to save the configuration now stalls/hangs with a dialog 'Getting Remote Interpreter Version'. This never completes. Also, I can't quit pycharm. This happens in Pycharm 2016.2 and 2016.3 EAP (2nd).
The help say "SFTP support is required for copying helpers to the server".
Does this mean I need to do something?
I'm not using docker-machine
The problem was that TCP access to the docker API is not established by default under ubuntu 16.04.
There are suggestions to enable TCP/IP access.
However, JetBrains gave me the simplest solution:
If you are using Linux it is most likely that Docker installed with
its default setup and Docker is expecting to be used through UNIX
domain file socket /var/run/docker.sock. And you should specify
unix:///var/run/docker.sock in the API URL field. Please comment
whether it helps!
This suggestion worked with my Ubuntu 16.04 -derived distribution.
This goes into the Docker entry in PyCharm preferences under Build, Execution, Deployment.
You can also edit this while setting up a remote interpreter, but only by making a new Docker entry.
TCP/IP Method
This method works if you want TCP/IP access, but this is a security risk. The socket approach is better, which is probably why it is the default.
https://coreos.com/os/docs/latest/customizing-docker.html
Customizing docker
The Docker systemd unit can be customized by overriding the unit that
ships with the default CoreOS settings. Common use-cases for doing
this are covered below.
Enable the remote API on a new socket
Create a file called /etc/systemd/system/docker-tcp.socket to make
Docker available on a TCP socket on port 2375.
[Unit]
Description=Docker Socket for the API
[Socket]
ListenStream=2375
BindIPv6Only=both
Service=docker.service
[Install]
WantedBy=sockets.target
Then enable this new socket:
systemctl enable docker-tcp.socket
systemctl stop docker
systemctl start docker-tcp.socket
systemctl start docker
Test that it’s working:
docker -H tcp://127.0.0.1:2375 ps
Once I thought to search for ubuntu 16.04 I came across simpler solutions, but I did not test them.
For instance:
https://www.ivankrizsan.se/2016/05/18/enabling-docker-remote-api-on-ubuntu-16-04/
Edit the file /lib/systemd/system/docker.service
Modify the line that starts with ExecStart to look like this:
ExecStart=/usr/bin/docker daemon -H fd:// -H tcp://0.0.0.0:2375
Where my addition is the “-H tcp://0.0.0.0:2375” part. Save the
modified file. Restart the Docker service:
sudo service docker restart
Test that the Docker API is indeed accessible:
curl http://localhost:2375/version
I - docker-compose up
I think PyCharm will run docker-compose up, have you try to run this command first in your terminal (from where your docker-compose.yml is) ?
Maybe if some errors occur, you will get more info in your terminal.
II - pycharm docker configuration
Otherwise it could be due to your docker machine configuration in PyCharm.
What I do to configure my machine and to be sure this one is correctly configured:
1 - run docker-machine ls in your shell
2 - copy paste the url without tcp://
3 - go to pycharm preferences -> Build, Execution, Deployement -> Docker -> + to create a new server, fill the server name field
4 - paste previously copied url keeping https://
5 - fill the path of your machine certificates folder
6 - tick Import credentials from Docker Machine
7 - click Detect -> your machine should appear in the selection list
8 - save this server
9 - select this server when configuring your remote interpreter, from PyCharm Preferences -> Project -> Project Interpreter -> wheel -> add remote -> Docker or Docker Compose
10 - you should be able to select a service name
11 - save your new interpreter
11 - try run your test twice, sometimes it could take time to initialize

Running docker in subprocess on Python PyCharm

I have a command that looks like:
p = subprocess.Popen(['docker', 'run', 'imagename'])
in a Python program. I am able to execute this successfully from terminal, however when I run it in PyCharm I receive this error:
Cannot connect to the Docker daemon. Is the docker daemon running on this host
How can I fix this error to run in the Python IDE?
The key is in understanding what eval "$(docker-machine env dockermachinename)" returns (where dockermachinename is your docker machine name(you can check the name with "docker-machine ls" command)).
When you run docker-machine env dockermachinename which is what you need to configure your shell to connect with Docker, it prints:
export DOCKER_TLS_VERIFY="1"
export DOCKER_HOST="tcp://999.999.99.999:999"
export DOCKER_CERT_PATH="/Users/enderland/.docker/machine/machines/dockermachinename"
export DOCKER_MACHINE_NAME="dockermachinename"
# Run this command to configure your shell:
# eval $(docker-machine env default)
These environment variables need to be within PyCharm. By adding them into your configuration environment variables list, you will be able to connect with Docker.
This assumes your Docker machine is running (if not, you need to do docker-machine start dockermachinename).

Categories