I'm very new to docker. I want to build my python application within a docker container. As I build the application I want to be testing / running it in Pycharm and in the container I build.
How do I connect Pycharm pro to a specific container or image (either python or Anaconda)?
When I create a project, click pure python and then add remote, then clicking docker I get the following result
I'm running on Mac OS X El Capitan (10.11.6) with Docker version 1.12.1 and Pycharm Pro 2016.2.3
Docker-for-mac only supports connections over the /var/run/docker.sock socket that is listening on your OSX host.
If you try to add this to pycharm, you'll get the following message:
"Cannot connect: java.lang.ExceptionInInitializerError, caused by: java.lang.IllegalStateException: Only supported on Linux"
So PyCharm really only wants to connect to a docker daemon over a TCP socket, and has support for the recommended TLS protection of that socket. The Certificates folder defaults to the certificate folder for the default docker-machine machine, "default".
It is possible to implement a workaround to expose Docker for Mac via a TCP server if you have socat installed on your OSX machine.
On my system, I have it installed via homebrew:
brew install socat
Now that's installed, I can run socat with the following parameters:
socat TCP-LISTEN:2376,reuseaddr,fork,bind=127.0.0.1 UNIX-CLIENT:/var/run/docker.sock
WARNING: this will make it possible for any process running as any user on your whole mac to access your docker-for-mac. The unix socket is protected by user permissions, while 127.0.0.1 is not.
This socat command tells it to listen on 127.0.0.1:2376 and pass connections on to /var/run/docker.sock. The reuseaddr and fork options allow this one command to service multiple connections instead of just the very first one.
I can test that socat is working by running the following command:
docker -H tcp://127.0.0.1:2376 ps
If you get a successful docker ps response back, then you know that the socat process is doing its job.
Now, in the PyCharm window, I can put the same tcp://127.0.0.1:2376 in place. I should get a "Connection successful" message back:
This workaround will require that socat command to be running any time you want to use docker from PyCharm.
If you wanted to do the same thing, but with TLS, you could set up certificates and make them available for both pycharm and socat, and use socat's OPENSSL-LISTEN instead of the TCP-LISTEN feature. I won't go into the details on that for this answer though.
Related
I'm building a Docker container that have python and import some packages using 'pip install'.
I want to allow 'pip' to communicate only with pypi.org, and block it from communicating with any other channel, including the host that runs the docker container (in which we run the pip).
I tried to create some custom outbound rules in firewall (including blocking the container's IP address, the container's gateway IP address, and the entire 'nat' docker subnet), but somehow it keeps going around the firewall and having access to wherever it wants (and imports any package it tries).
To clarify, I'm first trying to block 'pip' entirely, even for pypi.org
Is it possible to achieve this blocking behavior? What rules should I set in Firewall (or directly in the docker network)?
*I'm running on Windows desktop with Docker desktop, and the container is running with the following image - mcr.microsoft.com/windows/nanoserver:1809.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 1 year ago.
Improve this question
How do you get a Ruby, Python and Node.js development environments running on Apple Silicon architecture. What about virtualization software e.g. Docker?
Programming languages such as Ruby, Node and Python should run on Apple M1 Chip but Docker is not supported as of now (They are working on it)
Docker for Mac Issue
https://github.com/docker/for-mac/issues/4733
Docker team might be working on fixing the issue as per this:
https://github.com/docker/roadmap/issues/142
My Recommendation:
Get it now only if you want to build iOS application. Since most of the people don't have the laptop right now, we might end up with a lot of other issues.
UPDATE:
This one is more appropriate answer now:
https://stackoverflow.com/a/65253659/8216911
I've tried many things and had some real trouble getting things working, but in the end, here is the simplest way I've found to get Docker running on a new Mac Silicon M1 chip.
Docker does not natively work, VirtualBox doesn't work, Parallels doesn't work, ... in the end, it goes down to using UTM to create a Virtual Machine, install Ubuntu server on it. Once you have that, you can install whatever you want on it (Docker, Node JS, Apache, PHP, MariaDB, ...).
Then you set eveything up so you can use all your favorite Mac OS tools ( Terminal, Transmit, VS Code, Safari ... ) to work, just as if you had all that on your local file system.
1 - Download Linux installation disk
Grab an ARM linux distribution. I took Ubuntu server 20.04 LTS:
https://cdimage.ubuntu.com/releases/20.04/release/ubuntu-20.04.1-live-server-arm64.iso
It's 922 MB.
I got it from here: https://ubuntu.com/download/server/arm
You can chose any linux distribution, but just make sure you get the ARM version (some distributions don't have one).
2 - Download UTM
UTM is a virtualisation software that is mainly aimed at IOs devices, but it works on Mac OS too.
https://github.com/utmapp/UTM/releases/download/v2.0.14/UTM.dmg
That one is 255 MB.
Future versions will be available from here: https://github.com/utmapp/UTM/releases/
Simply download the package, open it, and launch the application that is inside.
3 - Create your VM
Create your new VM, attach the linux installation disk to it and launch the VM following these steps here:
https://github.com/utmapp/UTM/wiki/Install-Ubuntu-ARM64-on-Apple-M1
Basically:
click Create a New VM
in the Information tab: choose a name and an icon for your VM
in the System tab:
in Hardware choose ARM64 (aarch64) architecture
give it some memory (how about 4 GB ...)
in the Drives tab:
create your main drive with New Drive, interface VirtIO and choose the size you want (I chose 20 480 MB), then click Create.
create the CD drive with New Drive, check Removable, interface USB, click Create
Save the VM
Select your shiny new VM in the sidebar and in the bottom right corner, click Browse and select your Linux installation ISO virtual disk.
You can now launch the VM, it will boot on the Linux installation CD: install Linux.
During this classic installation process, you will be asked to create a user account on the linux system (let's call it bob). When the installation is finished, shut down the VM and extract the installation disk before rebooting.
4 - Working inside your VM
When you restart the VM, you get a terminal asking you to log into Linux, using the username and password you created during installation.
You can now install Docker, openSSH-server, nodeJS, using classic apt-get commands.
5 - Working in your VM from MacOS
If you failed like to me chose the right keyboard, you might have trouble typing some special characters. The best way to work with your VM is now to work from outside of it.
Stop the VM (sudo shutdown -h now if you are already inside the shell) and go back to UTM:
Select your VM in the left side panel and click the top right button to edit the VM again :
Go to the Network tab and in front of Port Forward, click New.
You need to manually add a new port forwarding directive for each port in your VM you want to access from your Mac OS Host.
For example for SSH: in the new port forward form, simply write 22 in Guest Port and what you want in Host Port (let's say 3022).
Now you can restart your VM and in a normal Mac OS Terminal, you can log into your VM with
ssh -p 3022 bob#localhost
If you don't want to type your password each time, copy the content of ~/.ssh/id_rsa.pub (from Mac OS) inside a newly created /home/bob/.ssh/authorized_keys text file, inside the VM.
6 - copying files via sftp
Sadly, I did not manage to access the content of the VM directly with the finder. I had to use the famous FTP client Transmit.
Create a new connection with:
Protocol: SFTP
Host: localhost
User: bob
Password: [your password]
Port: 3022
(yes, the port is the same as SSH)
You can now freely explore and copy files to and from your VM.
Oh but wait ... there is more!
7 - working with VS Code on your VM
Now you can also work on your VM, from your Mac OS VS Code, installing the Remote Development extention:
https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.vscode-remote-extensionpack
Once the extention is installed, click on the green >< sign at the bottom left of your VS Code window and choose Remote-SSH: connect to host ...
Choose add new SSH host and type ssh -p 3022 bob#localhost
Now you can work in VS Code on a project inside your VM just as if it was in your local file system.
I do some Nuxt.js development that calls an API powered with Apache / PHP / MySQL (had to switch to MariaDB because I could not find an ARM version of MySQL that was working) that is all running in different Docker containers inside the VM using docker-compose.
Having Port Forwarded the guest 3000 port to the host 3000 port, I can browser the front end with Safari just as if it was all running natively on Mac OS.
I hope this all saves you some time.
For docker there is a technical preview out https://docs.docker.com/docker-for-mac/apple-m1/.
You can run ruby, python etc. directly on a Mac M1 by setting up a terminal that runs under Rosetta mode. Then run home brew and you can use the existing x86_64 architecture brew taps. I'm using /bin/bash as my Mac shell rather than zsh but you could adapt the below for zsh if you prefer.
Log in to your normal shell and install ARM homebrew to /opt/homebrew
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
Now set up a new "Rosetta shell" terminal profile with the Shell tab Run command "env /usr/bin/arch -x86_64 /bin/bash --login".
Login again under rosetta shell and install x86_64 homebrew to /usr/local/homebrew
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
Edit your ~/.bash_profile and add some code to detect if your shell is under Rosetta and if so use the /usr/local/homebrew instead of the usual /opt/homebrew
if [ "$(sysctl -n sysctl.proc_translated)" = "1" ]; then
# run under rosetta 2 with
# env /usr/bin/arch -x86_64 /bin/bash --login
#local brew_path="/usr/local/homebrew/bin"
eval $(/usr/local/bin/brew shellenv)
export PS1="i \D{%I:%M %p}:\w $ "
else
#local brew_path="/opt/homebrew/bin"
eval $(/opt/homebrew/bin/brew shellenv)
fi
Now login to your Rosetta shell and do commands like
$ brew install ruby
then you can run ruby.
I have also managed to get a vagrant virtual machine Fedora 33 for ARM running on Mac M1 under Parallels virtualisation beta. That might help with apache and php. Instructions here
https://github.com/peterdragon/packer-M1-parallels-fedora33
Seems everything will work as is...
From the event presentation they said "Existing Mac apps that have not been updated to Universal will run seamlessly with Apple’s Rosetta 2 technology."
I'm new to Docker. I'm using Docker & docker-compose, going through a flask tutorial. The base docker image is python 2.7 slim.
It's running on Linux. docker 1.11.2
The application is working fine.
I want to get pycharm pro connecting to the remote interpreter, something I have never done before.
I followed the instructions for docker-compose. Initially it was failing because it could not connect to port 2376. I added this port to docker-compose.yml and the error went away.
However, trying to save the configuration now stalls/hangs with a dialog 'Getting Remote Interpreter Version'. This never completes. Also, I can't quit pycharm. This happens in Pycharm 2016.2 and 2016.3 EAP (2nd).
The help say "SFTP support is required for copying helpers to the server".
Does this mean I need to do something?
I'm not using docker-machine
The problem was that TCP access to the docker API is not established by default under ubuntu 16.04.
There are suggestions to enable TCP/IP access.
However, JetBrains gave me the simplest solution:
If you are using Linux it is most likely that Docker installed with
its default setup and Docker is expecting to be used through UNIX
domain file socket /var/run/docker.sock. And you should specify
unix:///var/run/docker.sock in the API URL field. Please comment
whether it helps!
This suggestion worked with my Ubuntu 16.04 -derived distribution.
This goes into the Docker entry in PyCharm preferences under Build, Execution, Deployment.
You can also edit this while setting up a remote interpreter, but only by making a new Docker entry.
TCP/IP Method
This method works if you want TCP/IP access, but this is a security risk. The socket approach is better, which is probably why it is the default.
https://coreos.com/os/docs/latest/customizing-docker.html
Customizing docker
The Docker systemd unit can be customized by overriding the unit that
ships with the default CoreOS settings. Common use-cases for doing
this are covered below.
Enable the remote API on a new socket
Create a file called /etc/systemd/system/docker-tcp.socket to make
Docker available on a TCP socket on port 2375.
[Unit]
Description=Docker Socket for the API
[Socket]
ListenStream=2375
BindIPv6Only=both
Service=docker.service
[Install]
WantedBy=sockets.target
Then enable this new socket:
systemctl enable docker-tcp.socket
systemctl stop docker
systemctl start docker-tcp.socket
systemctl start docker
Test that it’s working:
docker -H tcp://127.0.0.1:2375 ps
Once I thought to search for ubuntu 16.04 I came across simpler solutions, but I did not test them.
For instance:
https://www.ivankrizsan.se/2016/05/18/enabling-docker-remote-api-on-ubuntu-16-04/
Edit the file /lib/systemd/system/docker.service
Modify the line that starts with ExecStart to look like this:
ExecStart=/usr/bin/docker daemon -H fd:// -H tcp://0.0.0.0:2375
Where my addition is the “-H tcp://0.0.0.0:2375” part. Save the
modified file. Restart the Docker service:
sudo service docker restart
Test that the Docker API is indeed accessible:
curl http://localhost:2375/version
I - docker-compose up
I think PyCharm will run docker-compose up, have you try to run this command first in your terminal (from where your docker-compose.yml is) ?
Maybe if some errors occur, you will get more info in your terminal.
II - pycharm docker configuration
Otherwise it could be due to your docker machine configuration in PyCharm.
What I do to configure my machine and to be sure this one is correctly configured:
1 - run docker-machine ls in your shell
2 - copy paste the url without tcp://
3 - go to pycharm preferences -> Build, Execution, Deployement -> Docker -> + to create a new server, fill the server name field
4 - paste previously copied url keeping https://
5 - fill the path of your machine certificates folder
6 - tick Import credentials from Docker Machine
7 - click Detect -> your machine should appear in the selection list
8 - save this server
9 - select this server when configuring your remote interpreter, from PyCharm Preferences -> Project -> Project Interpreter -> wheel -> add remote -> Docker or Docker Compose
10 - you should be able to select a service name
11 - save your new interpreter
11 - try run your test twice, sometimes it could take time to initialize
Starting today for no discernible reason, Pycharm's remote console function will not connect with my remote server.
All other functions are working as normal, SSH session, deployment config, skeletons update, file sync etc.
I am running Pycharm 4.5.3 on Windows7x64 against a remote server running Centos6.5x64 on AWS, note that this setup has been working fine for months until today.
The following output appears in the console window when remote console is launched, it takes a minute or so to timeout:
sftp://user#FQDN:22/home/user/Envs/lab1/bin/python2.7 -u /home/user/.pycharm_helpers/pydev/pydevconsole.py 0 0
Couldn't connect to console process.
Process finished with exit code -1
Unhelpful Log output(C:\Users\user\.PyCharm40\system\log\idea.txt):
2015-07-09 17:15:07,910 [ 236325] INFO - esdk.transport.JschExecProcess - Executing ssh command: env "PYTHONIOENCODING"="UTF-8" "JETBRAINS_REMOTE_RUN"="1" "IPYTHONENABLE"="True" "PYTHONUNBUFFERED"="1" /home/user/Envs/lab1/bin/python2.7 -u /home/user/.pycharm_helpers/pydev/pydevconsole.py 0 0 for user#FQDN:22
The following troubleshooting steps have yielded nothing: Workstation / Server, reboot, Fresh Virtualenv, Different version of Python, reinstall of iPython, uninstall of iPython, reset of console / deployment configuration, connecting from a different workstation running same version of Pycharm, upgrade Pycharm from 4.5.2 to 4.5.3.
Suggestions for further troubleshooting steps gladly welcome while I wait for Jetbrains support to get around to my ticket!
The problem was within the security configuration on AWS.
My AWS instance has a friendly FQDN to go with the unfriendly AWS internal name, which appears to cause some havoc with the way their NAT works if you don't have a rule allowing traffic from the IP bound to the FQDN back into the server.
I created the issue yesterday when my office router reset to a new IP - I must've overwritten the recursive rule with the rule allowing my new office IP into the instance.
So, add a rule allowing the server public IP for all traffic and Pycharm connects again when pointed at the FQDN.
I have a simple django python server process which needs to be executed in linux environment (in a virtualenv python environment)
Currently one of my colleague manually logs into ssh console and starts the virtual environment via source bin/activate command. Thereafter python server is started using below command
/etc/init.d/start-python-server.sh
Note: This sh file starts the python server as a background process listening in port 8080
Can some one give some thoughts on improving this?
Please share in your thoughts.
you may include source <your_env_path>/bin/activate at the beginning of /etc/init.d/start-python-server.sh to automate this process