We Keep Coding

Running a python script inside an nginx docker container

I'm using nginx to serve some of my docs. I have a python script that processes these docs for me. I don't want to pre-process the docs and then add them in before the docker container is built since these docs can grow to be pretty big and they increase in number. What I want is to run my python (and bash) scripts inside the nginx container and have nginx just serve those docs. Is there a way to do this without pre-processing the docs before building the container?
I've attempted to execute RUN python3 process_docs.py, but I keep seeing the following error:
/bin/sh: 1: python: not found
The command '/bin/sh -c python process_docs.py' returned a non-zero code: 127
Is there a way to get python3 onto the Nginx docker container? I was thinking of installing python3 using:
apt-get update -y
apt-get install python3.6 -y
but I'm not sure that this would be good practice. Please let me know the best way to run my pre processing script.

You can use a bind mount to inject data from your host system into the container. This will automatically update itself when the host data changes. If you're running this in Docker Compose, the syntax looks like
version: '3.8'
services:
nginx:
image: nginx
volumes:
- ./html:/usr/share/nginx/html
- ./data:/usr/share/nginx/html/data
ports:
- '8000:80' # access via http://localhost:8000
In this sample setup, the html directory holds your static assets (checked into source control) and the data directory holds the generated data. You can regenerate the data from the host, outside Docker, the same way you would if Docker weren't involved
# on the host
. venv/bin/activate
./regenerate_data.py --output-directory ./data
You should not need docker exec in normal operation, though it can be extremely useful as a debugging tool. Conceptually it might help to think of a container as identical to a process; if you ask "can I run this Python script inside the Nginx process", no, you should generally run it somewhere else.

How do I build docker container behind company proxy?

I am trying to build a simple python based docker container. I am working at a corporate behind a proxy, on Windows 10. Below is my docker file:
FROM python:3.7.9-alpine3.11
WORKDIR ./
RUN pip install --proxy=http://XXXXXXX:8080 -r requirements.txt
COPY . /
EXPOSE 5000
CMD ["python", "application.py"]
But it's giving me the following errors in cmd :
"failed to solve with frontend dockerfile.v0: failed to build LLB: failed to load cache key: failed to do request: Head https://registry-1.docker.io/v2/library/python/manifests/3.7.9-alpine3.11: proxyconnect tcp: EOF"
I've tried to figure out how to configure docker's proxy, using many links but they keep referring to a file "/etc/sysconfig/docker" which I cannot find anywhere under Windows 10 or maybe I'm not looking at the right place.
Also I'm not sure this is only a proxy issue since I've seen people running into this issue without using a proxy.
I would highly appreciate anyone's help. Working at this corporate already made me spend >10 hours doing something that took me 10 minutes to do on my Mac... :(
Thank you

You're talking about the most basic of Docker functionality. Normally, it has to connect to the Docker Hub on the internet to get base images. If you can't make this work with your proxy, you can either
preload your local cache with the necessary images
set up a Docker registry inside your firewall that contains all the images you'll need
Obviously, the easiest thing, probably by far, would be to figure out how to get Docker to connect to Docker Hub through your proxy.
In terms of getting Docker on Windows to work with your proxy, might this help? - https://learn.microsoft.com/en-us/virtualization/windowscontainers/manage-docker/configure-docker-daemon
Here's what it says about configuring a proxy:
To set proxy information for docker search and docker pull, create a Windows environment variable with the name HTTP_PROXY or HTTPS_PROXY, and a value of the proxy information. This can be completed with PowerShell using a command similar to this:
In PowerShell:
[Environment]::SetEnvironmentVariable("HTTP_PROXY", "http://username:password#proxy:port/", [EnvironmentVariableTarget]::Machine)
Once the variable has been set, restart the Docker service.
In PowerShell:
Restart-Service docker
For more information, see Windows Configuration File on Docker.com.
I've also seen it mentioned that Docker for Windows allows you to set proxy parameters in its configuration GUI interface.

There is no need to pass proxy information in the Dockerfile.
There are predefined ARGs which can be used for this purpose.
HTTP_PROXY
HTTPS_PROXY
FTP_PROXY
You can pass the details when building the image
https://docs.docker.com/engine/reference/builder/#predefined-args
I do not see any run time dependency of your container on the Internet. So running the container will work without an issue.

How to Deploy Flask app on AWS EC2 Linux/UNIX instance

How to deploy Flask app on AWS Linux/UNIX EC2 instance.
With any way either
1> using Gunicorn
2> using Apache server

It's absolutely possible, but it's not the quickest process! You'll probably want to use Docker to containerize your flask app before you deploy it as well, so it boils down to these steps:
Install Docker (if you don't have it) and build an image for your application and make sure you can start the container locally and the app works as intended. You'll also need to write a Dockerfile that sets your runtime, copies all your directories and exposes port 80 (this will be handy for AWS later).
The command to build an image is docker build -t your-app-name .
Once you're ready to deploy the container, head over to AWS and launch an EC2 instance with the Linux 2 machine. You'll be required to create a security key (.pem file) and move it to somewhere on your computer. This acts like your credential to login to your instance. This is where things get different depending on what OS you use. On Mac, you need to cd into your directory where the key is and modify the permissions of it by running chmod 400 key-file-name.pem. On Windows, you have to go into the security settings and make sure only your account (ideally the owner of the computer) can use this file, basically setting it to private. At this point, you can connect to your instance from your command prompt with the command AWS gives you when you click connect to instance on the EC2 dashboard.
Once you're logged in, you can configure your instance to install docker and let you use it by running the following:
sudo amazon-linux-extras install docker
sudo yum install docker
sudo service docker start
sudo usermod -a -G docker ec2-user
Great, now you need to copy all your files from your local directory to your instance using SCP (secure transfer protocol). The long way is to use this command for each file: scp -i /path/my-key-pair.pem file-to-copy ec2-user#public-dns-name:/home/ec2-user. Another route is to install FileZilla or WinSCP to speed up this process.
Now that all your files are in the instance, build the docker container using the same command from the first step and activate it. If you go to the URL that AWS gives you, your app should be running on AWS!
Here's a reference I used when I did this for the first time, it might be helpful for you to look at too

How to deploy scrapyd to a network

I currently have an instance of scrapyd up and running locally on my machine. This instance of scrapyd needs to be available to other PC's on my employers network. I've read about scrapy-cloud (https://doc.scrapinghub.com/scrapy-cloud.html) and other cloud based services. However I'd much rather host scrapyd on our network, since the spiders I've built pull data from csv files stored on our servers.
I've searched through the scrapyd documentation (https://scrapyd.readthedocs.io/en/stable/) and understand how to install and run scrapyd. I am also comfortable with uploading scrapy projects to scrapyd and running specific spiders.
What steps do I need to take in order to make my scrapyd instance available to other machines on our network? All of our PC's and servers run on a windows OS
The answer doesn't need to be a specific step by step guide. I'm just looking for someone to point me in the right direction, because I am unsure how to proceed.

if you are in a lan in the same range of ip.
you can follow the manual and check your ip
ifconfig in linux
ipconfig in windows
and run the commands in the manual
curl http://localhost:6800/addversion.json -F project=myproject -F version=r23 -F egg=#myproject.egg
and change the localhost with your ipaddress
for example if your ip is 192.168.1.10
you will run
in other pc }.
curl http://192.168.1.10:6800/addversion.json -F project=myproject -F version=r23 -F egg=#myproject.egg
You need open the port if you use firewalls, and if you don't use cURL in windows can download and install it:
How do I install/set up and use cURL on Windows?
More information about the api check the manual

How do I run pycharm within my docker container?

I'm very new to docker. I want to build my python application within a docker container. As I build the application I want to be testing / running it in Pycharm and in the container I build.
How do I connect Pycharm pro to a specific container or image (either python or Anaconda)?
When I create a project, click pure python and then add remote, then clicking docker I get the following result
I'm running on Mac OS X El Capitan (10.11.6) with Docker version 1.12.1 and Pycharm Pro 2016.2.3

Docker-for-mac only supports connections over the /var/run/docker.sock socket that is listening on your OSX host.
If you try to add this to pycharm, you'll get the following message:
"Cannot connect: java.lang.ExceptionInInitializerError, caused by: java.lang.IllegalStateException: Only supported on Linux"
So PyCharm really only wants to connect to a docker daemon over a TCP socket, and has support for the recommended TLS protection of that socket. The Certificates folder defaults to the certificate folder for the default docker-machine machine, "default".
It is possible to implement a workaround to expose Docker for Mac via a TCP server if you have socat installed on your OSX machine.
On my system, I have it installed via homebrew:
brew install socat
Now that's installed, I can run socat with the following parameters:
socat TCP-LISTEN:2376,reuseaddr,fork,bind=127.0.0.1 UNIX-CLIENT:/var/run/docker.sock
WARNING: this will make it possible for any process running as any user on your whole mac to access your docker-for-mac. The unix socket is protected by user permissions, while 127.0.0.1 is not.
This socat command tells it to listen on 127.0.0.1:2376 and pass connections on to /var/run/docker.sock. The reuseaddr and fork options allow this one command to service multiple connections instead of just the very first one.
I can test that socat is working by running the following command:
docker -H tcp://127.0.0.1:2376 ps
If you get a successful docker ps response back, then you know that the socat process is doing its job.
Now, in the PyCharm window, I can put the same tcp://127.0.0.1:2376 in place. I should get a "Connection successful" message back:
This workaround will require that socat command to be running any time you want to use docker from PyCharm.
If you wanted to do the same thing, but with TLS, you could set up certificates and make them available for both pycharm and socat, and use socat's OPENSSL-LISTEN instead of the TCP-LISTEN feature. I won't go into the details on that for this answer though.