So I am having a dilemma. I made a Flask app that uses a mysql DB for storing username and passwords when people log into the app. My question is, is there a dynamic way to get the TCP Port address within my Python code it self? What I am currently doing is just hardcoding the path like so:
app.config['MYSQL_DATABASE_USER'] = 'root'
app.config['MYSQL_DATABASE_PASSWORD'] = ''
app.config['MYSQL_DATABASE_DB'] = 'UserList'
app.config['MYSQL_DATABASE_HOST'] = '172.17.0.3'
But what I would like is to make the host dynamic so if I ever build another mysql container, I won't have to manually change the IP every time. I know you can do this command,
env | grep MYSQL
MYSQL_PORT_5123_TCP_ADDR=172.17.0.3
MYSQL_ENV_MYSQL_ROOT_PASSWORD=test
MYSQL_PORT_5123_TCP=tcp://172.17.0.3:5123
MYSQL_PORT_5123_TCP_PROTO=tcp
MYSQL_ENV_GOSU_VERSION=1.7
MYSQL_PORT_3306_TCP_PORT=3306
MYSQL_PORT_3306_TCP=tcp://172.17.0.3:3306
MYSQL_PORT_5123_TCP_PORT=5123
MYSQL_ENV_MYSQL_VERSION=5.7.18-1debian8
MYSQL_NAME=/site-metrics/mysql
MYSQL_PORT_3306_TCP_PROTO=tcp
MYSQL_PORT_3306_TCP_ADDR=172.17.0.3
MYSQL_ENV_MYSQL_MAJOR=5.7
MYSQL_PORT=tcp://172.17.0.3:3306
But is there a way to do this within my Python script so I do not have to fiddle with this every time? Thanks for the help!
You can achieve this by running both containers on the same network. I'll assume a local network for now but overlay works in the same way.
$ docker network create my-network
$ docker run --name db --net my-network <other args...>
$ docker run --name web --net my-network <other args...>
Then in your app config:
app.config['MYSQL_DATABASE_HOST'] = 'db'
Because both containers are on the same network, they will be able to resolve each other via DNS (using their container names).
Edit: In fact an even better way of doing it would be with an env variable that you pass in. Then you have the best of both worlds. Something like:
app.config['MYSQL_DATABASE_HOST'] = os.getenv('DB_HOST', 'db')
The second argument to os.getenv() is a default value. So by default it will use 'db' but if for whatever reason in your environment you need to change it or have a different Docker service name, you can just run the container with:
$ docker run --name other_db --net my-network <other args...>
$ docker run --name web --net my-network -e DB_HOST=other_db <other args...>
Then it would try to connect to other_db:<port> instead.
Related
I am currently running my django app inside the docker container by using the below command
docker-compose run app sh -c "python manage.py runserver"
but I am not able to access the app with local host url, (not using any additional db server or ngnix or gunicorn, just simply running the django devlopment server inside the docker).
please let me know how to access the app
docker-compose run is intended to launch a utility container based on a service in your docker-compose.yml as a template. It intentionally does not publish the ports: declared in the Compose file, and you shouldn't need it to run the main service.
docker-compose up should be your go-to call for starting the services. Just docker-compose up on its own will start everything in the docker-compose.yml, concurrently, in the foreground; you can add -d to start the processes in the background, or a specific service name docker-compose up app to only start the app service and its dependencies.
The python command itself should be the main CMD in your image's Dockerfile. You shouldn't need to override it in your docker-compose.yml file or to provide it at the command line.
A typical Compose YAML file might look like:
version: '3.8'
services:
app:
build: . # from the Dockerfile in the current directory
ports:
- 5000:5000 # make localhost:5000 forward to port 5000 in the container
While Compose supports many settings, you do not need to provide most of them. Compose provides reasonable defaults for container_name:, hostname:, image:, and networks:; expose:, entrypoint:, and command: will generally come from your Dockerfile and don't need to be overridden.
Try 0.0.0.0:<PORT_NUMBER> (typically 80 or 8000), If you are still troubling to connect the server you should use the Docker Machine IP instead of localhost. Enter the following in terminal and navigate to the provided url:
docker-machine ip
I am using mysql-connector, when ever I run the container using docker I get this error:
mysql.connector.errors.InterfaceError: 2003: Can't connect to MySQL server on 'db:3306' (-5 No address associated with hostname)
but when I run the project using python only it executes with no errors
I want to use phpmyadmin only for the database please help.
To create a docker from your linux machine:
docker pull mysql:latest
To run it and mount a persistent folder with port access on 3306 (std port for mysql):
docker run --name=mysql_dockerdb --env="MYSQL_ROOT_PASSWORD=<your_password>" -p 3306:3306 -v /home/ubuntu/sql_db/<your_dbasename>:/var/lib/mysql -d mysql:latest
To connect to the docker instance so that you can create the database within the docker:
docker exec -it mysql_dockerdb mysql -uroot -p<your_password>
My SQL code to establish the database:
CREATE DATABASE dockerdb;
CREATE USER 'newuser'#'%' IDENTIFIED BY 'newpassword';
GRANT ALL PRIVILEGES ON dockerdb.* to 'newuser'#'%';
ALTER USER 'username'#'%' IDENTIFIED WITH mysql_native_password BY 'userpassword';
You will now have a docker running with a persistent SQL database. You connect to it from your Python code. I am running Flask mySql. You will want to keep your passwords in environment variables. I am using a Mac so therefore my ~/.bash_profile contains:
export RDS_LOGIN="mysql+pymysql://<username>:<userpassword>#<dockerhost_ip>/dockerdb"
Within Python:
import os
SQLALCHEMY_DATABASE_URI = os.environ.get('RDS_LOGIN')
And at that point you should be able to connect in your usual Python manner. Note that I've glossed over any security aspects on the presumption this is local behind a firewall.
I am trying to run a spider with portia in its docker version but i don't want to execute the spider using a terminal command like docker exec ... portiacrawl .... Is there any way I can run the spider, that is already created, by making a request at its localhost port and save it in an specific folder?
Something like:
https://localhost:9001/execute/spider_name/folder_path
Example of my own usage:
First what I do is run the container and leave it running, because i cant stop it for other reasons:
docker run -i -t -d --rm -v <PROJECTS_FOLDER>:/app/data/projects:rw -p 9001:9001 scrapinghub/portia
Next I execute the portiacrawl:
docker exec <CONTAINER_ID> portiacrawl <PROJECT_NAME_PATH> <SPIDER_NAME> -o /some/path/in/my/pc/<SPIDER_NAME>.json
Now, what i want is to replace the docker exec step with som http request to the localhost server that is running.
Thanks very much for your time
Yes, you can by doing a port mapping. While starting a docker container you wont have any ports published publicly or exposed internally unless you told docker to do so.
For example:
if you wish to expose a port internally (inside the docker network itself, you need to add EXPOSE in the dockerfile)
if you wish to publish a port publicly that can be access either through localhost or the public ip you can use -p option along with passing the ports so in your case it will be like this:
docker run -p 9001:9001 imagename
The command above will tell docker that you would like to do port mapping from 9001 (using localhost or any other interface) to 9001 (inside the container and you can change the ports according to your actual setup).
If you wish to expose it to localhost only you can change the command to something like this:
docker run -p 127.0.0.1:9001:9001 imagename
For more information check the following docs
According to the updated question, the other and safest way to accomplish this will be implementing an API inside portiacrawl that can be called through HTTP to do the needed tasks instead of using docker exec
I'm new to docker, redis and any kind of networking, (I know python at least!). Firstly I have figured out how to get a redis docker image and run it in a docker container:
docker run --name some-redis -d redis
As I understand this redis instance has port 6379 available to connect to other containers.
docker network inspect bridge
"Containers": {
"2ecceba2756abf20d5396078fd9b2ecf0d60ab04ca6b8df5e1b631b6fb5e9a85": {
"Name": "some-redis",
"EndpointID": "09f0069dae3632a2456cb4d82ad5e7c9782a2b58cb7a4ee655f57b5c410c3e87",
"MacAddress": "02:42:ac:11:00:02",
"IPv4Address": "172.17.0.2/16",
"IPv6Address": ""
}
If I run the following command I can interact with the redis instance and generate key:value pairs:
docker run -it --link some-redis:redis --rm redis redis-cli -h redis -p 6379
set 'a' 'abc'
>OK
get 'a'
>"abc"
quit
I have figured out how to make and run a docker container with the redis library installed that will run a python script as follows:
Here is my Dockerfile:
FROM python:3
ADD redis_test_script.py /
RUN pip install redis
CMD [ "python", "./redis_test_script.py" ]
Here is redis_test_script.py:
import redis
print("hello redis-py")
Build the docker image:
docker build -t python-redis-py .
If I run the following command the script runs in its container:
docker run -it --rm --name pyRed python-redis-py
and returns the expected:
>hello redis-py
It seems like both containers are working ok, the problem is connecting them both together, I would like to ultimately use python to perform operation on the redis container. If I modify the script as follows and rebuild the image for the python container it fails:
import redis
print("hello redis-py")
r = redis.Redis(host="localhost", port=6379, db=0)
r.set('z', 'xyz')
r.get('z')
I get several errors:
...
OSError: [Errno 99] Cannot assign requested address
...
redis.exceptions.ConnectionError: Error 99 connecting to localhost:6379. Cannot assign requested address.
.....
It looks like they're not connecting, I tried again using the bridge IP in the python script:
r = redis.Redis(host="172.17.0.0/16", port=6379, db=0)
and get this error:
redis.exceptions.ConnectionError: Error -2 connecting to 172.17.0.0/16:6379. Name or service not known.
and I tried the redis sub IP:
r = redis.Redis(host="172.17.0.2/16", port=6379, db=0)
and I get this error:
redis.exceptions.ConnectionError: Error -2 connecting to 172.17.0.2/16:6379. Name or service not known.
It feels like I'm fundamentally misunderstanding something about how to get the containers to talk to each other. I've read quite a lot of documentation and tutorials but as I say have no networking experience and have not previously used docker so any helpful explanations and/or solutions would be really great.
Many thanks
That's all about Docker networking. Fast solution - use host network mode for both containers. Drawback is low isolation, but you will get it working fast:
docker run -d --network=host redis ...
docker run --network=host python-redis-py ...
Then to connect from python to redis just use localhost as a hostname.
Better solution is to use docker user-defined bridge network
# create network
docker network create foo
docker run -d --network=foo --name my-db redis ...
docker run --network=foo python-redis-py ...
Note that in this case you cannot use localhost but instead use my-db as a hostname. That's why I've used --name my-db parameter when starting first container. In user-defined bridge networks containers reach each other by theirs names.
Do:
Explicitly create a Docker network for your application, and run your containers connected to that network. (If you use Docker Compose, this happens for you automatically and you don’t need to do anything.)
docker network create foo
docker run -d --net foo --name some-redis redis
docker run -it --rm --net foo --name pyRed python-redis-py
Use containers’ --name as DNS hostnames: you connect to some-redis:6379 to reach the container. (In Docker Compose the name of the service block works too.)
Make the locations of external services configurable, most likely using an environment variable. In your Python code you can connect to
redis.Redis(host=os.environ.get("REDIS_HOST", "localhost"),
port=int(os.environ.get("REDIS_PORT", "6379"))
docker run --rm -it \
--name py-red \
--net foo \
-e REDIS_HOST=some-redis \
python-redis-py
Don’t:
docker inspect anything to find the container-private IP addresses. Between containers you can always use hostnames as described above. The container-private IP addresses are unreachable from other hosts, and may even be unreachable from the same hosts on some platforms.
Use localhost in Docker for anything, expect the specific case of connecting from a browser or other process running directly on the host (not in a container) to a port you’ve published with docker run -p on the same host. (It generally means “this container”.)
Hard-code host names in your code like this; it makes it hard to run the service in a different environment. (For databases in particular it’s not uncommon to run them outside of Docker or even in a hosted cloud service.)
Use --link, it’s outdated and unnecessary.
I have used the following commands to create a Docker image with Postgres running on it:
docker pull postgres
docker run --name test-db -e POSTGRES_PASSWORD=my_secret_password -d postgres
I then created a table called test and inserted some random data into a couple of rows.
I am now trying to make a connection to this database table through psycopg2 in Python on my local machine.
I used the command docker-machine ip default to find out the IP address of the machine as 192.168.99.100 and am using the following to try and connect:
conn = psycopg2.connect("dbname='test-db' user='postgres' host='192.168.99.100' password='my_secret_password' port='5432'")
This is not working with the error message of "OperationalError: could not connect to server: Connection refused (0x0000274D/10061)"
.
.
Everything seems to be in order so I can't think why this would be refused.
According to the documentation for this postgres image, (at https://hub.docker.com/_/postgres/) this image includes EXPOSE 5432 (the postgres port) and the default username is postgres.
I also tried to get the IP address of the image itself with docker inspect test-db | grep IPAddress | awk 'print{$2}' | tr -d '",' that I found on SA to a slightly related article, but that IP address didn't work either.
The EXPOSE instruction may not be doing what you expect. It is used for links and inter-container communication inside the Docker network. When connecting to a container from outside the Docker bridge network you need to publish to port with -p. Try adding -p 5432:5432 to your docker run command so that it looks like:
docker run --name test-db -e POSTGRES_PASSWORD=my_secret_password -d -p 5432:5432 postgres
Here is a decent explanation of the differences between publish and exposed ports: https://stackoverflow.com/a/22150099/684908. Hope this helps!