We Keep Coding

Python requests in Docker Compose containers

Problem
I have a 2-container docker-compose.yml file.
One of the containers is a small FastAPI app.
The other is just trying to hit the API using Python's requests package.
I can access the app container from outside with the exact same code as is in the Python package trying to hit it, and it works, but it will not work within the package.
docker-compose.yml
version: "3.8"
services:
read-api:
build:
context: ./read-api
depends_on:
- "toy-api"
networks:
- ds-net
toy-api:
build:
context: ./api
networks:
- ds-net
ports:
- "80:80"
networks:
ds-net:
Relevant requests code
from requests import Session
def post_to_api(session, raw_input, path):
print(f"The script is sending: {raw_input}")
print(f"The script is sending it to: {path}")
response = session.post(path, json={"payload": raw_input})
print(f"The script received: {response.text}")
def get_from_api(session, path):
print(f"The datalake script is trying to GET from: {path}")
response = session.get(path)
print(f"The datalake script received: {response.text}")
session = Session()
session.trust_env = False ### I got that from here: https://stackoverflow.com/a/50326101/534238
get_from_api(session, path="http://localhost/test")
post_to_api(session, "this is a test", path="http://localhost/raw")
Running It REPL-Style
If I create an interactive session and run those exact commands above in the requests code portion, it works:
>>> get_from_api(session, path="http://localhost/test")
The script is trying to GET from: http://localhost/test
The script received: {"payload":"Yes, you reached here..."}
>>> post_to_api(session, "this is a test", path="http://localhost/raw")
The script is sending: this is a test
The script is sending it to: http://localhost/raw
The script received: {"payload":"received `raw_input`: this is a test"}
To be clear: the API code is still being run as a container, and that container was still created with the docker-compose.yml file. (In other words, the API container is working properly, when accessed from the host.)
Running Within Container
Doing the same thing within the container, I get the following (fairly long) errors:
read-api_1 | The script is trying to GET from: http://localhost/test
read-api_1 | Traceback (most recent call last):
read-api_1 | File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 159, in _new_conn
read-api_1 | conn = connection.create_connection(
read-api_1 | File "/usr/local/lib/python3.8/site-packages/urllib3/util/connection.py", line 84, in create_connection
read-api_1 | raise err
read-api_1 | File "/usr/local/lib/python3.8/site-packages/urllib3/util/connection.py", line 74, in create_connection
read-api_1 | sock.connect(sa)
read-api_1 | ConnectionRefusedError: [Errno 111] Connection refused
read-api_1 |
read-api_1 | During handling of the above exception, another exception occurred:
.
.
.
read-api_1 | Traceback (most recent call last):
read-api_1 | File "access_api.py", line 99, in <module>
read-api_1 | get_from_api(session, path="http://localhost/test")
read-api_1 | File "access_datalake.py", line 86, in get_from_api
read-api_1 | response = session.get(path)
read-api_1 | File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 543, in get
read-api_1 | return self.request('GET', url, **kwargs)
read-api_1 | File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 530, in request
read-api_1 | resp = self.send(prep, **send_kwargs)
read-api_1 | File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 643, in send
read-api_1 | r = adapter.send(request, **kwargs)
read-api_1 | File "/usr/local/lib/python3.8/site-packages/requests/adapters.py", line 516, in send
read-api_1 | raise ConnectionError(e, request=request)
read-api_1 | requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=80): Max retries exceeded with url: /test (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ffa9c69b3a0>: Failed to establish a new connection: [Errno 111] Connection refused'))
ai_poc_work_read-api_1 exited with code 1
Attempts to Solve
I thought it was with how the host identified itself within the container group, or whether that origin could be accessed, so I have already tried to change the following, with no success:
Instead of using localhost as the host, I used read-api.
Actually, I started with read-api, and had no luck, but once using localhost, I could at least use REPL on the host machine, as shown above.
I also tried 0.0.0.0, no luck. (I did not expect that to fix it.)
I have changed what CORS ORIGINS are allowed in the API, including all of the possible paths for the container that is trying to read, and just using "*" to flag all CORS origins. No luck.
What am I doing wrong? It seems the problem must be with the containers, or maybe how requests interacts with containers, but I cannot figure out what.
Here are some relevant GitHub issues or SO answers I found, but none solved it:
GitHub issue: Docker Compose problems with requests
GitHub issue: Solving high latency requests in Docker containers
SO problem: containers communicating with requests

Within the Docker network, applications must be accessed with the service names defined in the docker-compose.yml.
If you're trying to access the toy-api service, use
get_from_api(session, path="http://toy-api/test")
You can access the application via http://localhost/test on your host machine because Docker exposes the application to the host machine. However, loosely speaking, within the Docker network, localhost does not refer to the host's localhost but only to the container's localhost. And in the case of the read-api service, there is no application listening to http://localhost/test.

RuntimeError: Working outside of request context. with gunicorn

Whenever I run my code as python3 myapp.py it works fine but whenever I use gunicorn -w 4 myapp:index -b 10.91.1.230:5055 &
it throws
ion#aurora:~/TNQ$ [2019-02-05 14:26:34 +0530] [27107] [INFO] Starting
gunicorn 19.9.0
..............
27116
[2019-02-05 14:26:38 +0530] [27113] [ERROR] Error handling request /
Traceback (most recent call last):
File "/home/ion/.local/lib/python3.6/site-packages/gunicorn/workers/sync.py", line 135, in handle
self.handle_request(listener, req, client, addr)
File "/home/ion/.local/lib/python3.6/site-packages/gunicorn/workers/sync.py", line 176, in handle_request
respiter = self.wsgi(environ, resp.start_response)
File "/home/ion/TNQ/myapp.py", line 16, in index
f = request.files['file']
File "/home/ion/.local/lib/python3.6/site-packages/werkzeug/local.py", line 347, in __getattr__
return getattr(self._get_current_object(), name)
File "/home/ion/.local/lib/python3.6/site-packages/werkzeug/local.py", line 306, in _get_current_object
return self.__local()
File "/home/ion/.local/lib/python3.6/site-packages/flask/globals.py", line 37, in _lookup_req_object
raise RuntimeError(_request_ctx_err_msg)
RuntimeError: Working outside of request context.
This typically means that you attempted to use functionality that needed
an active HTTP request. Consult the documentation on testing for
information about how to avoid this problem.
myapp.py
from flask import Flask,request
from CXE.src.models import class_classic as cc
app = Flask(__name__)
#app.route('/', methods=['POST'])
#def index(environ, start_response):
def index(environ, start_response):
with app.app_context():
#if request.method == 'POST':
f = request.files['file']
a = cc.initiate(f)
return a
if __name__ == '__main__':
app.run(host = '0.0.0.0',port=5505,debug=True)
I need to put the code on gunicorn for serving it on threads. Any idea why isn't it working ?

It doesn't work because index is not a wsgi app - it's not enough for the function to have the correct signature. Do the following instead:
gunicorn -w 4 myapp:app -b 10.91.1.230:5055 &
You see no issue when you run python3 myapp.py because flask's dev server, app.run, does not require a wsgi app, whereas gunicorn does. With that said, you might as well go ahead and remove the index signature.

Google App Engine configuring Endpoints API

I have a problem configuring Endpoints API. Any code i use, from my own, to google's examples on site fail with the same traceback
WARNING 2016-11-01 06:16:48,279 client.py:229] no scheduler thread, scheduler.run() will be invoked by report(...)
Traceback (most recent call last):
File "/home/vladimir/projects/sb_fork/sb/lib/vendor/google/api/control/client.py", line 225, in start
self._thread.start()
File "/home/vladimir/sdk/google-cloud-sdk/platform/google_appengine/google/appengine/api/background_thread/background_thread.py", line 108, in start
start_new_background_thread(self.__bootstrap, ())
File "/home/vladimir/sdk/google-cloud-sdk/platform/google_appengine/google/appengine/api/background_thread/background_thread.py", line 87, in start_new_background_thread
raise ERROR_MAP[error.application_error](error.error_detail)
FrontendsNotSupported
INFO 2016-11-01 06:16:48,280 client.py:327] created a scheduler to control flushing
INFO 2016-11-01 06:16:48,280 client.py:330] scheduling initial check and flush
INFO 2016-11-01 06:16:48,288 client.py:804] Refreshing access_token
/home/vladimir/projects/sb_fork/sb/lib/vendor/urllib3/contrib/appengine.py:113: AppEnginePlatformWarning: urllib3 is using URLFetch on Google App Engine sandbox instead of sockets. To use sockets directly instead of URLFetch see https://urllib3.readthedocs.io/en/latest/contrib.html.
AppEnginePlatformWarning)
ERROR 2016-11-01 06:16:49,895 service_config.py:125] Fetching service config failed (status code 403)
ERROR 2016-11-01 06:16:49,896 wsgi.py:263]
Traceback (most recent call last):
File "/home/vladimir/sdk/google-cloud-sdk/platform/google_appengine/google/appengine/runtime/wsgi.py", line 240, in Handle
handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
File "/home/vladimir/sdk/google-cloud-sdk/platform/google_appengine/google/appengine/runtime/wsgi.py", line 299, in _LoadHandler
handler, path, err = LoadObject(self._handler)
File "/home/vladimir/sdk/google-cloud-sdk/platform/google_appengine/google/appengine/runtime/wsgi.py", line 85, in LoadObject
obj = __import__(path[0])
File "/home/vladimir/projects/sb_fork/sb/main.py", line 27, in <module>
api_app = endpoints.api_server([SolarisAPI,], restricted=False)
File "/home/vladimir/projects/sb_fork/sb/lib/vendor/endpoints/apiserving.py", line 497, in api_server
controller)
File "/home/vladimir/projects/sb_fork/sb/lib/vendor/google/api/control/wsgi.py", line 77, in add_all
a_service = loader.load()
File "/home/vladimir/projects/sb_fork/sb/lib/vendor/google/api/control/service.py", line 110, in load
return self._load_func(**kw)
File "/home/vladimir/projects/sb_fork/sb/lib/vendor/google/api/config/service_config.py", line 78, in fetch_service_config
_log_and_raise(Exception, message_template.format(status_code))
File "/home/vladimir/projects/sb_fork/sb/lib/vendor/google/api/config/service_config.py", line 126, in _log_and_raise
raise exception_class(message)
Exception: Fetching service config failed (status code 403)
INFO 2016-11-01 06:16:49,913 module.py:788] default: "GET / HTTP/1.1" 500 -
My app.yaml is configured like the new Endpoints Migrating to 2.0 document states:
- url: /_ah/api/.*
script: api.solaris.api_app
And main.py imports the API into the app:
api_app = endpoints.api_server([SolarisAPI,], restricted=False)
I use Google Cloud SDK with these versions:
Google Cloud SDK 132.0.0
app-engine-python 1.9.40
bq 2.0.24
bq-nix 2.0.24
core 2016.10.24
core-nix 2016.10.24
gcloud
gsutil 4.22
gsutil-nix 4.22

Have you tried generating and uploading the OpenAPI configuration for the service? See the sections named "Generating the OpenAPI configuration file" and "Deploying the OpenAPI configuration file" in the python library documentation.
Note that in step 2 of the generation process, you may need to prepend python to the command (e.g python lib/endpoints/endpointscfg.py get_swagger_spec ...), since the PyPi package doesn't preserve executable file permissions right now.

To get rid of the "FrontendsNotSupported" you need to use a "B*" instance class.
The error "Exception: Fetching service config failed" should be gone if you follow the steps in https://cloud.google.com/endpoints/docs/frameworks/python/quickstart-frameworks-python. As already pointed out by Brad, the section "OpenAPI configuration" and the resulting environment variables are required to make the service configuration work.

Unable to start Airflow worker/flower and need clarification on Airflow architecture to confirm that the installation is correct

Running a worker on a different machine results in errors specified below. I have followed the configuration instructions and have sync the dags folder.
I would also like to confirm that RabbitMQ and PostgreSQL only needs to be installed on the Airflow core machine and does not need to be installed on the workers (the workers only connect to the core).
The specification of the setup is detailed below:
Airflow core/server computer
Has the following installed:
Python 2.7 with
airflow (AIRFLOW_HOME = ~/airflow)
celery
psycogp2
RabbitMQ
PostgreSQL
Configurations made in airflow.cfg:
sql_alchemy_conn = postgresql+psycopg2://username:password#192.168.1.2:5432/airflow
executor = CeleryExecutor
broker_url = amqp://username:password#192.168.1.2:5672//
celery_result_backend = postgresql+psycopg2://username:password#192.168.1.2:5432/airflow
Tests performed:
RabbitMQ is running
Can connect to PostgreSQL and have confirmed that Airflow has created tables
Can start and view the webserver (including custom dags)
.
.
Airflow worker computer
Has the following installed:
Python 2.7 with
airflow (AIRFLOW_HOME = ~/airflow)
celery
psycogp2
Configurations made in airflow.cfg are exactly the same as in the server:
sql_alchemy_conn = postgresql+psycopg2://username:password#192.168.1.2:5432/airflow
executor = CeleryExecutor
broker_url = amqp://username:password#192.168.1.2:5672//
celery_result_backend = postgresql+psycopg2://username:password#192.168.1.2:5432/airflow
Output from commands run on the worker machine:
When running airflow flower:
ubuntu#airflow_client:~/airflow$ airflow flower
[2016-06-13 04:19:42,814] {__init__.py:36} INFO - Using executor CeleryExecutor
Traceback (most recent call last):
File "/home/ubuntu/anaconda2/bin/airflow", line 15, in <module>
args.func(args)
File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/airflow/bin/cli.py", line 576, in flower
os.execvp("flower", ['flower', '-b', broka, port, api])
File "/home/ubuntu/anaconda2/lib/python2.7/os.py", line 346, in execvp
_execvpe(file, args)
File "/home/ubuntu/anaconda2/lib/python2.7/os.py", line 382, in _execvpe
func(fullname, *argrest)
OSError: [Errno 2] No such file or directory
When running airflow worker:
ubuntu#airflow_client:~$ airflow worker
[2016-06-13 04:08:43,573] {__init__.py:36} INFO - Using executor CeleryExecutor
[2016-06-13 04:08:43,935: ERROR/MainProcess] Unrecoverable error: ImportError('No module named postgresql',)
Traceback (most recent call last):
File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/celery/worker/__init__.py", line 206, in start
self.blueprint.start(self)
File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/celery/bootsteps.py", line 119, in start
self.on_start()
File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/celery/apps/worker.py", line 169, in on_start
string(self.colored.cyan(' \n', self.startup_info())),
File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/celery/apps/worker.py", line 230, in startup_info
results=self.app.backend.as_uri(),
File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/kombu/utils/__init__.py", line 325, in __get__
value = obj.__dict__[self.__name__] = self.__get(obj)
File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/celery/app/base.py", line 626, in backend
return self._get_backend()
File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/celery/app/base.py", line 444, in _get_backend
self.loader)
File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/celery/backends/__init__.py", line 68, in get_backend_by_url
return get_backend_cls(backend, loader), url
File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/celery/backends/__init__.py", line 49, in get_backend_cls
cls = symbol_by_name(backend, aliases)
File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/kombu/utils/__init__.py", line 96, in symbol_by_name
module = imp(module_name, package=package, **kwargs)
File "/home/ubuntu/anaconda2/lib/python2.7/importlib/__init__.py", line 37, in import_module
__import__(name)
ImportError: No module named postgresql
When celery_result_backend is changed to the default db+mysql://airflow:airflow#localhost:3306/airflow and the airflow worker is run again the result is:
ubuntu#airflow_client:~/airflow$ airflow worker
[2016-06-13 04:17:32,387] {__init__.py:36} INFO - Using executor CeleryExecutor
-------------- celery#airflow_client2 v3.1.23 (Cipater)
---- **** -----
--- * *** * -- Linux-3.19.0-59-generic-x86_64-with-debian-jessie-sid
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> app: airflow.executors.celery_executor:0x7f5cb65cb510
- ** ---------- .> transport: amqp://username:**#192.168.1.2:5672//
- ** ---------- .> results: mysql://airflow:**#localhost:3306/airflow
- *** --- * --- .> concurrency: 16 (prefork)
-- ******* ----
--- ***** ----- [queues]
-------------- .> default exchange=default(direct) key=celery
[2016-06-13 04:17:33,385] {__init__.py:36} INFO - Using executor CeleryExecutor
Starting flask
[2016-06-13 04:17:33,737] {_internal.py:87} INFO - * Running on http://0.0.0.0:8793/ (Press CTRL+C to quit)
[2016-06-13 04:17:34,536: WARNING/MainProcess] celery#airflow_client2 ready.
What am I missing? How can I diagnose this further?

The ImportError: No module named postgresql error is due to the invalid prefix used in your celery_result_backend. When using a database as a Celery backend, the connection URL must be prefixed with db+. See
https://docs.celeryproject.org/en/stable/userguide/configuration.html#conf-database-result-backend
So replace:
celery_result_backend = postgresql+psycopg2://username:password#192.168.1.2:5432/airflow
with something like:
celery_result_backend = db+postgresql://username:password#192.168.1.2:5432/airflow

You need to ensure to install Celery Flower. That is, pip install flower.

Why does my Python subprocess error when managed by supervisord?

I wrote a small API for screen captures - a flask app that passes capture requests to CasperJS using subprocess.Popen
In development on a Mac, and when I run the server in the shell on my production server which is Ubuntu 13.04 - everything works great.
When I manage the server with supervisord, however, the subprocess call returns an error that CasperJS cannot find PhantomJS (Casper runs on Phantom).
The error thrown is:
Fatal: [Errno 2] No such file or directory; did you install phantomjs?
The code is all open source.
Here is the subprocess call:
https://github.com/pwalsh/moment/blob/master/moment/models.py#L215
Here is the supervisor conf file for the server (I generate the actual file with Fabric, but it should be clear):
https://github.com/pwalsh/moment/blob/master/fabfile/templates.py#L56
There are only two users on the system - root, and my app's user. When I log on the machine as either of these users, I can run a dev server successfully, and I can run PhantomJS and CasperJS successfully.
Why does my subprocess error with supervisord?
Edit: Adding code + stacktrace
Supervisord conf for the gunicorn server:
; Generated via Fabric on 2013-08-18 23:05:50.928087
; gunicorn configuration for Moment
[program:moment-gunicorn]
command=/srv/environments/moment/bin/gunicorn moment:app --bind 127.0.0.1:9000 --workers 4 --timeout 30 --access-logfile /srv/logs/moment_gunicorn_access.log --error-logfile /srv/logs/moment_gunicorn_error.log
environment=PATH="/srv/environments/moment/bin"
directory=/srv/projects/moment
user=moment
autostart=true
autorestart=true
The code that sends data to the CasperJS/PhantomJS subprocess. It is a method of a class, the full code is here:
def capture(self):
filename = '{key}.{format}'.format(key=self.get_key().lstrip(self.prefix),
format=self.arguments['format'])
image = os.path.join(conf.CAPTURES_ROOT, filename)
params = [conf.CASPER, conf.CAPTURE_SCRIPT, self.arguments['url'],
image, self.arguments['viewport'], self.arguments['target']]
casper = subprocess.Popen(params, stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
casper_output, casper_errors = casper.communicate()
logging.info(casper_output)
logging.info(casper_errors)
logging.info(casper.returncode)
# Here we are relying on convention:
# If success, subprocess.returncode == 0
# This could be fragile, need to investigate.
if casper.returncode:
raise Exception(casper_errors)
else:
return image
Traceback:
WARNING:root:Fatal: [Errno 2] No such file or directory; did you install phantomjs?
WARNING:root:
WARNING:root:1
ERROR:moment:Exception on /capture/ [GET]
Traceback (most recent call last):
File "/srv/environments/moment/local/lib/python2.7/site-packages/flask/app.py", line 1817, in wsgi_app
response = self.full_dispatch_request()
File "/srv/environments/moment/local/lib/python2.7/site-packages/flask/app.py", line 1477, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/srv/environments/moment/local/lib/python2.7/site-packages/flask/app.py", line 1381, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/srv/environments/moment/local/lib/python2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request
rv = self.dispatch_request()
File "/srv/environments/moment/local/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/srv/projects/moment/moment/views.py", line 45, in get_capture
image = capture.capture()
File "/srv/projects/moment/moment/models.py", line 229, in capture
raise Exception(casper_errors)
Exception
Note:
I am running in a virtualenv called "moment", and under a user called "moment".
The error is in the casper_output variable - those first three warning are the warning I log when I start the subprocess
I note that those warning are raised by root - I'd have expected them to be raised by "moment", the user that the supervisord process is supposed to run as

Although (for reasons you should investigate) the user has escalated from the original moment user to root, this does not mean that the process has the environment present when you login to a shell as root.
The chances are that your path is only that which is set in your supervisord.conf, and that's why phantomjs appears to be absent.
Environments are not looked up in some database of users; instead they are constructed by explicitly setting values (e.g. using a script) or by inheritance from the spawning process. In this case, you are inheriting from supervisor, and will get whatever environment supervisor had. If supervisor was run by something like cron, that environment will be empty.
Best practice in relation to supervisor is either to run it using a wrapper script which sets up the environment correctly, or just to explicitly set everything in the supervisord.conf. I generally recommend the latter, unless you have a common set of environment fixups in a file used by a lot of scripts (such as because you want it to run inside a virtualenv).