Localstack lambda is not triggered by SQS - python

I'm trying to use localstack to test a lambda triggered by an SQS event.
Here it is my localstack docker-compose:
version: "3.8"
services:
localstack:
container_name: "${LOCALSTACK_DOCKER_NAME-localstack_main}"
image: localstack/localstack:latest
network_mode: bridge
ports:
- "127.0.0.1:53:53"
- "127.0.0.1:53:53/udp"
- "127.0.0.1:443:443"
- "127.0.0.1:4566:4566"
- "127.0.0.1:4571:4571"
environment:
- SERVICES=${SERVICES- }
- DEBUG=1
- DATA_DIR=${DATA_DIR- }
- LAMBDA_EXECUTOR=docker
- LAMBDA_REMOTE_DOCKER=false
- LOCALSTACK_API_KEY=${LOCALSTACK_API_KEY- }
- KINESIS_ERROR_PROBABILITY=${KINESIS_ERROR_PROBABILITY- }
- DOCKER_HOST=unix:///var/run/docker.sock
- HOST_TMP_FOLDER=${TMPDIR}
volumes:
- "${TMPDIR:-/tmp/localstack}:/tmp/localstack"
- "/var/run/docker.sock:/var/run/docker.sock"
And here is my python lambda code in the handler.py file:
import json
import logging
import os
logger = logging.getLogger()
logger.setLevel(logging.INFO)
def handler(event, context):
try:
logger.info(f"### ENV VARS {os.environ}")
logger.info(f"### EVENT {event}")
logger.info(f"### ENV VARS {context}")
if event and "Records" in event:
for elt in event["Records"]:
logger.info(elt["body"])
return True
except Exception as e:
logger.error(e)
raise e
Here it is the lambda Dockerfile
FROM public.ecr.aws/lambda/python:3.9
RUN python -m pip install --upgrade pip
COPY requirements.txt ${LAMBDA_TASK_ROOT}
RUN pip install -r ${LAMBDA_TASK_ROOT}/requirements.txt --quiet
COPY src/ ${LAMBDA_TASK_ROOT}
CMD ["handler.handler"]
I run the following commands to create the lambda and the event
aws --endpoint http://localhost:4566 sqs create-queue --queue-name TEST_QUEUE
aws --endpoint http://localhost:4566 lambda create-function --timeout 120 --function-name python-lambda-test --handler handler.handler --code ImageUri=python-lambda-test:latest --role arn:aws:iam::000000000:role/lambda-ex
aws --endpoint http://localhost:4566 lambda create-event-source-mapping --function-name python-lambda-test --batch-size 1 --event-source-arn arn:aws:sqs:sa-east-1:000000000000:TEST_QUEUE
And finally, this command to test the Lambda function
aws --endpoint http://localhost:4566 sqs send-message --queue-url http://localhost:4566/000000000000/TEST_QUEUE --message-body 'this is a message'
But all this without success
I'm receiving this message in the logs
2021-08-30 18:30:23,439:API: 127.0.0.1 - - [30/Aug/2021 18:30:23] "POST / HTTP/1.1" 200 -
2021-08-30T18:30:23:DEBUG:localstack.services.awslambda.lambda_api: Found 1 source mappings for event from SQS queue arn:aws:sqs:sa-east-1:000000000000:TEST_QUEUE: ['arn:aws:lambda:sa-east-1:000000000000:function:python-lambda-test']
2021-08-30T18:30:23:DEBUG:localstack.services.awslambda.lambda_api: Starting SQS message polling thread for Lambda API
2021-08-30 18:30:23,467:API: 127.0.0.1 - - [30/Aug/2021 18:30:23] "POST / HTTP/1.1" 200 -
2021-08-30 18:30:23,520:API: 127.0.0.1 - - [30/Aug/2021 18:30:23] "POST / HTTP/1.1" 200 -
2021-08-30T18:30:23:DEBUG:localstack.services.awslambda.lambda_api: Sending event from event source arn:aws:sqs:sa-east-1:000000000000:TEST_QUEUE to Lambda arn:aws:lambda:sa-east-1:000000000000:function:python-lambda-test
2021-08-30T18:30:23:DEBUG:localstack.services.awslambda.lambda_executors: Lambda executed in Event (asynchronous) mode, no response will be returned to caller
2021-08-30T18:30:23:INFO:localstack.services.awslambda.lambda_executors: Determined main container target IP: 172.17.0.2
2021-08-30T18:30:23:INFO:localstack.services.awslambda.lambda_executors: Running lambda: arn:aws:lambda:sa-east-1:000000000000:function:python-lambda-test
2021-08-30T18:30:23:DEBUG:localstack.utils.docker: Running container with image: lambci/lambda:
2021-08-30T18:30:23:DEBUG:localstack.utils.docker: Creating container with image: lambci/lambda:
2021-08-30T18:30:23:DEBUG:localstack.services.awslambda.lambda_executors: Lambda arn:aws:lambda:sa-east-1:000000000000:function:python-lambda-test result / log output:
>
2021-08-30 18:30:23,635:API: 127.0.0.1 - - [30/Aug/2021 18:30:23] "POST / HTTP/1.1" 200 -
2021-08-30 18:30:23,658:API: 127.0.0.1 - - [30/Aug/2021 18:30:23] "POST / HTTP/1.1" 200 -
2021-08-30 18:30:23,677:API: 127.0.0.1 - - [30/Aug/2021 18:30:23] "POST / HTTP/1.1" 200 -
2021-08-30 18:30:23,701:API: 127.0.0.1 - - [30/Aug/2021 18:30:23] "POST / HTTP/1.1" 200 -
2021-08-30 18:30:23,734:API: 127.0.0.1 - - [30/Aug/2021 18:30:23] "POST / HTTP/1.1" 200 -
2021-08-30 18:30:23,754:API: 127.0.0.1 - - [30/Aug/2021 18:30:23] "POST / HTTP/1.1" 200 -
2021-08-30T18:30:23:INFO:localstack.utils.run: Thread run method <function LambdaExecutor.execute.<locals>.do_execute at 0x7f69de3770e0>(None) failed: Lambda process returned with error. Result: . Output:
Traceback (most recent call last):
File "/opt/code/localstack/.venv/lib/python3.7/site-packages/docker/api/client.py", line 268, in _raise_for_status
response.raise_for_status()
File "/opt/code/localstack/.venv/lib/python3.7/site-packages/requests/models.py", line 943, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: http+docker://localhost/v1.41/containers/create
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/code/localstack/localstack/utils/docker.py", line 1230, in create_container
container = create_container()
File "/opt/code/localstack/localstack/utils/docker.py", line 1226, in create_container
**kwargs,
File "/opt/code/localstack/.venv/lib/python3.7/site-packages/docker/models/containers.py", line 870, in create
resp = self.client.api.create_container(**create_kwargs)
File "/opt/code/localstack/.venv/lib/python3.7/site-packages/docker/api/container.py", line 430, in create_container
return self.create_container_from_config(config, name)
File "/opt/code/localstack/.venv/lib/python3.7/site-packages/docker/api/container.py", line 441, in create_container_from_config
return self._result(res, True)
File "/opt/code/localstack/.venv/lib/python3.7/site-packages/docker/api/client.py", line 274, in _result
self._raise_for_status(response)
File "/opt/code/localstack/.venv/lib/python3.7/site-packages/docker/api/client.py", line 270, in _raise_for_status
raise create_api_error_from_http_exception(e)
File "/opt/code/localstack/.venv/lib/python3.7/site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
raise cls(e, response=response, explanation=explanation)
docker.errors.APIError: 400 Client Error for http+docker://localhost/v1.41/containers/create: Bad Request ("invalid reference format")
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/code/localstack/localstack/services/awslambda/lambda_executors.py", line 336, in run_lambda_executor
func_details, env_vars, command, docker_flags=docker_flags, stdin=event_stdin_bytes
File "/opt/code/localstack/localstack/services/awslambda/lambda_executors.py", line 815, in execute_in_container
stdin=stdin,
File "/opt/code/localstack/localstack/utils/docker.py", line 1280, in run_container
additional_flags=additional_flags,
File "/opt/code/localstack/localstack/utils/docker.py", line 1238, in create_container
raise ContainerException()
localstack.utils.docker.ContainerException
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/opt/code/localstack/localstack/utils/run.py", line 138, in run
result = self.func(self.params, **kwargs)
File "/opt/code/localstack/localstack/services/awslambda/lambda_executors.py", line 247, in do_execute
return _run(func_arn=func_arn)
File "/opt/code/localstack/localstack/utils/cloudwatch/cloudwatch_util.py", line 157, in wrapped
raise e
File "/opt/code/localstack/localstack/utils/cloudwatch/cloudwatch_util.py", line 153, in wrapped
result = func(*args, **kwargs)
File "/opt/code/localstack/localstack/services/awslambda/lambda_executors.py", line 234, in _run
raise e
File "/opt/code/localstack/localstack/services/awslambda/lambda_executors.py", line 220, in _run
result = self._execute(func_arn, func_details, event, context, version)
File "/opt/code/localstack/localstack/services/awslambda/lambda_executors.py", line 450, in _execute
event=stdin, env_vars=environment, func_details=func_details, docker_flags=docker_flags
File "/opt/code/localstack/localstack/services/awslambda/lambda_executors.py", line 375, in run_lambda_executor
) from error
localstack.services.awslambda.lambda_executors.InvocationException: Lambda process returned with error. Result: . Output:
What is my mistake in this case?
Thanks in advance.

Related

Why does instabot in flask throw an error about wsgi?

Hello when i try to post on instagram flask doesn't let me do it and i have no idea why. I tested the instagram bot's library and it works but in flask it doesn't want to. Can someone help me solve this problem?
Python gives me this error:
2021-09-11 12:23:38,852 - INFO - Instabot version: 0.117.0 Started
2021-09-11 12:23:38,853 - INFO - Not yet logged in starting: PRE-LOGIN FLOW!
2021-09-11 12:23:41,559 - INFO - Logged-in successfully as 'user'!
2021-09-11 12:23:41,559 - INFO - LOGIN FLOW! Just logged-in: True
127.0.0.1 - - [11/Sep/2021 12:23:53] "POST /submit HTTP/1.1" 500 -
Traceback (most recent call last):
File "C:\Users\adamk\AppData\Local\Programs\Python\Python39\Lib\site-packages\flask\app.py", line 2088, in __call__
return self.wsgi_app(environ, start_response)
File "C:\Users\adamk\AppData\Local\Programs\Python\Python39\Lib\site-packages\flask\app.py", line 2073, in wsgi_app
response = self.handle_exception(e)
File "C:\Users\adamk\AppData\Local\Programs\Python\Python39\Lib\site-packages\flask\app.py", line 2070, in wsgi_app
response = self.full_dispatch_request()
File "C:\Users\adamk\AppData\Local\Programs\Python\Python39\Lib\site-packages\flask\app.py", line 1515, in full_dispatch_request
rv = self.handle_user_exception(e)
File "C:\Users\adamk\AppData\Local\Programs\Python\Python39\Lib\site-packages\flask\app.py", line 1513, in full_dispatch_request
rv = self.dispatch_request()
File "C:\Users\adamk\AppData\Local\Programs\Python\Python39\Lib\site-packages\flask\app.py", line 1499, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
File "D:\Programowanie\spotted\app.py", line 43, in submit
bot.login(username="user", password="password")
File "C:\Users\adamk\AppData\Local\Programs\Python\Python39\Lib\site-packages\instabot\bot\bot.py", line 450, in login
signal.signal(signal.SIGTERM, self.print_counters)
File "C:\Users\adamk\AppData\Local\Programs\Python\Python39\Lib\signal.py", line 47, in signal
handler = _signal.signal(_enum_to_int(signalnum), _enum_to_int(handler))
ValueError: signal only works in main thread of the main interpreter
127.0.0.1 - - [11/Sep/2021 12:23:53] "GET /submit?__debugger__=yes&cmd=resource&f=debugger.js HTTP/1.1" 200 -
127.0.0.1 - - [11/Sep/2021 12:23:53] "GET /submit?__debugger__=yes&cmd=resource&f=style.css HTTP/1.1" 200 -
127.0.0.1 - - [11/Sep/2021 12:23:54] "GET /submit?__debugger__=yes&cmd=resource&f=console.png HTTP/1.1" 200 -
127.0.0.1 - - [11/Sep/2021 12:23:54] "GET /submit?__debugger__=yes&cmd=resource&f=ubuntu.ttf HTTP/1.1" 200 -
Here's my code:
#app.route('/submit', methods=['POST', 'GET'])
def submit():
if request.method == 'POST':
message = request.form['text']
message = message.replace('\r', '')
if message.rstrip() == '':
return render_template('index.html', message='You cannot send an empty message!')
font = ImageFont.truetype("OpenSans-Regular.ttf", 40)
img = Image.open('image.jpg')
draw = ImageDraw.Draw(img)
draw.text((170, 280), message.rstrip(), (0, 0, 0), font=font)
i = 0
while os.path.exists(f'image-output-{i}.jpeg'):
i += 1
img.save(f'image-output-{i}.jpg')
cookie_del = glob.glob("config/*cookie.json")
os.remove(cookie_del[0])
bot = Bot()
bot.login(username="user", password="password")
bot.upload_photo('image.jpg', caption='')
return render_template('sent.html')
This is not a problem with flask but rather instabot. You can see in the traceback:
File "D:\Programowanie\spotted\app.py", line 43, in submit
Which is the only part of the traceback that actually references your own code. The login method is here. From the docstring, you'll see:
if login function is run threaded, for example in scheduled job,
signal will fail because it 'only works in main thread'.
In this case, you may want to call login(is_threaded=True).
flask uses thread-local objects so you need to add the argument suggested.

Airflow running inside a container cannot call another container

I'm trying to run a DAG that calls a docker container and executes a command inside it, but Airflow cannot execute the task. Follows the error launched:
*** Reading local file: /opt/airflow/logs/docker_operator_dag/docker_command_hello/2021-05-26T02:40:13.171571+00:00/2.log
[2021-05-26 02:45:26,001] {taskinstance.py:877} INFO - Dependencies all met for <TaskInstance: docker_operator_dag.docker_command_hello 2021-05-26T02:40:13.171571+00:00 [queued]>
[2021-05-26 02:45:26,030] {taskinstance.py:877} INFO - Dependencies all met for <TaskInstance: docker_operator_dag.docker_command_hello 2021-05-26T02:40:13.171571+00:00 [queued]>
[2021-05-26 02:45:26,031] {taskinstance.py:1068} INFO -
--------------------------------------------------------------------------------
[2021-05-26 02:45:26,031] {taskinstance.py:1069} INFO - Starting attempt 2 of 2
[2021-05-26 02:45:26,032] {taskinstance.py:1070} INFO -
--------------------------------------------------------------------------------
[2021-05-26 02:45:26,060] {taskinstance.py:1089} INFO - Executing <Task(DockerOperator): docker_command_hello> on 2021-05-26T02:40:13.171571+00:00
[2021-05-26 02:45:26,080] {standard_task_runner.py:52} INFO - Started process 67 to run task
[2021-05-26 02:45:26,083] {standard_task_runner.py:76} INFO - Running: ['airflow', 'tasks', 'run', 'docker_operator_dag', 'docker_command_hello', '2021-05-26T02:40:13.171571+00:00', '--job-id', '11', '--pool', 'default_pool', '--raw', '--subdir', 'DAGS_FOLDER/docker_job/docker-job.py', '--cfg-path', '/tmp/tmp3vl5dv7x', '--error-file', '/tmp/tmptazwx_tc']
[2021-05-26 02:45:26,084] {standard_task_runner.py:77} INFO - Job 11: Subtask docker_command_hello
[2021-05-26 02:45:26,181] {logging_mixin.py:104} INFO - Running <TaskInstance: docker_operator_dag.docker_command_hello 2021-05-26T02:40:13.171571+00:00 [running]> on host f1e5cfe4a07f
[2021-05-26 02:45:26,219] {taskinstance.py:1283} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_OWNER=airflow
AIRFLOW_CTX_DAG_ID=docker_operator_dag
AIRFLOW_CTX_TASK_ID=docker_command_hello
AIRFLOW_CTX_EXECUTION_DATE=2021-05-26T02:40:13.171571+00:00
AIRFLOW_CTX_DAG_RUN_ID=manual__2021-05-26T02:40:13.171571+00:00
[2021-05-26 02:45:26,227] {taskinstance.py:1482} ERROR - Task failed with exception
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 677, in urlopen
chunked=chunked,
File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 392, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/local/lib/python3.6/http/client.py", line 1287, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/local/lib/python3.6/http/client.py", line 1333, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/local/lib/python3.6/http/client.py", line 1282, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/local/lib/python3.6/http/client.py", line 1042, in _send_output
self.send(msg)
File "/usr/local/lib/python3.6/http/client.py", line 980, in send
self.connect()
File "/home/airflow/.local/lib/python3.6/site-packages/docker/transport/unixconn.py", line 43, in connect
sock.connect(self.unix_socket)
ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 727, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/util/retry.py", line 410, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/packages/six.py", line 734, in reraise
raise value.with_traceback(tb)
File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 677, in urlopen
chunked=chunked,
File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 392, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/local/lib/python3.6/http/client.py", line 1287, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/local/lib/python3.6/http/client.py", line 1333, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/local/lib/python3.6/http/client.py", line 1282, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/local/lib/python3.6/http/client.py", line 1042, in _send_output
self.send(msg)
File "/usr/local/lib/python3.6/http/client.py", line 980, in send
self.connect()
File "/home/airflow/.local/lib/python3.6/site-packages/docker/transport/unixconn.py", line 43, in connect
sock.connect(self.unix_socket)
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionRefusedError(111, 'Connection refused'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.6/site-packages/docker/api/client.py", line 207, in _retrieve_server_version
return self.version(api_version=False)["ApiVersion"]
File "/home/airflow/.local/lib/python3.6/site-packages/docker/api/daemon.py", line 181, in version
return self._result(self._get(url), json=True)
File "/home/airflow/.local/lib/python3.6/site-packages/docker/utils/decorators.py", line 46, in inner
return f(self, *args, **kwargs)
File "/home/airflow/.local/lib/python3.6/site-packages/docker/api/client.py", line 230, in _get
return self.get(url, **self._set_request_timeout(kwargs))
File "/home/airflow/.local/lib/python3.6/site-packages/requests/sessions.py", line 555, in get
return self.request('GET', url, **kwargs)
File "/home/airflow/.local/lib/python3.6/site-packages/requests/sessions.py", line 542, in request
resp = self.send(prep, **send_kwargs)
File "/home/airflow/.local/lib/python3.6/site-packages/requests/sessions.py", line 655, in send
r = adapter.send(request, **kwargs)
File "/home/airflow/.local/lib/python3.6/site-packages/requests/adapters.py", line 498, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionRefusedError(111, 'Connection refused'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/models/taskinstance.py", line 1138, in _run_raw_task
self._prepare_and_execute_task_with_callbacks(context, task)
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/models/taskinstance.py", line 1311, in _prepare_and_execute_task_with_callbacks
result = self._execute_task(context, task_copy)
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/models/taskinstance.py", line 1341, in _execute_task
result = task_copy.execute(context=context)
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/providers/docker/operators/docker.py", line 287, in execute
self.cli = self._get_cli()
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/providers/docker/operators/docker.py", line 319, in _get_cli
return APIClient(base_url=self.docker_url, version=self.api_version, tls=tls_config)
File "/home/airflow/.local/lib/python3.6/site-packages/docker/api/client.py", line 190, in __init__
self._version = self._retrieve_server_version()
File "/home/airflow/.local/lib/python3.6/site-packages/docker/api/client.py", line 215, in _retrieve_server_version
'Error while fetching server API version: {0}'.format(e)
docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', ConnectionRefusedError(111, 'Connection refused'))
[2021-05-26 02:45:26,230] {taskinstance.py:1532} INFO - Marking task as FAILED. dag_id=docker_operator_dag, task_id=docker_command_hello, execution_date=20210526T024013, start_date=20210526T024526, end_date=20210526T024526
[2021-05-26 02:45:26,261] {local_task_job.py:146} INFO - Task exited with return code 1
I'm using the Airflow docker-compose found here https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html and trying to run the following DAG:
from datetime import datetime, timedelta
from airflow import DAG
from airflow.operators.bash_operator import BashOperator
from airflow.operators.docker_operator import DockerOperator
from airflow.operators.python_operator import BranchPythonOperator
from airflow.operators.dummy_operator import DummyOperator
default_args = {
'owner' : 'airflow',
'description' : 'Use of the DockerOperator',
'depend_on_past' : False,
'start_date' : datetime(2021, 5, 1),
'email_on_failure' : False,
'email_on_retry' : False,
'retries' : 1,
'retry_delay' : timedelta(minutes=5)
}
with DAG('docker_operator_dag', default_args=default_args, schedule_interval="5 * * * *", catchup=False) as dag:
start_dag = DummyOperator(
task_id='start_dag'
)
end_dag = DummyOperator(
task_id='end_dag'
)
t1 = BashOperator(
task_id='print_current_date',
bash_command='date'
)
t2 = DockerOperator(
task_id='docker_command_sleep',
image='docker_image_task',
container_name='task___command_sleep',
api_version='auto',
auto_remove=True,
command="/bin/sleep 30",
docker_url="unix://var/run/docker.sock",
network_mode="bridge"
)
t3 = DockerOperator(
task_id='docker_command_hello',
image='docker_image_task',
container_name='task___command_hello',
api_version='auto',
auto_remove=True,
command="/bin/sleep 40",
docker_url="unix://var/run/docker.sock",
network_mode="bridge"
)
t4 = BashOperator(
task_id='print_hello',
bash_command='echo "hello world"'
)
start_dag >> t1
t1 >> t2 >> t4
t1 >> t3 >> t4
t4 >> end_dag
Also, I'm using Windows 10, I've tried adding the following in volumes: - "/var/run/docker.sock:/var/run/docker.sock" and I've succeeded in Ubuntu, but in Windows doesn't.
As requested, follow the docker-compose file:
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
#
# Basic Airflow cluster configuration for CeleryExecutor with Redis and PostgreSQL.
#
# WARNING: This configuration is for local development. Do not use it in a production deployment.
#
# This configuration supports basic configuration using environment variables or an .env file
# The following variables are supported:
#
# AIRFLOW_IMAGE_NAME - Docker image name used to run Airflow.
# Default: apache/airflow:master-python3.8
# AIRFLOW_UID - User ID in Airflow containers
# Default: 50000
# AIRFLOW_GID - Group ID in Airflow containers
# Default: 50000
# _AIRFLOW_WWW_USER_USERNAME - Username for the administrator account.
# Default: airflow
# _AIRFLOW_WWW_USER_PASSWORD - Password for the administrator account.
# Default: airflow
#
# Feel free to modify this file to suit your needs.
---
version: '3'
x-airflow-common:
&airflow-common
image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:2.0.2}
environment:
&airflow-common-env
AIRFLOW__CORE__EXECUTOR: CeleryExecutor
AIRFLOW__CORE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow#postgres/airflow
AIRFLOW__CELERY__RESULT_BACKEND: db+postgresql://airflow:airflow#postgres/airflow
AIRFLOW__CELERY__BROKER_URL: redis://:#redis:6379/0
AIRFLOW__CORE__FERNET_KEY: ''
AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION: 'true'
AIRFLOW__CORE__LOAD_EXAMPLES: 'false'
AIRFLOW__SCHEDULER__DAG_DIR_LIST_INTERVAL: 5 # Just to have a fast load in the front-end. Do not use it in production with those configurations.
AIRFLOW__API__AUTH_BACKEND: 'airflow.api.auth.backend.basic_auth'
AIRFLOW__CORE__ENABLE_XCOM_PICKLING: 'true' # "_run_image of the DockerOperator returns now a python string, not a byte string" Ref: https://github.com/apache/airflow/issues/13487
volumes:
- ./dags:/opt/airflow/dags
- ./logs:/opt/airflow/logs
- ./plugins:/opt/airflow/plugins
- "/var/run/docker.sock:/var/run/docker.sock" # We will pass the Docker Deamon as a volume to allow the webserver containers start docker images. Ref: https://stackoverflow.com/q/51342810/7024760
user: "${AIRFLOW_UID:-50000}:${AIRFLOW_GID:-50000}"
depends_on:
redis:
condition: service_healthy
postgres:
condition: service_healthy
services:
postgres:
image: postgres:13
environment:
POSTGRES_USER: airflow
POSTGRES_PASSWORD: airflow
POSTGRES_DB: airflow
volumes:
- postgres-db-volume:/var/lib/postgresql/data
healthcheck:
test: ["CMD", "pg_isready", "-U", "airflow"]
interval: 5s
retries: 5
restart: always
redis:
image: redis:latest
ports:
- 6379:6379
healthcheck:
test: ["CMD", "redis-cli", "ping"]
interval: 5s
timeout: 30s
retries: 50
restart: always
airflow-webserver:
<<: *airflow-common
command: webserver
ports:
- 8080:8080
healthcheck:
test: ["CMD", "curl", "--fail", "http://localhost:8080/health"]
interval: 10s
timeout: 10s
retries: 5
restart: always
airflow-scheduler:
<<: *airflow-common
command: scheduler
restart: always
airflow-worker:
<<: *airflow-common
command: celery worker
restart: always
airflow-init:
<<: *airflow-common
command: version
environment:
<<: *airflow-common-env
_AIRFLOW_DB_UPGRADE: 'true'
_AIRFLOW_WWW_USER_CREATE: 'true'
_AIRFLOW_WWW_USER_USERNAME: ${_AIRFLOW_WWW_USER_USERNAME:-airflow}
_AIRFLOW_WWW_USER_PASSWORD: ${_AIRFLOW_WWW_USER_PASSWORD:-airflow}
flower:
<<: *airflow-common
command: celery flower
ports:
- 5555:5555
healthcheck:
test: ["CMD", "curl", "--fail", "http://localhost:5555/"]
interval: 10s
timeout: 10s
retries: 5
restart: always
volumes:
postgres-db-volume:

Python DocuSign API receiving redirect URI error on the sample app

I am running the Python and React MyUni Education sample app and have it loading in my localhost3000. When I click on the logins though it is giving me redirect error.
I created a new Docusign integration key in case there was something incorrectly configured in my previous set up, but the error remains the same:
"The redirect URI is not registered properly with DocuSign" error
Docusign sandbox configured with:
Redirect URIs "http://localhost:5001/api"
.env file I believe has the right environment variables set but please let me know if I am missing anything.
# Configuration file for the example
DS_CLIENT_ID=blank
DS_CLIENT_SECRET=blank
DS_IMPERSONATED_USER_GUID=blank
DS_TARGET_ACCOUNT_ID=false
DS_PAYMENT_GATEWAY_ID=blank
DS_PAYMENT_GATEWAY_NAME=Stripe
DS_PAYMENT_GATEWAY_DISPLAY_NAME=Stripe
DS_PRIVATE_KEY=blank
# React environment variables
**REACT_APP_DS_RETURN_URL=http://localhost:3000
REACT_APP_API_BASE_URL=http://localhost:5001/api**
REACT_APP_DS_AUTH_SERVER=https://account-d.docusign.com
# Demo Docusign API URL
REACT_APP_DS_DEMO_SERVER=https://demo.docusign.net
REACT_APP_DS_CLICKWRAP_URL=//demo.docusign.net/clickapi/sdk/latest/docusign-click.js
I believe it is affecting the JWT, as when I selected the "continue with a preconfigured login" I received a "Request failed with status code 500" error as well. Logs from that below.
PS C:\Users\windowsuser\OneDrive\Documents\GitHub\sample-app-myuni> flask run --port 5001
* Environment: production
WARNING: This is a development server. Do not use it in a production deployment.
Use a production WSGI server instead.
* Debug mode: off
* Running on http://127.0.0.1:5001/ (Press CTRL+C to quit)
127.0.0.1 - - [27/Jul/2020 16:13:20] "GET /api/get_status HTTP/1.1" 200 -
127.0.0.1 - - [27/Jul/2020 16:13:51] "GET /api/code_grant_auth HTTP/1.1" 401 -
127.0.0.1 - - [27/Jul/2020 16:14:03] "OPTIONS /api/jwt_auth HTTP/1.1" 200 -
[2020-07-27 16:14:04,237] ERROR in app: Exception on /api/jwt_auth [POST]
Traceback (most recent call last):
File "c:\python38\lib\site-packages\flask\app.py", line 2446, in wsgi_app
response = self.full_dispatch_request()
File "c:\python38\lib\site-packages\flask\app.py", line 1951, in full_dispatch_request
rv = self.handle_user_exception(e)
File "c:\python38\lib\site-packages\flask_cors\extension.py", line 161, in wrapped_function
return cors_after_request(app.make_response(f(*args, **kwargs)))
File "c:\python38\lib\site-packages\flask\app.py", line 1820, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "c:\python38\lib\site-packages\flask\_compat.py", line 39, in reraise
raise value
File "c:\python38\lib\site-packages\flask\app.py", line 1949, in full_dispatch_request
rv = self.dispatch_request()
File "c:\python38\lib\site-packages\flask\app.py", line 1935, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "c:\python38\lib\site-packages\flask_cors\decorator.py", line 128, in wrapped_function
resp = make_response(f(*args, **kwargs))
File "C:\Users\windowsuser\OneDrive\Documents\GitHub\sample-app-myuni\app\api\auth.py", line 43, in jwt_auth
DsClient.update_token()
File "C:\Users\windowsuser\OneDrive\Documents\GitHub\sample-app-myuni\app\ds_client.py", line 36, in update_token
client.request_jwt_user_token(DsConfig.client_id(),
File "c:\python38\lib\site-packages\docusign_esign\client\api_client.py", line 679, in request_jwt_user_token
raise ArgumentException("Private key not supplied or is invalid!")
File "c:\python38\lib\site-packages\docusign_esign\client\api_exception.py", line 51, in __init__
super(Exception).__init__(*args, **kwargs)
TypeError: super() argument 1 must be type, not str
127.0.0.1 - - [27/Jul/2020 16:14:04] "POST /api/jwt_auth HTTP/1.1" 500 -
127.0.0.1 - - [27/Jul/2020 16:18:14] "GET /api/code_grant_auth HTTP/1.1" 401 -
The integration key settings must match exactly
I think what you can check the URL where you get this error, it has first the IK which you created (GUID value starting with 61C9) then it has the redirectUri. It is URI encoded, but you can figure it out and make sure the same exact matching URI is set in the IK settings in the DocuSign Developer account.
I think it's the localhost:3000 for the front-end and not the localhost:5001 for the back-end and maybe there's an endpoint under there, just check and confirm and add it, wait 30 seconds and try again.
Okay just tested this in MySure app. Adding the following will allow the log in function to work as expected.
http://localhost:5001/api/callback
http://localhost:3000/callback
http://localhost:3000/

Telnet to Python server gets stuck

I am preparing a demo, and I wanted to use a simple server for it, instead of nginx or any other popular server.
I got the following one from python official page:
import SimpleHTTPServer
import SocketServer
PORT = 8000
Handler = SimpleHTTPServer.SimpleHTTPRequestHandler
httpd = SocketServer.TCPServer(("", PORT), Handler)
print "serving at port", PORT
httpd.serve_forever()
So, I run it, and everything seems to be fine:
python -m SimpleHTTPServer 8000
If I go to the browser, or curl the IP address, it works just fine:
xx.xx.xx.xx - - [31/May/2019 09:36:33] "GET / HTTP/1.1" 200 -
xx.xx.xx.xx - - [31/May/2019 09:37:43] "GET / HTTP/1.1" 200 -
xx.xx.xx.xx - - [31/May/2019 09:37:46] "HEAD / HTTP/1.1" 200 -
xx.xx.xx.xx - - [31/May/2019 09:39:03] "HEAD / HTTP/1.1" 200 -
xx.xx.xx.xx - - [31/May/2019 09:39:06] "GET / HTTP/1.1" 200 -
xx.xx.xx.xx - - [31/May/2019 09:39:12] "GET / HTTP/1.1" 200 -
xx.xx.xx.xx - - [31/May/2019 09:39:13] "GET / HTTP/1.1" 200 -
Now, when I connect to the server through telnet, and I do GET, the server get stuck:
$ telnet xx.xx.xx.xx 8000
Trying xx.xx.xx.xx...
Connected to xx.xx.xx.xx.
Escape character is '^]'.
GET / HTTP/1.1
...It just hangs here foever...
When I hit on ctrl+c on the server, it shows me an exception that happened, that I don't know how to interpret in this context:
^C----------------------------------------
Exception happened during processing of request from ('xx.xx.xx.xx', 47346)
Traceback (most recent call last):
File "/usr/lib/python3.5/socketserver.py", line 313, in _handle_request_noblock
self.process_request(request, client_address)
File "/usr/lib/python3.5/socketserver.py", line 341, in process_request
self.finish_request(request, client_address)
File "/usr/lib/python3.5/socketserver.py", line 354, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/usr/lib/python3.5/socketserver.py", line 681, in __init__
self.handle()
File "/usr/lib/python3.5/http/server.py", line 422, in handle
self.handle_one_request()
File "/usr/lib/python3.5/http/server.py", line 400, in handle_one_request
if not self.parse_request():
File "/usr/lib/python3.5/http/server.py", line 334, in parse_request
_class=self.MessageClass)
File "/usr/lib/python3.5/http/client.py", line 206, in parse_headers
line = fp.readline(_MAXLINE + 1)
File "/usr/lib/python3.5/socket.py", line 576, in readinto
return self._sock.recv_into(b)
KeyboardInterrupt
----------------------------------------
I also SSH-ed into the same machine where the server is running and did the request connecting to the localhost. Hanged too.
I would like to know:
How is telnet GET different to other GETs.
At python server level, what does the exception mean, and how to solve it.
How is telnet GET different to other GETs.
telnet is not different here. Your server is hanging because it is waiting for you to finish typing your GET request. For this, you just need to press Enter twice, after typing your GET request.
This is because:
One CR-LF character is required to end the message header section of the HTTP request (as specified here)
another one is the actual CR-LF that tells telnet you finished entering input
At python server level, what does the exception mean, and how to solve it.
This is just a consequence of you pressing Ctrl + C, thus interrupting the Python process running your SimpleHTTPServer.

Django server is hanging in request handler

I do a poll once a second, and after some idle time (10min - 1h) the server stops responding. If I press Ctrl-C handler is terminated and the server is back alive. Error after pressing Ctrl-C (while trying to connect with a new client):
...same messages repeat...
[22/Sep/2016 21:54:27] "GET /ajax/?id=STATUS_POLL&_=1474566714286 HTTP/1.1" 200 0
[2016-09-22 21:54:28,217: DEBUG/MainProcess] NONE
[22/Sep/2016 21:54:28] "GET /ajax/?id=STATUS_POLL&_=1474566714287 HTTP/1.1" 200 0
[2016-09-23 09:33:14,657: INFO/MainProcess] Starting view
[23/Sep/2016 09:36:34] "GET / HTTP/1.1" 200 18970
Traceback (most recent call last):
File "c:\python27\lib\wsgiref\handlers.py", line 86, in run
self.finish_response()
File "c:\python27\lib\wsgiref\handlers.py", line 128, in finish_response
self.write(data)
File "c:\python27\lib\wsgiref\handlers.py", line 212, in write
self.send_headers()
File "c:\python27\lib\wsgiref\handlers.py", line 270, in send_headers
self.send_preamble()
File "c:\python27\lib\wsgiref\handlers.py", line 194, in send_preamble
'Date: %s\r\n' % format_date_time(time.time())
File "c:\python27\lib\socket.py", line 328, in write
self.flush()
File "c:\python27\lib\socket.py", line 307, in flush
self._sock.sendall(view[write_offset:write_offset+buffer_size])
error: [Errno 10053] An established connection was aborted by the software in your host machine
[23/Sep/2016 09:36:34] "GET / HTTP/1.1" 500 59
Request handler:
class CTrackerView(TemplateView):
__process = ProcessWrapper()
#staticmethod
def ajax_handler(request):
if request.GET.get('id', "") == "STATUS_POLL":
if CTrackerView.__process.is_alive(no_log=False):
return HttpResponse("UPDATING")
else:
return HttpResponse("")
process_wrapper.py:
class ProcessWrapper:
def is_alive(self, no_log=False):
if (not hasattr(self, "_process")) or not self._process:
if not no_log:
log("NONE", DEBUG)
return False
As you can see from the last code fragment, it just checks the variable is not defined (process is deleted) and return False during all the idle time. (Message "NONE" in log). The problem is difficult to reproduce.

Categories