I have a Web App built in Flask where tweets are captured (using Tweepy library) and displayed on the front-end. I used Socket IO to display the tweets live on the front-end.
My code works fine when I run this locally. The tweets appear instantly.
However, when i Dockerized the web app, the front-end doesn't update immediately. It takes some time to show the changes (sometimes I think tweets are lost due to the slowness)
Below are code extracts from my website:
fortsocket.js
$(document).ready(function () {
/************************************/
/*********** My Functions ***********/
/************************************/
function stream_active_setup() {
$("#favicon").attr("href", "/static/icons/fortnite-active.png");
$("#stream-status-ic").attr("src", "/static/icons/stream-active.png");
$("#stream-status-text").text("Live stream active");
}
function stream_inactive_setup() {
$("#favicon").attr("href", "/static/icons/fortnite-inactive.png");
$("#stream-status-ic").attr("src", "/static/icons/stream-inactive.png");
$("#stream-status-text").text("Live stream inactive");
}
/*********************************/
/*********** My Events ***********/
/*********************************/
// Socket connection to server
// Prometheus
//var socket = io.connect('http://104.131.173.145:8083');
// Local
var socket = io.connect(window.location.protocol + '//' + document.domain + ':' + location.port);
// Heroku
//var socket = io.connect('https://fortweet.herokuapp.com/');
// Send a hello to know
// if a stream is already active
socket.on('connect', () => {
socket.emit('hello-stream', 'hello-stream');
});
// Listene for reply from hello
socket.on('hello-reply', function (bool) {
if (bool == true) {
stream_active_setup()
} else {
stream_inactive_setup()
}
});
// Listens for tweets
socket.on('stream-results', function (results) {
// Insert tweets in divs
$('#live-tweet-container').prepend(`
<div class="row justify-content-md-center mt-3">
<div class="col-md-2">
<img width="56px" height="56px" src="${results.profile_pic !== "" ? results.profile_pic : "/static/icons/profile-pic.png"}" class="mx-auto d-block rounded" alt="">
</div>
<div class="col-md-8 my-auto">
<div><b>${results.author}</b></div>
<div>${results.message}</div>
</div>
</div>
`);
});
// Listener for when a stream of tweets starts
socket.on('stream-started', function (bool) {
if (bool == true) {
stream_active_setup()
}
});
// Listener for when a stream of tweets ends
socket.on('stream-ended', function (bool) {
if (bool == true) {
stream_inactive_setup()
}
});
});
init.py
# Create the app
app = create_app()
# JWT Configurations
jwt = JWTManager(app)
# Socket IO
socketio = SocketIO(app, cors_allowed_origins="*")
# CORS
CORS(app)
app.config["CORS_HEADERS"] = "Content-Type"
# Creates default admins and insert in db
create_default_admin()
# Main error handlers
#app.errorhandler(404) # Handling HTTP 404 NOT FOUND
def page_not_found(e):
return Err.ERROR_NOT_FOUND
# Listen for hello emit data
# from client
#socketio.on("hello-stream")
def is_stream_active(hello_stream):
emit("hello-reply", streamer.StreamerInit.is_stream_active(), broadcast=True)
streamer.py
import time
import tweepy
import threading as Coroutine
import app.messages.constants as Const
import app.setup.settings as settings_mod
import app.models.tweet as tweet_mod
import app.services.logger as logger
import app
class FStreamListener(tweepy.StreamListener):
def __init__(self):
self.start_time = time.time()
self.limit = settings_mod.TwitterSettings.get_instance().stream_time
logger.get_logger().debug("Live capture has started")
# Notify client that a live capture will start
app.socketio.emit(
"stream-started", True, broadcast=True,
)
super(FStreamListener, self).__init__()
def on_status(self, status):
if (time.time() - self.start_time) < self.limit:
# Create tweet object
forttweet = tweet_mod.TweetModel(
status.source,
status.user.name,
status.user.profile_background_image_url_https,
status.text,
status.created_at,
status.user.location,
)
# Emit to socket
app.socketio.emit(
"stream-results",
{
"profile_pic": forttweet.profile_pic,
"author": forttweet.author,
"message": forttweet.message,
},
broadcast=True,
)
# Add to database
forttweet.insert()
return True
else:
logger.get_logger().debug("Live capture has ended")
# Notify client that a live capture has ended
app.socketio.emit(
"stream-ended", True, broadcast=True,
)
# Stop the loop of streaming
return False
def on_error(self, status):
logger.get_logger().debug(f"An error occurred while fetching tweets: {status}")
raise Exception(f"An error occurred while fetching tweets: {status}")
class StreamerInit:
# [Private] Twitter configurations
def __twitterInstantiation(self):
# Get settings instance
settings = settings_mod.TwitterSettings.get_instance()
# Auths
auth = tweepy.OAuthHandler(settings.consumer_key, settings.consumer_secret,)
auth.set_access_token(
settings.access_token, settings.access_token_secret,
)
# Get API
api = tweepy.API(auth)
# Live Tweets Streaming
myStreamListener = FStreamListener()
myStream = tweepy.Stream(auth=api.auth, listener=myStreamListener)
myStream.filter(track=settings.filters)
def start(self):
for coro in Coroutine.enumerate():
if coro.name == Const.FLAG_TWEETS_LIVE_CAPTURE:
return False
stream = Coroutine.Thread(target=self.__twitterInstantiation)
stream.setName(Const.FLAG_TWEETS_LIVE_CAPTURE)
stream.start()
return True
#staticmethod
def is_stream_active():
for coro in Coroutine.enumerate():
if coro.name == Const.FLAG_TWEETS_LIVE_CAPTURE:
return True
return False
The streamer.py is called on a button click
Dockerfile
# Using python 3.7 in Alpine
FROM python:3.6.5-stretch
# Set the working directory to /app
WORKDIR /app
# Copy the current directory contents into the container at /app
ADD . /app
RUN apt-get update -y && apt-get upgrade -y && pip install -r requirements.txt
# Run the command
ENTRYPOINT ["uwsgi", "app.ini"]
#ENTRYPOINT ["./entry.sh"]
docker-compose.yml
version: "3.8"
services:
fortweet:
container_name: fortweet
image: mervin16/fortweet:dev
build: ./
env_file:
- secret.env
networks:
plutusnet:
ipv4_address: 172.16.0.10
expose:
- 8083
restart: always
nginx_fortweet:
image: nginx
container_name: nginx_fortweet
ports:
- "8083:80"
networks:
plutusnet:
ipv4_address: 172.16.0.100
volumes:
- ./nginx/nginx.conf:/etc/nginx/conf.d/default.conf
depends_on:
- fortweet
restart: always
networks:
plutusnet:
name: plutus_network
driver: bridge
ipam:
driver: default
config:
- subnet: 172.16.0.0/24
gateway: 172.16.0.1
app.ini
[uwsgi]
module = run:app
master = true
processes = 5
# Local & Prometheus
http-socket = 0.0.0.0:8083
http-websockets = true
chmod-socket = 660
vacuum = true
die-on-term = true
For a full, updated code, you can find it here under the branch dev/mervin
Any help is appreciated.
in order to see if ipv6 is responsible i would suggest you shutdown everything
open /etc/sysctl.conf and add the following lines to disable ipv6
net.ipv6.conf.all.disable_ipv6=1
net.ipv6.conf.default.disable_ipv6=1
net.ipv6.conf.lo.disable_ipv6=1
run sudo sysctl -p so changes can take effect
start nginx and the docker again
if you dont see any difference then you can just change the settings to 0 and rerun sysctl -p and let me know
Unfortunately I can't reproduce the issue without the configuration, so I can't verify my answer.
I was able to find a similar issue on a JP's blog: Performance problems with Flask and Docker
In short, it might be that having both IPv6 and IPv4 configs on the container are causing the issue.
In order to verify the issue:
Run the docker
Go inside the running container and change the hosts file so that it won't map IPv6 to localhost
Run application again inside of container
If the app runs smoothly then you've identified your issue.
The solution would be to tweak the uwsgi parameters.
What the author did in the blog post:
CMD uwsgi -s /tmp/uwsgi.sock -w project:app --chown-socket=www-data:www-data --enable-threads & nginx -g 'daemon off;'
Related
I have a Flask server that is running in an Azure VM (Ubuntu 20.04) that is supposed to be run on http://127.0.0.1:5000 and an Angular app that serves as a frontend and hosts 0.0.0.0/80. The Angular app is supposed to send GET/POST requests to the Flask server, but when I try to do so I get the following error [1].
I have CORS enabled for all domains on all routes in Flask.
And if I send the request using wget it works perfectly fine.
Here's how I'm sending my request from Angular:
this.http.post<ILoginResponse>('http://127.0.0.1:5000/login', {username: un, password: pswrd}).subscribe(data => {
this.loginResponse.success = data.success;
this.loginResponse.teamID = data.teamID;
})
With ILoginResponse being:
export interface ILoginResponse{
success: boolean;
teamID: string;
}
I have set rules in the Azure portal to allow connections on port 5000 and unlocked the port in the firewall in the VM itself. Running Flask with --host 0.0.0.0 does not help either.
Any idea of what could help or which direction I could look in?
[1]
scheduleTask # zone-evergreen.js:2845
scheduleTask # zone-evergreen.js:385
onScheduleTask # zone-evergreen.js:272
scheduleTask # zone-evergreen.js:378
scheduleTask # zone-evergreen.js:210
scheduleMacroTask # zone-evergreen.js:233
scheduleMacroTaskWithCurrentZone # zone-evergreen.js:1134
(anonymous) # zone-evergreen.js:2878
proto.<computed> # zone-evergreen.js:1449
(anonymous) # http.js:1785
_trySubscribe # Observable.js:42
subscribe # Observable.js:28
innerSubscribe # innerSubscribe.js:67
_innerSub # mergeMap.js:57
_tryNext # mergeMap.js:51
_next # mergeMap.js:34
next # Subscriber.js:49
(anonymous) # subscribeToArray.js:3
_trySubscribe # Observable.js:42
subscribe # Observable.js:28
call # mergeMap.js:19
subscribe # Observable.js:23
call # filter.js:13
subscribe # Observable.js:23
call # map.js:16
subscribe # Observable.js:23
checkCredentials # login.component.ts:63
login # login.component.ts:44
LoginComponent_Template_form_ngSubmit_6_listener # login.component.html:12
executeListenerWithErrorHandling # core.js:14994
wrapListenerIn_markDirtyAndPreventDefault # core.js:15029
schedulerFn # core.js:25687
__tryOrUnsub # Subscriber.js:183
next # Subscriber.js:122
_next # Subscriber.js:72
next # Subscriber.js:49
next # Subject.js:39
emit # core.js:25656
onSubmit # forms.js:5719
FormGroupDirective_submit_HostBindingHandler # forms.js:5774
executeListenerWithErrorHandling # core.js:14994
wrapListenerIn_markDirtyAndPreventDefault # core.js:15029
(anonymous) # platform-browser.js:582
invokeTask # zone-evergreen.js:399
onInvokeTask # core.js:28289
invokeTask # zone-evergreen.js:398
runTask # zone-evergreen.js:167
invokeTask # zone-evergreen.js:480
invokeTask # zone-evergreen.js:1621
globalZoneAwareCallback # zone-evergreen.js:1647
HttpErrorResponse {headers: HttpHeaders, status: 0, statusText: 'Unknown Error', url: 'http://127.0.0.1:5000/login', ok: false, …}
error: ProgressEvent {isTrusted: true, lengthComputable: false, loaded: 0, total: 0, type: 'error', …}
headers: HttpHeaders {normalizedNames: Map(0), lazyUpdate: null, headers: Map(0)}
message: "Http failure response for http://127.0.0.1:5000/login: 0 Unknown Error"
name: "HttpErrorResponse"
ok: false
status: 0
statusText: "Unknown Error"
url: "http://127.0.0.1:5000/login"
[[Prototype]]: HttpResponseBase
constructor: class HttpErrorResponse
[[Prototype]]: Object```
Thank you furas. Posting your suggestion as an answer to help other community members.
Address 127.0.0.1 can assess only programs which run on the same computer but Angular runs in the user's browser so it runs on user's computer so using 127.0.0.1 it tries to access user's computer, not the server with Flask.
Running Flask with --host 0.0.0.0 does not help either.
You can refer to the answer by nwillo to run the flask app on IPv4 address
So, I was trying to get some stats for the docker containers running on a server. I tried multiple approaches:
Docker SDK for python
Making curl request using subprocess module
Socket
The first two worked quite well but I could not employ the socket method. I want to compare these ways to figure out which one is faster and better for my usage.
Basically, I am trying to convert the curl command into python request.
Here's the curl command I have used:
curl -H --unix-socket /var/run/docker.sock http://localhost/containers/CONTAINER_ID/stats?stream=false
Can someone help me with this?
ensure process that you want to run Python SDK on has access to /var/run/docker.sock. In example I mount it as a volume into container
python code is then using sockets anyway so no need to spawn additional subprocesses to do curl. You have full access to all status information. Where more detailed into is required use
htpc:
container_name: htpc
build:
context: .
dockerfile: flask-Dockerfile
ports:
- "5000:5000"
restart: unless-stopped
volumes:
- /var/run/docker.sock:/var/run/docker.sock # map through socket
def docker_api():
client = docker.from_env()
if request.args["action"] == "list":
l = []
for c in client.containers.list(all=True):
# string conversion only works for 6 digit ms, strip back to 26 chars
if c.status == "running":
utc_time = datetime.strptime(client.api.inspect_container(c.short_id)["State"]["StartedAt"][:26]+"Z", "%Y-%m-%dT%H:%M:%S.%fZ")
else:
utc_time = datetime.utcnow()
l.append({"id":c.short_id, "name":c.name, "image":c.image.tags[0] if len(c.image.tags)>0 else "unknown",
"status":c.status, "logs":True, "inspect":True,
"stats":c.status=="running", "restart":c.status=="running", "stop":c.status=="running", "start":c.status!="running", "delete":True,
"uptime":(utc_time - datetime(1970, 1, 1)).total_seconds(),
})
return Response(json.dumps(l), mimetype="text/json")
elif request.args["action"] == "logs":
return Response(client.containers.get(request.args["id"]).logs().decode(), mimetype="text/plain")
elif request.args["action"] == "stats":
return Response(json.dumps(client.containers.get(request.args["id"]).stats(stream=False)), mimetype="text/json")
elif request.args["action"] == "inspect":
js = client.api.inspect_container(request.args["id"])
m = {"StartedAt":js["State"]["StartedAt"], "Mounts":js["Mounts"]}
for i, mp in enumerate(m["Mounts"]):
m["Mounts"][i] = {k: mp[k] for k in ('Source', 'Destination')}
e = {"Env":js["Config"]["Env"], 'RestartPolicy':js['HostConfig']['RestartPolicy']}
p = {"Ports":js["NetworkSettings"]["Ports"]}
js = {**m, **e, **p} #, **js}
return Response(json.dumps(js), mimetype="text/json")
I have docker-compose.yml to bring up two docker containers.
1.) front-end - reactjs - running on port 3000
2.) back-end - python (flask) - running on port 8083
The front-end is calling the python server for a POST method through the url "http://127.0.0.1:8083/api using axios, but now since these are running at two containers, how can I connect the front-end to the python server through that url?
This is my docker-compose.yml
version: "3.2"
services:
frontend:
build: ./frontend
environment:
CHOKIDAR_USEPOLLING: "true"
volumes:
- /app/node_modules
- ./frontend:/app
ports:
- 80:3000
depends_on:
- backend
backend:
build: ./backends/jananath
volumes:
- ./backends/jananath:/usr/src/app
environment:
FLASK_APP: /usr/src/app/server.py
FLASK_DEBUG: 1
FLASK_RUN_EXTRA_FILES: "/usr/src/app/banuka.txt"
ports:
- 5000:8083
And this is the server.py python server
from flask import Flask, render_template, request, url_for, redirect
import os
from os import path, walk
from flask_cors import CORS
app = Flask(__name__)
CORS(app)
PORT = 8083
#app.route('/api', methods=['POST', 'GET'])
def upload_file():
file = request.files['file']
filename = file.filename
print(filename)
filepath = os.path.abspath(filename)
print(filepath)
response_ = {}
response_['filename'] = str(filename)
response_['filepath'] = str(filepath)
return response_
if __name__ == '__main__':
app.run(host='0.0.0.0', port=PORT, extra_files=extra_files)
And this is the axios calling a POST method from front-end to this server.py
const onFormSubmit = async (e) => {
e.preventDefault();
const formData = new FormData();
formData.append("file", file);
try {
const res = await axios.post("http://127.0.0.1:8083/api", formData, { // this url should change now?
headers: {
"Content-Type": "multipart/form-data",
"Access-Control-Allow-Credentials": "true",
"crossorigin": "true",
},
});
const {filename, filepath} = res.data;
setUploadedFile({filename, filepath});
} catch (error) {
console.log(error);
}
};
I think the problem here is the URL is now changed, but I don't know how to fix that. Can someone tell me how to fix this and what is the correct URL I should hit?
Thank you!
try backend:8083 in your react app, you will need to rebuild your react image.
Does anybody know how to run Beam Python pipelines with Flink when Flink is running as pods in Kubernetes?
I have successfully managed to run a Beam Python pipeline using the Portable runner and the job service pointing to a local Flink server running in Docker containers.
I was able to achieve that mounting the Docker socket in my Flink containers, and running Flink as root process, so the class DockerEnvironmentFactory can create the Python harness container.
Unfortunately, I can't use the same solution when Flink is running in Kubernetes. Moreover, I don't want to create the Python harness container using the Docker command from my pods.
It seems that Bean runner automatically selects Docker for executing Python pipelines. However, I noticed there is an implementation called ExternalEnvironmentFactory, but I am not sure how to use it.
Is there a way to deploy a side container and use a different factory to run the Python harness process? What is the correct approach?
This is the patch for DockerEnvironmentFactory:
diff -pr beam-release-2.15.0/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/environment/DockerEnvironmentFactory.java beam-release-2.15.0-1/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/environment/DockerEnvironmentFactory.java
*** beam-release-2.15.0/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/environment/DockerEnvironmentFactory.java 2019-08-14 22:33:41.000000000 +0100
--- beam-release-2.15.0-1/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/environment/DockerEnvironmentFactory.java 2019-09-09 16:02:07.000000000 +0100
*************** package org.apache.beam.runners.fnexecut
*** 19,24 ****
--- 19,26 ----
import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects.firstNonNull;
+ import java.net.InetAddress;
+ import java.net.UnknownHostException;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.time.Duration;
*************** public class DockerEnvironmentFactory im
*** 127,133 ****
ImmutableList.<String>builder()
.addAll(gcsCredentialArgs())
// NOTE: Host networking does not work on Mac, but the command line flag is accepted.
! .add("--network=host")
// We need to pass on the information about Docker-on-Mac environment (due to missing
// host networking on Mac)
.add("--env=DOCKER_MAC_CONTAINER=" + System.getenv("DOCKER_MAC_CONTAINER"));
--- 129,135 ----
ImmutableList.<String>builder()
.addAll(gcsCredentialArgs())
// NOTE: Host networking does not work on Mac, but the command line flag is accepted.
! .add("--network=flink")
// We need to pass on the information about Docker-on-Mac environment (due to missing
// host networking on Mac)
.add("--env=DOCKER_MAC_CONTAINER=" + System.getenv("DOCKER_MAC_CONTAINER"));
*************** public class DockerEnvironmentFactory im
*** 222,228 ****
private static ServerFactory getServerFactory() {
ServerFactory.UrlFactory dockerUrlFactory =
! (host, port) -> HostAndPort.fromParts(DOCKER_FOR_MAC_HOST, port).toString();
if (RUNNING_INSIDE_DOCKER_ON_MAC) {
// If we're already running in a container, we need to use a fixed port range due to
// non-existing host networking in Docker-for-Mac. The port range needs to be published
--- 224,230 ----
private static ServerFactory getServerFactory() {
ServerFactory.UrlFactory dockerUrlFactory =
! (host, port) -> HostAndPort.fromParts(getCanonicalHostName(), port).toString();
if (RUNNING_INSIDE_DOCKER_ON_MAC) {
// If we're already running in a container, we need to use a fixed port range due to
// non-existing host networking in Docker-for-Mac. The port range needs to be published
*************** public class DockerEnvironmentFactory im
*** 237,242 ****
--- 239,252 ----
}
}
+ private static String getCanonicalHostName() throws RuntimeException {
+ try {
+ return InetAddress.getLocalHost().getCanonicalHostName();
+ } catch (UnknownHostException e) {
+ throw new RuntimeException(e);
+ }
+ }
+
/** Provider for DockerEnvironmentFactory. */
public static class Provider implements EnvironmentFactory.Provider {
private final boolean retainDockerContainer;
*************** public class DockerEnvironmentFactory im
*** 269,275 ****
public ServerFactory getServerFactory() {
switch (getPlatform()) {
case LINUX:
! return ServerFactory.createDefault();
case MAC:
return DockerOnMac.getServerFactory();
default:
--- 279,286 ----
public ServerFactory getServerFactory() {
switch (getPlatform()) {
case LINUX:
! return DockerOnMac.getServerFactory();
! // return ServerFactory.createDefault();
case MAC:
return DockerOnMac.getServerFactory();
default:
This is the Docker compose file I use to run Flink:
version: '3.4'
services:
jobmanager:
image: tenx/flink:1.8.1
command: 'jobmanager'
environment:
JOB_MANAGER_RPC_ADDRESS: 'jobmanager'
DOCKER_MAC_CONTAINER: 1
FLINK_JM_HEAP: 128
volumes:
- jobmanager-data:/data
- /var/run/docker.sock:/var/run/docker.sock
ports:
- target: 8081
published: 8081
protocol: tcp
mode: ingress
networks:
- flink
taskmanager:
image: tenx/flink:1.8.1
command: 'taskmanager'
environment:
JOB_MANAGER_RPC_ADDRESS: 'jobmanager'
DOCKER_MAC_CONTAINER: 1
FLINK_TM_HEAP: 1024
TASK_MANAGER_NUMBER_OF_TASK_SLOTS: 2
networks:
- flink
volumes:
- taskmanager-data:/data
- /var/run/docker.sock:/var/run/docker.sock
- /var/folders:/var/folders
volumes:
jobmanager-data:
taskmanager-data:
networks:
flink:
external: true
This is my Python pipeline:
import apache_beam as beam
import logging
class LogElements(beam.PTransform):
class _LoggingFn(beam.DoFn):
def __init__(self, prefix=''):
super(LogElements._LoggingFn, self).__init__()
self.prefix = prefix
def process(self, element, **kwargs):
logging.info(self.prefix + str(element))
yield element
def __init__(self, label=None, prefix=''):
super(LogElements, self).__init__(label)
self.prefix = prefix
def expand(self, input):
input | beam.ParDo(self._LoggingFn(self.prefix))
from apache_beam.options.pipeline_options import PipelineOptions
options = PipelineOptions(["--runner=PortableRunner", "--job_endpoint=localhost:8099"])
p = beam.Pipeline(options=options)
(p | beam.Create([1, 2, 3, 4, 5]) | LogElements())
p.run()
This is how I run the job service:
gradle :runners:flink:1.8:job-server:runShadow -PflinkMasterUrl=localhost:8081
Docker is automatically selected for executing the Python harness.
I can change the image used to run the Python container:
options = PipelineOptions(["--runner=PortableRunner", "--job_endpoint=localhost:8099", "--environment_type=DOCKER", "--environment_config=beam/python:latest"])
I can disable Docker and enable the ExternalEnvironmentFactory:
options = PipelineOptions(["--runner=PortableRunner", "--job_endpoint=localhost:8099", "--environment_type=EXTERNAL", "--environment_config=server"])
but I have to implement some callback answering on http://server:80.
Is there an implementation available?
To answer the question above, basically you want to add beam_worker_pool container along side with the flink task manager container in the same pods. So in the yaml file that you use to deploy flink task managers, add a new container:
- name: beam-worker-pool
image: apache/beam_python3.7_sdk:2.22.0
args: ["--worker_pool"]
ports:
- containerPort: 50000
name: pool
livenessProbe:
tcpSocket:
port: 50000
initialDelaySeconds: 30
periodSeconds: 60
volumeMounts:
- name: flink-config-volume
mountPath: /opt/flink/conf/
securityContext:
runAsUser: 9999
I found the solution. The new version of Apache Beam 2.16.0 provides an implementation to use in combination with environment type EXTERNAL. The implementation is based on worker_pool_main which has been created to support Kubernetes.
I know it is a bit outdated but there is a Flink operator for Kubernetes now.
Here are examples how to run Apache Beam with Flink using an operator:
https://github.com/GoogleCloudPlatform/flink-on-k8s-operator/tree/master/examples/beam
I have a rather simple test app:
import redis
import os
import logging
log = logging.getLogger()
log.setLevel(logging.DEBUG)
def test_redis(event, context):
redis_endpoint = None
if "REDIS" in os.environ:
redis_endpoint = os.environ["REDIS"]
log.debug("redis: " + redis_endpoint)
else:
log.debug("cannot read REDIS config environment variable")
return {
'statusCode': 500
}
redis_conn = None
try:
redis_conn = redis.StrictRedis(host=redis_endpoint, port=6379, db=0)
redis_conn.set("foo", "boo")
redis_conn.get("foo")
except:
log.debug("failed to connect to redis")
return {
'statusCode': 500
}
finally:
del redis_conn
return {
'statusCode': 200
}
which I have deployed as a HTTP endpoint with serverless
#
# For full config options, check the docs:
# docs.serverless.com
#
service: XXX
plugins:
- serverless-aws-documentation
- serverless-python-requirements
custom:
pythonRequirements:
dockerizePip: true
provider:
name: aws
stage: dev
region: eu-central-1
runtime: python3.6
environment:
# our cache
REDIS: xx-xx-redis-001.xxx.euc1.cache.amazonaws.com
functions:
hello:
handler: hello/hello_world.say_hello
events:
- http:
path: hello
method: get
# private: true # <-- Requires clients to add API keys values in the `x-api-key` header of their request
# authorizer: # <-- An AWS API Gateway custom authorizer function
testRedis:
handler: test_redis/test_redis.test_redis
events:
- http:
path: test-redis
method: get
When I trigger the endpoint via API Gateway, the lambda just times out after about 7 seconds.
The environmental variable is read properly, no error message displayed.
I suppose there's a problem connecting to the redis, but the tutorial are quite explicit - not sure what the problem could be.
The problem might need the need to set up a NAT, not sure how to accomplish this task with serverless
I ran into this issue as well. For me, there were a few problems that had to be ironed out
The lambda needs VPC permissions.
The ElastiCache security group needs an inbound rule from the Lambda security group that allows communication on the Redis port. I thought they could just be in the same security group.
And the real kicker: I had turned on encryption in-transit. This meant that I needed to pass redis.RedisClient(... ssl=True). The redis-py page mentions that ssl_cert_reqs needs to be set to None for use with ElastiCache, but that didn't seem to be true in my case. I did however need to pass ssl=True.
It makes sense that ssl=True needed to be set but the connection was just timing out so I went round and round trying to figure out what the problem with the permissions/VPC/SG setup was.
Try having the lambda in the same VPC and security group as your elastic cluster