We Keep Coding

Kubernetes : No difference in performances between 1 Pod and 3 Pods?

I have a very simple Kubernetes Cluster that uses GKE (Google Cloud Platform) with 1 node (4 vCPU / 16 Go RAM)
This morning I was load testing an API (written in Python) on this cluster with Locust. I only have one endpoint on my API, so I prepared a locust file to run differents configurations on that endpoint (with random parameters etc).
I first ran my Locust test with 10 users, 1 user generated every seconds (until limit reached), and the request wait time is between 1 and 2.5 seconds. I tested it first on just 1 pod and then I ran the exact same test on 3 pods. I noticed almost no improvement with 3 pods and the response times were just more spreaded.
Here are the results for 1 Pod :
And the charts:
And here are the results for 3 Pods
And the charts:
I am fairly new to K8s and I don't really understand everything, but my initial though was that increasing the number of running pods should improve the preformance right ? I should be able to handle 3 times more requests in the same time, but the results are far from that excepted conclusion.
And here is the yaml file defining the deployment on the K8s cluster (note: I set myself the replicas to 1 or 3 because for some reasons the autoscaler don't work) :
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: app_name
name: app_name
namespace: default
spec:
replicas: 3 or 1
selector:
matchLabels:
app: app_name
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
labels:
app: app_name
spec:
containers:
- image: >-
gcr.io/path_to_image
imagePullPolicy: IfNotPresent
name: app_name-v2-sha256-1
---
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
labels:
app: app_name
name: app_name-hpa-cbrt
namespace: default
spec:
maxReplicas: 3
metrics:
- resource:
name: cpu
targetAverageUtilization: 80
type: Resource
minReplicas: 1
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: app_name
Additional info : The API calls a MySQL database and perform a request that get approx. 70k rows and do some stuff on those results. It only returns a json with few rows at the end.
Thanks in advance for any help you could provide !
Edit : Here is the definition of the load balancer, it's GCP generated and untouched
apiVersion: v1
kind: Service
metadata:
annotations:
cloud.google.com/neg: '{"ingress":true}'
creationTimestamp: "2021-05-11T10:03:16Z"
finalizers:
- service.kubernetes.io/load-balancer-cleanup
labels:
app: app_name
app.kubernetes.io/managed-by: gcp-cloud-build-deploy
app.kubernetes.io/name: app_name
app.kubernetes.io/version: b8a19c9rth54erthe4rth8459633451fe8e73e038
name: app_service
namespace: default
resourceVersion: "87954231"
selfLink: /api/v1/namespaces/default/services/app_service
uid: service_uid
spec:
clusterIP: 661.661.661.661
externalTrafficPolicy: Cluster
ports:
- nodePort: 30129
port: 8080
protocol: TCP
targetPort: 8080
selector:
app: app_name
sessionAffinity: None
type: LoadBalancer
status:
loadBalancer:
ingress:
- ip: 666.666.666.666

How to read and process high priority messages in kafka consumer?

Is there any method to process messages with high priority first?
I tried creating three topics 'high', 'medium', and 'low', and subscribed to all three topics with one consumer and if there is an unprocessed message in the 'high' topic it will pause the other two. Is there any better way for implementing message priority?
I tried using the logic given below.
topics = ['high', 'medium', 'low']
consumer.subscribe(topics)
high_topic_partition = TopicPartition(priority['high'], 0)
medium_topic_partition = TopicPartition(priority['medium'], 0)
low_topic_partition = TopicPartition(priority['low'], 0)
while True:
messages = consumer.poll()
high_priotity_unprocessed_msg = consumer.end_offsets([high_topic_partition])[high_topic_partition] - consumer.position(high_topic_partition)
medium_priotity_unprocessed_msg = consumer.end_offsets([medium_topic_partition])[medium_topic_partition] - consumer.position(medium_topic_partition)
low_priotity_unprocessed_msg = consumer.end_offsets([low_topic_partition])[low_topic_partition] - consumer.position(low_topic_partition)
if high_priotity_unprocessed_msg >0:
consumer.pause(medium_topic_partition)
consumer.pause(low_topic_partition)
else:
consumer.resume(medium_topic_partition)
if medium_priotity_unprocessed_msg >0:
consumer.pause(low_topic_partition)
else:
consumer.resume(low_topic_partition)
if messages:
process(messages)

One option that you may evaluate is basically just having more parallelism on higher priority messages...
For example:
Topic1 (Priority Low): 1 partitions
Topic2 (Priority medium): 5 partitions
Topic3 (Priority High): 20 partitions
And then basically have:
1 consumers to get the data from topic1
5 consumers from topic2
20 consumers from topic3
👆Now, I would suggest you the easiest way to do this is basically write the code once... but externalize the configuration of the "topic name"... and then just scale it up (of course using containers)... Please refer to this reading:
https://12factor.net/config
For example, the code could be as simple as:
SuperAwesomeAppBinaryCode:
topic = %MY_TOPIC_NAME_INJECTED_BY_ENV_VAR%
consumer.subscribe(topic)
while True:
messages = consumer.poll()
if messages:
process(messages)
Now, if we have that code deployed on, let's say K8s, you could have 3 different deployments, running the same code, but injecting the right topic for each case, for example:
Low Priority Messages
apiVersion: apps/v1
kind: Deployment
metadata:
name: LowPriorityProcessor
labels:
app: LowPriorityProcessor
spec:
replicas: 1
selector:
matchLabels:
app: LowPriorityProcessor
template:
metadata:
labels:
app: LowPriorityProcessor
spec:
containers:
- name: LowPriorityProcessor
image: SuperAwesomeAppBinaryCode:1.0.0
env:
- name: MY_TOPIC_NAME_INJECTED_BY_ENV_VAR
value: topic1
ports:
- containerPort: 80
Medium Priority Messages
apiVersion: apps/v1
kind: Deployment
metadata:
name: MediumPriorityProcessor
labels:
app: MediumPriorityProcessor
spec:
replicas: 5
selector:
matchLabels:
app: MediumPriorityProcessor
template:
metadata:
labels:
app: MediumPriorityProcessor
spec:
containers:
- name: MediumPriorityProcessor
image: SuperAwesomeAppBinaryCode:1.0.0
env:
- name: MY_TOPIC_NAME_INJECTED_BY_ENV_VAR
value: topic2
ports:
- containerPort: 80
High Priority Messages
apiVersion: apps/v1
kind: Deployment
metadata:
name: HighPriorityProcessor
labels:
app: HighPriorityProcessor
spec:
replicas: 20
selector:
matchLabels:
app: HighPriorityProcessor
template:
metadata:
labels:
app: HighPriorityProcessor
spec:
containers:
- name: HighPriorityProcessor
image: SuperAwesomeAppBinaryCode:1.0.0
env:
- name: MY_TOPIC_NAME_INJECTED_BY_ENV_VAR
value: topic3
ports:
- containerPort: 80
And then just let the parallelism do its magic 😉
If you check carefully the only thing that changes from one "k8s deployment" to another is the topic and the number of replicas.
Notes:
You can achieve this without K8s.... using Docker Swarm or even just docker-compose or running manually the instances 🤷‍♂️, but why would you like to reinvent the wheel, but for sure in some edge cases, there is no much option...
A nice reading about this topic can be found here

How to set Browser Properties of Selenium Nodes?

I want to manipulate the browser properties of the Selenium Nodes. I use the selenium/node-chrome docker image for nodes (but potentially I want to do this on Firefox nodes, too) inside Selenium Grid on an Kubernetes (Minikube) Cluster.
The properties I want to manipulate are
navigator.webdriver
screen.width
screen.height
navigator.deviceMemory
but I am searching a way that works for most browser properties, I may later find other properties I want to change. The browser properties do not have to change during a scan, and they do not need to be set inside my python code, it is all fine for me to change them in the nodes configuration, docker image or spoof them elsewhere.
I wrote a python script to read out the current values of those properties
from selenium import webdriver
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
driver = webdriver.Remote(
command_executor='http://selenium-hub:4444/wd/hub',
desired_capabilities=getattr(DesiredCapabilities, "CHROME")
)
print("navigator.webdriver: "+str(driver.execute_script("return navigator.webdriver")))
print("screen.width: "+str(driver.execute_script("return screen.width")))
print("screen.height: "+str(driver.execute_script("return screen.height")))
print("navigator.deviceMemory: "+str(driver.execute_script("return navigator.deviceMemory")))
driver.quit()
How can I change these properties?
EDIT: The K8s deployment of the node looks like this:
kind: Deployment
metadata:
name: selenium-node-chrome
labels:
app: selenium-node-chrome
spec:
replicas: 2
selector:
matchLabels:
app: selenium-node-chrome
template:
metadata:
labels:
app: selenium-node-chrome
spec:
containers:
- name: selenium-node-chrome
image: selenium/node-chrome
ports:
- containerPort: 5555
volumeMounts:
- name: dshm
mountPath: /dev/shm
- name: config
mountPath: /opt/selenium/config.json
subPath: config.json
env:
- name: HUB_HOST
value: "selenium-hub"
- name: HUB_PORT
value: "4444"
resources:
limits:
memory: "1000Mi"
cpu: ".5"
volumes:
- name: dshm
emptyDir:
medium: Memory
- name: config
configMap:
name: selenium-node-chrome
---
apiVersion: v1
kind: ConfigMap
metadata:
name: selenium-node-chrome
data:
config.json: |
{
"capabilities": [
{
"version": "81.0.4044.92",
"browserName": "chrome",
"maxInstances": 1,
"seleniumProtocol": "WebDriver",
"applicationName": ""
}
],
"proxy": "org.openqa.grid.selenium.proxy.DefaultRemoteProxy",
"maxSession": 1,
"host": "0.0.0.0",
"port": 5555,
"register": true,
"registerCycle": 5000,
"nodePolling": 5000,
"unregisterIfStillDownAfter": 60000,
"downPollingLimit": 2,
"debug": false
}

uWSGI configuration in Kubernetes

I am running my backend using Python and Django with uWSGI. We recently migrated it to Kubernetes (GKE) and our pods are consuming a LOT of memory and the rest of the cluster is starving for resources. We think that this might be related to the uWSGI configuration.
This is our yaml for the pods:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: my-pod
namespace: my-namespace
spec:
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 10
maxUnavailable: 10
selector:
matchLabels:
app: my-pod
template:
metadata:
labels:
app: my-pod
spec:
containers:
- name: web
image: my-img:{{VERSION}}
imagePullPolicy: IfNotPresent
ports:
- containerPort: 8000
protocol: TCP
command: ["uwsgi", "--http", ":8000", "--wsgi-file", "onyo/wsgi.py", "--workers", "5", "--max-requests", "10", "--master", "--vacuum", "--enable-threads"]
resources:
requests:
memory: "300Mi"
cpu: 150m
limits:
memory: "2Gi"
cpu: 1
livenessProbe:
httpGet:
httpHeaders:
- name: Accept
value: application/json
path: "/healthcheck"
port: 8000
initialDelaySeconds: 15
timeoutSeconds: 5
periodSeconds: 30
readinessProbe:
httpGet:
httpHeaders:
- name: Accept
value: application/json
path: "/healthcheck"
port: 8000
initialDelaySeconds: 15
timeoutSeconds: 5
periodSeconds: 30
envFrom:
- configMapRef:
name: configmap
- secretRef:
name: secrets
volumeMounts:
- name: service-account-storage-credentials-volume
mountPath: /credentials
readOnly: true
- name: csql-proxy
image: gcr.io/cloudsql-docker/gce-proxy:1.11
command: ["/cloud_sql_proxy",
"-instances=my-project:region:backend=tcp:1234",
"-credential_file=/secrets/credentials.json"]
ports:
- containerPort: 1234
name: sql
securityContext:
runAsUser: 2 # non-root user
allowPrivilegeEscalation: false
volumeMounts:
- name: credentials
mountPath: /secrets/sql
readOnly: true
volumes:
- name: credentials
secret:
secretName: credentials
- name: volume
secret:
secretName: production
items:
- key: APPLICATION_CREDENTIALS_CONTENT
path: key.json
We are using the same uWSGI configuration that we had before the migration (when the backend was being executed in a VM).
Is there a best practice config for running uWSGI in K8s? Or maybe something that I am doing wrong in this particular config?

You activated 5 workers in uwsgi, that could mean 5 times the need of memory if your application is using lazy-loading techniques (my advice: load everything at startup and trust pre-fork check this). However, you could try reducing number of workers and instead raising number of threads.
Also, you should drop max-requests, this makes your app reload every 10 requests, that's non-sense in a production environment (doc). If you have troubles with memory leaks, use reload-on-rss instead.
I would do something like this, maybe less or more threads per worker depending on how your app uses it (adjust according to cpu usage/availability per pod in production):
command: ["uwsgi", "--http", ":8000", "--wsgi-file", "onyo/wsgi.py", "--workers", "2", "--threads", "10", "--master", "--vacuum", "--enable-threads"]
ps: as zerg said in comment you should of course ensure your app is not running DEBUG mode, together with low logging output.

How can I create kubernetes yaml file using pure python

Actually, i have kubernetes cluster set up. I want to generate yaml config file dynamically based on a template using python.
template.yaml
apiVersion: v1
kind: pod
metadata:
name: $name
spec:
replicas: $replicas
template:
metadata:
labels:
run: $name
spec:
containers:
- name: $name
image: $image
ports:
- containerPort: 80
Placeholders name, replicas and image are the input of my python method.
Any help will be appreciated.

If you want a way to do it using pure python, with no libraries, here's one using multiline strings and format:
def writeConfig(**kwargs):
template = """
apiVersion: v1
kind: pod
metadata:
name: {name}
spec:
replicas: {replicas}
template:
metadata:
labels:
run: {name}
spec:
containers:
- name: {name}
image: {image}
ports:
- containerPort: 80"""
with open('somefile.yaml', 'w') as yfile:
yfile.write(template.format(**kwargs))
# usage:
writeConfig(name="someName", image="myImg", replicas="many")

If you want to work only with templates, pure python and if your variables are already checked (safe) than you can use the format method of strings.
Here is a example:
# load your template from somewhere
template = """apiVersion: v1
kind: pod
metadata:
name: {name}
spec:
replicas: {replicas}
template:
metadata:
labels:
run: {name}
spec:
containers:
- name: {name}
image: {image}
ports:
- containerPort: 80
"""
# insert your values
specific_yaml = template.format(name="test_name", image="test.img", replicas="False")
print(specific_yaml)