I'm working on an app for my homelab, where I have an Intel NUC I'm using for some web scraping tasks. The NUC is accessible to my home network via 192.xxx.x.xx.
On the NUC I've set up nginx to proxy incoming http request to a docker container. In that container I've got a basic fastapi app to handle the request.
app.main.py
import os
from pathlib import Path
from fastapi import FastAPI
app = FastAPI()
cron_path = Path(os.getcwd(), "app", "cron.log")
#app.get("/cron")
def cron():
with cron_path.open("rt") as cron:
return {"cron_state": cron.read().split("\n")}
app.cronjob.py
import os
from pathlib import Path
from datetime import datetime
cron_path = Path(os.getcwd(), "app", "cron.log")
def append_time():
with cron_path.open("rt") as filein:
text = filein.read()
text += f"\n{datetime.utcnow().strftime('%Y-%m%dT%H:%M:%SZ')}"
with cron_path.open("wt") as fileout:
fileout.write(text)
if __name__ == "__main__":
append_time()
cron-job
* * * * * python3 /code/app/cronjob.py
# An empty line is required at the end of this file for a valid cron file.
Dockerfile
FROM python:3.10-slim-buster
#
WORKDIR /code
COPY ./cron-job /etc/cron.d/cron-job
COPY ./app /code/app
COPY ./requirements.txt /code/requirements.txt
# Give execution rights on the cron job
RUN chmod 0644 /etc/cron.d/cron-job
#Install Cron
RUN apt-get update
RUN apt-get -y install cron
# Apply cron job
RUN crontab /etc/cron.d/cron-job
#
RUN pip install --no-cache-dir --upgrade -r /code/requirements.txt
#
EXPOSE 8080
CMD crontab ; uvicorn app.main:app --host 0.0.0.0 --port 8080
I can access the the app without issues, but I can't seem to get the cron job to run while fastapi is running. Is what I'm attemping to do something better suited for pure python solution like from fastapi_utils.tasks import repeat_every or is there something I'm missing.
Related
When running my flask app, which uses Python's subprocess to use scrapy within a flask app as specified here (How to integrate Flask & Scrapy?), from a Docker Container and calling the appropriate endpoints specified in my flask app, I receive the error message: ERR_EMPTY_RESPONSE. Executing the flask app outside of my docker container (python app.py, where app.py has my flask code), everything works as intended and my spiders are called using subprocess within the flask app.
Instead of using flask & subprocess to call my spiders within a web app, I tried using twisted & twisted-klein python libraries, with the same result when called from a docker Container. I have also created a new, clean scrapy project, meaning no specific code of my own, just the standard scrapy code and project structure upon creation. This resulted in the same error. I am not quite certain whether my approach is anti-pattern, since flask and scrapy are in bundled into run image, resulting in one container for two purposes.
Here is my server.py code. When executing outside a container (using python interpreter) everything works as intended.
When running it from a container, then I receive the error message (ERR_EMPTY_RESPONSE).
# server.py
import subprocess
from flask import Flask
from medien_crawler.spiders.firstclassspider import FirstClassSpider
app = Flask(__name__)
#app.route("/")
def return_hello():
return "Hello!"
#app.route("/firstclass")
def return_firstclass_comments():
spider_name = "firstclass"
response = subprocess.call(['scrapy', 'crawl', spider_name, '-a', 'start_url=https://someurl.com'])
return "OK!"
if __name__ == "__main__":
app.run(debug=True)
FROM python:3
WORKDIR /usr/src/app
COPY requirements.txt ./
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
EXPOSE 5000
CMD [ "python", "./server.py" ]
Finally I run docker run -p 5000:5000 . It does not work. Any ideas?
Please try it.
.Dockerfile
FROM python:3.6
RUN apt-get update && apt-get install -y wget
WORKDIR /usr/src/app
ADD . /usr/src/app
RUN pip install --no-cache-dir -r requirements.txt
EXPOSE 5000
CMD [ "python", "./server.py" ]
I have data currently stored in an s3 bucket that I don't want to be public. I'm attempting to pandas.read_csv("s3_file_path") to load a pandas DataFrame in a script that runs in a docker container. I get a permission denied error. How do I pull the dataframe while giving aws the permissions it wants?
The end goal of this project is the create a RestApi that will process and use a statistical model on some data and return the results. I am also open to a completely different approach that avoids this problem altogether.
As I am the only user of this aws account, just to get it working, I tried putting my aws keys directly in the Dockerfile and running 'aws configure' to essentially copy the exact process I would use if I was doing this without docker. Obviously that is insecure, but I was simply trying to get it to work before I started implementing anything more complex. Unfortunately, it didn't.
Current Dockerfile
FROM python:3
WORKDIR /usr/src/app
COPY requirements.txt ./
RUN pip install -r requirements.txt
ARG AWS_KEY=My_Actual_Public_Key_In_Plain_Text
ARG AWS_SECRET_KEY=My_Actual_Secret_Key_In_Plain_Text
ARG AWS_REGION='us-east-1'
RUN aws configure set aws_access_key_id $AWS_KEY \
&& aws configure set aws_secret_access_key $AWS_SECRET_KEY \
&& aws configure set default.region $AWS_REGION
COPY . .
CMD [ "python", "./run.py" ]
run.py
from module import app
app.run(host="0.0.0.0", port = 80, debug = True)
from init.py in module
from flask import Flask
import pandas as pd
import numpy as np
file_name = "s3://foo/bar.csv"
df = pd.read_csv(file_name)
#app.route("/")
def index():
return("Hello World!")
The error I get is:
PermissionError: Access Denied
Assuming that you have s3fs installed as per the doc. Adding a print for debug:
from flask import Flask
import pandas as pd
import numpy as np
file_name = "s3://foo/bar.csv"
df = pd.read_csv(file_name)
print(df)
app = Flask(__name__)
#app.route("/")
def index():
return("Hello World!")
Ref. the Dockerfile reference: "The ARG instruction defines a variable that users can pass at build-time" - in this case you need the credentials to be available during the runtime, not during the build, you can pass them in the containers runtime environment for example:
FROM python:3
WORKDIR /usr/src/app
COPY requirements.txt ./
RUN pip install -r requirements.txt
ENV AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID}
ENV AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}
ENV AWS_REGION='us-east-1'
COPY . .
ENTRYPOINT ["flask"]
CMD ["run"]
Build the image: docker build --rm -t so:57700120 .
Run the container: docker run --rm -it -p 5000:5000 -e AWS_ACCESS_KEY_ID=... -e AWS_SECRET_ACCESS_KEY=... so:57700120
Note: boto does not recognize AWS_KEY / AWS_SECRET_KEY ref. the doc for additional information concerning the environment variables which are recognized.
I have the following simple Cloud Run service from the Python quickstart:
app.py:
import os
from flask import Flask
app = Flask(__name__)
#app.route('/')
def hello_world():
return 'Hello World!\n'
if __name__ == "__main__":
app.run(debug=True,host='0.0.0.0',port=int(os.environ.get('PORT', 8080)))
Dockerfile:
FROM python:3.7
ENV APP_HOME /app
WORKDIR $APP_HOME
COPY . .
RUN pip install Flask
CMD python app.py
How can I run & test this locally?
Similar to any other Dockerfile, you can use this two step command to build your image, and then run it locally:
$ docker build -t your_service .
$ docker run --rm -p 8080:8080 -e PORT=8080 your_service
It's important to specify the PORT environment variable here, and ensure that your app uses it appropriately.
Afterwards, your service will be running on http://localhost:8080
I'm building a simple app using: Dockerfile, app.py and requirements.txt. When the Dockerfile builds I get the error: "No such file or directory". However, when I change the ADD to COPY in the Dockerfile it works. Do you know why this is?
I'm using the tutorial: https://docs.docker.com/get-started/part2/#define-a-container-with-a-dockerfile
App.py
from flask import Flask
from redis import Redis, RedisError
import os
import socket
# Connect to Redis
redis = Redis(host="redis", db=0, socket_connect_timeout=2, socket_timeout=2)
app = Flask(__name__)
#app.route("/")
def hello():
try:
visits = redis.incr("counter")
except RedisError:
visits = "<i>cannot connect to Redis, counter disabled</i>"
html = "<h3>Hello {name}!</h3>" \
"<b>Hostname:</b> {hostname}<br/>" \
"<b>Visits:</b> {visits}"
return html.format(name=os.getenv("NAME", "world"), hostname=socket.gethostname(), visits=visits)
if __name__ == "__main__":
app.run(host='0.0.0.0', port=80)
requirements.txt
Flask
Redis
Dockerfile
# Use an official Python runtime as a parent image
FROM python:2.7-slim
# Set the working directory to /app
WORKDIR /app
# Copy the current directory contents into the container at /app
ADD . /app
# Install any needed packages specified in requirements.txt
RUN pip install -r requirements.txt
# Make port 80 available to the world outside this container
EXPOSE 80
# Define environment variable
ENV NAME World
# Run app.py when the container launches
CMD ["python", "app.py"]
In the first run, your working directory is /app inside container, and you copy contents to /tmp. To correct this behavior, you should be copying contents to /app and it will work fine.
Second one, where you are using add is correct since you are adding contents to /app., and not /tmp
I have a swagger server api in python that I can run on my pc and easily access to the user interface via web. I'm now trying to run this API into a docker container and place it into a remote server. After the doing the 'docker run' command int the remote server all seems to be working fine but when I try to connect I got a ERR_CONNECTION_REFUSED response. The funny thing is that if I enter into the container the swagger server is working and answer my requests.
Here is my Dockerfile:
FROM python:3
MAINTAINER Me
ADD . /myprojectdir
WORKDIR /myprojectdir
RUN pip install -r requirements.txt
RUN ["/bin/bash", "-c", "chmod 777 {start.sh,stop.sh,restart.sh,test.sh}"]
Here are my commands to build/run:
sudo docker build -t mycontainer .
sudo docker run -d -p 33788:80 mycontainer ./start.sh
Here is the start.sh script:
#!/bin/bash
echo $'\r' >> log/server_log_`date +%Y%m`.dat
python3 -m swagger_server >> log/server_log_`date +%Y%m`.dat 2>&1
And the main.py of the swagger server:
#!/usr/bin/env python3
import connexion
from .encoder import JSONEncoder
if __name__ == '__main__':
app = connexion.App(__name__, specification_dir='./swagger/')
app.app.json_encoder = JSONEncoder
app.add_api('swagger.yaml', arguments={'title': 'A title'})
app.run(port=80, threaded=True, debug=False)
Does anyone know why I can't acces to 'myremoteserver:33788/myservice/ui' and what to change for solving it.
Thanks in advance
I finally managed to find out the solution. It's needed to tell the flask server of connexion to run on 0.0.0.0 so that not only local connections are allowed and to change in the swagger.yaml the url with the name of the server where the docker container is located
app.run(port=80, threaded=True, debug=False, host='0.0.0.0')