I am trying to use Chalice to fit into an pre-existing build folder structure, the python source file (app.py) is one level deeper than the vanilla Chalice project
├── .chalice
│ └── config.json
└── src
├── app.py
├── requirements.txt
├── requirements_test.txt
When I run chalice local in src folder it says it cannot find the config file:
Unable to load the project config file. Are you sure this is a chalice project?
When I run chalice local in the project folder, it says it cannot find the source file:
No module named 'app'
I had a look at the config file doesn't seem to have an option to specify where the source file is.
Since there were no responses on Stackoverflow, so I went to the project and opened an issue ticket for it, from the horses mouth, this is not possible at this stage.
Related
I am not very familiar with python, I only done automation with so I am a new with packages and everything.
I am creating an API with Flask, Gunicorn and Poetry.
I noticed that there is a version number inside the pyproject.toml and I would like to create a route /version which returns the version of my app.
My app structure look like this atm:
├── README.md
├── __init__.py
├── poetry.lock
├── pyproject.toml
├── tests
│ └── __init__.py
└── wsgi.py
Where wsgi.py is my main file which run the app.
I saw peoples using importlib but I didn't find how to make it work as it is used with:
__version__ = importlib.metadata.version("__package__")
But I have no clue what this package mean.
You should not use __package__, which is the name of the "import package" (or maybe import module, depending on where this line of code is located), and this is not what importlib.metadata.version() expects. This function expects the name of the distribution package (the thing that you pip-install), which is the one you write in pyproject.toml as name = "???".
You can extract version from pyproject.toml using toml package to read toml file and then display it in a webpage.
I have run into an incredibly frustrating problem where a COPY command in my Dockerfile successfully copies all of my apps files except one. I do not have a .dockerignore file so I know the file isn't being excluded from the build that way.
Note: I do have a .gitignore which is excluding file2.json that I do not want to version. But as you will see below, I'm building from my local folder, not remotely from a clone/checkout so I don't see why .gitignore would influence the docker build in this case.
Below is what my directory looks like:
$ tree -a -I .git app
app
├── app
│ ├── data
│ │ ├── file1.txt
│ │ ├── file2.json
│ │ ├── file3.txt
│ │ └── file4.yml
│ ├── somefile2.py
│ └── somefile.py
├── Dockerfile
├── .gitignore
├── requirements.txt
└── setup.py
And this is what is in my Dockerfile looks like
FROM ubuntu:18.04
FROM python:3.7
COPY . /app
RUN cp app/app/data/file2.json ~/.somenewhiddendirectory
RUN pip install app/.
ENTRYPOINT ["python", "app/app/somefile.py"]
For some reason, file2.json is not being copied during the COPY . /app call and I am getting an error when I try to cp it somewhere else. I have done a call like RUN ls app/app/data/ and all the files except file2.json are in there. I checked the files permissions and made sure they are the same as all the other files. I have tried doing a direct COPY of that file which results in an error since Docker says that file doesn't exist.
On my system, that file exists, I can see it with ls, and I can cat its contents. I have played around with ensuring the context within the image is squarely within the root directory of my app, and like I said, all files are correctly copied except that json file. I can't for the life of my figure out why Docker hates this file.
For some added context, I am using Google's cloud build to build the image and the yaml config looks like this:
steps:
- name: gcr.io/cloud-builders/docker
id: base-image-build
waitFor: [-]
args:
- build
- .
- -t
- us.gcr.io/${PROJECT_ID}/base/${BRANCH_NAME}:${SHORT_SHA}
images:
- us.gcr.io/${PROJECT_ID}/base/${BRANCH_NAME}:${SHORT_SHA}
and the command I am executing looks like this:
gcloud builds submit --config=cloudbuild.yaml . \
--substitutions=SHORT_SHA="$(git rev-parse --short HEAD)",BRANCH_NAME="$(git rev-parse --abbrev-ref HEAD)"
Disclaimer: I have never used Google's cloud build so my answer is only based on read theory.
I don't see why .gitignore would influence the docker build in this case
Indeed, docker build in itself does not care about your .gitignore file. But you are building through Google's cloud build and this is a totally different story.
Quoting the documentation for the source specification in gcloud build command:
[SOURCE]
The location of the source to build. The location can be a directory on a local disk or a gzipped archive file (.tar.gz) in Google Cloud Storage. If the source is a local directory, this command skips the files specified in the --ignore-file. If --ignore-file is not specified, use .gcloudignore file. If a .gcloudignore file is absent and a .gitignore file is present in the local source directory, gcloud will use a generated Git-compatible .gcloudignore file that respects your .gitignore files. The global .gitignore is not respected. For more information on .gcloudignore, see gcloud topic gcloudignore
So in your given case, your file will be ignored even for a build from your local directory. At this point I see 2 options to workaround this problem:
Remove the entry for your file in .gitignore so that the default gcloud mechanism does not ignore it during your build
Provide a --ignore-file or a default .gcloudignore which actually re-includes the local file that is ignored for versioning.
I would personally go for the second option with something super simple like the following .gcloudignore file (crafted from the relevant documentation)
.git
.gcloudignore
.gitignore
I am trying to run a code from this repo without success. There are no instructions on how to run it. I suspect I should run FactcheckingRANLP/Factchecking_clean/classification/lstm_train.py and then run .../lstm_test.py.
The problem is that this code uses import statements as a module, referencing to folders and files that are in different directories, for example, in lstm_train.py:
File "lstm_train.py", line 3, in <module>
from classification.lstm_utils import *
ModuleNotFoundError: No module named 'classification'
This is the tree structure of the classification folder:
.
├── classification
│ ├── __init__.py
│ ├── __init__.pyc
│ ├── lstm_repres.py
│ ├── lstm_test.py
│ ├── lstm_train.py
│ ├── lstm_train.pyc
│ ├── lstm_utils.py
│ ├── lstm_utils.pyc
│ ├── __pycache__
│ │ ├── __init__.cpython-36.pyc
│ │ ├── lstm_train.cpython-36.pyc
│ │ └── lstm_utils.cpython-36.pyc
│ └── svm_run.py
I would like to know how can I make python run lsmt_train/test.py files in such a manner that the import statements contained within them are compiled correctly. I prefer not to modify the code as this could possibly generate a lot of errors..
You could add the path pointing to the classification folder to your python path variable.
I suggest using the sys package:
import sys
sys.path.append('<<<PathToRepo>>>/FactcheckingRANLP/Factchecking_clean')
With the repo classification directory added to your python path, the import statements should work.
EDIT:
Correction; in the initial post I suggested adding the path to .../classification to your path variable, instead the parent folder .../Factchecking_clean is required as the file imports the module 'classification'.
Also, in Lucas Azevedo's answer, the parent directory path is added in the repository lstm_train file itself. While this definitely works, I still think it should be possible without editing the original repository.
I took a look at the repo in question and files like lstm_train.py are scripts which should be executed with the python working directory set as '<<<PathToRepo>>>/FactcheckingRANLP/Factchecking_clean'.
There are a few ways to do so:
You could open the project in a python IDE and configure your execution to use the directory .../Factchecking_clean as the working directory. In pycharm for example this could be done by importing the repo directory .../Factchecking_clean as a project. The following image shows how to set a working directory for execution in pycharm:
I think the repository was developed with this execution configuration set up.
Another possibility is to execute the python script from within another python file. This seems to be rather inconvenient to me, regardless you could do so by creating a separate python file with:
import sys
import os
sys.path.append('<<<PathToRepo>>>/FactcheckingRANLP/Factchecking_clean')
os.chdir('<<<PathToRepo>>>/FactcheckingRANLP/Factchecking_clean')
exec(open('./classification/lstm_train.py').read())
This adds the Factchecking_clean directory to the python path (using sys.path.append()) to be able to import stuff like classification.utils. The working directory is set by os.chdir() and finally exec(open('<<<filepath>>>')).read() executes the lstm_train file with the correct working directory and path variable set up.
Executing the new python file with the code above works for me (without editing the original repository).
However, as scripts like lstm_train.py actually are used to execute specific parts of the code provided in the rest of the repository modules, I think editing these files for experimental purposes is fine. In general, when working with repositories like this I recommend using an IDE (like pycharm) with a correctly set up configuration (method 1).
Ended up having to modify the repository's code, following the suggestion of Jeff.
import sys,os
parent_dir_path = os.path.abspath(__file__+"/../..")
sys.path.append(parent_dir_path)
adding the path until classification doesnt work, as the import is done mentioning the classification folder. The parent directory is the one that has to be added.
__file__ gives the current file name and os.path.abspath resolves the path navigation done with /../..
I don't think my question is FastAPI-specific, but more Python-virtualenv generic, but here's my example use case with FastAPI anyway.
I'm running FastAPI from a virtualenv set up by Pipenv and I'd like it to serve static files for development purposes only. I know this is solved with a proper production deployment and I will be using a proper web server for that, but my question below is about the development environment.
File structure looks like this:
├── frontend <-- frontend code dir
│ ├── dist <-- frontend build output (static files)
│ ├── public
│ └── [..]
├── Pipfile <-- contains 'myproject = {editable = true, path = "."}' as dep
├── Pipfile.lock
├── setup.cfg
├── setup.py
├── myproject <-- backend code dir
│ ├── web.py <-- fastapi file
│ └── [...]
└── [...]
myproject/web.py:
import fastapi
app = fastapi.FastAPI()
app.mount("/", fastapi.staticfiles.StaticFiles(
directory="/absolute/path/to/frontend/dist"),
name="static",
)
Running the FastAPI dev server with pipenv run uvicorn myproject.web:app --reload.
Which works now, but I'd like to avoid this absolute path in my public source. Any relative path does not work, as it appears to be relative to the installed path in the virtualenv created by Pipenv, which is not deterministic. The same goes for the __file__ path; it's inside a virtualenv.
I've also considered to add the statics as data files to the Python package and load it from the directory as provided by pkg_resources (e.g. pkg_resources.resource_filename('myproject', 'data/')), but I don't plan to ship the statics with the Python package in production so it feels hackish.
Would it be possible to somehow link the right directory here by a relative path or do you have another clever idea in order to share the project in a team of devs (or publicly on GitHub for that matter) a little bit better?
Because you want to be flexible from where it will run and the statics location is unrelated, any constant relative path can not work.
I would go for:
Discovering the statics by some heuristics like walking back to the some known "anchor" folder (root folder?)
I've done something like this once, and even cached the found absolute path result in some persistent temp file so this discovery will happen only once per machine (unless the location moved or the temp cached file was deleted).
Setting the static location as config/command-line/both parameter that your server launched with.
my flask app is a package named app located at /Users/gexinjie/Codes/MyProject/xinnjie_blog
the file tree is like this
xinnjie_blog
├── app
| ├── __init__.py
│ ├── config.py
│ ├── exceptions.py
│ ├── model.py
│ ├── model_sqlalchemy.py
│ ├── static
│ ├── templates
│ ├── util.py
│ └── views
├── manage.py
I export it as PATHONPATH, so manage.py can import app
echo $PATHONPATH
/Users/gexinjie/Codes/MyProject/xinnjie_blog
and export FLASK_APP
echo $FLASK_APP
manage.py
current dir is /Users/gexinjie/Codes/MyProject/xinnjie_blog
pwd
/Users/gexinjie/Codes/MyProject/xinnjie_blog
here is the manage.py
import click
from app import create_app
app = create_app('development')
#app.cli.command()
def initdb():
click.echo('Init the db...')
here is app.__init__.py
from flask import Flask
from .model_sqlalchemy import db
def create_app(config_name='default'):
app = Flask(__name__)
... # init
return app
but then if I execute flask initdb, I get this error:
Usage: flask [OPTIONS] COMMAND [ARGS]...
Error: No such command "initdb".
and if I execute flask run, I get
Usage: flask run [OPTIONS]
Error: The file/path provided (manage) does not appear to exist. Please verify the path is correct. If app is not on PYTHONPATH, ensure the extension is .py
why manage.py is not found? And how can I fix it.
(actually it worked well when manage.py have flask app in itself )
# manage.py
# this work well
app = Flask(__name__) # not app = create_app('development')
Thank you
Thank to #Adam, This problem was solved after I uninstall Anaconda.
Because all the time I tested manage.py on Pycharm command tool, and that flask was installed by Anaconda(python version 3.6), it may lack some extensions(usually I use python3.5 on terminal).So I think the problem occur during importing.
The flask command tool complain about 'can't locate the app', but the real problem is import error.So that is so confusing.
"universal" solution:
So when if you come to this kind of problem like I do, I suggest you first check the location of your app(try both relative path and absolute path), relative path may cause the locate problem when you are not at the right working directory.So absolute path is recommend.
If every thing about path went well, then make sure that all the packages required by your app is installed and can be imported. if you are using some type of virtual environment or something like that (In my case, I use the other version of python lacking some flask extensions), it may be the import error makes flask complain.
Hope this can help you.