We Keep Coding

webpack and django on Heroku: bundling before collectstatic

I am building a django+react app on Heroku, using django-npm which automatically installs all modules from package.json to node-modules dir and then copies everything to staticfiles/ during python manage.py collectstatic (which is triggered by Heroku during deploy).
However, for this configuration to work I need to pre-bundle my React app before deployment and put it into my static folder along with all the CSS, fonts, etc. to be picked up by collectstatic later.
But I don't want to pollute my git diffs with new bundle versions. So, is there a way to make webpack create a bundle during deployment?
I know there is a release command on Heroku where I can put my npm run build. But the problem is it only fires AFTER collectstatic, so my bundle will only be created in static/ folder after this folder is scanned, and won't get copied to staticfiles dir.

Update:
Using bin/pre_compile should no longer be necessary as of March 11, 2019. Heroku will now automatically call an app's build script (if defined in package.json) during Heroku's build process. Source: Heroku Changelog.
Original: You can accomplish this with the (undocumented) pre_compile hook exposed by the heroku/python buildpack.
Add an executable shell file named bin/pre_compile at the top level of your app and it will be called automatically as part of the build process.
cd my-django-app
mkdir bin
echo '#!/usr/bin/env bash' >> bin/pre_compile
echo 'npm run build' >> bin/pre_compile
chmod +x bin/pre_compile

Heroku looking for Procfile in wrong directory

I'm trying to work with Django on Heroku and I'm following this tutorial with its Django template https://devcenter.heroku.com/articles/django-app-configuration
https://devcenter.heroku.com/articles/deploying-python
But when i run 'heroku local web' it always look in the wrong directory. I've tried to move the project
(env) D:\Study\Workbench\heroku-testing\testing\env\codeShareApp>heroku local web
[WARN] No ENV file found
[WARN] ENOENT: no such file or directory, open 'C:\Users\William\Procfile'
[FAIL] No Procfile and no package.json file found in Current Directory - See
run_foreman.js --help
The project directory is as follows
env/
codeShareApp/
.idea/
codeShareApp/
.env
manage.py
Procfile
requirements.txt
runtime
include/
Lib/
Scripts/
tcl/
EDIT:
I tried to make a new project with directory like below, but still got the same error
codeShareApp/
codeShareApp/
env/
.env
manage.py
Procfile
requirements.txt
runtime

I encounter this problem also in a project; I solved this by Making sure that the Procfile is using the correct syntax, and that there are no extra tabs or newlines in the file.
especially whitespcae.

I dont know why, but I tried it on another machine and it works just fine. It seems that it has something to do with heroku cli 6.13 because the error only occurs after it updates from version 5.12 to 6.12

The documentation says:
The file must be placed in the root directory of your application. It will not function if placed in a subdirectory.
Try to do this and it should work. See the docs for more information.

Python Heroku allow pushed .exe to run - OSError: [Errno 13] Permission denied

I had to push an .exe file to heroku to be able to create invoice pdfs. It works localy without any problems but on heroku I get an error:
OSError: [Errno 13] Permission denied
Probably because I am not allowed to execute .exe files. So I need somehow to create a rule that this file is allowed to execute.
I pushed wkhtmltopdf.exe to heroku and I access this file in my method to create a pdf:
MYDIR = os.path.dirname(__file__)
path_wkthmltopdf = os.path.join(MYDIR + "/static/executables/", "wkhtmltopdf.exe")
config = pdfkit.configuration(wkhtmltopdf=path_wkthmltopdf)
Was not able to find a solution yet.
EDIT:
Tryed giving permission with chmod through heroku bash and also adding a linux executable but still the same error:
~/static/executables $ chmod a+x wkhtmltopdf-linux.exe
~ $ chmod a+x static/executables/wkhtmltopdf-linux.exe
Using sudo gave me:
bash: sudo: command not found

I'm not very familiar with heroku, but if you can somehow get access to terminal of environment of your application (for example ssh to your server), you need to change permissions of that file so it can be executed. To do that, you need to run in that terminal:
sudo chmod a+x /path/to/file/FILENAME
Also,i'm pretty sure your app on Heroku runs on Linux, specifically on Ubuntu, since it's the default (link)
It means there might be difficulties with running Windows executables.

Okay I managed to fix this with a buildpack. In addition wkhtmltopdf-pack must be installed and added to the requirements.txt.
Then you have to set a config var in heroku for the wkhtmltopdf executable which will be generated from the files provided in the buildpack. Do not search for an .exe file.
heroku config:set WKHTMLTOPDF_BINARY=wkhtmltopdf-pack
You can see all your config vars also in the heroku dashboard under settings, you can also create it there and not use the CLI.
Then you have to tell the pdfkit configuration where to find the WKHTMLTOPDF_BINARY:
In my config.py:
import subprocess
WKHTMLTOPDF_CMD = subprocess.Popen(
['which', os.environ.get('WKHTMLTOPDF_BINARY', 'wkhtmltopdf')], # Note we default to 'wkhtmltopdf' as the binary name
stdout=subprocess.PIPE).communicate()[0].strip()
For the pdfkit configuration:
config = pdfkit.configuration(wkhtmltopdf=app.config['WKHTMLTOPDF_CMD'])
Now you should be able to create the pdf, example:
the_pdf = pdfkit.from_string("something", False, configuration=config)
Credit to this tutorial:
https://artandlogic.com/2016/12/generating-pdfs-wkhtmltopdf-heroku/

django manage.py script messed up directory permissions

I have a script that checks a json file and downloads photos to a folder.
This bit checks if the directory exists and if not, creates it:
folder = media_root + 'photos/' + now.strftime('%Y') + '/' + now.strftime('%m') + '/' + now.strftime('%d')
if not os.path.isdir(folder):
os.makedirs(folder)
everything worked like it should, but now the folder can't be written through the django admin interface, giving the error OSError: [Errno 13] Permission denied: media/photos/2014/08/31/OneOf.jpg
The only way this works is if I set the permissions to be writable by everyone.
I checked and gunicorn is running with user nobody, and this never caused a problem before.
What could have caused this issue and how do I stop it from happening in the future?
EDIT:
I tried out the suggestions bellow but running the script as nobody only returned an error:
File "./manage.py", line 8, in <module>
from django.core.management import execute_from_command_line
ImportError: No module named django.core.management
I also checked and the other folders, created by the django-admin interface, are also owned by root.
RESOLUTION:
the os.makedirs commands changed the whole directory tree to be owned by root, which is why that even with the same permissions as other folders django couldn't write to that directory.
To keep it from happening again the solution is to restore ownership to that directory tree once the new folder is created.

The problem is that you are running the script as root.
The user that you run a program with is the same user that owns any files created by that program.
Once you run a program as root, any output will also be owned by the root user, which means the user that is running the django app will not have access to those files.
You should run the manage.py script as the same user that is running your web app (nobody) so that files created are then readable by the gunicorn process - and by extension, django.

to unspool your first issue:
created some data with bad perms
$ sudo chown -R www-data:www-data /full/path/to/media
running manage.py as the webserver user
$ sudo -u www-data ./manage.py <your-management-command>
Note: If sudo is not set up, you should set it up.

How to deploy structured Flask app on AWS elastic beanstalk

After successfully deploying a test app using the steps outlined here:
http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/create_deploy_Python_flask.html
I tried to deploy my actual flask application which has the following structure:
myApp/
runServer.py
requirements.txt
myApp/
__init__.py
helpers.py
clean.sh
static/
myApp.css
handlers/
__init__.py
views.py
templates/
layout.html
viewOne.html
viewTwo.html
Where views.py contains my url mappings.
I have tried initializing the eb instance in the root directory as well as within the myApp module and git aws.push but I get the following error on the AWS dashboard:
ERROR Your WSGIPath refers to a file that does not exist. and the application does not work (404 for any path).
How can I deploy the above Flask application to elastic beanstalk?

I encountered a similar problem deploying a Flask application to EB, with a similar directory structure, and had to do 2 things:
Update my manage.py to create an object of name application, not app
import os
from application import create_app, db
from flask.ext.script import Manager, Shell
application = create_app(os.getenv('FLASK_CONFIG') or 'default')
manager = Manager(application)
Create .ebextensions/myapp.config, and define the following block to point to manage.py
option_settings:
"aws:elasticbeanstalk:container:python":
WSGIPath: manage.py
"aws:elasticbeanstalk:container:python:staticfiles":
"/static/": "application/static/"
This let Elastic Beanstalk find the application callable correctly.
This is described briefly at the official docs, and is described in more detail in this blog post
EDIT - see project structure below
ProjectRoot
.ebextensions
application.config
application
main
forms.py
views.py
static
templates
tests
manage.py
requirements.txt
config.py
etc, etc

Add the following to .ebextensions/<env-name>.config:
option_settings:
"aws:elasticbeanstalk:container:python":
WSGIPath: myApp/handlers/views.py
Update:
If you don't have .ebextensions directory, please create one for the project. You can find more information of what can be done regarding the container configuration in Customizing and Configuring AWS Elastic Beanstalk Environments guide.

Your WSGIPath refers to a file that does not exist.
This error appears because Beanstalk, by default, looks for application.py. Check at Beanstalk web UI, Configuration > Software Configuration, WSGIPath is mapped to application.py
Update the WSGIPath as shown in the previous replies or rename to application.py file.

As of awsebcli 3.0, you can actually edit your configuration settings to represent your WSGI path via eb config. The config command will then pull (and open it in your default command line text editor, i.e nano) an editable config based on your current configuration settings. You'll then search for WSGI and update it's path that way. After saving the file and exiting, your WSGI path will be updated automatically.

WSGI configuration was painful for me. I did changed WSCI settings using eb config command but it did not work. Below you can fix this in 5 easy steps.
1- Moved app.py function to the root of the directory (where I runned eb init command.
2- Also renamed app.py as application.py and in that initilized application as application = Flask(__name__) not app = Flask(__name__)
3- eb deploy did not worked after this (in the same project) I tried to fix config by using eb config but it was too hairy to sort it out. Delete all .extensions, .gitignore etc from your project.
4- re initialize your project on EB with eb init and follow the prompts. when deployment is done, eb open would launch your webapp (hopefully!)

When I encountered this problem it was because I was using the GUI to upload a zip of my project files. Initially I was zipping the project level directory and uploading that zip to EB.
Then I switched to simply uploading a zip of the project files themselves-ie select all files and send those to a zip-and then the GUI upload utility was able to find my application.py file without a problem because the application.py file was not in a subfolder.

Well, In my case I followed the entire process and conventions but was still getting 404. The problem was my virtual environment. I was ignoring all environment config related folders/files in my .gitignore but not in .ebignore. After creating .ebignore and ignoring all the folders/files which were not related to project code, fixed the issue.