I am in the process of migrating a Python flask web app from Heroku to Railway. The app uses a set of Flask CLI commands to initialize and update a Postgres database schema.
In the development environment, for example, I would run "flask db migrate" in order to create a db model to be copied to other locations.
In order to copy the model into staging or production environments, I would then need to run the command "flask db upgrade" on the staging/production app. Heroku's CLI "run" command allows you to do this without SSH by running the following command:
heroku run flask db upgrade --app NAME-OF-STAGING/PRODUCTION-APP
As far as I know, Railway's CLI run command does not allow you to select an online environment on which to run the command in the same way. Nor do they allow SSH access.
Is there any way to run a CLI command or Python file on a Railway live app, so that I can run Flask-migrate's "db upgrade" and get my Postgres DB working?
If you're still looking for an answer, here is what I did: I didn't use flask migrate, but instead I downloaded my database as a CSV file and created my own migration script. The script reads through the CSV file row by row and adds the data to the new database (you need to delete the old one). I called the script from my main.py, and then I removed the script after the database was migrated. This is definitely not the best way to do this, especially if your DB has a lot of data in it, but it's a good workaround.
Related
I have deployed my django website to Heroku but since the website fields are dependent upon a database that is on my local machine. I've tried using Postgres but the database on Heroku doesn't populate with the data I need it to before the app runs. Has anyone experienced this? Do I need to use an exteranl database on AWS or something (in which case, what is the best way to do this?)
Use load data and dump data: https://docs.djangoproject.com/en/3.2/ref/django-admin/#loaddata
First dump the data on your local machine
python manage.py dumpdata ..other_options > data.json
add it to git and push to herkou
git add data.json
git commit -m "Added data"
git push heroku master
now on Heroku can use loaddata to load your data to database
heroku run python manage.py loaddata data.json
And your done.
I have deployed a Django project on Heroku where I had data in models even after the deployment the data remained in the models. I worked on SQLite in production. Heroku works with Postgres but on deploying also my data remained safe. I just used git push heroku main
I have built an an app with Flask, without using SQLAlchemy (so that I don't get an ORM, which for me give less transparency, and understanding), so just plain SQL with SQLite.
Now that I want to publish it online (now I want to test it with a free Heroku), all the tutorials I find seem to use SQLAlchemy to be able to use PostgreSQL instead of SQLite.
I am also using a schema.sql file containing all the code to create the schema (CREATE TABLE user, etc.) which is launched with the command flask init-db.
I am also a bit worried that I will need to change way more code than simply the database link.
Using the db.py code that is in the Flask tutorial, I changed the connect URL to use the one which Heroku gave me.
def get_db():
if 'db' not in g:
if os.environ.get('DATABASE_URL') is None:
g.db = sqlite3.connect(
current_app.config['DATABASE'],
detect_types=sqlite3.PARSE_DECLTYPES
)
g.db.row_factory = sqlite3.Row
else:
DATABASE_URL = os.environ['DATABASE_URL']
g.db = psycopg2.connect(DATABASE_URL, sslmode='require')
return g.db
When pushing the code to Heroku with the initial commit, on the URL of my website I get a "Application error" message. And logs say:
2019-08-21T15:12:50.333541+00:00 app[init.1]: Usage: flask [OPTIONS] COMMAND [ARGS]...
2019-08-21T15:12:50.333564+00:00 app[init.1]: Try "flask --help" for help.
2019-08-21T15:12:50.333567+00:00 app[init.1]:
2019-08-21T15:12:50.333642+00:00 app[init.1]: Error: No such command "init_db".
2019-08-21T15:12:50.434989+00:00 heroku[init.1]: Process exited with status 2
2019-08-21T15:12:54.909027+00:00 heroku[init.1]: Starting process with command `FLASK_APP=myapp flask init_db`
2019-08-21T15:12:55.504819+00:00 heroku[init.1]: State changed from starting to up
2019-08-21T15:12:57.997833+00:00 heroku[init.1]: State changed from up to crashed
2019-08-21T15:12:57.888599+00:00 app[init.1]: Usage: flask [OPTIONS] COMMAND [ARGS]...
2019-08-21T15:12:57.888627+00:00 app[init.1]: Try "flask --help" for help.
2019-08-21T15:12:57.888630+00:00 app[init.1]:
2019-08-21T15:12:57.888639+00:00 app[init.1]: Error: No such command "init_db".
2019-08-21T15:12:57.978596+00:00 heroku[init.1]: Process exited with status 2
So can I easily deploy a Flask app to Heroku without using SQLAlchemy, or at least not an ORM, so that I can use my plain SQL commands?
You're conflating SQLAlchemy and PostgreSQL.
SQLAlchemy provides
a unified, relatively low-level interface for working with many different database backends via SQLAlchemy Core, and
an optional ORM
It can be used with both PostgreSQL and SQLite (and others).
PostgreSQL is a relational database management system, much like SQLite is.
You can use Postgres directly, e.g. with psycopg2, or with something like SQLAlchemy.
You can't use SQLite on Heroku, with or without SQLAlchemy. SQLite saves its data in a file on the filesystem, and this doesn't play nicely with Heroku's ephemeral filesystem.
To use Postgres without SQLAlchemy, simply
provision the add-on (this should happen automatically if you add psycopg2 to your dependencies),
and connect to it via the connection string given in the DATABASE_URL environment variable.
I am also using a schema.sql file containing all the code to create the schema (CREATE TABLE user, etc.) which is launched with the command flask init-db.
As long as the schema.sql is part of your application you should be able to do
heroku run flask init-db
If that doesn't work you should be able to do something like
cat schema.sql | heroku pg:psql
though personally I'd dig into why heroku run isn't doing what you want, first.
Finally, I strongly recommend switching to PostgreSQL in development as well.
SQLite and PostgreSQL are different products and don't always behave the same way. Some measure of uniformity can be achieved through an ORM or SQLAlchemy's Core (though even then I recommend using the same database everywhere), but if you're writing raw SQL the differences become more important.
I have deployed an app in Heroku using Django. The Django program uses a SQLite database db.sqlite3 on root directory to populate its page. Separately, there is also a Node.js scraper program that inserts to that database.
The problem is that the hard-refreshed webpage shows the same data even after the content of the database changed. Curiously, this does not happen when it is tested locally with python manage.py runserver. How can I fix this problem?
Thank you in advance!
For reference, here is my requirements.txt file:
Django==1.10.6
gunicorn==19.7.1
Pillow==4.0.0
selenium==3.3.1
whitenoise==3.3.0
You cannot use sqlite on Heroku.
An sqlite db is stored as a file on the local filesystem. But in Heroku the filesystem is ephemeral and is not shared between dynos. Every time you redeploy your app, or scale your process, or in your case launch a worker, you get a new filesystem with a different copy of the db file.
Use the proper Postgres support via the add-ons.
I have a worker running python script every 2 hour on Heroku.
The problem is each time I 'pull' the changes from git.
There is no changes at all for the sqlite3 database.
But I am sure the program is running and the database has changed by looking at the log file.
heroku log
How to retrieve the .db file then ?
It sounds like you have a little misconception. Heroku's git support is effectively one-way; you can use it to push new code to be run on the server, but you can't use it to copy files from Heroku back to your local tree.
Unfortunately it looks like there's not a good easy way to copy a file from your app to your local machine; you can use heroku run console to get a bash shell, and then scp a file out, but you're "pushing" it out of Heroku, and thus run can only copy to things with valid IP addresses.
If you're really using sqlite for your app's storage, though, you're going to run into a bigger problem. The filesystem for your app on Heroku is ephemeral, in that changes you make can be wiped out at any time. Heroku will delete your app's local storage and start over fresh whenever it wants to.
The right way to do it is use Heroku's built-in Postgres support and store your application's data there. Not only will it persist, but you'll be able to access it directly using the Postgres command-line tools.
Accessing the heroku console can now be done with:
heroku run bash
then i downloaded the linux gdrive application and ran in locally in the folder to upload my file to google drive. https://olivermarshall.net/how-to-upload-a-file-to-google-drive-from-the-command-line/ (skip step 4 and run with ./ like this ./gdrive upload my_file.txt
the other suggestion of heroku run console did not work for me (running a python flask app)
I am trying to install an app in Heroku cloud. This python based app is built on postgresql. I have set up this app in my local machine(Ubuntu). I have following queries.
How to install postgresql for this app on heroku?
Do I have to make same set up again on Heroku as I did on local machine? Like setting up of Flask etc.?
Can I submit code in many stages? e.g I have pushed code, after some time modified same code and pushed again.
I have already gone through following url https://devcenter.heroku.com/articles/git#deploying-code but didn't got any satisfactory answer.
Thanks
Sanjeev Yadav
Assuming you have heroku belt installed on your computer, create git repository inside your project(which you probably already have) then create Heroku app:
heroku create --stack cedar <name_of_your_app>
First add Postgresql to your project:
heroku addons:add heroku-postgresql
after that check if it has been added succesfully:
heroku pg:info
This should return something like:
HEROKU_POSTGRESQL_WHITE_URL <== Database name
Plan: Dev
Status: available
Then promote the database so heroku knows which one you will be using (you can have several databases in one project).
heroku pg:promote HEROKU_POSTGRESQL_WHITE_URL
At this point if you want to configure your database connections you can either get all the connection info from Heroku page of your project or from environment variable DATABASE_URL which was created when you promoted the database. If you use the latter you don't have to provide any info yourself, the function that i use frequently is something in this lines:
if "DATABASE_URL" in os.environ:
urlparse.uses_netloc.append("postgres")
url = urlparse.urlparse(os.environ.get("DATABASE_URL"))
DATABASE = {
"database" : url.path[1:],
"user" : url.username,
"password": url.password,
"host": url.hostname,
"port": url.port
}
return PostgresqlDatabase(**DATABASE)
This is for peewee ORM , but you can adapt it to any other ORM like SQLAlchemy, just take what's in DATABASE and plug it in to whatever you want.
A far as setting up your application, you need to write requirements.txt file with all dependencies that your app uses, this will automatically be parsed and all libraries installed when you first push your project to Heroku, also you need Procfile file inside your project, this tells Heroku which processes to run at which stage, the most important part is web part for instance:
web: gunicorn main:app
will start gunicorn server serving your site. If you need to run any scripts, for instance script that creates the tables in your database simply type:
heroku run create_tables.py
from within your repository.
As far as last question goes - yes you can push your code in many stages, your app will be restarted and all new code should work same as on your local machine. Hope this helps.