I've been trying for some days to connect my python 3 script to PostgresSQL database(psycopg2) in Heroku, without Django.
I found some article and related questions, but I had to invest a lot of time to get something that I thought should be very straightforward, even for a newbie like me.
I eventually made it work somehow but hopefully posting the question (and answer) will help other people to achieve it faster.
Of course, if anybody has a better way, please share it.
As I said, I had a python script that I wanted to make it run from the cloud using Heroku. No Django involved (just a script/scraper).
Articles that I found helpful at the beginning, even if they were not enough:
Running Python Background Jobs with Heroku
Simple twitter-bot with Python, Tweepy and Heroku
Main steps:
1. Procfile
Procfile has to be:
worker: python3 folder/subfolder/myscript.py
2. Heroku add-on
Add-on Heroku Postgres :: Database has to be added to the appropriate personal app in the heroku account.
To make sure this was properly set, this was quite helpful.
3. Python script with db connection
Finally, to create the connection in my python script myscript.py, I took this article as a reference and adapted it to Python 3:
import psycopg2
import urllib.parse as urlparse
import os
url = urlparse.urlparse(os.environ['DATABASE_URL'])
dbname = url.path[1:]
user = url.username
password = url.password
host = url.hostname
port = url.port
con = psycopg2.connect(
dbname=dbname,
user=user,
password=password,
host=host,
port=port
)
To create a new database, this SO question explains it. Key line is:
con.set_isolation_level(ISOLATION_LEVEL_AUTOCOMMIT)
You can do it using the SQLALCHEMY library.
First, you need to install the SQLALCHEMY library using pip, if you don't have pip on your computer install, you will know-how using a simple google search
pip install sqlalchemy
Here is the code snippet that do what you want:
from sqlalchemy import create_engine
from sqlalchemy.orm import scoped_session, sessionmaker
import os
# Put your URL in an environment variable and connect.
engine = create_engine(os.getenv("DATABASE_URL"))
db = scoped_session(sessionmaker(bind=engine))
# Some variables you need.
var1 = 12
var2 = "Einstein"
# Execute statements
db.execute("SELECT id, username FROM users WHERE id=:id, username=:username"\
,{"id": var1, "username": var2}).fetchall()
# Don't forget to commit if you did an insertion,etc...
db.commit()
I wasn't able to parse the DATABASE_URL provided by Heroku with the urllib.parse as suggested above, but the following worked for me:
The URL I retrieved from Heroku was in the format:
postgres://username:password#host:port/database
for example:
postgres://jticiuimwernbk:ff78903549d4c6ec13a53a8ffefcd201b937d54c35d976
#ec2-52-123-182-987.compute-1.amazonaws.com:5432/dbsd4fdf6c1awq
So I manually dissected it as follows:
user = 'jticiuimwernbk'
password = 'ff78903549d4c6ec13a53a8ffefcd201b937d54c35d976'
host = 'ec2-52-123-182-987.compute-1.amazonaws.com'
port = '5432'
database = 'dbsd4fdf6c1awq'
#Then created the connection using the above:
con = psycopg2.connect(database=database,
user=user,
password=password,
host=host,
port=port)
# and now I was able to perform queries:
cur = conn.cursor()
results = cur.execute("<some SQL query>;").fetchall()
cur.close()
conn.close()
Related
Hi I have created a tkinter app that uses MySQL database but I have two questions:
when I will share my app with others, will they be able to use MySQL database that I have in my computer? If not, what should I need to change in my code?
after I transform the app to .exe by using pyinstaller, then why I can not send it to others people by gmail?
To connect to my database I used this:
import mysql.connector
db = mysql.connector.connect(
host="localhost",
username = myusername,
password=mypassword,
database = "mydatabase"
)
Thank you so much for any help or answer.
I need an R script that allows me to connect to an Oracle DB without having to install anything needing admin powers, and preferrably nothing at all apart from package downloads. In python the following code works, I believe because it uses the cx_Oracle module as a portable driver. What would be a good R alternative?
import pandas as pd
import sqlalchemy
import sys
host = "xxx.intra"
database = "mydb"
user = "usr"
password = "pw"
def get_oracle_engine(host, database, user, password):
return sqlalchemy.create_engine("oracle+cx_oracle://{user}:{password}#{host}:1521/?service_name={database}".format(host=host, database=database, user=user, password=password))
engine=get_oracle_engine(host, database, user, password)
pd.read_sql_table("mytable", engine, schema= mydb,index.cols="id1")
I managed to install ROracle using the CRAN instructions but I keep getting the ORA-12154 TNS: cound not resolve the connect identifier specified when using:
library(ROracle)
con= DBI::dbconnect(dbDriver("Oracle"), user= user, password=password, host=host, dbname=database, port="1521")
By the way dbDriver("Oracle") returns
Driver name : Oracle (OCI)
Driver version: 1.3-1
Client version: 12.1.0.2.0
Try code like:
library(DBI)
library(ROracle)
drv <- Oracle()
con <- dbConnect(drv, 'cj', 'welcome', 'localhost:1521/orclpdb1')
dbGetQuery(con,"select count(*) from dual")
The connect string components are related to the {host}:1521/?service_name values you used with SQLAlchemy. Use a TNS alias or Easy Connect String, the same as other C based Oracle drivers, e.g. https://cx-oracle.readthedocs.io/en/latest/user_guide/connection_handling.html#connection-strings
The current ROracle code is at https://www.oracle.com/database/technologies/roracle-downloads.html There are some packaging glitches with uploading to CRAN and the CRAN maintainers haven't been responsive about resolving them.
ROracle still needs Oracle Client libraries such as from Oracle Instant Client.
I am attempting to create a flask web application that uses mysql to store users and some other data. When I use the following code
import MySQLdb
def connection():
conn = MySQLdb.connect(host='localhost', user='root', passwd='password', database='data')
c = conn.cursor()
return c, conn
I get a 500 internal service error. This code will connect on the command line, and my flask site works fine until I add the line
from dbconnect import connection
I have tried using different ways to connect to the database, (python connector, pymysql). They all give the same error. I have also tried updating permissions from inside mysql to the user. I have been following tutorials from https://pythonprogramming.net/flask-connect-mysql-using-mysqldb-tutorial/ but they did not get these errors in the tutorial and I have followed most of the tutorial exactly.
I have been having major trouble connecting my python shell to my postgres. I am doing this on windows. I have downloaded psycopg2 and everything for this to process, however it still is not working.
import psycopg2
conn=psycopg2.connect("dbname = 'test' user ='postgres' host ='localhost' password = 'mypassword'")
It gives me an error telling me that the database "test" does not exist, however it does! If you guys have any advice at all on what I should test out, that would be amazing. Thank you!
You can layout connection parameters as a string and pass it to the connect() function as like:
conn = psycopg2.connect("dbname=test user=postgres password=postgres")
Or you can use a list of keyword arguments like
conn = psycopg2.connect(host="localhost",database="test", user="postgres", password="postgres")
If its still fails then you should check on PostgreSQL side. You should try to connect the db in question using command line and see if error re appears or not. if it appears then something is missing on DB server side.
I have Python 3.6, pymysql 0.7.11 and sqlalchemy 1.2.4
I am having an issue creating an engine with sqlalchemy.
When I try (credentials changed for privacy, except for ":3306" as the port and the encd variable):
import pymysql
import sqlalchemy
login = ‘username’
passwd = ‘password’
server = '1.1.1.1:3306'
db = 'db_name'
encd = 'charset=utf8'
engine_str = 'mysql+pymysql://{}:{}#{}/{}?{}'.format(login, passwd, server, db, encd)
engine = sqlalchemy.create_engine(engine_str)
I get:
AttributeError: module 'sqlalchemy.sql.sqltypes' has no attribute 'NativeForEmulated'
Note that I also tried without the port :3306 and had same error.
The error occurs at this point, not when connecting the engine or using the connection.
When I create an engine with mypysql using the same credentials, it works fine:
engine = pymysql.connect(user=login, password=passwd,
host='1.1.1.1',
database=db, port=‘3306’)
but I need sqlalchemy for this project.
I haven't found anything searching for this error message. I tried running the exact same code on a different computer and it worked fine. Does anyone have any insight?