I am trying to take a dataframe and import it into a postgresql database. I've done this before and can always see the data in the database using pgAdmin. However, this time, there's no data visable in pgAdmin.
This is the SQL code I've used to create my table in the postgresql database:
CREATE TABLE Twitterapp
(
tweetid character varying(255),
tweettext character varying(255),
tweetretweetct integer,
tweetfavoritect integer,
tweetsource character varying(255),
tweetcreated character varying(255),
userid character varying(255),
userscreen character varying(255),
username character varying(255),
usercreatedt character varying(255),
user_desc character varying(255),
userfollowerct integer,
userfriendsct integer,
userlocation character varying(255),
usertimezone character varying(255)
)
WITH (
OIDS=FALSE
);
Then, using the sqlalchemy library in python, I tried to push my dataframe to the database:
#First attempt to load data
engine =create_engine('postgresql://username:password#link:5432/db_name')
conn = engine.connect()
a.to_sql('public.twitterapp', con=conn, if_exists='replace',
index=False)
Since that operation was successful, I try to read a table from the database to check sure it worked:
##call data from database
result_set = engine.execute("SELECT * FROM twitterapp")
for r in result_set:
print(r)
This code successfully returns the data in my dataframe but I still can't see it in my pgAdmin database. So I tried this second approach:
b = a.values.tolist()
sql2 = """INSERT INTO twitterapp VALUES(%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s);"""
conn = psycopg2.connect("dbname='db_name' user='username' host='link' password='password'")
c = conn.cursor()
for row in b:
c.execute(sql2, row)
print(row)
conn.commit()
Again, it successfully executes (ie, doesn't throw an error) but no data shows in pgAdmin.
What am I doing wrong?!
If you can show data during the session but not from pgAdmin (a different session) it's sure that your mistake is a wrong commit.
I'm not an expert of sqlalchemy but I think you have to read how to manage the session and how to commit it.
This is a good link:
http://docs.sqlalchemy.org/en/latest/orm/session_basics.html
You have to create a session object and execute a commit command:
session.commit()
You should check the port in psql if it's right or not. In my case local postgresql was listening to port 5433
Related
I have an issue trying to upload files from Python to my database. I can insert the binary data into the table just fine using %s, but when I try to update the record, I am unable to get this to work. Am I doing something wrong?
You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00\x00\x01\x90\x00\x00\x00\xc8\x08\x06\x00\' at line 8
Now, I know this is because the bytestring contains a ' so this doesn't work, it excepts, and leaves some security holes open if it were to work:
dbc.execute(f"""UPDATE userdata SET
emailaddr ='{email}',
firstname ='{namee}',
lastname ='{laste}',
username ='{usere}',
password ='{passe}',
phonenum ='{phone}',
photoimg ='{u_pho}'
WHERE user_id = '{user_ident}';""")
but I'm wondering if I can update a value like this:
insertcommand = f"""UPDATE userdata SET (emailaddr,firstname,lastname,username,password,phonenum,photoimg) VALUES (%s, %s, %s, %s, %s, %s, %s) WHERE user_id = '{user_ident}'"""
insertrecord = email,namee,laste,usere,passe,phone,u_pho
dbc.execute(insertcommand,insertrecord)
How would I go about updating bytestrings in MySQL with Python?
First-time question. I am writing a Python application for personal use, which reads metadata from MP3 files I've collected on CD-ROMs and inserts it into a MySQL database. At least, it should--but when it comes to the actual INSERT statement, the program is throwing a ProgrammingError: "1064: You have an error in your SQL syntax," etc.
In trying to write this program, I've learned how to create parameterized queries, instantiate the cursor object with cursor_class = MySQLCursorPrepared, and instantiate the database connection with use_pure = True. I've searched the Web for similar problems but come up dry.
Here is the offending code (it's the cursor.execute line specifically that throws the exception; for debugging purposes I've temporarily removed the try/except blocks):
table = "mp3_t"
# Parameterized query for SQL INSERT statement
query = '''
INSERT INTO %s
(track_num, title, artist, album, album_year, genre, discname)
VALUES
(%s, '%s', '%s', '%s', %s, '%s', '%s')
'''
conn = self.opendb(self.config)
cursor = conn.cursor(cursor_class = MySQLCursorPrepared)
for track in tracklist:
print("Counter: {}".format(counter))
# Tuple for parameterized query
input = (table, track['track_num'], track['title'],
track['artist'], track['album'], track['album_year'],
track['genre'], track['discname'])
print(query % input) # What is the actual query?
cursor.execute(query, input)
The database table is defined with the following SQL:
CREATE TABLE mp3_t (
id int NOT NULL PRIMARY KEY AUTO_INCREMENT,
track_num int NOT NULL,
title VARCHAR(255) NOT NULL,
artist VARCHAR(255) NOT NULL,
album VARCHAR(255) NOT NULL,
album_year int NOT NULL,
genre VARCHAR(255) NOT NULL,
discname VARCHAR(255) NOT NULL);
For debugging, I've got a print statement that outputs the query that's being sent to MySQL (the error message is particularly unhelpful for trying to pinpoint the cause of the problem), for example:
INSERT INTO mp3_t
(track_num, title, artist, album, album_year, genre, discname)
VALUES
(1, 'Moribund the Burgermeister', 'Peter Gabriel', 'I', 1977, 'Rock', 'Rock 19')
I don't see any error visually, and if I paste directly into the MySQL CLI and add the required semicolon, it inserts a row into the table as expected.
I'm completely stymied where the problem lies.
If it's any help, I'm running Python 3.6.7 and MariaDB 10.1.37 with Connector/Python 8.0.15, on Ubuntu 18.04 64-bit.
Table name should not be replaced by %s. I think your error message should like:
You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near "mp3_t"
So just use table name in your query template.
INSERT INTO mp3_t
(track_num, title, artist, album, album_year, genre, discname)
VALUES
(%s, %s, %s, %s, %s, %s, %s)
If you want to know the query will be executed, you should use these codes:
conn = connect()
cur = conn.cursor()
print(query % cur._get_db().literal(params))
Don't use %s for table name. Use the table name directly.
I created a database in psql and in it, created a table called "tweet".
CREATE TABLE tweet
( tid CHARACTER VARYING NOT NULL, DATA json,
CONSTRAINT tid_pkey PRIMARY KEY (tid) );
Then when I use
SELECT * FROM tweet;
in the psql window it works and shows an empty table.
Now I have a python script that takes JSON data and is loading it into this table.
conn_string = "host='localhost' port=5432 dbname='tweetsql' user='tweetsql' password='tweetsql'"
conn = psycopg2.connect(conn_string)
cur = conn.cursor()
That sets up the connection and I don't think it had any issues.
Now I have some logic to read in the JSON file and then to add it in, I say:
cur.execute("INSERT INTO tweet (tid, data) VALUES (%s, %s)", (cur_tweet['id'], json.dumps(cur_tweet, cls=DecimalEncoder), ))
But this always says that the relation tweet doesn't exist. Am I missing something here? Is there an issue with my connection or can my script somehow not see the table? For reference I'm using psycopg2 for the connection.
EDIT: I updated the DDL to include a transaction I could commit but that didn't fix it either. Is it a schema issue?
This is what I did regarding the table creation to commit:
BEGIN;
CREATE TABLE tweet
( tid CHARACTER VARYING NOT NULL, DATA json,
CONSTRAINT tid_pkey PRIMARY KEY (tid) );
COMMIT;
EDIT 2: I'm posting some code here...
import psycopg2
import json
import decimal
import os
import ctypes
conn_string = "host='localhost' port=5432 dbname='tweetsql' user='tweetsql' password='tweetsql'"
conn = psycopg2.connect(conn_string)
cur = conn.cursor()
cur.execute("CREATE TABLE tweet (tid CHARACTER VARYING NOT NULL, DATA json, CONSTRAINT tid_pkey PRIMARY KEY (tid) );")
cur.commit()
for file in os.listdir(path):
if not is_hidden(file):
with open(path+file, encoding='utf-8') as json_file:
tweets = json.load(json_file, parse_float=decimal.Decimal)
for cur_tweet in tweets:
cur.execute("INSERT INTO tweet (tid, data) VALUES (%s, %s)", (cur_tweet['id'], json.dumps(cur_tweet, cls=DecimalEncoder), ))
cur.commit()
cur.close()
conn.close()
You're probably not committing the table creation, and, (I'm assuming; not seeing your complete code) you're starting a new connection via psycopg2 each time. You need to commit right after the table creation, and not in a new connection, as each connection is its own implicit transaction. So, your code flow should be something like this:
connect to the db
create the table using the cursor
fill the table
commit and disconnect from db.
Or, if you must separate creation from filling, just commit and disconnect after (2) and then reconnect before (3).
From the PostgreSQL documentation if you do a INSERT without specifying a schema it should be a public schema.
conn = psycopg2.connect(dbname = 'orion',
host = 'localhost',
port = 5432,
user = 'earthling',
password = 'mysupersecretpassword')
sql = conn.cursor()
def INSERT(table, info, text):
date = datetime.date.today()
query = "INSERT INTO %s (info, text, date) " \
"VALUES (%s, %s, %s)" %(table, info, text, date)
sql.execute(query)
INSERT("main", "www.capecod.edu", "test")
For some reason I'm seeing the following error?
psycopg2.ProgrammingError: schema "www" does not exist
You are using string interpolation to create the query. This is what psycopg2 executes:
INSERT INTO main (info, text, date)
VALUES (www.capecod.edu, test, 2015-09-12)
If it's not obvious what's wrong here, it's that none of the values are quoted. Here is the properly quoted version:
INSERT INTO main (info, text, date)
VALUES ('www.capecod.edu', 'test', '2015-09-12')
The error is caused by the unquoted www.capecod.edu. Due to the dots, it's being interpreted as schema.table.column.
The "right" way to do this is with a parameterized query.
query = "INSERT INTO main (info, text, date) VALUES (%s, %s, %s)"
params = (info, text, date)
sql.execute(query, params)
psycopg2 will figure out what should be quoted and how. This is a safer option than simply interpolating the string yourself, which often leaves you open to SQL injection attack.
http://initd.org/psycopg/articles/2012/10/01/prepared-statements-psycopg/
Unfortunately, you can't just toss identifiers such as the table name in as a parameter, because then they are quoted as string values, which is bad SQL syntax. I found an answer (python adds "E" to string) that points to psycopg2.extensions.AsIs as a way to pass identifiers such as table names safely as parameters. I wasn't able to make this work in my testing, though.
If you go the AsIs route, you should be cautious about checking the table names are valid, if they somehow come from user input. Something like
valid_tables = ["main", "foo", "bar", "baz"]
if table not in valid_tables:
return False
I have an form which has FirstName, Lastname, Age and Gender. I am using MySQL db.
While using MySQl db, do we need to create table , do the insert operation in the single pythonic script ?
For example :
#!/usr/bin/python
import MySQLdb
db = MySQLdb.connect("localhost", "usename", "password", "TESTDB")
cursor = db.cursor()
cursor.execute ( """
CREATE TABLE PERSON
(
F_NAME CHAR(20) NOT NULL,
L_NAME CHAR(20),
AGE INT,
GENDER CHAR(4)
)
""")
cursor.execute( """
INSERT INTO PERSON (F_NAME, L_NAME, AGE, GENDER)
VALUES
('Neeraj','Lad','22','Male'),
('Vivek','Pal','24','Male')
""")
print cursor.rowcount
Edited Code:
#!/usr/bin/python
import MySQLdb
import cgi
print "Content-type: text/html\n"
form = cgi.FieldStorage()
f_Name = form.getvalue('firstname', '')
l_Name = form.getvalue('lastname', '')
age = form.getvalue('age', 0)
gender = form.getvalue('gender', '')
db = MySQLdb.connect(host="", user="", password="", db="")
cursor = db.cursor()
sql = "INSERT INTO PERSON (F_NAME, L_NAME, Age, Gender) VALUES (%s, %s, %s, %s)" %(f_name, l_name, age, gender)
cursor.execute(sql)
db.commit()
db.close()
I'm not 100% clear on what you're asking, but I'll take a guess.
You have to create a table exactly once in the database before you can insert into it.
If your Python script is talking to a brand-new database each time it runs, then it needs a CREATE TABLE statement.
If your Python script might be talking to a brand-new database, but will usually be talking to an already-existing one, then you can use CREATE TABLE IF NOT EXISTS.
But, except in toy learning projects, both of these are rare. Normally, you create the database once, then you write Python scripts that connect to it and assume it's already been created. In that case, you will not have a CREATE TABLE statement in your form handler.
If you're asking about inserting multiple values in a single INSERT statement… normally, you won't be inserting hard-coded values like 'Neeraj', but rather values that you get dynamically (e.g., from the web form). So you will be using parameterized SQL statements like this:
cursor.execute("""
INSERT INTO PERSON (F_NAME, L_NAME, AGE, GENDER)
VALUES (%s, %s, %s, %s)
""", (f_name, l_name, age, gender))
In that case, if you have, say, a list of 4-tuples, each representing a person, and you want to insert all of them, you do that not by putting multiple copies of the parameter lists in the SQL statement, but by putting a single parameter list, and using the executemany function:
cursor.execute("""
INSERT INTO PERSON (F_NAME, L_NAME, AGE, GENDER)
VALUES (%s, %s, %s, %s)
""", list_of_people)
You only need to create the table once. You can do that using the mysql CLI tool, or phpmyadmin, or python/MySQLdb, or by some other method.
CREATE TABLE PERSON will raise an error if the table PERSON already exists. If you'd like a way to create it in your python script, use IF NOT EXISTS, so subsequent runs of your program do not raise an error:
cursor.execute ( """
CREATE TABLE IF NOT EXISTS PERSON
(
F_NAME CHAR(20) NOT NULL,
L_NAME CHAR(20),
AGE INT,
GENDER CHAR(4)
)
""")