I created a database in psql and in it, created a table called "tweet".
CREATE TABLE tweet
( tid CHARACTER VARYING NOT NULL, DATA json,
CONSTRAINT tid_pkey PRIMARY KEY (tid) );
Then when I use
SELECT * FROM tweet;
in the psql window it works and shows an empty table.
Now I have a python script that takes JSON data and is loading it into this table.
conn_string = "host='localhost' port=5432 dbname='tweetsql' user='tweetsql' password='tweetsql'"
conn = psycopg2.connect(conn_string)
cur = conn.cursor()
That sets up the connection and I don't think it had any issues.
Now I have some logic to read in the JSON file and then to add it in, I say:
cur.execute("INSERT INTO tweet (tid, data) VALUES (%s, %s)", (cur_tweet['id'], json.dumps(cur_tweet, cls=DecimalEncoder), ))
But this always says that the relation tweet doesn't exist. Am I missing something here? Is there an issue with my connection or can my script somehow not see the table? For reference I'm using psycopg2 for the connection.
EDIT: I updated the DDL to include a transaction I could commit but that didn't fix it either. Is it a schema issue?
This is what I did regarding the table creation to commit:
BEGIN;
CREATE TABLE tweet
( tid CHARACTER VARYING NOT NULL, DATA json,
CONSTRAINT tid_pkey PRIMARY KEY (tid) );
COMMIT;
EDIT 2: I'm posting some code here...
import psycopg2
import json
import decimal
import os
import ctypes
conn_string = "host='localhost' port=5432 dbname='tweetsql' user='tweetsql' password='tweetsql'"
conn = psycopg2.connect(conn_string)
cur = conn.cursor()
cur.execute("CREATE TABLE tweet (tid CHARACTER VARYING NOT NULL, DATA json, CONSTRAINT tid_pkey PRIMARY KEY (tid) );")
cur.commit()
for file in os.listdir(path):
if not is_hidden(file):
with open(path+file, encoding='utf-8') as json_file:
tweets = json.load(json_file, parse_float=decimal.Decimal)
for cur_tweet in tweets:
cur.execute("INSERT INTO tweet (tid, data) VALUES (%s, %s)", (cur_tweet['id'], json.dumps(cur_tweet, cls=DecimalEncoder), ))
cur.commit()
cur.close()
conn.close()
You're probably not committing the table creation, and, (I'm assuming; not seeing your complete code) you're starting a new connection via psycopg2 each time. You need to commit right after the table creation, and not in a new connection, as each connection is its own implicit transaction. So, your code flow should be something like this:
connect to the db
create the table using the cursor
fill the table
commit and disconnect from db.
Or, if you must separate creation from filling, just commit and disconnect after (2) and then reconnect before (3).
Related
I know I can call a database in the mysql.connector.connect method. However, I'm trying to programmatically create the database if it doesn't already exist, then connect too to upload table data.
Currently my line cursor.execute("SELECT DATABASE myDatabase") is generating an error in my "SQL syntax".
How can I check if a database exists and then connect to it?
db = mysql.connector.connect(
host="localhost",
user=os.environ.get('MYSQL_DB_USER'),
password=os.environ.get('MYSQL_DB_USER_PASS')
)
cursor = db.cursor()
cursor.execute("CREATE DATABASE IF NOT EXISTS myDatabase")
cursor.execute("SELECT DATABASE myDatabase")
# create table on database
sql = "CREATE TABLE IF NOT EXISTS `trades` (`id` INT(11) UNSIGNED NOT NULL AUTO_INCREMENT, `objectid` VARCHAR(100), `type` VARCHAR(5), `ownerid` VARCHAR(100), `color` VARCHAR(15), `shape` VARCHAR(15), `size` VARCHAR(15), PRIMARY KEY (`id`) );"
print(sql)
cursor.execute(sql)
Alternatively, the database property can be used of the MySQLConnection object, as per the MySQL Connector documentation (which executes the USE command for you):
db = mysql.connector.connect(
host="localhost",
user=os.environ.get('MYSQL_DB_USER'),
password=os.environ.get('MYSQL_DB_USER_PASS')
)
cursor = db.cursor()
cursor.execute("CREATE DATABASE IF NOT EXISTS myDatabase")
db.database = 'myDatabase'
(I was dealing with the same issue and stumbled upon this post, after which I found the described solution. So thought I'd share this alternative solution for anyone interested.)
Per MySQL documentation, the syntax is:
CREATE DATABASE `db_name`;
Then to select/change the database, use:
USE `db_name`;
To list all available databases, use:
SHOW DATABASES;
Alternatively, you can skip the USE command and reference the database in the statement itself as:
CREATE TABLE `db_name`.`table1` ... ;
Or, in the case of a SELECT statement:
SELECT * FROM `db_name`.`table1`;
I am trying to just create a temporary table in my SQL database, where I then want to insert data (from a Pandas DataFrame), and via this temporary table insert the data into a 'permanent' table within the database.
So far I have something like
""" Database specific... """
import sqlalchemy
from sqlalchemy.sql import text
dsn = 'dsn-sql-acc'
database = "MY_DATABASE"
connection_str = """
Driver={SQL Server Native Client 11.0};
Server=%s;
Database=%s;
Trusted_Connection=yes;
""" % (dsn,database)
connection_str_url = urllib.quote_plus(connection_str)
engine = sqlalchemy.create_engine("mssql+pyodbc:///?odbc_connect=%s" % connection_str_url, encoding='utf8', echo=True)
# Open connection
db_connection = engine.connect()
sql_create_table = text("""
IF OBJECT_ID('[MY_DATABASE].[SCHEMA_1].[TEMP_TABLE]', 'U') IS NOT NULL
DROP TABLE [MY_DATABASE].[SCHEMA_1].[TEMP_TABLE];
CREATE TABLE [MY_DATABASE].[SCHEMA_1].[TEMP_TABLE] (
[Date] Date,
[TYPE_ID] nvarchar(50),
[VALUE] nvarchar(50)
);
""")
db_connection.execute("commit")
db_connection.execute(sql_create_table)
db_connection.close()
The "raw" SQL-snippet within sql_create_table works fine when executed in SQL Server, but when running the above in Python, nothing happens in my database...
What seems to be the issue here?
Later on I would of course want to execute
BULK INSERT [MY_DATABASE].[SCHEMA_1].[TEMP_TABLE]
FROM '//temp_files/temp_file_data.csv'
WITH (FIRSTROW = 2, FIELDTERMINATOR = ',', ROWTERMINATOR='\n');
in Python as well...
Thanks
These statements are out of order:
db_connection.execute("commit")
db_connection.execute(sql_create_table)
Commit after creating your table and your table will persist.
When I'm using pymysql to perform operations on MySQL database, it seems that all the operations are temporary and only visible to the pymysql connection, which means I can only see the changes through cur.execute('select * from qiushi') and once I cur.close() and conn.close() and log back in using pymysql, everything seems unchanged.
However, when I'm looking at the incremental id numbers, it does increased, but I can't see the rows that were inserted from pymysql connection. It seems that they were automatically deleted?!
Some of my code is here:
import pymysql
try:
conn = pymysql.connect(host='127.0.0.1',port=3306,user='pymysql',passwd='pymysql',charset='utf8')
cur = conn.cursor()
#cur.execute('CREATE TABLE qiushi (id INT NOT NULL AUTO_INCREMENT, content_id BIGINT(10) NOT NULL, content VARCHAR(1000), created TIMESTAMP DEFAULT CURRENT_TIMESTAMP, PRIMARY KEY(id));')
#cur.execute('DESCRIBE content')
#cur.fetchall()
cur.execute('USE qiushibaike')
for _ in range(0,len(content_ids)):
cur.execute("INSERT INTO qiushi (content,content_id) VALUES (\"%s\",%d)"%(jokes[_],int(content_ids[_])))
finally:
cur.close()
conn.close()
I solved the problem by myself...
Because the config is automatically committed, so after each SQL sentence we should commit the changes.
Approach 1:
add cur.commit() after the cur.execute()
Approach 2:
edit the connection config, add autocommit=True
I am able to connect Python 3.4 and Postgres but I am the query is not successfully getting executed. For e.g, the table below is not getting created
import psycopg2
from psycopg2 import connect
try:
conn = psycopg2.connect("dbname='postgres' user='postgres' host='localhost' password='postgres'")
print("Database connected!")
cur = conn.cursor()
cur.execute("""CREATE TABLE DEPARTMENT(
ID INT PRIMARY KEY NOT NULL,
DEPT CHAR(50) NOT NULL,
EMP_ID INT NOT NULL
)""")
except:
print("I am unable to connect to the database")
Just add
conn.commit()
after you've run the execute.
Relational databases have the concept of transaction, which happen (if at all) "atomically" (all-or-none). You need to commit a transaction to actually make it take place; until you've done that, you keep the option to rollback it instead, to have no changes made to the DB if you've found something iffy on the way.
I am trying to add all words of a text file into a column such that one row has one word. my code is as :
import MySQLdb
conn = MySQLdb.connect (host = "localhost",user = "root", db = "pcorpora")
c = conn.cursor()
file = open('C:\Users\Admin\Desktop\english.txt', 'r')
words = list(file.read())
i=0
for value in words:
c.execute("""INSERT INTO tenglish (`english words`) VALUES (%s)""" % (words[i]) i=i+1)`
The code run without error but table is still empty.
You should use commit
c.execute("""INSERT INTO tenglish (`english words`) VALUES (%s)""" % (value))
con.commit()
This method sends a COMMIT statement to the MySQL server, committing
the current transaction. Since by default Connector/Python does not
autocommit, it is important to call this method after every
transaction that modifies data for tables that use transactional
storage engines.