I was hoping someone could help me, I have a task to do for my College course using Python and SQlite.
I have two excel sheets I would like to make into two tables and a left join on some finance information. I have started by importing the two sheets into pandas and what I thought was creating two tables, but if I'm honest I don't know where I am going wrong. The spreadsheets contain a lot more Columns than I would like to use in the SQL database. Could this be a problem?
I am eventually presented with error code "OperationalError: table Lockedlist has no column named Site Ref"
import pandas as pd
Lockedlist_panda = pd.read_excel (r'Location of Lockedlist_panda')
PO1report_panda = pd.read_excel (r'Location of PO1report_panda')
import sqlite3
conn = sqlite3.connect("data_superstore1.db")
cursor = conn.cursor()
sql='''CREATE TABLE Lockedlist (
NR TEXT,
Programme TEXT NOT NULL,
Sub_Region TEXT NOT NULL,
Site_Type TEXT NOT NULL,
MS13_Actual TEXT NOT NULL,
PRIMARY KEY(NR)
)'''
cursor.execute(sql)
print("Table created successfully........")
#Commit your changes in the database
conn.commit()
#Closing the connection
conn.close()
conn = sqlite3.connect("data_superstore1.db")
cursor = conn.cursor()
sql='''CREATE TABLE PoReportTable (
Tracking_Field TEXT,
VENDOR TEXT NOT NULL,
Short_Text TEXT NOT NULL,
COST INT NOT NULL,
PRIMARY KEY(Tracking_Field)
)'''
cursor.execute(sql)
print("Table created successfully........")
#Commit your changes in the database
conn.commit()
#Closing the connection
conn.close()
conn = sqlite3.connect("data_superstore1.db")
Lockedlist_panda.to_sql('Lockedlist', conn, if_exists='append', index=False)
PO1report_panda.to_sql('PoReportTable', conn, if_exists='append', index=False)
#Commit your changes in the database
conn.commit()
#Closing the connection
conn.close()
Indeed pandas try to insert all columns from the dataframe into the database, hence if it has more columns that what you defined in the database it will complain for the missing columns,
Try to limit your dataframe to the columns you want first:
Lockedlist_panda = Lockedlist_panda[["NR","Programme","Sub_Region","Site_Type","MS13_Actual" ]]
Related
When I'm using pymysql to perform operations on MySQL database, it seems that all the operations are temporary and only visible to the pymysql connection, which means I can only see the changes through cur.execute('select * from qiushi') and once I cur.close() and conn.close() and log back in using pymysql, everything seems unchanged.
However, when I'm looking at the incremental id numbers, it does increased, but I can't see the rows that were inserted from pymysql connection. It seems that they were automatically deleted?!
Some of my code is here:
import pymysql
try:
conn = pymysql.connect(host='127.0.0.1',port=3306,user='pymysql',passwd='pymysql',charset='utf8')
cur = conn.cursor()
#cur.execute('CREATE TABLE qiushi (id INT NOT NULL AUTO_INCREMENT, content_id BIGINT(10) NOT NULL, content VARCHAR(1000), created TIMESTAMP DEFAULT CURRENT_TIMESTAMP, PRIMARY KEY(id));')
#cur.execute('DESCRIBE content')
#cur.fetchall()
cur.execute('USE qiushibaike')
for _ in range(0,len(content_ids)):
cur.execute("INSERT INTO qiushi (content,content_id) VALUES (\"%s\",%d)"%(jokes[_],int(content_ids[_])))
finally:
cur.close()
conn.close()
I solved the problem by myself...
Because the config is automatically committed, so after each SQL sentence we should commit the changes.
Approach 1:
add cur.commit() after the cur.execute()
Approach 2:
edit the connection config, add autocommit=True
I created a database in psql and in it, created a table called "tweet".
CREATE TABLE tweet
( tid CHARACTER VARYING NOT NULL, DATA json,
CONSTRAINT tid_pkey PRIMARY KEY (tid) );
Then when I use
SELECT * FROM tweet;
in the psql window it works and shows an empty table.
Now I have a python script that takes JSON data and is loading it into this table.
conn_string = "host='localhost' port=5432 dbname='tweetsql' user='tweetsql' password='tweetsql'"
conn = psycopg2.connect(conn_string)
cur = conn.cursor()
That sets up the connection and I don't think it had any issues.
Now I have some logic to read in the JSON file and then to add it in, I say:
cur.execute("INSERT INTO tweet (tid, data) VALUES (%s, %s)", (cur_tweet['id'], json.dumps(cur_tweet, cls=DecimalEncoder), ))
But this always says that the relation tweet doesn't exist. Am I missing something here? Is there an issue with my connection or can my script somehow not see the table? For reference I'm using psycopg2 for the connection.
EDIT: I updated the DDL to include a transaction I could commit but that didn't fix it either. Is it a schema issue?
This is what I did regarding the table creation to commit:
BEGIN;
CREATE TABLE tweet
( tid CHARACTER VARYING NOT NULL, DATA json,
CONSTRAINT tid_pkey PRIMARY KEY (tid) );
COMMIT;
EDIT 2: I'm posting some code here...
import psycopg2
import json
import decimal
import os
import ctypes
conn_string = "host='localhost' port=5432 dbname='tweetsql' user='tweetsql' password='tweetsql'"
conn = psycopg2.connect(conn_string)
cur = conn.cursor()
cur.execute("CREATE TABLE tweet (tid CHARACTER VARYING NOT NULL, DATA json, CONSTRAINT tid_pkey PRIMARY KEY (tid) );")
cur.commit()
for file in os.listdir(path):
if not is_hidden(file):
with open(path+file, encoding='utf-8') as json_file:
tweets = json.load(json_file, parse_float=decimal.Decimal)
for cur_tweet in tweets:
cur.execute("INSERT INTO tweet (tid, data) VALUES (%s, %s)", (cur_tweet['id'], json.dumps(cur_tweet, cls=DecimalEncoder), ))
cur.commit()
cur.close()
conn.close()
You're probably not committing the table creation, and, (I'm assuming; not seeing your complete code) you're starting a new connection via psycopg2 each time. You need to commit right after the table creation, and not in a new connection, as each connection is its own implicit transaction. So, your code flow should be something like this:
connect to the db
create the table using the cursor
fill the table
commit and disconnect from db.
Or, if you must separate creation from filling, just commit and disconnect after (2) and then reconnect before (3).
I am trying to connect to SQLITE database and it would seem that the code does create the database, but however it does not create the tables nor does it insert anything in them (most probably because they were never created)
I have SQLite DB Browser and in it I open the database but lack any tables or anything at all. Also I see my database in the folder next to my project.
I have the following code:
import sqlite3
conn = sqlite3.connect("dfg.db")
c = conn.cursor()
def create_tabe():
c.execute('CREATE TABLE IF NOT EXISTS tabl(city TEXT, temp REAL)')
def data_entry():
c.execute("INSERT INTO tabl VALUES ('dasfsd', 32434)")
conn.commit()
c.close()
conn.close()
I am inserting a couple thousand records into a table via the python code below:
values = ''
for row in cursor:
values = values + "(" + self.quoted_comma_separate(row) + "),"
values = values[:-1]
insert_statement = "INSERT INTO t1 ({0}) VALUES {1};".format(
self.comma_separate(members), values)
db = Database()
conn = db.get_db()
cursor = conn.cursor()
cursor.execute(insert_statement)
conn.commit()
conn.close()
When I check the database after it runs none of the records show up in the database. If I go into an MySQL editor and manually commit the transaction all of the records appear. Why is my conn.commit() not working?
The insert statements were fine. Turns out I had another database connection open and it was getting confused and committing to the wrong connection or something like that. Sorry for the pointless question :)
db = MySQLdb.connect("XXXXXXXX","root", "XXXXXX", database)
cursor = db.cursor()
cursor.execute('INSERT INTO media_files (ID, DATA) VALUES ("test", "test")')
cursor.execute("commit")
This statement errors and I cannot seem to figure out why. Any thoughts? The table media_files btw only has the two columns, ID and DATA each of which are VARCHAR(255)
Thanks
Trent
'INSERT INTO media_files (ID, DATA) VALUES ("test", "test")'
Using the parentheses will fix the problem. You were inserting one piece of data but provided 2 fields.