I'm trying to make a mysql table with python 2.7 using MySQLdb.
I've written a program to retrieve tweets based off of a search or "query" from twitter.
I want the tweets to be stored in a MySQL database for later retrieval by their query.
Things to know beforehand: the variable "tweets" is a list of strings. I want the table to have two parts, the query (what the person searched) and the tweets.
What is supposed to happen, is it should cycle through the list of tweets and insert them into the table. I keep getting Syntax errors and I'm not sure why.
Maybe there is a better way to do this than how I am currently trying to? Or maybe I'm just doing it wrong? All suggestions appreciated.
Thanks
#Does the MySQL stuff
db=MySQLdb.connect(host = 'localhost',user='root',passwd="swag",db="tweets")
cursor = db.cursor()
cursor.execute("DROP TABLE IF EXISTS TWEETS")
sql = """CREATE TABLE TWEETS (
QUERY CHAR(40) NOT NULL,
TWEET BLOB,
)"""
cursor.execute(sql)
#Inserts tweets into MySQLdb
for i in range(0,len(tweets)):
sql = "INSERT INTO TWEETS(QUERY, \
TWEET) \
VALUES ('%s', '%s')" % \
(query, tweets[i])
try:
cursor.execute(sql)
db.commit()
except:
db.rollback()
Related
I'm trying to transfer a user input from a python code to a table in postgresql
What I want to do is place an input() in this code and make it's value go to the comment (#) in the code.
conn = psycopg2.connect(
host="localhost",
database="Twitterzuil",
user="postgres",
password="")
cur = conn.cursor()
cur.execute("INSERT INTO Bericht2 (name) VALUES (#THIS IS WHERE I WANT THE INPUT TO GO)");
conn.commit()
I have no idea how, I'm really a beginner in all this so any help is appreciated
I believe what you are asking about is called string interpolation. Using f-style format, this might look like
new_name = "'bob'" # need single quotes for SQL strings
sql = f"INSERT INTO Bericht2 (name) VALUES ({new_name})" # => sql == "INSERT INTO Bericht2 (name) VALUES ('bob')"
cur.execute(sql)
Note the f at the start of the string, when you do this expressions inside {} pairs get replaced with their python values (tutorial). There are also string formatting approaches involving % substitution and the .format method on strings.
If you are doing anything beyond the basics you should look into using the SQLAlchemy package; here's the link to their insert api. Using SQLAlchemy will help reduce the risks that can come with manually constructing SQL queries.
Example from "Inserting Rows with SQLAlchemy"
from sqlalchemy import insert
stmt = insert(user_table).values(name='spongebob', fullname="Spongebob Squarepants")
with engine.connect() as conn:
result = conn.execute(stmt)
conn.commit()
So I'm just trying to play around in MySQL to start getting some experience. I downloaded the Yelp dataset which is in JSON format. Since I already know Python, I figured it wouldn't be too hard to import the data into MySQL using Python because I couldn't find an easy way to do it with standalone MySQL. I set up a loop to add in the JSON row by row, and this loop ran for more than 12 hours. But now that I go to query the database in MySQL, the table appears to be empty.
SELECT COUNT(*) FROM reviews;
Returns a big fat 0.
I've tried querying different things to no avail--the database is empty. I've checked to make sure that the INSERT statement is correct with the given variables, and the string turned out to be exactly what it should.
import mysql.connector as mysql
import json
import os
db = mysql.connect(
user=('root'),
password=('{secret_password}'),
database='yelpdb',
host='localhost'
)
cursor = db.cursor()
with open('filepath', 'r') as f:
for line in f:
data = json.loads(line)
cursor.execute("CREATE TABLE reviews (review_id VARCHAR(255), user_id VARCHAR(255), business_id VARCHAR(255), stars FLOAT(2), useful INT(6), funny INT(6), cool INT(6), text VARCHAR(5001), date DATETIME(2))")
table = 'reviews'
with open('{path}', 'r') as f:
for line in f:
data = json.loads(line)
placeholders = ', '.join(['%s'] * len(data))
columns = ', '.join(data.keys())
values = [each for each in data.values()]
sql = "INSERT INTO %s (%s) VALUES (%s)" % (table, columns, placeholders)
cursor.execute(sql, values)
I thought--especially since the code took so long--that it was a lot of inserts, so it would just take a long time. But empty. It's hard for me to troubleshoot when it takes so long to do.
import sqlite3
con = sqlite3.connect("Demo.db")
cursor = con.cursor()
query = "UPDATE Table set demoName = %s Where demoId = %s"
cursor.execute(query,("demoName",1)
con.commit()
This method is not work.
I want to keep the current values in the column and add more values to those values. I can do that with what code. Updating doesn't work here.
UPDATE sql command is to update existing rows. Use the INSERT sql command to create new rows.
import sqlite3
con = sqlite3.connect("Demo.db")
cursor = con.cursor()
query = "INSERT INTO Table (demoName, demoId) VALUES (%s, %s)"
cursor.execute(query,("demoName",1)
con.commit()
Python official sqlite3 documentation suggests the following links for learning sql:
https://www.w3schools.com/sql/ - Tutorial, reference and examples for learning SQL syntax.
https://www.sqlite.org - The SQLite web page; the documentation describes the syntax and the available data types for the supported SQL dialect.
I would like to check if a database table exists or not, but I don't know how to do.
I wrote (for example with SQLite, although I use MySQL mainly),
import sqlite3
table_name = "some_table"
connection = sqlite3.connect(db)
cursor = connection.cursor()
table_check = "SELECT name FROM sqlite_master WHERE type='table' AND name={};".format(table_name)
if not cursor.execute(table_check).fetchone(): # if the table doesn't exist
# OR if cursor.execute(table_check).fetchone() == "":
create_table()
else:
update_table()
But, an Error occured and I cannot proceed.
sqlite3.OperationalError: no such column: some_table
I read several Q&A here, but I couldn't get those.
Any advice can help me.
Thank you.
Python 3.5.1
The answer is depending on what rdbms product (mysql, sqlite, ms sql, etc.) you use.
You are getting this particular error in your above query because you do not enclose the value of table_name variable in single quotes.
In mysql you can use information_schema.tables table to query if a table exists.
I am trying to create a table in Python with MySQL, and then insert new values into the table!
The program works with no error message, but the table stays empty all the time. Here are my python scripts:
cursor = conn.cursor()
cursor.execute('DROP TABLE twitter')
cursor.execute("CREATE TABLE twitter (id INT UNSIGNED NOT NULL AUTO_INCREMENT, user_id CHAR, state CHAR(2), city CHAR, tweet CHAR, PRIMARY KEY (id))")
### after getting some data, then insert the data into the table:
cursor.execute('INSERT INTO twitter VALUES ({0},{1},{2},{3})'.format(tw_userid, tw_state, tw_city, tw_text))
cursor.close()
conn.commit()
conn.close()
But any time I try to select data from this table, I get an empty set.
I also tried without using format option, but still I get an empty set:
cursor.execute('INSERT INTO twitter VALUES ({0},{1},{2},{3})', (tw_userid, tw_state, tw_city, tw_text))
Any idea why the insert command doesn;t change the table at all?
You need to use a comma to separate parameters in python.
cursor.execute('INSERT INTO twitter VALUES ({0},{1},{2},{3})'.format(tw_userid, tw_state, tw_city, tw_text))
Should be:
data = (tw_userid, tw_state, tw_city, tw_text)
cursor.execute('INSERT INTO twitter VALUES ({0},{1},{2},{3})', data)
You can see the documentation for the python connector on the mysql documentation page.