I searched for this error and was able to find some information. I am importing Excel into postgreSQL and below is my script. I know the error is regarding how Python is handling a blank field. The script None if not has functioned fine for other tables I have imported and it also works fine for the Traffic field for this table but for some odd reason throws out an error for Week_Ending date field. I am new to Python and StackOverflow so if you decide to down vote my question please tell me why.
import psycopg2
import xlrd
book = xlrd.open_workbook("consolidateddata.xlsx")
sheet = book.sheet_by_name("Builder_Traffic")
database = psycopg2.connect (database = "", user="", password="", host="", port="")
cursor = database.cursor()
delete = """Drop table if exists "Python".buildertraffic"""
print (delete)
mydata = cursor.execute(delete)
cursor.execute('''CREATE TABLE "Python".buildertraffic
(Builder_Name varchar(55),
Traffic integer,
Week_Ending date,
Project_ID integer
);''')
print "Table created successfully"
query = """INSERT INTO "Python".buildertraffic (Builder_Name, Traffic, Week_Ending, Project_ID)
VALUES (%s, %s, %s, %s)"""
for r in range(1, sheet.nrows):
Builder_Name = sheet.cell(r,0).value
Traffic = None if not sheet.cell(r,1).value else sheet.cell(r,1).value
Week_Ending = None if not sheet.cell(r,2).value else xlrd.xldate.xldate_as_datetime(sheet.cell(r,2).value,book.datemode)
Project_ID = sheet.cell(r,3).value
# Assign values from each row
values = (Builder_Name, Traffic, Week_Ending, Project_ID)
# Execute sql query
cursor.execute(query, values)
cursor.close()
database.commit()
database.close()
print ""
print "All Done! Bye, for now."
print ""
columns = str(sheet.ncols)
rows = str(sheet.nrows)
print "I just imported Excel into postgreSQL"
When run, this fails with the stack frame
Week_Ending = None if not sheet.cell(r,2).value else xlrd.xldate.xldate_as_datetime(sheet.cell(r,2).value,book.datemode)
File "C:\Python27\lib\site-packages\xlrd\xldate.py", line 123, in xldate_as_datetime
days = int(xldate)
ValueError: invalid literal for int() with base 10: ''
Related
I'm using snscrape lib to scrape twitter data off twitter. I want to insert this data into my database but I seem to be failing no matter what method I try. when I use a loop and create a sql query after the loop to insert the values 1 by 1. I get an IndexError and a TypeError. When I try to append the data into a list. I can't loop in to each value 1 by 1. Now I'm stuck and don't know what to do.
method 1
class Tweet_list():
def tweets_list1(self):
dbname = '******'
user = '******'
password = '******'
host = '*******'
port = ****
cur = None
conn = None
try:
conn = psycopg2.connect(
dbname = dbname,
user = user,
password = password,
host = host,
port = port
)
cur = conn.cursor()
cur.execute('DROP TABLE IF EXISTS Machine_twitter')
create_table = '''CREATE TABLE IF NOT EXISTS Machine_twitter (
id int PRIMARY KEY,
Tweet text,
Tweet_id int,
Timestamp timestamp,
Replys int,
Retweets int,
Likes int,
Username char)'''
cur.execute(create_table)
for i, tweet in enumerate(sntwitter.TwitterSearchScraper('from:TheHoopCentral').get_items()):
if i > 5:
break
insert_tweet = 'INSERT INTO Machine_twitter (Tweet, Tweet_id, Timestamp, Replys, Retweets, Likes, Username) VALUES (%s, %s, %s, %s,%s, %s, %s)'
insert_values = (tweet.content, tweet.id, tweet.date, tweet.replyCount, tweet.retweetCount, tweet.likeCount, tweet.user.username)
cur.execute(insert_tweet, insert_values)
conn.commit()
print('completed')
except Exception as error:
print(error)
finally:
if cur is not None:
cur.close()
if conn is not None:
conn.close()
tweets = Tweet_list()
tweets2 = Tweet_list()
tweets2.tweets_list1()
error
IndexError: list index out of range
method 2
def update_list1(self):
tweets_list2 = []
for i, tweet in enumerate(sntwitter.TwitterSearchScraper('from:TheHoopCentral').get_items()):
if i > 100:
break
tweets_list2.append([tweet.content, tweet.id,tweet.likeCount, tweet.retweetCount, tweet.replyCount, tweet.user.username])
tweet_df = pd.DataFrame(tweets_list2, columns=('tweet', 'tweet id', 'likeCount', 'retweetCount', 'replyCount', 'username'))
tweet_df.head()
the problem with the second method is that after the list gets appended. I can't access the values(eg. tweet.content) so I can insert them into the database. I've tried every method under the sun but I'm failing miserably can somebody help.
I have an ever growing and changing database that reflects a permits passed by the State and EPA.
As the database changes and updates I need to transfer the relevant information.
The script does two things; first it checks which fields are the same and creates a list of fields and data that will be inserted into the new database. Second to insert the data into the new database.
Problem is I cannot get it to insert. I have matched everything like it says online in various ways but i get error ('42000', '[42000] [Microsoft][ODBC Microsoft Access Driver] Syntax error in INSERT INTO statement. (-3502) (SQLExecDirectW)').
I cannot figure out how to prevent it.
Code:
import pyodbc
importDatabase = r"J:\ENVIRO FIELD\AccessDatabases\MS4\MS4 Town Databases\~Template\MS4_Apocalypse Import DEV 1.accdb"
"Create the Import Database Connection"
connectionImport = pyodbc.connect(r'Driver={Microsoft Access Driver (*.mdb, *.accdb)};DBQ=%s;' %(importDatabase))
cursorImport = connectionImport.cursor()
"####---Outfall Section---####"
"Import the outfall names into the new database"
tbl = "tbl_Outfall_1_Profile"
exportList = []
importList = []
for row in cursorImport.columns(table = "tblExportMigration_Outfall_1_Profile"):
field = row.column_name
exportList.append(field)
for row in cursorImport.columns(table = "tbl_Outfall_1_Profile"):
field = row.column_name
importList.append(field)
matchingList = []
for field in exportList:
if field != "outfallID":
if field in importList:
matchingList.append(field)
else:
continue
sqlValue = ""
for field in matchingList:
sqlValue += "[%s], " %(field)
sqlValue = sqlValue[:-2]
sql = "SELECT %s from %s" %(sqlValue, "tblExportMigration_Outfall_1_Profile")
for rowA in cursorImport.execute(sql):
tupleList = list(rowA)
tupleList = ["" if i == None else i for i in tupleList]
tupleValues = tuple(tupleList)
sqlUpdate = """INSERT INTO tbl_Outfall_1_Profile (%s) Values %s;""" %(sqlValue, tupleValues)
cursorImport.execute(sqlUpdate)
cursorImport.close()
This is the sql string I create
"INSERT INTO tbl_Outfall_1_Profile ([profile_OutfallName], [profile_HistoricalName1], [profile_HistoricalName2], [profile_HistoricalName3], [profile_HistoricalName4]) Values ('756', '', '', '', '');"
Taking what #Gord Thompson said I was actually able to create a dynamic parameter flow
First created a module to create the ?
def Defining_Paramters(length):
parameterString = ""
for x in range(1,length):
parameterString += "?, "
parameterString += "?"
return parameterString
Then stuck it into the string for the sql update
sqlUpdate = sqlUpdate = "INSERT INTO %s (%s) Values (%s);" %(table, sqlFrameworkSubStr, parameters)
Run the cursor and commit it
cursorTo.execute(sqlUpdate, (dataTuple))
connectionTo.commit()
It would seem that you have to create the query in its entirety then have your data in tuple format for entry
This is the sql string [I think] I create
Try this:
sqlUpdate = """INSERT INTO tbl_Outfall_1_Profile (%s) Values (%s);""" %(sqlValue, tupleValues)
or perhaps:
sqlUpdate = "INSERT INTO tbl_Outfall_1_Profile (%s) Values (%s);" %(sqlValue, tupleValues)
Im trying to insert / update a very long sql, tried to format to string, but wont take any effect on MySQL table:
import MySQLdb
db = MySQLdb.connect(host=host, user=user, passwd=passwd, db=db)
cur = db.cursor()
str1 = 'x'
str2 = 'y'
s = """a:150:{i:0;b:0;s:12:"social_icons";a:9:{s:8:"facebook";s:1:"1";s:7:"twitter";s:1:"1";s:5:"email";s:1:"1";s:9:"pinterest";s:1:"1";s:10:"googleplus";s:1:"1";s:8:"linkedin";s:1:"0";s:2:"vk";s:1:"0";s:6:"tumblr";s:1:"0";s:8:"whatsapp";s:1:"0";}s:7:"backups";N;s:9:"smof_init";s:31:"Tue, 16 Aug 2016 23:03:59
+0000";s:17:"minified_flatsome";i:0;s:16:"flatsome_builder";i:1;s:13:"flatsome_docs";i:1;s:16:"maintenance_mode";i:0;s:21:"maintenance_mode_text";s:24:"Please check back soon..";s:19:"html_scripts_header";s:0:"";s:19:"html_scripts_footer";s:0:"";s:15:"html_custom_css";s:6:"div {}";s:22:"html_custom_css_mobile";s:0:"";s:9:"site_logo";s:57:"[site_url]/wp-content/uploads/2016/08/Logomakr_8dvMrU.png";s:0:"";s:165:"Favicon upload has moved to: <br/> <a href=''[site_url]/wp-admin/customize.php?&autofocus%5Bpanel%5Dof-option-logoandicons''>Appearance
> Customize > Site Identity</a>";s:16:"custom_cart_icon";s:0:"";s:14:"site_logo_dark";s:0:"";s:16:"site_logo_sticky";s:0:"";s:11:"body_layout";s:10:"full-width";s:10:"box_shadow";i:0;s:7:"body_bg";s:0:"";s:13:"body_bg_image";s:0:"";s:12:"body_bg_type";s:12:"bg-full-size";s:13:"content_color";s:5:"light";s:10:"content_bg";s:4:"#FFF";s:13:"header_preset";s:0:"";s:13:"header_height";s:3:"120";s:10:"logo_width";s:3:"210";s:13:"logo_position";s:6:"center";s:10:"search_pos";s:4:"left";s:12:"nav_position";s:3:"top";s:8:"nav_size";s:3:"80%";s:18:"myaccount_dropdown";i:1;s:19:"account_login_style";s:4:"link";s:9:"show_cart";i:1;s:14:"top_right_text";s:0:"";s:13:"header_sticky";i:1;s:20:"header_height_sticky";s:2:"70";s:12:"header_color";s:5:"light";s:9:"header_bg";s:4:"#fff";s:13:"header_bg_img";s:0:"";s:17:"header_bg_img_pos";s:8:"repeat-x";s:11:"topbar_show";i:1;s:9:"topbar_bg";s:0:"";s:11:"topbar_left";s:0:"";s:12:"topbar_right";s:0:"";s:15:"nav_position_bg";s:4:"#eee";s:18:"nav_position_color";s:5:"light";s:17:"nav_position_text";s:0:"";s:21:"nav_position_text_top";s:0:"";s:17:"html_after_header";s:0:"";s:10:"html_intro";s:0:"";s:16:"footer_left_text";s:158:"Copyright [ux_current_year] © <strong>asd.com</strong>.<br> <img src=''http://www.aasd.com/wp-content/uploads/2015/12/creditcard-icons.png''>";s:17:"footer_right_text";s:137:"<img src="http://gninggel.com/wp-content/uploads/2016/08/comodo_secure_seal_100x85_transp.png"><br> %s <br>%s";s:14:"footer_1_color";s:5:"light";s:17:"footer_1_bg_color";s:7:"#ff3233";s:17:"footer_1_bg_image";s:0:"";s:16:"footer_1_columns";s:7:"large-3";s:14:"footer_2_color";s:4:"dark";s:17:"footer_2_bg_color";s:7:"#772222";s:17:"footer_2_bg_image";s:0:"";s:16:"footer_2_columns";s:7:"large-3";s:19:"footer_bottom_style";s:4:"dark";s:19:"footer_bottom_color";s:4:"#333";s:18:"html_before_footer";s:0:"";s:17:"html_after_footer";s:0:"";s:13:"disable_fonts";i:0;s:13:"type_headings";s:4:"Lato";s:10:"type_texts";s:4:"Lato";s:8:"type_nav";s:4:"Lato";s:8:"type_alt";s:14:"Dancing Script";s:11:"type_subset";a:7:{s:5:"latin";s:1:"1";s:12:"cyrillic-ext";s:1:"0";s:9:"greek-ext";s:1:"0";s:5:"greek";s:1:"0";s:10:"vietnamese";s:1:"0";s:9:"latin-ext";s:1:"0";s:8:"cyrillic";s:1:"0";}s:11:"custom_font";s:0:"";s:13:"color_primary";s:7:"#dd3333";s:15:"color_secondary";s:7:"#eeee22";s:13:"color_success";s:7:"#7a9c59";s:11:"color_links";s:0:"";s:13:"button_radius";s:3:"0px";s:15:"dropdown_border";s:0:"";s:11:"dropdown_bg";s:0:"";s:13:"dropdown_text";s:5:"light";s:11:"blog_layout";s:13:"right-sidebar";s:10:"blog_style";s:11:"blog-normal";s:18:"blog_archive_title";i:1;s:11:"blog_header";s:1:" ";s:15:"blog_after_post";s:1:" ";s:16:"blog_post_layout";s:13:"right-sidebar";s:15:"blog_post_style";s:7:"default";s:15:"blog_author_box";i:1;s:10:"blog_share";i:0;s:13:"blog_parallax";i:0;s:19:"featured_items_page";i:0;s:22:"featured_items_pr_page";s:2:"12";s:22:"featured_items_related";s:7:"default";s:29:"featured_items_related_height";s:5:"250px";s:16:"wc_account_links";i:1;s:14:"facebook_login";i:0;s:17:"facebook_login_bg";s:0:"";s:14:"color_checkout";s:0:"";s:10:"color_sale";s:0:"";s:16:"color_new_bubble";s:7:"#7a9c59";s:12:"color_review";s:0:"";s:15:"product_sidebar";s:12:"left_sidebar";s:25:"product_offcanvas_sidebar";s:1:"0";s:15:"product_display";s:4:"tabs";s:18:"cart_dropdown_show";s:1:"1";s:16:"shop_aside_title";s:0:"";s:12:"product_zoom";s:1:"0";s:16:"related_products";s:6:"slider";s:23:"related_products_pr_row";s:1:"4";s:20:"max_related_products";s:2:"12";s:15:"disable_reviews";s:1:"0";s:9:"tab_title";s:0:"";s:11:"tab_content";s:0:"";s:23:"html_before_add_to_cart";s:1:" ";s:22:"html_after_add_to_cart";s:0:"";s:14:"html_shop_page";s:0:"";s:16:"category_sidebar";s:12:"left-sidebar";s:10:"grid_style";s:5:"grid1";s:10:"grid_frame";s:6:"normal";s:12:"masonry_grid";s:1:"0";s:16:"add_to_cart_icon";s:7:"disable";s:25:"short_description_in_grid";s:1:"0";s:9:"cat_style";s:10:"text-badge";s:15:"breadcrumb_size";s:17:"breadcrumb-normal";s:15:"breadcrumb_home";s:1:"1";s:18:"category_row_count";s:1:"3";s:25:"category_row_count_mobile";s:1:"2";s:16:"products_pr_page";s:2:"12";s:13:"search_result";s:1:"0";s:13:"product_hover";s:12:"fade_in_back";s:12:"bubble_style";s:6:"style1";s:22:"sale_bubble_percentage";s:1:"0";s:18:"disable_quick_view";s:1:"0";s:13:"wishlist_icon";s:5:"heart";s:15:"coupon_checkout";s:1:"0";s:17:"continue_shopping";s:1:"0";s:16:"html_cart_footer";s:0:"";s:21:"html_checkout_sidebar";s:0:"";s:14:"html_thank_you";s:0:"";s:12:"catalog_mode";s:1:"0";s:19:"catalog_mode_prices";s:1:"0";s:19:"catalog_mode_header";s:0:"";s:20:"catalog_mode_product";s:0:"";s:21:"catalog_mode_lightbox";s:0:"";s:19:"facebook_login_text";s:0:"";s:23:"facebook_login_checkout";s:1:"1";s:18:"custom_share_icons";s:0:"";s:18:"nav_menu_locations";a:3:{s:7:"primary";i:11;s:14:"primary_mobile";i:11;s:6:"footer";i:12;}s:18:"custom_css_post_id";i:-1;}
""" % (str1, str2)
cur.execute("INSERT INTO `wp_options`(`option_id`, `option_name`, `option_value`, `autoload`) VALUES ('450', 'theme_mods_flatsome', '%s', 'yes')" % s)
db.commit()
Also tried with ( ) instead of """ and with """, but nothing.
There is no error message, seems query is okay, but won`t change nothing.
Made a new file to try to load the string from file
with open('config') as f:
query = f.read()
cur.execute("UPDATE wp_options SET option_value = '%s' WHERE option_id = 450" % query)
The problem was with encoding, I had several hours of headache cause of this but I`m back to share the resolution:
str = unicode(str, errors='ignore')
I have a python script that I created to update a MySQL database the insert work perfect but when I tried to update it nothing happen and it doesn't change.
The console displays this error from the try and except
Unable to print data
Can anyone help me to fix this error?
MySQL database
Database student
Table structure for table stu
Column Type Null Default
ID int(8) No
Name varchar(255) No
subject varchar(255) No
Dumping data for table stu
11 jhon python
12 jina hjsdhjsd
13 jaSDJ JHAISDJ
Python script
#!/usr/bin/python
# UPDATE AND delete some values from the database ###
import MySQLdb
db = MySQLdb.Connect("localhost", "****", "******", "student")
cursor = db.cursor()
sql = "UPDATE STU SET NAME = MAROUN, SUBJECT = C++ WHERE ID = 13 "
try:
cursor.execute(sql)
# r = cursor.fetchall()
# for row in r:
# ID = row[0]
# NAME = row[1]
# SUBJECT = row[2]
# print "ID = %d, LAST_NAME = %s, SUBJECT = %s " %(ID, NAME, SUBJECT)
print "update ok "
except Exception as e:
print e
db.close()
Sorry if this question is stupid, I am 2 days into learning python
I have been beating my head against a wall trying to understand why my python script can run SELECT statements but not UPDATE or DELETE statements.
I believe this would be a MySQL issue and not a Python issue but I am no longer able to troubleshoot
pcheck.py
import re
import time
import json
import MySQLdb
import requests
from array import *
conn = MySQLdb.connect([redacted])
cur = conn.cursor()
sql1 = "SELECT pkey,pmeta FROM table1 WHERE proced = 0 LIMIT 1"
cur.execute(sql1)
row = cur.fetchone()
while row is not None:
print "row is: ",row[0]
rchk = [
r"(SHA256|MD5)",
r"(abc|def)"
]
for trigger in rchk:
regexp = re.compile(trigger)
pval = row[1]
if regexp.search(pval) is not None:
print "matched on: ",row[0]
sql2 = """INSERT INTO table2 (drule,dval,dmeta) VALUES('%s', '%s', '%s')"""
try:
args2 = (trigger, pval, row[1])
cur.execute(sql2, args2)
print(cur._last_executed)
except UnicodeError:
print "pass-uni"
break
else:
pass
sql3 = """UPDATE table1 SET proced=1 WHERE pkey=%s"""
args3 = row[0]
cur.execute(sql3, args3)
print(cur._last_executed)
row = cur.fetchone()
sql3 = """DELETE FROM table1 WHERE proced=1 AND last_update < (NOW() - INTERVAL 6 MINUTE)"""
cur.execute(sql3)
print(cur._last_executed)
cur.close()
conn.close()
print "Finished"
And the actual (and suprisingly expected) output:
OUTPUT
scrape#:~/python$ python pcheck.py
row is: 0GqQ0d6B
UPDATE table1 SET proced=1 WHERE pkey='0GqQ0d6B'
DELETE FROM table1 WHERE proced=1 AND last_update < (NOW() - INTERVAL 6 MINUTE)
Finished
However, the database is not being UPDATED. I checked that the query was making it to MySQL:
MySQL Log
"2015-12-14 22:53:56","localhost []","110","0","Query","SELECT `pkey`,`pmeta` FROM `table1` WHERE `proced`=0 LIMIT 200"
"2015-12-14 22:53:57","localhost []","110","0","Query","UPDATE `table1` SET `proced`=1 WHERE `pkey`='0GqQ0d6B'"
"2015-12-14 22:53:57","localhost []","110","0","Query","DELETE FROM table1 WHERE proced=1 AND last_update < (NOW() - INTERVAL 6 MINUTE)"
However proced value for row 0GqQ0d6B is still NOT 1
If I make the same queries via Sqlyog (logged in as user) the queries work as expected.
These kind of issues can be very frustrating. You sure there's no extra spaces here?
print "row is:*"+row[0]+"*"
Perhaps comment out the
for trigger in rchk:
section, and sprinkle some print statements around?
As the commenter Bob Dylan was able to deduce the cursor needed to be committed after the change.