Hello and thank you for clicking on this question. I want to insert into the database the contents from a text file encoded with utf-8. When proceeding to insert the text content into the DB it tells me that it is binary data for some reason. When I created the DB in sqlite3, I have specified that the description (the column in question) to be TEXT, therefore I do not know what could be the problem...
The code is the following [it only contains the part where I insert into the DB]:
(quick summary of the code: it is looking into a folder with many text files and then it collects, some variables from the text's name and contents and then if the text has not been added to the DB then add a new row that contains the missing variables corresponding to the text file )
def put_inside_db():
counter = 0
for item in list_txt:
item_components = item.split("-")
item_year = item_components[-1].split(".")
unique_key = str(item_components[0]) + str(item_year[0])
cik = item_components[0]
comp_name = item_components[1]
year = item_year[0]
file_path = path_to_10k + item
file = open(file_path, "r+", encoding="utf-8")
description = file.read()
description = str(description)
print(description)
file.close()
if unique_key not in keys_db:
c.execute("INSERT INTO finaldata (cik, comp_name, year, unique_key, description) "
"VALUES(?,?,?,?,?)", (cik, comp_name, year, unique_key, description))
print("This key is not inside: " + unique_key)
counter += 1
else:
"do nothing"
# print("This key is inside: " + unique_key)
if counter % 50 == 0:
conn.commit()
conn.commit()
I even printed to the console the insides of the text files and they are strings, therefore I do not know why this issue is present. Below you can see the message that the DB displays when I click on the a value from the column "description"
UPDATE
I tried implementing the solution from the other question answered Forcing a data type (BLOB or TEXT) when inserting values into an SQLite table. Meaning I have done the following:
1) Tried fixing the values in the database by rewriting them as per the solution number one, but that did not fixed my problem
2) Another suggestion from the other post was that I should make sure that I insert text values into the DB. To my knowledge, the values that I try to insert into the DB are strings. To make sure I even forced the extracted description from the text files to be a string. However, that does not fix my problem..
Therefore, in my opinion I think that my question is not a duplicate since I insert strings into a column with affinity to text and it stores it as binary. If I am wrong about this, can someone please explain in a more detailed manner what is exactly happening and why I am getting this result. I have used similar code for other database insertions, but I have never received such an error...
Thank you!
Thanks to
https://stackoverflow.com/users/570339/ramy-al-zuhouri
and
sqlite for swift is unstable
sqlite3_bind_int(statement, 1, Int32(id))
sqlite3_bind_text(statement, 2,(type as NSString).UTF8String, -1, nil)
let desc = (description ?? "") as NSString
sqlite3_bind_text(statement, 3, desc.UTF8String, -1, nil)
Related
Whenever i run any select query in snowflake the result set is having auto generated row number column (as a first column).. how to ignore this column from the code...
Like : select * from emp ignore row;
If you're referring to the unnamed column just before TABLE_CATALOG in the below picture.
I'm pretty sure that's not something we can not change -> maybe if you wrote some custom JS to fiddle with the page you might be able to hide it by perhaps changing the TEXT color to white or something. But that seems like a lot of work.
If you extract the data to a CSV (or any file format) this number does not appear in the payload.
When you query Snowflake, regardless of which client you use, that column won't be returned.
It is a pure UI thing in the Snowflake Editor for readability.
If this is happening in python. then below could help. Also refer https://docs.snowflake.com/en/user-guide/python-connector-example.html#using-cursor-to-fetch-values
result = cur.execute("select * from table")
#assign to a variable
result_var = result.fetchall()
#write into a file
result.fetch_pandas_all().to_csv(multiple_table_path, index=False, header=None, sep='|')
this seems like a basic function, but I'm new to Python so maybe I'm not googling this correctly.
In Microsoft SQL Server, when you have a statement like
SELECT top 100 * FROM dbo.Patient_eligibility
you get a result like
Patient_ID | Patient_Name | Patient_Eligibility
67456 | Smith, John | Eligible
...
etc.
Etc.
I am running a connection to SQL through Python as such, and would like the output to look exactly the same as in SQL. Specifically - with column names and all the data rows specified in the SQL query. It doesn't have to appear in the console or the log, I just need a way to access it to see what's in it.
Here is my current code attempts:
import pyodbc
conn = pyodbc.connect(connstr)
cursor = conn.cursor()
sql = "SELECT top 100 * FROM [dbo].[PATIENT_ELIGIBILITY]"
cursor.execute(sql)
data = cursor.fetchall()
#Query1
for row in data :
print (row[1])
#Query2
print (data)
#Query3
data
My understanding is that somehow the results of PATIENT_ELIGIBILITY are stored in the variable data. Query 1, 2, and 3 represent methods of accessing that data that I've googled for - again seems like basic stuff.
The results of #Query1 give me the list of the first column, without a column name in the console. In the variable explorer, 'data' appears as type List. When I open it up, it just says 'Row object of pyodbc module' 100 times, one for each row. Not what I'm looking for. Again, I'm looking for the same kind of view output I would get if I ran it in Microsoft SQL Server.
Running #Query2 gets me a little closer to this end. The results appear like a .csv file - unreadable, but it's all there, in the console.
Running #Query3, just the 'data' variable, gets me the closest result but with no column names. How can I bring in the column names?
More directly, how do i get 'data' to appear as a clean table with column names somewhere? Since this appears a basic SQL function, could you direct me to a SQL-friendly library to use instead?
Also note that neither of the Queries required me to know the column names or widths. My entire method here is attempting to eyeball the results of the Query and quickly check the data - I can't see that the Patient_IDs are loading properly if I don't know which column is patient_ids.
Thanks for your help!
It's more than 1 question, I'll try help you and give advice.
I am running a connection to SQL through Python as such, and would like the output to look exactly the same as in SQL.
You are mixing SQL as language and formatted output of some interactive SQL tool.
SQL itself does not have anything about "look" of data.
Also note that neither of the Queries required me to know the column names or widths. My entire method here is attempting to eyeball the results of the Query and quickly check the data - I can't see that the Patient_IDs are loading properly if I don't know which column is patient_ids.
Correct. cursor.fetchall returns only data.
Field informations can be read from cursor.description.
Read more in PEP-O249
how do i get 'data' to appear as a clean table with column names somewhere?
It depends how do you define "appear".
You want: text output, html page or maybe GUI?
For text output: you can read column names from cursor.description and print them before data.
If you want html/excel/pdf/other - find some library/framework suiting your taste.
If you want an interactive experience similar to SQL tools - I recommend to look on jupyter-notebook + pandas.
Something like:
pandas.read_sql_query(sql)
will give you "clean table" nothing worse than SQLDeveloper/SSMS/DBeaver/other gives.
We don't need any external libraries.
Refer to this for more details.
Print results in MySQL format with Python
However, the latest version of MySQL gives an error to this code. So, I modified it.
Below is the query for the dataset
stri = "select * from table_name"
cursor.execute(stri)
data = cursor.fetchall()
mycon.commit()
Below it will print the dataset in tabular form
def columnnm(name):
v = "SELECT LENGTH("+name+") FROM table_name WHERE LENGTH("+name+") = (SELECT MAX(LENGTH("+name+")) FROM table_name) LIMIT 1;"
cursor.execute(v)
data = cursor.fetchall()
mycon.commit()
return data[0][0]
widths = []
columns = []
tavnit = '|'
separator = '+'
for cd in cursor.description:
widths.append(max(columnnm(cd[0]), len(cd[0])))
columns.append(cd[0])
for w in widths:
tavnit += " %-"+"%ss |" % (w,)
separator += '-'*w + '--+'
print(separator)
print(tavnit % tuple(columns))
print(separator)
for row in data:
print(tavnit % row)
print(separator)
I just started learning SQLite3 for python today, and I'm having trouble figuring out why this won't work.
import sqlite3, os
if not os.path.isfile("G:\\Python\\My first database.db"):
dtabse = sqlite3.connect("G:\\Python\\My first database.db")
cursr = dtabse.cursor()
cursr.execute("""CREATE TABLE Students
(first_name text,
surname text,
DOB text,
Form text)
""")
cursr.execute(""" INSERT INTO Students
VALUES ("Dave", "Edwards", "16", "11AB")""")
dtabse.commit()
dtabse.close()
else:
dtabse = sqlite3.connect("G:\\Python\\My first database.db")
cursr = dtabse.cursor()
print(cursr.fetchall())
In a powerpoint I was viewing, it said fetchall() should retrieve everything and display it. On the first go of this program it won't find a file in this directory, so the if area gets executed. When I next run the program the else area gets executed.
That much works, on the first go the program ends and begins. On the second go it prints an empty list, when I was expecting the table. I checked the database file and the data is there, so why can't I print it?
You need a SELECT statement to retrieve the data you want.
The problem:
I am working on a bit of code that is meant to create a new record in a linking table. The linking table has a primary key that is an auto number, the other two fields are VARCHAR with a length of 10.
The issue I'm having is I cannot seem to get partID into the table. If you look at the sql output you can clearly see it write None and u'1' (the orderID) to the table. So that tells me its recieving the orderID just fine. Also you can see that I did a print to find out what is inside my variable before passing it to the new object. It has 3 in it which is the correct partId. Somewhere between creating the new object and writing to the table it passes a null.
I've tried to cast it, Ive tried different ways of pulling the partID from the database, etc and I cannot for the life of me figure out what is wrong.
The code:
def updateOrderParts_view(request):
part = None
partToOrder = None
idnum = None
part = DBSession.execute(\
"SELECT partID "+\
"FROM tblParts "+\
"WHERE partName = " + "'" +request.POST['partName'] +"'").fetchone()
print "<---DEBUG--->"
print part['partID']
partToOrder = PartsByOrder(part['partID'], request.POST['orderID'])
DBSession.add(partToOrder)
return{}
The terminal output:
<---DEBUG--->
3
2013-04-24 08:14:47,985 INFO [sqlalchemy.engine.base.Engine][Dummy-2] INSERT INTO "tblPartsByOrder" ("partID", "orderID") VALUES (?, ?)
2013-04-24 08:14:47,985 INFO [sqlalchemy.engine.base.Engine][Dummy-2] (None, u'1')
2013-04-24 08:14:47,986 INFO [sqlalchemy.engine.base.Engine][Dummy-2] COMMIT
I would appreciate any thoughts or comments on this issue
Thanks for your time
First, I would look at doing the SQL lookup a little different, if you can. (I guess it depends, do you have a model based on "tblParts"? I'm going to use an example assuming there is a model object "Part"):
part = DBSession.query(Part.partID).filter(Part.partName == request.POST['partName']).first()
From there, I'm rusty on the exact syntax but I think you could do something like
print "<---DEBUG--->"
print part
partToOrder = PartsByOrder(part, request.POST['orderID'])
DBSession.add(partToOrder)
return{}
You might need to case "part" to a string (str(part)) if it's a casting problem.
Good luck,
I am a beginner at python and using SQLite. So please be patient with me. I am not entirely sure how much information i should provide, so i have decided on putting up as much code as i think is related. Like the saying goes; better safe than sorry.
Basically, what I have is a python script running a cherrypy server for a kind of peer to peer social networking web app. I have a method which logs three kind of Updates i make to my profile; New Post, New Photo, or New Event.
Each Update contains the following fields:
messageID: A 16 letter string containing a unique identifier
creator: My user name
created: A time stamp, UNIX Epoch time, of when the update took place
body: A short message about the update.
Link: A link to the update. e.g.. "/gallery/photo5"
Type: type 1 means new post, 2 means photo, 3 means event.
I have made these fields into columns of a table inside a database created with SQLite, here is the method i used to do it:
#cherrypy.expose
def writeUpdate(self, type=None):
"""This method is called whenever a change takes place on the Acebook
It takes an input 'type' to know what kind of update it is.
The method then make appropriet change to the 'Updates' database
"""
con = lite.connect('static/database/Updates.db')
messageID = self.randomword()
creator = trueUser
created = time.time()
if type == 1:
link = "/homepage"
body = "New Status Update"
elif type == 2:
link = "/portfolio"
body = "New Photo Update"
elif type ==3:
link = "/event"
body = "New Event Update"
else:
link = "/homepage"
body = "If you are seeing this, something is not quite right with by server"
with con:
cur = con.cursor()
cur.execute("CREATE TABLE IF NOT EXISTS Updates(messageID TEXT, creator TEXT, created INT, link TEXT, type INT, body TEXT)")
cur.execute("INSERT INTO Updates VALUES(?, ?, ?, ?, ?, ?)", (messageID, creator, created, link, type, body))
"Debugging check"
cur.execute('select * from Updates')
print "The Updates database now contains:"
for row in cur:
print row
return
I have another method which my friends can call in order to get a newsfeed of my latest updates. This method is:
#cherrypy
def getActivity(self, minutes=48*60):
“”” Return any updates since last time this user requested them. Optional argument returns the last updates in the given time period instead.
“””
# current_user = getAuthenticatedUser(): # if not current_user:
# return “Please Authenticate”
# updates = getUpdatesByUser(current_user)
ExampleUpdate = [ {
‘messageID’: “ccog001-1332889924-839”, ‘creator’: “ccog001”,
‘created’: 1332889924,
‘link’: “/updates?id=839”,
‘type’: 1,
‘body’: “Hello, is anybody out there?”
},{
‘messageID’: “ccog001-1332890482-840”, ‘creator’: “ccog001”,
‘created’: 1332890482,
‘link’: “/updates?id=840”, ‘type’: 1,
‘body’: “Seriously, is this thing on?” }
]
reply = json.dumps(updates)
return reply
My question is, how do i read individual rows of the database into a separate dictionary, and then combine all the dictionary together into the format of the variable Example Update, before encoding it all with json.dumps?
Or would it maybe be easier if I write the messageID, creator, created ... etc... into a dictionary first, before writing that dictionary into a database? So I end up with a database containing only one column of dictionaries? If so, what would be the code like?
I am quite new, so please be detailed in your answer, preferably with code and comments to help me understand.
Thank you very much for you time
The column names are stored in Cursor.description after executing a SELECT statement. According to the docs, each entry in this list is a 7-tuple where the first element is populated by the column's name.
You can extract the column names and form a dict as follows:
cur.execute('select * from Updates')
# extract column names
column_names = [d[0] for d in cur.description]
for row in cur:
# build dict
info = dict(zip(column_names, row))
# dump it to a json string
reply = json.dumps(info)
Here, zip takes the two lists column_names and row and stitches them together element-wise into a list of tuples. dict then turns this into a dictionary ready for json to dump.