I'm trying to read from a file and insert the data to a postgresql table in python using the psycopg2.
Here's the function I wrote:
def insert(file, table, col, conn):
sql = "INSERT INTO "+table+"("+col+") VALUES(%s)"
cur = conn.cursor()
with open(os.path.join(DEFAULTS_FOLDER, file)) as fp:
line = fp.readline()
while line:
cur.execute(sql, (line.rstrip(),))
line = fp.readline()
conn.commit()
cur.close()
return
For some reason I get an error:
cur.execute(sql, (line.rstrip(),)) psycopg2.DataError: malformed array literal: "hello" LINE 1: INSERT INTO greetings(gname) VALUES('hello')
I also tried to insert a plain string and I still get the same error.
The error message means that the column gname of the table greetings is an array, not a plain text. If it is a text array, the query should look like this:
INSERT INTO greetings(gname) VALUES('{hello}')
You should change the relevant fragment of your code, e.g.:
cur.execute(sql, ("{{{}}}".format(line.rstrip()),))
Related
I tried to insert each element of the json api into my postgres table.
But I get the follwoing error:
Traceback (most recent call last):
File "c:/Users/myname/Documents/repos/docker-playground/parse_json_to_postgres.py", line 20, in <module>
cursor.execute(f"INSERT into catfacts(data) VALUES ( {cat_fact} )")
psycopg2.errors.SyntaxError: syntax error at or near "{"
LINE 1: INSERT into catfacts(data) VALUES ( {'status': {'verified':...
^
My postgres table:
CREATE TABLE cat_facts (
id serial NOT NULL PRIMARY KEY,
data jsonb NOT NULL
);
My Python code to insert the data into the table:
import requests, json, psycopg2
cat_facts_json = requests.get('https://cat-fact.herokuapp.com/facts').json
conn = psycopg2.connect(user="postgres",
password="password",
host="localhost",
port="5432",
database="postgres")
cursor = conn.cursor()
for cat_fact in cat_facts_json():
cursor.execute(f"INSERT into catfacts(data) VALUES ( \' {cat_fact} \' )")
API = https://cat-fact.herokuapp.com/facts
What I am trying to achieve:
INSERT INTO cat_facts(data) VALUES ('{"status":{"verified":true,"sentCount":1},"type":"cat","deleted":false,"_id":"58e008800aac31001185ed07","user":"58e007480aac31001185ecef","text":"Wikipedia has a recording of a cat meowing, because why not?","__v":0,"source":"user","updatedAt":"2020-08-23T20:20:01.611Z","createdAt":"2018-03-06T21:20:03.505Z","used":false}');
INSERT INTO cat_facts(data) VALUES ('{"status":{"verified":true,"sentCount":1},"type":"cat","deleted":false,"_id":"58e008630aac31001185ed01","user":"58e007480aac31001185ecef","text":"When cats grimace, they are usually \"taste-scenting.\" They have an extra organ that, with some breathing control, allows the cats to taste-sense the air.","__v":0,"source":"user","updatedAt":"2020-08-23T20:20:01.611Z","createdAt":"2018-02-07T21:20:02.903Z","used":false},{"status":{"verified":true,"sentCount":1},"type":"cat","deleted":false,"_id":"58e00a090aac31001185ed16","user":"58e007480aac31001185ecef","text":"Cats make more than 100 different sounds whereas dogs make around 10.","__v":0,"source":"user","updatedAt":"2020-08-23T20:20:01.611Z","createdAt":"2018-02-11T21:20:03.745Z","used":false}');
....
See here JSON Adaption.
So something like:
from psycopg2.extras import Json
cursor.execute("INSERT into catfacts(data) VALUES (%s)", [Json(cat_fact)])
I got it working now:
for cat_fact in cat_facts_json:
data = json.dumps(cat_fact)
insert_query = "insert into cat_facts (data) values (%s) returning data"
cursor.execute(insert_query, (data,))
conn.commit()
conn.close()
I considered your comments #Stefano Frazzetto and #Adrian Klaver.
json.dumps works !
I didn't execute the parameters directly in the execute query
I still think, this is a pretty odd syntax with the comma after data:
cursor.execute(insert_query, (data,))
I am trying to import txt file into sql, but i have an error:
TypeError: can only concatenate str (not "list") to str
My code:
import psycopg2
con = psycopg2.connect(
host = "",
database="",
user = "",
password = "")
cursor = con.cursor()
with open("pom1.txt") as infile:
for line in infile:
data = line.split()
print(data)
query = ("INSERT INTO Pomiary_Obwod_90(Znacznik, Pomiar_x, Pomiar_y, Pomiar_z) VALUES"
"(" + data + ");")
cursor.execute(query, *data)
con.commit()
Does anyone have an idea how can i solve it? :)
You don't put the actual values into the parameterized query; you put whatever placeholders are appropriate for your library.
data = line.split()
place_holders = ', '.join("%s" for _ in data) # Assuming %s is correct
query = ("INSERT INTO Pomiary_Obwod_90(Znacznik, Pomiar_x, Pomiar_y, Pomiar_z) VALUES"
"(" + place_holders + ");")
cursor.execute(query, *data)
cursor.execute takes care of inserting each value where a placeholder occurs, ensure things are properly quoted/escaped/etc.
There are several problems here. First, as the error says, you are trying to concatenate a List (which is data) directly to a string.
Second, you should not use + to concatenate your values and your query.
The doc says:
Warning: Never, never, NEVER use Python string concatenation (+) or string parameters interpolation (%) to pass variables to a SQL query string. Not even at gunpoint.
You should only pass the values to the query via %s.
I'm not sure about the use of * in front of data in cursor.execute(query, *data).
Here is a code that should work, though I have nothing at hand for testing it right now:
import psycopg2
con = psycopg2.connect(
host = "",
database="",
user = "",
password = "")
cursor = con.cursor()
with open("pom1.txt") as infile:
for line in infile:
data = line.split()
print(data)
query = ("INSERT INTO Pomiary_Obwod_90(Znacznik, Pomiar_x, Pomiar_y, Pomiar_z) VALUES (%s, %s, %s, %s);")
cursor.execute(query, data)
con.commit()
I have the following script:
create_table_WAC = """
create table if not exists %s (
w_geocode text,
C000 text,
CFS04 text,
CFS05 text,
createdate text
)
"""
target_directory = Path(sys.argv[1]).resolve()
for file in target_directory.rglob('*.csv'):
table_name = 'opendata_uscensus_lodes_' + str(file.stem)
print(table_name)
# MAKE SURE THIS IS THE RIGHT TABLE FOR THE FILES
cur.execute(create_table_WAC, (table_name,))
with open(file,'r') as file_in:
# MAKE SURE THIS HAS THE RIGHT TABLE NAME IN THE COPY STATEMENT
cur.copy_expert("copy %s from stdin with csv header delimiter ','", table_name, file_in)
conn.commit()
conn.close()
When I run it, it throws this error related to the CREATE TABLE command. I don't understand why there are '' added -- and how do I remove them?
Here is the error:
psycopg2.ProgrammingError: syntax error at or near "'opendata_uscensus_lodes_ca_wac_SA02_JT03_2003'"
LINE 2: create table if not exists 'opendata_uscensus_lodes_ca_wac_S...
Use SQL string composition:
from psycopg2 import sql
create_table_WAC = """
create table if not exists {} ( -- note changed placeholder
w_geocode text,
C000 text,
CFS04 text,
CFS05 text,
createdate text
)
"""
# ...
cur.execute(sql.SQL(create_table_WAC).format(sql.Identifier(table_name)))
Read the comprehensive explanation in the documentation.
I am trying to connect to a DB and get a data.
But when I am trying to write into a file it gives me the below error.
TypeError: must be string or read-only character buffer, not list
Below is my sample code.
import pyodbc
path= "M:/xyz/"
file_write=open( path +"files.test.log", "w")
#print "Connection started"
db=pyodbc.connect('DSN=DB_NAME;PWD=xyz')
#print "Connection done"
cursor = db.cursor()
book_value="WPZ.N"
query="SELECT book_id FROM Table_name WHERE book_value='"+book_value+"'"
#print query
cursor.execute(query)
book_id= cursor.fetchall()
file_write.write(book_id)
#str(file_write)
#print file_write
Please Guide on This.
Thanks.
fetchall() returns a list of rows, what you should do is to get the really 'book_id' with string type from this list and write it to the file.
Try this:
rows = cursor.fetchall()
for row in rows:
file_write.write(row.book_id) # not sure if row[0] works as well
file_write.write('\n') # new line or anything else you like
source: pyodbc example
book_id= cursor.fetchall() #List of rows
file_write.write(book_id) # you will get error `TypeError: expected a character buffer object`
file_write.write(', '.join(book_id))
I am trying to write a csv file into a table in SQL Server database using python. I am facing errors when I pass the parameters , but I don't face any error when I do it manually. Here is the code I am executing.
cur=cnxn.cursor() # Get the cursor
csv_data = csv.reader(file(Samplefile.csv')) # Read the csv
for rows in csv_data: # Iterate through csv
cur.execute("INSERT INTO MyTable(Col1,Col2,Col3,Col4) VALUES (?,?,?,?)",rows)
cnxn.commit()
Error:
pyodbc.DataError: ('22001', '[22001] [Microsoft][ODBC SQL Server Driver][SQL Server]String or binary data would be truncated. (8152) (SQLExecDirectW); [01000] [Microsoft][ODBC SQL Server Driver][SQL Server]The statement has been terminated. (3621)')
However when I insert the values manually. It works fine
cur.execute("INSERT INTO MyTable(Col1,Col2,Col3,Col4) VALUES (?,?,?,?)",'A','B','C','D')
I have ensured that the TABLE is there in the database, data types are consistent with the data I am passing. Connection and cursor are also correct. The data type of rows is "list"
Consider building the query dynamically to ensure the number of placeholders matches your table and CSV file format. Then it's just a matter of ensuring your table and CSV file are correct, instead of checking that you typed enough ? placeholders in your code.
The following example assumes
CSV file contains column names in the first line
Connection is already built
File name is test.csv
Table name is MyTable
Python 3
...
with open ('test.csv', 'r') as f:
reader = csv.reader(f)
columns = next(reader)
query = 'insert into MyTable({0}) values ({1})'
query = query.format(','.join(columns), ','.join('?' * len(columns)))
cursor = connection.cursor()
for data in reader:
cursor.execute(query, data)
cursor.commit()
If column names are not included in the file:
...
with open ('test.csv', 'r') as f:
reader = csv.reader(f)
data = next(reader)
query = 'insert into MyTable values ({0})'
query = query.format(','.join('?' * len(data)))
cursor = connection.cursor()
cursor.execute(query, data)
for data in reader:
cursor.execute(query, data)
cursor.commit()
I modified the code written above by Brian as follows since the one posted above wouldn't work on the delimited files that I was trying to upload. The line row.pop() can also be ignored as it was necessary only for the set of files that I was trying to upload.
import csv
def upload_table(path, filename, delim, cursor):
"""
Function to upload flat file to sqlserver
"""
tbl = filename.split('.')[0]
cnt = 0
with open (path + filename, 'r') as f:
reader = csv.reader(f, delimiter=delim)
for row in reader:
row.pop() # can be commented out
row = ['NULL' if val == '' else val for val in row]
row = [x.replace("'", "''") for x in row]
out = "'" + "', '".join(str(item) for item in row) + "'"
out = out.replace("'NULL'", 'NULL')
query = "INSERT INTO " + tbl + " VALUES (" + out + ")"
cursor.execute(query)
cnt = cnt + 1
if cnt % 10000 == 0:
cursor.commit()
cursor.commit()
print("Uploaded " + str(cnt) + " rows into table " + tbl + ".")
You can pass the columns as arguments. For example:
for rows in csv_data: # Iterate through csv
cur.execute("INSERT INTO MyTable(Col1,Col2,Col3,Col4) VALUES (?,?,?,?)", *rows)
If you are using MySqlHook in airflow , if cursor.execute() with params throw san error
TypeError: not all arguments converted during string formatting
use %s instead of ?
with open('/usr/local/airflow/files/ifsc_details.csv','r') as csv_file:
csv_reader = csv.reader(csv_file)
columns = next(csv_reader)
query = '''insert into ifsc_details({0}) values({1});'''
query = query.format(','.join(columns), ','.join(['%s'] * len(columns)))
mysql = MySqlHook(mysql_conn_id='local_mysql')
conn = mysql.get_conn()
cursor = conn.cursor()
for data in csv_reader:
cursor.execute(query, data)
cursor.commit()
I got it sorted out. The error was due to the size restriction restriction of table. It changed the column capacity like from col1 varchar(10) to col1 varchar(35) etc. Now it's working fine.
Here is the script and hope this works for you:
import pandas as pd
import pyodbc as pc
connection_string = "Driver=SQL Server;Server=localhost;Database={0};Trusted_Connection=Yes;"
cnxn = pc.connect(connection_string.format("DataBaseNameHere"), autocommit=True)
cur=cnxn.cursor()
df= pd.read_csv("your_filepath_and_filename_here.csv").fillna('')
query = 'insert into TableName({0}) values ({1})'
query = query.format(','.join(df.columns), ','.join('?' * len(df1.columns)))
cur.fast_executemany = True
cur.executemany(query, df.values.tolist())
cnxn.close()
You can also import data into SQL by using either:
The SQL Server Import and Export Wizard
SQL Server Integration Services (SSIS)
The OPENROWSET function
More details can be found on this webpage:
https://learn.microsoft.com/en-us/sql/relational-databases/import-export/import-data-from-excel-to-sql?view=sql-server-2017