We Keep Coding

inserting dataset entries into PostgreSQL database server

I have a problem with attempting to pipeline some entries into a Postgresql database. The loader is in this file movie_loader.py provided to me:
import csv
"""
This program generates direct SQL statements from the source Netflix Prize files in order
to populate a relational database with those files’ data.
By taking the approach of emitting SQL statements directly, we bypass the need to import
some kind of database library for the loading process, instead passing the statements
directly into a database command line utility such as `psql`.
"""
# The INSERT approach is best used with a transaction. An introductory definition:
# instead of “saving” (committing) after every statement, a transaction waits on a
# commit until we issue the `COMMIT` command.
print('BEGIN;')
# For simplicity, we assume that the program runs where the files are located.
MOVIE_SOURCE = 'movie_titles.csv'
with open(MOVIE_SOURCE, 'r+', encoding='iso-8859-1') as f:
reader = csv.reader(f)
for row in reader:
id = row[0]
year = 'null' if row[1] == 'NULL' else int(row[1])
title = ', '.join(row[2:])
# Watch out---titles might have apostrophes!
title = title.replace("'", "''")
print(f'INSERT INTO movie VALUES({id}, {year}, \'{title}\');')
sys.stdout.reconfigure(encoding='UTF08')
# We wrap up by emitting an SQL statement that will update the database’s movie ID
# counter based on the largest one that has been loaded so far.
print('SELECT setval(\'movie_id_seq\', (SELECT MAX(id) from movie));')
# _Now_ we can commit our transation.
print('COMMIT;')
However, when attempting to pipeline this file into my database, I get the following error, which seems to be some kind of encoder error. I am using git bash as my terminal.
$ python3 movie_loader.py | psql postgresql://localhost/postgres
stdin is not a tty
Traceback (most recent call last):
File "C:\Users\dhuan\relational\movie_loader.py", line 28, in <module>
print(f'INSERT INTO movie VALUES({id}, {year}, \'{title}\');')
OSError: [Errno 22] Invalid argument
Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='cp1252'>
OSError: [Errno 22] Invalid argument
It seems as if maybe my dataset has an error? I'm not sure specifically what the error is pointing at. Any insight is appreciated

Python SQL Query Error: TypeError: expecting string or bytes object

I have a python program that makes an SQL query to an Oracle DB using a query saved in a text file *.sql as shown below. I'm getting a "type error" as TypeError: expecting string or bytes object pointing to line with sql = query. I have not been able to solve it. Here is the program:
import pandas as pd
import cx_Oracle
from sys import exit
conn= cx_Oracle.connect('DOMINA/S#U#ex021-orc.corp.mycompany.com:1540/domp_domi_bi')
# READ SQL QUERY FROM FILE
with open(r'C:\Users\Au321103\.spyder-py3\Validation\DOMINA_query.sql') as f:
query = f.readlines()
# IMPORT INTO PANDAS DATA FRAME
try:
df = pd.read_sql(con = conn,
sql = query) # QUERY READ FROM .sql FILE
finally:
conn.close()
df.head()
exit()
Thank you in advance as I'm trying to learn these db queries.

Python: Execute Stored Procedure with Parameters

I'm working on a Python script that writes records from a stored procedure to a text file. I'm having issues executing the stored procedure with parameters.
I'm not sure what I could do differently to execute this stored procedure with both parameters. You can see the error I'm getting below.
Any insight would be appreciated.
Here's my code
# Import Python ODBC module
import pyodbc
# Create connection
cnxn = pyodbc.connect(driver="{SQL Server}",server="<server>",database="<database>",uid="<username>",pwd="<password>")
cursor = cnxn.cursor()
# Execute stored procedure
storedProc = "exec database..stored_procedure('param1', 'param2')"
# Loop through records
for irow in cursor.execute(storedProc):
# Create a new text file for each ID
myfile = open('c:/Path/file_' + str(irow[0]) + '_' + irow[1] + '.txt', 'w')
# Write retrieved records to text file
myfile.write(irow[2])
# Close the file
myfile.close()
Here's the error
Traceback (most recent call lst):
File "C:\Path\script.py", line 12, in <module>
for irow in cursor.execute(storedProc):
pyodbc.ProgrammingError: ('42000', "[4200] [Microsoft][ODBC SQL Server Driver][SQL Server]Incorrect syntax near 'param1'. <102> <SQLExecDirectW>">

I was able to fix the syntax error by removing the parenthesis from the query string.
# Execute stored procedure
storedProc = "exec database..stored_procedure('param1', 'param2')"
should be
# Execute stored procedure
storedProc = "exec database..stored_procedure 'param1','param2'"

This worked for me
query = "EXEC [store_proc_name] #param1='param1', #param2= 'param2'"
cursor.execute(query)

For SQL Server:
cursor.execute('{call your_sp (?)}',var_name)

Passing properly formatted windows pathname to SQL server via PYODBC

I need python to issue a query via pyodbc to insert a .PNG file into a table as a blob. The problem seems to have something to do with how the path to the file is represented. Here's some code:
OutFilePath = 'd:\\DilerBW_images\\'
OutFileName = SubjID+'_'+str(UserID)+'_'+DateTime+'.png'
print OutFilePath+OutFileName
qstring = ('insert into [wpic-smir].[Diler_BW].[Beckwith].[PlotImages](ID, UserID, ImageType, MoodType, ImageIndex, ImageData)'
'select '+str(SubjID)+' as ID, '+str(UserID)+' as UserID,'
'1 as ImageType,'
'NULL as MoodType,'
'NULL as ImageIndex,'
'* from OPENROWSET(Bulk \''+OutFilePath+OutFileName+'\', SINGLE_BLOB) as ImageData')
print qstring
cursor.execute(qstring)
conn.commit()
`
And here's the output:
d:\DilerBW_images\999123_999123_2015-01-20_14-25-07.013000.png
insert into [wpic-smir].[Diler_BW].[Beckwith].[PlotImages](ID, UserID, ImageType, MoodType, ImageIndex, ImageData)select 999123 as ID, 999123 as UserID,1 as ImageType,NULL as MoodType,NULL as ImageIndex,* from OPENROWSET(Bulk 'd:\DilerBW_images\999123_999123_2015-01-20_14-25-07.013000.png', SINGLE_BLOB) as ImageData
Now, here's the error I get:
Traceback (most recent call last):
File "c:\pythonscripts\DilerBW_Plot_Single_report_2_EH.py", line 253, in <module>
cursor.execute(qstring)
pyodbc.ProgrammingError: ('42000', '[42000] [Microsoft][SQL Server Native Client 11.0][SQL Server]Cannot bulk load because the file "d:\\DilerBW_images\\999123_999123_2015-01-20_14-25-07.013000.png" could not be opened. Operating system error code 3(The system cannot find the path specified.). (4861) (SQLExecDirectW)')
Sorry this is so long. Notice that the file path in the error message includes double backslashes, where the query only includes singles. I have looked extensively at the various methods to build the string (os.path.sep, using a raw string, os.path.join), but it doesn't seem to matter, and I'm not certain the input string is the problem. Again, if I cut and paste the query as it's presented in the output into SSMS and execute it, it works fine.
Thanx.

error to use django cursor to save escaped characters

I have a url which I want to save into the MySQL database using the "cursor" tool offered by django, but I keep getting the "not enough arguments for format string" error because this url contains some escaped characters (non-ascii characters). The testing code is fairly short:
test.py
import os
import runconfig #configuration file
os.environ['DJANGO_SETTINGS_MODULE'] = runconfig.django_settings_module
from django.db import connection,transaction
c = connection.cursor()
url = "http://www.academicjournals.org/ijps/PDF/pdf2011/18mar/G%C3%B3mez-Berb%C3%ADs et al.pdf"
dbquery = "INSERT INTO main_crawl_document SET url="+url
c.execute(dbquery)
transaction.commit_unless_managed()
The full error message is
Traceback (most recent call last):
File "./test.py", line 14, in <module>
c.execute(dbquery)
File "/usr/local/lib/python2.6/site-packages/django/db/backends/util.py", line 38, in execute
sql = self.db.ops.last_executed_query(self.cursor, sql, params)
File "/usr/local/lib/python2.6/site-packages/django/db/backends/__init__.py", line 505, in last_executed_query
return smart_unicode(sql) % u_params
TypeError: not enough arguments for format string
Can anybody help me?

You're opening yourself up for a possible SQL injection. Instead, use c.execute() properly:
url = "http://www.academicjournals.org/ijps/PDF/pdf2011/18mar/G%C3%B3mez-Berb%C3%ADs et al.pdf"
dbquery = "INSERT INTO main_crawl_document SET url=?"
c.execute(dbquery, (url,))
transaction.commit_unless_managed()
The .execute method should accept an iterable of parameters to use for escaping, assuming it's the normal dbapi method (which it should be with Django).