We Keep Coding

Sqlite Database Access : No such table (Within Django no models)

I have a django & docker server running on my computer and I have created a database with code from outside this server. I am trying to access this database ('test.sqlite3') within the server.
I made sure the path was the correct one and that the file name was correct as well. When I open the database with DB Browser, I can see the tables and all my data. But I still get the following error text:
OperationalError no such table: NAMEOFTABLE
When I use the exact same code from another python IDE (spyder) it works fine. I'm guessing there's something weird going on with django?
Here is some of the code:
conn = sqlite3.connect("../test.sqlite3")
c = conn.cursor()
c.execute("SELECT firstName, lastName FROM RESOURCES")
conn.close()
(Yes, I have also tried using the absolute path and I get the same error.)
Also to be noted: I get this same error when I try to create the database file & table from within the django code (the path should then be the same but it still get the error in this case).
Update: it seems I have a problem with my path because I can't even open a text file with python and it's absolute path. So if anyone has any idea why that'd be great.
try:
f = open("/Users/XXXXX/OneDrive/XXXXX/XXXX/Autres/argon-dashboard-django/toto.txt")
# Do something with the file
except IOError:
q="File not accessible"
finally:
f.close()
always return the following error 'f referenced before assignment' and q = "File not accesible" so that means I can't even find the text file.

To answer this problem, I used two things:
I moved the sqlite3 file within the app folder and used '/app/db.sqlite3' as the path
Added ; at the ends of my SQL requests
c.execute("SELECT firstName, lastName FROM RESOURCES;")
Not sure which one solved the problem but everything works for me now.

Had a similar issue, possibly something about leaving Django Model metadata files outside of the image. I needed to synchronize the model with the DB using a run-syncdb
RUN ["python", "manage.py", "migrate"]
RUN ["python", "manage.py", "migrate", "--run-syncdb"]
CMD ["python", "manage.py", "runserver", "0.0.0.0:8000"]

flask sqlalchemy update sql raw command does not provide any response

I'm trying to perform an update using flask-sqlalchemy but when it gets to the update script it does not return anything. it seems the script is hanging or it is not doing anything.
I tried to wrap a try catch on the code that does not complete but there are no errors.
I gave it 10 minutes to complete the update statement which only updates 1 record and still, it will not do anything for some reason.
When I cancel the script, it provides an error Communication link failure (0) (SQLEndTran) but I don't think this is the root cause of the error because on the same script, I have other sql scripts that works ok so the connection to db is good
what my script does is get some list of filenames that I need to process (I have no issues with this). then using the retrieved list of filenames, I will look into the directory to check if the file exists. if it does not exists, I will update the database to tag the file as it is not found. this is where I get the issue, it does not perform the update nor provide an error message of some sort.
I even tried to create a new engine just for the update script, but still I get the same behavior.
I also tried to print out the sql script first in python before executing. I ran the printed sql command on my sql browser and it worked ok.
The code is very simple, I'm not really sure why it's having the issue.
#!/usr/bin/env python3
from flask_sqlalchemy import sqlalchemy
import glob
files_directory = "/files_dir/"
sql_string = """
select *
from table
where status is null
"""
# ommited conn_string
engine1 = sqlalchemy.create_engine(conn_string)
result = engine1.execute(sql_string)
for r in result:
engine2 = sqlalchemy.create_engine(conn_string)
filename = r[11]
match = glob.glob(f"{files_directory}/**/{filename}.wav")
if not match:
print('no match')
script = "update table set status = 'not_found' where filename = '" + filename + "' "
engine2.execute(script)
engine2.dispose()
continue
engine1.dispose()

it appears that if I try to loop through 26k records, the script doesn't work. but when I try to do by batches of 2k records per run, then the script will work. so my sql string will become (added top 2000 on the query)
sql_string = """
select top 2000 *
from table
where status is null
"""
it's manual yeah, but it works for me since I just need to run this script once. (I mean 13 times)

Connect / Store sqlite3 database in specified directory other than default -- "conn = sqlite3.connect('name_of_database')"

I'm working with and Sqlite3 database trying to access its data from a different directory than it was originally created.
The python script(test case) that I have ran through our Squish GUI Automation IDE is located in a directory
C:\Squish_Automation_Functional_nVP2\suite_Production_Checklist\tst_dashboard_functional_setup
There I create a database with the following table inside of that same script:
def get_var_value_pass():
conn = sqlite3.connect('test_result.db')
c = conn.cursor()
c.execute('CREATE TABLE IF NOT EXISTS test_result_pass (pass TEXT,result INTEGER)')
c.execute("INSERT INTO test_result_pass VALUES('Pass', 0)")
conn.commit()
c.execute("SELECT * FROM test_result_pass WHERE pass='Pass'", )
pass_result = c.fetchone()
test.log(str(pass_result[1]))
test_result = pass_result[1]
c.close()
conn.close()
return test_result
Now I'd like to access the same database which I've created formerly by "conn = sqlite3.connect('test_result.db')" inside of another test case which is located in a different directory
C:\Squish_Automation_Functional_nVP2\suite_Production_Checklist\tst_Calling_In-Call-Options_Text_Share-Text_Update-Real-Time
The problem is, when I try a select statement for the same database-- inside of a different script(test case) like so:
def get_var_value_pass(pass_value):
conn = sqlite3.connect('test_result.db')
c = conn.cursor()
c.execute("SELECT * FROM test_result_pass WHERE pass='Pass'", )
My test fails as soon as I try the c.execute("") statement because the table can't be found. Instead, the most recent "conn = sqlite3.connect('test_result.db')" has just created a new empty database, instead of referring to my original created database. Therefore, I've come to the conclusion that I'll want to try and store my original database where both test cases can use them as a test_suite_resource--basically inside of another directory where the other test cases will have reference. Ideally here:
C:\Squish_Automation_Functional_nVP2\suite_Production_Checklist
Is there a sqlite3 function that lets you declare the place where you'd like to Connect / Store your database? Similar to sqlite3.connect('test_result.db') except you define its path?
BTW-- I have tried the second snippet of code inside of the same test script and it runs perfectly. I have also tried an in-memory approach by sqlite3.connect(':memory:') -- still no luck. Please help! Thanks.

It's not clear to me that you received an answer. You have a number of options. I happen to have a sqlite database stored in one directory which I can open in that or another directory by specifying it in one of the following way.
import sqlite3
conn = sqlite3.connect(r'C:\Television\programs.sqlite')
conn.close()
print('Hello')
conn = sqlite3.connect('C:\\Television\\programs.sqlite')
conn.close()
print('Hello')
conn = sqlite3.connect('C:/Television/programs.sqlite')
conn.close()
print('Hello')
All three connection attempts succeed. I see three Hellos as output.
Stick an 'r' ahead of the string. See the documents about string for the reason.
Replace each backward stroke with a pair of backward strokes.
Replace each backward stroke with a forward stroke.

Use python and psycopg2 to execute a sql file that contains a DROP DATABASE statement

I am trying to use a python function to execute a .sql file.
The sql file begins with a DROP DATABASE statement.
The first lines of the .sql file look like this:
DROP DATABASE IF EXISTS myDB;
CREATE DATABASE myDB;
The rest of the .sql file defines all the tables and views for 'myDB'
Python Code:
def connect():
conn = psycopg2.connect(dbname='template1', user='user01')
conn.set_isolation_level(psycopg2.extensions.ISOLATION_LEVEL_AUTOCOMMIT)
cursor = conn.cursor()
sqlfile = open('/path/to/myDB-schema.sql', 'r')
cursor.execute(sqlfile.read())
db = psycopg2.connect(dbname='myDB', user='user01')
cursor = db.cursor()
return db,cursor
When I run the connect() function, I get an error on the DROP DATABASE statement.
ERROR:
psycopg2.InternalError: DROP DATABASE cannot be executed from a function or multi-command string
I spent a lot of time googling this error, and I can't find a solution.
I also tried adding an AUTOCOMMIT statement to the top of the .sql file, but it didn't change anything.
SET AUTOCOMMIT TO ON;
I am aware that postgreSQL doesn't allow you to drop a database that you are currently connected to, but I didn't think that was the problem here because I begin the connect() function by connecting to the template1 database, and from that connection create the cursor object which opens the .sql file.
Has anyone else run into this error, is there any way to to execute the .sql file using a python function?

This worked for me for a file consisting of SQL Queries in each line:
sql_file = open('file.sql','r')
cursor.execute(sql_file.read())

You are reading in the entire file and passing the whole thing to PostgreSQL as one string (as the error message says, "multi-command string". Is that what you are intending to do? If so, it isn't going to work.
Try this:
cursor.execute(sqlfile.readline())
Or, shell out to psql and let it do the work.

In order to deploy scripts using CRON that serve as ETL files that use .SQL, we had to expand how we call the SQL file itself.
sql_file = os.path.join(os.path.dirname(__file__), "../sql/ccd_parcels.sql")
sqlcurr = open(sql_file, mode='r').read()
curDest.execute(sqlcurr)
connDest.commit()
This seemed to please the CRON job...

Returning Output of Python CGI MySQL Script

I'm very new to Python and MySQL and this is my first Stack question. So, apologies in advance if I'm missing something obvious. But, I really did try to research this before asking.
I'm trying to learn the basics of Python, MySQL, and CGI scripting. To that end, I've been reading tutorials at http://www.tutorialspoint.com/python/python_cgi_programming.htm and http://www.tutorialspoint.com/python/python_database_access.htm, among others.
I'm trying to have a CURL GET or Python Requests GET call a Python CGI script on a test server. That Python CGI script would then perform a Read action on a local MySQL database and return the results to CURL or to Python Requests.
The Python CGI script I've created outputs the MySQL Read data perfectly to the terminal on the remote test server. But, it won't return that output to the CURL or to the Python Requests that triggered the CGI script.
This is the Python CGI script I've cobbled together:
#!/usr/bin/python
import MySQLdb
# Open database connection
db = MySQLdb.connect("localhost","testuser","test123","TESTDB" )
# prepare a cursor object using cursor() method
cursor = db.cursor()
# Prepare SQL query to INSERT a record into the database.
sql = "SELECT * FROM EMPLOYEE \
WHERE INCOME > '%d'" % (1000)
try:
# Execute the SQL command
cursor.execute(sql)
# Fetch all the rows in a list of lists.
results = cursor.fetchall()
for row in results:
fname = row[0]
lname = row[1]
age = row[2]
sex = row[3]
income = row[4]
# Now print fetched result
print "fname=%s,lname=%s,age=%d,sex=%s,income=%d" % \
(fname, lname, age, sex, income )
except:
print "Error: unable to fetch data"
# disconnect from server
db.close()
My guess is that I need to somehow pass the data from the SQL query back to Python or back to the requesting process through some sort of Return or Output statement. But, all my best efforts to research this have not helped. Can anyone point me in the right direction? Many thanks for any help!
Marc :-)

First, as per the CGI tutorial you link, you need to output the content type you are working with:
print "Content-type: text/plain\r\n\r\n",
If you don't at least have the newlines in there, HTTP clients will think your content is supposed to be part of the headers and get confused, they will probably assume the document you asked for is empty.
Next, you need a CGI server. Python comes with one. Place your script in a subdirectory called cgi-bin and run this command (from the parent directory):
python -m CGIHTTPServer
The url to call will look something like this:
curl http://localhost:8000/cgi-bin/cgitest.py