We Keep Coding

sqlite database didn't update table in my flask webhook python code

I try to update my sqlite database using flask webhook.
It seems commands line work fine if I type manually in the python console but my flask webhook didn't update my SQLite database. It seems the apps fail at the "cursor.execute()" line.
here is my webhook code:
#app.route('/trendanalyser', methods=['POST'])
def trendanalyser():
data = json.loads(request.data)
if data['passphrase'] == config.WEBHOOK_PASSPHRASE:
#Init update variables
tastate = data['TrendAnalyser']
date_format = datetime.today()
date_update = date_format.strftime("%d/%m/%Y %H:%M:%S")
update_data = ((tastate), (date_update))
#Database connection
connection = sqlite3.connect('TAState15min.db')
cursor = connection.cursor()
#Database Update
update_query = """Update TrendAnalyser set state = ?, date = ? where id = 1"""
cursor.execute(update_query, update_data)
connection.commit()
return("Record Updated successfully")
cursor.close()
else:
return {"invalide passphrase"}
Can you please tell me what's wrong with my code ?
if it's can help, here is my database structure (my db creation):
#Database connection
conn = sqlite3.connect("TAState15min.db")
cursor = conn.cursor()
#Create table
sql_query = """ CREATE TABLE TrendAnalyser (
id integer PRIMARY KEY,
state text,
date text
)"""
cursor.execute(sql_query)
#Create empty row with ID at 1
insert_query = """INSERT INTO TrendAnalyser
(id, state, date)
VALUES (1, 'Null', 'Null');"""
cursor.execute(insert_query)
conn.commit()
#Close database connexion
cursor.close()
**I finally found the issue, webhooks need the full path to the SQLite database to work fine. I just start to code in python, it was a noob issue... **

I finally found the issue, webhooks need the full path to the SQLite database to work fine. I just start to code in python, it was a noob issue...

Can't create a postgresql table using python

I am trying to create tables out of json files containing the field names and types of each table of a database downloaded from Bigquery. The SQL request semt fine to me and but no table was created according to psql command-line interpreter typing \d
So, to begin I've just tried with a simpler sql request that doesn't work neither,
Here is the code :
import pandas as pd
import psycopg2
# information used to create a database connection
sqluser = 'postgres'
dbname = 'testdb'
pwd = 'postgres'
# Connect to postgres database
con = psycopg2.connect(dbname=dbname, user=sqluser, password=pwd )
curs=con.cursor()
q="""set search_path to public,public ;
CREATE TABLE tab1(
i INTEGER
);
"""
curs.execute(q)
q = """
SELECT table_name
FROM information_schema.tables
WHERE table_schema='public'
AND table_type='BASE TABLE';
"""
df = pd.read_sql_query(q, con)
print(df.head())
print("End of test")
The code written above displays this new table tab1, but actually this new table doesn't appear listed when typing \d within the psql command line interpreter. If I type in the psql interpreter :
SELECT table_name
FROM information_schema.tables
WHERE table_type='BASE TABLE';
it doesn't get listed neither , seems it's not actually created, Thanks in advance for your help

There was a commit() call missing, that must be written after the table creation sql request,
This code works:
import pandas as pd
import psycopg2
# information used to create a database connection
sqluser = 'postgres'
dbname = 'testdb'
pwd = 'postgres'
# Connect to postgres database
con = psycopg2.connect(dbname=dbname, user=sqluser, password=pwd )
curs=con.cursor()
q="""set search_path to public,public ;
CREATE TABLE tab1(
i INTEGER
);
"""
curs.execute(q)
con.commit()
q = """
SELECT table_name
FROM information_schema.tables
WHERE table_schema='public'
AND table_type='BASE TABLE';
"""
df = pd.read_sql_query(q, con)
print(df.head())
print("End of test")

Python cx_Oracle module: unable to format query in code

I am using cx_Oracle module to connect to oracle database. In the script i use two variables schema_name and table_name. The below query works fine
cur1.execute("select owner,table_name from dba_tables where owner ='schema_name'")
But i need to query the num of rows of a table, where i need to qualify the table_name with the schema_name and so the query should be
SELECT count(*) FROM "schema_name"."table_name"
This does not work when using in the code, i have tried to put it in triple quotes, single quotes and other options but it does not format the query as expected and hence errors out with table does not exist.
Any guidance is appreciated.

A prepared statement containing placeholders with variables of the form ...{}.{}".format(sc,tb) might be used
sc='myschema'
tb='mytable'
cur1.execute("SELECT COUNT(*) FROM {}.{}".format(sc,tb))
print(cur1.fetchone()[0])

In this particular case, you could also try setting Connection.current_schema, see the cx_Oracle API doc
For example, if you create table in your own schema:
SQL> show user
USER is "CJ"
SQL> create table ffff (mycol number);
Table created.
SQL> insert into ffff values (1);
1 row created.
SQL> commit;
Commit complete.
Then run Python code that connects as a different user:
import cx_Oracle
import os
import sys, os
if sys.platform.startswith("darwin"):
cx_Oracle.init_oracle_client(lib_dir=os.environ.get("HOME")+"/Downloads/instantclient_19_8")
username = "system"
password = "oracle"
connect_string = "localhost/orclpdb1"
connection = cx_Oracle.connect(username, password, connect_string)
connection.current_schema = 'CJ';
with connection.cursor() as cursor:
sql = """select * from ffff"""
for r in cursor.execute(sql):
print(r)
sql = """select sys_context('USERENV','CURRENT_USER') from dual"""
for r in cursor.execute(sql):
print(r)
the output will be:
(1,)
('SYSTEM',)
The last query shows that it is not the user that is being changed, but just the first query is automatically changed from 'ffff' to 'CJ.ffff'.

PostgreSQL relation doesn't exist (Python)

I created a database in psql and in it, created a table called "tweet".
CREATE TABLE tweet
( tid CHARACTER VARYING NOT NULL, DATA json,
CONSTRAINT tid_pkey PRIMARY KEY (tid) );
Then when I use
SELECT * FROM tweet;
in the psql window it works and shows an empty table.
Now I have a python script that takes JSON data and is loading it into this table.
conn_string = "host='localhost' port=5432 dbname='tweetsql' user='tweetsql' password='tweetsql'"
conn = psycopg2.connect(conn_string)
cur = conn.cursor()
That sets up the connection and I don't think it had any issues.
Now I have some logic to read in the JSON file and then to add it in, I say:
cur.execute("INSERT INTO tweet (tid, data) VALUES (%s, %s)", (cur_tweet['id'], json.dumps(cur_tweet, cls=DecimalEncoder), ))
But this always says that the relation tweet doesn't exist. Am I missing something here? Is there an issue with my connection or can my script somehow not see the table? For reference I'm using psycopg2 for the connection.
EDIT: I updated the DDL to include a transaction I could commit but that didn't fix it either. Is it a schema issue?
This is what I did regarding the table creation to commit:
BEGIN;
CREATE TABLE tweet
( tid CHARACTER VARYING NOT NULL, DATA json,
CONSTRAINT tid_pkey PRIMARY KEY (tid) );
COMMIT;
EDIT 2: I'm posting some code here...
import psycopg2
import json
import decimal
import os
import ctypes
conn_string = "host='localhost' port=5432 dbname='tweetsql' user='tweetsql' password='tweetsql'"
conn = psycopg2.connect(conn_string)
cur = conn.cursor()
cur.execute("CREATE TABLE tweet (tid CHARACTER VARYING NOT NULL, DATA json, CONSTRAINT tid_pkey PRIMARY KEY (tid) );")
cur.commit()
for file in os.listdir(path):
if not is_hidden(file):
with open(path+file, encoding='utf-8') as json_file:
tweets = json.load(json_file, parse_float=decimal.Decimal)
for cur_tweet in tweets:
cur.execute("INSERT INTO tweet (tid, data) VALUES (%s, %s)", (cur_tweet['id'], json.dumps(cur_tweet, cls=DecimalEncoder), ))
cur.commit()
cur.close()
conn.close()

You're probably not committing the table creation, and, (I'm assuming; not seeing your complete code) you're starting a new connection via psycopg2 each time. You need to commit right after the table creation, and not in a new connection, as each connection is its own implicit transaction. So, your code flow should be something like this:
connect to the db
create the table using the cursor
fill the table
commit and disconnect from db.
Or, if you must separate creation from filling, just commit and disconnect after (2) and then reconnect before (3).

Creation pyodbc cursor to multiple databases

I am processing events stored in MS-Access databases using pyodbc.
Each month is a seperate file / database and I would like to process events from multiple months.
Is it possible to create a cursor to a view containing multiple months i.e. database connections?
Edit 1: And without having to write a new database? (Something like UNION VIEW maybe?)

You'll need to make multiple connections and cursors, but you should be able to process the data.
Let's say the files are stored as month_1.mdb, month_2.mdb, etc. in C:\access.
# Set up each connection, must have a way to access each file's name
connect_string = "Driver={Microsoft Access Driver (*.mdb, *.accdb)};DBQ=C:\\access\\month_{}.mdb;"
# Assuming that you'll get the same data from each database
sql = "SELECT column_1, column_2 FROM table"
# Connect to each file
connections = [pyodbc.connect(connect_string.format(n)) for n in range(1, 12 + 1)]
# Create a cursor for each file
cursors = [conn.cursor() for conn in connections]
# Query each file and save the data
data = []
for cur in cursors:
cur.execute(sql)
data.extend(cur.fetchall())
OK, so now you have all the data. You can create an in-memory database with the sqlite3 module and then do queries against it.
import sqlite3
# Create your temporary database
connection = sqlite3.connect(":memory:")
cursor = connection.cursor()
# Set up a place to hold the data fetched previously
_ = cur.execute("CREATE TABLE t(x INTEGER, y INTEGER)")
# Dump all the data into the database
for column_1, column_2 in data:
_ = cursor.execute("INSERT INTO t VALUES (?, ?)", [column_1, column_2])
# Now you can run queries against the new view of your data
sql = "SELECT t.column_1, t.count(*) FROM t GROUP BY t.column_1"