We Keep Coding

Plotting values against time in MatPlotlib using SQL

I am trying to plot a list of numerical values against a list of timestamps, which are stored in a SQL database called 'scores'. After SELECTING two columns and fetching the data from the database, I then convert the tuple of tuples into lists, and then remove the "None" records, so that it could be plotted using plt.plot(x,y).
However, when I run the program below, I get this error:
ValueError: x and y must have same first dimension, but have shapes (5,) and (4,)
Here is some sample data for yaxis_list and xaxis_final:
yaxis_list = [20,40,60,80]
xaxis_final = ['2023-02-18 02:09:15', '2023-02-18 02:18:00', '2023-02-18 18:52:10']
my imports:
from matplotlib import pyplot as plt
import _sqlite3
I am a beginner at matplotlib and any other code rebuilding suggestions would really help me :)
with _sqlite3.connect("scores.db") as db:
cursor = db.cursor()
cursor.execute("SELECT (phy) FROM scores WHERE phy > 0")
yaxis_tuple = cursor.fetchall()
yaxis_list = [item for t in yaxis_tuple for item in t]
cursor.execute("SELECT (phystamp) FROM scores")
xaxis_tuple = cursor.fetchall()
xaxis_list = [item for t in xaxis_tuple for item in t]
xaxis_final = [i for i in xaxis_list if i is not None]
plt.plot(xaxis_final,yaxis_list)
plt.show()

Summing database column in python

I have recently encountered the problem of adding the elements of a database column. Here is the following code:
import sqlite3
con = sqlite3.connect("values.db")
cur = con.cursor()
cur.execute('SELECT objects FROM data WHERE firm = "sony"')
As you can see, I connect to the database (sql) and I tell to Python to select the column "objects".
The problem is that I do not know the appropriate command for summing the selected objects.
Any ideas/ advices are highly reccomended.
Thank you in advance!!

If you can, have the database do the sum, as that reduces data transfer and lets the database do what it's good at.
cur.execute("SELECT sum(objects) FROM data WHERE firm = 'sony'")
or, if you're really just looking for the total count of objects.
cur.execute("SELECT count(objects) FROM data WHERE firm = 'sony'")
either way, your result is simply:
count = cur.fetchall()[0][0]

Try the following line:
print sum([ row[0] for row in cur.fetchall()])
If you want the items instead adding them together:
print ([ row[0] for row in cur.fetchall()])

Retrieve SQLite result as ndarray

I am retrieving a set of latitude and longitudinal points from an sqlite database as this:
cur = con.execute("SELECT DISTINCT latitude, longitude FROM MessageType1 WHERE latitude>{bottomlat} AND latitude<={toplat} AND longitude>{bottomlong} AND longitude<={toplong}".format(bottomlat = bottomlat, toplat = toplat, bottomlong = bottomlong, toplong = toplong))
However, as I am supposed to make a ConvexHull (http://docs.scipy.org/doc/scipy/reference/generated/scipy.spatial.ConvexHull.html#scipy.spatial.ConvexHull) which has an ndarray as input, I need to save the results for cur as a ndarray. How would I do that?
Right now I fetch the values as :
positions = [[floats(x[0]) floats(x[1])] for x in cur]
Is this correct?

You don't need to convert positions to an ndarray. scipy.spatial.ConvexHull can accept a list of lists as well:
import scipy.spatial as spatial
hull = spatial.ConvexHull(positions)
Also, if your MessageType1 table has latitude, longitude fields of type float, then you should not need to call float explicitly. Instead of
positions = [[floats(x[0]) floats(x[1])] for x in cur]
you could use
positions = cur.fetchall()
Note that if you are using NumPy slicing syntax, such as positions[:, 0],
then you would need to convert the list of lists/tuples to a NumPy array:
positions = np.array(cur.fetchall())

How to convert numpy array to postgresql list

I am trying to use python to insert 2 columns of a numpy array into a postgresql table as two arrays.
postgresql table is DOS:
primary_key
energy integer[]
dos integer[]
I have a numpy array that is a 2d array of 2x1D arrays:
finArray = np.array([energy,dos])
I am trying to use the following script for inserting into a database and I keep getting errors with the insert. I can't figure out how to format the array so that it properly formats in the form: INSERT INTO dos VALUES(1,'{1,2,3}','{1,2,3}')"
Script:
import psycopg2
import argparse
import sys
import re
import numpy as np
import os
con = None
try:
con = psycopg2.connect(database='bla', user='bla')
cur = con.cursor()
cur.execute("INSERT INTO dos VALUES(1,'{%s}')", [str(finArray[0:3,0].tolist())[1:-1]])
con.commit()
except psycopg2.DatabaseError, e:
if con:
con.rollback()
print 'Error %s' % e
sys.exit(1)
finally:
if con:
con.close()
The part I can't figure out is I will get errors like this:
Error syntax error at or near "0.31691105000000003"
LINE 1: INSERT INTO dos VALUES(1,'{'0.31691105000000003, -300.0, -19...
I can't figure out where that inner ' ' is coming from in the bracket.

Too late, but putting this out anyway.
I was trying to insert a numpy array into Redshift today. After trying odo, df.to_sql() and what not, I finally got this to work at a pretty fast speed (~3k rows/minute). I won't talk about the issues I faced with those tools but here's something simple that works:
cursor = conn.cursor()
args_str = b','.join(cursor.mogrify("(%s,%s,...)", x) for x in tuple(map(tuple,np_data)))
cursor.execute("insert into table (a,b,...) VALUES "+args_str.decode("utf-8"))
cursor.commit()
cursor.close()
The 2nd line will need some work based on the dimensions of your array.
You might want to check these answers too:
Converting from numpy array to tuple
Multiple row inserts in psycopg2

You probably have an array of strings, try changing your command adding astype(float), like:
cur.execute("INSERT INTO dos VALUES(1,'{%s}')", [str(finArray[0:3,0].astype(float).tolist())[1:-1]])

The quotes come during the numpy.ndarray.tolist() and come because you actually have strings. If you don't want to assume that data is float-typed as #Saullo Castro suggested you could also do a simple str(finArray[0:3,0].tolist()).replace("'","")[1:-1] to get rid of them.
However, more appropriately, if you are treating the data in finArray in any way in your script and assume they are numbers, you should probably make sure they are imported into the array as numbers to start with.
You can require the array to have a certain datatype while initiating it by specifying, e.g. finArray = np.array(..., dtype=np.float) and then work backwards towards where it is suitable to enforce the type.

Psycopg will adapt a Python list to an array so you just have to cast the numpy array to a Python list and pass it to the execute method
import psycopg2
import numpy as np
energy = [1, 2, 3]
dos = [1, 2, 3]
finArray = np.array([energy,dos])
insert = """
insert into dos (pk, energy) values (1, %s);
;"""
conn = psycopg2.connect("host=localhost4 port=5432 dbname=cpn")
cursor = conn.cursor()
cursor.execute(insert, (list(finArray[0:3,0]),))
conn.commit()
conn.close()

You need convert the numpy array to a list, example:
import numpy as np
import psycopg2
fecha=12
tipo=1
precau=np.array([20.35,25.34,25.36978])
conn = psycopg2.connect("dbname='DataBase' user='Administrador' host='localhost' password='pass'")
cur = conn.cursor()
#make a list
vec1=[]
for k in precau:
vec1.append(k)
#make a query
query=cur.mogrify("""UPDATE prediccioncaudal SET fecha=%s, precaudal=%s WHERE idprecau=%s;""", (fecha,vec1,tipo))
#execute a query
cur.execute(query)
#save changes
conn.commit()
#close connection
cur.close()
conn.close()

Replacing value in all cursor rows

Using SQLite and Python 3.1, I want to display currency data in a HTML table via. a template which accepts a cursor as a parameter. Hence all currency values must have 2 decimal places, but SQLite stores them as float type (even though the structure states decimal :-( ) so some must be converted before display (eg. I want 12.1 displayed as 12.10).
The code goes something like this (simplified for illustration)...
import sqlite3
con = sqlite3.connect("mydb")
con.row_factory = sqlite3.Row
cur = con.cursor()
cur.execute("select order_no, amount from orders where cust_id=123")
for row in cur:
row['amount'] = format(row['amount'],'%.2f')
The last command throws the error "# builtins.TypeError: 'sqlite3.Row' object does not support item assignment"
How can I solve the problem whereby the row object values cannot be changed? Could I convert the cursor to a list of dictionaries (one for each row, eg. [{'order_no':1, 'amount':12.1}, {'order_no':2, 'amount':6.32}, ...]), then format the 'amount' value for each item? If so, how can I do this?
Are there any better solutions for achieving my goal? Any help would be appreciated.
TIA,
Alan

Yep:
cur.execute("select order_no, amount from orders where cust_id=123")
dictrows = [dict(row) for row in cur]
for r in dictrows:
r['amount'] = format(r['amount'],'%.2f')
There are other ways, but this one seems the simplest and most direct one.

An alternative is to store your value as an integer number of cents (which is always an exact amount, no rounding), and then convert to dollars when displaying for reports using divmod:
>>> value_in_cents = 133
>>> print "$%d.%d" % divmod(value_in_cents,100)
$1.33