We Keep Coding

psycopg2: cursor.execute storing only table structure, no data

I am trying to store some tables I create in my code in an RDS instance using psycopg2. The script runs without issue and I can see the table being stored correctly in the DB. However, if I try to retrieve the query, I only see the columns, but no data:
import pandas as pd
import psycopg2
test=pd.DataFrame({'A':[1,1],'B':[2,2]})
#connect is a function to connect to the RDS instance
connection= connect()
cursor=connection.cursor()
query='CREATE TABLE test (A varchar NOT NULL,B varchar NOT NULL);'
cursor.execute(query)
connection.commit()
cursor.close()
connection.close()
This script runs without issues and, printing out file_check from the following script:
connection=connect()
# check if file already exists in SQL
sql = """
SELECT "table_name","column_name", "data_type", "table_schema"
FROM INFORMATION_SCHEMA.COLUMNS
WHERE "table_schema" = 'public'
ORDER BY table_name
"""
file_check=pd.read_sql(sql, con=connection)
connection.close()
I get:
table_name column_name data_type table_schema
0 test a character varying public
1 test b character varying public
which looks good.
Running the following however:
read='select * from public.test'
df=pd.read_sql(read,con=connection)
returns:
Empty DataFrame
Columns: [a, b]
Index: []
Anybody have any idea why this is happening? I cannot seem to get around this

Erm, your first script has a test_tbl dataframe, but it's never referred to after it's defined.
You'll need to
test_tbl.to_sql("test", connection)
or similar to actually write it.
A minimal example:
$ createdb so63284022
$ python
>>> import sqlalchemy as sa
>>> import pandas as pd
>>> test = pd.DataFrame({'A':[1,1],'B':[2,2], 'C': ['yes', 'hello']})
>>> engine = sa.create_engine("postgres://localhost/so63284022")
>>> with engine.connect() as connection:
... test.to_sql("test", connection)
...
>>>
$ psql so63284022
so63284022=# select * from test;
index | A | B | C
-------+---+---+-------
0 | 1 | 2 | yes
1 | 1 | 2 | hello
(2 rows)
so63284022=# \d+ test
Table "public.test"
Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
--------+--------+-----------+----------+---------+----------+--------------+-------------
index | bigint | | | | plain | |
A | bigint | | | | plain | |
B | bigint | | | | plain | |
C | text | | | | extended | |
Indexes:
"ix_test_index" btree (index)
Access method: heap
so63284022=#

I was able to solve this:
As it was pointed out by #AKX, I was only creating the table structure, but I was not filling in the table.
I now import import psycopg2.extras as well and, after this:
query='CREATE TABLE test (A varchar NOT NULL,B varchar NOT NULL);'
cursor.execute(query)
I add something like:
update_query='INSERT INTO test(A, B) VALUES(%s,%s) ON CONFLICT DO NOTHING'
psycopg2.extras.execute_batch(cursor, update_query, test.values)
cursor.close()
connection.close()
My table is now correctly filled after checking with pd.read_sql

Is there any ways to combine two rows of table into one row using Django ORM?

I have a table which has columns named measured_time, data_type and value.
In data_type, there is two types, temperature and humidity.
I want to combine two rows of data if they have same measured_time using Django ORM.
I am using Maria DB.
Using Raw SQL, The following Query does what I want to.
SELECT T1.measured_time, T1.temperature, T2.humidity
FROM ( SELECT CASE WHEN data_type = 1 then value END as temperature,
CASE WHEN data_type = 2 then value END as humidity ,
measured_time FROM data_table) as T1,
( SELECT CASE WHEN data_type = 1 then value END as temperature ,
CASE WHEN data_type = 2 then value END as humidity ,
measured_time FROM data_table) as T2
WHERE T1.measured_time = T2.measured_time and
T1.temperature IS NOT null and T2.humidity IS NOT null and
DATE(T1.measured_time) = '2019-07-01'
Original Table
| measured_time | data_type | value |
|---------------------|-----------|-------|
| 2019-07-01-17:27:03 | 1 | 25.24 |
| 2019-07-01-17:27:03 | 2 | 33.22 |
Expected Result
| measured_time | temperaure | humidity |
|---------------------|------------|----------|
| 2019-07-01-17:27:03 | 25.24 | 33.22 |

I've never used it and so can't answer in detail, but you can feed a raw SQL query into Django and get the results back through the ORM. Since you have already got the SQL this may be the easiest way to proceed. Documentation here

django unique together does not work with none datetime [duplicate]

This question already has answers here:
Django unique together constraint failure?
(4 answers)
Closed 9 years ago.
I am using django 1.5.5 and python 2.7 and MySQL
this is my model
class Foo(models.Model):
user = models.ForeignKey(User)
date = models.DateTimeField(null=True, blank=True,editable=True)
class Meta:
unique_together = ('user', 'date')
If i use this i can add 2 records with the same user and an empty date.
I would like the uniqueness to be enforced even for empty date.
table create command
CREATE TABLE `management_foo`
( `id` int(11) NOT NULL AUTO_INCREMENT,
`user_id` int(11) NOT NULL,
`date` datetime DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `user_id` (`user_id`,`date`),
KEY `management_foo_6232c63c` (`user_id`),
CONSTRAINT `user_id_refs_id_d47e5747` FOREIGN KEY (`user_id`)
REFERENCES `auth_user` (`id`))
ENGINE=InnoDB AUTO_INCREMENT=4 DEFAULT CHARSET=latin1
and table describe
+-------------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-------------+--------------+------+-----+---------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| user_id | int(11) | NO | MUL | NULL | |
| date | datetime | YES | | NULL | |
+-------------+--------------+------+-----+---------+----------------+

In InnoDB each NULL treats as unique value.
example:
mysql> create table x (i int, j int, primary key (i), unique key (j));
mysql> insert into x (i,j) values (NULL,NULL);
ERROR 1048 (23000): Column 'i' cannot be null
mysql> insert into x (i,j) values (1,NULL);
mysql> insert into x (i,j) values (2,NULL);
mysql> insert into x (i,j) values (3,3);
mysql> select * from x;
+---+------+
| i | j |
+---+------+
| 1 | NULL |
| 2 | NULL |
| 3 | 3 |
+---+------+
3 rows in set (0.01 sec)
mysql> insert into x (i,j) values (4,3);
ERROR 1062 (23000): Duplicate entry '3' for key 'j'
You need to add not null constraint <=> remove null=True from field definition (replace it with default for example)
BTW, it's better to write in such style: unique_together = (("field1", "field2"),) – it will be much easier to extend unique pairs

Just for the record: the SQL standard states that NULL is not a value and so must not be taken in account for unique constraints. IOW it has nothing to do with Django and actually is the expected behavior.
For a technical solution see akaRem's answer.

Django: Make a column which is not primary key unique

I have a list of numeric codes with corresponding mnemonic names and I want to have a Django model for them so the names are primary keys, but there is also a constraint that the values in the code column are unique.
What I tried is the following:
class Constant(models.Model):
name = models.CharField(max_length=70)
name.primary_key = True
code = models.IntegerField()
description = models.CharField(max_length=100)
unique_together = (("code",),)
I realize that unique_together is meant to enforce uniqueness of values in a set of columns, but I thought I would try with just one and it seemed to work, i.e. no error when doing python manage.py syncdb, but it doesn't really enforce the constraint I want:
mysql> describe constant;
+-------------+--------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-------------+--------------+------+-----+---------+-------+
| name | varchar(70) | NO | PRI | | |
| code | int(11) | NO | | | |
| description | varchar(100) | NO | | | |
+-------------+--------------+------+-----+---------+-------+
3 rows in set (0.01 sec)
mysql> insert into constant values ('x',1,'fooo');
Query OK, 1 row affected (0.00 sec)
mysql> insert into constant values ('y',1,'foooo');
Query OK, 1 row affected (0.00 sec)
What can I do to make sure values in both columns are unique?

Add the unique option to your code field.
class Constant(models.Model):
name = models.CharField(max_length=70, primary_key=True)
code = models.IntegerField(unique=True)
description = models.CharField(max_length=100)

MySQL and Python Select Statement Issues

Thanks for taking the time to read this. It's going to be a long post to explain the problem. I haven't been able to find an answer in all the usual sources.
Problem:
I am having an issue with using the select statement with python to recall data from a table in a mysql database.
System and versions:
Linux ubuntu 2.6.38-14-generic #58-Ubuntu SMP Tue Mar 27 20:04:55 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
Python: 2.7.1+
MySql: Server version: 5.1.62-0ubuntu0.11.04.1 (Ubuntu)
Here's the table:
mysql> describe hashes;
+-------+--------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-------+--------------+------+-----+---------+-------+
| id | varchar(20) | NO | PRI | NULL | |
| hash | varbinary(4) | NO | MUL | NULL | |
+-------+--------------+------+-----+---------+-------+
Here are responses that I want via a normal mysql query:
mysql> SELECT id FROM hashes WHERE hash='f';
+------+
| id |
+------+
| 0x67 |
+------+
mysql> SELECT id FROM hashes WHERE hash='ff';
+--------+
| id |
+--------+
| 0x6700 |
+--------+
As before, these are the responses that are expected and how I designed the DB.
My code:
import mysql.connector
from database import login_info
import sys
db = mysql.connector.Connect(**login_info)
cursor = db.cursor()
data = 'ff'
cursor.execute("""SELECT
* FROM hashes
WHERE hash=%s""",
(data))
rows = cursor.fetchall()
print rows
for row in rows:
print row[0]
This returns the result I expect:
[(u'0x67', 'f')]
0x67
If I change data to :
data = 'ff'
I receive the following error:
Traceback (most recent call last):
File "test.py", line 11, in <module>
(data))
File "/usr/local/lib/python2.7/dist-packages/mysql_connector_python-0.3.2_devel- py2.7.egg/mysql/connector/cursor.py", line 310, in execute
"Wrong number of arguments during string formatting")
mysql.connector.errors.ProgrammingError: Wrong number of arguments during string formatting
OK. So, I add a string formatting character to my SQL statement as so:
cursor.execute("""SELECT
* FROM hashes
WHERE hash=%s%s""",
(data))
And I get the following response:
[(u'0x665aa6', "f'f")]
0x665aa6
and it should by 0x6700.
I know that I should be passing the data with one %s character. That is how I built my database table, using one %s per variable:
cursor.execute("""
INSERT INTO hashes (id, hash)
VALUES (%s, %s)""", (k, hash))
Any ideas how to fix this?
Thanks.

Your execute statement doesn't seem quite correct. My understanding is that it should follow the pattern cursor.execute( <select statement string>, <tuple>) and by putting only a single value in the tuple location it is actually just a string. To make the second argument the correct data type you need to put a comma in there, so your statement would look like:
cursor.execute("""SELECT
* FROM hashes
WHERE hash=%s""",
(data, ))