We Keep Coding

How to update all column values with new values in a MySQL database

I am using Python to pull data from an API and update a MySQL database with those values. I realized that as my code is right now, the data I want from the API is only being INSERTED into the database the first time the code is ran but the database needs to be updated with new values whenever the code is executed after the first time. The API provides close to real - time values of position, speed, altitude etc. of current airlines in flight. My database looks like this:
table aircraft:
+-------------+------------+------------+-----------+-----------+
| longitude | latitude | velocity | altitude | heading |
+-------------+------------+------------+-----------+-----------+
| | | | | |
| | | | | |
I am quite new to MySQL and I am having trouble finding how to do this the right way. The point is to run the code in order to update the table whenever I want. Or possibly every 10 seconds or so. I am using Python's MySQLdb module in order to execute SQL commands within the python code. Here is the main part of what I currently have.
#"states" here is a list of state vectors from the api that have the data I want
states = api.get_states()
#creates a cursor object to execute SQL commands
#the parameter to this function should be an SQL command
cursor = db.cursor()
#"test" is the database name
cursor.execute("USE test")
print("Adding states from API ")
for s in states.states:
if( s.longitude is not None and s.latitude is not None):
cursor.execute("INSERT INTO aircraft(longitude, latitude) VALUES (%r, %r);", (s.longitude, s.latitude))
else:
("INSERT INTO aircraft(longitude, latitude) VALUES (NULL, NULL);")
if(s.velocity is not None):
cursor.execute("INSERT INTO aircraft(velocity) VALUES (%r);" % s.velocity)
else:
cursor.execute("INSERT INTO aircraft(velocity) VALUES (NULL);")
if(s.altitude is not None):
cursor.execute("INSERT INTO aircraft(altitude) VALUES (%r);" % s.altitude)
else:
cursor.execute("INSERT INTO aircraft(altitude) VALUES (NULL);")
if(s.heading is not None):
cursor.execute("INSERT INTO aircraft(heading) VALUES (%r);" % s.heading)
else:
cursor.execute("INSERT INTO aircraft(heading) VALUES (NULL);")
db.commit()

Returning data from many rows with different values

so i have been having a problem with my code.
What im trying to do is i have a list set up like:
locationList = [['mt', 'moon', 'mt-moon'], ['misty', 'mis'], ['surge', 'sur'], ect...]
And what this is is they are inputs the people use, so instead of having to input the whole word, they just do an abbreviation.
Now i have these stored in a DB, with The persons name, their points, the bet loctaion, and betted points. Like this:
**Name** | **Points** | **BetLocation** | **BetPoints**
-------- | ---------- | --------------- | -------------
James | 1000 | mis | 100
Mike | 3000 | misty | 700
Dave | 400 | mt | 200
Now what im trying to do is when i choose a winning location, like say i choose misty, it will return both James and Mike but not Dave.
if any(winner in s for s in locationList):
c = conn.cursor()
search = winner
for sublist in locationList:
if sublist[0] == search:
print(sublist)
print(str(sublist)[1:-1])
test1 = '(' + str(sublist)[1:-1] + ')'
print(test1)
c.execute("SELECT Name, CurrentBetValue FROM Users WHERE CurrentBetLocation IN('misty', 'mis')")
data = c.fetchall()
print(data)
for f in data:
kek = (f[1], f[0])
c.execute('UPDATE Users SET Points = Points + (? * 2) WHERE Name = ?', kek)
conn.commit()
betReset()
betsOpen = False
return send_message('"'+str(winner) + '"' + ' is the winning location, points have been updated.')
Each print returns:
['misty', 'mis']
'misty', 'mis'
('misty', 'mis')
Now this works due to me having manually put in IN('misty', 'mis')
And returns what i want it to.
However if i change it to c.execute("SELECT Name, CurrentBetValue FROM Users WHERE CurrentBetLocation IN ?", test1)
c.execute("SELECT Name, CurrentBetValue FROM Users WHERE CurrentBetLocation IN ?", test1)
sqlite3.OperationalError: near "?": syntax error
I was wondering if anybody could help!

Just taking a guess from "Its just a Local .db file" that you're using SQLite and the built-in sqlite3 Python module.
If so, I think you want this:
c.execute("SELECT Name, CurrentBetValue FROM Users WHERE CurrentBetLocation IN ({})".format(",".join("?"*len(sublist))), sublist)
"?" * len(sublist) gives you a sequence of n question marks for n elements in the sublist. Joining that with commas, you get a nice parameterized command: "... IN (?,?)". Finally, you can pass in your sublist as those parameters.
Here's a complete working example with a couple comments inline so you can see what's happening:
import sqlite3
conn = sqlite3.connect("test.db")
c = conn.cursor()
c.execute("create table test (name text, location text)")
c.execute("insert into test values (?, ?)", ("James", "mis"))
c.execute("insert into test values (?, ?)", ("Mike", "misty"))
c.execute("insert into test values (?, ?)", ("Dave", "mt"))
sublist = ["misty", "mis"]
# select name from test where location in (?,?)
command = "select name from test where location in ({})".format(",".join("?" * len(sublist)))
# select name from test where location in ('misty', 'mis')
c.execute(command, sublist)
print(c.fetchall()) # [('James',), ('Mike',)]
If my assumptions were wrong, and you're using a database other than SQLite or a module other than sqlite3, please tell us what you're using. (For future questions: this is quite important information to include. Even better would be to provide a complete runnable example.)

Storing the data from text file into mysql table

I have a text file and a MySQL table. the text file look like below.
new.txt
apple| 3
ball | 4
cat | 2
like this. from this text file I want to store data in the below MySQL table.
| query | count | is_prod_ready | time_of_created | last_updated |
I want to store apple,ball,cat in the column query, and all the number 3,4,2 in count column. in is_prod_ready column will be false by default, in time_of_created will take the current time. and last_updated_column will take the update time.
I have already made the table. and i am not able to store all the data into database from the text file. i have tried the below code.
import MySQLdb
con = MySQLdb.connect(host="localhost",user="root",passwd="9090547207",db="Test")
cur = con.cursor()
query = 'load data local infile "new.txt" into table data field terminated by "|" lines terminated by "\n" '
cur.execute(query)
con.commit()
con.close()
here my data base name is Test and table name is data.

PostgreSQL - how do you get the column formats?

I'm using PostgreSQL 9.3.3 on x86_64-unknown-linux-gnu, compiled by gcc (GCC) 4.1.2 20080704 (Red Hat 4.1.2-52), 64-bit.
I've done
psycopg2.connect
, got the cursor and can run lines of code similar to
cur.execute('SELECT latitude, longitude, date from db')
table = cur.fetchall()
From what I understand at http://initd.org/psycopg/docs/cursor.html, running
print(cur.description)
should show the type_code of each column. How come I don't get this?
I get
(Column(name='column_name', type_code=1043, display_size=None, internal_size=-1, precision=None, scale=None, null_ok=None), Column(name='data_type', type_code=1043, display_size=None, internal_size=-1, precision=None, scale=None, null_ok=None))
Another solution I've been suggested is running
cur.execute('select column_name, data_type from information_schema.columns')
cur.fetchall()
cols = cur.fetchall()
but this returns an empty list.
That's what I've tried. What do you suggest for getting the column formats?

information_schema.columns should provide you with the column data-type info.
For example, given this DDL:
create table foo
(
id serial,
name text,
val int
);
insert into foo (name, val) values ('narf', 1), ('poit', 2);
And this query (filtering out the meta tables to get at your tables):
select *
from information_schema.columns
where table_schema NOT IN ('information_schema', 'pg_catalog')
order by table_schema, table_name;
Will yield 4 rows, for the table foo -- the three columns I defined, plus a FK.
SQL fiddle
Regarding psycopg2, the information_schema-related code that you have shown looks like it should work... What's the entirety of the code? I would also recommend trying to step through the code in a debugger (the built-in pdb is OK, but I would recommend pudb, as it's more full featured and easier to use, but still terminal-based. It only runs on *nix platforms, though, due to the underlying modules it uses.
Edit:
I was able to get the data_type info from information_schema using psycopg2 with the following code:
#!/usr/bin/env python
import psycopg2
import psycopg2.extras
conn = psycopg2.connect("host=<host> dbname=<dbname> user=<user> password=<password>")
cur = conn.cursor(cursor_factory=psycopg2.extras.DictCursor)
cur.execute("""select *
from information_schema.columns
where table_schema NOT IN ('information_schema', 'pg_catalog')
order by table_schema, table_name""")
for row in cur:
print "schema: {schema}, table: {table}, column: {col}, type: {type}".format(
schema = row['table_schema'], table = row['table_name'],
col = row['column_name'], type = row['data_type'])
I prefer to use DictCursors, as I find them much easier to work with, but it should work with a regular cursor, too -- you would just need to change how you accessed the rows.
Also, regarding cur.description, that returns a tuple of tuples. If you want to get at the type_code there, you can do so like this:
print cur.description[0][1]
Where the first dimension in the index of the column you want to look at, and the second dimension is the datum within that column. type_code is always 1. So you could iterate over the outer tuple and always look at its second item, for example.

select oid,typname from pg_type;
oid | typname
------+-------------
16 | bool
23 | int4
25 | text
1043 | varchar
1184 | timestamptz

MySQL and Python Select Statement Issues

Thanks for taking the time to read this. It's going to be a long post to explain the problem. I haven't been able to find an answer in all the usual sources.
Problem:
I am having an issue with using the select statement with python to recall data from a table in a mysql database.
System and versions:
Linux ubuntu 2.6.38-14-generic #58-Ubuntu SMP Tue Mar 27 20:04:55 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
Python: 2.7.1+
MySql: Server version: 5.1.62-0ubuntu0.11.04.1 (Ubuntu)
Here's the table:
mysql> describe hashes;
+-------+--------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-------+--------------+------+-----+---------+-------+
| id | varchar(20) | NO | PRI | NULL | |
| hash | varbinary(4) | NO | MUL | NULL | |
+-------+--------------+------+-----+---------+-------+
Here are responses that I want via a normal mysql query:
mysql> SELECT id FROM hashes WHERE hash='f';
+------+
| id |
+------+
| 0x67 |
+------+
mysql> SELECT id FROM hashes WHERE hash='ff';
+--------+
| id |
+--------+
| 0x6700 |
+--------+
As before, these are the responses that are expected and how I designed the DB.
My code:
import mysql.connector
from database import login_info
import sys
db = mysql.connector.Connect(**login_info)
cursor = db.cursor()
data = 'ff'
cursor.execute("""SELECT
* FROM hashes
WHERE hash=%s""",
(data))
rows = cursor.fetchall()
print rows
for row in rows:
print row[0]
This returns the result I expect:
[(u'0x67', 'f')]
0x67
If I change data to :
data = 'ff'
I receive the following error:
Traceback (most recent call last):
File "test.py", line 11, in <module>
(data))
File "/usr/local/lib/python2.7/dist-packages/mysql_connector_python-0.3.2_devel- py2.7.egg/mysql/connector/cursor.py", line 310, in execute
"Wrong number of arguments during string formatting")
mysql.connector.errors.ProgrammingError: Wrong number of arguments during string formatting
OK. So, I add a string formatting character to my SQL statement as so:
cursor.execute("""SELECT
* FROM hashes
WHERE hash=%s%s""",
(data))
And I get the following response:
[(u'0x665aa6', "f'f")]
0x665aa6
and it should by 0x6700.
I know that I should be passing the data with one %s character. That is how I built my database table, using one %s per variable:
cursor.execute("""
INSERT INTO hashes (id, hash)
VALUES (%s, %s)""", (k, hash))
Any ideas how to fix this?
Thanks.

Your execute statement doesn't seem quite correct. My understanding is that it should follow the pattern cursor.execute( <select statement string>, <tuple>) and by putting only a single value in the tuple location it is actually just a string. To make the second argument the correct data type you need to put a comma in there, so your statement would look like:
cursor.execute("""SELECT
* FROM hashes
WHERE hash=%s""",
(data, ))