We Keep Coding

Inner Join in Django

I have two tables
front_employee (Employee model in Django)
+-----------------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-----------------+--------------+------+-----+---------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| order | int(11) | NO | | NULL | |
| name | varchar(100) | YES | | NULL | |
| position | varchar(100) | YES | | NULL | |
| description | longtext | YES | | NULL | |
| employee_img_id | int(11) | NO | MUL | NULL | |
| language_id | int(11) | NO | MUL | NULL | |
+-----------------+--------------+------+-----+---------+----------------+
And front_employeepicture (EmployeePicture in Django)
+-------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-------+--------------+------+-----+---------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| order | int(11) | NO | | NULL | |
| img | varchar(100) | YES | | NULL | |
+-------+--------------+------+-----+---------+----------------+
I would like to perform this query:
SELECT a.id, a.name, b.img
FROM front_employee a
INNER JOIN front_employeepicture b
ON a.employee_img_id = b.id
For now I have
context['employee'] = Employee.objects.all().order_by('order')
And I tried something like
context['employee'] = Employee.objects.select_related('EmployeePicture')
Without result. Any idea?

Django's ORM is really powerful. You actually don't have to do joins like that, ever. When you access the field, the ORM performs the join on the fly, returning the result you want.
first_employee = Employee.objects.all().first()
employee_picture = first_employee.employee_img
And then employee_picture should have the picture of the first employee. It needed to internally do an inner join to figure that out, but Django hides all that for you. (I may have gotten your variable names wrong, sorry).
What select_related does is pre-fetch a relation for every object in a queryset. That means less trips back and forth to the database, which makes your functions faster. But, Django allows you to traverse relationships just by accessing attributes.
Try messing around with this stuff in your django shell. I would add django_extensions to your app (pip install django-extensions, then add django_extensions to your INSTALLED_APPS, then run python manage.py shell_plus). And if not, python manage.py shell works fine, but you have to import your models manually.

Flask SqlAlchemy MySql Boolean Type Always Returns True

I have a Flask application connected to a MySql DB using SqlAlchemy. The table has 3 x boolean (bit) fields as shown below:
+------------------------+---------------+------+-----+-------------------+----------------+
| Field | Type | Null | Key | Default |
Extra |
+------------------------+---------------+------+-----+-------------------+----------------+
| ID | int(11) | NO | PRI | NULL |
auto_increment |
| clientID | int(11) | YES | | NULL |
|
| accountType | varchar(2) | YES | | NULL |
|
| systemType | varchar(1) | YES | | NULL |
|
| clientName | varchar(400) | YES | | NULL |
|
| clientURL | varchar(5000) | YES | | NULL |
|
| clientTelephone | varchar(300) | YES | | NULL |
|
| clientAddressLine1 | varchar(500) | YES | | NULL |
|
| clientAddressLine2 | varchar(500) | YES | | NULL |
|
| clientAddressLine3 | varchar(500) | YES | | NULL |
|
| clientPostcode | varchar(50) | YES | | NULL |
|
| clientCountry | varchar(100) | YES | | NULL |
|
| accessBenchmarking | bit(1) | YES | | NULL |
|
| accessTechnicalSupport | bit(1) | YES | | NULL |
|
| accountLive | bit(1) | YES | | NULL |
|
| clientTown | varchar(100) | YES | | NULL |
|
| clientCounty | varchar(100) | YES | | NULL |
|
| dateTimeStamp | timestamp | YES | | CURRENT_TIMESTAMP |
|
+------------------------+---------------+------+-----+-------------------+----------------+
Each of the bit fields has a value set to 0.
The SqlAlchemy Model for this is:
class ClientAccounts(db.Model):
id = db.Column(db.Integer, primary_key=True)
clientID = db.Column(db.Integer)
accountType = db.Column(db.Text(2))
systemType = db.Column(db.Text(1))
clientName = db.Column(db.Text(400))
clientURL = db.Column(db.Text(5000))
clientTelephone = db.Column(db.Text(300))
clientAddressLine1 = db.Column(db.Text(500))
clientAddressLine2 = db.Column(db.Text(500))
clientAddressLine3 = db.Column(db.Text(500))
clientPostcode = db.Column(db.Text(50))
clientCountry = db.Column(db.Text(100))
accessBenchmarking = db.Column(db.Boolean)
accessTechnicalSupport = db.Column(db.Boolean)
accountLive = db.Column(db.Boolean)
clientTown = db.Column(db.Text(100))
clientCounty = db.Column(db.Text(100))
The code to retrieve the values is here:
#check for valid and live user account
CheckAccount = ClientAccounts.query.filter_by(
clientID=accountNo,
).first()
if not CheckAccount is None:
accessBenchmarking = CheckAccount.accessBenchmarking
accessTechnicalSupport = CheckAccount.accessTechnicalSupport
accountLive = CheckAccount.accountLive
print 'db return ...'
print accessBenchmarking
print accessTechnicalSupport
print accountLive
The values are always returned as True even though they are set to False in the DB. The returned vales can be seen here:
INFO:sqlalchemy.engine.base.Engine:('11111111', 1)
db return ...
True
True
True
Does anybody have any idea what's causing this?

I figured out a fix for this. Changing the field data type from bit to tinyint for each boolean field did the trick. I'm still none the wiser as to why bit doesn't work with SqlAlchemy. Maybe it's the version of MySql Python I'm using?

For those who come across this thread without finding the solid solution for this:
I fixed this issue by changing the MYSQL connector to mysql-connector from pymysql.
pip3 install mysql-connector
'mysql+mysqlconnector://username:password!!#127.0.0.1:3306/'
I was lost for a long time, making this work. Didn't know the connector would be the issue.

Pandas + MySQL On Duplicate Key Update Broken

My ON DUPLICATE UPDATE clause stopped updating and I'm not sure why.
Below is my code to create a temporary table via Pandas:
#connect to mysql database
engine = sqlalchemy.create_engine('mysql://username:#localhost/db?charset=utf8')
conn = engine.connect()
#Create df and write to temp table
df = pd.DataFrame(item_bank,columns=['email','id', 'mbid','artist','track','plays','track_count'])
df.to_sql(con=conn, name='temp', if_exists='replace',index=False)
It successfully creates a MySQL table with all of the data types as 'Text' except for user_tracks which writes as a bigint(20).
I then run this, but the table does not update. It is especially strange to me because I have many scripts that use a similar method, and the only thing I remember changing was to stop updating the other static columns.
mysql_statement = """
INSERT INTO pickaresk.permanent
(email, id, mbid, artist, track, plays, track_count)
SELECT * FROM temp
ON DUPLICATE KEY UPDATE
plays=temp.plays,
track_count=temp.track_count,
lastfm_last_update=NOW()
;
"""
conn.execute(mysql_statement)
conn.close()
The permanent table column's schema that are being updated is shown below. The multiple unique key constraint is the combination of id and email. I also confirmed that there are duplicate keys in both tables
| Field | Type | Null | Key | Default | Extra |
+-------------------------------+--------------+------+-----+---------+-------+
|
| id | varchar(255) | NO | | NULL | |
| email | varchar(120) | NO | MUL | NULL | |
| mbid | varchar(120) | YES | | NULL | |
| artist | varchar(250) | YES | | NULL | |
| track | varchar(250) | YES | | NULL | |
| plays | float | YES | | NULL | |
| track_count | int(11) | YES | | NULL | |
| lastfm_last_update | datetime | YES | | NULL | |
+-------------------------------+--------------+------+-----+---------+-------+

Django: Field 'X' doesn't have a default value

The models.py of my table is :
class TestCaseGlobalMetricData(models.Model):
testRunData = models.ForeignKey(TestRunData)
testCase = models.CharField(max_length=200)
cvmTotalFree = models.IntegerField(default=0)
systemFree = models.IntegerField(default=0)
sharedMemory = models.IntegerField(default=0)
And the part of the code which uses this table is
TestCaseGlobalMetricData(testRunData=testRunDataObj,
testCase=tokens['tc_name'],
timestamp=tokens['timestamp'],
cvmTotalFree=totalFree).save()
When this line is executed the following error is seen,
File "/web/memmon/eink-memmon/projects/process.py", line 382, in process_cvm_usage
cvmTotalFree=totalFree).save()
.....
Warning: Field 'sharedMemory' doesn't have a default value
I tried inserting into the table using the python manage.py shell and it works fine.
The schema of the table as seen in mysql is
+-----------------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-----------------+--------------+------+-----+---------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| testRunData_id | int(11) | NO | MUL | NULL | |
| testCase | varchar(200) | NO | | NULL | |
| cvmTotalFree | int(11) | NO | | NULL | |
| systemFree | int(11) | NO | | NULL | |
| sharedMemory | int(11) | NO | | NULL | |
+-----------------+--------------+------+-----+---------+----------------+
Note:
The field sharedMemory was added in the models.py today. I did proper migrations using makemigrations and migrate commands.

Insert Data to multiple MySQL database tables from .csv file

I have a database with tables: person, player, coach, and team. All the tables have an auto-increment id field as the primary key. Person has id, firstname, lastname. Player and coach both have the id field, as well as person_id and team_id as foreign keys to tie them to a team.id or person.id field in the other tables.
Now in order to fully populate these tables, I have several csv files with the list of names of the players in each team. Can I write a bash or python script to take this data and input not only the names to the person table, but also have the proper person and team id values put into the player table?
If the question isn't clear just ask and I'll do what I can to clarify. Thanks.
mysql> describe person;
+-----------+-------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-----------+-------------+------+-----+---------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| firstname | varchar(30) | NO | | NULL | |
| lastname | varchar(30) | NO | | NULL | |
+-----------+-------------+------+-----+---------+----------------+
mysql> describe player;
+-----------+---------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-----------+---------+------+-----+---------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| person_id | int(11) | NO | MUL | NULL | |
| team_id | int(11) | NO | MUL | NULL | |
+-----------+---------+------+-----+---------+----------------+
mysql> describe team;
+-----------+-------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-----------+-------------+------+-----+---------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| teamname | varchar(25) | NO | | NULL | |
| location | varchar(40) | NO | | NULL | |
| city | varchar(25) | NO | | NULL | |
| state | varchar(2) | NO | | NULL | |
| venue | varchar(35) | NO | | NULL | |
| league_id | int(11) | NO | MUL | NULL | |
+-----------+-------------+------+-----+---------+----------------+
And here is an example of the csv file content:
(AL-Central-Indians.csv)
Fausto,Carmona
Carlos,Carrasco
Kelvin,De La Cruz
Chad,Durbin

You can do this directly using the mysql command as follows:
load data local infile 'AL-Central-Indians.csv' into table player
fields terminated by ','
enclosed by '"'
lines terminated by '\n'
(person_id, team_id)
I got that from here. Although that page also deals with exporting excel into CSV first.

Using an ORM might be an overkill for your purpose, but it makes your life easy if you need to do real work with data. It will require you to install some software, but if you are willing to learn some new stuff, you will probably gain a lot down the road. Luckily, it is not very hard to start using it, for example, using Django:
Download and install django
Create a new project using django-admin startproject myproject
Create a new app: ./manage.py startapp myapp
Change the database connection parameters in settings.py
./manage.py inspectdb should create the models for you. Use ./manage.py inspectdb > myapp/models.py to save it.
Execute export DJANGO_SETTINGS_MODULE=settings to allow you to use django from command line scripts
Now you can create an import_players.py script in this fashion:
from myapp.models import Player, Person, Coach, Team
for my_file in my_files: # TODO: Iterate through your files
team = Team.objects.create(name=my_team_name) # creates a db record for a team
for line in lines_in_my_file: # TODO: Iterate through lines in your file
player = Player.objects.create(name=my_player_name, team=team) creates a db record for a player
See this to learn how to work with models: https://docs.djangoproject.com/en/dev/topics/db/models/