I have two tables
front_employee (Employee model in Django)
+-----------------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-----------------+--------------+------+-----+---------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| order | int(11) | NO | | NULL | |
| name | varchar(100) | YES | | NULL | |
| position | varchar(100) | YES | | NULL | |
| description | longtext | YES | | NULL | |
| employee_img_id | int(11) | NO | MUL | NULL | |
| language_id | int(11) | NO | MUL | NULL | |
+-----------------+--------------+------+-----+---------+----------------+
And front_employeepicture (EmployeePicture in Django)
+-------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-------+--------------+------+-----+---------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| order | int(11) | NO | | NULL | |
| img | varchar(100) | YES | | NULL | |
+-------+--------------+------+-----+---------+----------------+
I would like to perform this query:
SELECT a.id, a.name, b.img
FROM front_employee a
INNER JOIN front_employeepicture b
ON a.employee_img_id = b.id
For now I have
context['employee'] = Employee.objects.all().order_by('order')
And I tried something like
context['employee'] = Employee.objects.select_related('EmployeePicture')
Without result. Any idea?
Django's ORM is really powerful. You actually don't have to do joins like that, ever. When you access the field, the ORM performs the join on the fly, returning the result you want.
first_employee = Employee.objects.all().first()
employee_picture = first_employee.employee_img
And then employee_picture should have the picture of the first employee. It needed to internally do an inner join to figure that out, but Django hides all that for you. (I may have gotten your variable names wrong, sorry).
What select_related does is pre-fetch a relation for every object in a queryset. That means less trips back and forth to the database, which makes your functions faster. But, Django allows you to traverse relationships just by accessing attributes.
Try messing around with this stuff in your django shell. I would add django_extensions to your app (pip install django-extensions, then add django_extensions to your INSTALLED_APPS, then run python manage.py shell_plus). And if not, python manage.py shell works fine, but you have to import your models manually.
Related
My ON DUPLICATE UPDATE clause stopped updating and I'm not sure why.
Below is my code to create a temporary table via Pandas:
#connect to mysql database
engine = sqlalchemy.create_engine('mysql://username:#localhost/db?charset=utf8')
conn = engine.connect()
#Create df and write to temp table
df = pd.DataFrame(item_bank,columns=['email','id', 'mbid','artist','track','plays','track_count'])
df.to_sql(con=conn, name='temp', if_exists='replace',index=False)
It successfully creates a MySQL table with all of the data types as 'Text' except for user_tracks which writes as a bigint(20).
I then run this, but the table does not update. It is especially strange to me because I have many scripts that use a similar method, and the only thing I remember changing was to stop updating the other static columns.
mysql_statement = """
INSERT INTO pickaresk.permanent
(email, id, mbid, artist, track, plays, track_count)
SELECT * FROM temp
ON DUPLICATE KEY UPDATE
plays=temp.plays,
track_count=temp.track_count,
lastfm_last_update=NOW()
;
"""
conn.execute(mysql_statement)
conn.close()
The permanent table column's schema that are being updated is shown below. The multiple unique key constraint is the combination of id and email. I also confirmed that there are duplicate keys in both tables
| Field | Type | Null | Key | Default | Extra |
+-------------------------------+--------------+------+-----+---------+-------+
|
| id | varchar(255) | NO | | NULL | |
| email | varchar(120) | NO | MUL | NULL | |
| mbid | varchar(120) | YES | | NULL | |
| artist | varchar(250) | YES | | NULL | |
| track | varchar(250) | YES | | NULL | |
| plays | float | YES | | NULL | |
| track_count | int(11) | YES | | NULL | |
| lastfm_last_update | datetime | YES | | NULL | |
+-------------------------------+--------------+------+-----+---------+-------+
This seems to be the most obscure issue I have dealt with so far. I am struggling hard.
I have an application that collects remote data from some hardware and saves it to a database. I renamed one of the columns using alembic. I have a development database and a testing database. Both are on the same server (MySQL via MariaDB on CentOS 7).
I interact with the development server via the app running on my laptop and the testing db is interacted with via the app running on a clone of the production server.
All servers and databases are set up using Ansible, so differences are limited to usernames and passwords.
Below is the ansible snippet from the upgrade script
def upgrade():
op.alter_column('units', 'ip_address', new_column_name='ipv4', existing_type=sa.String(100))
If I run the app from my laptop (using my IDE), the data is saved ok.
If I run the script on the testing server manually (env)$ python app.py, the data is saved ok.
But, here's the problem, If I run the script using supervisord, I get an SQLAlchemy error. (excerpts below, full traceback)
...
File "thefile.py", line 64, in run
self.data['my_wellsites'][w.name]['ipv4'] = wellsite.unit.ipv4
...
sqlalchemy.exc.InternalError: (InternalError) (1054, u"Unknown column 'units.ipv4' in 'field list'") 'SELECT units.id AS units_id, units.unittype_id AS units_unittype_id, units.serial AS units_serial, units.ipv4 AS units_ipv4, units.mac_address AS units_mac_address, units.engine_hours AS units_engine_hours, units.gen_odometer AS units_gen_odometer, units.gen_periodic AS units_gen_periodic, units.notes AS units_notes \nFROM units \nWHERE units.id = %s' (1,)
...
SQLAlchemy models..
class Unit(Model):
__tablename__ = 'units'
__table_args__ = {'mysql_engine': 'InnoDB'}
id = Column(Integer, Sequence('unit_id_seq'), primary_key=True)
wellsites = relationship('Wellsite', order_by='Wellsite.id', backref='unit')
ipv4 = Column(String(16), unique=True)
...
class Wellsite(Model):
__tablename__ = 'wellsites'
__table_args__ = {'mysql_engine': 'InnoDB'}
id = Column(Integer, Sequence('wellsite_id_seq'), primary_key=True)
unit_id = Column(Integer, ForeignKey('units.id'), nullable=False)
...
etc/supervisord.conf..
...
[program:datacollect]
command = /home/webdev/mydevelopment/git/ers_data_app/env/bin/python /home/webdev/mydevelopment/git/ers_data_app/data_monitoring/collection_manager.py
stdout_logfile=/home/webdev/logs/datacollect.log
stderr_logfile=/home/webdev/logs/datacollecterr.log
autostart=true
autorestart=unexpected
startsecs=10
I tried running an additional alembic upgrade with
op.create_unique_constraint('uq_ipv4', 'units', ['ipv4'])
No dice.
The traceback is identical (using diff program) except timestamps
Here are the two database descriptions of the units table (identical)
MariaDB [ers_DEV]> show columns from units;
+--------------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+--------------+--------------+------+-----+---------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| unittype_id | int(11) | YES | MUL | NULL | |
| serial | varchar(10) | YES | UNI | NULL | |
| ipv4 | varchar(100) | YES | UNI | NULL | |
| mac_address | varchar(17) | YES | UNI | NULL | |
| engine_hours | int(11) | YES | | NULL | |
| gen_odometer | tinyint(1) | YES | | NULL | |
| gen_periodic | tinyint(1) | YES | | NULL | |
| notes | mediumtext | YES | | NULL | |
+--------------+--------------+------+-----+---------+----------------+
MariaDB [ers_TEST]> show columns from units;
+--------------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+--------------+--------------+------+-----+---------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| unittype_id | int(11) | YES | MUL | NULL | |
| serial | varchar(10) | YES | UNI | NULL | |
| ipv4 | varchar(100) | YES | UNI | NULL | |
| mac_address | varchar(17) | YES | UNI | NULL | |
| engine_hours | int(11) | YES | | NULL | |
| notes | mediumtext | YES | | NULL | |
| gen_odometer | tinyint(1) | YES | | NULL | |
| gen_periodic | tinyint(1) | YES | | NULL | |
+--------------+--------------+------+-----+---------+----------------+
The issue was at the top of the /etc.supervisord.conf file. In it, an environment variable is set that was pointing the script to the wrong database, overriding the environment variable that was set and examined from anywhere else. This environment variable is only set when the script is being run from supervisor and was causing the trouble
I have 2 Models in my django app.
The corresponding tables for the two models are:
Table 1:
+-------------+---------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-------------+---------+------+-----+---------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| post_id | int(11) | NO | MUL | NULL | |
| user_id | int(11) | NO | MUL | NULL | |
+-------------+---------+------+-----+---------+----------------+
Table2:
+---------------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+---------------+--------------+------+-----+---------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| p_text | varchar(200) | NO | | NULL | |
| p_slug | varchar(50) | YES | MUL | NULL | |
| user_id | int(11) | NO | MUL | NULL | |
+---------------+--------------+------+-----+---------+----------------+
Now what i want is to write the equivalent of the below query in my django view in the best way possible? The query i want to write is a simple join as:
select B.p_slug from Table1 A, Table2 B where A.post_id = B.id;
I tried but could not get it working. Any help please? How to implement the above query in Django views
The models are: Model1:
class Model1(models.Model):
user = models.ForeignKey(settings.AUTH_USER_MODEL)
post = models.ForeignKey(Model1)
class Model2(models.Model):
p_text = models.CharField(max_length=200)
user = models.ForeignKey(settings.AUTH_USER_MODEL)
p_slug = models.SlugField(null=True,blank=True)
Try this: Model2.objects.filter(pk__in=Model1.objcts.values_list('post_id', flat=True)).values('p_slug'). I hope it helps.
I have a database with tables: person, player, coach, and team. All the tables have an auto-increment id field as the primary key. Person has id, firstname, lastname. Player and coach both have the id field, as well as person_id and team_id as foreign keys to tie them to a team.id or person.id field in the other tables.
Now in order to fully populate these tables, I have several csv files with the list of names of the players in each team. Can I write a bash or python script to take this data and input not only the names to the person table, but also have the proper person and team id values put into the player table?
If the question isn't clear just ask and I'll do what I can to clarify. Thanks.
mysql> describe person;
+-----------+-------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-----------+-------------+------+-----+---------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| firstname | varchar(30) | NO | | NULL | |
| lastname | varchar(30) | NO | | NULL | |
+-----------+-------------+------+-----+---------+----------------+
mysql> describe player;
+-----------+---------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-----------+---------+------+-----+---------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| person_id | int(11) | NO | MUL | NULL | |
| team_id | int(11) | NO | MUL | NULL | |
+-----------+---------+------+-----+---------+----------------+
mysql> describe team;
+-----------+-------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-----------+-------------+------+-----+---------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| teamname | varchar(25) | NO | | NULL | |
| location | varchar(40) | NO | | NULL | |
| city | varchar(25) | NO | | NULL | |
| state | varchar(2) | NO | | NULL | |
| venue | varchar(35) | NO | | NULL | |
| league_id | int(11) | NO | MUL | NULL | |
+-----------+-------------+------+-----+---------+----------------+
And here is an example of the csv file content:
(AL-Central-Indians.csv)
Fausto,Carmona
Carlos,Carrasco
Kelvin,De La Cruz
Chad,Durbin
You can do this directly using the mysql command as follows:
load data local infile 'AL-Central-Indians.csv' into table player
fields terminated by ','
enclosed by '"'
lines terminated by '\n'
(person_id, team_id)
I got that from here. Although that page also deals with exporting excel into CSV first.
Using an ORM might be an overkill for your purpose, but it makes your life easy if you need to do real work with data. It will require you to install some software, but if you are willing to learn some new stuff, you will probably gain a lot down the road. Luckily, it is not very hard to start using it, for example, using Django:
Download and install django
Create a new project using django-admin startproject myproject
Create a new app: ./manage.py startapp myapp
Change the database connection parameters in settings.py
./manage.py inspectdb should create the models for you. Use ./manage.py inspectdb > myapp/models.py to save it.
Execute export DJANGO_SETTINGS_MODULE=settings to allow you to use django from command line scripts
Now you can create an import_players.py script in this fashion:
from myapp.models import Player, Person, Coach, Team
for my_file in my_files: # TODO: Iterate through your files
team = Team.objects.create(name=my_team_name) # creates a db record for a team
for line in lines_in_my_file: # TODO: Iterate through lines in your file
player = Player.objects.create(name=my_player_name, team=team) creates a db record for a player
See this to learn how to work with models: https://docs.djangoproject.com/en/dev/topics/db/models/
I have a django project for which I'm trying to update the database tables without using any migration tools. I'm adding a 'slug' field to a model which simply references it's name, and I was intending on doing so by replicating what was in another table that already has a slug for.
So, I have an existing table 'gym' in the database as follows
+----------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+----------+--------------+------+-----+---------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| gym_name | varchar(50) | NO | UNI | NULL | |
| gym_slug | varchar(50) | NO | MUL | NULL | |
| created | datetime | NO | | NULL | |
| modified | datetime | NO | | NULL | |
+----------+--------------+------+-----+---------+----------------+
and I have another table 'wall' in the database as follows:
+-----------+-------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-----------+-------------+------+-----+---------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| wall_name | varchar(50) | NO | UNI | NULL | |
| gym_id | int(11) | NO | MUL | NULL | |
| created | datetime | NO | | NULL | |
| modified | datetime | NO | | NULL | |
+-----------+-------------+------+-----+---------+----------------+
This would get me part of the way there:
ALTER TABLE wall ADD COLUMN wall_slug varchar(50);
But I'm not certain how to figure out where the foreign key in the first table is referencing and thus where I should point the new one.
End goal: the wall_slug field tied to a unique wall_name field. Hope that makes sense.
from django.template.defaultfilters import slugify
class Wall(Model):
name = CharField(max_length=60)
slug = CharField(max_length=60, unique=True)
...
def save(self, *args, **kwargs):
if self.slug == '':
self.slug = slugify(self.name)
super(Wall, self).save(*args, **kwargs)
This will generate for you a slug based on the name when saving. It also protects you from changing the slug next time you change the name, so URLs remain stable and unchanged.