I have a SQL database and I want to fill tables with a script. The following source code shows what I have done so far:
#!/bin/tcsh
mysql -h dbs1 -D my_devel -u USER --password=XYZ
insert into my_table ( col_id, type, file, result, signature) values (***,'###','+++','$$$','...');
exit
According that I pass the parameters '*','#','+','$' and '.' via a method defined in python.
But somehow this is not working for me. Can someone tell me why, or suggest an alternative?
This seems to work fine for me:
#!/bin/tcsh
# count-rows.csh
mysql -h 127.0.0.1 -D tpcw -u root <<EOF
select count(*) from $1;
exit
EOF
You need to send the commands to execute to the standard input of the mysql utility. There are several ways to do that, but using a here document is probably the easiest method.
That said, if your data comes from a Python program, you should look at using a proper MySQL connector for Python. You should see a definite performance increase, but more importantly you will be able to handle any errors properly.
Expanding on Martijn's comment, one of the easiest ways to connect to a MySQL database is using python-mysqldb.
Related
I have the following lines of code:
sql = "source C:\\My Dropbox\\workspace\\projects\\hosted_inv\\create_site_db.sql"
cursor.execute (sql)
When I execute my program, I get the following error:
Error 1064: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'source C:\My Dropbox\workspace\projects\hosted_inv\create_site_db.sql' at line 1
Now I can copy and past the following into mysql as a query:
source C:\\My Dropbox\\workspace\\projects\\hosted_inv\\create_site_db.sql
And it works perfect. When I check the query log for the query executed by my script, it shows that my query was the following:
source C:\\My Dropbox\\workspace\\projects\\hosted_inv\\create_site_db.sql
However, when I manually paste it in and execute, the entire create_site_db.sql gets expanded in the query log and it shows all the sql queries in that file.
Am I missing something here on how mysqldb does queries? Am I running into a limitation. My goal is to run a sql script to create the schema structure, but I don't want to have to call mysql in a shell process to source the sql file.
Any thoughts? Thanks!
As others said, you cannot use the command source in MySQLdb Python API
So, instead of running that, load the file and execute it
Lets say your .sql file has
create database test;
Read the content like
sql=open("test.sql").read()
And then execute it
cursor.execute(sql);
You will get new database "test"
The source command is one of the built-in commands recognized only by the mysql command-line client. It is not supported as a statement you can execute via any API.
Some people think you can simply split an SQL script file on the ";" statement terminator and call execute() on each line you get. But there are numerous exception cases:
Statements that are built-in commands like CONNECT, SOURCE, CHARSET, WARNINGS, QUIT, etc.
Note that built-in commands don't need to terminate in ; for example DELIMITER.
Statements that contain ; but not as a terminator, like CREATE TRIGGER.
Statements that contain ; inside string literals or comments or even quoted identifiers.
Comments lines.
To load an SQL script programmatically, you'd have to duplicate a fair amount of the functionality of the mysql client. So it's best if you just fork a process to actually execute that client program with the script as input.
See also:
Loading .sql files from within PHP
is it possible to call a sql script from a stored procedure in another sql script?
composing multiple mysql scripts
Running Database scripts in C#
'source' is not an SQL command, but an internal command of the mysql command line client.
I ran into the same problem!
As a solution I installed the library sqlparse and used the sqlparse.split( sql ) results. I had to check that sql_parts don't include blank lines as solo statements... Otherwise "WOW" sqlparse is pretty great and exactly what I needed!
import sqlparse
....
sql = open("test.sql").read()
sql_parts = sqlparse.split( sql )
for sql_part in sql_parts:
if sql_part.strip() == '':
continue
cursor.execute( sql_part )
FYI: If you do not run each statement on its own you may get the error "Commands out of sync; you can't run this command now". I only got this error after I added some more queries to my sql file - not the first time around.
I believe the "source" command is specific to the mysql shell executable - it is not an sql command and cannot be interpreted correctly when executed as an sql statement.
To achieve your goal, you probably need to read your script file and parse it into individual sql statements, then execute them one at a time with your cursor.
I need to create a bash(or python) script which gives me the availability status of multiple databases which are on different servers. I found that I can get the status using this url "http://marklogic:8002/manage/v2/database/$DBNAME/?view=status". But I have for about twenty different DBs. When you open this link it generates an xml with database details. Can you please advise how can I loop all the links and grep only the status row ? Or if you have any other idea please advise
It might be worth looking into the MarkLogic Python API project on Github:
https://github.com/marklogic/python_api
HTH!
You can keep the dbnames in a file and then use for loop around it.
for a in `cat dbname.txt`
do
status = `wget -qO- "http://marklogic:8002/manage/v2/database/${a}/?view=status"`
echo $a, $status
done
yep, I did it through curl --anyauth --user user:pass "http://marklogic:8002/manage/v2/database/${a}/?view=status
I am trying to run a mysql command through python pipe
cmd='CREATE DATABASE {0} ; GRANT ALL PRIVILEGES ON {1}.* TO "{2}"#"%" identified by password "{3}";'.format(requested_database_name,requested_database_name,'xyz',requested_password)
where the password is the hash generated from mysql. I always get
["ERROR 1064 (42000) at line 1: You have an error in your SQL syntax; check the manual that
corresponds to your MySQL server version for the right syntax to use near '% identified by password *ASSFS3FS2' at line 1"]
But when I see the output for the cmd I am trying to run
CREATE DATABASE db1 ; GRANT ALL PRIVILEGES ON db1.* TO "xyz"#"%" identified by password "*ASSFS3FS2";
it looks good and runs well in the mysql prompt directly.
I use pipe = pexpect.spawn(cmd)
to run these commands. I am not sure where I am going wrong in terms of formatting.
You don't show us the exact sequence of commands you use (surely you don't pexpect.spawn that specific MySQL command!) but it sure looks like you're passing cmd through a shell which, of course, is eating up those indispensable double-quote characters.
With no idea of what Python version you're using (since you chose not to tell us), it's hard to know what to suggest. In 3.3 or later, shlex.quote is good. If you're stuck on an earlier version no later than 2.7, pipes.quote might help. But probably your best bet might be simply to replace each of those " with an escaped version:
cmd = cmd.replace('"', r'\"')
and try using the new version of cmd instead of your previous one -- this one should work independently of your Python version.
I'm doing a script to add many dotSql files that I have to MySQL. So far, I have the path of each file ready, the name of each file ready, the join between those too ready, but im struggling at handling the command, I must execute to add them to MySQL.
system("mysql -u root -p database < "+ files[i] +"")
This is running inside a for, the problem is that I have to input the password each loop, is there a way to automatically do the input which is the same every time? It's the password...
figured out by looking the mysql documentation, that you can actually do this:
mysql -u root -ppassword
and it doesnt ask for the password anymore, if every one still knows how to handle the responding input of the past command please let me know, it would be interesting to know how handle it from python
I have a pretty long query i need to execute, and i'm using prepared statements to insert the values to the query. As far as i know, it's suppose to work, but it's not working!
Here's the code:
cursor = connection.cursor()
rows = cursor.execute(r"""SELECT... %s.. %s.. %s...""",(val1,val2,val3,))
rows = cursor.fetchall()
rows = text_position(rows)
If i insert a simple-quote it breaks, and same thing if i insert any non-english characters. It works with double-quotes as i wrapped the statement in them. Any suggestions?
This is a general technique for debugging SQL (especially when generated from another language, like Python):
Stop the mysql server (assuming you run mysqld without logging):
sudo invoke-rc.d mysql stop
Run mysqld with logging turned on:
sudo mysqld --log=/tmp/myquery.log
Run your script.
Look in /tmp/myquery.log for the SQL command that MySQLdb sent to mysqld.
This might give you enough of a clue to solve the problem, otherwise, please post the complete SQL command in Python
cursor.execute(r"""SELECT... %s.. %s.. %s...""",(val1,val2,val3,))
and the corresponding SQL as seen by mysqld in /tmp/myquery.log.
Once you are done debugging, you can shutdown the logging mysqld and restart it with:
sudo invoke-rc.d mysql restart