We Keep Coding

How to specify length prefix in bcp command?

I'm using bcp tool to import CSV into a sql server table. I'm using python subprocess to execute the bcp command. My sample bcp command is like below:
bcp someDatabase.dbo.sometable IN myData.csv -n -t , -r \n -S mysqlserver.com -U myusername -P 'mypassword'
the command executes and says
0 rows copied.
Even if i remove the -t or -n option, the message is still the same. I read from sql server docs that there is something called length prefix(if bcp tool is used with -n (native) mode).
How can i specify that length prefix with bcp command?
My goal is to import CSV into a sql server table using bcp tool. I first create my table according to my date in the CSV file and i dont create a format file for bcp. I want all my data to be inserted correctly(according to the data type i have sepecified in my table).

If it is a csv file then do not use -n, -t or -r options. Use -e errorFileName to catch the error(s) you may be encountering. You can then take the appropriate steps.

It is a very common practice with ETL tasks to first load text files into a "load" table that has all varchar/char data types. This avoids any possible implied data conversion errors that are more difficult/time-consuming to troubleshoot via BCP. Just pass the character data in the text file into character datatype columns in SQL Server. Then you can move data from the "load" table into your final destination table. This will allow you to use the MUCH more functional T-SQL commands to handle transformation of data types. Do not force BCP/SQL Server to transform your data-types for you by going from text file directly into your final table via BCP.
Also, I would also suggest visually inspecting your incoming data file to confirm it is formatted as specified. I often see mixups betweeen \n and \r\n for line terminator.
Last, when loading the data, you should also use the -e option as Neeraj has stated. This will capture "data" errors (it does not report command/syntax errors; just data/formatting errors). Since you incoming file is an ascii text file, you DO want to use the -c option for loading into the all-varchar "load" table.

Python - Dynamic variable in Redshift SQL Query

I'm developing in a python environment and I want to call an sql query using psycopg2
Lets say I have the following UNLOAD command in an .sql file:
UNLOAD
(
'
Some SQL Query
'
)
TO 's3://%PATH%'
...
In the sql file the %PATH% should be explicit like: 'folder1/folder3/file_name'.
But I want the python program to set this %PATH% in runtime. which means that the .sql file containts something like %PATH% and will be set only in runtime.
Any idea of how to do it?

Implementing it this way will give you a tough time.
The best way to do is to dump the file at a static location:
UNLOAD
(
'
Some SQL Query
'
)
TO 's3://path/to/static/s3_bucket'
...
and then use (via a shellscript / or opt for a suitable command for any other script)
aws s3 mv $source $destination
Here, you may pass any value for $destination which can be easily populated during run-time.
In short, you've dumped the file in s3 at a fixed location (using
UNLOAD) and moved it to the location of your choice or a location
populated at run time (using aws s3 mv...)

You simply specify a replacement field in your SQL file, and the use a format command.
Create your file like this
UNLOAD ('Some SQL Query')
TO 's3://{bucket}/{key}'
And use this file in python like
template = open('file1.sql', 'r').read()
query = template.format(bucket='mybucket', key='folder/file.csv')

You would not be able to set up the UNLOAD path dynamically at runtime, however you could put your SQL statement in a something like a shell/python script where you can create variables with the path you'd like and then pass them into the query.
This UNLOAD utility by AWS will get you started if you decide to go with a python script.

Error with .output in sqlite3

I'm getting the following error
conn = sqlite3.connect('./mydb.db')
c = conn.cursor()
c.execute('.output ./mytable.sql')
conn.close()
c.execute('.output ./mytable.sql') sqlite3.OperationalError: near ".":
syntax error

That's because .output is a command for the command line sqlite tool. It is not a valid SQL command. Hence it cannot be used when you are using sqlite through a library, only interactively through the command prompt.
None of the shell commands listed at https://www.sqlite.org/cli.html can work as they are something totally separate from the sqlite itself. You can think of them as if they were part of a GUI program - it would not make sense to be able to access something in a GUI program through the library.
What you have to do is fetch the data yourself and parse it yourself and output in the way you want.
Another option is to call the sqlite shell and pass the commands you want it to execute. Something like:
sqlite3 < '.output FILE \n SELECT * FROM TABLE'
(this is untested...)

Using Postgres's COPY FROM file query in Python without writing to a temporary file

I need to load data from some source data sources to a Postgres database.
To do this task, I first write the data to a temporary CSV file and then load data from the CSV file to Postgres database using COPY FROM query. I do all of this on Python.
The code looks like this:
table_name = 'products'
temp_file = "'C:\\Users\\username\\tempfile.csv'"
db_conn = psycopg2.connect(host, port, user, password, database)
cursor = db_conn.cursor()
query = """COPY """ + table_name + """ FROM """ + temp_file + " WITH NULL AS ''; """
cursor.execute(query)
I want to avoid the step of writing to the intermediate file. Instead, I would like to write to a Python object and then load data to postgres database using COPY FROM file method.
I am aware of this technique of using psycopg2's copy_from method which copies data from a StringIO object to the postgres database. However, I cannot use psycopg2 for a reason and hence, I don't want my COPY FROM task to be dependent on a library. I want it to be Postgres query which can be run by any other postgres driver as well.
Please advise a better way of doing this without writing to an intermediate file.

You could call the psql command-line tool from your script (i.e. using subprocess.call) and leverage its \copy command, piping the output of one instance to the input of another, avoiding a temp file. i.e.
psql -X -h from_host -U user -c "\copy from_table to stdout" | psql -X -h to_host -U user -c "\copy to_table from stdin"
This assumes the table exists in the destination database. If not, a separate command would first need to create it.
Also, note that one caveat of this method is that errors from the first psql call can get swallowed by the piping process.

psycopg2 has integrated support for the COPY wire-protocol, allowing you to use COPY ... FROM STDIN / COPY ... TO STDOUT.
See Using COPY TO and COPY FROM in the psycopg2 docs.
Since you say you can't use psycopg2, you're out of luck. Drivers must understand COPY TO STDOUT / COPY FROM STDIN in order to use them, or must provide a way to write raw data to the socket so you can hijack the driver's network socket and implement the COPY protocol yourself. Driver specific code is absolutely required for this, it is not possible to simply use the DB-API.
So khampson's suggestion, while usually a really bad idea, seems to be your only alternative.
(I'm posting this mostly to make sure that other people who find this answer who don't have restrictions against using psycopg2 do the sane thing.)
If you must use psql, please:
Use the subprocess module with the Popen constructor
Pass -qAtX and -v ON_ERROR_STOP=1 to psql to get sane behaviour for batching.
Use the array form command, e.g. ['psql', '-v', 'ON_ERROR_STOP=1', '-qAtX', '-c', '\copy mytable from stdin'], rather than using a shell.
Write to psql's stdin, then close it, and wait for psql to finish.
Remember to trap exceptions thrown on command failure. Let subprocess capture stderr and wrap it in the exception object.
It's safer, cleaner, and easier to get right than the old-style os.popen2 etc.

automation : Script to take a mysqldump into a file named by date/time of backup

I tried fabric with a '>' in the command string. It always gives out an error code 2. Currently dabbling with subprocess.call, subprocess.check_output and keeping stdout="filesocket". Not working. The only thing that gets written in the file is the USAGE for mysqldump. Using shlex to parse 'mysqldump -uroot -ppassword database table1 table2'
All this because I don't know shell scripting with string variables from the 'date' utility. How do I take the current date and use it to name the backup file in shell script. OR how do I get this thing done in python?
Thanks in advance.
regards.

You can get a custom date out of date using the following syntax.
CUSTOM_DATE=$(date "+%Y-%m-%d_%H_%M_%S")
The easiest way to accomplish this is to put a script on the remote end that does 'everything'
#!/bin/bash
CUSTOM_DATE=$(date "+%Y-%m-%d_%H_%M_%S")
mysqldump -u admin -p password database table1 table2 >/path/to/backups/mysqldump.${CUSTOM_DATE}.db

"How do I take the current date and use it to name the backup file in shell script. OR how do I get this thing done in python?"
from datetime import datetime
filename = 'mysql_backup_{0:%Y%m%d_%H%M}.sql'.format(datetime.now())
# filename == 'mysql_backup_20120227_0952.sql'

My Answer from related stackoverflow post.
In Microsoft Windows, run below command in CMD
mysqldump -u USERNAME -pYOURPASSWORD --all-databases > "C:/mysql_backup_%date:~-10,2%-%date:~-7,2%-%date:~-4,4%-%time:~0,2%_%time:~3,2%_%time:~6,2%.sql"
Output file will look like,
mysql_backup_21-02-2015-13_07_18.sql
If you want to automate the backup process, then you can use Windows Task Scheduler, and put above command in .bat file - task scheduler will run the .bat file at specified interval.