Currently I have two separate statements being passed through to Postgres (Greenplum).
1. Truncates a table
2. loads data using \copy
myStr="export PGPASSWORD=" + dbPass + "; psql -h " + dbHost + " -p " + dbPort + " -d " + dbName + " -U " + dbUser + " -c " + "\"" + "truncate table " + dbTable + ";\""
print(myStr)
subprocess.call(myStr,shell=True)
myStr="export PGPASSWORD=" + dbPass + "; psql -h " + dbHost + " -p " + dbPort + " -d " + dbName + " -U " + dbUser + " -c " + "\"" + "\\" + "copy " + dbTable + " from " + "'" + csvfile + "' with " + copyOpts + ";" + "select count(*) from " + dbTable + ";\""
print(myStr)
subprocess.call(myStr,shell=True)
Sometimes the load has errors but the truncate already happened, so I'm trying to run the two statements in one connection so I can put a transcation block (BEGIN ... COMMIT;) that way if the data load fails it will rollback to before the truncate happens.
I tried the below method:
myStr="export PGPASSWORD=" + dbPass + "; psql -h " + dbHost + " -p " + dbPort + " -d " + dbName + " -U " + dbUser + " -c " + "\"" + "truncate table " + dbTable + ";" + " \\" + "copy " + dbTable + " from " + "'" + csvfile + "' with " + copyOpts + ";" + "select count(*) from " + dbTable + ";\""
print(myStr)
Which resolves to the command:
export PGPASSWORD=abcde;
psql -h abcde.testserver.corp
-p 5432 -d namem -U username -c
"truncate table schema.example;
\copy schema.example from
'/home/testing/schema/schema.example_export.csv'
with header null as '' escape 'off' delimiter E',' ;
select count(*) from schema.example;"
However I am getting the error:
ERROR: syntax error at or near "\"
I believe this is due to the \ commands have to be on a separate line.
Is there a way to split the command into separate lines so I can execute ll the commands in a single connection?
The problem is that you can't separate backslash commands from other commands if you are using the -c option. You can send your commands via STDIN to psql using echo:
export PGPASSWORD=abcde;
echo "truncate table schema.example;
\copy schema.example from '/home/testing/schema/schema.example_export.csv' with header null as '' escape 'off' delimiter E',' ;
select count(*) from schema.example;" | psql -h abcde.testserver.corp -p 5432 -d namem -U username
That's a little bit clumsy. It's better to use subprocess.Popen
theCommand = """truncate table schema.example;
\copy schema.example from
'/home/testing/schema/schema.example_export.csv'
with header null as '' escape 'off' delimiter E',' ;
select count(*) from schema.example;"""
theProcess = subprocess.Popen("psql -h abcde.testserver.corp -p 5432 -d namem -U username",
stdin = subprocess.PIPE, stdout = subprocess.PIPE, stderr = subprocess.PIPE)
theOutput, theErrors = theProcess.communicate(input = theCommand)
But the best way should be avoiding shell commands and using a database adaptor like PyGreSQL.
Related
I have a .sql file which gets an input and runs, I should run it from a python code with some inputs, but it doesn't work. What is problem?
sql file:
declare
--define variables
v_workspace_id NUMBER;
BEGIN
select workspace_id into v_workspace_id
from apex_workspaces
where workspace = upper('&1');
DBMS_OUTPUT.PUT_LINE(v_workspace_id);
apex_application_install.set_workspace_id( v_workspace_id );
apex_application_install.generate_application_id;
apex_application_install.generate_offset;
EXCEPTION
WHEN OTHERS
THEN
RAISE;
END;
part of python file to run this file:
cmd_sql = 'echo quit | sqlplus -S ' + DB_USER + '/' + DB_USER_PWD + '#' + DB_HOST + ' #' + SQL_PATH + '\\' + 'install_apex_apps.sql ' + user_name + ' >> ' + LOG_FILE
os.system(cmd_sql)
user_name is given as an input to sql file.
PL/SQL
Change code to (note the /). Also as #Belayer sugessted in the comment section remove the EXCEPTION section.
SET SERVEROUTPUT ON;
declare
--define variables
v_workspace_id NUMBER;
BEGIN
select workspace_id into v_workspace_id
from apex_workspaces
where workspace = upper('&1');
DBMS_OUTPUT.PUT_LINE(v_workspace_id);
apex_application_install.set_workspace_id( v_workspace_id );
apex_application_install.generate_application_id;
apex_application_install.generate_offset;
END;
/
Python
Linux
import os
DB_USER = 'xxx'
DB_USER_PWD = 'xxx'
DB_HOST = 'xxx'
SQL_PATH = '/home/xxx/Documents/stack/'
LOG_FILE = '/home/xxx/Documents/stack/log.txt'
user_name = 'xxx'
# I override the log file with > for appending use >>
cmd_sql = 'echo quit | sqlplus -S ' + DB_USER + '/' + DB_USER_PWD + '#' + DB_HOST + ' #' + SQL_PATH + 'install_apex_apps.sql ' + user_name + ' > ' + LOG_FILE
os.system(cmd_sql)
def putCardMon():
interfaceName = input(('Type an interface name to put in monitor mode: '))
print(colored('[+] ', 'green') + 'Trying to put the interface ' + colored(interfaceName, 'yellow') + ' in monitor mode\n')
call(["sudo ifconfig " + interfaceName + " down; sudo iwconfig " + interfaceName + " mode monitor; sudo ifconfig " + interfaceName + " up"], shell=True)
interfaceMonCheck = Popen(["iwconfig " + interfaceName + " | grep Mode | cut -d ':' -f2 | cut -d ' ' -f1"], shell=True, stdout=PIPE, universal_newlines=True).communicate()[0].rstrip()
sleep (1)
---//lots of code before the putCardMon is called//---
interfacesList = Popen("ifconfig -a | grep Link | cut -d ' ' -f1", shell=True, stdout=PIPE, universal_newlines=True).communicate()[0].rstrip()
print('The available interfaces are:\n' + interfacesList + '\n')
putCardMon()
print('Checking if ' + interfaceName + ' is really in monitor mode.')
if interfaceMonCheck == 'Managed':
print('The interface ' + colored(interfaceName, "green") + ' is not in monitor mode! Check if you typed the interface name correctly or contact support.\n')
putCardMon()
elif interfaceMonCheck == 'Monitor':
print(colored('The interface ' + colored(interfaceName, "green") + ' is in monitor mode!'))
pass
else:
print(colored('There was an unexpected error. Contact support!\n', "red"))
exit()
The script works fine, the function does its job, but then when it gets to the checking part everything goes downhill.
Traceback (most recent call last):
File "script.py", line 76, in <module>
print('Checking if ' + interfaceName + ' is really in monitor mode.')
NameError: name 'interfaceName' is not defined
How come interfaceName is not defined if the function that assigns a string to it has already been called and successfully assigned a value to it?
I searched stack overflow for the same error, but all the threads were answered and it was either an indentation error or the function was defined after being invoked, which neither is the case here.
I'm really out of options. I tried everything.
From https://python-textbok.readthedocs.io/en/1.0/Variables_and_Scope.html#variable-scope-and-lifetime
A variable which is defined inside a function is local to that
function. It is accessible from the point at which it is defined until
the end of the function
Your variable interfaceName is only defined inside the scope of your function putCardMon(). It ceases to exist outside the scope of the function. Hence you are getting the error.
If you want to use the variable outside the function body, consider returning it and saving its value.
def putCardMon():
interfaceName = input(('Type an interface name to put in monitor mode: '))
print(colored('[+] ', 'green') + 'Trying to put the interface ' + colored(interfaceName, 'yellow') + ' in monitor mode\n')
call(["sudo ifconfig " + interfaceName + " down; sudo iwconfig " + interfaceName + " mode monitor; sudo ifconfig " + interfaceName + " up"], shell=True)
interfaceMonCheck = Popen(["iwconfig " + interfaceName + " | grep Mode | cut -d ':' -f2 | cut -d ' ' -f1"], shell=True, stdout=PIPE, universal_newlines=True).communicate()[0].rstrip()
sleep (1)
return interfaceName
# Later you can do this
interfaceName = putCardMon()
I'm getting this error when running my code:
prefix = ip.split('.')[0] + '.' + ip.split('.')[1] + '.' + ip.split('.')[2] + '.'
IndexError: list index out of range
I didn't write the script myself. I need to perform an arp scan on intranet and get as output every IP-addr that responds. My code is this:
#!/usr/bin/python
import logging
import subprocess
logging.getLogger("scapy.runtime").setLevel(logging.ERROR)
from scapy.all import *
if len(sys.argv) != 2:
print("Usage - ./arp_disc.py [interface]")
print("Example - ./arp_disc.py eth0")
print("Example will perform an ARP scan of the local subnet to which eth0 is assigned")
sys.exit()
interface = str(sys.argv[1])
ip = subprocess.check_output("ifconfig " + interface + " | grep 'inet addr' | cut -d ':' -f 2 | cut -d ' ' -f 1", shell=True).strip()
prefix = ip.split('.')[0] + '.' + ip.split('.')[1] + '.' + ip.split('.')[2] + '.'
for addr in range(0,254):
answer=sr1(ARP(pdst=prefix+str(addr)),timeout=1,verbose=0)
if answer == None:
pass
else:
print prefix+str(addr)
Use the following instead:
ip = subprocess.check_output("ifconfig " + interface + " | grep 'inet' | cut -d ':' -f 2 | cut -d '' -f 1", shell=True).strip()
ip= ip[5:]
I'm new at exporting data, I research all over the net but it was really hard for me to understand, can someone help me to know the basic about it.
This is my main problem: I want to download a specific data from mysql base on the date range I choose in my client, then when I click the download button, I want these data from mysql to be save in my computer together the user have the option to save it as CSV/Excel, I'm using python for my webservice. Thank you
This is my code right know in my webservice:
#api.route('/export_file/', methods=['GET', 'POST'])
def export_file():
if request.method == 'POST':
selectAttendance = """SELECT * FROM attendance"""
db.session.execute(selectAttendance)
db.session.commit()
f = csv.writer(open("file.csv", "w"))
for row in selectAttendance:
f.writerow([str(row)])
return jsonify({'success': True})
In general:
Set the mime header "Content-Type" part of the http header to the corresponding MIME-Type matching your data.
This tells the browser what type of data the webserver is going to send.
Send the actual data in the 'body'
With flask:
Forcing application/json MIME type in a view (Flask)
http://flask.pocoo.org/docs/0.10/patterns/streaming/
def get(self):
try:
os.stat(BACKUP_PATH)
except:
os.mkdir(BACKUP_PATH)
now = datetime.now() # current date and time
year = now.strftime("%Y")
month = now.strftime("%m")
day = now.strftime("%d")
time = now.strftime("%H:%M:%S")
date_time = now.strftime("%d_%m_%Y_%H:%M:%S")
TODAYBACKUPPATH = BACKUP_PATH + '/' + date_time
try:
os.stat(TODAYBACKUPPATH)
except:
os.mkdir(TODAYBACKUPPATH)
print ("checking for databases names file.")
if os.path.exists(DB_NAME):
file1 = open(DB_NAME)
multi = 1
print ("Databases file found...")
print ("Starting backup of all dbs listed in file " + DB_NAME)
else:
print ("Databases file not found...")
print ("Starting backup of database " + DB_NAME)
multi = 0
if multi:
in_file = open(DB_NAME,"r")
flength = len(in_file.readlines())
in_file.close()
p = 1
dbfile = open(DB_NAME,"r")
while p <= flength:
db = dbfile.readline() # reading database name from file
db = db[:-1] # deletes extra line
dumpcmd = "mysqldump -h " + DB_HOST + " -u " + DB_USER + " -p" + DB_USER_PASSWORD + " " + db + " > " + pipes.quote(TODAYBACKUPPATH) + "/" + db + ".sql"
os.system(dumpcmd)
gzipcmd = "gzip " + pipes.quote(TODAYBACKUPPATH) + "/" + db + ".sql"
os.system(gzipcmd)
p = p + 1
dbfile.close()
else:
db = DB_NAME
dumpcmd = "mysqldump -h " + DB_HOST + " -u " + DB_USER + " -p" + DB_USER_PASSWORD + " " + db + " > " + pipes.quote(TODAYBACKUPPATH) + "/" + db + ".sql"
os.system(dumpcmd)
gzipcmd = "gzip " + pipes.quote(TODAYBACKUPPATH) + "/" + db + ".sql"
os.system(gzipcmd)
# t = ("Your backups have been created in '" + TODAYBACKUPPATH + "' directory")
return "Your Folder have been created in '" + TODAYBACKUPPATH + "'."
I have code like:
if len(sys.argv)>1:
host = sys.argv[1]
number = sys.argv[2] if len(sys.argv)>2 else "1"
size = sys.argv[3] if len(sys.argv)>3 else "56"
timeout = sys.argv[4] if len(sys.argv)>4 else "1"
proc = subprocess.Popen(["ping \"" + host + "\" -c \"" + number + "\" -s \"" + size + "\" -W \"" + timeout + "\" 2>/dev/null | grep packets | sed 's/[^0-9,%]*//g'"], stdout=subprocess.PIPE, shell=True)
(out, err) = proc.communicate()
print "address=\"" + host + "\" data=" + out
I need to split out to list. How I can achieve this?
Everything I try causes error: cannot concatenate 'str' and ... objects
Like when I tried:
...
res=list(out)
print "address=\"" + host + "\" data=" + res
I got error:
TypeError: cannot concatenate 'str' and 'list' objects
To unite strings and lists you first must make a string out of the list:
res=list(out)
print "address=\"" + str(host) + "\" data=" + str(res)