Wrote a function that tries to reconnect to SSH when a disconnect happens. Basically expanded my existing function that simply saved the images, which works fine. The code runs but does not work to re-establish connectivity. Any help would be appreciated.
def get_image_id_and_upload_folder_of_images(db_name, table_name, selector_list,
condition_label, condition_val, img_address):
"""
get image id using project name from db
:param db_name: str - name of the data-base (usually 'server')
:param table_name: str - name of the table (usually 'images')
:param selector_list: list of str - list of selectors for the query (usually ["id", "path"])
:param condition_label: str - condition for the sql statement (like 'path')
:param condition_val: list of str - value for the condition of the condition_label (like ['name_of_file.png'])
:param img_address: str - address of the images to send them to the ftp server
:return: returns image or project id
"""
client = paramiko.client.SSHClient()
client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
retry_interval = 1
retry_interval = float(retry_interval)
timeout = int(20)
timeout_start = time.time()
while time.time() < timeout_start + timeout:
time.sleep(retry_interval)
try:
db, cur = db_info(db_name)
cond_list = ["'" + str(x) + "'" for x in condition_val]
condition = ",".join(cond_list)
selector = ",".join(selector_list)
# make a query for sql
query_stmt = "SELECT " + selector + " FROM " + table_name + " WHERE `" + \
condition_label + "` IN (" + str(condition) + ");"
image_ids = get_image_ids(cur, db, query_stmt, condition_val)
for idx, ids in enumerate(image_ids):
print(ids)
save_img_new_server(img_address + '/' + condition_val[idx], str(ids))
save_img_new_server(img_address + '/' + condition_val[idx], str(ids), hst="site.com",
folder_path='images/')
except paramiko.ssh_exception.NoValidConnectionsError as e:
print('SSH transport is not ready...')
continue
# print(img_address + '/' + condition_val[idx], str(ids))
return image_ids
Your code never calls client.connect(). In fact it doesn't interact with any paramiko module at all inside the while loop.
Related
So, in my server side, I created a dict that takes records from a .txt file:
records = {}
with open("data.txt") as f:
for line in f:
(name1, age, location, number) = line.split("|")
records[str(name1)] = [age, location, number]
As of now, I am able to print this DB on the server side :
print("** Python DB contents **\n")
for keys in records.keys():
value_list = records[keys]
print(keys + "|" + value_list[0] + "|" + value_list[1] + "|" + value_list[2])
However, I want to send this to the client side, but nothing is working for me.
I have tried sending it to the client side by creating a list that contains the contents, then sending it to the client like so:
Server:
for keys in records.keys():
value_list = records[keys]
string = keys + "|" + value_list[0] + "|" + value_list[1] + "|" + value_list[2]
client.send(string.encode())
Client:
buffer = server.recv(1024).decode() # receive from server
print(f"Server: {buffer}") # print in server what was sent
Anyone know what exactly is wrong here? This is how I've been sending and receiving stuff my whole code but in this case it just does not work!
I have subscribed to multiple topics using paho mqtt client. On receiving the messages from broker, I want to store the messages into mysql database. I want to gather the messages altogether before inserting into DB.I have set the threshold say 1000 messages. Only when the threshold is reached the messages has to be inserted into DB altogether at once. I am checking the row_count after cursor.execute(). But it shows the count as 1. So the bulk insert is not happening. here is my sample code snippet
//main.py
#mysql database class
db = MySQLDBClass()
#mqtt client class where subscription,connection to broker,some callbacks
mqttclient = MyMQTTClient()
mqttclient.on_message = db.onMessage
mqttclient.loop_forever()
//MySQLDBClass.py
def __init__(self):
self.insertcounter = 0
self.insertStatement = ''
self.bulkpayload = ''
self.maxInsert = 1000
def onMessage(self, client, userdata, msg):
if msg.topic.startswith("topic1/"):
self.bulkpayload += "(" + msg.payload.decode("utf-8") + "," + datetime + "),"
elif msg.topic.startswith("topic2/"):
self.insertStatement += "INSERT INTO mydatabase.table1 VALUES (" + msg.payload.decode("utf-8") + "," + datetime + ");"
elif msg.topic.startswith("topic3/")
self.insertStatement += "INSERT INTO mydatabase.table2 VALUES (" +msg.payload.decode("utf-8") + "," + datetime + ");"
elif msg.topic.startswith("messages"):
self.insertStatement += "INSERT INTO mydatabase.table3 VALUES ('" + msg.topic + "'," + msg.payload.decode("utf-8") + "," + datetime + ");"
else:
return # do not store in DB
self.insertcounter += 1
if ( self.insertcounter > self.maxInsert ):
if ( self.bulkpayload != '' ):
self.insertStatement += "INSERT INTO mydatabase.table4 VALUES" + self.bulkpayload + ";"
self.bulkpayload = ''
cursor.execute(self.insertStatement)
cursor.commit()
print (cursor.rowcount) #prints always count as one , expecting bulk count
self.insertcounter = 0
self.insertStatement = ''
with pymysql module, execute can only execute only one query at a time whereas by using mysql-connector-python, we can set multi=True in execute(mutli=true) to execute multiple statements.
https://dev.mysql.com/doc/connector-python/en/connector-python-api-mysqlcursor-execute.html
This question already has answers here:
Flask view return error "View function did not return a response"
(3 answers)
Closed 2 years ago.
My application when it runs the url http://localhost:81/ip_update/00178r it of the error ValueError: View function did not return a response.
I am unable to solve, if anyone can help me I will be very grateful
def ip_update (client):
ip = request.remote_addr
if 'r' in client:
customer = cliente.rstrip ('r')
urllib2.urlopen ('http://localhost:85/administration/ip_update/' + client + '/' + ip)
time.sleep (3)
else:
server = 'localhost'
port = 3306
login = root
password = 'password'
mdb.connect con = (host = server, port = port = user login, passwd = password, use_unicode = True)
cur = con.cursor ()
cur.execute ("SELECT FROM tar_cliente ebas.tarefa WHERE tar_status = 'processing' and tar_cliente = '" + client $
if cur.rowcount == 0:
Data2 data = ()
server = data2 ['clients'] [customer] ['server']
urllib2.urlopen ('http://' + server + '/administration/ip_update/' + client + '/' + ip)
time.sleep (3)
return 'request for ip received for updating client code' + client
else:
return 'denied updating, client operation underway'
Your function returns None when the if test is true:
if 'r' in client:
customer = cliente.rstrip ('r')
urllib2.urlopen ('http: // localhost: 85 / administration / ip_update /' + client + '/' + ip)
time.sleep (3)
You need to return a response in that case too. There is no point in using sleep here; all you are doing is delaying returning something to the browser.
I'm new at exporting data, I research all over the net but it was really hard for me to understand, can someone help me to know the basic about it.
This is my main problem: I want to download a specific data from mysql base on the date range I choose in my client, then when I click the download button, I want these data from mysql to be save in my computer together the user have the option to save it as CSV/Excel, I'm using python for my webservice. Thank you
This is my code right know in my webservice:
#api.route('/export_file/', methods=['GET', 'POST'])
def export_file():
if request.method == 'POST':
selectAttendance = """SELECT * FROM attendance"""
db.session.execute(selectAttendance)
db.session.commit()
f = csv.writer(open("file.csv", "w"))
for row in selectAttendance:
f.writerow([str(row)])
return jsonify({'success': True})
In general:
Set the mime header "Content-Type" part of the http header to the corresponding MIME-Type matching your data.
This tells the browser what type of data the webserver is going to send.
Send the actual data in the 'body'
With flask:
Forcing application/json MIME type in a view (Flask)
http://flask.pocoo.org/docs/0.10/patterns/streaming/
def get(self):
try:
os.stat(BACKUP_PATH)
except:
os.mkdir(BACKUP_PATH)
now = datetime.now() # current date and time
year = now.strftime("%Y")
month = now.strftime("%m")
day = now.strftime("%d")
time = now.strftime("%H:%M:%S")
date_time = now.strftime("%d_%m_%Y_%H:%M:%S")
TODAYBACKUPPATH = BACKUP_PATH + '/' + date_time
try:
os.stat(TODAYBACKUPPATH)
except:
os.mkdir(TODAYBACKUPPATH)
print ("checking for databases names file.")
if os.path.exists(DB_NAME):
file1 = open(DB_NAME)
multi = 1
print ("Databases file found...")
print ("Starting backup of all dbs listed in file " + DB_NAME)
else:
print ("Databases file not found...")
print ("Starting backup of database " + DB_NAME)
multi = 0
if multi:
in_file = open(DB_NAME,"r")
flength = len(in_file.readlines())
in_file.close()
p = 1
dbfile = open(DB_NAME,"r")
while p <= flength:
db = dbfile.readline() # reading database name from file
db = db[:-1] # deletes extra line
dumpcmd = "mysqldump -h " + DB_HOST + " -u " + DB_USER + " -p" + DB_USER_PASSWORD + " " + db + " > " + pipes.quote(TODAYBACKUPPATH) + "/" + db + ".sql"
os.system(dumpcmd)
gzipcmd = "gzip " + pipes.quote(TODAYBACKUPPATH) + "/" + db + ".sql"
os.system(gzipcmd)
p = p + 1
dbfile.close()
else:
db = DB_NAME
dumpcmd = "mysqldump -h " + DB_HOST + " -u " + DB_USER + " -p" + DB_USER_PASSWORD + " " + db + " > " + pipes.quote(TODAYBACKUPPATH) + "/" + db + ".sql"
os.system(dumpcmd)
gzipcmd = "gzip " + pipes.quote(TODAYBACKUPPATH) + "/" + db + ".sql"
os.system(gzipcmd)
# t = ("Your backups have been created in '" + TODAYBACKUPPATH + "' directory")
return "Your Folder have been created in '" + TODAYBACKUPPATH + "'."
I am trying to execute a wlst script to scaleup managed server i am getting below error.Since I am new to python not able to figure out whats the issue any pointers will surely help thanks
Problem invoking WLST - Traceback (innermost last):
File "/MiddlewareOSB/MyOSB/common/bin/extendOSBDomain.py", line 214, in ?
File "MiddlewareOSB/MyOSB/common/bin/extendOSBDomain.py", line 27, in main
File "MiddlewareOSB/MyOSB/common/bin/extendOSBDomain.py", line 110, in createJMSResources
TypeError: unsupported operand type(s) for +: 'NoneType' and 'str'
import sys
import os
def main():
try:
#load properties file
loadProperties('extend.properties')
print('connecting ' + adminUser + ' to Admin server')
connect(adminUser,adminPassword,'t3://'+adminHost+':'+adminPort)
exists = 1
index = 1
while exists == 1:
serverName = 'osb_server%s' % index
print "serverName is: " + serverName
try:
print('Check if server exists...')
cd('/Servers/'+serverName)
index += 1
except:
exists = 0
managedServerName = 'osb_server%s' % index
createManagedServer(managedServerName)
addNewMStoClusterList(managedServerName)
createJMSResources(managedServerName, index)
except:
traceback.format_exc()
exit()
def createManagedServer(managedServerName):
print ('*** Creating managed server ' + managedServerName + ' ...')
edit()
startEdit()
managedServer = create(managedServerName,'Server')
managedServer.setListenAddress(managedServerAddress)
managedServer.setListenPort(int(managedServerPort))
managedServer.setCluster(getMBean('/Clusters/' + clusterName))
# To add additional properties, such as SSL, web server information, Coherence specific startup parameters,
# server logging details or channels this would be the place to add it.
# For instance the following (once edited) adds SSL to the server
#
# server.setKeyStores('CustomIdentityAndCommandLineTrust')
# server.setCustomIdentityKeyStoreFileName(identityFileLocationAndPath)
# server.setCustomIdentityKeyStoreType('JKS')
# server.setCustomIdentityKeyStorePassPhrase(SSLPASSWORD)
#
# and this adds the coherence settings
#
# domainname=soaDomain+'_domain'
# domainlocation='/usr/appl/'+'%s'%tla+'/weblogic'
# cd('/Servers/' + managedServerName + '/ServerStart/' + managedServerName)
# cmo.setArguments('-DOSB.coherence.localhost=' + managedServerName + ' - DOSB.coherence.localport=7890 -DOSB.coherence.wka1=' + managedServerName + ' -DOSB.coherence.wka1.port=7890 -DOSB.coherence.wka2=' + managedServerName + ' -DOSB.coherence.wka1.port=7890 ')
activate()
print ('managed server created and added to the cluster')
def addNewMStoClusterList(managedServerName):
edit()
startEdit()
cd('/Clusters/' + clusterName)
activeCluster=cmo
try:
#increasing servers in list by 1
servers = activeCluster.getNumberOfServersInClusterAddress()
servers = servers + 1
#adding new server address and port
clusterList = activeCluster.getClusterAddress() + "," + managedServerAddress + ':' + managedServerPort
#if the cluster list not empty, add the new server to the list and increase number of servers in cluster list by 1
print ('*** Adding Server ' + managedServerName + ' to cluster address of cluster ' + clusterName + ' ...')
activeCluster.setNumberOfServersInClusterAddress(servers)
activeCluster.setClusterAddress(clusterList)
activate()
print ('cluster list updated')
except:
print('*** Cluster list failed to be updated - is the cluster address for the cluster empty?')
def createJMSResources(managedServerName, index):
edit()
startEdit()
targetServer = jarray.array([ObjectName('com.bea:Name='+ managedServerName +',Type=Server')], ObjectName)
# creating persistance stores for jms servers
# PLEASE NOTE - On some versions of OSB, you will need to create the physical folder for the filestore
print ('*** Creating FileStores')
filestore_list = ["FileStore", "WseeFileStore"]
count = 0
limit = 2
while count < limit:
filestore = filestore_list[count] + '_auto_%s' % index
print ('*** Adding ' + filestore )
cd('/')
cmo.createFileStore(filestore)
cd('/Deployments/' + filestore)
cmo.setDirectory(filestore)
set('Targets', targetServer)
print ('Filestore ' + filestore + ' created')
# create filestore directory, if it doesn't exist
dir = System.getenv('DOMAIN_HOME') + '/' + filestore
try:
os.stat(dir)
except:
os.mkdir(dir)
count += 1
print ('Filestores created')
# creating jms servers
print ('*** Creating JMS Servers')
# create wlsbJMSServer_auto_x jms server
cd('/')
cmo.createJMSServer('wlsbJMSServer_auto_%s' % index)
cd('/JMSServers/wlsbJMSServer_auto_%s' % index)
cmo.setPersistentStore(getMBean('/Deployments/FileStore_auto_%s' % index))
set('Targets', targetServer)
cd('/Deployments/wlsbJMSServer_auto_%s' % index)
cmo.setTemporaryTemplateResource('jmsResources')
cmo.setTemporaryTemplateName('TemporaryTmplt')
# create wseeJMSServer_auto_x jms server
cd('/')
cmo.createJMSServer('WseeJmsServer_auto_%s' % index)
cd('/JMSServers/WseeJmsServer_auto_%s' % index)
cmo.setPersistentStore(getMBean('/Deployments/WseeFileStore_auto_%s' % index))
set('Targets', targetServer)
targetWseeJmsServer = jarray.array([ObjectName('com.bea:Name=WseeJmsServer_auto_%s' % index + ',Type=JMSServer')], ObjectName)
targetWlsbJmsServer = jarray.array([ObjectName('com.bea:Name=wlsbJMSServer_auto_%s' % index + ',Type=JMSServer')], ObjectName)
print ('JMS Servers created')
# creating WseeJMSModule Components
print ('*** Create Queues: DefaultCallbackQueue-WseeJmsServer_auto_%s' % index + ' and DefaultQueue-WseeJmsServer_auto_%s' % index)
# create subdeployment within WseeJmsModule
cd('/JMSSystemResources/WseeJmsModule')
cmo.createSubDeployment('WseeJmsServer_auto_%s' % index + '-Sub')
# create DefaultQueue-WseeJmsServer_auto_x queue
cd('/JMSSystemResources/WseeJmsModule/JMSResource/WseeJmsModule')
cmo.createQueue('DefaultQueue-WseeJmsServer_auto_%s' % index)
cd('/JMSSystemResources/WseeJmsModule/JMSResource/WseeJmsModule/Queues/DefaultQueue-WseeJmsServer_auto_%s' % index)
cmo.setJNDIName('weblogic.wsee.DefaultQueue-WseeJmsServer_auto_%s' % index)
cmo.setSubDeploymentName('WseeJmsServer_auto_%s' % index + '-Sub')
cd('/JMSSystemResources/WseeJmsModule/SubDeployments/WseeJmsServer_auto_%s' % index + '-Sub')
set('Targets',targetWseeJmsServer)
# create DefaultCallbackQueue-WseeJmsServer_auto_x queue
cd('/JMSSystemResources/WseeJmsModule/JMSResource/WseeJmsModule')
cmo.createQueue('DefaultCallbackQueue-WseeJmsServer_auto_%s' % index)
cd('/JMSSystemResources/WseeJmsModule/JMSResource/WseeJmsModule/Queues/DefaultCallbackQueue-WseeJmsServer_auto_%s' % index)
cmo.setJNDIName('weblogic.wsee.DefaultCallbackQueue-WseeJmsServer_auto_%s' % index)
cmo.setSubDeploymentName('WseeJmsServer_auto_%s' % index + '-Sub')
cd('/JMSSystemResources/WseeJmsModule/SubDeployments/WseeJmsServer_auto_%s' % index + '-Sub')
set('Targets',targetWseeJmsServer)
print ('Queues within the WseeJMSModule created')
print ('*** Creating queues within the jmsResources module')
# create subdeployment within jmsResources
wlsJmsServerSubName = 'wlsbJMSServer%s' % index + '_sub'
cd('/JMSSystemResources/jmsResources')
cmo.createSubDeployment(wlsJmsServerSubName)
cd('SubDeployments/' + wlsJmsServerSubName)
set('Targets',targetWlsbJmsServer)
# create queues within jmsResources
queue_list = ["wli.reporting.jmsprovider.queue", "wli.reporting.jmsprovider_error.queue", "wlsb.internal.transport.task.queue.email", "wlsb.internal.transport.task.queue.file", "wlsb.internal.transport.task.queue.ftp", "wlsb.internal.transport.task.queue.sftp","QueueIn"]
count = 0
limit = 7
while count < limit:
queue = queue_list[count] + '_auto_%s' % index
dist_queue = 'dist_' + queue_list[count] + '_auto'
print ('*** Adding ' + queue + ' to ' + dist_queue)
cd('/JMSSystemResources/jmsResources/JMSResource/jmsResources/Queues')
cmo.createQueue(queue)
cd('/JMSSystemResources/jmsResources/JMSResource/jmsResources/Queues/' + queue)
cmo.setJNDIName(queue)
cmo.setSubDeploymentName(wlsJmsServerSubName)
cd('/JMSSystemResources/jmsResources/JMSResource/jmsResources/DistributedQueues/' + dist_queue)
cmo.createDistributedQueueMember(queue)
cd('/JMSSystemResources/jmsResources/JMSResource/jmsResources/DistributedQueues/' + dist_queue + '/DistributedQueueMembers/' + queue)
cmo.setWeight(1)
print ('Queue created and added to UDQ')
count += 1
# create SAF Agent
cd('/')
cmo.createSAFAgent('ReliableWseeSAFAgent_auto_%s' % index)
cd('/SAFAgents/ReliableWseeSAFAgent_auto_%s' % index)
cmo.setStore(getMBean('/FileStores/WseeFileStore_auto_%s' % index))
set('Targets',targetServer)
cmo.setServiceType('Both')
activate()
main()
I see you've cranked out a lot of code, however, it's littered with syntax errors. I'd recommend you try writing smaller chunks at a time, getting each step working as you plug along.
First, your while loop is infinte.
Second, if you want to use parenthesis with print (which I recommend), for Python 2.x, use this at the top of your files:
from future import print_function
Third, you should always use os.path for portable filesystem path manipulations.
What your error means is that, somewhere, you're trying to add a None type with a string. That's not possible, of course, so Python throws an exception. What you need to do, is to figure out why whatever you're adding is None, and fix that ;)
Hope that helps!
You may need to set the environment variable DOMAIN_HOME or to change System.getenv('DOMAIN_HOME') to System.getenv('DOMAIN_HOME', '.') or the like.