The function creates a folder and saves a file into it. Then the folder is packed into a rar archive and sent to the user, and the newly created folder and archive are deleted from the server after.
code.py
new_file_name = self.generate_file_name(rfi, vendor, current_scoring_round)
path_to_temp_folder = os.path.dirname(BASE_DIR)
if not os.path.exists(f'{path_to_temp_folder}/temp_folder'):
pathlib.Path(f'{path_to_temp_folder}/temp_folder').mkdir(parents=True, exist_ok=True)
wb.save(f'{path_to_temp_folder}/temp_folder/{new_file_name}') #save xlsx file from openpyxl library
archive = self.generate_zip_name(rfi) # generate name for archive
to_rar = f'{path_to_temp_folder}/temp_folder'
patoolib.create_archive(archive, (to_rar,)) # patoolib - to .rar converter
to_download = f'{path_to_temp_folder}/{archive}'
if os.path.exists(to_download):
try:
with open(to_download, 'rb') as fh:
response = HttpResponse(fh.read(),
content_type="content_type='application/vnd.rar'")
response['Content-Disposition'] = 'attachment; filename= "{}"'.format(archive)
return response
finally:
shutil.rmtree(to_rar, ignore_errors=True)
default_storage.delete(to_download)
Everything work, but the problem is that the downloaded archive contains subfolders - paths to the saved file.
Expected result:
folder.rar
file.xlsx
Actual result:
folder.rar
/home
/y700
/projects
file.xlsx
The documentation to patool is minimal. It certainly seems to suggest that this should be possible by passing the path to the file in the create-archive command. I've tried this though, and it appears not.
So the only option, probably, is to change the working directory to the location of the test.xlsx file:
import patoolib
import os
new_file_name = self.generate_file_name(rfi, vendor, current_scoring_round)
path_to_temp_folder = os.path.dirname(BASE_DIR)
if not os.path.exists(f'{path_to_temp_folder}/temp_folder'):
pathlib.Path(f'{path_to_temp_folder}/temp_folder').mkdir(parents=True, exist_ok=True)
wb.save(f'{path_to_temp_folder}/temp_folder/{new_file_name}') #save xlsx file from openpyxl library
archive = self.generate_zip_name(rfi) # generate name for archive
to_rar = f'{path_to_temp_folder}/temp_folder'
cwd=os.getcwd()
os.chdir('to_rar')
patoolib.create_archive(cwd+archive, ({new_file_name},)) # patoolib - to .rar converter
os.chdir('cwd')
to_download = f'{path_to_temp_folder}/{archive}'
if os.path.exists(to_download):
try:
with open(to_download, 'rb') as fh:
response = HttpResponse(fh.read(),
content_type="content_type='application/vnd.rar'")
response['Content-Disposition'] = 'attachment; filename= "{}"'.format(archive)
return response
finally:
shutil.rmtree(to_rar, ignore_errors=True)
default_storage.delete(to_download)
This works on my system, for example, and I get a single file in the archive (using tar, because I don't have rar installed):
import patoolib
import os
cwd=os.getcwd()
os.chdir('foo/bar/baz/qux/')
patoolib.create_archive(cwd+'/foo.tar.gz',('test.txt',))
os.chdir(cwd)
Note that you should really use os.path.join rather than concatenating strings, but this was just a quick & dirty test.
I have about 200k files with .tif extension in my root folder of google drive that i need to delete.
The python code i wrote only transfers / deletes the few files that we can see at an instance (we need to scroll down in the drive and let them 'load' to see more of them)
I am willing to delete all other files as well if there is a shortcut to do so.
Cntl + A does not work either, it just selects a same few files that we can see in an instance.
import shutil
import os
source = '/content/gdrive/My Drive'
dest1 = '/content/gdrive/My Drive/toDelete'
files = os.listdir(source)
for f in files:
if (f.endswith(".tif")):
shutil.move(f, dest1)
dir_name = "/content/gdrive/My Drive"
test = os.listdir(dir_name)
for item in test:
if item.endswith(".tif"):
os.remove(os.path.join(dir_name, item))
First you need to search for all the files that contain in the name and are in your root directory once you have those you can start deleting them.
I recommend you test this without the delete first to make sure its listing the files your after I am not responsible for this deleting stuff :)
page_token = None
while True:
response = drive_service.files().list(q="name contains '.tif' and 'root' in parents",
spaces='drive',
fields='nextPageToken, files(id, name)',
pageToken=page_token).execute()
for file in response.get('files', []):
# Process change
print 'Found file: %s (%s)' % (file.get('name'), file.get('id'))
#drive_service.files().delete(fileId=file.get('id')).execute()
page_token = response.get('nextPageToken', None)
if page_token is None:
break
Use glob.
Use pathlib for path manipulation.
import pathlib
import shutil
source = pathlib.Path('/content/gdrive/My Drive')
dest1 = pathlib.Path('/content/gdrive/My Drive/toDelete')
dest1.mkdir(exist_ok=True)
for f in source.glob("*.tif"):
shutil.move(f, dest1.joinpath(f.name))
I am new to Python scripting. I need to copy few folders from my local machine (windows) to Linux server. As of now, I am copying the folders by opening WinSCP console. I need to automate this process. I have written a below code in Python using Paramiko module library.
import paramiko
import os
transport = paramiko.Transport(('10.10.10.10', 22))
transport.connect(username='weblogic', password='weblogic')
sftp = paramiko.SFTPClient.from_transport(transport)
filepath = '/apps/logs'
localpath = 'C:\\Users\\Public\\test'
sftp.put(localpath,filepath)
Above is not working properly and giving below error. Can you please help me to copy the folder present in the windows path C:\Users\Public\test to Linux server path /apps/logs?
Traceback (most recent call last):
File "C:\Users\Desktop\python\execute_script.py", line 28, in <module>
sftp.put(localpath,filepath)
File "C:\Python27\lib\paramiko\sftp_client.py", line 548, in put
fl = file(localpath, 'rb')
IOError: [Errno 13] Permission denied: 'C:\\Users\\Public\\test'
Please check the below code from the link https://gist.github.com/johnfink8/2190472. I have used put_all method in the snippet.
import paramiko
import socket
import os
from stat import S_ISDIR
class SSHSession(object):
# Usage:
# Detects DSA or RSA from key_file, either as a string filename or a
# file object. Password auth is possible, but I will judge you for
# using it. So:
# ssh=SSHSession('targetserver.com','root',key_file=open('mykey.pem','r'))
# ssh=SSHSession('targetserver.com','root',key_file='/home/me/mykey.pem')
# ssh=SSHSession('targetserver.com','root','mypassword')
# ssh.put('filename','/remote/file/destination/path')
# ssh.put_all('/path/to/local/source/dir','/path/to/remote/destination')
# ssh.get_all('/path/to/remote/source/dir','/path/to/local/destination')
# ssh.command('echo "Command to execute"')
def __init__(self,hostname,username='root',key_file=None,password=None):
#
# Accepts a file-like object (anything with a readlines() function)
# in either dss_key or rsa_key with a private key. Since I don't
# ever intend to leave a server open to a password auth.
#
self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.sock.connect((hostname,22))
self.t = paramiko.Transport(self.sock)
self.t.start_client()
keys = paramiko.util.load_host_keys(os.path.expanduser('~/.ssh/known_hosts'))
key = self.t.get_remote_server_key()
# supposed to check for key in keys, but I don't much care right now to find the right notation
if key_file is not None:
if isinstance(key,str):
key_file=open(key,'r')
key_head=key_file.readline()
key_file.seek(0)
if 'DSA' in key_head:
keytype=paramiko.DSSKey
elif 'RSA' in key_head:
keytype=paramiko.RSAKey
else:
raise Exception("Can't identify key type")
pkey=keytype.from_private_key(key_file)
self.t.auth_publickey(username, pkey)
else:
if password is not None:
self.t.auth_password(username,password,fallback=False)
else: raise Exception('Must supply either key_file or password')
self.sftp=paramiko.SFTPClient.from_transport(self.t)
def command(self,cmd):
# Breaks the command by lines, sends and receives
# each line and its output separately
#
# Returns the server response text as a string
chan = self.t.open_session()
chan.get_pty()
chan.invoke_shell()
chan.settimeout(20.0)
ret=''
try:
ret+=chan.recv(1024)
except:
chan.send('\n')
ret+=chan.recv(1024)
for line in cmd.split('\n'):
chan.send(line.strip() + '\n')
ret+=chan.recv(1024)
return ret
def put(self,localfile,remotefile):
# Copy localfile to remotefile, overwriting or creating as needed.
self.sftp.put(localfile,remotefile)
def put_all(self,localpath,remotepath):
# recursively upload a full directory
os.chdir(os.path.split(localpath)[0])
parent=os.path.split(localpath)[1]
for walker in os.walk(parent):
try:
self.sftp.mkdir(os.path.join(remotepath,walker[0]))
except:
pass
for file in walker[2]:
self.put(os.path.join(walker[0],file),os.path.join(remotepath,walker[0],file))
def get(self,remotefile,localfile):
# Copy remotefile to localfile, overwriting or creating as needed.
self.sftp.get(remotefile,localfile)
def sftp_walk(self,remotepath):
# Kindof a stripped down version of os.walk, implemented for
# sftp. Tried running it flat without the yields, but it really
# chokes on big directories.
path=remotepath
files=[]
folders=[]
for f in self.sftp.listdir_attr(remotepath):
if S_ISDIR(f.st_mode):
folders.append(f.filename)
else:
files.append(f.filename)
print (path,folders,files)
yield path,folders,files
for folder in folders:
new_path=os.path.join(remotepath,folder)
for x in self.sftp_walk(new_path):
yield x
def get_all(self,remotepath,localpath):
# recursively download a full directory
# Harder than it sounded at first, since paramiko won't walk
#
# For the record, something like this would gennerally be faster:
# ssh user#host 'tar -cz /source/folder' | tar -xz
self.sftp.chdir(os.path.split(remotepath)[0])
parent=os.path.split(remotepath)[1]
try:
os.mkdir(localpath)
except:
pass
for walker in self.sftp_walk(parent):
try:
os.mkdir(os.path.join(localpath,walker[0]))
except:
pass
for file in walker[2]:
self.get(os.path.join(walker[0],file),os.path.join(localpath,walker[0],file))
def write_command(self,text,remotefile):
# Writes text to remotefile, and makes remotefile executable.
# This is perhaps a bit niche, but I was thinking I needed it.
# For the record, I was incorrect.
self.sftp.open(remotefile,'w').write(text)
self.sftp.chmod(remotefile,755)
In addition to the answer #user1041177, but here a way to do it when you are on windows to linux host (not really sure which kind of host actually).
I don't know why, but if I keep backslash onto remote path, I get a FileNotFoundException. The only way to work was to replace all '\' by '/'
Maybe someone could tell me the proper way to avoid this situation at all ?
Here a part of the exact same code above to give you breadcrumbs if you encounter the same issue :
def sftp_walk(socket, remotepath):
remotepath = remotepath.replace('\\', '/')
path = remotepath
files = []
folders = []
for f in socket.listdir_attr(remotepath.replace('\\', '/')):
if S_ISDIR(f.st_mode):
folders.append(f.filename)
else:
files.append(f.filename)
print(path, folders, files)
yield path, folders, files
for folder in folders:
new_path = os.path.join(remotepath.replace('\\', '/'), folder)
for x in sftp_walk(socket, new_path):
yield x
def get_all(socket, remotepath, localpath):
remotepath = remotepath.replace('\\', '/')
socket.chdir(os.path.split(remotepath)[0])
parent = os.path.split(remotepath)[1]
try:
os.mkdir(localpath)
except:
pass
for walker in sftp_walk(socket, parent):
try:
os.mkdir(os.path.join(localpath, walker[0]).replace('\\', '/'))
except:
pass
for file in walker[2]:
socket.get(os.path.join(walker[0], file).replace('\\', '/'), os.path.join(localpath, walker[0], file).replace('\\', '/'))
BTW, I am not using those function inside an object, that's why their is 'socket' instead of 'self' because I call those function by passing the SFTP socket to them.
Finally have to say thank you to #user1041177, working like a charm.
I was trying to copy from a windows box to a linux box and got the same error as #Apex above. I was using the put_all method and I had to do some "replace" on parts of the code.
def put_all(self,localpath,remotepath):
remotepath = remotepath.replace('\\', '/')
# recursively upload a full directory
os.chdir(os.path.split(localpath)[0])
parent=os.path.split(localpath)[1]
for walker in os.walk(parent):
try:
self.sftp.mkdir(os.path.join(remotepath,walker[0]).replace('\\', '/'))
except:
pass
for file in walker[2]:
self.put(os.path.join(walker[0],file).replace('\\', '/'),os.path.join(remotepath,walker[0],file).replace('\\', '/'))
I found a few shortcomings with the above methods - first, the putter/getter doesn't function in the way you'd expect - if you want to put /foo/bar into /some/folder, you can't as it won't let you put files from a source folder to a different destination folder - the only thing you can do is put /foo/bar into /some/bar. In addition, you have to specify the source as /foo/bar and the destination as /some to end up with /some/bar - I find this confusing as it's not how most operating/ftp systems handle putting/getting/copying/etc. So, I improved on the answers listed above:
If you're going from Windows to Linux:
def put_dir(source, dest):
source = os.path.expandvars(source).rstrip('\\').rstrip('/')
dest = os.path.expandvars(dest).rstrip('\\').rstrip('/')
for root, dirs, files in os.walk(source):
for dir in dirs:
try:
sftp.mkdir(posixpath.join(dest, ''.join(root.rsplit(source))[1:].replace('\\', '/'), dir))
except:
pass
for file in files:
sftp.put(os.path.join(root, file), posixpath.join(dest, ''.join(root.rsplit(source))[1:].replace('\\', '/'), file))
source = '%USERPROFILE%\\Downloads\\'
dest = '/foo/bar'
put_dir(source, dest)
If you're just doing Windows then swap out posixpath.join with os.path.join and remove .replace('\\', '/'):
def put_dir(source, dest):
source = os.path.expandvars(source).rstrip('\\').rstrip('/')
dest = os.path.expandvars(dest).rstrip('\\').rstrip('/')
for root, dirs, files in os.walk(source):
for dir in dirs:
try:
sftp.mkdir(os.path.join(dest, ''.join(root.rsplit(source))[1:], dir))
except:
pass
for file in files:
sftp.put(os.path.join(root, file), os.path.join(dest, ''.join(root.rsplit(source))[1:], file))
source = '%USERPROFILE%\\Downloads\\'
dest = 'foo\\bar'
put_dir(source, dest)
The reason for the try statement is that sftp.mkdir errors out if the folder already exists.
Paramiko does not support recursive operations.
You can use pysftp. It's a wrapper around Paramiko that has more Python-ish look and feel and supports recursive operations. See
pysftp.Connection.put_r()
pysftp.Connection.get_r()
Or you can just base your code on pysftp source code. Or see my answer to Python pysftp get_r from Linux works fine on Linux but not on Windows.
I'm currently putting together a script in Python which will do the following:-
Create a directory in my Dropbox folder called 'Spartacus'
Create a subdirectory in this location with the naming convention of the date and time of creation
Within this directory, create a file called iprecord.txt and information will then be written to this file.
Here is my code thusfar using Python v2.7 on Windows 7:-
import os
import time
import platform
import urllib
current_dir = os.getcwd()
targetname = "Spartacus"
target_dir = os.path.join(current_dir, targetname)
timenow = time.strftime("\%d-%b-%Y %H-%M-%S")
def directoryVerification():
os.chdir(current_dir)
try:
os.mkdir('Spartacus')
except OSError:
pass
try:
os.system('attrib +h Spartacus')
except OSError:
pass
def gatherEvidence():
os.chdir(target_dir)
try:
evidential_dir = os.mkdir(target_dir + timenow)
os.chdir(evidential_dir)
except OSError:
pass
f = iprecord.txt
with f as open:
ip_addr = urllib.urlopen('http://www.biranchi.com/ip.php').read()
f.write("IP Address:\t %s\t %s" % ip_addr, time.strftime("\%d-%b-%Y %H-%M-%S"))
x = directoryVerification()
y = gatherEvidence()
I keep on getting an error in line 26 whereby it cannot resolve the full path to the dynamically named directory (date and time) one. I've printed out the value of 'evidential_dir' and it shows as being Null.
Any pointers as to where I am going wrong? Thanks
PS: Any other advice on my code to improve it would be appreciated
PPS: Any advice on how to locate the default directory for 'Dropbox'? Is there a way of scanning a file system for a directory called 'Dropbox' and capturing the path?
os.mkdir() does not return a pathname as you might be thinking. It seems like you do inconsistent methods of the same thing at different spots of your code.
Try this:
evidential_dir = os.path.join(target_dir, timenow)
os.mkdir(evidential_dir)
And fix your other line:
f = "iprecord.txt"
os.mkdir doesn't return anything.
evidential_dir = target_dir + timenow
try:
os.mkdir(evidential_dir)
except OSError:
pass
os.chdir(evidential_dir)
Have you ever tried this feedback calling an external zip.py script to work? My CGITB does not show any error messages. It simply did not invoke external .py script to work. It simply skipped over to gush. I should be grateful if you can assist me in making this zip.py callable in feedback.py.
Regards. David
#**********************************************************************
# Description:
# Zips the contents of a folder.
# Parameters:
# 0 - Input folder.
# 1 - Output zip file. It is assumed that the user added the .zip
# extension.
#**********************************************************************
# Import modules and create the geoprocessor
#
import sys, zipfile, arcgisscripting, os, traceback
gp = arcgisscripting.create()
# Function for zipping files. If keep is true, the folder, along with
# all its contents, will be written to the zip file. If false, only
# the contents of the input folder will be written to the zip file -
# the input folder name will not appear in the zip file.
#
def zipws(path, zip, keep):
path = os.path.normpath(path)
# os.walk visits every subdirectory, returning a 3-tuple
# of directory name, subdirectories in it, and filenames
# in it.
#
for (dirpath, dirnames, filenames) in os.walk(path):
# Iterate over every filename
#
for file in filenames:
# Ignore .lock files
#
if not file.endswith('.lock'):
gp.AddMessage("Adding %s..." % os.path.join(path, dirpath, file))
try:
if keep:
zip.write(os.path.join(dirpath, file),
os.path.join(os.path.basename(path),
os.path.join(dirpath, file)[len(path)+len(os.sep):]))
else:
zip.write(os.path.join(dirpath, file),
os.path.join(dirpath[len(path):], file))
except Exception, e:
gp.AddWarning(" Error adding %s: %s" % (file, e))
return None
if __name__ == '__main__':
try:
# Get the tool parameter values
#
infolder = gp.GetParameterAsText(0)
outfile = gp.GetParameterAsText(1)
# Create the zip file for writing compressed data. In some rare
# instances, the ZIP_DEFLATED constant may be unavailable and
# the ZIP_STORED constant is used instead. When ZIP_STORED is
# used, the zip file does not contain compressed data, resulting
# in large zip files.
#
try:
zip = zipfile.ZipFile(outfile, 'w', zipfile.ZIP_DEFLATED)
zipws(infolder, zip, True)
zip.close()
except RuntimeError:
# Delete zip file if exists
#
if os.path.exists(outfile):
os.unlink(outfile)
zip = zipfile.ZipFile(outfile, 'w', zipfile.ZIP_STORED)
zipws(infolder, zip, True)
zip.close()
gp.AddWarning(" Unable to compress zip file contents.")
gp.AddMessage("Zip file created successfully")
except:
# Return any python specific errors as well as any errors from the geoprocessor
#
tb = sys.exc_info()[2]
tbinfo = traceback.format_tb(tb)[0]
pymsg = "PYTHON ERRORS:\nTraceback Info:\n" + tbinfo +
"\nError Info:\n " + str(sys.exc_type) +
": " + str(sys.exc_value) + "\n"
gp.AddError(pymsg)
msgs = "GP ERRORS:\n" + gp.GetMessages(2) + "\n"
gp.AddError(msgs)
zip() is a built-in function in Python. Therefore it is a bad practice to use zip as a variable name. zip_ can be used instead of.
execfile() function reads and executes a Python script.
It is probably that you actually need just import zip_ in feedback.py instead of execfile().
Yay ArcGIS.
Just to clarify how are you trying to call this script using popen, can you post some code?
If your invoking this script via another script in the ArcGIS environment, then the thing is, when you use Popen the script wont be invoked within the ArcGIS environment, instead it will be invoked within windows. So you will loose all real control over it.
Also just another ArcGIS comment you never initalize a license for the geoprocessor.
My suggestion refactor your code, into a module function that simply attempts to zip the files, if it fails print the message out to ArcGIS.
If you want post how you are calling it, and how this is being run.