I'm successfully able to download CSV files from a folder on Exavault, using the files provided by Exavault, but the download gets saved in a temporary folder on my Mac.
How do I define the directory to save it to? This is an excerpt of the script.
resources_api = ResourcesApi()
resources_api.api_client.configuration.host = ACCOUNT_URL
try:
list_result = resources_api.list_resources(
API_KEY, ACCESS_TOKEN, "/files", offset=0, type='file', name='*.csv')
if list_result.returned_results == 0:
print("Found no files to download")
sys.exit(0)
else:
print("Found {} CSV files to download".format(list_result.returned_results))
except Exception as e:
raise e
print('Exception when calling Api:', str(e))
sys.exit(1)
downloads = []
listed_files = list_result.data
for listed_file in listed_files:
downloads.append("id:{}".format(listed_file.id))
print(listed_file.attributes.path)
try:
downloaded_file = resources_api.download(API_KEY, ACCESS_TOKEN, downloads, download_archive_name="sample-csvs")
print("File(s) downloaded to", os.path(downloaded_file))
Separately there's a resources_api.py file, which might hold the answer, but if I edit that, I'm not sure how I would invoke the changes.
Any help would be appreciated.
When looking at the code of the API, you can see that it is written such that it always creates a temporary folder for you:
# Write the file to the temporary folder configured for our client
fd, path = tempfile.mkstemp(dir=self.api_client.configuration.temp_folder_path)
You could check if you can try to change api_client.configuration.temp_folder_path and see if that works for you.
Or even better, just copy the file to a location of your choice. You can do it with shutil
import shutil
import os
targetFolder= "/your/folder/"
filename = os.path.basename(downloaded_file)
destination = os.path.join(targetFolder,filename )
shutil.copy(downloaded_file, destination)
I am aiming to create a function that does the following:
Declare a path with a file, not just a folder. e.g. 'C:/Users/Lampard/Desktop/Folder1/File.py'
Create a folder in same folder as the declared file path - Calling it 'Archive'
Cut the file and paste it into the new folder just created.
If the folder 'Archive' already exists - then simply cut and paste the file into there
I have spent approx. 15-20min going through these:
https://www.programiz.com/python-programming/directory
Join all except last x in list
https://docs.python.org/3/library/pathlib.html#operators
And here is what I got to:
import os
from pathlib import Path, PurePath
from shutil import copy
#This path will change every time - just trying to get function right first
path = 'C:/Users/Lampard/Desktop/Folder1/File.py'
#Used to allow suffix function
p = PurePath(path)
#Check if directory is a file not a folder
if not p.suffix:
print("Not an extension")
#If it is a file
else:
#Create new folder before last file
#Change working directory
split = path.split('/')
new_directory = '/'.join(split[:-1])
apply_new_directory = os.chdir(new_directory)
#If folder does not exist create it
try:
os.mkdir('Archive')#Create new folder
#If not, continue process to copy file and paste it into Archive
except FileExistsError:
copy(path, new_directory + '/Archive/' + split[-1])
Is this code okay? - does anyone know a simpler method?
Locate folder/file in path
print [name for name in os.listdir(".") if os.path.isdir(name)]
Create path
import os
# define the name of the directory to be created
path = "/tmp/year"
try:
os.mkdir(path)
except OSError:
print ("Creation of the directory %s failed" % path)
else:
print ("Successfully created the directory %s " % path)
To move and cut files you can use this library
As you're already using pathlib, there's no need to use shutil:
from pathlib import Path
path = 'C:/Users/Lampard/Desktop/Folder1/File.py' # or whatever
p = Path(path)
target = Path(p.with_name('Archive')) # replace the filename with 'Archive'
target.mkdir() # create target directory
p.rename(target.joinpath(p.name)) # move the file to the target directory
Feel free to add appriopriate try…except statements to handle any errors.
Update: you might find this version more readable:
target = p.parent / 'Archive'
target.mkdir()
p.rename(target / p.name)
This is an example of overloading / operator.
I am new to Python scripting. I need to copy few folders from my local machine (windows) to Linux server. As of now, I am copying the folders by opening WinSCP console. I need to automate this process. I have written a below code in Python using Paramiko module library.
import paramiko
import os
transport = paramiko.Transport(('10.10.10.10', 22))
transport.connect(username='weblogic', password='weblogic')
sftp = paramiko.SFTPClient.from_transport(transport)
filepath = '/apps/logs'
localpath = 'C:\\Users\\Public\\test'
sftp.put(localpath,filepath)
Above is not working properly and giving below error. Can you please help me to copy the folder present in the windows path C:\Users\Public\test to Linux server path /apps/logs?
Traceback (most recent call last):
File "C:\Users\Desktop\python\execute_script.py", line 28, in <module>
sftp.put(localpath,filepath)
File "C:\Python27\lib\paramiko\sftp_client.py", line 548, in put
fl = file(localpath, 'rb')
IOError: [Errno 13] Permission denied: 'C:\\Users\\Public\\test'
Please check the below code from the link https://gist.github.com/johnfink8/2190472. I have used put_all method in the snippet.
import paramiko
import socket
import os
from stat import S_ISDIR
class SSHSession(object):
# Usage:
# Detects DSA or RSA from key_file, either as a string filename or a
# file object. Password auth is possible, but I will judge you for
# using it. So:
# ssh=SSHSession('targetserver.com','root',key_file=open('mykey.pem','r'))
# ssh=SSHSession('targetserver.com','root',key_file='/home/me/mykey.pem')
# ssh=SSHSession('targetserver.com','root','mypassword')
# ssh.put('filename','/remote/file/destination/path')
# ssh.put_all('/path/to/local/source/dir','/path/to/remote/destination')
# ssh.get_all('/path/to/remote/source/dir','/path/to/local/destination')
# ssh.command('echo "Command to execute"')
def __init__(self,hostname,username='root',key_file=None,password=None):
#
# Accepts a file-like object (anything with a readlines() function)
# in either dss_key or rsa_key with a private key. Since I don't
# ever intend to leave a server open to a password auth.
#
self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.sock.connect((hostname,22))
self.t = paramiko.Transport(self.sock)
self.t.start_client()
keys = paramiko.util.load_host_keys(os.path.expanduser('~/.ssh/known_hosts'))
key = self.t.get_remote_server_key()
# supposed to check for key in keys, but I don't much care right now to find the right notation
if key_file is not None:
if isinstance(key,str):
key_file=open(key,'r')
key_head=key_file.readline()
key_file.seek(0)
if 'DSA' in key_head:
keytype=paramiko.DSSKey
elif 'RSA' in key_head:
keytype=paramiko.RSAKey
else:
raise Exception("Can't identify key type")
pkey=keytype.from_private_key(key_file)
self.t.auth_publickey(username, pkey)
else:
if password is not None:
self.t.auth_password(username,password,fallback=False)
else: raise Exception('Must supply either key_file or password')
self.sftp=paramiko.SFTPClient.from_transport(self.t)
def command(self,cmd):
# Breaks the command by lines, sends and receives
# each line and its output separately
#
# Returns the server response text as a string
chan = self.t.open_session()
chan.get_pty()
chan.invoke_shell()
chan.settimeout(20.0)
ret=''
try:
ret+=chan.recv(1024)
except:
chan.send('\n')
ret+=chan.recv(1024)
for line in cmd.split('\n'):
chan.send(line.strip() + '\n')
ret+=chan.recv(1024)
return ret
def put(self,localfile,remotefile):
# Copy localfile to remotefile, overwriting or creating as needed.
self.sftp.put(localfile,remotefile)
def put_all(self,localpath,remotepath):
# recursively upload a full directory
os.chdir(os.path.split(localpath)[0])
parent=os.path.split(localpath)[1]
for walker in os.walk(parent):
try:
self.sftp.mkdir(os.path.join(remotepath,walker[0]))
except:
pass
for file in walker[2]:
self.put(os.path.join(walker[0],file),os.path.join(remotepath,walker[0],file))
def get(self,remotefile,localfile):
# Copy remotefile to localfile, overwriting or creating as needed.
self.sftp.get(remotefile,localfile)
def sftp_walk(self,remotepath):
# Kindof a stripped down version of os.walk, implemented for
# sftp. Tried running it flat without the yields, but it really
# chokes on big directories.
path=remotepath
files=[]
folders=[]
for f in self.sftp.listdir_attr(remotepath):
if S_ISDIR(f.st_mode):
folders.append(f.filename)
else:
files.append(f.filename)
print (path,folders,files)
yield path,folders,files
for folder in folders:
new_path=os.path.join(remotepath,folder)
for x in self.sftp_walk(new_path):
yield x
def get_all(self,remotepath,localpath):
# recursively download a full directory
# Harder than it sounded at first, since paramiko won't walk
#
# For the record, something like this would gennerally be faster:
# ssh user#host 'tar -cz /source/folder' | tar -xz
self.sftp.chdir(os.path.split(remotepath)[0])
parent=os.path.split(remotepath)[1]
try:
os.mkdir(localpath)
except:
pass
for walker in self.sftp_walk(parent):
try:
os.mkdir(os.path.join(localpath,walker[0]))
except:
pass
for file in walker[2]:
self.get(os.path.join(walker[0],file),os.path.join(localpath,walker[0],file))
def write_command(self,text,remotefile):
# Writes text to remotefile, and makes remotefile executable.
# This is perhaps a bit niche, but I was thinking I needed it.
# For the record, I was incorrect.
self.sftp.open(remotefile,'w').write(text)
self.sftp.chmod(remotefile,755)
In addition to the answer #user1041177, but here a way to do it when you are on windows to linux host (not really sure which kind of host actually).
I don't know why, but if I keep backslash onto remote path, I get a FileNotFoundException. The only way to work was to replace all '\' by '/'
Maybe someone could tell me the proper way to avoid this situation at all ?
Here a part of the exact same code above to give you breadcrumbs if you encounter the same issue :
def sftp_walk(socket, remotepath):
remotepath = remotepath.replace('\\', '/')
path = remotepath
files = []
folders = []
for f in socket.listdir_attr(remotepath.replace('\\', '/')):
if S_ISDIR(f.st_mode):
folders.append(f.filename)
else:
files.append(f.filename)
print(path, folders, files)
yield path, folders, files
for folder in folders:
new_path = os.path.join(remotepath.replace('\\', '/'), folder)
for x in sftp_walk(socket, new_path):
yield x
def get_all(socket, remotepath, localpath):
remotepath = remotepath.replace('\\', '/')
socket.chdir(os.path.split(remotepath)[0])
parent = os.path.split(remotepath)[1]
try:
os.mkdir(localpath)
except:
pass
for walker in sftp_walk(socket, parent):
try:
os.mkdir(os.path.join(localpath, walker[0]).replace('\\', '/'))
except:
pass
for file in walker[2]:
socket.get(os.path.join(walker[0], file).replace('\\', '/'), os.path.join(localpath, walker[0], file).replace('\\', '/'))
BTW, I am not using those function inside an object, that's why their is 'socket' instead of 'self' because I call those function by passing the SFTP socket to them.
Finally have to say thank you to #user1041177, working like a charm.
I was trying to copy from a windows box to a linux box and got the same error as #Apex above. I was using the put_all method and I had to do some "replace" on parts of the code.
def put_all(self,localpath,remotepath):
remotepath = remotepath.replace('\\', '/')
# recursively upload a full directory
os.chdir(os.path.split(localpath)[0])
parent=os.path.split(localpath)[1]
for walker in os.walk(parent):
try:
self.sftp.mkdir(os.path.join(remotepath,walker[0]).replace('\\', '/'))
except:
pass
for file in walker[2]:
self.put(os.path.join(walker[0],file).replace('\\', '/'),os.path.join(remotepath,walker[0],file).replace('\\', '/'))
I found a few shortcomings with the above methods - first, the putter/getter doesn't function in the way you'd expect - if you want to put /foo/bar into /some/folder, you can't as it won't let you put files from a source folder to a different destination folder - the only thing you can do is put /foo/bar into /some/bar. In addition, you have to specify the source as /foo/bar and the destination as /some to end up with /some/bar - I find this confusing as it's not how most operating/ftp systems handle putting/getting/copying/etc. So, I improved on the answers listed above:
If you're going from Windows to Linux:
def put_dir(source, dest):
source = os.path.expandvars(source).rstrip('\\').rstrip('/')
dest = os.path.expandvars(dest).rstrip('\\').rstrip('/')
for root, dirs, files in os.walk(source):
for dir in dirs:
try:
sftp.mkdir(posixpath.join(dest, ''.join(root.rsplit(source))[1:].replace('\\', '/'), dir))
except:
pass
for file in files:
sftp.put(os.path.join(root, file), posixpath.join(dest, ''.join(root.rsplit(source))[1:].replace('\\', '/'), file))
source = '%USERPROFILE%\\Downloads\\'
dest = '/foo/bar'
put_dir(source, dest)
If you're just doing Windows then swap out posixpath.join with os.path.join and remove .replace('\\', '/'):
def put_dir(source, dest):
source = os.path.expandvars(source).rstrip('\\').rstrip('/')
dest = os.path.expandvars(dest).rstrip('\\').rstrip('/')
for root, dirs, files in os.walk(source):
for dir in dirs:
try:
sftp.mkdir(os.path.join(dest, ''.join(root.rsplit(source))[1:], dir))
except:
pass
for file in files:
sftp.put(os.path.join(root, file), os.path.join(dest, ''.join(root.rsplit(source))[1:], file))
source = '%USERPROFILE%\\Downloads\\'
dest = 'foo\\bar'
put_dir(source, dest)
The reason for the try statement is that sftp.mkdir errors out if the folder already exists.
Paramiko does not support recursive operations.
You can use pysftp. It's a wrapper around Paramiko that has more Python-ish look and feel and supports recursive operations. See
pysftp.Connection.put_r()
pysftp.Connection.get_r()
Or you can just base your code on pysftp source code. Or see my answer to Python pysftp get_r from Linux works fine on Linux but not on Windows.
Is there any way to make it so that IPython searches through your system path to find a file if only the name of the file is given?
I have a file foo_060112.dat that lives in a folder containing a lot of data files that are uniquely named in a folder in my path. I want to be able to simply call a load or open function on this file without specifying the full path---is it possible?
You can write your own function to search through the path and return an open file object according to the mode you select:
import sys
import os
def my_open(filename, mode):
for path in sys.path:
try:
return open(os.path.join(path, filename), mode)
except IOError:
continue
return None
Example: my_open('foo_060112.dat', 'rb')
I thinks this may help you:
import os,sys
path = os.sytem('locate foo_060112.dat')
print path
or you can also use :
_SOURCE_FILE_PATH = '/home/admin/'
_each_folder = os.walk("%s/" %(_SOURCE_FILE_PATH))
for x in _each_folder:
print x
if x[1] == []:
for y in x[2]:
if y == 'locate foo_060112.dat'
f = open(y,'r')
data = f.read()