I'm successfully able to download CSV files from a folder on Exavault, using the files provided by Exavault, but the download gets saved in a temporary folder on my Mac.
How do I define the directory to save it to? This is an excerpt of the script.
resources_api = ResourcesApi()
resources_api.api_client.configuration.host = ACCOUNT_URL
try:
list_result = resources_api.list_resources(
API_KEY, ACCESS_TOKEN, "/files", offset=0, type='file', name='*.csv')
if list_result.returned_results == 0:
print("Found no files to download")
sys.exit(0)
else:
print("Found {} CSV files to download".format(list_result.returned_results))
except Exception as e:
raise e
print('Exception when calling Api:', str(e))
sys.exit(1)
downloads = []
listed_files = list_result.data
for listed_file in listed_files:
downloads.append("id:{}".format(listed_file.id))
print(listed_file.attributes.path)
try:
downloaded_file = resources_api.download(API_KEY, ACCESS_TOKEN, downloads, download_archive_name="sample-csvs")
print("File(s) downloaded to", os.path(downloaded_file))
Separately there's a resources_api.py file, which might hold the answer, but if I edit that, I'm not sure how I would invoke the changes.
Any help would be appreciated.
When looking at the code of the API, you can see that it is written such that it always creates a temporary folder for you:
# Write the file to the temporary folder configured for our client
fd, path = tempfile.mkstemp(dir=self.api_client.configuration.temp_folder_path)
You could check if you can try to change api_client.configuration.temp_folder_path and see if that works for you.
Or even better, just copy the file to a location of your choice. You can do it with shutil
import shutil
import os
targetFolder= "/your/folder/"
filename = os.path.basename(downloaded_file)
destination = os.path.join(targetFolder,filename )
shutil.copy(downloaded_file, destination)
I am getting this error from this code.
import os
import requests
import shutil
path = "/Users/mycode/Documents/API upload/"
api_endpoint = "xxxxxx"
files = {
'file': open(p,'rb') for p in os.path.abspath(path)
}
for file in os.path.abspath(path):
response = requests.post(url=api_endpoint, files=files)
if response.status_code == 200:
print(response.status_code)
print("success!")
else:
print("did not work")
IsADirectoryError: [Errno 21] Is a directory: '/'
^ what does this error mean? I tried googling it but still do not understand in my case. It has something to do with the paths but not sure why.
anything helps!
for p in os.path.abspath(path)
doesn't do what you think it does.
It does not iterate over all files in a given directory. Use os.listdir for that. You can combine the directory path and the filename inside the directory using os.path.join. The pathlib module has an IMHO simpler to use / higher level interface to all of this.
What your code does is iterate over all characters in the string returned by os.path.abspath(path). And the first character is /. Which you then try to open as a file. And that doesn't work, because / is a directory.
You might want to consider doing this in chunks because if your directory contents are very large, you could run out of file descriptors.
Something like this should work:
from requests import post
from glob import glob
from os.path import join, isfile
DIR = '/Users/mycode/Documents/API upload/'
CHUNK = 10
API_ENDPOINT = '...'
filelist = [filename for filename in glob(join(DIR, '*')) if isfile(filename)]
for idx in range(0, len(filelist), CHUNK):
files = [('file', open(fn, 'rb')) for fn in filelist[idx:idx+CHUNK]]
post(API_ENDPOINT, files=files).raise_for_status()
for _, fd in files:
fd.close()
Note:
For improved efficiency you should consider multithreading for this
HI i have a small python script which untars a list of files present in a folder.Below is the script.
app = Flask(__name__)
#app.route('/untarJson')
def untarJson():
outdir="C:\\Users\\esrilka\\Documents\\Tar Files\\Untar"
inputfilefolder="C:\\Users\\esrilka\\Documents\\Tar Files\\New tar files\\"
jsonfiles=[]
for filenames in os.listdir(inputfilefolder):
if filenames.endswith(".tar.gz"):
head,tail= os.path.split(filenames)
basename=os.path.splitext(os.path.splitext(tail)[0])[0]
t = tarfile.open(os.path.join(inputfilefolder,filenames), 'r')
for member in t.getmembers():
if "autodiscovery/report.json" in member.name:
with open(os.path.join(outdir,basename + '.json' ), 'wb') as f:
f.write(t.extractfile('autodiscovery/report.json').read())
if __name__ == '__main__':
app.run(debug=True)
It works fine without flask and in the folder i have four tar files and all 4 files are untarred.
But when i use flask only one file is untarred and the only one file name is displayed.
how can i untar all files inside a folder and also return the name of the files(i.,. only short names and not with full path)
See if the below code works for you, I have changed only little bit to your original code and it works without any issues. All the available tar.gz files are untared and file names gets displayed after request completes,
from flask import Flask, jsonify
import tarfile
import os
app = Flask(__name__)
#app.route('/untarJson')
def untarJson():
outdir = "C:\\tests\\untared"
inputfilefolder = "C:\\tests"
jsonfiles = []
for filenames in os.listdir(inputfilefolder):
if filenames.endswith(".tar.gz"):
head, tail = os.path.split(filenames)
basename = os.path.splitext(os.path.splitext(tail)[0])[0]
t = tarfile.open(os.path.join(inputfilefolder, filenames), 'r')
for member in t.getmembers():
if "autodiscovery/report.json" in member.name:
with open(os.path.join(outdir, basename + '.json'), 'wb') as f:
f.write(t.extractfile('autodiscovery/report.json').read())
jsonfiles.append(os.path.join(outdir, basename + '.json'))
return jsonify(jsonfiles), 200
if __name__ == '__main__':
app.run(debug=True)
After request completed, something like below will be returned (output will be different in your case),
[
"C:\tests\untared\autodiscovery1.json",
"C:\tests\untared\autodiscovery2.json",
"C:\tests\untared\autodiscovery3.json"
]
I upload file to dropbox api, but it post on dropbox all directories from my computer since root folder. I mean you have folder of your project inside folder home, than user until you go to file sours folder. If I cut that structure library can't see that it is file, not string and give mistake message.
My code is:
def upload_file(project_id, filename, dropbox_token):
dbx = dropbox.Dropbox(dropbox_token)
file_path = os.path.abspath(filename)
with open(filename, "rb") as f:
dbx.files_upload(f.read(), file_path, mute=True)
link = dbx.files_get_temporary_link(path=file_path).link
return link
It works, but I need something like:
file_path = os.path.abspath(filename)
chunks = file_path.split("/")
name, dir = chunks[-1], chunks[-2]
which gives me mistake like:
dropbox.exceptions.ApiError: ApiError('433249b1617c031b29c3a7f4f3bf3847', GetTemporaryLinkError('path', LookupError('not_found', None)))
How could I make only parent folder and filename in the path?
For example if I have
/home/user/project/file.txt
I need
/project/file.txt
you have /home/user/project/file.txt and you need /project/file.txt
I would split according to os default separator (so it would work with windows paths as well), then reformat only the 2 last parts with the proper format (sep+path) and join that.
import os
#os.sep = "/" # if you want to test that on Windows
s = "/home/user/project/file.txt"
path_end = "".join(["{}{}".format(os.sep,x) for x in s.split(os.sep)[-2:]])
result:
/project/file.txt
I assume the following code should works:
def upload_file(project_id, filename, dropbox_token):
dbx = dropbox.Dropbox(dropbox_token)
abs_path = os.path.abspath(filename)
directory, file = os.path.split(abs_path)
_, directory = os.path.split(directory)
dropbox_path = os.path.join(directory, file)
with open(abs_path, "rb") as f:
dbx.files_upload(f.read(), dropbox_path, mute=True)
link = dbx.files_get_temporary_link(path=dropbox_path).link
return link
I am new to Python scripting. I need to copy few folders from my local machine (windows) to Linux server. As of now, I am copying the folders by opening WinSCP console. I need to automate this process. I have written a below code in Python using Paramiko module library.
import paramiko
import os
transport = paramiko.Transport(('10.10.10.10', 22))
transport.connect(username='weblogic', password='weblogic')
sftp = paramiko.SFTPClient.from_transport(transport)
filepath = '/apps/logs'
localpath = 'C:\\Users\\Public\\test'
sftp.put(localpath,filepath)
Above is not working properly and giving below error. Can you please help me to copy the folder present in the windows path C:\Users\Public\test to Linux server path /apps/logs?
Traceback (most recent call last):
File "C:\Users\Desktop\python\execute_script.py", line 28, in <module>
sftp.put(localpath,filepath)
File "C:\Python27\lib\paramiko\sftp_client.py", line 548, in put
fl = file(localpath, 'rb')
IOError: [Errno 13] Permission denied: 'C:\\Users\\Public\\test'
Please check the below code from the link https://gist.github.com/johnfink8/2190472. I have used put_all method in the snippet.
import paramiko
import socket
import os
from stat import S_ISDIR
class SSHSession(object):
# Usage:
# Detects DSA or RSA from key_file, either as a string filename or a
# file object. Password auth is possible, but I will judge you for
# using it. So:
# ssh=SSHSession('targetserver.com','root',key_file=open('mykey.pem','r'))
# ssh=SSHSession('targetserver.com','root',key_file='/home/me/mykey.pem')
# ssh=SSHSession('targetserver.com','root','mypassword')
# ssh.put('filename','/remote/file/destination/path')
# ssh.put_all('/path/to/local/source/dir','/path/to/remote/destination')
# ssh.get_all('/path/to/remote/source/dir','/path/to/local/destination')
# ssh.command('echo "Command to execute"')
def __init__(self,hostname,username='root',key_file=None,password=None):
#
# Accepts a file-like object (anything with a readlines() function)
# in either dss_key or rsa_key with a private key. Since I don't
# ever intend to leave a server open to a password auth.
#
self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.sock.connect((hostname,22))
self.t = paramiko.Transport(self.sock)
self.t.start_client()
keys = paramiko.util.load_host_keys(os.path.expanduser('~/.ssh/known_hosts'))
key = self.t.get_remote_server_key()
# supposed to check for key in keys, but I don't much care right now to find the right notation
if key_file is not None:
if isinstance(key,str):
key_file=open(key,'r')
key_head=key_file.readline()
key_file.seek(0)
if 'DSA' in key_head:
keytype=paramiko.DSSKey
elif 'RSA' in key_head:
keytype=paramiko.RSAKey
else:
raise Exception("Can't identify key type")
pkey=keytype.from_private_key(key_file)
self.t.auth_publickey(username, pkey)
else:
if password is not None:
self.t.auth_password(username,password,fallback=False)
else: raise Exception('Must supply either key_file or password')
self.sftp=paramiko.SFTPClient.from_transport(self.t)
def command(self,cmd):
# Breaks the command by lines, sends and receives
# each line and its output separately
#
# Returns the server response text as a string
chan = self.t.open_session()
chan.get_pty()
chan.invoke_shell()
chan.settimeout(20.0)
ret=''
try:
ret+=chan.recv(1024)
except:
chan.send('\n')
ret+=chan.recv(1024)
for line in cmd.split('\n'):
chan.send(line.strip() + '\n')
ret+=chan.recv(1024)
return ret
def put(self,localfile,remotefile):
# Copy localfile to remotefile, overwriting or creating as needed.
self.sftp.put(localfile,remotefile)
def put_all(self,localpath,remotepath):
# recursively upload a full directory
os.chdir(os.path.split(localpath)[0])
parent=os.path.split(localpath)[1]
for walker in os.walk(parent):
try:
self.sftp.mkdir(os.path.join(remotepath,walker[0]))
except:
pass
for file in walker[2]:
self.put(os.path.join(walker[0],file),os.path.join(remotepath,walker[0],file))
def get(self,remotefile,localfile):
# Copy remotefile to localfile, overwriting or creating as needed.
self.sftp.get(remotefile,localfile)
def sftp_walk(self,remotepath):
# Kindof a stripped down version of os.walk, implemented for
# sftp. Tried running it flat without the yields, but it really
# chokes on big directories.
path=remotepath
files=[]
folders=[]
for f in self.sftp.listdir_attr(remotepath):
if S_ISDIR(f.st_mode):
folders.append(f.filename)
else:
files.append(f.filename)
print (path,folders,files)
yield path,folders,files
for folder in folders:
new_path=os.path.join(remotepath,folder)
for x in self.sftp_walk(new_path):
yield x
def get_all(self,remotepath,localpath):
# recursively download a full directory
# Harder than it sounded at first, since paramiko won't walk
#
# For the record, something like this would gennerally be faster:
# ssh user#host 'tar -cz /source/folder' | tar -xz
self.sftp.chdir(os.path.split(remotepath)[0])
parent=os.path.split(remotepath)[1]
try:
os.mkdir(localpath)
except:
pass
for walker in self.sftp_walk(parent):
try:
os.mkdir(os.path.join(localpath,walker[0]))
except:
pass
for file in walker[2]:
self.get(os.path.join(walker[0],file),os.path.join(localpath,walker[0],file))
def write_command(self,text,remotefile):
# Writes text to remotefile, and makes remotefile executable.
# This is perhaps a bit niche, but I was thinking I needed it.
# For the record, I was incorrect.
self.sftp.open(remotefile,'w').write(text)
self.sftp.chmod(remotefile,755)
In addition to the answer #user1041177, but here a way to do it when you are on windows to linux host (not really sure which kind of host actually).
I don't know why, but if I keep backslash onto remote path, I get a FileNotFoundException. The only way to work was to replace all '\' by '/'
Maybe someone could tell me the proper way to avoid this situation at all ?
Here a part of the exact same code above to give you breadcrumbs if you encounter the same issue :
def sftp_walk(socket, remotepath):
remotepath = remotepath.replace('\\', '/')
path = remotepath
files = []
folders = []
for f in socket.listdir_attr(remotepath.replace('\\', '/')):
if S_ISDIR(f.st_mode):
folders.append(f.filename)
else:
files.append(f.filename)
print(path, folders, files)
yield path, folders, files
for folder in folders:
new_path = os.path.join(remotepath.replace('\\', '/'), folder)
for x in sftp_walk(socket, new_path):
yield x
def get_all(socket, remotepath, localpath):
remotepath = remotepath.replace('\\', '/')
socket.chdir(os.path.split(remotepath)[0])
parent = os.path.split(remotepath)[1]
try:
os.mkdir(localpath)
except:
pass
for walker in sftp_walk(socket, parent):
try:
os.mkdir(os.path.join(localpath, walker[0]).replace('\\', '/'))
except:
pass
for file in walker[2]:
socket.get(os.path.join(walker[0], file).replace('\\', '/'), os.path.join(localpath, walker[0], file).replace('\\', '/'))
BTW, I am not using those function inside an object, that's why their is 'socket' instead of 'self' because I call those function by passing the SFTP socket to them.
Finally have to say thank you to #user1041177, working like a charm.
I was trying to copy from a windows box to a linux box and got the same error as #Apex above. I was using the put_all method and I had to do some "replace" on parts of the code.
def put_all(self,localpath,remotepath):
remotepath = remotepath.replace('\\', '/')
# recursively upload a full directory
os.chdir(os.path.split(localpath)[0])
parent=os.path.split(localpath)[1]
for walker in os.walk(parent):
try:
self.sftp.mkdir(os.path.join(remotepath,walker[0]).replace('\\', '/'))
except:
pass
for file in walker[2]:
self.put(os.path.join(walker[0],file).replace('\\', '/'),os.path.join(remotepath,walker[0],file).replace('\\', '/'))
I found a few shortcomings with the above methods - first, the putter/getter doesn't function in the way you'd expect - if you want to put /foo/bar into /some/folder, you can't as it won't let you put files from a source folder to a different destination folder - the only thing you can do is put /foo/bar into /some/bar. In addition, you have to specify the source as /foo/bar and the destination as /some to end up with /some/bar - I find this confusing as it's not how most operating/ftp systems handle putting/getting/copying/etc. So, I improved on the answers listed above:
If you're going from Windows to Linux:
def put_dir(source, dest):
source = os.path.expandvars(source).rstrip('\\').rstrip('/')
dest = os.path.expandvars(dest).rstrip('\\').rstrip('/')
for root, dirs, files in os.walk(source):
for dir in dirs:
try:
sftp.mkdir(posixpath.join(dest, ''.join(root.rsplit(source))[1:].replace('\\', '/'), dir))
except:
pass
for file in files:
sftp.put(os.path.join(root, file), posixpath.join(dest, ''.join(root.rsplit(source))[1:].replace('\\', '/'), file))
source = '%USERPROFILE%\\Downloads\\'
dest = '/foo/bar'
put_dir(source, dest)
If you're just doing Windows then swap out posixpath.join with os.path.join and remove .replace('\\', '/'):
def put_dir(source, dest):
source = os.path.expandvars(source).rstrip('\\').rstrip('/')
dest = os.path.expandvars(dest).rstrip('\\').rstrip('/')
for root, dirs, files in os.walk(source):
for dir in dirs:
try:
sftp.mkdir(os.path.join(dest, ''.join(root.rsplit(source))[1:], dir))
except:
pass
for file in files:
sftp.put(os.path.join(root, file), os.path.join(dest, ''.join(root.rsplit(source))[1:], file))
source = '%USERPROFILE%\\Downloads\\'
dest = 'foo\\bar'
put_dir(source, dest)
The reason for the try statement is that sftp.mkdir errors out if the folder already exists.
Paramiko does not support recursive operations.
You can use pysftp. It's a wrapper around Paramiko that has more Python-ish look and feel and supports recursive operations. See
pysftp.Connection.put_r()
pysftp.Connection.get_r()
Or you can just base your code on pysftp source code. Or see my answer to Python pysftp get_r from Linux works fine on Linux but not on Windows.