My python script is running on Windows without any issues. But the same script has to run on a Linux machine as well. When ran, it prompts that the specified path is non-existent. Note that variable "path" is pointing to a cloud server
I subsequently tried the os.path.join function after looking through some forums, which also failed
import os
import re
import sys
#List .xlsx files followed by the string ESC
path = '\\\\cloudnetworkonlinuxserver'
path2 = 'DBX'
path3 = 'SrcFiles'
path4 = 'MEBilling'
path5 = 'ParmFiles'
filenames = os.listdir(os.path.join(path, path2, path3, path4))
for filename in filenames:
getdate = re.search('(?<=ESC_)\w+', filename)
#Replace '_' with '-'
if getdate:
date = getdate.group(0).replace('_', '-')
print('The following ESC file has date', date)
#Create .prm file with following body
f = open(os.path.join(path, path2, path5, "wf_SC_Monthend_Billing_XLS" + "." + "prm"), 'w')
#f.write cannot take more than one argument. Write variables such
a = '$$WF_PERIOD='
b = date
#Write in body of file
f.write("[Global]\n")
f.write('%s%s' % (a,b,))
#Close writing process
f.close
What other methods are there in specifying a path that is compatible with those both operating systems?
Assuming that \\cloudnetworkonlinuxserver is a Samba share: This path alone is Windows-specific. Depending on the platform, there may be a different way how this share would be accessed.
On Linux, you'd have to mount this share at some physical path first, for example /mnt/cloudshare. Then you'd access this path instead.
You should take this path from a command line argument or environment variable so that the appropriate path can be passed depending on what is correct in each environment.
Or, in case this should be part of the script, you'd have to make your script take care of mounting the share in case the environment is Linux (and ideally also unmounting it when no longer needed).
Related
I am trying to access a file from a Box folder as I am working on two different computers. So the file path is pretty much the same except for the username.
I am trying to load a numpy array from a .npy file and I could easily change the path each time, but it would be nice if I could make it universal.
Here is what the line of code looks like on my one computer:
y_pred_walking = np.load('C:/Users/Eric/Box/CMU_MBL/Data/Calgary/4_Best_Results/Walking/Knee/bidir_lstm_50_50/predictions/y_pred_test.npy')
And here is what the line of code looks like on the other computer:
y_pred_walking = 'C:/Users/erapp/Box/CMU_MBL/Data/Calgary/4_Best_Results/Walking/Knee/bidir_lstm_50_50/predictions/y_pred_test.npy'
The only difference is that the username on one computer is Eric and the other is erapp, but is there a way where I can make the line universal to all computers where all computers will have the Box folder?
You could either save the file to a path that doesn't depend on the user: e.g. 'C:/Box/CMU_MBL/Data/Calgary/4_Best_Results/Walking/Knee/bidir_lstm_50_50/predictions/y_pred_test.npy'
Or you could do some string formatting. One way would be with an environment or configuration variable that indicates which is the relevant user, and then for your load statement:
import os
current_user = os.environ.get("USERNAME") # assuming you're running on the Windows box as the relevant user
# Now load the formatted string. f-strings are better, but this is more obvious since f-strings are still very new to Python
y_pred_walking = 'C:/Users/{user}/Box/CMU_MBL/Data/Calgary/4_Best_Results/Walking/Knee/bidir_lstm_50_50/predictions/y_pred_test.npy'.format(user=current_user)
Yes, there is a way, at least for the problem as it is right now solution is pretty simple: to use f-strings
user='Eric'
y_pred_walking =np.load(f'C:/Users/{user}/Box/CMU_MBL/Data/Calgary/4_Best_Results/Walking/Knee/bidir_lstm_50_50/predictions/y_pred_test.npy')
or more general
def pred_walking(user):
return np.load(f'C:/Users/{user}/Box/CMU_MBL/Data/Calgary/4_Best_Results/Walking/Knee/bidir_lstm_50_50/predictions/y_pred_test.npy')
so on any machine you just do
y_pred_walking=pred_walking(user)
with defined user before, to receive the result
Simply search the folders recursivly for your file:
filename = 'y_pred_test.npy'
import os
import random
# creates 1000 directories with a 1% chance of having the file as well
for k in range(20):
for i in range(10):
for j in range(5):
os.makedirs(f"./{k}/{i}/{j}")
if random.randint(1,100) == 2:
with open(f"./{k}/{i}/{j}/{filename}","w") as f:
f.write(" ")
# search the directories for your file
found_in = []
# this starts searching in your current folder - you can give it your c:\Users\ instead
for root,dirs,files in os.walk("./"):
if filename in files:
found_in.append(os.path.join(root,filename))
print(*found_in,sep = "\n")
File found in:
./17/3/1/y_pred_test.npy
./3/8/1/y_pred_test.npy
./16/3/4/y_pred_test.npy
./16/5/3/y_pred_test.npy
./14/2/3/y_pred_test.npy
./0/5/4/y_pred_test.npy
./11/9/0/y_pred_test.npy
./9/8/1/y_pred_test.npy
If you get read errors because of missing file/directory permissions you can start directly in the users folder:
# Source: https://stackoverflow.com/a/4028943/7505395
from pathlib import Path
home = str(Path.home())
found_in = []
for root,dirs,files in os.walk(home):
if filename in files:
found_in.append(os.path.join(root,filename))
# use found_in[0] or break as soon as you find first file
You can use the expanduser function in the os.path module to modify a path to start from the home directory of a user
https://docs.python.org/3/library/os.path.html#os.path.expanduser
i am rather new to python and i have the following problem (just an example):
import os
mypath = 'I:\Folder1'
for dirpath,_,filenames in os.walk(mypath):
for f in filenames:
getpath = os.path.abspath(os.path.join(dirpath, f))
returns the path in the form:
I:\Folder1\Folder2
which is normally ok for me
However "I:\" is one of our servers at work and for further processing (html stuff) i would need the exact address in such a form
file://///Servername/Subfolder/Folder1/Folder2
Edit: In other words:
My program may be used locally or on different servers - it just depends on the user. Rather stupidly said I need a function that returns what in win10 goes like this: "right click on a folder --> Path Copy --> file:////....". And I only know that this path on my computer is called "I:\Folder1" ..but "I:\Folder1" is the server name
Edit 2: Solved (see comments)
If you are on a Windows platform and need forward slashes, it is actually possible to import the OS specific version. For example you could use posixpath.
To solve your problem you would need to first strip off mypath from each return dirpath. Next split this into folder components using split with your operating system's seperator i.e. \. This can then be all rejoined with a server prefix using the posixpath.join() command. For example:
import posixpath
import os
mypath = r'I:\Folder1'
server = 'file://///Servername/Subfolder'
for dirpath,_,filenames in os.walk(mypath):
for f in filenames:
subfolder = dirpath[len(mypath):]
server_path = posixpath.join(server, *subfolder.split(os.sep), f)
print(server_path)
I am trying to save a file in the QCTestFiles directory, but my script is not writing the files to the specified directory. Is my path file correct for Linux and if it is not how can I change my code to write and save a file in a different directory? My current script is running in the brperez location.
filename = (GPN + '_' + inspector + '_' + date + '.txt')
save_path = 'c:/usr/local/home/brperez/QCTestFiles'
complete_name = os.path.join(save_path, filename)
file1 = open(complete_name, "w")
I don't think that path is correct for linux. I can't remember ever preceding a linux path with 'c:', it's been a while since I spent a lot of time in a true linux env, but I suppose it could depend on what you are doing.
Do you have access to the machine? Navigate to the folder in terminal and type 'pwd' to get the proper path. Also, make sure your application has write permissions to the folder.
No, it is not correct for linux. There is no concept of a 'c:' drive in linux. You should only have to specify the path, e.g. '/usr/local/home/brperez/QCTestFiles'
For more information about drive naming in linux, take a look at this question, and the links in the top answer: https://askubuntu.com/questions/56929/what-is-the-linux-drive-naming-scheme
Since you are using Linux, your path will not have a drive letter, it will just be something like /usr/local/home/brperez.
Hope that helps.
giving that you're running your script in the brperez location, you can replace your snippet with the following one:
filename = (GPN + '_' + inspector + '_' + date + '.txt')
save_path = os.path.join(os.path.dirname(os.path.realpath(__file__)), 'QCTestFiles')
complete_name = os.path.join(save_path, filename)
file1 = open(complete_name, "w")
I´m trying to save a file, which I create with the "open" function.
Well I tried nearly everything to change the directory, but nothing works. The file gets always saved in the folder of my file, which I read in before.
file = open(fname[0] + ft, 'w')
file.write("Test")
file.close()
So this is it simple, but what do I have to add, to change the path of creation?
The File Dialog in a individual Function:
global fname
fname = QFileDialog.getOpenFileName(None, 'Please choose your File.',"C:\\Program Files", "Text-Files(*.txt)")
And the File Typ ( in a individual Function too) I set the file type by ticking a check box and ft will set to .py or .pyw
if self.exec_py.isChecked() == True:
global ft
ft = ".py"
I should have mentioned that I already tried os.path.join and os.chdir, but the file will get printed in the file anyway. Any solutions or approaches how to fix it? Here is how i tried it:
tmppath = "C:/temp"
tmp = os.path.join(tmppath,fname[0]+ft)
file = open(tmp, 'w')
Your question is a little short on details, but I am guessing that fname is the tuple returned by QFileDialog, and so fname[0] is the absolute path of the original file. So if you display fname[0], you will see something like this:
>>> fname[0]
'C:\\myfolder\\file.txt'
Now look what happens when you try to use that with os.path.join:
>>> tmppath = 'C:\\temp'
>>> os.path.join(tmppath, fname[0])
'C:\\myfolder\\file.txt'
Nothing! Conclusion: attempting to join two absolute paths will simply return the original path unchanged. What you need to do instead is take the basename of the original path, and join it to the folder where you want to save it:
>>> basename = os.path.basename(fname[0])
>>> basename
'file.txt'
>>> os.path.join(tmppath, basename)
'C:\\tmp\\file.txt'
Now you can use this new path to save your file in the right place.
You need to provide the full filepath
with open(r'C:\entire\path\to\file.txt', 'w') as f:
f.write('test')
If you just provide a file name without a path, it will use the current working directory, which isn't necessarily the directory where the python script your running is located. It will be the directory where you launched the script from.
C:\Users\admin> python C:\path\to\my_script.py
In this instance, the current working directory is C:\Users\admin, not C:\path\to.
I need to scan a file system for a list of files, and log those who don't exist. Currently I have an input file with a list of the 13 million files which need to be investigated. This script needs to be run from a remote location, as I do not have access/cannot run scripts directly on the storage server.
My current approach works, but is relatively slow. I'm still fairly new to Python, so I'm looking for tips on speeding things up.
import sys,os
from pz import padZero #prepends 0's to string until desired length
output = open('./out.txt', 'w')
input = open('./in.txt', 'r')
rootPath = '\\\\server\share\' #UNC path to storage
for ifid in input:
ifid = padZero(str(ifid)[:-1], 8) #extracts/formats fileName
dir = padZero(str(ifid)[:-3], 5) #exracts/formats the directory containing the file
fPath = rootPath + '\\' + dir + '\\' + ifid + '.tif'
try:
size = os.path.getsize(fPath) #don't actually need size, better approach?
except:
output.write(ifid+'\n')
Thanks.
dirs = collections.defaultdict(set)
for file_path in input:
file_path = file_path.rjust(8, "0")
dir, name = file_path[:-3], file_path
dirs[dir].add(name)
for dir, files in dirs.iteritems():
for missing_file in files - set(glob.glob("*.tif")):
print missing_file
Explanation
First read the input file into a dictionary of directory: filename. Then for each directory, list all the TIFF files in that directory on the server, and (set) subtract this from the collection of filenames you should have. Print anything that's left.
EDIT: Fixed silly things. Too late at night when I wrote this!
That padZero and string concatenation stuff looks to me like it would take a good percent of time.
What you want it to do is spend all its time reading the directory, very little else.
Do you have to do it in python? I've done similar stuff in C and C++. Java should be pretty good too.
You're going to be I/O bound, especially on a network, so any changes you can make to your script will result in very minimal speedups, but off the top of my head:
import os
input, output = open("in.txt"), open("out.txt", "w")
root = r'\\server\share'
for fid in input:
fid = fid.strip().rjust(8, "0")
dir = fid[:-3] # no need to re-pad
path = os.path.join(root, dir, fid + ".tif")
if not os.path.isfile(path):
output.write(fid + "\n")
I don't really expect that to be any faster, but it is arguably easier to read.
Other approaches may be faster. For example, if you expect to touch most of the files, you could just pull a complete recursive directory listing from the server, convert it to a Python set(), and check for membership in that rather than hitting the server for many small requests. I will leave the code as an exercise...
I would probably use a shell command to get the full listing of files in all directories and subdirectories in one hit. Hopefully this will minimise the amount of requests you need to make to the server.
You can get a listing of the remote server's files by doing something like:
Linux: mount the shared drive as /shared/directory/ and then do ls -R /shared/directory > ~/remote_file_list.txt
Windows: Use Map Network Drive to mount the shared drive as drive letter X:, then do dir /S X:/shared_directory > C:/remote_file_list.txt
Use the same methods to create a listing of your local folder's contents as local_file_list.txt. You python script will then reduce to an exercise in text processing.
Note: I did actually have to do this at work.