I have recently begun working on a new computer. All my python files and my data are in the dropbox folder, so having access to the data is not a problem. However, the "user" name on the file has changed. Thus, none of my os.chdir() operations work. Obviously, I can modify all of my scripts using a find and replace, but that won't help if I try using my old computer.
Currently, all the directories called look something like this:
"C:\Users\Old_Username\Dropbox\Path"
and the files I want to access on the new computer look like:
"C:\Users\New_Username\Dropbox\Path"
Is there some sort of try/except I can build into my script so it goes through the various path-name options if the first attempt doesn't work?
Thanks!
Any solution will involve editing your code; so if you are going to edit it anyway - its best to make it generic enough so it works on all platforms.
In the answer to How can I get the Dropbox folder location programmatically in Python? there is a code snippet that you can use if this problem is limited to dropbox.
For a more generic solution, you can use environment variables to figure out the home directory of a user.
On Windows the home directory is location is stored in %UserProfile%, on Linux and OSX it is in $HOME. Luckily Python will take care of all this for you with os.path.expanduser:
import os
home_dir = os.path.expanduser('~')
Using home_dir will ensure that the same path is resolved on all systems.
Thought the file sq.py with these codes(your olds):
C:/Users/Old_Username/Dropbox/Path
for x in range:
#something
def Something():
#something...
C:/Users/Old_Username/Dropbox/Path
Then a new .py file run these codes:
with open("sq.py","r") as f:
for x in f.readlines():
y=x
if re.findall("C:/Users/Old_Username/Dropbox/Path",x) == ['C:/Users/Old_Username/Dropbox/Path']:
x="C:/Users/New_Username/Dropbox/Path"
y=y.replace(y,x)
print (y)
Output is:
C:/Users/New_Username/Dropbox/Path
for x in range:
#something
def Something():
#something...
C:/Users/New_Username/Dropbox/Path
Hope its your solution at least can give you some idea dealing with your problem.
Knowing that eventually I will move or rename my projects or scripts, I always use this code right at the beginning:
import os, inspect
this_dir = os.path.dirname(os.path.abspath(inspect.getfile(inspect.currentframe())))
this_script = inspect.stack()[0][1]
this_script_name = this_script.split('/')[-1]
If you call your script not with the full but a relative path, then this_script will also not contain a full path. this_dir however will always be the full path to the directory.
Related
I'm trying to run a script that works without issue when I run using in console, but causes issue if I try to run it from another directory (via IPython %run <script.py>)
The issue comes from this line, where it references a folder called "Pickles".
with open('Pickles/'+name+'_'+date.strftime('%y-%b-%d'),'rb') as f:
obj = pickle.load(f)
In Console:
python script.py <---works!
In running IPython (Jupyter) in another folder, it causes a FileNotFound exception.
How can I make any path references within my scripts more robust, without putting the whole extended path?
Thanks in advance!
Since running in the console the way you show works, the Pickles directory must be in the same directory as the script. You can make use of this fact so that you don't have to hard code the location of the Pickles directory, but also don't have to worry about setting the "current working directory" to be the directory containing Pickles, which is what your current code requires you to do.
Here's how to make your code work no matter where you run it from:
with open(os.path.join(os.path.dirname(__file__), 'Pickles', name + '_' + date.strftime('%y-%b-%d')), 'rb') as f:
obj = pickle.load(f)
os.path.dirname(__file__) provides the path to the directory containing the script that is currently running.
Generally speaking, it's a good practice to always fully specify the locations of things you interact with in the filesystem. A common way to do this as shown here.
UPDATE: I updated my answer to be more correct by not assuming a specific path separator character. I had chosen to use '/' only because the original code in the question already did this. It is also the case that the code given in the original question, and the code I gave originally, will work fine on Windows. The open() function will accept either type of path separator and will do the right thing on Windows.
You have to use absolute paths. Also to be cross platform use join:
First get the path of your script using the variable __file__
Get the directory of this file with os.path.dirname(__file__)
Get your relative path with os.path.join(os.path.dirname(__file__), "Pickles", f"{name}_{date.strftime('%y-%b-%d')}")
it gives you:
with open(os.path.join(os.path.dirname(__file__), "Pickles", f"{name}_{date.strftime('%y-%b-%d')}"), 'rb') as f:
obj = pickle.load(f)
I have a python file, converted from a Jupiter Notebook, and there is a subfolder called 'datasets' inside this file folder. When I'm trying to open a file that is inside that 'datasets' folder, with this code:
import pandas as pd
# Load the CSV data into DataFrames
super_bowls = pd.read_csv('/datasets/super_bowls.csv')
It says that there is no such file or folder. Then I add this line
os.getcwd()
And the output is the top-level folder of the project, and not the subfolder when is this python file. And I think maybe that's the reason why it's not working.
So, how can I open that csv file with relative paths? I don't want to use absolute path because this code is going to be used in another computers.
Why os.getcwd() is not getting the actual folder path?
My observation, the dot (.) notation to move to the parent directory sometimes does not work depending on the operating system. What I generally do to make it os agnostic is this:
import pandas as pd
import os
__location__ = os.path.realpath(os.path.join(os.getcwd(), os.path.dirname(__file__)))
super_bowls = pd.read_csv(__location__ + '/datasets/super_bowls.csv')
This works on my windows and ubantu machine equally well.
I am not sure if there are other and better ways to achieve this. Would like to hear back if there are.
(edited)
Per your comment below, the current working directory is
/Users/ivanparra/AprendizajePython/
while the file is in
/Users/ivanparra/AprendizajePython/Jupyter/datasets/super_bowls.csv
For that reason, going to the datasets subfolder of the current working directory (CWD) takes you to /Users/ivanparra/AprendizajePython/datasets which either doesn't exist or doesn't contain the file you're looking for.
You can do one of two things:
(1) Use an absolute path to the file, as in
super_bowls = pd.read_csv("/Users/ivanparra/AprendizajePython/Jupyter/datasets/super_bowls.csv")
(2) use the right relative path, as in
super_bowls = pd.read_csv("./Jupyter/datasets/super_bowls.csv")
There's also (3) - use os.path.join to contact the CWD to the relative path - it's basically the same as (2).
(you can also use
The answer really lies in the response by user2357112:
os.getcwd() is working fine. The problem is in your expectations. The current working directory is the directory where Python is running, not the directory of any particular source file. – user2357112 supports Monica May 22 at 6:03
The solution is:
data_dir = os.path.dirname(__file__)
Try this code
super_bowls = pd.read_csv( os.getcwd() + '/datasets/super_bowls.csv')
I noticed this problem a few years ago. I think it's a matter of design style. The problem is that: your workspace folder is just a folder, not a project folder. Most of the time, your relative reference is based on the current file.
VSCode actually supports the dynamic setting of cwd, but that's not the default. If your work folder is not a rigorous and professional project, I recommend you adding the following settings to launch.json. This is the simplest answer you need.
"cwd": "${fileDirname}"
Thanks to everyone that tried to help me. Thanks to the Roy2012 response, I got a code that works for me.
import pandas as pd
import os
currentPath = os.path.dirname(__file__)
# Load the CSV data into DataFrames
super_bowls = pd.read_csv(currentPath + '/datasets/super_bowls.csv')
The os.path.dirname gives me the path of the current file, and let me work with relative paths.
'/Users/ivanparra/AprendizajePython/Jupyter'
and with that it works like a charm!!
P.S.: As a side note, the behavior of os.getcwd() is quite different in a Jupyter Notebook than a python file. Inside the notebook, that function gives the current file path, but in a python file, gives the top folder path.
I'm trying to load a simple text file with an array of numbers into Python. A MWE is
import numpy as np
BASE_FOLDER = 'C:\\path\\'
BASE_NAME = 'DATA.txt'
fname = BASE_FOLDER + BASE_NAME
data = np.loadtxt(fname)
However, this gives an error while running:
OSError: C:\path\DATA.txt not found.
I'm using VSCode, so in the debug window the link to the path is clickable. And, of course, if I click it the file opens normally, so this tells me that the path is correct.
Also, if I do print(fname), VSCode also gives me a valid path.
Is there anything I'm missing?
EDIT
As per your (very helpful for future reference) comments, I've changed my code using the os module and raw strings:
BASE_FOLDER = r'C:\path_to_folder'
BASE_NAME = r'filename_DATA.txt'
fname = os.path.join(BASE_FOLDER, BASE_NAME)
Still results in error.
Second EDIT
I've tried again with another file. Very basic path and filename
BASE_FOLDER = r'Z:\Data\Enzo\Waste_Code'
BASE_NAME = r'run3b.txt'
And again, I get the same error.
If I try an alternative approach,
os.chdir(BASE_FOLDER)
a = os.listdir()
then select the right file,
fname = a[1]
I still get the error when trying to import it. Even though I'm retrieving it directly from listdir.
>> os.path.isfile(a[1])
False
Using the module os you can check the existence of the file within python by running
import os
os.path.isfile(fname)
If it returns False, that means that your file doesn't exist in the specified fname. If it returns True, it should be read by np.loadtxt().
Extra: good practice working with files and paths
When working with files it is advisable to use the amazing functionality built in the Base Library, specifically the module os. Where os.path.join() will take care of the joins no matter the operating system you are using.
fname = os.path.join(BASE_FOLDER, BASE_NAME)
In addition it is advisable to use raw strings by adding an r to the beginning of the string. This will be less tedious when writing paths, as it allows you to copy-paste from the navigation bar. It will be something like BASE_FOLDER = r'C:\path'. Note that you don't need to add the latest '\' as os.path.join takes care of it.
You may not have the full permission to read the downloaded file. Use
sudo chmod -R a+rwx file_name.txt
in the command prompt to give yourself permission to read if you are using Ubuntu.
For me the problem was that I was using the Linux home symbol in the link (~/path/file). Replacing it with the absolute path /home/user/etc_path/file worked like charm.
I try to write a script in Python that saves the file in each user directory.
Example for user 1, 2 and 3.
C:\Users\user1\Documents\ArcGIS\file1.gdb
C:\Users\user2\Documents\ArcGIS\file1.gdb
C:\Users\user3\Documents\ArcGIS\file1.gdb
How can I do this?
As one commenter pointed out, the simplest solution is to use the USERPROFILE environment variable to write the file path. This would look something like:
import os
userprofile = os.environ['USERPROFILE']
path = os.path.join(userprofile, 'Documents', 'ArcGIS', 'file1.gdb')
Or even more simply (with better platform-independence, as this will work on Mac OSX/Linux, too; credit to Abhijit's answer below):
import os
path = os.path.join(os.path.expanduser('~'), 'Documents', 'ArcGIS', 'file1.gdb')
Both of the above may have some portability issues across Windows versions, since Microsoft has been known to change the name of the "Documents" folder back and forth from "My Documents".
If you want a Windows-portable way to get the "Documents" folder, see the code here: https://stackoverflow.com/questions/3858851#3859336
In Python you can use os.path.expanduser to get the User's home directory.
>>> import os
>>> os.path.expanduser("~")
This is a platform independent way of determining the user's home directory.
You can then concatenate the result to create your final path
os.path.join(os.path.expanduser("~"), 'Documents', 'ArcGIS', 'file1.gdb')
You want to use the evironment variable HOME, something like this:
import os
homeDir = os.environ["HOMEPATH"]
file = open(homeDir+"Documents\ArcGIS\file1.gdb")
file.write("Hello, World")
file.close()
Notice that I've used HOMEPATH considering you're using Windows, it may be wrong depending on your OS. Take a look at this: http://en.wikipedia.org/wiki/Environment_variable
I'm not sure even where to start.
I have a list of output files from a program, lets call them foo. They are numbered outputs like foo_1.out
I'd like to make a directory for each file, move the file to its directory, run a bash script within that directory, take the output from each script, copy it to the root directory as a concatenated single file.
I understand that this is not a forum for "hey, do my work for me", I'm honestly trying to learn. Any suggestions on where to look are sincerely appreciated!
Thanks!
You should probably look up the documentation for the python modules os - specifically os.path and a couple of others - and subprocess which can be found here and here respectively.
Without wanting to do it all for you as you stated - you'll be wanting to do something like:
for f in filelist:
[pth, ext] = os.path.splitext(f)
os.mkdir(pth)
out = subprocess.Popen(SCRIPTNAME, stdout=...)
# and so on...
To get a list of all files in a directory or make folders, check out the os module. Specifically, try os.listdir and os.mkdir
To copy files, you could either manually open each file, copy the contents to a string, and rewrite it to a different file. Alternatively, look at the shutil module
To run bash scripts, use the subprocess library.
All three of those should be a part of python's standard library.