We Keep Coding

Why does Python not find/recognise a file saved in the same directory as the file where the code is written?

I am quite new to working with Python files, and having a small issue. I am simply trying to print the name of a text file and its 'mode'.
Here is my code:
f = open('test.txt','r')
print(f.name)
print(f.mode)
f.close()
I have a saved a text file called 'test.txt' in the same directory as where I wrote the above code.
However, when I run the code, I get the following file not found error:
FileNotFoundError: [Errno 2] No such file or directory: 'test.txt'
Any ideas what is causing this?
I have also tried to replace the first argument above with the path of the test.txt file, but I get the same error?

open (on pretty much any operating system) doesn't care where your program lies, but from which directory you are running it. (This is not specific to python, but how file operations work, and what a current working directory is.)
So, this is expected. You need to run python from the directory that test.txt is in.
I have also tried to replace the first argument above with the path of the test.txt file, but I get the same error?
In that case, you must have mistyped the path. Make sure there's no special characters (like backslashes) that python interprets specially in there, or use the raw string format r'...' instead of just '...'.

It depend from where the python command is launched for instance :
let suppose we have this 2 files :
dir1/dir2/code.py <- your code
dir1/dir2/test.txt
if you run your python commande from the dir1 directory it will not work because it will search for dir1/test.txt
you need to run the python commande from the same directory(dir2 in the example).

The 'os' module in Python says "No such file or directory" even when there definitely is one

I am trying to open an .npy file. However, Python keeps saying that there exists no such file or directory, even when there is one...
To make sure that it isn't an issue of giving correct path names, I changed my directory to the folder that contains the .npy file that I want. Then used list.dir() and used it to use np.load (the code is below):
os.chdir(filename_dir) #filename_dir is the directory I want to get to that contains the npy file I want)
the_path=os.path.join(os.getcwd()+os.listdir()[-1]) #i.e. I did this to make sure that the directory is correct
data=np.load(the_path)
However, I got the error, [Errno 2] No such file or directory: .......
The error occurs when I try np.load. I couldn't understand this because I explicitly made the path the_path from what Python says exists.

I think this is the problem...
Windows (by default) has a maximum path length of 260 characters. The absolute path to this file exceeds that limit. The filename alone is 143 characters.
It looks as though even if you try to access the file using a relative path (i.e., chdir() to the appropriate folder then specify just the filename), numpy is probably working out the absolute path then failing.
You have said that renaming the file to something much shorter solves the problem but is impractical due to the high numbers of files that you need to process.
There is a Windows Registry key that can be modified to enable long pathnames: Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem\LongPathsEnabled
Changing that may help.
Or you could buy a Mac

Try adding r. For example,
file_path = r"filePath"
This may work...

os.path.join accesses file preventing rename operation

I am trying to handle a JSON decode by backing up a malformed file when the decode fails, but I'm experiencing some strange behaviour that I did not expect from the os.path.join method.
The following code fails with an exception: PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'file.txt' -> 'file.txt\\.bak'
file_path = "file.txt"
try:
with open(file_path, 'r') as f:
json.load(f)
except json.JSONDecodeError as e:
os.rename(file_path, os.path.join(file_path, '.bak'))
If I change the argument like this: os.rename(file_path, file_path + '.bak') The code executes as expected without the permission error. It seems like the os.path.join method actually accesses the file rather than being a strict string operation. Is this expected behaviour?

os.path.join(file_path, '.bak')) actually will give you file.txt\\.bak like you see in the error code, but file_path + '.bak' gives you the correct file name file.txt.bak
os.path.join appends a separator between it's arguments, hence it ends up adding that separator in your case too
Example in MacOS, you can see that it adds a separator between each of it's arguments. os.path.join is more useful to append directory names, of the full filename with the directory paths.
In [4]: import os
In [5]: os.path.join('filename','.bak')
Out[5]: 'filename/.bak'
In [6]: os.path.join('folder1', 'folder2')
Out[6]: 'folder1/folder2'
The error happens since the Windows OS is trying to make a file .bak in a folder named file.txt, which isn't possible since file.txt is a plain file and not a directory, which is correct.
Using file_path+'.bak creates the file.path.bak correctly in the folder you want, hence you don't see an error there!

The error message is the key. As usually on Windows the cause (because it is being used by another process) is wrong, but the names ('file.txt' -> 'file.txt\.bak') are correct.
Join is not a string concatenation but expects that all path members except the last represent folders. So here you are trying to make a file .bak in a folder named file.txt. It is not possible because file.txt is a plain file and not a directory.
On another hand, when you use os.rename(file_path, file_path + '.bak') you are renaming file.txt to file.txt.bak in the same folder which is allowed by the underlying file system, hence no error.
So the behaviour is exactly what is expected, except for the beginning of the error message.
As I am not a core Microsoft Developper, the following is a wild guess. The number of error given by the system is limited. The rename C function received 2 strings and passed it to the system call for rename. As expected the file system generated an error but as it was neither a physical error nor a file system full error, it just choosed a permission refused cause. Which is not really wrong because it is not allowed to create folders under a plain file. But the message for that error is unfortunately because it is being used by another process which is stupid here

Python: How do I interact with unicode filenames on Windows? (Python 2.7)

My problem:
Start with US Windows 10 install
Create a Japanese filename in Windows explorer
Open the Python shell, and os.listdir('.')
The listed filename is full of question marks.
os.path.exists() unsurprisingly reports file not found.
NTFS stores the filename as Unicode. I'm sure if I used the win32api CreateFile() series of functions I will get my Unicode filename back, however those APIs are too cumbersome (and not portable). I'd prefer that I get utf-8 encoded filenames, or the Unicode bytes from the FS directory structure, but in default mode this doesn't seem to happen.
I have tried playing around with setlocale() but I haven't stumbled upon the correct arguments to make my program work. I do not want to (and cannot) install additional code pages onto the Windows machine. This needs to work with a stock install of Windows.
Please note this has nothing to do with the console. A repr() shows that the ? chars that end up in the filename listed by os.listdir('.') are real question marks and not some display artifact. I assume they have been added by the API that listdir() uses under the hood.

You may be getting ?s while displaying that filename in the console using os.listdir() but you can access that filename without any problems as internally everything is stored in binary. If you are trying to copy the filename and paste it directly in python, it will be interpreted as mere question marks...
If you want to open that file and perform any operations, then, have a look at this...
files = os.listdir(".")
# Possible output:
# ["a.txt", "file.py", ..., "??.html"]
filename = files[-1] # The last file in this case
f = open(filename, 'r')
# Sample file operation
lines = f.readlines()
print(lines)
f.close()
EDIT:
In Python 2, you need to pass current path as Unicode which could be done using: os.listdir(u'.'), where the . means current path. This will return the list of filenames in Unicode...

Primer needed in python pathnames

I am a very novice coder, and Python is my first (and, practically speaking, only) language. I am charged as part of a research job with manipulating a collection of data analysis scripts, first by getting them to run on my computer. I was able to do this, essentially by removing all lines of coding identifying paths, and running the scripts through a Jupyter terminal opened in the directory where the relevant modules and CSV files live so the script knows where to look (I know that Python defaults to the location of the terminal).
Here are the particular blocks of code whose function I don't understand
import sys
sys.path.append('C:\Users\Ben\Documents\TRACMIP_Project\mymodules/')
import altdata as altdata
I have replaced the pathname in the original code with the path name leading to the directory where the module is; the file containing all the CSV files that end up being referenced here is also in mymodules.
This works depending on where I open the terminal, but the only way I can get it to work consistently is by opening the terminal in mymodules, which is fine for now but won't work when I need to work by accessing the server remotely. I need to understand better precisely what is being done here, and how it relates to the location of the terminal (all the documentation I've found is overly technical for my knowledge level).
Here is another segment I don't understand
import os.path
csvfile = 'csv/' + model +'_' + exp + '.csv'
if os.path.isfile(csvfile): # csv file exists
hcsvfile = open(csvfile )
I get here that it's looking for the CSV file, but I'm not sure how. I'm also not sure why then on some occasions depending on where I open the terminal it's able to find the module but not the CSV files.
I would love an explanation of what I've presented, but more generally I would like information (or a link to information) explaining paths and how they work in scripts in modules, as well as what are ways of manipulating them. Thanks.

sys.path
This is simple list of directories where python will look for modules and packages (.py and dirs with __init__.py file, look at modules tutorial). Extending this list will allow you to load modules (custom libs, etc.) from non default locations (usually you need to change it in runtime, for static dirs you can modify startup script to add needed enviroment variables).
os.path
This module implements some useful functions on pathnames.
... and allows you to find out if file exists, is it link, dir, etc.
Why you failed loading *.csv?
Because sys.path responsible for module loading and only for this. When you use relative path:
csvfile = 'csv/' + model +'_' + exp + '.csv'
open() will look in current working directory
file is either a string or bytes object giving the pathname (absolute or relative to the current working directory)...
You need to use absolute paths by constucting them with os.path module.

I agree with cdarke's comment that you are probably running into an issue with backslashes. Replacing the line with:
sys.path.append(r'C:\Users\Ben\Documents\TRACMIP_Project\mymodules')
will likely solve your problem. Details below.
In general, Python treats paths as if they're relative to the current directory (where your terminal is running). When you feed it an absolute path-- which is a path that includes the root directory, like the C:\ in C:\Users\Ben\Documents\TRACMIP_Project\mymodules-- then Python doesn't care about the working directory anymore, it just looks where you tell it to look.
Backslashes are used to make special characters within strings, such as line breaks (\n) and tabs (\t). The snag you've hit is that Python paths are strings first, paths second. So the \U, \B, \D, \T and \m in your path are getting misinterpreted as special characters and messing up Python's path interpretation. If you prefix the string with 'r', Python will ignore the special characters meaning of the backslash and just interpret it as a literal backslash (what you want).
The reason it still works if you run the script from the mymodules directory is because Python automatically looks in the working directory for files when asked. sys.path.append(path) is telling the computer to include that directory when it looks for commands, so that you can use files in that directory no matter where you're running the script. The faulty path will still get added, but its meaningless. There is no directory where you point it, so there's nothing to find there.
As for path manipulation in general, the "safest" way is to use the function in os.path, which are platform-independent and will give the correct output whether you're working in a Windows or a Unix environment (usually).
EDIT: Forgot to cover the second part. Since Python paths are strings, you can build them using string operations. That's what is happening with the line
csvfile = 'csv/' + model +'_' + exp + '.csv'
Presumably model and exp are strings that appear in the filenames in the csv/ folder. With model = "foo" and exp = "bar", you'd get csv/foo_bar.csv which is a relative path to a file (that is, relative to your working directory). The code makes sure a file actually exists at that path and then opens it. Assuming the csv/ folder is in the same path as you added in sys.path.append, this path should work regardless of where you run the file, but I'm not 100% certain on that. EDIT: outoftime pointed out that sys.path.append only works for modules, not opening files, so you'll need to either expand csv/ into an absolute path or always run in its parent directory.
Also, I think Python is smart enough to not care about the direction of slashes in paths, but you should probably not mix them. All backslashes or all forward slashes only. os.path.join will normalize them for you. I'd probably change the line to
csvfile = os.path.join('csv\', model + '_' + exp + '.csv')
for consistency's sake.