I started moving parts of code outside my jupyter notebooks. I am new in Python and want to make everything right from the beginning. I have a following project structure.
Using data/raw/filename obviously doesn't work, I have to go one level up. Like:
with open ('../data/dumps/df_aggregated.dmp', 'rb') as fp:
df_aggregated = pickle.load(fp)
I think I can save my absolute path to data somewhere in a python script, but I have some doubts if it is right. What if I execute a script/a notebook from a nested folder?
Please help me to understand how to arrange my folders and make it flexible
Related
I am wondering if there is an easy way to access 'parallel' directories (See photo for what I am talking about... I don't know what else to call them, please correct me if they are called something else!) from a Python file without having to input the string path.
The basic structure I intend to use is shown in the picture. The structure will be used across different computers, so I need to avoid just typing in "C:\stuff_to_get_there\parent_directory\data\file.txt" because "C:\stuff_to_get_there" will not be the same on different computers.
I want to store the .py files in their own directory, then access the data files in data directory, and save figures to figures directory. I was thinking of trying os module but not sure if that's the correct way to go.
parent directory
scripts
.py files
figures
save files here
data
.txt files stored here
Thanks for any help!
I have a program that makes calculus, converts different data and plots it while saving it in different directories. So far, so good BUT. Whenever I want to move the files generated to another folder without closing Python it doesn't let me only with one of them, telling me that a program still has it open and I can't move it. The weird part is that I can take out all the plots generated so it's the folder itself that can't be moved. I don't know what to do because other folders are generated and treated the same way and don't give me problems. Any idea?
TL;DR: Can't move a empty folder while Python is still open
path_data_anaconv = path_data + r'TheProblem/'
if not os.path.exists(path_data_anaconv):
os.makedirs(path_data_anaconv)
I am creating a program in Python that requires the user to place images into an Input folder, and then take images out of an Output folder. As this will become an application, the Input and Output folders will be very difficult to navigate to, being buried in the app's contents.
I am looking for a way to open folders onscreen so that a user can add or remove their own files from these folders, without knowing the exact location of the folders they are interacting with.
I am thoroughly stumped on this problem, and I appreciate all of your time.
Thank you very much.
*Edit: I am working on MacOS
If the program is intended for use on windows, it seems like you could use the solution here, where you could open explorer as a sub process and then open the path to the file.
For example
import subprocess
subprocess.Popen(r'explorer /select, FilePath')
You could also use os, and os.startfile(FilePath).
I'm using Geany as my editor and when I first started using it, writing to files worked fine but somehow it randomly stopped working. The code executes without any errors but the file isn't created / is empty if already created and I've no idea why.
Simple code as below doesn't work:
filename = 'dogs'
with open(filename, 'w') as f:
f.write('tester')
with open(filename, 'r') as f:
contents = f.read()
print(contents)
The output I get from the 'read' method looks correct on the console output (it just prints 'tester'), but no file is created / edited in my directory.
Geany also has a weirdly complex debugger (if anyone has any helpful guides on how to use it please let me know) so I can't debug properly. I've tried all that I know including using an absolute file path, running in admin mode. The issue is also present when I try to use Pygal to render_to_file(), which is the project I'm working on so right now I can't go any further because anything that requires writing doesn't work. FYI it reads fine.... It's like Geany doesn't have admin rights or something?
EDIT: I've run this code on a python shell (without a .py file) and it worked fine, creating the file as desired. I then ran it using CMD with the .py file and it didn't work. Also ran using Pycharm, it doesn't work when I run it normal but it works when I run it in debug mode? It doesn't seem to be a Geany specific issue, but I am so confused!
Because the code works in the shell I strongly suspect the file is being created somewhere but you are looking in the wrong place.
I know you mentioned absolute paths but I just want to reiterate that you are not currently using an absolute path. An absolute path must start with a '/' (linux/mac) or something like: 'C:/' (windows).
Here are 2 things to try:
1)
Change the name 'dogs' to something really obscure and then do a global search on your whole hard disk for that name. You'll probably find it.
2) Get python to tell you where the file is like this:
import os
filename = 'dogs'
with open(filename, 'w') as f:
f.write('tester')
print(os.path.realpath(f.name))
Got the solution - the files were being created the whole time every time I ran my code, however my antivirus software Comodo contained them within a hidden folder in my drive which couldn't be found by a normal search.
I'm not sure if this a common problem with AV software or just Comodo, can't find anything on the net about it but there you go. I removed these files and the programs from its radar and it now works perfectly.
Although one mystery is how the python shell bypassed that problem. That threw me off thinking it was AV.
Just a brief outline of what I'm doing: I'm trying to automate some pdf merging routine with python in a network directory, which involves copying, deleting and creating files at a specific network location. Apologies if my language is not very precise.
I'm coding on windows 7, using python 3.6. The program will need to be distributed on other machines, so local and temporary fixes will probably not help. The code I wrote is fully functional and works fine with all the local folders and files, however, now that I need to make use of it on the network, I am having some difficulties accessing the folder I need.
Here is what I have tried:
os.system("pushd " + "\\" + "\\netWorkDrive\Reports")
check_output("pushd " + "\\" + "\\netWorkDrive\Reports", shell=True)
pushd and popd work fine when entered just in the cmd, but when I do system calls through python, they just don't go through. I send a system call, and it runs correctly, but then when I "cd" a current directory, it shows that I'm still in my previous one. If done through the cmd manually, everything works as desired. I have googled the issue, but did not end up finding anything working/useful. I would really appreciate any suggestions, and let me know if I need to clarify my problem further.
Thank you!
I would not use pushd/popd in such a way, I would just include the full paths, including network paths in the paths of whatever file operation I need to do
However if I really need to change working directory, I would do this with python:
import os
original_working_directory = os.getcwd()
# do stuff
new_networked_directory = r'\\server\share\folder'
# change to the networked directory
os.chdir(new_networked_directory)
# do stuff
#changeback to original working directory
os.chdir(original_working_directory)
# do more stuff
There should be no need for "temp drives" or the like really.