We Keep Coding

Getting dir path for file in argument in Python

I'm using argparse to send .pcap's to a script that scrapes through them. Wanted to organize what I was doing better, so I threw the data I was using in one folder and my scripts in another. Interfered with the way I was saving IP lists/hostnames (filename now tries to just add 'IP-list-' before the path to the .pcap file I sent as an argument).
new_ips_filename = '/IP-list-' + self.pcap
new_ips_file = open(new_ips_filename[:-5], 'w')
for i in range(len(self.new_ips)):
new_ips_file.write(self.new_ips[i] + ':' + self.new_hostnames[i] + '\n')
new_ips_file.close()
self.pcap is the path to the .pcap (which would just be ./file.pcap if it were in the same dir), is there any easy way to pull the dir from the filename here?

Are you looking for the absolute path of the file?
You might try os.path.abspath("file.pcap")
Or if you want only the directory name os.path.dirname(..)

Regarding file io in Python

I mistakenly, typed the following code:
f = open('\TestFiles\'sample.txt', 'w')
f.write('I just wrote this line')
f.close()
I ran this code and even though I have mistakenly typed the above code, it is however a valid code because the second backslash ignores the single-quote and what I should get, according to my knowledge is a .txt file named "\TestFiles'sample" in my project folder. However when I navigated to the project folder, I could not find a file there.
However, if I do the same thing with a different filename for example. Like,
f = open('sample1.txt', 'w')
f.write('test')
f.close()
I find the 'sample.txt' file created in my folder. Is there a reason for the file to not being created even though the first code was valid according to my knowledge?
Also is there a way to mention a file relative to my project folder rather than mentioning the absolute path to a file? (For example I want to create a file called 'sample.txt' in a folder called 'TestFiles' inside my project folder. So without mentioning the absolute path to TestFiles folder, is there a way to mention the path to TestFiles folder relative to the project folder in Python when opening files?)
I am a beginner in Python and I hope someone could help me.
Thank you.

What you're looking for are relative paths, long story short, if you want to create a file called 'sample.txt' in a folder 'TestFiles' inside your project folder, you can do:
import os
f = open(os.path.join('TestFiles', 'sample1.txt'), 'w')
f.write('test')
f.close()
Or using the more recent pathlib module:
from pathlib import Path
f = open(Path('TestFiles', 'sample1.txt'), 'w')
f.write('test')
f.close()
But you need to keep in mind that it depends on where you started your Python interpreter (which is probably why you're not able to find "\TestFiles'sample" in your project folder, it's created elsewhere), to make sure everything works fine, you can do something like this instead:
from pathlib import Path
sample_path = Path(Path(__file__).parent, 'TestFiles', 'sample1.txt')
with open(sample_path, "w") as f:
f.write('test')
By using a [context manager]{https://book.pythontips.com/en/latest/context_managers.html} you can avoid using f.close()

When you create a file you can specify either an absolute filename or a relative filename.
If you start the file path with '\' (on Win) or '/' it will be an absolute path. So in your first case you specified an absolute path, which is in fact:
from pathlib import Path
Path('\Testfile\'sample.txt').absolute()
WindowsPath("C:/Testfile'sample.txt")
Whenever you run some code in python, the relative paths that will be generate will be composed by your current folder, which is the folder from which you started the python interpreter, which you can check with:
import os
os.getcwd()
and the relative path that you added afterwards, so if you specify:
Path('Testfiles\sample.txt').absolute()
WindowsPath('C:/Users/user/Testfiles/sample.txt')
In general I suggest you use pathlib to handle paths. That makes it safer and cross platform. For example let's say that your scrip is under:
project
src
script.py
testfiles
and you want to store/read a file in project/testfiles. What you can do is get the path for script.py with __file__ and build the path to project/testfiles
from pathlib import Path
src_path = Path(__file__)
testfiles_path = src_path.parent / 'testfiles'
sample_fname = testfiles_path / 'sample.txt'
with sample_fname.open('w') as f:
f.write('yo')

As I am running the first code example in vscode, I'm getting a warning
Anomalous backslash in string: '\T'. String constant might be missing an r prefix.
And when I am running the file, it is also creating a file with the name \TestFiles'sample.txt. And it is being created in the same directory where the .py file is.
now, if your working tree is like this:
project_folder
-testfiles
-sample.txt
-something.py
then you can just say: open("testfiles//hello.txt")
I hope you find it helpful.

python script expected an indent block in if statement

I'm trying to write a basic backup script from one folder to another, and I got it to work - but the directory structure was not being copied over, just the files. I'm trying to copy in the subfolder as well, so that, for example, c:\temp\docs\file.txt goes to d:\temp\docs\file.txt instead of just d:\temp\file.txt
My issue exists in indentation with my if/else statement, but everything looks good to me. What am I doing wrong?
import datetime, time, string, os, shutil
COPY_FROM_LOCATION = 'C:\\xampp\\htdocs\\projects'
folder_date = time.strftime("%Y-%m-%d")
BACKUP_TO_LOCATION = 'D:\\BACKUP\\' + folder_date
#Create a new directory in D:\BACKUP based on today's date so the folder you're trying to copy to actually exists:
if not os.path.exists(BACKUP_TO_LOCATION):
os.makedirs(BACKUP_TO_LOCATION)
#copy function
def backup(source_folder, target_folder):
for subdir, dirs, files in os.walk(source_folder):
if subdir == source_folder :
new_target_folder = target_folder
else:
folder_name = subdir.split("C:\\xampp\\htdocs\\projects\\",1)[-1]
new_target_folder = target_folder + "\\" + folder_name
for file in files:
print "backing up: " + folder_name
shutil.copy2(os.path.join(subdir, file), new_target_folder)
backup(COPY_FROM_LOCATION,BACKUP_TO_LOCATION)
Here's the error I'm getting:
File "backup.py", line 15
new_target_folder = target_folder
^
IndentationError: expected an indented block

You're intermixing tabs and spaces.
Use one or the other, not both. Preferably spaces.

This error typically means there is an error in indentation. Check you don't mix tabs and spaces.
You can use https://www.pylint.org/ to detect them or if it something simple paste the code at http://pep8online.com, it will show you what you can enhance.

what's up with the weirdness with the space before the semi-colon? I've not seen it done that way before, that appears to be where this script is choking up.
change
if subdir == source_folder :
with
if subdir == source_folder:

How does one rename multiple files using python?

I have lots of programming experience but this is my first python script. I am trying to add the prefix "00" to all the files in a specific folder. First I read the names of all the files and save them in an array. Then I sort through the array and add the prefix "00" then use the os.rename function but somewhere along the way I've messed up something.
import sys, os
file_list = []
for file in os.listdir(sys.argv[1]):
file_list.append(file)
for i in file_list:
file_list[i] = prevName
newName = '00' + file_list[i]
os.rename(prevName, newName)
I have a .py file in the folder with all the files I want to rename. The .py file contains the script above. When i double click the .py file a cmd window flashes and disappears and none of the file names have been changed. Any help would be appreciated, sorry if this is a very obvious mistake, my python level is quite n00b at the moment.

In addition to the answer by #Padraic, also make following changes to your code.
import sys, os
file_list = []
for f in os.listdir(sys.argv[1]):
file_list.append(f)
for i in range(len(file_list)):
prevName = file_list[i]
if prevName != 'stackoverflow.py': # Mention .py file so that it doesnt get renamed
newName = '00' + file_list[i]
os.rename(prevName, newName)

Check your indentation. The second for loop is not indented correctly.
for i in file_list:
file_list[i] = prevName
You are not iterating correctly. for loops in Python are like foreach loops you may know from other programming languages. i in for i in file_list actually gives you the list's elements, so you should be doing
for i in range(len(file_list)):
file_list[i] = ......
although it is not very pythonic nor generally a good idea to modify the collection that you're currently iterating over.

Your code errors because you provide no args so sys.argv[1] would give an IndexError, you would need to call the script with the dir name from a cmd prompt not double click it:
python your_script directory <- argv[1]
Or change the code and specify the path, you also need to join the path to the filename.
path = "full_path"
for f in os.listdir(path):
curr,new = os.path.join(path,f), os.path.join(path,"00{}".format(f))
os.rename(curr,new)
os.listdir returns a list so just iterate over that, you don't need to create a list and append to it.
for i in file_list: would also make each i a filename not an index so that would cause another error but as above you don't need to do it anyway.

Python File System Reader Performance

I need to scan a file system for a list of files, and log those who don't exist. Currently I have an input file with a list of the 13 million files which need to be investigated. This script needs to be run from a remote location, as I do not have access/cannot run scripts directly on the storage server.
My current approach works, but is relatively slow. I'm still fairly new to Python, so I'm looking for tips on speeding things up.
import sys,os
from pz import padZero #prepends 0's to string until desired length
output = open('./out.txt', 'w')
input = open('./in.txt', 'r')
rootPath = '\\\\server\share\' #UNC path to storage
for ifid in input:
ifid = padZero(str(ifid)[:-1], 8) #extracts/formats fileName
dir = padZero(str(ifid)[:-3], 5) #exracts/formats the directory containing the file
fPath = rootPath + '\\' + dir + '\\' + ifid + '.tif'
try:
size = os.path.getsize(fPath) #don't actually need size, better approach?
except:
output.write(ifid+'\n')
Thanks.

dirs = collections.defaultdict(set)
for file_path in input:
file_path = file_path.rjust(8, "0")
dir, name = file_path[:-3], file_path
dirs[dir].add(name)
for dir, files in dirs.iteritems():
for missing_file in files - set(glob.glob("*.tif")):
print missing_file
Explanation
First read the input file into a dictionary of directory: filename. Then for each directory, list all the TIFF files in that directory on the server, and (set) subtract this from the collection of filenames you should have. Print anything that's left.
EDIT: Fixed silly things. Too late at night when I wrote this!

That padZero and string concatenation stuff looks to me like it would take a good percent of time.
What you want it to do is spend all its time reading the directory, very little else.
Do you have to do it in python? I've done similar stuff in C and C++. Java should be pretty good too.

You're going to be I/O bound, especially on a network, so any changes you can make to your script will result in very minimal speedups, but off the top of my head:
import os
input, output = open("in.txt"), open("out.txt", "w")
root = r'\\server\share'
for fid in input:
fid = fid.strip().rjust(8, "0")
dir = fid[:-3] # no need to re-pad
path = os.path.join(root, dir, fid + ".tif")
if not os.path.isfile(path):
output.write(fid + "\n")
I don't really expect that to be any faster, but it is arguably easier to read.
Other approaches may be faster. For example, if you expect to touch most of the files, you could just pull a complete recursive directory listing from the server, convert it to a Python set(), and check for membership in that rather than hitting the server for many small requests. I will leave the code as an exercise...

I would probably use a shell command to get the full listing of files in all directories and subdirectories in one hit. Hopefully this will minimise the amount of requests you need to make to the server.
You can get a listing of the remote server's files by doing something like:
Linux: mount the shared drive as /shared/directory/ and then do ls -R /shared/directory > ~/remote_file_list.txt
Windows: Use Map Network Drive to mount the shared drive as drive letter X:, then do dir /S X:/shared_directory > C:/remote_file_list.txt
Use the same methods to create a listing of your local folder's contents as local_file_list.txt. You python script will then reduce to an exercise in text processing.
Note: I did actually have to do this at work.