We Keep Coding

I'm using os and pandas to modify multiple excel spreadsheets by iterating through a folder but ~$ keeps getting added to the file name

My initial code is here:
import pandas as pd
import os
directory_in_str = input('\n\nEnter the name of the folder you would like to use. If there are spaces, replace with underscores: ')
directory_in_str.strip()
directory = os.fsencode(directory_in_str)
user = input('\nEnter your first initial and last name as one word (ex: username): ')
user.strip()
path1 = '/Users/'
path2 = '/Desktop/DataScience/'
dspath = path1 + user + path2
slash = '/'
for file in os.listdir(directory):
filename = os.fsdecode(file)
if filename.endswith(".xls") or filename.endswith(".xlsx"):
print(directory)
pathname = dspath + directory_in_str + slash + filename
print(filename)
#Global = pd.read_excel(pathname, sheet_name=0)
Stats = pd.read_excel(pathname, sheet_name=1)
listorder = ['1', '2', '3']
Stats = Stats.reindex(columns=listorder)
Stats.to_excel(filename, sheet_name='Statistics', index=False)
continue
else:
continue
I've included the filename print statement to insure that the correct path is being used. However, the print statement happens twice.
These are the statements printed.
b'testrearrange'
Testname.xlsx
b'testrearrange'
~$Testname.xlsx
Why are the two characters '~$' added? The error originates from the line
Stats = pd.read_excel(pathname, sheet_name=1)
with the error
ValueError: File is not a recognized excel file
Does anyone know how to fix this?

I think the files starting with "~$# are temporary excel files that are created when you open the file in excel. One option is to close the file, in which case the temporary file is deleted. Other option is to change the logic by which you list the files to be read so that it ignores files that start with ~. I like to use glob for this
from glob import glob
path = "C:/Users/Wolf/[!~]*.xls*"
files = glob(path)
for file in files:
print("Do your thing here")

Creating and writing to a file using another files name

I'm new to Python and I'm trying to automate some of my work.
I need to create a .wst file (dictation data file) with the same name as it's corresponding .DS2 (dictation file) and then populate the .WST file with the data input (author code, job type).
Perhaps I need to generate a txt then change the extension to .wst? I'm not sure...
Nothing is being created when I run the following, can anyone offer any advice?
import os
print('Dictation Zipper 1.0\n')
print('**Warning** What you set in the following fields will apply to ALL dictations in the current folder, please make any manual adjustments after running the tool.\n')
get_directory = input('Enter the file path where the dictations are stored, please use a NEW folder outwith the Share...\n')
author_id = input('Enter the four digit author id...\n')
jobtype_id = input('Enter the job type...\n')
for f in os.listdir():
file_name, file_ext = os.path.splitext(f) #splitting file name and extension
wst_file = open(file_name + ".wst", "w+") #creating a wst file
wst_file.write("[JobParameters]\nAuthorId=" + author_id + "\nJobtypeId=" + jobtype_id +"\nPriority=NORMAL\nKeyfield=\nUserfield1=\nUserfield2=\nUserfield3=\nUserfield4=\nNotes=\n")
wst_file.close() #closing wst file

You need to give the directory name as an argument to os.listdir(). And you need to prepend the directory name to the filenames when opening the file.
for f in os.listdir(get_directory):
file_name, file_ext = os.path.splitext(f) #splitting file name and extension
path = os.path.join(get_directory, file_name + ".wst")
wst_file = open(path, "w+") #creating a wst file
wst_file.write("[JobParameters]\nAuthorId=" + author_id + "\nJobtypeId=" + jobtype_id +"\nPriority=NORMAL\nKeyfield=\nUserfield1=\nUserfield2=\nUserfield3=\nUserfield4=\nNotes=\n")
wst_file.close() #closing wst file

renaming the extracted file from zipfile

I have lots of zipped files on a Linux server and each file includes multiple text files.
what I want is to extract some of those text files, which have the same name across zipped files and save it a folder; I am creating one folder for each zipped file and extract the text file to it. I need to add the parent zipped folder name to the end of file names and save all text files in one directory. For example, if the zipped folder was March132017.zip and I extracted holding.txt, my filename would be holding_march13207.txt.
My problem is that I am not able to change the extracted file's name.
I would appreciate if you could advise.
import os
import sys
import zipfile
os.chdir("/feeds/lipper/emaxx")
pwkwd = "/feeds/lipper/emaxx"
for item in os.listdir(pwkwd): # loop through items in dir
if item.endswith(".zip"): # check for ".zip" extension
file_name = os.path.abspath(item) # get full path of files
fh = open(file_name, "rb")
zip_ref = zipfile.ZipFile(fh)
filelist = 'ISSUERS.TXT' , 'SECMAST.TXT' , 'FUND.TXT' , 'HOLDING.TXT'
for name in filelist :
try:
outpath = "/SCRATCH/emaxx" + "/" + os.path.splitext(item)[0]
zip_ref.extract(name, outpath)
except KeyError:
{}
fh.close()

import zipfile
zipdata = zipfile.ZipFile('somefile.zip')
zipinfos = zipdata.infolist()
# iterate through each file
for zipinfo in zipinfos:
# This will do the renaming
zipinfo.filename = do_something_to(zipinfo.filename)
zipdata.extract(zipinfo)
Reference:
https://bitdrop.st0w.com/2010/07/23/python-extracting-a-file-from-a-zip-file-with-a-different-name/

Why not just read the file in question and save it yourself instead of extracting? Something like:
import os
import zipfile
source_dir = "/feeds/lipper/emaxx" # folder with zip files
target_dir = "/SCRATCH/emaxx" # folder to save the extracted files
# Are you sure your files names are capitalized in your zip files?
filelist = ['ISSUERS.TXT', 'SECMAST.TXT', 'FUND.TXT', 'HOLDING.TXT']
for item in os.listdir(source_dir): # loop through items in dir
if item.endswith(".zip"): # check for ".zip" extension
file_path = os.path.join(source_dir, item) # get zip file path
with zipfile.ZipFile(file_path) as zf: # open the zip file
for target_file in filelist: # loop through the list of files to extract
if target_file in zf.namelist(): # check if the file exists in the archive
# generate the desired output name:
target_name = os.path.splitext(target_file)[0] + "_" + os.path.splitext(file_path)[0] + ".txt"
target_path = os.path.join(target_dir, target_name) # output path
with open(target_path, "w") as f: # open the output path for writing
f.write(zf.read(target_file)) # save the contents of the file in it
# next file from the list...
# next zip file...

You could simply run a rename after each file is extracted right? os.rename should do the trick.
zip_ref.extract(name, outpath)
parent_zip = os.path.basename(os.path.dirname(outpath)) + ".zip"
new_file_name = os.path.splitext(os.path.basename(name))[0] # just the filename
new_name_path = os.path.dirname(outpath) + os.sep + new_file_name + "_" + parent_zip
os.rename(outpath, new_namepath)
For the filename, if you want it to be incremental, simply start a count and for each file, go up by on.
count = 0
for file in files:
count += 1
# ... Do our file actions
new_file_name = original_file_name + "_" + str(count)
# ...
Or if you don't care about the end name you could always use something like a uuid.
import uuid
random_name = uuid.uuid4()

outpath = '/SCRATCH/emaxx'
suffix = os.path.splitext(item)[0]
for name in filelist :
index = zip_ref.namelist().find(name)
if index != -1: # check the file exists in the zipfile
filename, ext = os.path.splitext(name)
zip_ref.filelist[index].filename = f'{filename}_{suffix}.{ext}' # rename the extracting file to the suffix file name
zip_ref.extract(zip_ref.filelist[index], outpath) # use the renamed file descriptor to extract the file

I doubt this is possible to rename file during their extraction.
What about renaming files once they are extracted ?
Relying on linux bash, you can achieve it in a one line :
os.system("find "+outpath+" -name '*.txt' -exec echo mv {} `echo {} | sed s/.txt/"+zipName+".txt/` \;")
So, first we search all txt files in the specified folder, then exec the renaming command, with the new name computed by sed.
Code not tested, i'm on windows now ^^'

How to add characters to filename using glob in Python?

The following is a snippet of code. The script takes the input files from a "Test" folder, runs a function and then outputs the files with the same name in the "Results" folder (i.e. "Example_Layer.shp"). How could I set it so that the output file would instead read "Example_Layer(A).shp"?
#Set paths
path_dir = home + "\Desktop\Test\\"
path_res = path_dir + "Results\\"
def run():
#Set definitions
input = path_res + "/" + "input.shp"
output = path_res + "/" + fname
#Set current path to path_dir and search for only .shp files then run function
os.chdir(path_dir)
for fname in glob.glob("*.shp"):
run_function, input, output
run()

You currently calculate the output variable once (which IMO wouldn't work since you don't have any fname defined yet).
Move the statement where you compute the output variable within the for loop, like below:
#Set paths
path_dir = home + "\Desktop\Test\\"
path_res = path_dir + "Results\\"
def run():
#Set definitions
input = path_res + "/" + "input.shp"
#Set current path to path_dir and search for only .shp files then run function
os.chdir(path_dir)
for fname in glob.glob("*.shp"):
output = path_res + "/" + fname
run_function, input, output
run()

To answer your question:
How could I set it so that the output file would instead read "Example_Layer(A).shp"
You can use shutil.copy to copy the file to the new directory, adding a "(A)" to each file name using os.path.join to join the path and new filename:
path_dir = home + "\Desktop\Test\\"
path_res = path_dir + "Results\\"
import os
import shutil
def run():
os.chdir(path_dir)
for fname in glob.glob("*.shp"):
name,ex = fname.rsplit(".",1) # split on "." to rejoin later adding a ("A")
# use shutil.copy to copy the file after adding ("A")
shutil.copy(fname,os.path.join(path_res,"{}{}{}".format(name,"(A)",ex)))
# to move and rename in one step
#shutil.move(fname,os.path.join(path_res,"{}{}{}".format(name,"(A)",ex)))

error 00865! ArcMap geoprocessing. Raster to Polygon

I am trying to write a python script to process image files into shapefiles and then to buffer these files with a 5 meter buffer. I first made the script in model builder in arcmap but I am trying to run it for multiple image files, all beginning with the letters LG. I keep getting the error 00865, which states that the input raster (image file) does not exist!! I have checked the folder a million times and it definitely does exist! Here is my code:
# Import system modules
import sys, string, os, arcgisscripting
# Create the Geoprocessor object
gp = arcgisscripting.create()
# Load required toolboxes...
gp.AddToolbox("C:/Program Files/ArcGIS/ArcToolbox/Toolboxes/Conversion Tools.tbx")
gp.AddToolbox("C:/Program Files/ArcGIS/ArcToolbox/Toolboxes/Analysis Tools.tbx")
# Script arguments...
folder = "D:\\J04-0083\\IMAGEFILES"
for root, dirs, filenames in os.walk(folder): # returms root, dirs, and files
for filename in filenames:
filename_split = os.path.splitext(filename) # filename and extensionname (extension in [1])
filename_zero = filename_split[0]
try:
first_2_letters = filename_zero[0] + filename_zero[1]
except:
first_2_letters = "XX"
if first_2_letters == "LG":
Output_polygon_features = "D:\\J04-0083\\ShapeFiles.gdb\\" + "SH_" + filename + ".shp"
# Process: Raster to Polygon...
InRaster = filename_zero + ".png"
gp.RasterToPolygon_conversion(InRaster, Output_polygon_features, "SIMPLIFY", "VALUE") # FILL IN THE CORRECT VALUES!
Distance__value_or_field_ = "5 Meters"
Raster_Buffer_shp = "D:\\J04-0083\\ShapeFiles.gdb\\" + "SB_" + filename + ".shp"
# Process: Buffer...
gp.Buffer_analysis(Output_polygon_features, Raster_Buffer_shp, Distance__value_or_field_, "FULL", "ROUND", "NONE", "")
Does anyone have any idea why it doesn't work? thank you!

I don't know where you're running the script from, but when you call gp.RasterToPolygon_conversion, you're only giving it the file name, not the full path. If the file's not in the working directory, it won't find it. Try replacing the line:
InRaster = filename_zero + ".png"
With:
InRaster = os.path.join(root, filename_zero + ".png")