We Keep Coding

Merge multiple files from server folder into one

I have tried to merge some files in server folder into a new file, saving under same server folder.
In my below script, I keep receiving unexpected indent error. I would like to seek some expert guidance.
import pandas as pd # import need to be in lower case
import numpy as np
import openpyxl
from openpyxl.workbook import workbook #save to excel doc
#>>> 1.1 Define common file path and filename
path= '\hbap.adroot.abb\HK\Finance\00210602\AMH_A2R\1KY\Drv Reengine\Python\'
#>>> 1.2 Define list of files
filenames = [path+'100_6.xlsx', path+'101_6.xlsx']
# Open file3 in write mode
with Open(r path+'file3.xlsx','w') as outfile:
# Iterate through list
for names in filenames:
# Open each file in read mode
with open(names) as infile:
# read the data from file1 and
# file2 and write it in file3
outfile.write(infile.read())
# Add '\n' to enter data of file2
# from next line
outfile.write("\n")

please take a look at proposed solution.
In this case I take all files present in /sql directory, read all of them one by one and append the result to the output file.
import os
files_list = list()
output = r"E:\Downloads\output.txt"
for (dirpath, dirnames, filenames) in os.walk(r'E:\Downloads\sql'):
files_list += [os.path.join(dirpath, file) for file in filenames]
for file in files_list:
fin = open(file, "rt")
data = fin.read()
fin.close()
fin = open(output, "a+")
fin.write(data)
fin.write("\n ---------- \n")
fin.close()
Also I might suggest that you are dealing with .xlsx files which is a bit different topic and merging excel files should be treated in another way.

Having a correct indentation is important in python as the interpreter uses it to read the code.
This has an ordered indent level:
# Open file3 in write mode
with Open(r path+'file3.xlsx','w') as outfile:
# Iterate through list
for names in filenames:
# Open each file in read mode
with open(names) as infile:
# read the data from file1 and
# file2 and write it in file3
outfile.write(infile.read())
# Add '\n' to enter data of file2
# from next line
outfile.write("\n")

how to write csv file into specific folder

i am trying to write several .csv file into one specific directory
here is my code
with open(f+'.csv', 'w', newline='') as csvfile:
writer = csv.writer(csvfile)
writer.writerow(["index", "B", "G", "R"])
for row in rows:
writer.writerow(row)
writer.writerow(["Mean", mean_b/total_b, mean_g/total_g, mean_r/total_r])
writer.writerow("STD", np.sqrt(var_b/total_b), np.sqrt(var_g/total_g), np.sqrt(var_r/total_r))
i have created the csv file into the directory which is same as the .py file
however i would like to create a directory and create my csv file in it
i know i need to us os.makedirs() function
but i don't know whether i have to create the directory first and designate the path for the csv file or i simply put the directory name into the open() function
please help me

Instead of using os I recommend using the pathlib module.
You can create a directory with:
path = Path('path/to/dir')
path.mkdir(parents=True)
to create the directory and all its missing parent dirs.
After doing this you can create a file in the new directory with
fpath = (path / 'filename').with_suffix('.csv')
with fpath.open(mode='w+') as csvfile:
# your csv writer code

I would simply create the directory and except directory exists error
try:
os.mkdir("./CSV")
except OSError as e:
print("Directory exists")
with open("./CSV/" + f + ".csv", newline="") as csvfile:
[...]

You can add a check for the directory like this just before open statement
dir_path = 'folder_to_save_csv_file_in'
if not os.path.isdir(dir_path):
os.makedirs(dir_path)
with open('{file_path}.csv'.format(file_path=os.path.join(dir_path, file_name), 'w+') as csv_file:
....

Creating folders from a single column in a csv and using other column to rename files into folders

I have a csv file that looks like this:
dc_identifier,aubrey_identifier
AR0776-280206-LT513-01,metadc1084267
AR0776-280206-LT513-02,metadc1083385
AR0776-280206-LT513-03,metadc1084185
AR0776-280206-LT513-04,metadc1083449
AR0776-280206-LT513-05,metadc1084294
AR0776-280206-LT513-06,metadc1083393
AR0776-280206-LT513-07,metadc1083604
AR0776-280206-LT513-08,metadc1083956
AR0776-280206-LT513-09,metadc1083223
AR0776-280206-LT513-10,metadc1084224
I need to create folders with the "metadc#######" names within the directory that the script will live in.
Here's what I have so far:
import os
import fileinput
path = 'C:\Users\gpp0020\Desktop\TestDir'
textFile = 'C:\Users\gpp0020\Desktop\TestDir\kxas_ids.csv'
myList = open(textFile, 'rb+')
for line in myList:
for item in line.strip().split(','):
os.makedirs(os.path.join(path, item))
print 'created', item
However! I also need the program to grab files that are named with the identifiers (AR0776-280206-LT513-01, etc) and put them in the corresponding metadc number, according to the csv. Each file is doubled (one .mkv file, and one .mkv.md5 checksum file) and both need to go into the folder.
What's the best way to go about this?

Use the csv library to help with reading the file in:
import csv
import os
import shutil
path = r'C:\Users\gpp0020\Desktop\TestDir'
with open('kxas_ids.csv', 'r', newline='') as f_input:
csv_input = csv.reader(f_input)
header = next(csv_input)
for dv, aubrey in csv_input:
os.makedirs(os.path.join(path, aubrey), exist_ok=True)
mkv = '{}.mkv'.format(dv)
shutil.copy2(os.path.join(path, mkv), os.path.join(path, aubrey, mkv))
mkv_md5 = '{}.mkv.md5'.format(dv)
shutil.copy2(os.path.join(path, mkv_md5), os.path.join(path, aubrey, mkv_md5))
This would for example:
Create a folder called C:\Users\gpp0020\Desktop\TestDir\metadc108426
Copy a file called AR0776-280206-LT513-01.mkv into it.
Copy a file called AR0776-280206-LT513-01.mkv.md5 into it.
It assumes that all files are found in path

error 2 No such file or directory

i'm trying to run python script which already ran on test environment.
already checked if the path correct and if the file in it.
checked in shell that the file exist.
current code is :
# Open a file
path = 'C:\\Users\\tzahi.k\\Desktop\\netzer\\'
dirs = os.listdir( path )
fileslst = []
alertsCode = (some data)
# loop over to search the relative file
for file in dirs:
if "ALERTS" in file.upper() :
fileslst.append(file)
fileslst.sort()
#open and modify the latest file
with open(fileslst[-1], 'rb') as csvfile:
csvReader = csv.reader(csvfile)
clean_rows = [row for row in csvReader if not any(alert in row[2] for alert in alertsCode)]
error :
IOError:error 2 no such file or directory:'file name'
when i debug in shell i see the path and files
what am i doing wrong?

os.listdir() lists the files relative to the directory.
You need to add the full directory path to the filename for it to be an absolute path again:
with open(os.path.join(path, fileslst[-1]), 'rb') as csvfile:

Reading/Writing Text from Multiple Files to Master File

In the code below I'm trying to open a series of text files and copy their contents into a single file. I'm getting an error on the "os.write(out_file, line)" in which it asks me for an integer. I haven't defined what "line" is, so is that the problem? Do I need to specify somehow that "line" is a text string from the in_file? Also, I open the out_file through each iteration of the for-loop. Is that bad? Should I open it once at the beginning? Thanks!
import os
import os.path
import shutil
# This is supposed to read through all the text files in a folder and
# copy the text inside to a master file.
# This defines the master file and gets the source directory
# for reading/writing the files in that directory to the master file.
src_dir = r'D:\Term Search'
out_file = r'D:\master.txt'
files = [(path, f) for path,_,file_list in os.walk(src_dir) for f in file_list]
# This for-loop should open each of the files in the source directory, write
# their content to the master file, and finally close the in_file.
for path, f_name in files:
open(out_file, 'a+')
in_file = open('%s/%s' % (path, f_name), 'r')
for line in in_file:
os.write(out_file, line)
close(file_name)
close(out_file)
print 'Finished'

You're doing it wrong:
You did:
open(out_file, 'a+')
but that doesn't save the reference as a variable, so you have no way to access the file object you just created. What you need to do:
out_file_handle = open(out_file, 'a+')
...
out_file_handle.write(line)
...
out_file_handle.close()
Or, more pythonically:
out_filename = r"D:\master.txt"
...
with open(out_filename, 'a+') as outfile:
for filepath in files:
with open(os.path.join(*filepath)) as infile:
outfile.write(infile.read())
print "finished"