We Keep Coding

Python - how to re-export a file as utf-8

I have csv & excel files that were not correctly saved as UTF-8 so i cannot simply load them into pandas. Manually, I can open it and save as excel or csv and select utf-8 and then it works fine in pandas but I have too many files to do this manually and I don't want to replace the raw file (so overwriting it is out of the question). How can I accomplish this programmatically?
I thought of one solution could be to do something like this:
import pandas as pd
with open('path/to/bad_file.csv', 'rb') as f:
text = f.read()
with open('fixed-temp.csv', 'w', encoding='utf8') as f:
f.write(text.decode(encoding="latin-1"))
df = pd.read_csv('fixed-temp.csv')
But this leaves behind a temporary file or a new file that i don't want. I guess I could write more code to then delete this temporary file but that seems unclean and I'd rather encapsulate all this into one convenience function.

Need a push to start with a function about text files, I can't figure this out on my own

I don't need the entire code but I want a push to help me on the way, I've been searching on the internet for clues on how to start to write a function like this but I haven't gotten any further then just the name of the function.
So I haven't got the slightest clue on how to start with this, I don't know how to work with text files. Any tips?

These text files are CSV (Comma Separated Values). It is a simple file format used to store tabular data.
You may explore Python's inbuilt module called csv.
Following code snippet an example to load .csv file in Python:
import csv
filename = 'us_population.csv'
with open(filename, 'r') as csvfile:
csvreader = csv.reader(csvfile)

Writing a Semicolon containing String to CSV with "," as Delimiter still causes Separation in CSV-File

I want to write strings that include semicolons to CSV.
This is what I have:
name = ["Peter;Parker"]
file = open("Test.csv", "a", newline='')
writer = csv.writer(file, delimiter=',')
writer.writerow(name)
So the problem is, I want the complete string name in column 1, but in fact the semicolon splits the part before and after into column 1 and column 2

So the comment from #snakecharmerb was correct.
In my case I was trying to open the csv file with Excel and Excel has Semicolon as a default delimiter.
Instead of double clicking the csv file to open Excel you can check if the raw text is correct by editing the file via notepad.
If you want to open it with Excel you need to load an empty excel file and go to Data and import from Text. Now load the csv file and you are asked which delimiter you want to use, no you can unselect semicolon and choose none or just ,

Saving data from python to excel file as CSV UTF-8 file format

I have been trying to save the data as a excel file as a type of CSV UTF-8 (Comma delimited) (*.csv) which is different then the normal
CSV (Comma delimited) (*.csv) file. It display the unicode text when opened in excel. I can save as that file easily from excel but from python i am only able to save it as normal csv. Which will not cause loss of data but when opened it shows this kind of text "à¤à¤‰à¤Ÿà¤¾" instead of "एउटा" this text.
If I copied the text opening it with notepad to the excel file and then manually save the file as CSV UTF-8 then it preserves the correct display. But doing so is time consuming since all values appear in same line in notepad and i have to separate it in excel file.
So i just want to know how can i save data as CSV UTF-8 format of excel using python.
I have tried the follwing code but it results in normal csv file.
import codecs
import unicodecsv as csv
input_text = codecs.open('input.txt', encoding='utf-8')
all_text = input_text.read()
text_list = all_text.split()
output_list = [['Words','Tags']]
for input_word in text_list:
word_tag_list = [input_word,'O']
output_list.append(word_tag_list)
with codecs.open("output.csv", "wb") as f:
writer = csv.writer(f)
writer.writerows(output_list)

You need to indicate to Excel that this is a UTF-8 file. Unfortunately the only way to do this is by prepending a special byte sequence to the front of the file. Python will do this automatically if you use a special encoding.
with codecs.open("output.csv", "w", "encoding="utf_8_sig") as f:

I have found the answer. The encoding="utf_8_sig" should be given to csv.writer method to write the excel file as CSV UTF-8 file. Previous code can be witten as:
with open("output.csv", "wb") as f:
writer = csv.writer(f, dialect='excel', encoding='utf_8_sig')
writer.writerows(output_list)
However there was problem when data has , at the end Eg: "भने," For this case i didn't need the comma so i removed it with following code within the for loop.
import re
if re.search(r'.,$',input_word):
input_word = re.sub(',$','',input_word)
Finally I was able to obtain the output as desired with Unicode character correctly displayed and removing extra comma which is present at the end of data. So, if anyone know how to ignore comma at the end of data in excel file then you can comment here. Thanks.

Exporting nested lists to an excel file in Python

So I have a list of lists.
For example:
l = [[1,2,3],['a','b','c'],['a,b','d,c','a,e']]
I am having trouble exporting this properly because I need the two letters in the same cell, so if I use commas as a delimiter, it separates them. I have attached a photo of what i would want it to look like. I have already tried a few things but I am not sure how to do this. Any help would be great.

Export it as CSV first, which you can then open and save as a normal Excel workbook.
You use the built-in csv module to do that:
import csv
with open(file_name, 'w') as f:
fc = csv.writer(f, lineterminator='\n')
fc.writerows(l)