We Keep Coding

How to get pool.py to accept non ascii characters?

I am using Python 2.7.18
The idea is to use python to gather songs from specified directories, then create and run the commands to run them through a bunch of converters and sound processors.
Some of my songs have characters with accents and any song with a ? in the title gets changed to a ¿ (Inverted Question Mark) in the file name.
My convert_song function works correctly when ran, but when I try to run it in a Pool and the file name or directory has a non ascii character in it, it fails with:
Traceback (most recent call last):
File "C:\StreamLine.py", line 270, in <module>
result = pool.map(convert_song, qTheStack)
File "C:\Python27\lib\multiprocessing\pool.py", line 253, in map
return self.map_async(func, iterable, chunksize).get()
File "C:\Python27\lib\multiprocessing\pool.py", line 572, in get
raise self._value
UnicodeEncodeError: 'ascii' codec can't encode character u'\xbf' in position 27: ordinal not in range(128)
Here's my main where I set up the pool:
if __name__ == '__main__':
print('Reading artists.')
predir = 'G:\\Vault\\The Music\\'
artistfile = open('C:\\Controls\\ArtistList.txt', 'r')
artistlist = artistfile.readlines()
dirs = []
for artist in artistlist:
dirs.append(predir + artist.strip())
qTheStack = []
for currentPath in dirs:
for wFile in generate_next_file(currentPath):
print(repr(wFile))
#print(convert_song(wFile))
qTheStack.append(wFile)
print('List loaded.')
pool = Pool(12)
result = pool.map(convert_song, qTheStack)
for item in result:
print(item)
The print(repr(wFile)) looks like this when ran:
'G:\\Vault\\The Music\\Chicago\\1989 - Greatest Hits 1982-1989\\04 - Will You Still Love Me\xbf.flac'
'G:\\Vault\\The Music\\Chicago\\1989 - Greatest Hits 1982-1989\\06 - What Kind of Man Would I Be\xbf [Remix].flac'
How can I get the built-in Pool from multiprocessing to accept my input?

Change to Python 3, dude.
As much as I wanted there to be an answer that stayed on Python 2.7, I tried Python 3 and it didn't disappoint.
I did have to go back through the obscure steps I found to generate a file that will run a COM/DLL in Python, and I had to remove all the str.decode and encode calls throughout my script. After only one import change, I hit run and it ran as expected.

Python Error code in Notebook, [Errno 22] Invalid argument: '.....\test python.xlsx' [duplicate]

I'm trying to open a file in Python, but I got an error, and in the beginning of the string I got a /u202a character... Does anyone know how to remove it?
def carregar_uml(arquivo, variaveis):
cadastro_uml = {}
id_uml = 0
for i in open(arquivo):
linha = i.split(",")
carregar_uml("‪H:\\7 - Script\\teste.csv", variaveis)
OSError: [Errno 22] Invalid argument: '\u202aH:\7 - Script\teste.csv'

When you initially created your .py file, your text editor introduced a non-printing character.
Consider this line:
carregar_uml("‪H:\\7 - Script\\teste.csv", variaveis)
Let's carefully select the string, including the quotes, and copy-paste it into an interactive Python session:
$ python
Python 3.6.1 (default, Jul 25 2017, 12:45:09)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> "‪H:\\7 - Script\\teste.csv"
'\u202aH:\\7 - Script\\teste.csv'
>>>
As you can see, there is a character with codepoint U-202A immediately before the H.
As someone else pointed out, the character at codepoint U-202A is LEFT-TO-RIGHT EMBEDDING. Returning to our Python session:
>>> s = "‪H:\\7 - Script\\teste.csv"
>>> import unicodedata
>>> unicodedata.name(s[0])
'LEFT-TO-RIGHT EMBEDDING'
>>> unicodedata.name(s[1])
'LATIN CAPITAL LETTER H'
>>>
This further confirms that the first character in your string is not H, but the non-printing LEFT-TO-RIGHT EMBEDDING character.
I don't know what text editor you used to create your program. Even if I knew, I'm probably not an expert in that editor. Regardless, some text editor that you used inserted, unbeknownst to you, U+202A.
One solution is to use a text editor that won't insert that character, and/or will highlight non-printing characters. For example, in vim that line appears like so:
carregar_uml("<202a>H:\\7 - Script\\teste.csv", variaveis)
Using such an editor, simply delete the character between " and H.
carregar_uml("H:\\7 - Script\\teste.csv", variaveis)
Even though this line is visually identical to your original line, I have deleted the offending character. Using this line will avoid the OSError that you report.

you can use this sample code to remove u202a from file path
st="‪‪F:\\somepath\\filename.xlsx"
data = pd.read_excel(st)
if i try to do this it gives me a OSError and
In detail
Traceback (most recent call last):
File "F:\CodeRepo\PythonWorkSpace\demo\removepartofstring.py", line 14, in <module>
data = pd.read_excel(st)
File "C:\Users\Admin\AppData\Local\Programs\Python\Python37\lib\site-packages\pandas\util\_decorators.py", line 188, in wrapper
return func(*args, **kwargs)
File "C:\Users\Admin\AppData\Local\Programs\Python\Python37\lib\site-packages\pandas\util\_decorators.py", line 188, in wrapper
return func(*args, **kwargs)
File "C:\Users\Admin\AppData\Local\Programs\Python\Python37\lib\site-packages\pandas\io\excel.py", line 350, in read_excel
io = ExcelFile(io, engine=engine)
File "C:\Users\Admin\AppData\Local\Programs\Python\Python37\lib\site-packages\pandas\io\excel.py", line 653, in __init__
self._reader = self._engines[engine](self._io)
File "C:\Users\Admin\AppData\Local\Programs\Python\Python37\lib\site-packages\pandas\io\excel.py", line 424, in __init__
self.book = xlrd.open_workbook(filepath_or_buffer)
File "C:\Users\Admin\AppData\Local\Programs\Python\Python37\lib\site-packages\xlrd\__init__.py", line 111, in open_workbook
with open(filename, "rb") as f:
OSError: [Errno 22] Invalid argument: '\u202aF:\\somepath\\filename.xlsx'
but if i do that like this
st="‪‪F:\\somepath\\filename.xlsx"
data = pd.read_excel(st.strip("‪u202a")) #replace your string here
Its working for me

The problem is the directory path of the file is not read properly. Use raw strings to pass it as argument and it should work.
carregar_uml(r'H:\7 - Script\teste.csv', variaveis)

try strip(),
def carregar_uml(arquivo, variaveis):
cadastro_uml = {}
id_uml = 0
for i in open(arquivo):
linha = i.split(",")
carregar_uml("‪H:\\7 - Script\\teste.csv", variaveis)
carregar_uml = carregar_uml.strip("\u202a")

Or you can slice out that character
file_path = r"‪C:\Test3\Accessing_mdb.txt"
file_path = file_path[1:]
with open(file_path, 'a') as f_obj:
f_obj.write('some words')

use small letter when you write your hard-disk-drive name! not big letter!
ex) H: -> error
ex) h: -> not error

I tried all of the above solutions. Problem is when we copy path or any string from left to write, extra character is added . It does not show in our IDE. this extra added character denotes Right to Left mark (RLM)
https://en.wikipedia.org/wiki/Right-to-left_mark
, i.e. you selected the text at time of copying from Right to left.
check the image Linked to my answer.
I also did try copying left to right ,then this extra character is not added. So either type your path manually or copy it left to right to avoid this type of issue.

The following is a simple function to remove the "\u202a"and "\u202c" characters.
you can add any characters you want to be removed to the list.
def cleanup(inp):
new_char = ""
for char in inp:
if char not in ["\u202a", "\u202c"]:
new_char += char
return new_char
example = '\u202a7551\u202c'
print(cleanup(example)) # prints 7551

Python Chatterbot "Errno 22"

I'm trying to train a chatbot, and most of the data is in text files.
I pull:
Matt said you have a "shit load" of dining dollars. I have almost none so if you're willing to sell, I'm willing to buy.
from the text file, but when the chatterbot corpus tries to train the bot, it reads the above as:
'Matt said you have a "shit load" of dining dollars\\ I have almost none so if you\'re willing to sell, I\'m willing to buy\\\n'
How can I fix this?
This is my code:
def train_from_text():
#chatbot.set_trainer(ListTrainer)
directory = basedir + "Text Trainers"
files = find_files_in_directory(directory)
for file in files:
conversation = []
file_name = directory+"/"+file
with open(file_name, 'r') as to_read:
for line in to_read:
conversation.append(line)
chatbot.train(conversation)
Please excuse the swearing, its the data I was given.
Edit: Full error
Traceback (most recent call last):
File "E:/Jason Chatterbot/Jason Chat.py", line 102, in <module>
control()
File "E:/Jason Chatterbot/Jason Chat.py", line 96, in control
train_from_text()
File "E:/Jason Chatterbot/Jason Chat.py", line 58, in train_from_text
chatbot.train(conversation)
File "C:\Python27\lib\site-packages\chatterbot\trainers.py", line 119, in train
corpora = self.corpus.load_corpus(corpus_path)
File "C:\Python27\lib\site-packages\chatterbot_corpus\corpus.py", line 98, in load_corpus
corpus_data = self.read_corpus(file_path)
File "C:\Python27\lib\site-packages\chatterbot_corpus\corpus.py", line 63, in read_corpus
with io.open(file_name, encoding='utf-8') as data_file:
IOError: [Errno 22] Invalid argument: 'Matt said you have a "shit load" of dining dollars\\ I have almost none so if you\'re willing to sell, I\'m willing to buy\\\r\n'

Without looking at a larger subset of the data, seems like it's replacing single quotes (') with escaped single quotes (\'), actual newline characters, with escaped newlines (\n) and periods with double backslashes (\)
A simple string replace might fix it for you, depending on how bad the data is getting munged. Try changing
conversation.append(line)
to
conversation.append(line.replace("\\'","'").replace('\\\\','.').replace("\\n","\n"))
We're basically trying to reverse those substitutions that are being made automatically.

with open() throwing errors on a json file

So I'm super new to coding and I wanted to design a text based RPG as sort of a fun way to learn some stuff and I picked out the language Python because it was named after Monty Python. How perfect right? Well, that is what I thought until trying to get rooms to load.
I am using json files to store my room names, descriptions, and exits then trying to call them in python via a method I saw on YouTube, here is the code:
def getRoom(id):
ret = None
with open(str(id)+".json", "r") as f:
jsontext = f.read()
d = json.loads(jsontext)
d['id'] = id
ret = Room(**d)
This threw an IOError directory or file not found, so I added a try statement like so:
def getRoom(id):
ret = None
try:
with open(str(id)+".json", "r") as f:
jsontext = f.read()
d = json.loads(jsontext)
d['id'] = id
ret = Room(**d)
except IOError:
print("An error occured")
However now I am getting an "AttributeError: 'NoneType' object has no attribute 'name'" off my look command which I have coded like so:
def look(player, args):
print(player.loc.name)
print("")
print (player.loc.description)
In case this matters here is my json file that I have named 1.json:
{
"name": "A Small Bedroom",
"description": "The old bed room has probably seen many people over the years as the inn sits along a major trade route. The floor boards show wear and creak as you walk over them.",
"neighbors": {"w":2}
}
EDIT:
Full traceback:
Traceback (most recent call last):
File "game.py", line 79, in <module>
main(player) File "game.py", line 68, in main
player.loc = getRoom(1)
File "/home/illyduss/Urth/Locations/room.py", line 6, in getRoom
with open(str(id)+".json", "r") as f:
IOError: [Errno 2] No such file or directory: '1.json'

The error clearly says that the file is not to be found. Try the following.
1. make sure that the filename 1.json is available from where you are calling the python interpretor.
for example: if you are calling $ python game/game.py, then the file should be in the present working directory, not in game dir
Try using absolute paths if you can
import os
base_dir = /path/to/json/dir
filename = str(id)+".json"
abs_file = os.path.join(base_dir, filename)
with open(abs_file, "r"):
#do stuff
If you need the json files to be relative to the game.py file and still need the game file to be called from elsewhere, a good practice would be to define base_dir using __file__ attribute of the python file
base_dir = os.path.dirname(__file__)
The reason you're geting NoneType error is that somehow the loc variable is being set to None. which means that you are passing None to the Player's constructor. Since you haven't provided the code where you initialize player, I am assuming that you're passing the result of getRoom() as loc to the constructor. If that is the case, make sure that the value returned by getRoom is not None. you need an explicit return statement at the end of the function. return ret . by default any function without a return statement returns None. That could be your issue

Syntax error with json process ?

I have been working a solution for a game which creates stats for two different players then saves to a .txt, For some reason, unbeknownst to me a sytax error keeps on appearing, here ;
Traceback (most recent call last):
File "E:\CA2 solution.py", line 29, in <module>
json.dump(char_data,open("character_data.dat","wb"))
File "C:\Python33\lib\json\__init__.py", line 184, in dump
fp.write(chunk)
TypeError: 'str' does not support the buffer interface
Im not sure what is wrong but here is the code as well,
import random
char1=str(input('Please enter a name for character 1: '))
strh1=((random.randrange(1,4))//(random.randrange(1,12))+10)
skl1=((random.randrange(1,4))//(random.randrange(1,12))+10)
line = '%s has a strength value of %s and a skill value of %s'%(char1,strh1,skl1)
char2=str(input('Please enter a name for character 2: '))
strh2=((random.randrange(1,4))//(random.randrange(1,12))+10)
skl2=((random.randrange(1,4))//(random.randrange(1,12))+10)
line = '%s has a strength value of %s and a skill value of %s'%(char1,strh1,skl1)
char_data1 = {
"name":char1,
"STRENGTH":strh1,
"SKILL":skl1,
};
char_data2 = {
"name":char2,
"STRENGTH":strh2,
"SKILL":skl2,
};
char_data = [char_data1,char_data2]
import json
json.dump(char_data,open("character_data.dat","wb"))
char_data_loaded = json.load(open("character_data.dat"))
I don't know what is wrong, so if anyone does could they please help me and point it out and suggest a way to fix it ? Thanks !

You are giving a string input to a file opened in binary mode. json.dump serializes the char_data as a string as noted in the doc here
So open the file in w mode instead of wb and the error will be fixed.