I'm learning python from Think Python by Allen Downey and I'm stuck at Exercise 6 here. I wrote a solution to it, and at first look it seemed to be an improvement over the answer given here. But upon running both, I found that my solution took a whole day (~22 hours) to compute the answer, while the author's solution only took a couple seconds.
Could anyone tell me how the author's solution is so fast, when it iterates over a dictionary containing 113,812 words and applies a recursive function to each to compute a result?
My solution:
known_red = {'sprite': 6, 'a': 1, 'i': 1, '': 0} #Global dict of known reducible words, with their length as values
def compute_children(word):
"""Returns a list of all valid words that can be constructed from the word by removing one letter from the word"""
from dict_exercises import words_dict
wdict = words_dict() #Builds a dictionary containing all valid English words as keys
wdict['i'] = 'i'
wdict['a'] = 'a'
wdict[''] = ''
res = []
for i in range(len(word)):
child = word[:i] + word[i+1:]
if nword in wdict:
res.append(nword)
return res
def is_reducible(word):
"""Returns true if a word is reducible to ''. Recursively, a word is reducible if any of its children are reducible"""
if word in known_red:
return True
children = compute_children(word)
for child in children:
if is_reducible(child):
known_red[word] = len(word)
return True
return False
def longest_reducible():
"""Finds the longest reducible word in the dictionary"""
from dict_exercises import words_dict
wdict = words_dict()
reducibles = []
for word in wdict:
if 'i' in word or 'a' in word: #Word can only be reducible if it is reducible to either 'I' or 'a', since they are the only one-letter words possible
if word not in known_red and is_reducible(word):
known_red[word] = len(word)
for word, length in known_red.items():
reducibles.append((length, word))
reducibles.sort(reverse=True)
return reducibles[0][1]
wdict = words_dict() #Builds a dictionary containing all valid English words...
Presumably, this takes a while.
However, you regenerate this same, unchanging dictionary many times for every word you try to reduce. What a waste! If you make this dictionary once, and then re-use that dictionary for every word you try to reduce like you do for known_red, the computation time should be greatly reduced.
Related
I have been working on writing a Wordle bot, and wanted to see how it preforms with all 13,000 words. The problem is that I am running this through a for loop and it is very inefficient. After running it for 30 minutes, it only gets to around 5%. I could wait all that time, but it would end up being 10+ hours. There has got to be a more efficient way. I am new to python, so any suggestions would be greatly appreciated.
The code here is the code that is used to limit down the guesses each time. Would there be a way to search for a word that contains "a", "b", and "c"? Instead of running it 3 separate times. Right now the containts, nocontains, and isletter will each run every time I need to search for a new letter. Searching them all together would greatly reduce the time.
#Find the words that only match the criteria
def contains(letter, place):
list.clear()
for x in words:
if x not in removed:
if letter in x:
if letter == x[place]:
removed.append(x)
else:
list.append(x)
else:
removed.append(x)
def nocontains(letter):
list.clear()
for x in words:
if x not in removed:
if letter not in x:
list.append(x)
else:
removed.append(x)
def isletter(letter, place):
list.clear()
for x in words:
if x not in removed:
if letter == x[place]:
list.append(x)
else:
removed.append(x)
The performance problems can be massively reduced by using sets. Any time that you want to repeatedly test for membership (even only a few times), e.g. if x not in removed, you want to try to make a set. Lists require checking every element to find x, which is bad if the list has thousands of elements. In a Python set, if x not in removed should take as long to run if removed has 100 elements or 100,000, a small constant amount of time.
Besides this, you're running into problems by trying to use mutable global variables everywhere, like for list (which needs to be renamed) and removed. There's no benefit to doing that and several downsides, such as making it harder to reason about your code or optimize it. One benefit of Python is that you can pass large containers or objects to functions without any extra time or space cost: calling a function f(huge_list) is as fast and uses as much memory as f(tiny_list), as if you were passing by reference in other languages, so don't hesitate to use containers as function parameters or return types.
In summary, here's how your code could be refactored if you take away 'list' and 'removed' and instead store this as a set of possible words:
all_words = [] # Huge word list to read in from text file
current_possible_words = set(all_words)
def contains_only_elsewhere(possible_words, letter, place):
"""Given letter and place, remove from possible_words
all words containing letter but not at place"""
to_remove = {word for word in possible_words
if letter not in word or word[place] == letter}
return possible_words - to_remove
def must_not_contain(possible_words, letter):
"""Given a letter, remove from possible_words all words containing letter"""
to_remove = {word for word in possible_words
if letter in word}
return possible_words - to_remove
def exact_letter_match(possible_words, letter, place):
"""Given a letter and place, remove from possible_words
all words not containing letter at place"""
to_remove = {word for word in possible_words
if word[place] != letter}
return possible_words - to_remove
The outside code will be different: for example,
current_possible_words = exact_letter_match(current_possible_words, 'a', 2)`
Further optimizations are possible (and much easier now): storing only indices to words rather than the strings; precomputing, for each letter, the set of all words containing that letter, etc.
I just wrote a wordle bot that runs in about a second including the web scraping to fetch a list of 5 letter words.
import urllib.request
from bs4 import BeautifulSoup
def getwords():
source = "https://www.thefreedictionary.com/5-letter-words.htm"
filehandle = urllib.request.urlopen(source)
soup = BeautifulSoup(filehandle.read(), "html.parser")
wordslis = soup.findAll("li", {"data-f": "15"})
words = []
for k in wordslis:
words.append(k.getText())
return words
words = getwords()
def hasLetterAtPosition(letter,position,word):
return letter==word[position]
def hasLetterNotAtPosition(letter,position,word):
return letter in word[:position]+word[position+1:]
def doesNotHaveLetter(letter,word):
return not letter in word
lettersPositioned = [(0,"y")]
lettersMispositioned = [(0,"h")]
lettersNotHad = ["p"]
idx = 0
while idx<len(words):
eliminated = False
for criteria in lettersPositioned:
if not hasLetterAtPosition(criteria[1],criteria[0],words[idx]):
del words[idx]
eliminated = True
break
if eliminated:
continue
for criteria in lettersMispositioned:
if not hasLetterNotAtPosition(criteria[1],criteria[0],words[idx]):
del words[idx]
eliminated = True
break
if eliminated:
continue
for letter in lettersNotHad:
if not doesNotHaveLetter(letter,words[idx]):
del words[idx]
eliminated = True
break
if eliminated:
continue
idx+=1
print(words) # ["youth"]
The reason yours is slow is because you have a lot of calls to check if word in removed in addition to a number of superfluous logical conditions in addition to going through all the words for each of your checks.
Edit: Here's a get words function that gets more words.
def getwords():
source = "https://wordfind-com.translate.goog/length/5-letter-words/?_x_tr_sl=es&_x_tr_tl=en&_x_tr_hl=en&_x_tr_pto=wapp"
filehandle = urllib.request.urlopen(source)
soup = BeautifulSoup(filehandle.read(), "html.parser")
wordslis = soup.findAll("a", {"rel": "nofollow"})
words = []
for k in wordslis:
words.append(k.getText())
return words
I'm trying to create a function in Python that will generate anagrams of a given word. I'm not just looking for code that will rearrange the letters aimlessly. All the options given must be real words. I currently have a solution, which to be honest I took most of this code from a YouTube video, but it is very slow for my purpose and can only provide one word responses to a single word given. It uses a 400,000 word dictionary to compare the words it is going though, called "dict.txt".
My goal is to get this code to mimic how well this website's code works:
https://wordsmith.org/anagram/
I could not find the javascript code when reviewing the network activity using Google Chrome's developer tool, so I believe the code is probably in the background, and is possibly using Node.js. This would perhaps make it faster than Python, but given how much faster it is I believe there is more to it than just the programming language. I assume they are using some type of search algorithm rather than just going through each line one by one like I am. I also like the fact that their response is not limited to a single word, but can break up the word given to provide more options to the user. For example, an anagram of "anagram" is "nag a ram".
Any suggestions or ideas would be appreciated.
Thank you.
def init_words(filename):
words = {}
with open(filename) as f:
for line in f:
word = line.strip()
words[word] = 1
return words
def init_anagram_dict(words):
anagram_dict = {}
for word in words:
sorted_word = ''.join(sorted(list(word)))
if sorted_word not in anagram_dict:
anagram_dict[sorted_word] = []
anagram_dict[sorted_word].append(word)
return anagram_dict
def find_anagrams(word, anagram_dict):
key = ''.join(sorted(list(word)))
if key in anagram_dict:
return set(anagram_dict[key]).difference(set([word]))
return set([])
#This is the first function called.
def make_anagram(user_word):
x = str(user_word)
lower_user_word = str.lower(x)
word_dict = init_words('dict.txt')
result = find_anagrams(lower_user_word, init_anagram_dict(word_dict.keys()))
list_result = list(result)
count = len(list_result)
if count > 0:
random_num = random.randint(0,count -1)
anagram_value = list_result[random_num]
return ('An anagram of %s is %s. Would you like me to search for another word?' %(lower_user_word, anagram_value))
else:
return ("Sorry, I could not find an anagram for %s." %(lower_user_word))
You can build a dictionary of anagrams by grouping words by their sorted text. All words that have the same sorted text are anagrams of each other:
from collections import defaultdict
with open("/usr/share/dict/words","r") as wordFile:
words = wordFile.read().split("\n")
anagrams = defaultdict(list)
for word in words:
anagrams["".join(sorted(word))].append(word)
aWord = "spear"
result = anagrams["".join(sorted(aWord))]
print(aWord,result)
# ['asper', 'parse', 'prase', 'spaer', 'spare', 'spear']
Using 235,000 words, the response time is instantaneous
In order to obtain multiple words forming an anagram of the specified word, you will need to get into combinatorics. A recursive function is probably the easiest way to go about it:
from itertools import combinations,product
from collections import Counter,defaultdict
with open("/usr/share/dict/words","r") as wordFile:
words = wordFile.read().split("\n")
anagrams = defaultdict(set)
for word in words:
anagrams["".join(sorted(word))].add(word)
counters = { w:Counter(w) for w in anagrams }
minLen = 2 # minimum word length
def multigram(word,memo=dict()):
sWord = "".join(sorted(word))
if sWord in memo: return memo[sWord]
result = anagrams[sWord]
wordCounts = counters.get(sWord,Counter())
for size in range(minLen,len(word)-minLen+1):
seen = set()
for combo in combinations(word,size):
left = "".join(sorted(combo))
if left in seen or seen.add(left): continue
left = multigram(left,memo)
if not left: continue
right = multigram("".join((wordCounts-Counter(combo)).elements()),memo)
if not right: continue
result.update(a+" "+b for a,b in product(left,right) )
memo[sWord] = list(result)
return memo[sWord]
Performance is good up to 12 character words. Beyond that the exponential nature of combinations start to take a heavy toll
result = multigram("spear")
print(result)
# ['parse', 'asper', 'spear', 'er spa', 're spa', 'se rap', 'er sap', 'sa per', 're asp', 'ar pes', 'se par', 'pa ers', 're sap', 'er asp', 'as per', 'spare', 'spaer', 'as rep', 'sa rep', 'ra pes', 'pa ser', 'es rap', 'es par', 'prase']
len(multigram("mulberries")) # 15986 0.1 second 10 letters
len(multigram("raspberries")) # 60613 0.2 second 11 letters
len(multigram("strawberries")) # 374717 1.3 seconds 12 letters
len(multigram("tranquillizer")) # 711491 7.6 seconds 13 letters
len(multigram("communications")) # 10907666 52.2 seconds 14 letters
In order to avoid any delay, you can convert the function to an iterator. This will allows you to get the first few anagrams without having to generate them all:
def iMultigram(word,prefix=""):
sWord = "".join(sorted(word))
seen = set()
for anagram in anagrams.get(sWord,[]):
full = prefix+anagram
if full in seen or seen.add(full): continue
yield full
wordCounts = counters.get(sWord,Counter(word))
for size in reversed(range(minLen,len(word)-minLen+1)): # longest first
for combo in combinations(sWord,size):
left = "".join(sorted(combo))
if left in seen or seen.add(left): continue
for left in iMultigram(left,prefix):
right = "".join((wordCounts-Counter(combo)).elements())
for full in iMultigram(right,left+" "):
if full in seen or seen.add(full): continue
yield full
from itertools import islice
list(islice(iMultigram("communications"),5)) # 0.0 second
# ['communications', 'cinnamomic so ut', 'cinnamomic so tu', 'cinnamomic os ut', 'cinnamomic os tu']
I'm currently trying to generate a list of words that rhyme with an input word according to the CMU Pronouncing dictionary I have managed to arrange all the words into a dictionary with their keys being a list of strings representing their values. However, due to something rhyming based on the last vowel, I'm sort of stuck on finding how to go about this in the case of words that contain more than one
def dotheyrhyme(filename,word):
rhymes = {}
list = []
with open(filename) as f:
text = f.readlines()[56:]
for line in text:
splitline = line.split(" ")
rhymes[str(splitline[0])] = "".join(splitline[1:])
f.close()
comparer = rhymes[word.upper()].rstrip().split(" ")
return comparer
I plan to use the comparer variable as a baseline and believe reversing this variable could also be a good way to go about it but I'm lost or overthinking ways to compare if the last vowel and letters after are the same and append accordingly?
Example:
{SECOND: 'S' 'EH1' 'K' 'AH0' 'N' 'D'}
Would rhyme with
{'AND': 'AH0' 'N' 'D'}
but these two wouldn't rhyme
{'YELLOW': 'Y' 'EH1' 'L' 'OW0'}
And
{HELLO: 'HH' 'AH0' 'L' 'OW1'}
But the methods I can't think of ways to counter varying lengths and multiple vowels.
Thanks for your help!
Finding last vowel requires you to have a set of vowels. After that you only got to iterate over the list backwards.
vowels = {...} # some list of vowels
word = ['S', 'EH1', 'K', 'AH0', 'N', 'D']
for i in word[::-1]:
if i in vowels:
last_vowel = i
break
If open to other idea you can also look at this library which finds the rhymes for you : https://pypi.org/project/pronouncing/
You would have to start comparing from the end. There are special algorithms and data structures that can help in cases like yours - you can check Aho-Corasick algorithm.
But in the simple case, you would need to compare the words in the reverse order and find common substring above some threshold to call these words a rhyme, e.g.:
def if_rhymes(word1, word2):
r1 = reverse(rhymes[word2])
r2 = reverse(rhymes[word1])
the_same = 0
for sound1, sound2 in zip(r1, r2):
if sound1 == sound2:
the_same += 1
else:
break
if the_same < threshold:
return 'no rhyme' # or False if you want
else:
return 'rhymes' # or True
What the algorithm does
It takes the list of sounds from the rhymes dictionary that you populated from file (for clarity I recommend doing it outside the rhyme testing function).
Then it reverses the order of elements in lists of sounds for both words and creates a list of pairs (or tuples) using zip.
Each of the tuples (sounds from the words in the reverse order) is compared. We count the ones that are the same and stop comparing on the first different pair of sounds from the back.
Depending on the threshold (you may want to substitute the variable for an actual value) you consider given pair of words as a rhyme or not.
I'm trying to write an algorithm that by given to it a bunch of letters is giving you all the words that can be constructed of the letters, for instance, given 'car' should return a list contains [arc,car,a, etc...] and out of it returns the best scrabble word. The problem is in finding that list which contains all the words.
I've got a giant txt file dictionary, line delimited and I've tried this so far:
def find_optimal(bunch_of_letters: str):
words_to_check = []
c1 = Counter(bunch_of_letters.lower())
for word in load_words():
c2 = Counter(word.lower())
if c2 & c1 == c2:
words_to_check.append(word)
max_word = max_word_value(words_to_check)
return max_word,calc_word_value(max_word)
max_word_value - returns the word with the maximum value of the list given
calc_word_value - returns the word's score in scrabble.
load_words - return a list of the dictionary.
I'm currently using counters to do the trick but, the problem is that I'm currently on about 2.5 seconds per search and I don't know how to optimize this, any thoughts?
Try this:
def find_optimal(bunch_of_letters):
bunch_of_letters = ''.join(sorted(bunch_of_letters))
words_to_check = [word for word in load_words() if ''.join(sorted(word)) in bunch_of_letters]
max_word = max_word_value(words_to_check)
return max_word, calc_word_value(max_word)
I've just used (or at least tried to use) a list comprehension. Essentially, words_to_check will (hopefully!) be a list of all of the words which are in your text file.
On a side note, if you don't want to use a gigantic text file for the words, check out enchant!
from itertools import permutations
theword = 'car' # or we can use input('Type in a word: ')
mylist = [permutations(theword, i)for i in range(1, len(theword)+1)]
for generator in mylist:
for word in generator:
print(''.join(word))
# instead of .join just print (word) for tuple
Output:
c
a
r
ca
cr
...
ar rc ra car cra acr arc rca rac
This will give us all the possible combinations (i.e. permutations) of a word.
If you're looking to see if the generated word is an actual word in the English dictionary we can use This Answer
import enchant
d = enchant.Dict("en_US")
for word in mylist:
print(d.check(word), word)
Conclusion:
If want to generate all the combinations of the word. We use this code:
from itertools import combinations, permutations, product
word = 'word' # or we can use input('Type in a word: ')
solution = permutations(word, 4)
for i in solution:
print(''.join(i)) # just print(i) if you want a tuple
So I was exploring on coderbyte.com and one of the challenges is to find the longest word in a string. My code to do so is the following:
def LongestWord(sen):
current="";
currentBest=0
numberOfLettersInWord=0
longestWord=0
temp=sen.split()
for item in temp:
listOfCharacters=list(item)
for currentChar in listOfCharacters:
if currentChar.isalpha():
numberOfLettersInWord+=1
if numberOfLettersInWord>longestWord:
longestWord=numberOfLettersInWord
numberOfLettersInWord=0
currentBest=item
z = list(currentBest)
x=''
for item in z:
if item.isalpha(): x+=item
return x
testCase="a confusing /:sentence:/ this"
print LongestWord(testCase)
when testCase is "a confusing /:sentence:/"
The code returns confusing, which is the correct answer. But when the test case is the one in the current code, my code is returning 'this' instead of 'confusing'
Any ideas as to why this is happening?
I know that this is not the answer to your question, but this is how I would calculate the longest word. And not sharing it, wouldn't help you, either:
import re
def func(text: str) -> str:
words = re.findall(r"[\w]+", text)
return max(words, key=len)
print(func('a confusing /:sentence:/ this'))
Let me suggest another approach, which is more modular and more Pythonic.
Let's make a function to measure word length:
def word_length(w):
return sum(ch.isalpha() for ch in w)
So it will count (using sum()) how many characters there are for which .isalpha() is True:
>>> word_length('hello!!!')
5
>>> word_length('/:sentence:/')
8
Now, from a list of words, create a list of lengths. This is easily done with map():
>>> sen = 'a confusing /:sentence:/ this'.split()
>>> map(word_length, sen)
[1, 9, 8, 4]
Another builtin useful to find the maximum value in a list is max():
>>> max(map(word_length, sen))
9
But you want to know the word which maximizes the length, which in mathematical terms is called argument of the maximum.
To solve this, zip() the lengths with the words, and get the second argument found by max().
Since this is useful in many cases, make it a function:
def arg_max(func, values):
return max(zip(map(func, values), values))[1]
Now the longest word is easily found with:
>>> arg_max(word_length, sen)
'confusing'
Note: PEP-0008 (Style Guide for Python Code) suggests that function names be lower case and with words separated by underscore.
You loop through the words composing the sentence. However, numberOfLettersInWord is never reseted so it keeps increasing while you iterate among the words.
You have to set the counter to 0 each time you start a new word.
for item in temp:
numberOfLettersInWord = 0
It solves your issue as you can see: https://ideone.com/y1cmHX
Here's a little function I just wrote that will return the longest word, using a regular expression to remove non alpha-numeric characters
import re
def longest_word(input):
words = input.split()
longest = ''
for word in words:
word = re.sub(r'\W+', '', word)
if len(word) > len(longest):
longest = word
return longest
print(longest_word("a confusing /:sentence:/ this"))