I'm working on python/nltk with (OMW) wordnet specifically for The Arabic language. All the functions work fine with the English language yet I can't seem to be able to perform any of them when I use the 'arb' tag. The only thing that works great is extracting the lemma_names from a given Arabic synset.
The code below works fine with u'arb':
The output is a list of Arabic lemmas.
for synset in wn.synsets(u'عام',lang=('arb')):
for lemma in synset.lemma_names(u'arb'):
print lemma
When I try to perform the same logic as the code above with synset, definitions, example, hypernyms, I get an error which says:
TypeError: hyponyms() takes exactly 1 argument (2 given)
(if I supply the 'arb' flag) or
KeyError: u'arb'
This is one of the codes that will not work if I write synset.hyponyms(u'arb'):
for synset in wn.synsets(u'عام',lang=('arb')):
for hypo in synset.hyponyms(): #print the hyponyms in English not Arabic
print hypo
Does this mean that I can't get to use wn.all_synsets and other built-in functions to extract all the Arabic synsets, hypernyms, etc?
The nltk's Open Multilingual Wordnet has English names for all the synsets, since it is a multilingual database centered on the original English Wordnet. Synsets model meanings, hence they are language-independent and cannot be requested in a specific language. But each synset is linked to lemmas for the languages covered by the OMW. Once you have some synsets (original, hyponyms, etc.), just ask for the Arabic lemmas again:
>>> for synset in wn.synsets(u'عام',lang=('arb')):
... for hypo in synset.hyponyms():
... for lemma in hypo.lemmas("arb"):
... print(lemma)
...
Lemma('waft.v.01.إِنْبعث')
Lemma('waft.v.01.انبعث')
Lemma('waft.v.01.إنبعث_كالرائحة_العطرة')
Lemma('waft.v.01.إِنْدفع')
Lemma('waft.v.01.إِنْطلق')
Lemma('waft.v.01.انطلق')
Lemma('waft.v.01.حمل_بخفة')
Lemma('waft.v.01.دفع')
Lemma('calendar_year.n.01.سنة_شمْسِيّة')
Lemma('calendar_year.n.01.سنة_مدنِيّة')
Lemma('fiscal_year.n.01.سنة_ضرِيبِيّة')
Lemma('fiscal_year.n.01.سنة_مالِيّة')
In other words, the lemmas are multilingual, the synsets are not.
Related
I am new to NLP, NLTK and Python. I am using wordnet to get the synonyms for a word in given sentence. I am using the below code to get the synonyms and lemma names of those words
synonyms = wordnet.synsets(w,pos)
lemmas.append(list( set(chain.from_iterable([w.lemma_names() for w in synonyms]))))
eg : wordnet.synsets("get",'v')
The lemma_names for this word "get" returns many things which are irrelevant for me.
My search string is "error getting the report". lemma_names has even "buzz off", "gets under one's skin" which are not correct for my statement.
So is there a way get synonyms which are relevant to the statement? is there any concept or algorithms that I can check for?
I'm starting to program with NLTK in Python for Natural Italian Language processing. I've seen some simple examples of the WordNet Library that has a nice set of SynSet that permits you to navigate from a word (for example: "dog") to his synonyms and his antonyms, his hyponyms and hypernyms and so on...
My question is:
If I start with an italian word (for example:"cane" - that means "dog") is there a way to navigate between synonyms, antonyms, hyponyms... for the italian word as you do for the english one? Or... There is an Equivalent to WordNet for the Italian Language ?
Thanks in advance
You are in luck. The nltk provides an interface to the Open Multilingual Wordnet, which does indeed include Italian among the languages it describes. Just add an argument specifying the desired language to the usual wordnet functions, e.g.:
>>> cane_lemmas = wn.lemmas("cane", lang="ita")
>>> print(cane_lemmas)
[Lemma('dog.n.01.cane'), Lemma('cramp.n.02.cane'), Lemma('hammer.n.01.cane'),
Lemma('bad_person.n.01.cane'), Lemma('incompetent.n.01.cane')]
The synsets have English names, because they are integrated with the English wordnet. But you can navigate the web of meanings and extract the Italian lemmas for any synset you want:
>>> hypernyms = cane_lemmas[0].synset().hypernyms()
>>> print(hypernyms)
[Synset('canine.n.02'), Synset('domestic_animal.n.01')]
>>> print(hypernyms[1].lemmas(lang="ita"))
[Lemma('domestic_animal.n.01.animale_addomesticato'),
Lemma('domestic_animal.n.01.animale_domestico')]
Or since you mentioned "cattiva_persona" in the comments:
>>> wn.lemmas("bad_person")[0].synset().lemmas(lang="ita")
[Lemma('bad_person.n.01.cane'), Lemma('bad_person.n.01.cattivo')]
I went from the English lemma to the language-independent synset to the Italian lemmas.
Since I found myself wondering how to actually use the wordnet resources after reading this question and its answer, I'm going to leave here some useful information:
Here is a link to the nltk guide.
The two necessary commands to download wordnet data and thus proceed with the usage explained in the other answer are:
import nltk
nltk.download('wordnet')
nltk.download('omw')
I am trying to get the synonyms for arabic words in a sentence
If the word is in English it works perfectly, and the results are displayed in Arabic language, I was wondering if its possible to get the synonym of an Arabic word right away without writing it in english first.
I tried that but it didn't work & I would prefer without tashkeel انتظار instead of اِنْتِظار
from nltk.corpus import wordnet as omw
jan = omw.synsets('انتظار ')[0]
print(jan)
print(jan.lemma_names(lang='arb'))
Wordnet used in nltk doesnt support arabic. If you are looking for Arabic Wordnet so this is a totally different thing.
For Arabic wordnet, download:
http://nlp.lsi.upc.edu/awn/get_bd.php
http://nlp.lsi.upc.edu/awn/AWNDatabaseManagement.py.gz
You run it with:
$ python AWNDatabaseManagement.py -i upc_db.xml
Now to get something like wn.synset('إنتظار'). Arabic Wordnet has a function wn.get_synsets_from_word(word), but it gives offsets. Also it accepts the words only as vocalized in the database. For example, you should use جَمِيل for جميل:
>> wn.get_synsets_from_word(u"جَمِيل")
[(u'a', u'300218842')]
300218842 is the offset of the synset of جميل .
I checked for the word إنتظار and seems it doesn't exist in AWN.
More details about using AWN to get synonyms here.
Like this question, I am interested in getting a large list of words by part of speech (a long list of nouns; a list of adjectives) to be used programmatically elsewhere. This answer has a solution using the WordNet database (in SQL) format.
Is there a way to get at such list using the corpora/tools built into the Python NLTK. I could take a large bunch of text, parse it and then store the nouns and adjectives. But given the dictionaries and other tools built in, is there a smarter way to simply extract the words that are already present in the NLTK datasets, encoded as nouns/adjectives (whatever)?
Thanks.
It's worth noting that Wordnet is actually one of the corpora included in the NLTK downloader by default. So you could conceivably just use the solution you already found without having to reinvent any wheels.
For instance, you could just do something like this to get all noun synsets:
from nltk.corpus import wordnet as wn
for synset in list(wn.all_synsets('n')):
print synset
# Or, equivalently
for synset in list(wn.all_synsets(wn.NOUN)):
print synset
That example will give you every noun that you want and it will even group them into their synsets so you can try to be sure that they're being used in the correct context.
If you want to get them all into a list you can do something like the following (though this will vary quite a bit based on how you want to use the words and synsets):
all_nouns = []
for synset in wn.all_synsets('n'):
all_nouns.extend(synset.lemma_names())
Or as a one-liner:
all_nouns = [word for synset in wn.all_synsets('n') for word in synset.lemma_names()]
You should use the Moby Parts of Speech Project data. Don't be fixated on using only what is directly in NLTK by default. It would be little work to download the files for this and pretty easy to parse them with NLTK once loaded.
I saw a similar question earlier this week (can't find the link), but like I said then, I don't think maintaining a list of nouns/adjectives/whatever is a great idea. This is primarily because the same word can have different parts of speech, depending on the context.
However, if you are still dead set on using these lists, then here's how I would do it (I don't have a working NLTK install on this machine, but I remember the basics):
nouns = set()
for sentence in my_corpus.sents():
# each sentence is either a list of words or a list of (word, POS tag) tuples
for word, pos in nltk.pos_tag(sentence): # remove the call to nltk.pos_tag if `sentence` is a list of tuples as described above
if pos in ['NN', "NNP"]: # feel free to add any other noun tags
nouns.add(word)
Hope this helps
I am trying to do lemmatization on words with NLTK.
What I can find now is that I can use the stem package to get some results like transform "cars" to "car" and "women" to "woman", however I cannot do lemmatization on some words with affixes like "acknowledgement".
When using WordNetLemmatizer() on "acknowledgement", it returns "acknowledgement" and using .PorterStemmer(), it returns "acknowledg" rather than "acknowledge".
Can anyone tell me how to eliminate the affixes of words?
Say, when input is "acknowledgement", the output to be "acknowledge"
Lemmatization does not (and should not) return "acknowledge" for "acknowledgement". The former is a verb, while the latter is a noun. Porter's stemming algorithm, on the other hand, simply uses a fixed set of rules. So, your only way there is to change the rules at source. (NOT the right way to fix your problem).
What you are looking for is the derivationally related form of "acknowledgement", and for this, your best source is WordNet. You can check this online on WordNet.
There are quite a few WordNet-based libraries that you can use for this (e.g. in JWNL in Java). In Python, NLTK should be able to get the derivationally related form you saw online:
from nltk.corpus import wordnet as wn
acknowledgment_synset = wn.synset('acknowledgement.n.01')
acknowledgment_lemma = acknowledgment_synset.lemmas[1]
print(acknowledgment_lemma.derivationally_related_forms())
# [Lemma('admit.v.01.acknowledge'), Lemma('acknowledge.v.06.acknowledge')]