We Keep Coding

How to look up information from a website with python

I want to know if its possible to write a code in python that will allow me to look up information from an online source and add it too my code as a dictionary. (I want to use this so I have a dictionary consisting of all the spells listed on the harry potter wiki as the key and their descriptions as associated values (https://harrypotter.fandom.com/wiki/List_of_spells))
I am beginning python student and really don't know how to start, I guess I could copy the information as a text file and manipulate it from there but I really want it too change should the online source change etc.

You can grab the wiki data here and parse it:
https://harrypotter.fandom.com/wiki/List_of_spells?action=edit
It looks like the spells follow the same format making it easy for parsing. They are separated by a new line so you can split the data by \n, and parse each line out. There seems to be two different type of spells, ones that start with '|' and others that start with ':', so you have to parse differently for the type. Does that help you get started?
===''[[Water-Making Spell|Aguamenti]]'' (Water-Making Spell)===
[[File:Aguamenti.gif|235px|thumb]]
{{spell sum
|t=Charm, Conjuration
|p=AH-gwah-MEN-tee
|d=Produces a clean, drinkable jet of water from the wand tip.
|sm=Used by [[Fleur Delacour]] in [[1994]] to extinguish her skirt, which had caught flame during a fight against a [[dragon]]. [[Harry Potter|Harry]] used this spell twice in [[1997]], both on the same night; once to attempt to provide a drink for [[Albus Dumbledore|Dumbledore]], then again to help douse [[Rubeus Hagrid|Hagrid]]'s hut after it was set aflame by [[Thorfinn Rowle]], who used the [[Fire-Making Spell]].
|e=Possibly a hybrid of [[Latin]] words ''aqua'', which means "water", and ''mentis'', which means "mind".}}
===''[[Alarte Ascendare]]''===
[[File:Alarte Ascendare.gif|250px|thumb]]
{{spell sum
|t=Charm
|p=a-LAR-tay a-SEN-der-ay
|d=Shoots the target high into the air.
|sm=Used by [[Gilderoy Lockhart]] in [[Harry Potter and the Chamber of Secrets (film)|1992]]
|e=''Ascendere'' is a [[Latin]] infinitive meaning "to go up,""to climb," "to embark," "to rise(figuratively);" this is the origin of the English word "ascend".}}
===([[Albus Dumbledore's forceful spell|Albus Dumbledore's Forceful Spell]])===
:'''Type:''' Spell
:'''Description:''' This spell was, supposedly, quite powerful as when it was cast, [[Tom Riddle|the opponent]] was forced to conjure a [[Silver shield spell|silver shield]] to deflect it.
:'''Seen/Mentioned:''' This incantation was used only once throughout the series, and that was by Dumbledore in the [[British Ministry of Magic|Ministry of Magic]], immediately following the [[Battle of the Department of Mysteries]] on [[17 June]], [[1996]], while he duelled Voldemort.
===''[[Unlocking Charm|Alohomora]]'' (Unlocking Charm)===
[[File:Unlocking charm1.gif|235px|thumb]]
:'''Type:''' Charm
:'''Pronunciation:''' ah-LOH-ho-MOR-ah
:'''Description:''' Unlocks doors and other objects. It can also unlock doors that have been sealed with a [[Locking Spell]], although it is possible to bewitch doors to become unaffected by this spell.
:'''Seen/Mentioned:''' Used by [[Hermione Granger]] in [[1991]] to allow [[Trio|her and her friends]] to access the [[Third-floor corridor]]] at [[Hogwarts School of Witchcraft and Wizardry|her school]], which was at the time forbidden; she used it again two years later to free [[Sirius Black|Sirius]]'s cell in [[Filius Flitwick's office|her teacher's prison room]].
:'''Etymology:''' The incantation is derived from the West African Sidiki dialect used in geomancy; it means "friendly to thieves", as stated by [[J. K. Rowling|the author]] in testimony during a court case.<ref name=alomohoracourt>{{cite web |url=http://cyberlaw.stanford.edu/files/blogs/Trial%20Transcript%20Day%201.txt |title=Warner Bros Entertainment, Inc. and J.K. Rowling v. RDR Books (Transcript) |author=United States District Court, Southern District of New York |date=April 14, 2008 |publisher=Stanford Law School |access-date=October 1, 2015 |quote=Alohomora is a Sidiki word from West Africa, and it is a term used in geomancy. It is a figure -- the figure alohomora means in Sidiki "favorable to thieves." Which is obviously a very appropriate meaning for a spell that enables you to unlock a locked door by magic.}}</ref>
:'''Notes:''' Whilst in the first book, when the spell is cast the lock or door must be tapped once, in the fifth, [[Miriam Strout|a healer]] simply points her wand at the door to cast it, and on {{PM}} the wand motion is seen as a backward 'S'.

Regular expression to extract chunks of text from a text file?

I need to extract headings and the chunk of text beneath them from a text file in Python using regular expression but I'm finding it difficult.
I converted this PDF to text so that it now looks like this:
So far I have been able to get all the numerical headers (12.4.5.4, 12.4.5.6, 13, 13.1, 13.1.1, 13.1.12) using the following regex:
import re
with open('data/single.txt', encoding='UTF-8') as file:
for line in file:
headings = re.findall(r'^\d+(?:\.\d+)*\.?', line)
print(headings)`
I just don't know how to get the worded part of those headings or the paragraph of text beneath them.
EDIT - Here is the text:
I.S. EN 60601-1:2006&A1:2013&AC:2014&A12:2014
60601-1 © IEC:2005
60601-1 © IEC:2005
– 337 –
– 169 –
12.4.5.4 Other ME EQUIPMENT producing diagnostic or therapeutic radiation
When applicable, the MANUFACTURER shall address in the RISK MANAGEMENT PROCESS the
RISKS associated with ME EQUIPMENT producing diagnostic or therapeutic radiation other than
for diagnostic X-rays and radiotherapy (see 12.4.5.2 and 12.4.5.3).
Compliance is checked by inspection of the RISK MANAGEMENT FILE.
12.4.6 Diagnostic or therapeutic acoustic pressure
When applicable, the MANUFACTURER shall address in the RISK MANAGEMENT PROCESS the
RISKS associated with diagnostic or therapeutic acoustic pressure.
Compliance is checked by inspection of the RISK MANAGEMENT FILE.
13 * HAZARDOUS SITUATIONS and fault conditions
13.1 Specific HAZARDOUS SITUATIONS
General
13.1.1
When applying the SINGLE FAULT CONDITIONS as described in 4.7 and listed in 13.2, one at a
time, none of the HAZARDOUS SITUATIONS in 13.1.2 to 13.1.4 (inclusive) shall occur in the
ME EQUIPMENT.
The failure of any one component at a time, which could result in a HAZARDOUS SITUATION, is
described in 4.7.
Emissions, deformation of ENCLOSURE or exceeding maximum temperature
13.1.2
The following HAZARDOUS SITUATIONS shall not occur:
– emission of flames, molten metal, poisonous or ignitable substance in hazardous
quantities;
– deformation of ENCLOSURES to such an extent that compliance with 15.3.1 is impaired;
–
temperatures of APPLIED PARTS exceeding the allowed values identified in Table 24 when
measured as described in 11.1.3;
temperatures of ME EQUIPMENT parts that are not APPLIED PARTS but are likely to be
touched, exceeding the allowable values in Table 23 when measured and adjusted as
described in 11.1.3;
–
– exceeding the allowable values for “other components and materials” identified in Table 22
times 1,5 minus 12,5 °C. Limits for windings are found in Table 26, Table 27 and Table 31.
In all other cases, the allowable values of Table 22 apply.
Temperatures shall be measured using the method described in 11.1.3.
The SINGLE FAULT CONDITIONS in 4.7, 8.1 b), 8.7.2 and 13.2.2, with regard to the emission of
flames, molten metal or ignitable substances, shall not be applied to parts and components
where:
– The construction or the supply circuit limits the power dissipation in SINGLE FAULT
CONDITION to less than 15 W or the energy dissipation to less than 900 J.

You could use your pattern and match a space after it followed by the rest of the line.
Then repeat matching all following lines that do not start with a heading.
^\d+(?:\.\d+)* .*(?:\r?\n(?!\d+(?:\.\d+)* ).*)*
^\d+(?:.\d+)* Your pattern to match a heading followed by a space
.* Match any char except a newline 0+ times
(?: Non capturing group
\r?\n Match a newline
(?! Negative lookahead, assert what is directly to the right is not
\d+(?:.\d+)* The heading pattern
) Close lookahead
.* Match any char except a newline 0+ times
)* Close the non capturing group and repeat 0+ times to match all the lines
Regex demo

Maybe,
^(\d+(?:\.\d+)*)\s+([\s\S]*?)(?=^\d+(?:\.\d+)*)|^(\d+(?:\.\d+)*)\s+([\s\S]*)
might be somewhat close to get those desired texts that I'm guessing.
Here we'd simply look for lines that'd start with,
^(\d+(?:\.\d+)*)\s+
then, we'd simply collect anything afterwards using
([\s\S]*?)
upto the next line that'd start with,
(?=^\d+(?:\.\d+)*)
Then, we may or may not, depending on how our input may look like, have only one last element left, which we would collect that using this last:
^(\d+(?:\.\d+)*)\s+([\s\S]*)
which we would then alter (using |) to the prior expression.
Even though, this method is simple to code, it's pretty slow performance-wise since we're using lookarounds, so the other answer here is much better, if time complexity would be a concern, which is likely to be.
Demo 1
Test
import re
regex = r"^(\d+(?:\.\d+)*)\s+([\s\S]*?)(?=^\d+(?:\.\d+)*)|^(\d+(?:\.\d+)*)\s+([\s\S]*)"
string = """
I.S. EN 60601-1:2006&A1:2013&AC:2014&A12:2014
60601-1 © IEC:2005
60601-1 © IEC:2005
– 337 –
– 169 –
12.4.5.4 Other ME EQUIPMENT producing diagnostic or therapeutic radiation
When applicable, the MANUFACTURER shall address in the RISK MANAGEMENT PROCESS the
RISKS associated with ME EQUIPMENT producing diagnostic or therapeutic radiation other than
for diagnostic X-rays and radiotherapy (see 12.4.5.2 and 12.4.5.3).
Compliance is checked by inspection of the RISK MANAGEMENT FILE.
12.4.6 Diagnostic or therapeutic acoustic pressure
When applicable, the MANUFACTURER shall address in the RISK MANAGEMENT PROCESS the
RISKS associated with diagnostic or therapeutic acoustic pressure.
Compliance is checked by inspection of the RISK MANAGEMENT FILE.
13 * HAZARDOUS SITUATIONS and fault conditions
13.1 Specific HAZARDOUS SITUATIONS
* General
13.1.1
When applying the SINGLE FAULT CONDITIONS as described in 4.7 and listed in 13.2, one at a
time, none of the HAZARDOUS SITUATIONS in 13.1.2 to 13.1.4 (inclusive) shall occur in the
ME EQUIPMENT.
The failure of any one component at a time, which could result in a HAZARDOUS SITUATION, is
described in 4.7.
* Emissions, deformation of ENCLOSURE or exceeding maximum temperature
13.1.2
The following HAZARDOUS SITUATIONS shall not occur:
– emission of flames, molten metal, poisonous or ignitable substance in hazardous
quantities;
– deformation of ENCLOSURES to such an extent that compliance with 15.3.1 is impaired;
–
temperatures of APPLIED PARTS exceeding the allowed values identified in Table 24 when
measured as described in 11.1.3;
temperatures of ME EQUIPMENT parts that are not APPLIED PARTS but are likely to be
touched, exceeding the allowable values in Table 23 when measured and adjusted as
described in 11.1.3;
–
– exceeding the allowable values for “other components and materials” identified in Table 22
times 1,5 minus 12,5 °C. Limits for windings are found in Table 26, Table 27 and Table 31.
In all other cases, the allowable values of Table 22 apply.
Temperatures shall be measured using the method described in 11.1.3.
The SINGLE FAULT CONDITIONS in 4.7, 8.1 b), 8.7.2 and 13.2.2, with regard to the emission of
flames, molten metal or ignitable substances, shall not be applied to parts and components
where:
– The construction or the supply circuit limits the power dissipation in SINGLE FAULT
CONDITION to less than 15 W or the energy dissipation to less than 900 J.
"""
print(re.findall(regex, string, re.M))
Output
[('12.4.5.4', 'Other ME EQUIPMENT producing diagnostic or therapeutic
radiation \nWhen applicable, the MANUFACTURER shall address in
the RISK MANAGEMENT PROCESS the \nRISKS associated with ME
EQUIPMENT producing diagnostic or therapeutic radiation other than
\nfor diagnostic X-rays and radiotherapy (see 12.4.5.2 and 12.4.5.3).
\n\nCompliance is checked by inspection of the RISK MANAGEMENT
FILE.\n\n', '', ''), ('12.4.6', 'Diagnostic or therapeutic acoustic
pressure \nWhen applicable, the MANUFACTURER shall address in
the RISK MANAGEMENT PROCESS the \nRISKS associated with diagnostic
or therapeutic acoustic pressure. \n\nCompliance is checked by
inspection of the RISK MANAGEMENT FILE.\n\n', '', ''), ('13', '*
HAZARDOUS SITUATIONS and fault conditions\n\n', '', ''), ('13.1',
'Specific HAZARDOUS SITUATIONS\n\n* General \n\n', '', ''),
('13.1.1', 'When applying the SINGLE FAULT CONDITIONS as
described in 4.7 and listed in 13.2, one at a \ntime, none
of the HAZARDOUS SITUATIONS in 13.1.2 to 13.1.4 (inclusive)
shall occur in the \nME EQUIPMENT.\n\nThe failure of any one
component at a time, which could result in a HAZARDOUS SITUATION, is
\ndescribed in 4.7. \n\n* Emissions, deformation of ENCLOSURE or
exceeding maximum temperature \n\n', '', ''), ('', '', '13.1.2', 'The
following HAZARDOUS SITUATIONS shall not occur: \n– emission of
flames, molten metal, poisonous or ignitable substance in
hazardous \n\nquantities; \n\n– deformation of ENCLOSURES to such an
extent that compliance with 15.3.1 is impaired; \n– \n\ntemperatures
of APPLIED PARTS exceeding the allowed values identified in
Table 24 when \nmeasured as described in 11.1.3; \ntemperatures of
ME EQUIPMENT parts that are not APPLIED PARTS but are likely
to be \ntouched, exceeding the allowable values in Table 23
when measured and adjusted as \ndescribed in 11.1.3; \n\n– \n\n–
exceeding the allowable values for “other components and materials”
identified in Table 22 \ntimes 1,5 minus 12,5 °C. Limits for windings
are found in Table 26, Table 27 and Table 31. \nIn all other cases,
the allowable values of Table 22 apply. \n\nTemperatures shall be
measured using the method described in 11.1.3. \n\nThe SINGLE FAULT
CONDITIONS in 4.7, 8.1 b), 8.7.2 and 13.2.2, with regard to
the emission of \nflames, molten metal or ignitable substances,
shall not be applied to parts and components \nwhere: \n– The
construction or the supply circuit limits the power
dissipation in SINGLE FAULT \n\nCONDITION to less than 15 W or the
energy dissipation to less than 900 J. \n\n')]

Thanks to their detailed answers and helpful explanations I ended up combining parts of both #The-fourth-bird's code and #Emma's code into this regex which seems to work nicely for what I need.
(^\d+(?:\.\d+)*\s+)((?![a-z])[\s\S].*(?:\r?\n))([\s\S]*?)(?=^\d+(?:\.\d+)*\s+(?![a-z]))
Here is the REGEX DEMO.
I does what I want, which is splitting the (numerical heading), (worded heading) and the (body of text) into groups separated by commas which allow me to separate them into columns in Excel by using the custom delimiter ), ( and some other post processing.
The nice thing about this new regex is that it skips numbered headings that are just references and not actually headings as seen here:

import pdfplumber
import re
pdfToString = ""
with pdfplumber.open(r"sample.pdf") as pdf:
for page in pdf.pages:
print(page.extract_text())
pdfToString += page.extract_text()
matches = re.findall(r'^\d+(?:\.\d+)* .*(?:\r?\n(?!\d+(?:\.\d+)* ).*)*',pdfToString, re.M)
for i in matches:
if "word_to_extract" in i[:50]:
print(i)
This solution is to extract all the headings which has same format of headings in the question and to extract the required heading and the paragraphs that follows it.

NLTK - Python extract names from csv

i've got a CSV which contains article's text in different raws.
Like we have column 1:
Hello i am John
Tom has got a Dog
... more text.
I'm trying the extract the first names and surname from those text and i was able to do that if i copy and paste the single text in the code.
But i don't know how to read the csv in the code and then it has to processes the different texts in the raws extracting name and surname.
Here is my code working with the text in it:
import operator,collections,heapq
import csv
import pandas
import json
import nltk
from nameparser.parser import HumanName
def get_human_names(text):
tokens = nltk.tokenize.word_tokenize(text)
pos = nltk.pos_tag(tokens)
sentt = nltk.ne_chunk(pos, binary = False)
person_list = []
person = []
name = ""
for subtree in sentt.subtrees(filter=lambda t: t.label() == 'PERSON'):
for leaf in subtree.leaves():
person.append(leaf[0])
if len(person) > 1: #avoid grabbing lone surnames
for part in person:
name += part + ' '
if name[:-1] not in person_list:
person_list.append(name[:-1])
name = ''
person = []
return (person_list)
text = """
M.F. Husain, Untitled, 1973, oil on canvas, 182 x 122 cm. Courtesy the Pundole Family Collection
In her essay ‘Worlding Asia: A Conceptual Framework for the First Delhi Biennale’, Arshiya Lokhandwala explores Gayatri Spivak’s provocation of ‘worlding’, which has been defined as imperialism’s epistemic violence of inscribing meaning upon a colonized space to bring it into the world through a Eurocentric framework. Lokhandwala extends this concept of worlding to two anti-cartographical terms: ‘de-worlding’, rejecting or debunking categories that are no longer useful such as the binaries of East-West, North-South, Orient-Occidental, and ‘re-worlding’, re-inscribing new meanings into the spaces that have been de-worlded to create one’s own worlds. She offers de-worlding and re-worlding as strategies for active resistance against epistemic violence of all forms, including those that stem from ‘colonialist strategies of imperialism’ or from ‘globalization disguised within neo-imperialist practices’.
Lokhandwala writes: Fourth World. The presence of Arshiya is really the main thing here.
Re-worlding allows us to reach a space of unease performing the uncanny, thereby locating both the object of art and the postcolonial subject in the liminal space, which prevents these categorizations as such… It allows an introspected view of ourselves and makes us seek our own connections, and look at ourselves through our own eyes.
In a recent exhibition on the occasion of the seventieth anniversary of India’s Independence, Lokhandwala employed the term to seemingly interrogate this proposition: what does it mean to re-world a country through the agonistic intervention of art and activism? What does it mean for a country and its historiography to re-world? What does this re-worlded India, in active resistance and a state of introspection, look like to itself?
The exhibition ‘India Re-Worlded: Seventy Years of Investigating a Nation’ at Gallery Odyssey in Mumbai (11 September 2017–21 February 2018) invited artists to select a year from the seventy years since the country’s independence that had personal import or resonated with them because of the significance of the events that occurred at the time. The show featured works that responded to or engaged with these chosen years. It captured a unique history of post-independent India told through the perspective of seventy artists. The works came together to collectively reflect on the history and persistence of violence from pre-independence to the present day and made reference to the continued struggle for political agency through acts of resistance, artistic and otherwise. Through the inclusion of subaltern voices, imagined geographies, particular experiences, solidarities and critical dissent, the exhibition offered counter-narratives and multiple histories.
Anita Dube, Missing Since 1992, 2017, wood, electrical wire, holders, bulbs, voltage stabilizers, 223 x 223 cm. Courtesy the artist and Gallery Odyssey
Lokhandwala says she had been thinking hard about an appropriate response to the seventy years of independence. ‘I wanted to present a new curatorial paradigm, a postcolonial critique of the colonisation and an affirmation of India coming into her own’, she says. ‘I think the fact that I tried to include seventy artists to [each take up] one year in the lifetime of the nation was also a challenging task to take on curatorially.’
Her previous undertaking ‘After Midnight: Indian Modernism to Contemporary India: 1947/1997’ at the Queens Museum in New York in 2015 juxtaposed two historical periods in Indian art: Indian modern art that emerged in the post-independence period from 1947 through the 1970s, and contemporary art from 1997 onwards when the country experienced the effects of economic liberalization and globalization. The 'India Re-Worlded' exhibition similarly presented art practices that emerged from the framework of postcolonial Indian modernity. It attempted to explore the self-reflexivity of the Indian artist as a postcolonial subject and, as Lokhandwala described in the curatorial note, the artists’ resulting ‘sense of agency and renewed connection with the world at large’. The exhibition included works by Progressive Artists' Group core members F.N. Souza, S.H. Raza, M.F. Husain and their peers Krishen Khanna, Tyeb Mehta and V.S. Gaitonde, presented under the year in which they were produced. Other important and pioneering pieces included work from Somnath Hore’s paper pulp print series Wounds (1970); a blowtorch on plywood work by abstractionist Jeram Patel, who was one of the founding members of Group 1890 ; and a video documenting one of Rummana Husain’s last performances.
The methodology of their display removed the didactic, art historical preoccupation with chronology and classification, instead opting to intersperse them amongst contemporary works. This fits in with Lokhandwala’s curatorial impulses and vision: to disrupt and resist single narratives, to stage dialogues and interactions between the works, to offer overlaps, intersections and nuances in the stories, but also in the artistic impetuses.
Jeram Patel, Untitled, 1970, blowtorch Fourht World on plywood, 61 x 61 cm. Courtesy the artist and Gallery Odyssey
The show opened with Jitish Kallat’s Death of Distance (2006), then we have Arshiya, which through lenticular prints presented two overlaid found texts from 2005 and 2006. One was a harrowing news story of a twelve-year-old Indian girl committing suicide after her mother tells her she cannot afford one rupee – two US cents – for a school meal. The other one was a news clipping in which the head of the state-run telecommunications company announces a new one-rupee-per-minute tariff plan for interstate phone calls and declares the scheme as ‘the death of distance’. The images offer two realities that are distant from and at odds with each other. They highlight an economic disparity heightened by globalization. A rupee coin, enlarged to a human scale and covered in black lead, stood poised on the gallery floor in front of the prints.
Bose Krishnamachari chose 1962, the year of his birth, to discuss the relationship between memory and age. As a visual representation of the country’s past through a timeline, within which he situated his own identity-questioning experiences as an artist, his work epitomized the themes and intentions of the exhibition. In Shilpa Gupta’s single channel video projection 100 Hand drawn Maps of India (2007–8) ordinary Indian people sketch outlines of the country from memory. The subjective maps based on the author’s impression and perception of space show how each person sees the country and articulates its borders. The work seems to ask, what do these incongruent representations reveal about our collective identities and our ideas about nationhood?
The repetition of some of the years selected, or even the absence of certain years, suggested that the parameters set by the curatorial concept sought to guide rather than clamp down on. This allowed greater freedom for the artists and curator, and therefore more considered and wide responses.
Surekha’s photographic series To Embrace (2017) celebrated the Chipko tree-hugging movement that originated on 25 March 1974, when 27 women from Reni village in Uttar Pradesh in northern India staged a self-organised, non-violent resistance to the felling of trees by clinging to them and linking arms around them. The photographs showed women embracing the branches of the giant, 400-year-old Dodda Alada Mara (Big Banyan Tree) in rural Bengaluru – paying a homage to both the pioneering eco-feminist environmental movement and the grand old tree.
Anita Dube’s Missing Since 1992 (2017) hung from the ceiling like a ghost of a terrible, dark past. Its electrical wires and bulbs outlined a sombre dome to represent the demolition of the Babri Masjid on 6 December 1992, which Dube calls ‘the darkest day I have experienced as a citizen’. This piece was one of several works in the exhibition that dealt with this event and the many episodes of communal riots that followed. These works document a decade when the country witnessed economic reform and growth but also the rise of a religious right-wing.
Riyas Komu, Fourth World, 2017, rubber and metal, 244 x 45 cm each. Courtesy the artist and Gallery Odyssey
Near the end of the exhibition, Riyas Komu’s sculptural installation Fourth World (2017) alerted us to the divisive forces that are threatening to dismantle the ethical foundations of the Republic symbolized by its official emblem, the Lion Capital – a symbol seen also on the blackened rupee coin featured in Kallat’s work – and in a way rounded off the viewing experience.
The seventy works that attempted to represent seventy years of the country’s history built a dense and complicated network of voices and stories, and also formed a cross section of the art emerging during this period. Although the show’s juxtaposition of modern and contemporary art made it seem like an extension of the themes presented in the curator’s previous exhibition at the Queens Museum, here the curatorial concept made the process of staging the exhibition more democratic blurring the sequence of modern and contemporary Indian art. Furthermore, the multi-pronged curatorial intentions brought renewed criticality to the events of past and present, always underscoring the spirit of resistance and renegotiation as the viewer could actively de-world and re-world.
"""
names = get_human_names(text)
print ("LAST, FIRST")
namex=[]
for name in names:
last_first = HumanName(name).last + ' ' + HumanName(name).first
print (last_first)
namex.append(last_first)
print (namex)
print('Saving the data to the json file named Names')
try:
with open('Names.json', 'w') as outfile:
json.dump(namex, outfile)
except Exception as e:
print(e)
So i would like to remove all the text from the code and want the code to process the text from my csv.
Thanks a lot :)

CSV stands for Comma Separated Values and is a text format used to represent tabular data in plain text. Commas are used as column separators and line breaks as row separators. Your string does not look like a real csv file. Nevermind the extension you can still read your text file like this:
with open('your_file.csv', 'r') as f:
my_text = f.read()
Your text file is now available as my_text in the rest of your code.

Pandas has read_csv command:
yourText= pandas.read_csv("csvFile.csv")

Cleaning Up (or Avoiding) Extra Whitespace with PyPDF2

I've been extracting text from PDFs using PyPDF2. However it seems to be inputting erroneous white space in between words. Does anyone know of way to avoid this, or clean it after the fact? Here is an example:
'IN THE MATTER OF an application submitted by 1113 York Avenue Realty
Company, L.L.C. and 60th Street Devel opment LLC pursuant to Sections
197-c and 201 of the New York City Charter for an amendment of th e
Zoning Map, Section Nos. 8c and 8d:'
Here "development" is spelt "devel opment" and "the" is the spelt "th e". I'd like to correct this.
Here is PDF. The example text is from list item number 1, on the first page.

Cleaning text string after getting body text using Beautifulsoup

I'm trying to get text from articles on various webpages and write them as clean text documents. I don't want all visible text because that often includes irrelevant links on the side of webpages. I'm using Beautifulsoup to extract the information from pages. But, extra links not just on the side of the page but also those sometimes in the middle of the body text and at the bottom of the articles sometimes make it into the final product.
Does anyone know how to deal with the problem of extra links that are converted into text that are not actually a part of the real article's text?
#Some of the imports are for other portions of the code not shown here.
#I'm new to Python and am bad at remembering which library has which functions.
import os
import sys
import urllib2
import webbrowser
from bs4 import BeautifulSoup
from os import path
from cookielib import CookieJar
#I made an opener to deal with proxies and put *** instead of my information
#cookielib helps me get articles from nytimes
proxy = urllib2.ProxyHandler({'http': '***' % '***'})
auth = urllib2.HTTPBasicAuthHandler()
cj = CookieJar()
opener = urllib2.build_opener(proxy, auth, urllib2.HTTPHandler, urllib2.HTTPCookieProcessor(cj))
urllib2.install_opener(opener)
#Uses url input as a string to upen a webpage and and pulls out all the information.
def baumeister(url):
req = urllib2.Request(url)
opened = urllib2.urlopen(req)
html_doc = opened.read()
soup = BeautifulSoup(html_doc)
return soup
#Gets the body text from that html information.
def substanz(url):
soup = baumeister(url)
body = soup.find_all("p") #This is where I have tried to fix the problem and failed
result = ""
for e in body:
i = e.getText().replace("\t", "").replace(" ", " ").strip().encode(errors="ignore")
result += i + "\r\n\r\n"
return result
One article that I have used to test substanz that gets cleaned in the exact way I want is:
http://blogs.hbr.org/2014/06/do-you-really-want-to-be-yourself-at-work/
I'm trying to test with more articles from different sites. So I'm trying to clean the result of substanz (the result is a big string). The problem I have is with this article:
http://www.cnbc.com/id/101790001?__source=yahoo%7Cfinance%7Cheadline%7Cheadline%7Cstory&par=yahoo&doc=101790001%7CThink%20college%20is%20expensiv#.
I've just used the print substanz('url') to see what the result looks like. With the cnbc article I get extra links turned into text that are not really a part of the article. Whereas in the Harvard Business Review Article everything works out just fine as included links are part of the actual text.
I'm not going to attach the full result for each article here for viewing because they are each a full page of text long.
If you try exactly the code I have posted above the opener is not going to work, so use whatever opener you like to access websites. I have to access a certain proxy at work so that's the format that works for me.
Final note, I'm using python 3.4, and am writing the code in ipython notebook.

import requests
from bs4 import BeautifulSoup
r = requests.get("http://www.cnbc.com/id/101790001?__source=yahoo%7Cfinance%7Cheadline%7Cheadline%7Cstory&par=yahoo&doc=101790001%7CThink%20college%20is%20expensiv#")
soup = BeautifulSoup(r.content)
text =[''.join(s.findAll(text=True))for s in soup.findAll('p')]
print (text)
['>> View All Results for ""', 'Enter multiple symbols separated by commas', 'London quotes now available', 'Interest rates on loans to jump', "Because federal student loans are tied to the 10-year Treasury note, CNBC's Sharon Epperson reports borrowers will see the impact of the rise in Treasury yields over the past year.", ' Congratulations, graduates, on your diploma. Now what about that $29,000 student loan debt? ', ' More than 70 percent of graduates will carry student debt into the real world, according to the Institute for College Access and Success. And the average debt is just shy of $30,000. ', ' But the news will get worse next week when interest rates on student loans are set to rise again. ', ' Though federal student loan rates are fixed for the life of the loan, these rates reset for new borrowers every July 1, thanks to legislation that ties the rates to the performance of the financial markets. ', ' The interest rate on federal Stafford loans will go from its current fixed rate of just under 4 percent to 4.66 percent for loans that are distributed between July 1 and June 30, 2015. ', ' Read MoreStudent loan problem an easy fix: Sen. Warren ', ' For graduate students, the rate on Stafford loans will rise from just over 5 percent to 6.21 percent. ', ' Direct PLUS Loans for graduates and parents are still the most expensive, with rates rising to 7.21 percent.', 'Which college major pays off most?', "CNBC's Sharon Epperson reports majoring in engineering is the most lucrative. ", " The increase in monthly federal student loan payments can add up quickly, but shouldn't be too burdensome for most students. For every $10,000 in loans, new borrowers will pay about $4 more a month based on a 10-year repayment period. ", " Read MoreWhy millennial women don't save for retirement ", ' Still, experts warn that this is only just the beginning. ', ' "Federal student loan rates will continue to increase in the next few years and will likely hit the maximum rate caps which are as high as 10.5 percent for some loans," said Mark Kantrowitz, senior vice president and publisher of Edvisors.com. ', ' For sophomore student Samantha Cook, the decision to go to George Washington University was a big one financially. She says she had doubts about it. ', ' "My parents wanted to assure me that no matter what I picked, we\'d find a way to make it work," Cook said. Like most families, Cook and her parents are making it work by combining their household savings, scholarships and grants—and student loans. ', ' Read MoreCramer: Offset high cost of higher education ', ' Despite rising tuition and borrowing costs, the Cook family decided against Samantha transferring to an in-state university. ', ' Despite the debt load she is taking on, she said, "the value of a GW degree for me at least would be more valuable when looking for jobs later on." ', " —By CNBC's Sharon Epperson ", 'Hosting a yard sale may not be the most profitable way to get rid of your old junk.', 'Many Americans with debit cards tied to their checking accounts are still confused about how these programs work. ', "Here's how to avoid these deadly sins if you're contemplating or already in a divorce.", "The IRS offers a lot of help for students. Problem is, the educational tax breaks and how they work together -- or don't -- are confusing.", 'Get the best of CNBC in your inbox', 'Tips for home buyers that will help you find the right home for your bank account.', 'Complaints about movers are down. How to find the right one—and save.', "Forget bathing suit season. Why it's really time to join the gym. ", 'Drivers might see lower gas prices this year, but smart shopping tactics could help them save even more.', 'Data is a real-time snapshot *Data is delayed at least 15 minutesGlobal Business and Financial News, Stock Quotes, and Market Data and Analysis', '© 2014 CNBC LLC. All Rights Reserved.', 'A Division of NBCUniversal']
From the website in your link to get text from the main article.
import requests
from bs4 import BeautifulSoup
r = requests.get("http://www.cnbc.com/id/101790001?__source=yahoo%7Cfinance%7Cheadline%7Cheadline%7Cstory&par=yahoo&doc=101790001%7CThink%20college%20is%20expensiv#")
soup = BeautifulSoup(r.content)
text =[''.join(s.findAll(text=True)) for s in soup.findAll("div", {"class":"group"})]
print (text)
['\n Congratulations, graduates, on your diploma. Now what about that $29,000 student loan debt? \n More than 70 percent of graduates will carry student debt into the real world, according to the Institute for College Access and Success. And the average debt is just shy of $30,000. \n But the news will get worse next week when interest rates on student loans are set to rise again. \n Though federal student loan rates are fixed for the life of the loan, these rates reset for new borrowers every July 1, thanks to legislation that ties the rates to the performance of the financial markets. \n The interest rate on federal Stafford loans will go from its current fixed rate of just under 4 percent to 4.66 percent for loans that are distributed between July 1 and June 30, 2015. \n Read MoreStudent loan problem an easy fix: Sen. Warren \n For graduate students, the rate on Stafford loans will rise from just over 5 percent to 6.21 percent. \n Direct PLUS Loans for graduates and parents are still the most expensive, with rates rising to 7.21 percent.\n', '\n The increase in monthly federal student loan payments can add up quickly, but shouldn\'t be too burdensome for most students. For every $10,000 in loans, new borrowers will pay about $4 more a month based on a 10-year repayment period. \n Read MoreWhy millennial women don\'t save for retirement \n Still, experts warn that this is only just the beginning. \n "Federal student loan rates will continue to increase in the next few years and will likely hit the maximum rate caps which are as high as 10.5 percent for some loans," said Mark Kantrowitz, senior vice president and publisher of Edvisors.com. \n For sophomore student Samantha Cook, the decision to go to George Washington University was a big one financially. She says she had doubts about it. \n "My parents wanted to assure me that no matter what I picked, we\'d find a way to make it work," Cook said. Like most families, Cook and her parents are making it work by combining their household savings, scholarships and grants—and student loans. \n Read MoreCramer: Offset high cost of higher education \n Despite rising tuition and borrowing costs, the Cook family decided against Samantha transferring to an in-state university. \n Despite the debt load she is taking on, she said, "the value of a GW degree for me at least would be more valuable when looking for jobs later on." \n —By CNBC\'s Sharon Epperson \n']