We Keep Coding

Finding the line number of specific values in pandas dataframe - python

I am working on a school project and I am trying to simulate a library's catalogue system. I have .csv files that hold all the data I need but I am having a problem with checking if an inputted title, author, bar code, etc. is in the data set. I have searched around for quite a while trying different solutions but nothing is working.
The idea that I have right now is that if I can find at what line the inputted data, then I can use .loc[] to get the needed info.
Is this the right track? is there another, more efficient way to do this?
import pandas
mainData = pandas.read_csv("mainData.csv")
barcodes = mainData["Barcode"]
authors = mainData["Author"]
titles = mainData["Title/Subtitle"]
callNumbers = mainData["Call Number"]
k = "Han, Jenny,"
for i in authors:
if k == i:
print("Success")
k = authors.index[k]
print(authors[k])
else:
print("Fail" + k)
# Please Note: This code only checks for an author match and has all other fields left out as I thought this code was too inefficient to add the rest of the fields. The code also does not find the line on witch the matched are located, therefore .loc[] can not be used to print out all the data found.
This is the code I am using right now, It outputs the result along with an error Python IndexError: only integers, slices (\`:\`), ellipsis (\`\...\`), numpy.newaxis (\`None\`) and integer or boolean arrays are valid indices and is very slow. I would like the code to be able to output the books and their respective info. I have found the the .loc[] feature (mentioned above) outputs the info quite nicely. Here is the data I am using .
Edit: I have been able to reduce the time it takes for the program to run and made a functional "prototype"
authorFirst = authorFirst.lower()
authorFirst = authorFirst.title()
authorFirst += ","
authorSecond = input("Enter author's last name: ")
authorSecond = authorSecond.lower()
authorSecond = authorSecond.title()
authorSecond += ", "
authorInput = authorSecond + authorFirst
print(mainData[mainData["Author"].isin([authorInput])])
bookChoice = input("Please Enter the number to the left of the barcode to select a book: ")
print(mainData.loc[int(bookChoice)])
id provides the functionality that I am looking for but I feel that there has to be a better way of doing it. (Not asking the user to input the row number). Idk if this is possible tho.
I am new to python and this is my first time using pandas so i'm sorry if this is really shitty and hurts your brain.
Thank-you so much for your time!

Pandas does not really need to find the numeric index of something, to do indexing.
Since you have not provided any starting point or data, I'll just provide a few pointers here as there are mans ways to match and index things in pandas.
import pandas as pd
# build a library
library = pd.DataFrame({
"Author": ["H.G. Wells", "Hubert Selby Jr.", "Ken Kesey"],
"Title": [
"The War of the Worlds",
"Requiem for a Dream",
"One Flew Over the Cuckoo's Nest",
],
"Published": [1898, 1979, 1962],
})
# find on some characteristics
mask_wells = library.Author.str.contains("Wells")
mask_rfad = library["Title"] == "Requiem for a Dream"
mask_xixth = library["Published"] < 1900

Python Dictionaries stuck

hope you are having a nice day, so I need to write a code that takes an input, which is going to be a key in an already existing dictionary, and prints the value as a str as described in the pic. I am stuck rn, I am a beginner and tried everything I see on the web related, appriciate any help, thank you.
[and these are the codes I tried][1]
def read_dataset(filename):
lines = open(filename).readlines()
lines_splitted = [line.strip().split(',') for line in lines]
return {lst[0]: lst[1:] for lst in lines_splitted}
movie_dict = read_dataset('dataset.txt')
# DO_NOT_EDIT_ANYTHING_ABOVE_THIS_LINE
hey = input('Enter a movie name: ')
actors = movie_dict[hey]
# DO_NOT_EDIT_ANYTHING_BELOW_THIS_LINE
print(','.join(sorted(actors)))
So this is my assignment

If you begin, try using pandas to make a dataframe of your base. It will be easier to use, and you should reuse it in a lot of cases.
Here an example of what you could do. If you don't understand a point, do not hesitate to ask me:
# import the package
import pandas as pd
# build the dataframe for the example
film = pd.DataFrame({'film':['Goldorak','Mickey Mouse'],'actors':[['Actarus','Alkor'],['Mickey, Minnie, Pluto, Goofy']]})
# the input
query = input('Enter the film')
# the 'search' engine
if len(film[film.film==query].actors)>0:
print(film[film.film==query].actors)
else:
print('not found in base')
EDIT with Dictionary :
FILM = {'Forest Gump':['Tom Hanks', 'Gary Sinise']}
query = None
query = input('Enter the film')
try:
print(FILM.get(query))
except:
pass

sqlite: Calling database each time function is called

I got a function that is called to calculate a response every time the user inputs something. The function gets the response from a database. What I don't understand is, why I have to redefine my variable (I have called it intents_db) that contains all the data from the database each time the function is called? I have tried putting it outside the function, but then my program only works the first time, but the returns an empty answer the second time the user inputs something.
def response(sentence, user_id='1'):
results = classify_intent(sentence)
intents_db = c.execute("SELECT row_num, responses, tag, responses, intent_type, response_type, context_set,\
context_filter FROM intents")
if results:
# loop as long as there are matches to process
while results:
if results[0][1] > answer_threshold:
for i in intents_db:
# print('tag:', i[2])
if i[2] == results[0][0]:
print(i[6])
if i[6] != 'N/A':
if show_details:
print('context: ', i[6])
context[user_id] = i[6]
responses = i[1].split('&/&')
print(random.choice(responses))
if i[7] == 'N/A' in i or \
(user_id in context and i[7] in i and i[7] == context[
user_id]):
# a random response from the intent
responses = i[1].split('&/&')
print(random.choice(responses))
print(i[4], i[5])
print(results[0][1])
elif results[0][1] <= answer_threshold:
print(results[0][1])
for i in intents_db:
if i[2] == 'unknown':
# a random response from the intent
responses = i[1].split('&/&')
print(random.choice(responses))
initial_comm_output = random.choice(responses)
return initial_comm_output
else:
initial_comm_output = "Something unexpected happened when calculating response. Please restart me"
return initial_comm_output
results.pop(0)
return results
Also, I started getting into databases and sqlite3 because I want to make a massive database long term. Therefore it also seems inefficient that I have to load the whole database at all. Is there some way I can only load the row of data I need? I got a row_number column in my database, so if it was somehow possible to say like:
"SELECT WHERE row_num=2 FROM intents"
that would be great, but I can't figure out how to do it.

cursor.execute() returns an iterator, and you can only loop over it once.
If you want to reuse it, turn it into a list:
intents_db = list(c.execute("..."))

Therefore it also seems inefficient that I have to load the whole database at all. Is there some way I can only load the row of data I need? I got a row_number column in my database, so if it was somehow possible to say like: "SELECT WHERE row_num=2 FROM intents" that would be great, but I can't figure out how to do it.
You nearly got it: it is
intents_db = c.execute("SELECT row_num, responses, tag, responses, intent_type,
response_type, context_set, context_filter
FROM intents WHERE row_num=2")
But don't do the mistake other database beginners make and try to put in some variable from your program directly into that string. This makes the program prone to SQL injections.
Rather, do
row_num = 2
intents_db = c.execute("SELECT row_num, responses, tag, responses, intent_type,
response_type, context_set, context_filter
FROM intents WHERE row_num=?", (row_num,))
Of course, you can also set conditions for other fields.

My function doesn't return a value, can't figure out why

The function is supposed to open a csv file that has data in this format
"polling company,date range,how many polled,margin of error,cruz,kasich,rubio,trump"
When I run this function, read_data_file, there is no output, which I don't understand since I am returning the poll_data. I don't believe there is an issue with the rest of the code as if I replaced 'return poll data' with 'print(poll_data)' there is the desired output.
I am a noob at this and don't have a full grasp of return.
def read_data_file(filename):
file = open(filename, 'r')
poll_data = []
for data in file:
data = data.strip('\n')
data = data.split(',')
poll_data.append(data)
return poll_data
read_data_file('florida-gop.csv')

you changed that last line in the function from print to return. So, when you call your function as such:
read_data_file('florida-gop.csv')
it does return that data. It's sitting right there! But then your script ends, doing nothing with that data. so, instead, do something like this:
data = read_data_file('florida-gop.csv')
print(data)
a short addendum - political data is an excellent way to learn data manipulation with Python and, if so inclined, Python itself. I'd recommend the O'Reilly books on data & Python - but that's outside the scope of this question.

You have two options here:
Replacing return poll_data with print poll_data.
Instead of read_data_file('florida-gop.csv'), you can do print read_data_file('florida-gop.csv').
Why do you need to do this?
Print vs Return
print actually shows you the result, while return only gives the result to the computer, if that makes sense. The computer knows it, but it doesn't print it, which is why the second solution works - the computer has the data you want, and it is able to print it if you command it too. However, in your case, the first solution is probably easier.
Hope this helps!

Continuing from the above(or not so above anymore xD) answer...
the full code would now be,
def read_data_file(filename):
file = open(filename, 'r')
poll_data = []
for data in file:
data = data.strip('\n')
data = data.split(',')
poll_data.append(data)
return poll_data
print(read_data_file('florida-gop.csv')) # Before you forgot to print it.
or exactly like the above answer,
def read_data_file(filename):
file = open(filename, 'r')
poll_data = []
for data in file:
data = data.strip('\n')
data = data.split(',')
poll_data.append(data)
return poll_data
data = read_data_file('florida-gop.csv')
print(data)

Paginating requests to an API

I'm consuming (via urllib/urllib2) an API that returns XML results. The API always returns the total_hit_count for my query, but only allows me to retrieve results in batches of, say, 100 or 1000. The API stipulates I need to specify a start_pos and end_pos for offsetting this, in order to walk through the results.
Say the urllib request looks like http://someservice?query='test'&start_pos=X&end_pos=Y.
If I send an initial 'taster' query with lowest data transfer such as http://someservice?query='test'&start_pos=1&end_pos=1 in order to get back a result of, for conjecture, total_hits = 1234, I'd like to work out an approach to most cleanly request those 1234 results in batches of, again say, 100 or 1000 or...
This is what I came up with so far, and it seems to work, but I'd like to know if you would have done things differently or if I could improve upon this:
hits_per_page=100 # or 1000 or 200 or whatever, adjustable
total_hits = 1234 # retreived with BSoup from 'taster query'
base_url = "http://someservice?query='test'"
startdoc_positions = [n for n in range(1, total_hits, hits_per_page)]
enddoc_positions = [startdoc_position + hits_per_page - 1 for startdoc_position in startdoc_positions]
for start, end in zip(startdoc_positions, enddoc_positions):
if end > total_hits:
end = total_hits
print "url to request is:\n ",
print "%s&start_pos=%s&end_pos=%s" % (base_url, start, end)
p.s. I'm a long time consumer of StackOverflow, especially the Python questions, but this is my first question posted. You guys are just brilliant.

I'd suggest using
positions = ((n, n + hits_per_page - 1) for n in xrange(1, total_hits, hits_per_page))
for start, end in positions:
and then not worry about whether end exceeds hits_per_page unless the API you're using really cares whether you request something out of range; most will handle this case gracefully.
P.S. Check out httplib2 as a replacement for the urllib/urllib2 combo.

It might be interesting to use some kind of generator for this scenario to iterate over the list.
def getitems(base_url, per_page=100):
content = ...urllib...
total_hits = get_total_hits(content)
sofar = 0
while sofar < total_hits:
items_from_next_query = ...urllib...
for item in items_from_next_query:
sofar += 1
yield item
Mostly just pseudo code, but it could prove quite useful if you need to do this many times by simplifying the logic it takes to get the items as it only returns a list which is quite natural in python.
Save you quite a bit of duplicate code also.