I am writing a code to get specific information from yahoo finance website
page = requests.get('http://finance.yahoo.com/q/pr?s=%s')
tree=html.fromstring(page.text)
annual_report = tree.xpath('//td[#class="yfnc_datamoddata1"]/text()')
annual_report
Where %s is a name of a stock. If I manually input a name of a stock everything works great. But if I try to run a for loop for a list that I made,
for x in my_list:
page = requests.get('http://finance.yahoo.com/q/pr?s=%s'),(x,)
tree=html.fromstring(page.text)
annual_report = tree.xpath('//td[#class="yfnc_datamoddata1"]/text()')
print annual_report
I get an error on tree line 'tuple' object has no attribute 'text'
Your mistake is:
page = requests.get('http://finance.yahoo.com/q/pr?s=%s'),(x,)
Instead of formatting the string, you made 'page' a tuple.
This should work:
page = requests.get('http://finance.yahoo.com/q/pr?s=%s' % (x,))
page = requests.get('http://finance.yahoo.com/q/pr?s=%s'),(x,)
This is not how to define a format string. You accidentally created a tuple ('http://finance.yahoo.com/q/pr?s=%s', x)
To incorporate x into the string write it like this:
page = requests.get('http://finance.yahoo.com/q/pr?s=%s' % x)
Or even better because not needing specifiers:
page = requests.get('http://finance.yahoo.com/q/pr?s={0}'.format(x))
page = requests.get('http://finance.yahoo.com/q/pr?s=' + x)
Related
I have been trying to get the multiple elements with the same class using 'nested Loop' and Selenium for Python.
The DataFrame used has a column that shows the URL, so I'm using this column in order to connect 'driver'
for i, url in enumerate(df['naver_store_url']):
driver.get(url+'/review/visitor')
#try
all_elements= driver.find_elements(by=By.CLASS_NAME, value = "WoYOw")
for comment in all_elements:
default_comment += comment.text +'/'
review_text_list.append(default_comment)
The code has been modified as follows:
try:
search_review = driver.find_elements(by=By.CLASS_NAME, value= "WoYOw")
for comment in search_review:
x = comment.get_attribute('innerText')
default_comment = default_comment +'/' + x
comment_house.append(default_comment)
Even if the element has characters, the old code can return an empty string. This is because the element does not have a text property and the value is set to a different property.
so,I had modified a piece of code as follows : .text -> .get_attribute('innerText')
*Please let me know if you have a better way :)
I have written a dynamic XPath as shown below:-
_FIRST_LIKE_TEXT_XPATH = "(//*[starts-with(#class,'%s-message')]/div/p)[%s]"
print(f"FIRST_LIKE_TEXT_XPATH with action and iteration = {self._FIRST_LIKE_TEXT_XPATH % action % str(itr)}")
action = like and itr = 1
expected output = (//*[starts-with(#class,'like-message')]/div/p)[1]
The error I am getting:-
TypeError: not enough arguments for format string
You use the python format function to pass variable in a string.
_FIRST_LIKE_TEXT_XPATH = "(//*[starts-with(#class,'{}-message')]/div/p)[{}]".format(action,str(itr))
I'm currently working on a selenium bot that gets a random english noun from a list of 1000 nouns, puts it in a site that gets similar instagram tags, then goes on instagram, logs into my account, and starts liking pictures with that hashtag, then starts the process all over again
My problem is the function that returns similar hashtags, this is the code for it:
def get_similar_tags(tag):
url = "https://top-hashtags.com/hashtag/" + str(tag).lower() + "/"
page = requests.get(url).text
parsed_page = BeautifulSoup(page, "html.parser")
parsed_page = parsed_page.find("div", class_="tht-tags")
text = parsed_page.text
hashtag = text.split(" ")
hashtag = [s.strip('#') for s in hashtag]
hashtag = hashtag.pop(len(hashtag) - 1)
return hashtag
When i try to print the hashtag variable inside of the function, then the output is filled with hashtags, which is exactly what i need.
When i use the function to assign the hashtag variable to another variable, like this:
foo = get_similar_tags(random_noun)
and then i do
print(foo)
i get absolutely nothing, not even an empty list, just an empty line.
i've already tried assigning the value returned by the get_similar_tags function using a global variable, but that doesn't work either.
Any help would be really apprecciated,
Thanks
hashtag = hashtag.pop(len(hashtag) - 1)
don't do that, try :
hashtag.pop(len(hashtag) - 1)
list.pop([i])
Remove the item at the given position in the list, and return it.
If no index is specified, a.pop() removes and returns the last item in
the list. (The square brackets around the i in the method signature
denote that the parameter is optional, not that you should type square
brackets at that position. You will see this notation frequently in
the Python Library Reference.)
https://docs.python.org/3/tutorial/datastructures.html
Say I have a BeautifulSoup Object named bs4. To use bs4's find_all_next function on a p tag I would just do:
bs4.p.find_all_next(string = True)
I want to throw this in a for loop for all tags available in the webpage:
temp_set = set()
for x in bs4.find_all():
temp_set.add(x.name) # Store only tag name, no dupes, order doesn't matter
However when it comes time use it in a bs4 object:
for x in temp_set:
bs4.x.find_all_next(string = True) # x is supposed to represent the tag name; attribute error
I know that there is no tag 'x' that exists, which is why I get the attribute error, is there anything I can do to make the 'x' in the loop symbolize the tag it's supposed to be representing as if I print it to console?
document states here:
getattr(x, 'foobar') is equivalent to x.foobar
So when I tried it, I just looped through:
getattr(bs4, x).find_all_next(string = True)
as the document says is equivalent to bs4.x.find_all_next(string = True) with x as the variable
I'm not sure what you want to do with it from there.
Not all the items in your set will work, so I just looped, threw them into a list, and then threw the exceptions into a list too just to see which didn't work. And 'select' will give a AttributeError: 'function' object has no attribute 'find_all_next'
so basically, here's what I did in that last section. Again, do what you want, but you'll loop through getattr(bs4, x).find_all_next(string = True)
final_list = []
failed_x = []
for x in temp_set:
try:
final_list.append(list(getattr(bs4, x).find_all_next(string = True)))
except:
failed_x.append(x)
continue
How do i parse my received variables into arrays. I receive them from a site and i want to insert them into my firebird database but it would be a lot faster if i could do that through parsing it into a list.
This is how my flask code looks:
#app.route('/fddatumupdate', methods=['GET'])
def fddatumupdate():
datums = request.args.get('datums')
IDS1 = request.args.get('ids1')
IDS2 = request.args.get('ids2')
IDS3 = request.args.get('ids3')
print datums
print IDS1
print IDS2
print IDS3
#cur.execute("UPDATE OR INSERT INTO T_FOOD_DUTY (F_FD_DATE, F_US_ID1, F_US_ID2, F_US_ID3) values(%s, %s, %s) matching (F_FD_DATE)")
return("great succes")
This is the printing output so you can see how my data looks:
2017-5-15,2017-5-16,2017-5-17,2017-5-18,2017-5-19,2017-5-20,2017-5-21
27,36,26,435,26,30,31
27,28,30,435,27,28,26
30,28,30,28,29,28,27
I always get the error when i try to parse them from an NoneType to a string or array:
TypeError: unsupported operand type(s) for +: 'NoneType' and ...
You can split your string by , character and you will get a list:
print datums.split(',')
Alternatively, you can use list comprehensions to construct your list with some extra checking:
# example code
if datums: # this will check if 'datums' is None
print [i if i > 0 for i in datums.split(',')] # include element in list only if it is larger than 0
Found the answer, I parsed my received variables while i was receiving them en putthing them into my local variables. It needs to be like this then:
datums = request.args.get('datums')
IDS1 = request.args.get('ids1')
IDS2 = request.args.get('ids2')
IDS3 = request.args.get('ids3')
datumArray = str(datums).split(',')
IDS1Array = str(IDS1).split(',')
IDS2Array = str(IDS2).split(',')
IDS3Array = str(IDS3).split(',')