We Keep Coding

Python - Recursive Loop is stuck

I want to set up a recursive loop to iterate over different levels of a transaction tree until a predefined depth is reached.
Example:
A transfers funds to B and C (depth 1)
- B transfers funds to D and E (depth 2)
- C transfers funds to F and G (depth 2)
etc.
To do so, I have set up the following function:
def traverse(target, depth):
try:
print('Starting analysis of parent address {}'.format(target))
result=callAddress(target)
inner_transaction_targets=analyseTransaction(result)
traversed_addresses=compileAddresses(compiled_receiver_addresses,inner_transaction_targets) #Compile list of start addresses
if depth > 0:
for inner in inner_transaction_targets:
print('Starting analysis of child address {} at depth {}.'.format(inner,depth))
inner_transaction_targets=inner_transaction_targets.pop(0)
print(inner_transaction_targets)
return traverse(inner, depth-1)
else:
return traversed_addresses
except:
print('Could not traverse.')
The functions callAdress, analyseTransaction and compileAdresses themselves should not be relevant to this problem. However to offer a brief description:
analyseTransaction will offer a list of all receiver addresses involved in a given transaction
compileAdresses collects the scanned addresses in a master list
See here the other functions:
def callAddress(bitcoin_address): #This function serves to retrieve the JSON respose from blockchain.info
try:
r = requests.get("https://blockchain.info/address/{}?format=json".format(bitcoin_address))
unparsed=r.content
parsed = json.loads(unparsed)
return parsed
except: #unparsed.status=="400":
print ("Incorrect bitcoin address")
def compileAddresses(compiled_receiver_addresses,input_addr): #This function serves to add extracted transactions to a master list (compiled_receiver_addresses)
try:
compiled_receiver_addresses.append(input_addr)
return compiled_receiver_addresses
except:
print('Failed to compile')
def analyseTransaction(response): #This function serves to extract the wallet addresses that the specified wallet has sent funds to (transaction targets)
try:
transaction_targets=[]
wallet_transactions=response['txs']
for txs_entry in wallet_transactions:
print ("{:=^22}".format(""))
print ("Checking outgoing transaction with Hash: {}".format(txs_entry['hash']))
print ("Transaction occured at: {}".format(datetime.fromtimestamp(
int(txs_entry['time']) ).strftime('%Y-%m-%d %H:%M:%S')))
print ("{:=^22}".format(""))
for out in txs_entry['out']:
outbound_addr=out['addr']
print ("In this transaction funds were sent to this wallet: {}".format(out['addr']))
print ("Amount sent: {}".format(int(out['value']) * 10**-8))
transaction_targets.append(outbound_addr)
cleaned_transaction_targets = list(dict.fromkeys(transaction_targets))
return cleaned_transaction_targets
except:
print('Dead End')
My issue is that the recursive loop keeps getting stuck on the first value, i.e. it is looping the very first value of "inner" over and over again instead of going down the list that is inner_transaction_targets
I have tried to drop the very first value in this list by using inner_transaction_targets.pop(0) but to no effect.
How do I have to modify this function so that it does what I want?

Your code has some errors that are most likely hindering its successful completion, such as:
The return is within a loop.
The object returned is in fact a self-reference to the traverse() function.
The .pop() method is not a good practice.
I would suggest you create a separate function that will return a single object for each inner transaction, much like what compile_addresses() does.
You then add this new function into the for loop so that it iterates over the inner_transaction_targets and appends the results to a list. Finally, you return the list with all the results, indented aligned with the if statement, outside the for loop.

How to use If/Then with IndexError

This program is using a website's API to scrape the latest sale. This program works fine for products that have recent sales, but not for one's that don't have a recent sale within the last day.
The array is [] and I of course get the IndexError: list index is out of range.
Here is my code:
import requests
cybersole_url = 'https://www.botbroker.io/bots/6/chart?key_type=lifetime&days=1'
response = requests.get(cybersole_url)
response.raise_for_status()
if (response.json()[0][1] == None):
cyber = "No recent sales."
else:
cyber = "$" + str(response.json()[0][1])
How can I work around this error to get one of the two results listed in my if statement? I believe I used try and except, but it only performed the except even when it had objects in the array.

import requests
cybersole_url = 'https://www.botbroker.io/bots/6/chart?key_type=lifetime&days=1'
response = requests.get(cybersole_url)
response.raise_for_status()
# Try to index the result, otherwise set result=None
try:
result = response.json()[0][1]
except IndexError:
result = None
cyber = 'No recent sales.' if not result else f'${result}'
Note you might want to add another layer of try-catching since you not only want to grab the element at [0], but also the element at [0][1] – there are two layers of indexing here.

Loop iterating over a list of BeautifulSoup Parse Tree Elements terminating too early

I am taking an Intro to Data Science Course and my first task is to extract certain Data Fields from each country page from the CIA World Factbook. Although I have recently become aware that there is easier ways to locate the data, I would like to follow through on my initial thought process which is as follows.
I developed a function which iterates over the result of:
for link in fulllink:
with urlopen(link) as countrypage:
countrysoup = BeautifulSoup(countrypage, "lxml")
data = countrysoup.find_all(class_="category_data")
I have confirmed that if the String Values I require exist on a country page they will be present in "data". The following function takes a tag and works with .parent and .previous_sibling to determine that the string value attached to tag is the one I am interested in extracting.
def get_wfb_data(x):
country_data = [0,0,0,0]
for tag in x:
try:
if 'Area:' in tag.parent.previous_sibling.previous_sibling.strings and 'total: ' in tag.parent.strings:
country_data[0]=tag.string
elif 'GDP (purchasing power parity):' in tag.previous_sibling.previous_sibling.strings:
country_data[1]=tag.string
elif 'Roadways:' in tag.parent.previous_sibling.previous_sibling.strings and 'total: ' in tag.parent.strings:
country_data[2]=tag.string
elif 'Railways:' in tag.parent.previous_sibling.previous_sibling.strings and 'total: ' in tag.parent.strings:
country_data[3]=tag.string
else:
continue
except:
pass
return country_data
Exception Handling is used to deal with NavigableString Objects which do not have such attributes and thus raise an exception. Replacing values in a list of zeroes allows me to handle situations where a particular region has no data listed in one of the four categories I am interested in extracting. Furthermore, defined as four separate functions, the respective criterion work, but together are extremely slow as the list must be iterated over at most our times. However, the end result from this function is always a list where the first zero has been replaced but the others have not like as follows
["Total Area #",0,0,0].
I believe the loop terminates after matching the first if statement to a tag in x, how can I repair my function to continue down x?

I'm making the assumption that you're making this call: get_wfb_data(data)
Your function never fails, it simply never matches the rest of your conditions.
I tried it on a couple of different links (each contained data for GDP, Roadways, etc.)
By checking the length of x and printing the number of iterations through the for loop you can be assured that the loop is not terminating after matching the first condition and then failing to reach the last data element. It is in fact the remaining conditions that are never satisfied.
def get_wfb_data(x):
country_data = [0, 0, 0, 0]
print(len(x))
i = 0
for tag in x:
i += 1
try:
if 'Area:' in tag.parent.previous_sibling.previous_sibling.strings \
and 'total: ' in tag.parent.strings:
country_data[0] = tag.string
elif 'GDP (purchasing power parity):' in tag.previous_sibling.previous_sibling.strings:
country_data[1] = tag.string
elif 'Roadways:' in tag.parent.previous_sibling.previous_sibling.strings \
and 'total: ' in tag.parent.strings:
country_data[2] = tag.string
elif 'Railways:' in tag.parent.previous_sibling.previous_sibling.strings \
and 'total: ' in tag.parent.strings:
country_data[3] = tag.string
else:
continue
except:
pass
print(str(i))
return country_data

Extracting data from dictionary in python

I have a program from Dr.Chuck to print the sum of the counts from this data. The problem is. The count of the JSON is showing "2" when there are many..
import json
import urllib
url="http://python-data.dr-chuck.net/comments_42.json"
uh = urllib.urlopen(url)
data = uh.read()
print 'Retrieved',len(data),'characters'
print data
info = json.loads(data)
print 'User count:', len(info)
This line print 'User count:', len (info) is showing an output of 2. When there is a lot of data, hence I can only access 2 datas and not the rest.
I have no idea why. I can solve the counting sum part. Just not getting why am I only getting access to the first 2 data and the rest of the JSON is getting ignored.

The json has two top level properties: note and comments. That is why you get a length of 2.
This will probably give you what you want:
len(info["comments"])

To count the number of comments:
print 'User count:', len(info["comments"])
To print the total "count":
count = 0
for comment in info["comments"]:
count += comment["count"]
print 'Total count:', count

So, your json parsed to dict like
{"note":"bla", "comments":[...]}
Length of this should be 2 because it's only two keys in this dict. Right way to do you case is get comments itself and count them.
For example:
len(data.get('comments',[]))

The Json is composed from note and comments. Inside comments there's another array of object.
if you wanna access to that array you have to use this info['comments'] and then, if you want the length of that array, as you do, you can use len(info['comments'])

Filter twitter files by location

I am trying to find lat/long information for numerous tweets. One path to a tweet lat/long data in a json tweet is,
{u'location: {u'geo': {u'coordinates : [120.0,-5.0]}}}
I want to be able to check each tweet if this location path exists. If it does then I want to use that information in a function later on. If it does not then I want to check another location path and finally move on to the next tweet.
Here is the code I have currently to check if this path exists and if there is corresponding data. 'data' is a list of twitter files that I opened using the data.append(json.loads(line)) method.
counter = 0
for line in data:
if u'coordinates' in data[counter][u'location'][u'geo']:
print counter, "HAS FIELD"
counter += 1
else:
counter += 1
print counter, 'no location data'
I get a KeyError error with this code. If I just do the below code it works, but is not specific enough to actually get me to the information that I need.
counter = 0
for line in data:
if u'location' in data[counter]:
print counter, "HAS FIELD"
counter += 1
else:
counter += 1
print counter, 'no location data'
Does anyone have a way to do this.
Below is some more background on what I am doing overall, but the above sums up where I am stuck.
Background: I have access to 12 billion tweets, purchased through gnip, that are divided up into multiple files. I am trying to comb through those tweets one-by-one and find which ones have location (lat/long) data and then see if the corresponding coordinates fall in a certain country. If that tweet does fall in that country I will add it to a new database which is a subset of the my larger database.
I have successfully created a function to test if a lat/long falls in the bounding box of my target country, but I am having difficulty populating the lat/long for each tweet for 2 reasons. 1) There are multiple places that long/lat data are stored in each json file, if it exists at all. 2) The tweets are organized in a complex dictionary of dictionaries which I have difficulty maneuvering through.
I need to be able to loop through each tweet and see if a specific lat/long combination exists for the different location paths so that I can pull it and feed it into my function that tests if that tweet originated in my country of interest.

I get a KeyError error with this code
Assume keys should be in double quotes because they have ':
counter = 0
for line in data:
if "u'coordinates" in data[counter]["u'location"]["u'geo"]:
print counter, "HAS FIELD"
counter += 1
else:
counter += 1
print counter, 'no location data'

The solution that I found may not be the most efficient, but is functional. It uses if statements nested in try-except statements. This allows me to check different location paths but push through KeyError s so that I can move on to other tweets and pathways. Below is my code. It goes through multiple tweets and checks for ones that have an available Lat/Long combos in any of 3 pathways. It works with my addTOdb function that checks to see if that Lat/Long combo is in my target country. It also builds a separate dictionary called Lat Long where I can view all the tweets that had Lat/Long combos and what path I pulled them through.
#use try/except function to see if entry is in json files
#initialize counter that numbers each json entry
counter = 0
#This is a test dict to see what lat long was selected
Lat_Long = {}
for line in data:
TweetLat = 0
TweetLong = 0
#variable that will indicate what path was used for coordinate lat/long
CoordSource = 0
#Sets while variable to False. Will change if coords are found.
GotCoord = False
while GotCoord == False:
#check 1st path using geo function
try:
if u'coordinates' in data[counter][u'geo'] and GotCoord == False:
TweetLat = data[counter][u'geo'][u'coordinates'][0]
TweetLong = data[counter][u'geo'][u'coordinates'][1]
#print 'TweetLat',TweetLat
print counter, "HAS FIELD"
addTOdb(TweetLat,TweetLong,North,South,East,West)
CoordSource = 1
GotCoord = True
except KeyError:
pass
#check 2nd path using gnip info
try:
if u'coordinates' in data[counter][u'gnip'][u'profileLocations'][0][u'geo'] and GotCoord == False:
TweetLat = data[counter][u'gnip'][u'profileLocations'][0][u'geo'][u'coordinates'][1]
TweetLong = data[counter][u'gnip'][u'profileLocations'][0][u'geo'][u'coordinates'][0]
print counter, "HAS FIELD"
addTOdb(TweetLat,TweetLong,North,South,East,West)
CoordSource = 2
GotCoord = True
except KeyError:
pass
#check 3rd path using location polygon info
try:
if u'coordinates' in data[counter][u'location'][u'geo'] and GotCoord == False:
TweetLat = data[counter][u'location'][u'geo'][u'coordinates'][0][0][1]
TweetLong = data[counter][u'location'][u'geo'][u'coordinates'][0][0][0]
print counter, "HAS FIELD"
addTOdb(TweetLat,TweetLong,North,South,East,West)
CoordSource = 3
GotCoord = True
except KeyError:
pass
if GotCoord==True:
Lat_Long[counter] = [CoordSource,TweetLat, TweetLong]
else:
print counter, "no field"
GotCoord = True
counter += 1