We Keep Coding

I would like to get all transactions given an address

Given a Solana wallet address I would like to verify every single transaction ever confirmed to check other information, such as the receiver (or sender) and the amount sent (or received). So, as usual, I searched for some APIs. I found the following:
Solana py
PySolana
After that, I went to look which methods they offer. The one that seems to be close to what I wish is solana_client.get_confirmed_signature_for_address2 (available in 1), however my results do not match what its documentation shows. Here it is:
from solana.rpc.api import Client
solana_client = Client("https://api.devnet.solana.com")
solana_client.get_signatures_for_address("2AQdpHJ2JpcEgPiATUXjQxA8QmafFegfQwSLWSprPicm", limit=1)
I get this:
{'jsonrpc': '2.0', 'result': [], 'id': 1}
However, I should get its last signature, which seems to be this:
4SNQ4h1vL9GkmSnojQsf8SZyFvQsaq62RCgops2UXFYag1Jc4MoWrjTg2ELwMqM1tQbn9qUcNc4tqX19EGHBqC5u
Anyways, we can use SolanaBeach and check. Further, if we code as the documentation explains:
from solana.rpc.api import Client
solana_client = Client("https://api.devnet.solana.com")
solana_client.get_signatures_for_address("Vote111111111111111111111111111111111111111", limit=1)
I get this:
{'jsonrpc': '2.0', 'result': [{'blockTime': 1637328065, 'confirmationStatus': 'finalized', 'err': {'InstructionError': [0, {'Custom': 0}]}, 'memo': None, 'signature': '5yaeqDRCHWCGQMqNWhq3g6zqw63MBkri9i86hjK954YFFvnG2VCQJfszXsozDVUJbePagJieAzwsSY5H7Xd1jJhC', 'slot': 95301596}], 'id': 1}
Weird thing is "Vote111...11" seems not to be an address. Nevertheless, I get expected results, that is a signature, even though such signature can't be found by Solana Explorer...
Please, tell me what to fix. I have no idea what to do. I even tried to check if all Solana Explorers have their own API, but they do not. Probably because Solana already shares it, right?
EDIT
Well, it seems I need to enter the "account address as base-58 encoded string", thus the address becomes: HLiBGYYxaQqQx8UTPHEahqcd7aZjkDgN3bihc3hYM3SDUBGU9LFrQSnx9eje.
I also did that and I get:
{'jsonrpc': '2.0', 'error': {'code': -32602, 'message': 'Invalid param: WrongSize'}, 'id': 1}

I have implemented the function to get all the transactions of a given address on javaScript this might help you out.
async getTransactionsOfUser(address, options, connection) {
console.log({ address, options });
try {
const publicKey = new PublicKey(address);
const transSignatures =
await connection.getConfirmedSignaturesForAddress2(publicKey, options);
console.log({ transSignatures });
const transactions = [];
for (let i = 0; i < transSignatures.length; i++) {
const signature = transSignatures[i].signature;
const confirmedTransaction = await connection.getConfirmedTransaction(
signature,
);
if (confirmedTransaction) {
const { meta } = confirmedTransaction;
if (meta) {
const oldBalance = meta.preBalances;
const newBalance = meta.postBalances;
const amount = oldBalance[0] - newBalance[0];
const transWithSignature = {
signature,
...confirmedTransaction,
fees: meta?.fee,
amount,
};
transactions.push(transWithSignature);
}
}
}
return transactions;
} catch (err) {
throw err;
}
}

Problem is not module nor function but endpoint.
In Solana Doc I found endpoint for mainnet:
https://api.mainnet-beta.solana.com
https://solana-api.projectserum.com
and it gives all values.
On other page you can see that
devnet is only playground for tests and tokens are not real
testnet is only for stress test and tokens are not real
#Devnet#
- Devnet serves as a playground for anyone who wants to take Solana for a test drive, as a user, token holder, app developer, or validator.
- Application developers should target Devnet.
- Potential validators should first target Devnet.
- Key differences between Devnet and Mainnet Beta:
- Devnet tokens are not real
- Devnet includes a token faucet for airdrops for application testing
- Devnet may be subject to ledger resets
- Devnet typically runs a newer software version than Mainnet Beta
#Testnet#
-Testnet is where we stress test recent release features on a live cluster, particularly focused on network performance, stability and validator behavior.
- Testnet tokens are not real
- Testnet may be subject to ledger resets.
- Testnet includes a token faucet for airdrops for application testing
- Testnet typically runs a newer software release than both Devnet and Mainnet Beta
Minimal working example for tests:
from solana.rpc.api import Client
all_addresses = [
'2AQdpHJ2JpcEgPiATUXjQxA8QmafFegfQwSLWSprPicm',
'Vote111111111111111111111111111111111111111',
'fake address',
]
#endpoint = 'https://api.devnet.solana.com' # probably for `developing`
#endpoint = 'https://api.testnet.solana.com' # probably for `testing`
endpoint = 'https://api.mainnet-beta.solana.com'
#endpoint = 'https://solana-api.projectserum.com'
solana_client = Client(endpoint)
for address in all_addresses:
print('address:', address)
#result = solana_client.get_confirmed_signature_for_address2(address, limit=1)
result = solana_client.get_signatures_for_address(address)#, before='89Tv9s2uMGaoxB8ZF1LV9nGa72GQ9RbkeyCDvfPviWesZ6ajZBFeHsTPfgwjGEnH7mpZa7jQBXAqjAfMrPirHt2')
if 'result' in result:
print('len:', len(result['result']))
# I use `[:5]` to display only first 5 values
for number, item in enumerate(result['result'][:5], 1):
print(number, 'signature:', item['signature'])
# check if there is `4SNQ4h1vL9GkmSnojQsf8SZyFvQsaq62RCgops2UXFYag1Jc4MoWrjTg2ELwMqM1tQbn9qUcNc4tqX19EGHBqC5u`
for number, item in enumerate(result['result'], 1):
if item['signature'].startswith('4SN'):
print('found at', number, '>>>', item['signature'])
else:
# error message
print(result)
print('---')
#solana_client.get_account_info(address)
Result:
address: 2AQdpHJ2JpcEgPiATUXjQxA8QmafFegfQwSLWSprPicm
len: 1000
1 signature: 89Tv9s2uMGaoxB8ZF1LV9nGa72GQ9RbkeyCDvfPviWesZ6ajZBFeHsTPfgwjGEnH7mpZa7jQBXAqjAfMrPirHt2
2 signature: 3Ku2rDnAVo5Mj3r9CVSGHJjvn4H9rxzDvc5Cg5uyeCC9oa6p7enAG88pSfRfxcqhBh2JiWSo7ZFEAD3mP8teS1Yg
3 signature: 3wiYCmfXb9n6pT3mgBag7jx6jBjeKZowkYmeakMibw4GtERFyyitrmmoPU6t28HpJJgWkArymWEGWQj8eiojswoD
4 signature: 5vjV1wKU3ZEgyzqXCKrJcJx5jGC8LPqRiJBwhPcu62HQU64mkrvkK8LKYaTzX4x4p26UXSufWM57zKSxRrMgjWn3
5 signature: 3aLk4xZPcWRogtvsFe8geYC177PK8s47mgqUErteRc9NJ4EF2iHi3GPsaj5guTwyiabhwivFhrrEk4YQgiE2hZs8
found at 970 >>> 4SNQ4h1vL9GkmSnojQsf8SZyFvQsaq62RCgops2UXFYag1Jc4MoWrjTg2ELwMqM1tQbn9qUcNc4tqX19EGHBqC5u
---
address: Vote111111111111111111111111111111111111111
len: 1000
1 signature: 67RRbUWGCrwmJ3hhLL7aB2K8gc6MewxwgAdfG7FeXQBaSstacqvuo9QUPZ6nhqXjJwYpKHihNJwFfcaAZHuyFmMc
2 signature: 67PsyRRw8bXgtsB49htxcW2FE9cyyBrocUKacnrxJpqaBpFT6QDLrCkyovWnM8XyGKxTv3kqzmW72SH7gj3N8YJr
3 signature: 675FWqrAjE5Bt6rf3KD2H2PCKUmEtrcD8BRRypdS7m2V22zXhrGn3SktP6JYW4ws6xEqDj52MZMH8RwNjoqgW4mt
4 signature: 671K7N9FwaMAyBC4MEYbYb1ACYAendBbRMqKPvr3h63dt5ybAPHyppjHwxq1yPDjqaRUwCBVU9o5dVqgsdVabint
5 signature: 666jBXXLwmB5tuvufhNn8Q7A3eCzGo6CBFD5BYJkuGfBf1bRoAGz4DeEpUAKsUrRk4NdRBhYkwfrhyZjgFmo3Dp2
---
address: fake address
{'jsonrpc': '2.0', 'error': {'code': -32602, 'message': 'Invalid param: Invalid'}, 'id': 3}
---
BTW:
Because it gets only 1000 values you may not see 4SNQ... which is at position ~1200 at this moment, but if you use before=
get_signatures_for_address(address, before='89Tv9s2uMGaoxB8ZF1LV9nGa72GQ9RbkeyCDvfPviWesZ6ajZBFeHsTPfgwjGEnH7mpZa7jQBXAqjAfMrPirHt2')
then it should be at position ~970
BTW:
On Solana Explorer you have big button to change Mainnet to Devnet and when you use Devnet then
2AQdpHJ2JpcEgPiATUXjQxA8QmafFegfQwSLWSprPicm also gives 0 items.
The same on Solana Beach. There is also big button to change Mainnet to Devnet and when you use Devnet then
2AQdpHJ2JpcEgPiATUXjQxA8QmafFegfQwSLWSprPicm gives 0 items.

funny I was working on this exact same issue this morning.. and just like furas pointed out, it's the endpoint, need to use the mainnet endpoint:
https://api.mainnet-beta.solana.com
And I found it's bit confusing even though the doc says you need to input base-58 address, I tried same as you did it gives me the same error, turns out I just need to copy paste my address directly

Retrieving all comments from a thread on Reddit

I’m new to API’s and working with JSON and would love some help here.
I know everything I’m trying to accomplish can be done using the PRAW library, but I’m trying to figure it out without PRAW.
I have a for loop that pulls post titles from a specific subreddit, inputs all the post titles into a pandas data frame, and after the limit is reached, changes the ‘after parameter to the last post id so it repeats with the next batch.
Everything worked perfectly, but when I tried the same technique with a specific thread and gathering the comments, the ‘after’ parameter doesn’t work to grab the next batch.
I’m assuming ‘after’ works differently with threads than with a subreddits posts. I saw in the JSON ‘more’ with a list of ids. Do I need to use this somehow? When I looked at the JSON for the thread, the ‘after’ says ‘none’ even with the updated parameters.
Any idea on what I need to change here? It’s probably something simple.
Working code for getting the subreddit posts with limit 5:
params = {"t":"day","limit":5}
for i in range(2):
response = requests.get('https://oauth.reddit.com/r/stocks/new',
headers=headers, params = params)
response = response.json()
for post in response['data']['children']:
name = post['data']['name']
print('name',name)
params['after'] = name
print(params)
Giving the output:
name t3_lifixn
name t3_lifg68
name t3_lif6u2
name t3_lif5o2
name t3_lif3cm
{'t': 'day', 'limit': 5, 'after': 't3_lif3cm'}
name t3_lif26d
name t3_lievhr
name t3_liev9i
name t3_liepud
name t3_lie41e
{'t': 'day', 'limit': 5, 'after': 't3_lie41e'}
Code for the Reddit thread with limit 10
params = {"limit":10}
for i in range(2):
response = requests.get('https://oauth.reddit.com/r/wallstreetbets/comments/lgrc39/',
params = params,headers=headers)
response = response.json()
for post in response[1]['data']['children']:
name = post['data']['name']
print(name)
params['after'] = name
print(params)
Giving the output:
t1_gmt20i4
t1_gmzo4xw
t1_gmzjofk
t1_gmzjkcy
t1_gmtotfl
{'limit': 10, 'after': 't1_gmtotfl'}
t1_gmt20i4
t1_gmzo4xw
t1_gmzjofk
t1_gmzjkcy
t1_gmtotfl
{'limit': 10, 'after': 't1_gmtotfl'}
Even though the limit was set to 10, it only gave 5 id's before continuing the loop. Also, rather than updating the 'after' parameter, it just restarted.

I ended up figuring out how to do it. Reading the documentation for Reddit's API, when in a thread and you want to pull more comments, you have to compile a list of the id's from the more sections in the JSON. It's a nested tree and looks like the following:
{'kind': 'more', 'data': {'count': 161, 'name': 't1_gmuram8', 'id': 'gmuram8', 'parent_id': 't1_gmt20i4', 'depth': 1, 'children': ['gmuram8', 'gmt6mf6', 'gmubxmr', 'gmt63gl', 'gmutw5j', 'gmtpitn', 'gmtoec3', 'gmtnel0', 'gmt4p79', 'gmupqhx', 'gmv70rm', 'gmtu2sj', 'gmt2vc7', 'gmtmjai', 'gmtje0b', 'gmtkzzj', 'gmt93n5', 'gmtvsqa', 'gmumhat', 'gmuj73q', 'gmtor7c', 'gmuqcwv', 'gmt3lxe', 'gmt4l78', 'gmum9cm', 'gmt857f', 'gmtjrz3', 'gmu0qcl', 'gmt9t9i', 'gmt8jc7', 'gmurron', 'gmt3ysv', 'gmt6neb', 'gmt4v3x', 'gmtoi6t']}}
When using the get request, you would use the following url and format
requests.get(https://oauth.reddit.com/api/morechildren/.json?api_type=json&link_id=t3_lgrc39&children=gmt20i4,gmuram8....etc)

JSON from (RIOT) API Formatted Incorrectly

I am importing JSON data into Python from an API and ran into the following decode error:
JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)
Looking at online examples its immediately clear my JSON data has ' where others have have ".
Ideally, I'd like to know why it's being downloaded in this way. It seems highly likely its an error on my end, not theirs.
I decided it should be easy to correct the JSON format but I have failed here too. Please see the below code for how I obtain the JSON data and my attempt at fixing it.
#----------------------------------------
#---Read this only if you want to download
#---the data yourself.
#----------------------------------------
#Built from 'Towards Data Science' guide
#https://towardsdatascience.com/how-to-use-riot-api-with-python-b93be82dbbd6
#Must first have installed riotwatcher
#Info in my example is made up, I can't supply a real API code or
#I get in trouble. Sorry about this. You could obtain one from their website
#but this would be a lot of faff for what is probably a simple StackOverflow
#question
#If you were to get/have a key you could use the following information:
#<EUW> for region
#<Agurin> for name
#----------------------------------------
#---Code
#----------------------------------------
#--->Set Variables
#Get installed riotwatcher module for
#Python
import riotwatcher
#Import riotwatcher tools.
from riotwatcher import LolWatcher, ApiError
#Import JSON (to read the JSON API file)
import json
# Global variables
# Get new API from
# https://developer.riotgames.com/
api_key = 'RGAPI-XXXXXXX-XXX-XXXX-XXXX-XXXX'
watcher = LolWatcher(api_key)
my_region = 'MiddleEarth'
#need to give path to where records
#are to be stored
records_dir = "/home/solebaysharp/Projects/Riot API/Records"
#--->Obtain initial data, setup new varaibles
#Use 'watcher' to get basic stats and setup my account as a variable (me)
me = watcher.summoner.by_name(my_region, "SolebaySharp")
# Setup retrieval of recent match info
my_matches = watcher.match.matchlist_by_account(my_region, me["accountId"])
print(my_matches)
#--->Download the recent match data
#Define where the JSON data is going to go
recent_matches_index_json = (records_dir + "/recent_matches_index.json")
#get that JSON data
print ("Downloading recent match history data")
file_handle = open(recent_matches_index_json,"w+")
file_handle.write(str(my_matches))
file_handle.close()
#convert it to python
file_handle = open(recent_matches_index_json,)
recent_matches_index = json.load(file_handle)
Except this giver the following error...
JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)
So instead to correct this I tried:
file_handle = open(recent_matches_index_json)
json_sanitised = json.loads(file_handle.replace("'", '"'))
This returns...
AttributeError: '_io.TextIOWrapper' object has no attribute 'replace'
For the sake of completeness, beneath is a sample of what the JSON looks like. I have added the paragraphs to enhance readability. It does not come this way.
{'matches': [
{'platformId': 'NA1',
'gameId': 5687555181,
'champion': 235,
'queue': 400,
'season': 13,
'timestamp': 1598243995076,
'role': 'DUO_SUPPORT',
'lane': 'BOTTOM'
},
{'platformId': 'NA1',
'gameId': 4965733458,
'champion': 235,
'queue': 400,
'season': 13,
'timestamp': 1598240780841,
'role': 'DUO_SUPPORT',
'lane': 'BOTTOM'
},
{'platformId': 'NA1',
'gameId': 4583215645,
'champion': 111,
'queue': 400,
'season': 13,
'timestamp': 1598236666162,
'role': 'DUO_SUPPORT',
'lane': 'BOTTOM'
}],
'startIndex': 0,
'endIndex': 100,
'totalGames': 186}

This is occurring because Python is converting the JSON to a string (str).
file_handle.write(str(my_matches))
As it doesn't see the difference between ' and " it just goes with its default of '.
We can stop this from happening by using JSON.dumps. This partially (certainly not fully) answers the second part of the question as well, as we're using a correctly formatted JSON command.
The above-mentioned line simply has to be replaced with this:
file_handle.write(json.dumps(my_matches))
This will preserve the JSON formatting.

How to add a local variable to pymongo.find if it's not None?

I run a web service with an api function which uses a method I created to interact with MongoDB, using pymongo.
The json data comes with post may or may not include a field: firm. I don't want to create a new method for posts that does not include a firm field.
So I want to use that firm in pymongo.find if it does exists, or I want to just skip it if it doesn't. How can I do this with using one api function and one pymongo method?
API function:
#app.route(f'/{API_PREFIX}/wordcloud', methods=['POST'])
def generate_wc():
request_ = request.get_json()
firm = request_.get("firm").lower()
source = request_["source"]
since = datetime.strptime(request_["since"], "%Y-%m-%d")
until = datetime.strptime(request_["until"], "%Y-%m-%d")
items = mongo.get_tweets(firm, since, until)
...
The pymongo method:
def get_tweets(self, firm: str, since: datetime, until: datetime):
tweets = self.DB.tweets.find(
{
# use firm here if it exists (I mean not None), else just get items by date
'date': {'$gte': since, '$lte': until}
})
...
Here in the second code, comment line in find.
Thanks.

Since it involves two different queries: {date: ...} and {date: ..., firm: ...} depending on the existence of firm in the input, you would have to check if firm is not None in get_tweets and execute the proper query.
For example:
def get_tweets(self, since, until, firm=None):
query = { 'date': { '$gte': since, '$lte': until } }
if firm is not None:
query['firm'] = firm
tweets = self.DB.tweets.find(query)
....
Note that since firm has a default value, it needs to be last in the get_tweets parameter list.

Boto - How to delete a record set from route53 -Tried to delete resource record set but it was not found

I am using the following to delete route53 records. I get no error messages.
conn = Route53Connection(aws_access_key_id, aws_secret_access_key)
changes = ResourceRecordSets(conn, zone_id)
change = changes.add_change("DELETE",sub_domain, "A", 60,weight=weight,identifier=identifier)
change.add_value(ip_old)
changes.commit()
all required fields are present and they match..weight, identifier,
ttl=60 etc.\
e.g.
test.com. A 111.111.111.111 60 1 id1
test.com. A 111.111.111.222 60 1 id2
I want to delete 111.111.111.222 and the record set.
So, what is the proper way to delete a record set?
For a record set, I will have multiple values that are distinguished
by a unique identifier. When an ip address becomes in active I want
to remove from route53. I am using a a poor mans load balancing.
Here is the meta of the record want to delete.
{'alias_dns_name': None,
'alias_hosted_zone_id': None,
'identifier': u'15754-1',
'name': u'hui.com.',
'resource_records': [u'103.4.xxx.xxx'],
'ttl': u'60',
'type': u'A',
'weight': u'1'}
Traceback (most recent call last):
File "/home/ubuntu/workspace/rtbopsConfig/classes/redis_ha.py", line 353, in <module>
deleteRedisSubDomains(aws_access_key_id, aws_secret_access_key,platform=platform,sub_domain=sub_domain,redis_domain=redis_domain,zone_id=zone_id,ip_address=ip_address,weight=1,identifier=identifier)
File "/home/ubuntu/workspace/rtbopsConfig/classes/redis_ha.py", line 341, in deleteRedisSubDomains
changes.commit()
File "/usr/local/lib/python2.7/dist-packages/boto-2.3.0-py2.7.egg/boto/route53/record.py", line 131, in commit
return self.connection.change_rrsets(self.hosted_zone_id, self.to_xml())
File "/usr/local/lib/python2.7/dist-packages/boto-2.3.0-py2.7.egg/boto/route53/connection.py", line 291, in change_rrsets
body)
boto.route53.exception.DNSServerError: DNSServerError: 400 Bad Request
<?xml version="1.0"?>
<ErrorResponse xmlns="https://route53.amazonaws.com/doc/2011-05-05/"><Error><Type>Sender</Type><Code>InvalidChangeBatch</Code><Message>Tried to delete resource record set hui.com., type A, SetIdentifier 15754-1 but it was not found</Message></Error><RequestId>9972af89-cb69-11e1-803b-7bde5b9c457d</RequestId></ErrorResponse>
Thanks

Are you sure you need all of those parameters for add_change?
Look at add_change here.
Default parameters are given to the function, so you may be over-specifying by providing weight and TTL.
Try leaving weight and TTL out (you may need to keep identifier). This blog provides a simple example of deleting records:
Also, I can't see the values of your parameters that you're passing, but ensure their integrity and try including a '.' at the end of your subdomain

I tried similar example and had to specify all fields including weight and ttl for a successful deletion. (By keeping it default, it did not work). Could not produce the original problem with weighted DNS record and explicitly passed ttl.

boto3
import boto3
client = boto3.connect('route53')
hosted_zone_id = "G5LEP7LWYS8WL2"
response = client.change_resource_record_sets(
ChangeBatch={
'Changes': [
{
'Action': 'DELETE',
'ResourceRecordSet': {
'Name': 'test.example.com',
'ResourceRecords': [
{
'Value': '10.1.1.1',
},
],
'Type': 'A',
},
},
]
},
HostedZoneId=hosted_zone_id,
)
print(response)
Source: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/route53.html#Route53.Client.change_resource_record_sets
boto
import boto
from boto.route53.record import ResourceRecordSets
conn = boto.connect_route53()
hosted_zone_id = "G5LEP7LWYS8WL2"
record_sets = ResourceRecordSets(conn, hosted_zone_id)
change = record_sets.add_change("DELETE", "test.example.com", "A")
change.add_value("10.1.1.1")
response = record_sets.commit()
print(response)
Source: https://danieljamesscott.org/17-software/development/33-manipulating-aws-route53-entries-using-boto.html