Compare 2 JSON files in python and check if item is changed - python

Maybe the title is a bit confusing but my brain is stuck on this for like days now. So here is my "problem". I am redoing my Discord Bot and I have a small task that will check 2 JSON files (same format, different data) and there is a difference between them it will send a message.
This code needs to load 2 JSON files ("VATSIM_DATA_old.json & VATSIM_Data_new.json") and compare them and find differences. Now for simplicity, I reduced code so I just get if someone came online and if they are no longer online.
So my flow for this is something like this:
-When bots start, fetch JSON data and save it to file so we can compare updated data with this file,
-Now it is time to check if someone came online by fetching new data from the same link and saving it to the new file,
-Load both files as JSON so we can find all items inside of them and compare if some items are present in both or only in one of them,
-After comparing them and doing actions such as sending messages to the channel let's save that new data to our Old_Data.json file and go to sleep for X min (I set it to 5),
The problem I have is that it will send the same message every time a task is started. So my logic here was ok let's check if files are changing and they are but the code still somehow ignores part of the code that tells him if data is not changed here simply pass.
So here is my code:
atc = ["ADR_CTR", "ADR_W_CTR", "ADR_U_CTR", "ADR_E_CTR"] #this is short version of array which needs to be checked. This will most likely be moved to the config file so code is cleaner
#tasks.loop(minutes=5)
async def atcAnnoucements(self):
#Load old and new VATSIM Data for comparing!
with open("VATSIM_Data_old.json", encoding="utf-8") as f:
data = json.load(f)
file = open("VATSIM_Data_new.json", "w", encoding="utf-8")
file.write(requests.get('https://data.vatsim.net/vatsim-data.json').text)
file.close()
with open("VATSIM_Data_new.json", encoding="utf-8") as s:
data1 = json.load(s)
#Check if new VATSIM data contains some changes in online atc
#If callsign from atc array is present in VATSIM_Data_old it will check is same controller active or not
#If the controller is not changed it will skip, if new controller comes online it will send a message that the ATC station came online
for item in data1['clients']:
if item['callsign'] in data['clients']:
pass
else:
if item['callsign'] in atc:
time_start = item['time_logon']
time_logg = time_start[:19]
logon_start = datetime.strptime(time_logg, '%Y-%m-%dT%H:%M:%S')
time_logon = logon_start.strftime('%H:%M')
channel = self.bot.get_channel(781834455765483562)
embed = Embed(title="ATC Annoucement", description="`New controller just logged in!`", color=0x00ff4c)
embed.add_field(name='Position', value=f"`{item['callsign']}`", inline=False)
embed.add_field(name='Frequency', value=f"`{item['frequency']}`", inline=False)
embed.add_field(name='Controller', value=f"`{item['realname']}`", inline=False)
embed.set_footer(text=f"Logged on at: {time_logon}")
await channel.send(embed=embed)
if item['callsign'] in data['clients'] and atc and not data1['clients']:
channel = self.bot.get_channel(781834455765483562)
embed2 = Embed(title="ATC Logged off!", description=f"{item['callsign']} just logged off!", color=0xff8800)
await channel.send(embed=embed2)
file = open("VATSIM_Data_old.json", "w", encoding="utf-8")
file.write(requests.get('https://data.vatsim.net/vatsim-data.json').text)
file.close()
print("Saved new VATSIM Data!")
EDIT:
In my first attempt to just get 1 message, I did something like the code below and it worked. But as soon as I changed to the code above it simply sends the same message all over again
json1 = open("VATSIM_Data_old.json", "r", encoding="utf-8").read()
xy = json.dumps(json1)
s1 = json.loads(xy)
t = requests.get('https://data.vatsim.net/vatsim-data.json').json()
xx = json.dumps(t)
s = json.loads(xx)
for item in s['clients']:
if item['callsign'] in atc:
if item['callsign'] in s1:
pass
else:
if item['callsign'] in atc:
time_start = item['time_logon']
time_logg = time_start[:19]
logon_start = datetime.strptime(time_logg, '%Y-%m-%dT%H:%M:%S')
time_logon = logon_start.strftime('%H:%M')
channel = self.bot.get_channel(781834455765483562)
embed = Embed(title="ATC Annoucement", description="`New controller just logged in!`", color=0x00ff4c)
embed.add_field(name='Position', value=f"`{item['callsign']}`", inline=False)
embed.add_field(name='Frequency', value=f"`{item['frequency']}`", inline=False)
embed.add_field(name='Controller', value=f"`{item['realname']}`", inline=False)
embed.set_footer(text=f"Logged on at: {time_logon}")
await channel.send(embed=embed)```

Related

Why is it bringing the Userid back and not the text_max?

Hi so im having a problem where when I try to use this command its bringing the USERID instead of what im trying to get heres the error
File "goonie.py", line 421, in spam
spammers = users[str(ctx.author.id)]['text_max']
KeyError: '932428536176848938'
The numbers are the "ctx.author.id" and im trying to get the "text_max"
async def spam(ctx, message,amount):
users = await get_paid_data()
spammers = users[str(ctx.author.id)]['text_max']
if amount > spammers:
if str(ctx.author.id) in users:
if users[str(ctx.author.id)]['vip'] == 'Y':
timeleft = datetime.strptime(users[str(ctx.author.id)]['expDate'],'%Y-%m-%d') - datetime.now()
if timeleft.days <= 0:
em = discord.Embed(color=0x000000, description="Sorry your plan on this tool has expired")
await ctx.send(embed=em) ```
users is a dictionary.
You first access the str(ctx.author.id) key in the dictionary.
The error is telling you the dictionary does not contain a key corresponding to the value of string value of ctx.author.id. Therefore, you cannot access text_max because the first part of your dict navigation is incorrect.
You should print users and see what it contains:
print(users)
My initial guess is that you're converting the id to a string when it in-fact doesn't need to be converted, but it's impossible to say without seeing the contents of the dictionary.

Issue with if and else statement in a loop

Hi I'm having an issue checking through a list codes that are stored in a temp dictionary and matching ones that exist and if one doesn't exist then print/await a different result. I'm doing this by comparing the uses for each invite before the user joined and after they joined.
It seems as if the if statement if invite.uses < find_invite_by_code(invites_after_join, invite.code).uses: is being ignored and the else statement is being printed/awaited instead for both expired and none expired code. I'm not too sure why this is happening. Help would be appreciated.
I have commented parts of the code for a better understanding.
Here is my code:
invites = {} #temporary dict to store the codes
# A function to find an invite in a guilds.invites() list
def find_invite_by_code(invite_list, code):
# Simply looping through each invite in an
# invite list which we will get using guild.invites()
for inv in invite_list:
# Check if the invite code in this element
# of the list is the one we're looking for
if inv.code == code:
# If it is, we return it.
return inv
#commands.Cog.listener()
async def on_member_join(self, member):
time = datetime.utcnow()
users = len([e.name for e in member.guild.members])
name = member
name = " ~ ".join((name.name, name.nick)) if name.nick else name.name
# Getting the invites before the user joining
# from our cache for this specific guild
invites_before_join = invites[member.guild.id]
# Getting the invites after the user joining
# so we can compare it with the first one, and
# see which invite uses number increased
invites_after_join = await member.guild.invites()
# Loops for each invite we have for the guild
# the user joined.
for invite in invites_before_join:
# Now, we're using the function we created just
# before to check which invite count is bigger
# than it was before the user joined.
if invite.uses < find_invite_by_code(invites_after_join, invite.code).uses:
# Now that we found which link was used we'll start the embed
joinmsg = discord.Embed(title=f'{member.name}#{member.discriminator} has joined', timestamp=time, colour=discord.Color(random.randint(0x000000, 0xFFFFFF)))
joinmsg.add_field(name="**Account Created:**", value=member.created_at.strftime("%#d %b %Y, at %I:%M%p"), inline=True)
joinmsg.add_field(name="**Invite:**", value=f"||{invite.code}||", inline=True)
joinmsg.add_field(name="**Inviter:**", value=f"{invite.inviter}", inline=False)
joinmsg.set_footer(text=f"Member {users} • User ID: {member.id}")
joinmsg.set_thumbnail(url=member.avatar_url)
await self.bot.send_log_message(member.guild, embed=joinmsg)
#We use an else statement if a NoneType is returned this would happen if the invite link has expired.
else:
# Create the embed to be sent
joinmsg = discord.Embed(title=f'{member.name}#{member.discriminator} has joined', timestamp=time, colour=discord.Color(random.randint(0x000000, 0xFFFFFF)))
joinmsg.add_field(name="**Account Created:**", value=member.created_at.strftime("%#d %b %Y, at %I:%M%p"), inline=True)
joinmsg.add_field(name="**Invite:**", value=f"Expired", inline=True)
joinmsg.set_footer(text=f"Member {users} • User ID: {member.id}")
joinmsg.set_thumbnail(url=member.avatar_url)
await self.bot.send_log_message(member.guild, embed=joinmsg)
# We will now update our cache so it's ready
# for the next user that joins the guild
invites[member.guild.id] = invites_after_join
# return to prevent looping when we already got what we want
return
It looks like something isn't happening quite the way you want it to, I would add a print function before it and print the values of invite.uses and find_invite_by_code(invites_after_join, invite.code).uses and then print the type of them too, if you still can't spot the error, then we'll have a problem
print(invite.uses, find_invite_by_code(invites_after_join, invite.code).uses)
print(type(invite.uses), type(find_invite_by_code(invites_after_join, invite.code).uses))

Downloading emails with UTF-8 B encoded header

I have a problem with a code which is supposed to download your emails in eml files.
Its supposed to go through the INBOX email listing, retrieve the email content and attachments(if any) and create an .eml file which contains all that.
What it does is that it works with content type of text and the majority multiparts. If an email in the listing contains utf-8B in its header, it simply acts like its the end of the email listing, without displaying any error.
The code in question is:
result, data = p.uid('search',None, search_criteria) # search_criteria is defined earlier in code
if result == 'OK':
data = get_newer_emails_first(data) # get_newer_emails_first() is a function defined to return the list of UIDs in reverse order (newer first)
context['emailsum'] = len(data) # total amount of emails based on the search_criteria parameter.
for num in data:
mymail2 = {}
result,data1 = p.iud('fetch', num, '(RFC822)')
email_message = email.message_from_bytes(data[0][1])
fullemail = email_message.as_bytes()
default_charset = 'ASCII'
if email_message.is_multipart():
m_subject = make_header(decode_header(email_message['Subject']))
else:
m_subject = r''.join([ six.text_type(t[0], t[1] or default_charset) for t in email.header.decode_header(email_message['Subject']) ])
m_from = string(make_header(decode_header(email_message['From'])))
m_date = email_message['Date']
I have done my tests and discovered that while the fullemail variable contains the email properly (thus it reads the data from the actual email successfully), the problem should be in the if else immediately after, but I cannot find what the problem is exactly.
Any ideas?
PS: I accidentally posted this question as a guest, but I opted to delete it and repost it from my account.
Apparently the error lay in my code in the silliest of ways.
Instead of:
m_from = string(make_header(decode_header(email_message['From'])))
m_date = email_message['Date']
It should be:
m_from = str(make_header(decode_header(email_message['From'])))
m_date = str(make_header(decode_header(email_message['Date'])))

Pulling the 'ExternalImageId' data when running search_faces_by_image

I'm fairly new to AWS and for the past week, been following all the helpful documentation on the site.
I am currently stuck on bring unable to pull the External Image Id data from a Reko collection after a 'search face by image', I just need to be able to put that data into a variable or to print it, does anybody know how I could do that?
Basically, this is my code:
import boto3
if name == "main":
bucket = 'bucketname'
collectionId = 'collectionname'
fileName = 'test.jpg'
threshold = 90
maxFaces = 2
admin = 'test'
targetFile = "%sTarget.jpg" % admin
imageTarget = open(targetFile, 'rb')
client = boto3.client('rekognition')
response = client.search_faces_by_image(CollectionId=collectionId,
Image={'Bytes': imageTarget.read()},
FaceMatchThreshold=threshold,
MaxFaces=maxFaces)
faceMatches = response['FaceMatches']
print ('Matching faces')
for match in faceMatches:
print ('FaceId:' + match['Face']['FaceId'])
print ('Similarity: ' + "{:.2f}".format(match['Similarity']) + "%")
at the end of it, I receive:
Matching faces
FaceId:8081ad90-b3bf-47e0-9745-dfb5a530a1a7
Similarity: 96.12%
Process finished with exit code 0
What I need is the External Image Id instead of the FaceId.
Thanks!

Tweepy: get all friends of a sample of twitter accounts: how to handle protected users

I want to look up all the friends (meaning the twitter users one is following) of a sample of friends of one twitter account, to see what other friends they have in common. The problem is that I don't know how to handle protected accounts, and I keep running into this error:
tweepy.error.TweepError: Not authorized.
This is the code I have:
...
screen_name = ----
file_name = "followers_data/follower_ids-" + screen_name + ".txt"
with open(file_name) as file:
ids = file.readlines()
num_samples = 30
ids = [x.strip() for x in ids]
friends = [[] for i in range(num_samples)]
for i in range(0, num_samples):
id = random.choice(ids)
for friend in tweepy.Cursor(api.friends_ids, id).items():
print(friend)
friends[i].append(friend)
I have a list of all friends from one account screen_name, from which I load the friend ids. I then want to sample a few of those and look up their friends.
I have also tried something like this:
def limit_handled(cursor, name):
try:
yield cursor.next()
except tweepy.TweepError:
print("Something went wrong... ", name)
pass
for i in range(0, num_samples):
id = random.choice(ids)
items = tweepy.Cursor(api.friends_ids, id).items()
for friend in limit_handled(items, id):
print(friend)
friends[i].append(friend)
But then it seems like only one friend per sample friend is stored before moving on to the next sample. I'm pretty new to Python and Tweepy so if anything looks weird, please let me know.
First of all, a couple of comments on naming. The names file and id are protected, so you should avoid using them to name variables - I have changes these.
Secondly, when you initialise your tweepy API, it's clever enough to deal with rate limits if you use wait_on_rate_limit=True and will inform you when it's delayed due to rate limits if you use wait_on_rate_limit_notify=True.
You also lose some information when you set friends = [[] for i in range(num_samples)], as you then won't be able to associate the friends you find with the account they relate to. You can instead use a dictionary, which will associate each ID used with the friends found, allowing for better processing.
My corrected code is as follows:
import tweepy
import random
consumer_key = '...'
consumer_secret = '...'
access_token = '...'
access_token_secret = '...'
# OAuth process, using the keys and tokens
auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_token_secret)
# Creation of the actual interface, using authentication. Use rate limits.
api = tweepy.API(auth, wait_on_rate_limit=True, wait_on_rate_limit_notify=True)
screen_name = '----'
file_name = "followers_data/follower_ids-" + screen_name + ".txt"
with open(file_name) as f:
ids = [x.strip() for x in f.readlines()]
num_samples = 30
friends = dict()
# Initialise i
i = 0
# We want to check that i is less than our number of samples, but we also need to make
# sure there are IDs left to choose from.
while i <= num_samples and ids:
current_id = random.choice(ids)
# remove the ID we're testing from the list, so we don't pick it again.
ids.remove(current_id)
try:
# try to get friends, and add them to our dictionary value if we can
# use .get() to cope with the first loop.
for page in tweepy.Cursor(api.friends_ids, current_id).pages():
friends[current_id] = friends.get(current_id, []) + page
i += 1
except tweepy.TweepError:
# we get a tweep error when we can't view a user - skip them and move onto the next.
# don't increment i as we want to replace this user with someone else.
print 'Could not view user {}, skipping...'.format(current_id)
The output is a dictionary, friends, with keys of user IDs and items of the friends for each user.

Categories