We Keep Coding

How to send Properties details in EventHub message using python?

I am referring to this article "https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-python-get-started-send" related to sending messages to EventHub using Python.
A Message has the following components: offset, body, systemProperties, properties.
offset is auto-generated but we can provide the other.
For my project - apart from message body I also need to send "Properties" which is not part of Body, How to do that? I checked the class EventData(object): and looks like application_properties can be used to do this but i'm not sure how to implement this.
Is application_properties right analogy to obtain "Properties" in message?
Could you please add more details to the article with example displaying how to use the python EventData class for sending detailed information apart from message body like Properties and SystemProperties. As of now Properties is being sent as empty list.
Sample message format that i need to send to EventHub using Python:
Sender.send(EventData('{"**offset**":"2415248","**body**":"TESTone:100,
Temperature:553.0","**systemProperties**":[{"key":{"string":"x-opt-
sequence-number"},"value":{"string":"23512"}},{"key":{"string":"x-opt-
offset"},"value":{"string":"2415248"}},{"key":{"string":"x-opt-
enqueued-time"},"value":{"string":"Fri Feb 22 02:14:23 UTC
2019"}}],"**properties**":[]}'))
And I want to send this dict values in properties: {"key":{"string":"Type"},"value":{"string":"iPhone"}}

I asked this question elsewhere as the solution provided by PraveenS does not actually solve the problem. The correct answer which I received is below, sharing to help others:
The solution is to assign the properties you want to be sent with the message to EventData.properties, applying that to the above code would look like this:
from azure.eventhub import EventHubClient, Receiver, Offset, Sender, EventData
from uamqp import Message
ADDRESS = "amqps://<>.windows.net/<>"
USER = "RootManageSharedAccessKey"
KEY = "<>"
client = EventHubClient(ADDRESS, debug=True, username=USER, password=KEY)
Sender = client.add_sender(partition="0")
client.run()
event = EventData(body="TESTTWO:100, Temperature:-127.0")
### THIS IS WHERE THE CHANGE IS ###
event.properties = {"Type": "iPhone"}
Sender.send(event)

Found solution to this, yes we can use "application_properties" to send "properties" of message.
This is my sample code that worked:
from azure.eventhub import EventHubClient, Receiver, Offset, Sender, EventData
from uamqp import Message
ADDRESS = "amqps://<>.windows.net/<>"
USER = "RootManageSharedAccessKey"
KEY = "<>"
client = EventHubClient(ADDRESS, debug=True, username=USER, password=KEY)
Sender = client.add_sender(partition="0")
client.run()
event = EventData(body="TESTTWO:100, Temperature:-127.0")
event.application_properties = {"Type": "iPhone"}
Sender.send(event)

You may also find useful the following way, which basically do the same:
props = {"Type": "iPhone"} # properties you want to send
columns = ['body', 'properties']
values = [(write_binneddata, props)] # You can also send multiple messages adding tuples to the list
df = spark.createDataFrame(values, columns)
conn_string = f"Endpoint=sb://{event_hub_namespace}.servicebus.windows.net/;SharedAccessKeyName={shared_acc_keyname};SharedAccessKey={shared_acc_key};EntityPath={event_hub_name}"
conf = { 'eventhubs.connectionString' : conn_string }
ds = (
df
.write
.format("eventhubs")
.options(**conf)
.option("checkpointLocation", f"{checkpoint_path}")
.save()
)
PS: it needs this library installed.

schedule email with sendgrid python

I am using python module of sendgrid (https://github.com/sendgrid/sendgrid-python) to send transactional emails. I need help with the syntax to send the email at a scheduled time.
The general sendgrid documentation asks to modify json as "{ “send_at”: 1409348513 }". I believe this json is not directly accessible in sendgrid-python. I need syntax for doing equivalent thing with the python library.
My current code is equivalent to the one copied below. It would be great if some one can suggest how to modify this code to schedule it at a particular time e.g. datetime.dateime.now() + datetime.timedelta(days=1)
import sendgrid
from sendgrid.helpers.mail import Email, Content, Substitution, Mail
import urllib2 as urllib
def send_email_custom():
sg = sendgrid.SendGridAPIClient(apikey=myApiKey)
from_email = Email(sendEmail)
to_email = Email(custEmail)
reply_to_email = Email(receiveEmail)
content = Content("text/html", "Introduction")
mail = Mail(from_email, subject="Hi!", to_email=to_email, content=content)
mail.personalizations[0].add_substitution(Substitution("_firstName_", firstName))
mail.set_template_id(templateId)
try:
response = sg.client.mail.send.post(request_body=mail.get())
except urllib.HTTPError as e:
print(e.read())
return False
if response.status_code >=200 and response.status_code < 300:
return True
else:
return False

send_at is a component of the personalizations Object, so you can define it at that level, allowing you to set distinct sending times for each recipient/personalization set.
If you don't need that, you can also set it at the top mail level.

Looks like you should be able to do that by following this example. You can specify all of the raw fields you need.

Exact code solution:
mail = Mail(from_email, subject="Hi!", to_email=to_email, content=content)
mail.send_at = SendAt(1461775053) # Time is specified in UNIX form.
# This takes advantage of Sendgrid's inbuilt email scheduler
# but you can't schedule more than 72 hours in advance.
This removes the need to deal with personalizations for send time.
Source: line 308 of https://github.com/sendgrid/sendgrid-python/blob/main/examples/helpers/mail_example.py
Posting this for those who Googled their way here in 2022 or later.

Adding Content-Disposition header in Python - email isn't sent

Following the directions in Python's email examples and in several Stack Overflow questions, I wrote the following function (sending through the Gmail SMTP server):
def send_email(emailaddr, message, attachmentfile = None, subject = None):
try:
smtpconn = smtplib.SMTP(mainconf["SMTPHOST"], mainconf["SMTPPORT"])
smtpconn.set_debuglevel(1)
smtpconn.ehlo()
smtpconn.starttls()
smtpconn.login(mainconf["SMTPUSER"], mainconf["SMTPPASS"])
if not attachmentfile:
message = '\n' + message
smtpconn.sendmail(mainconf["EMAILFROM"], emailaddr, message)
else:
multipart = MIMEMultipart()
multipart['Subject'] = subject if subject else "Attachment"
multipart['From'] = mainconf["EMAILFROM"]
multipart['To'] = emailaddr
with open(attachmentfile, 'rb') as fp:
filepart = MIMEApplication(fp.read())
multipart.attach(filepart)
multipart.attach(message)
smtpconn.sendmail(mainconf["EMAILFROM"], emailaddr, multipart.as_string())
generallog.info("Sent an email to {0}".format(emailaddr))
except:
generallog.warn("Email sending to {0} failed with error message {1}".format(emailaddr, traceback.format_exception_only(sys.exc_info()[0], sys.exc_info()[1])))
This works fine for sending an email with an attachment, but it results in the famous noname problem (that's just one of several SO questions on the subject). So I add in this:
filepart.add_header('Content-Disposition','attachment',filename=os.basename(attachmentfile))
Then Gmail simply refuses to send the email. When I run this code in the Python shell, I get the standard message accepted message, but the message is never delivered.
Why might this be happening?

Writing code using graph APIs

I am extremely new to python , scripting and APIs, well I am just learning. I came across a very cool code which uses facebook api to reply for birthday wishes.
I will add my questions, I will number it so that it will be easier for someone else later too. I hope this question will clear lots of newbies doubts.
1) Talking about APIs, in what format are the usually in? is it a library file which we need to dowload and later import? for instance, twitter API, we need to import twitter ?
Here is the code :
import requests
import json
AFTER = 1353233754
TOKEN = ' <insert token here> '
def get_posts():
"""Returns dictionary of id, first names of people who posted on my wall
between start and end time"""
query = ("SELECT post_id, actor_id, message FROM stream WHERE "
"filter_key = 'others' AND source_id = me() AND "
"created_time > 1353233754 LIMIT 200")
payload = {'q': query, 'access_token': TOKEN}
r = requests.get('https://graph.facebook.com/fql', params=payload)
result = json.loads(r.text)
return result['data']
def commentall(wallposts):
"""Comments thank you on all posts"""
#TODO convert to batch request later
for wallpost in wallposts:
r = requests.get('https://graph.facebook.com/%s' %
wallpost['actor_id'])
url = 'https://graph.facebook.com/%s/comments' % wallpost['post_id']
user = json.loads(r.text)
message = 'Thanks %s :)' % user['first_name']
payload = {'access_token': TOKEN, 'message': message}
s = requests.post(url, data=payload)
print "Wall post %s done" % wallpost['post_id']
if __name__ == '__main__':
commentall(get_posts())`
Questions:
importing json--> why is json imported here? to give a structured reply?
What is the 'AFTER' and the empty variable 'TOKEN' here?
what is the variable 'query' and 'payload' inside get_post() function?
Precisely explain almost what each methods and functions do.
I know I am extremely naive, but this could be a good start. A little hint, I can carry on.
If not going to explain the code, which is pretty boring, I understand, please tell me how to link to APIs after a code is written, meaning how does a script written communicate with the desired API.
This is not my code, I copied it from a source.

json is needed to access the web service and interpret the data that is sent via HTTP.
The 'AFTER' variable is supposed to get used to assume all posts after this certain timestamp are birthday wishes.
To make the program work, you need a token which you can obtain from Graph API Explorer with the appropriate permissions.

imaplib.error: command FETCH illegal in state AUTH

I'm trying to download attachments from Gmail, using a combination of pieces of code I found online, and some editing from myself. However, the following code:
import email, getpass, imaplib, os, random, time
import oauth2 as oauth
import oauth2.clients.imap as imaplib
MY_EMAIL = 'example#gmail.com'
MY_TOKEN = "token"
MY_SECRET = "secret"
consumer = oauth.Consumer('anonymous', 'anonymous')
token = oauth.Token(MY_TOKEN, MY_SECRET)
url = "https://mail.google.com/mail/b/"+MY_EMAIL+"/imap/"
m = imaplib.IMAP4_SSL('imap.gmail.com')
m.authenticate(url, consumer, token)
m.select('INBOX')
items = m.select("UNSEEN");
items = items[0].split()
for emailid in items:
data = m.fetch(emailid, "(RFC822)")
returns this error:
imaplib.error: command FETCH illegal
in state AUTH
Why would Fetch be illegal while I'm authorized?

You're lacking error checking on your calls to select. Typically, this is how I'll structure the first parts of a connection to a mailbox:
# self.conn is an instance of IMAP4 connected to my server.
status, msgs = self.conn.select('INBOX')
if status != 'OK':
return # could be break, or continue, depending on surrounding code.
msgs = int(msgs[0])
Essentially, the trouble you're encountering is that you've selected a mailbox that doesn't exist, your status message is probably not "OK" as it should be, and the value you're iterating over isn't valid. Remember, select expects a mailbox name. It does not search based on a flag (which may be what you're attempting with "UNSEEN"). When you select a non-existent mail box you actually get this as a response:
('NO', ['The requested item could not be found.'])
In which case, for email id in items is not operating properly. Not what you're after in any way, unfortunately. What you'd get on a valid mailbox would be like this:
('OK', ['337'])
Hope that helps.
To address the question in comments, if you want to actually retrieve the unseen messages in the mailbox you'd use this:
status, msgs = self.conn.select('INBOX')
# returns ('OK', ['336'])
status, ids = self.conn.search(None, 'UNSEEN')
# returns ('OK', ['324 325 326 336'])
if status == 'OK':
ids = map(int, ids[0].split())
The response is going to be similar to the response from select, but instead of a single integer for the number of messages you'll get a list of ids.

Why would Fetch be illegal while I'm authorized?
IMAP has a state concept.
You can only fetch messages if you have selected a folder.
Here is a nice picture which shows this:
Image source: http://www.tcpipguide.com/free/t_IMAP4GeneralOperationClientServerCommunicationandS-2.htm