We Keep Coding

Bittorrent and sockets: how to handle multiple messages?

I'm writing a bittorrent client in python, and have been using a loop to continually read messages from the peer sockets using recv().
When I run my program I look in wireshark to see what bittorrent messages I'm getting. It's pretty easy to tell what kind of message you got from the first 5 bytes of the message, since the length and message ID are specified there.
I'm running into some problems when dealing with receiving data containing multiple messages.
I've tried tackling it by writing a method like this:
def handleMultiple(self, message, peer):
total_length = len(message)
parsed = 0
while parsed < total_length:
m_len, m_id = struct.unpack(">IB", message[parsed:parsed + 5])
m_total = m_len + 4
print(m_len, total_length, parsed, m_id, peer.made_handshake, peer.ip)
self.handleMessage(message[parsed:m_total + parsed], peer)
parsed += m_total
The function just breaks down the received bytes into its constituent messages and hands it off to the message handler that knows how to deal with individual messages.
The problem is that when I printed out the length prefix and message ID from a message I received using recv(), sometimes it looks like just garbage numbers.
This is really my first time experimenting with sockets, so I lack the intuition to know what I'm really getting when calling recv(). Should I just call receive on the first 5 bytes of data I get, then do some checking to make sure that the length and ID are valid, then call recv() on the rest of the message?
How should I go about handling multiple messages incoming at a time?
Edit:
I wanted to provide some images of the results I'm seeing to see if anyone can help identify the issue I'm having.
Here's a picture of the bittorrent messages I'm receiving:
Here's a corresponding logging output:
The columns are supposed to be message length + 4, total message length, message id, and the IP from the sender:
As I can see, the length prefix for the first messages, (the ones that are multiple messages sent to me at a time) are completely too large. The fifth message I got from 95.211.212.26 is a well formed bitfield message.
Another thing I noticed is that the supposed message ID from each of the multi-message messages is 255. Also given that the total length of a bitfield message for this given torrent is 126, the total lengths (303, 328, 325) are not inconceivable for messages of a bitfield followed by several have messages.

Alright so I've managed to figure out where I was going wrong. I was reading from the socket assuming that my message would be there in full. In reality, I was reading the initial snippet of the message, and at a later time I was reading the middle of the message. The 255 values I was seeing weren't message IDs but actually the middle of the peer's bitfield (0xff).
I changed my approach to store the read in bytes from the socket to the peer's message buffer. Once the message buffer was at least as long as the expected payload, I read the message and trimmed the buffer to exclude what I just read. Now all of my messages' IDs are looking as I expect.

forward message to super group with telethon

Recently I wrote code that should forward every message from a certain user to all groups that I joined but it doesn't.
Here my code:
for message in client.iter_messages('aliakhtari78'):
try:
dialogs = client.get_dialogs()
for dialog in dialogs:
id_chat = dialog.message.to_id.channel_id
entity = client.get_entity(id_chat)
client.forward_messages(
entity, # to which entity you are forwarding the messages
message.id, # the IDs of the messages (or message) to forward
'somebody' # who sent the messages?
)
except:
pass
in this code first I take every message which send to me by 'aliakhtari78' and then get entity of the groups that I joined to and in the end it should forward the message to all groups but it doesn't, I check my code and replace the entity with a user entity and it worked and I know the problem is because of entity, but I cant find out what is my problem.
In addition, I'm sorry for writing mistakes in my question.

In order to send messages to any entities in Telegram, you need two pieces of information:
the constant unique ID of the entity (It's an integer. It's NOT username string)
the access_hash which is different for each user for each entity
You can only pass #username to client.get_entity, and Telethon automatically resolves the #username to an entity with id and access_hash. That's why it works when you change your code like that. However, in your code, you have passed channel_id (which is the constant unique ID of the entity) to client.get_entity, not username.
Note that client.get_dialogs returns entities along with dialogs. You have just ignored the entities! This is how you can get an array of all entities:
dialogs, entities = client.get_dialogs()
Then simply pass the corresponding entity from the entities array to client.forward_messages.

Maildir - open latest email and reply to sender

I've configured postfix on the email server with .forward file which saves a copy of email and invokes a python script. These emails are stored in Maildir format.
I want to use this Python script to send a reply to the sender acknowledging that the email has been received. I was wondering if there is any way I can open/access that e-mail, get the header info and sender address and send email back.
I looked at several examples of Maildir functions of Python, but they mostly add/delete e-mails. How can I open the latest e-mail received in Maildir/new and get the required information?
The program I have so far:
md = mailbox.Maildir('/home/abcd/Maildir')
message = md.iterkeys().next()
#print message
#for msg in md:
# subject = msg.get('Subject',"")
# print subject
print message
sender = message.get('From',"")
print sender
When I execute this, I do get the sender name. But It is rather the oldest email arrived in Maildir/new folder not the latest one.
Also, if I use get_date function, what if two (or more) e-mails arrive on the same day?

The MaildirMessage's method .get_date() gets you the timestamp of the
message file on disc. Depending on your filesystem, this may have anywhere between two second and nanosecond accuracy. The changes of two messages giving the same value with .get_date() are vastly smaller than when this actually returned a date only.
However if the message files were touched for some reason the return from .get_date() would not be relevant at all. Dovecot e.g. explicitly states that a files mtime should not be changed.
There are several dates associated with a MaildirMessage:
The arrival time timestamp, as encoded in the name of message (the part before the first dot, these are "whole" seconds). If the part
between the first and second dot has a segment of the form Mn than n is the microsecond arrival time, and be used to improve the resolution of the timestamp.
The timestamp of the file on disc
The 'Date:' header field as set by the sending program (or added by some
MTA)
The dates added by intermediate MTA in the 'Received:' header field
The last of these might not be available e.g. if you and the sender are on the same mail server. The third can be easily faked/incorrect (ever got spam in your inbox dated many years ago?). And the second is incorrect if the file ever got touched.
That leaves selecting on the first option:
d = {}
for name in md.keys():
d.setdefault(int(name.split('.', 1)[0]), []).append(name)
result = sorted(d.items())[-1][1]
assert len(result) == 1 # might fail
msg = md.get_message(result[0])
If you are lucky result is a list with a single item. But this value has only second resolution, so you might have multiple emails and then you have to decide on how to decide which message to select based on one of the other values (e.g. by sorting using the files timestamp .get_date()) or just select the first, randomly select one. (If you have the log file, you can search for the result messages' keys in there to determine which one arrived latest).
If you wouldn't convert to int, and have old emails (i.e. before 2001-09-09 03:46:40) a string comparison would probably not give you the message with the latest arrival time.

Some hints for this:
You can open a Maildir with the mailbox.Maildir class (see the Documentation for mailbox)
You can iterate over all the mails in a Maildir via the method itervalues
Now you get all the mails in the Maildir. One of them is the most recent one.
The mails are objects of the class MaildirMessage, which is a subclass of Message. For these classes, also a documentation exists (on the same page as mailbox, currently)
With the method "get_date" on those objects, you can find out, which one is the most recent one. You still have to select it yourself.
So much as beginners help: A little bit you should also do by yourself.
You should make yourself familiar with the Python documentation - I agree, that it is not easy to find the right packages and how to use them, but you can try them directly in the Python shell.
Ok, here another code snippet:
newest = None
for message in md.itervalues():
if newest == None or message.get_date() > newest.get_date():
newest = message
# now newest should contain the newest message
Did not see your last question: get_date does not only contain the date, but also the time, because it gives the number of seconds since (normally) 1970.

Receiving "role" and/or "affiliation" with message in SleekXMPP

Apologies for the amateur question. I'm just learning Python and I'm fumbling around this XMPP bot script using XMPP.
I have a bot built using the MUC bot example from SleekXMPP: http://sleekxmpp.com/getting_started/muc.html
Where my bot differs from the example is my script creates a SQLite database and on each group_message event, parses the XML to retrieve the nick and message body text and write it to the database with a timestamp.
Here is the part of my bot that's recording the msg output from the XMPP channel:
def groupchat_message(self, msg):
if msg['type'] in ('groupchat'):
raw = str(msg) # Save raw XML as a string in the database for debugging purposes
timestamp = datetime.utcnow().strftime('%Y-%m-%d %H:%M:%S')
fromuser = str(msg['from']) # Convert "from" attribute to string so it can be split
author = fromuser.split('/')[1] # Split "from" attribute to remove channel address leaving only nick behind
body = msg['body']
msginsert = [timestamp, author, body, raw] # Database input list to be handed to placeholders.
db.execute("INSERT INTO messages VALUES (?,?,?,?)", msginsert) # "?" placeholder is REQUIRED for automatic database sanitization.
dbconn.commit()
print("[",timestamp,"]",author,": ",body, sep='')
else:
print(msg)
The print statements are just for debugging purposes so I can watch the messages tick by in the terminal so I know the script is running.
In the recorded information I would like to also include the user's role or affiliation in the XMPP channel so admins and moderators can be singled out. This is what I'm having trouble wrapping my head around. It seems like I should be able to do it based on the SleekXMPP stanza docs but I'm having trouble figuring out how I get from the message XML to the role/affiliation information.
How can I take the information provided in the message XML and get back the role and/or affiliation of the user who posted that message?
For reference, this is the raw XML output from the XMPP channel for messages:
<message to="username#example.com" from="channelname#channels.example.com/User Nick" id="1453" type="groupchat">
<body>This is the message body text</body>
<x xmlns="jabber:x:event">
<composing />
</x>
</message>

The role/affiliation isn't included in the message, it is part of the state of the chat.
As a reminder, XMPP has three different stanzas that get send: <message>, which you had as an example, <iq>, which are used to retrieve or set things and <presence>, which indicates the presence of things. The role/affiliation information is included in a presence stanza. For example in the first presence stanzas your client receives to inform it of who is present in the room already, see Example 21 from XEP-0045 (Multi-User Chat). Your client also receives new presence stanzas whenever someone's nickname, role or affiliation changes or when they leave the room.
You should make sure you store this information yourself, because SleekXMPP doesn't do it for you. This can be done by creating a dictionary that stores, for every nickname, their role and one for their affiliation. On presence changes you should make sure to update this information. Then you can use those dictionaries in your message handler to log their role/affiliation.
So something like:
def __init__(...):
self.roles = dict()
self.affiliations = dict()
self.add_event_handler(""groupchat_presence"", self.muc_presence)
...
def muc_presence(self, presence):
nick = presence['muc']['nick']
self.roles[nick] = presence['muc']['role']
self.affiliations[nick] = presence['muc']['affiliation']
This is the general idea, you'll need to do some more work to make it handle nickname changes and people leaving the room properly.

get email unread content, without affecting unread state [duplicate]

This question already has answers here:
Fetch an email with imaplib but do not mark it as SEEN
(4 answers)
Closed 7 years ago.
Right now its a gmail box but sooner or later I want it to scale.
I want to sync a copy of a live personal mailbox (inbox and outbox) somewhere else, but I don't want to affect the unread state of any unread messages.
what type of access will make this easiest? I can't find any information if IMAP will affect the read state, but it appears I can manually reset a message to unread. Pop by definition doesn't affect unread state but nobody seems to use pop to access their gmail, why?

In the IMAP world, each message has flags. You can set the individual flags on each message. When you Fetch a message, it's actually possible to read the message, without applying the \Seen flag.
Most mail clients will apply the \Seen flag when the message is read. So, if the message has already been read, outside of your app, then you will need to remove the \Seen flag.
Just as fyi...here is the relevant part about flags from the RFCs:
A system flag is a flag name that is pre-defined in this
specification. All system flags begin with "\". Certain system
flags (\Deleted and \Seen) have special semantics described
elsewhere. The currently-defined system flags are:
\Seen
Message has been read
\Answered
Message has been answered
\Flagged
Message is "flagged" for urgent/special attention
\Deleted
Message is "deleted" for removal by later EXPUNGE
\Draft
Message has not completed composition (marked as a draft).
\Recent
Message is "recently" arrived in this mailbox. This session
is the first session to have been notified about this
message; if the session is read-write, subsequent sessions
will not see \Recent set for this message. This flag can not
be altered by the client.
If it is not possible to determine whether or not this
session is the first session to be notified about a message,
then that message SHOULD be considered recent.
If multiple connections have the same mailbox selected
simultaneously, it is undefined which of these connections
will see newly-arrived messages with \Recent set and which
will see it without \Recent set.

There is a .PEEK option on the FETCH command in IMAP that will explicitly not set the /Seen flag.
Look at the FETCH command in RFC 3501 and scroll down a bit to page 57 or search for "BODY.PEEK".

You need to specify section when you use BODY.PEEK. Sections are explained in IMAP Fetch Command documentations under BODY[<section>]<<partial>>
import getpass, imaplib
M = imaplib.IMAP4()
M.login(getpass.getuser(), getpass.getpass())
M.select()
typ, data = M.search(None, 'ALL')
for num in data[0].split():
typ, data = M.fetch(num, '(BODY.PEEK[])')
print 'Message %s\n%s\n' % (num, data[0][5])
M.close()
M.logout()
PS: I wanted to fix answer given Gene Wood but was not allowed because edit was smaller than 6 characters (BODY.PEEK -> BODY.PEEK[])

Nobody uses POP because typically they want the extra functionality of IMAP, such as tracking message state. When that functionality is only getting in your way and needs workarounds, I think using POP's your best bet!-)

if it helps anyone, GAE allows you to receive email as an HTTP request, so for now i'm just forwarding emails there.

To follow up on Dan Goldstein's answer above, in python the syntax to use the ".PEEK" option would be to call IMAP4.fetch and pass it "BODY.PEEK"
To apply this to the example in the python docs :
import getpass, imaplib
M = imaplib.IMAP4()
M.login(getpass.getuser(), getpass.getpass())
M.select()
typ, data = M.search(None, 'ALL')
for num in data[0].split():
typ, data = M.fetch(num, '(BODY.PEEK)')
print 'Message %s\n%s\n' % (num, data[0][5])
M.close()
M.logout()