We Keep Coding

How to replace/overwrite default header of EmailMultiAlternatives

Environment: Ubuntu 18.10, Python 2.7.15, Django 1.11.16
I'm trying to send an email containing an inline image. I have the following code:
msg = EmailMultiAlternatives(some_subject, some_body, 'from#some-domain.com', ['to#some#domain'])
img_data = open('path/to/image.png', 'rb').read()
img = MIMEImage(img_data)
msg.attach(img)
msg.send()
(I've only included the code that I think is relevant but I can add more on demand.)
The above properly works and the image is properly displayed on most of the email clients (about 7 of them, both mobile, desktop or webmail ones) that I tested on, with two exceptions: Mozilla Thunderbird 60 and some macOS native email client.
On Thunderbird the image is not displayed inline but at the very end of the message. On the macOS client, the image is displayed inline but additionally it is also displayed at the very end of the message.
I composed and sent a test message from another email client, containing an inline image which was properly displayed on both Thunderbird and macOS. I compared the headers of this message with the headers of the message generated by my code.
I noticed that the faulty message has the 'Content-Type' set to 'multipart/mixed' while the properly displayed message had the same header set to 'multipart/related'.
I saved the faulty message in an eml file and manually changed the value of that header and then loaded the message in Thunderbird. The message was properly displayed and the image was in the right place.
If I could set that header to the proper value, the problem would be solved.
So, my question is: is there any possibility to tell EmailMultiAlternatives to set 'Content-Type' : 'multipart/related' instead of the default value of 'multipart/mixed'?
I tried to add the header like this but it is not working:
msg = EmailMultiAlternatives(some_subject, some_body, 'from#some-domain.com', ['to#some#domain'], headers={'Content-Type' : 'multipart/related'})
I got the following error ( I use Amazon SES):
400 Bad Request
<ErrorResponse xmlns="http://ses.amazonaws.com/doc/2010-12-01/">
<Error>
<Type>Sender</Type>
<Code>InvalidParameterValue</Code>
<Message>Duplicate header 'Content-Type'.</Message>
</Error>
<RequestId>xxxxxxxxxx</RequestId>
</ErrorResponse>
If I can't modify that header, do you suggest any alternatives?

If you look at the source code, you'll see that EmailMultiAlternatives is a subclass of EmailMessage, which itself has a class attribute:
mixed_subtype = 'mixed'
So if you create your own subclass to override this, you should get what you need:
class EmailMultiAlternativesRelated(EmailMultiAlternatives):
mixed_subtype = 'related'
That's it, now you just use this new class, and it will use "multipart/related".
(the _create_attachments() method passes this subtype to python's SafeMIMEMultipart which creates the actual headers for each attachment.)

How to obtain packet count of TCP packets?

I am retrieving flow statistics using a _flow_stats_reply_handler as demonstrated in the Ryu Traffic Monitor example.
I print using the following:
file.write("\n{},{},{},{},{},{},{},{},{}"
.format(ev.msg.datapath.id,
stat.match['in_port'], stat.match['eth_src'], stat.match['eth_dst'],
stat.instructions[0].actions[0].port,
stat.packet_count, stat.byte_count,
stat.duration_sec, stat.duration_nsec))
Note the stat.packet_count.
How could I change this to count TCP packets? I understand there is an ip_proto field and a tcp_flags field but I don't know how to code the match/count.
Edit:
I have further investigated this and added a flow match to my request flow stats function:
def _request_stats(self, datapath):
self.logger.debug('send stats request: %016x', datapath.id)
ofp = datapath.ofproto
parser = datapath.ofproto_parser
cookie = cookie_mask = 0
match = parser.OFPMatch(eth_type=0x0800)
req = parser.OFPFlowStatsRequest(datapath, 0, ofp.OFPTT_ALL, ofp.OFPP_ANY, ofp.OFPG_ANY,
cookie, cookie_mask, match)
datapath.send_msg(req)
This unfortunately still doesn't work, any ideas as to why not would be greatly appreciated.

You should add more data to your match, like ip_proto in order to match with tcp, as you may know, IP protocol number of TCP is 6, for more information about IP Protocol numbers check Wikipedia.
Please use the code below, You don't need to settcp_flags in this case.
match = parser.OFPMatch(
eth_type=0x0800,
ip_proto=6,
)

Ntrip client not receiving RTCM corrections

I'm searching for help understanding how to develope a complete and functional NTRIP Client in order to receive RTCM corrections.
I'm using Python 3.4, for now on Windows 7. Searching the net, I found some sample code and I used it to write a basic client. The problem is... it doesn't work.
I have access to a rtk correction service. The service is active and functioning.
This is a snippet of my code.
dummyNMEA = "$GPGGA,143741.356,7839.493,S,07627.626,W,0,00,,,M,,M,,*45"
username = my_username #username for RTCM correction service
password = my_password #password for RTCM correction service
port = 2101 #port for the service
'''Generate an encoding of the username:password for the service.
The string must be first encoded in ascii to be correctly parsed by the
base64.b64encode function.'''
pwd = base64.b64encode("{}:{}".format(username, password).encode('ascii'))
#The following decoding is necessary in order to remove the b' character that
#the ascii encoding add. Othrewise said character will be sent to the net and misinterpreted.
pwd = pwd.decode('ascii')
print("Header sending... \n")
header =\
"GET /mountpoint HTTP/1.1\r\n" +\
"Host my_host\r\n" +\
"Ntrip-Version: Ntrip/1.0\r\n" +\
"User-Agent: ntrip.py/0.1\r\n" +\
"Accept: */*" +\
"Connection: close\r\n" +\
"Authorization: Basic {}\r\n\r\n".format(pwd)
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((my_host,int(port)))
s.send(header.encode('ascii'))
print("Waiting answer...\n")
data = s.recv(2048).decode('ascii')
print(data)
s.send(dummyNMEA.encode('ascii'))
data = s.recv(2048).decode('ascii')
print(data)
s.close()
Now, the code is partially funcitoning. The request goes to the rtk server and I am correctly authenticated. I receive the correct answer as from ntrip protocol:
ICY 200 OK
Server: "Server of the mountpoint"
Date: "The date"
After this, I have to send a NMEA GGA sentence, in order to start receiving the RTCM corrections. I created various dummy NMEA sentences with a generator and tested sending them. I send the sentence and.... nothing happens. I receive no answer from the server.
Somebody has some idea? Perhaps I do something wrong when encoding the sentence?
I read that perhaps I should send the NMEA sentence continuosly, but I'm new in Python programming and I am not sure how to do that with sockets.
English is not my mother language, so please excuse my errors :)
Thnak you everyone.

When you are sending the GGA is it a position that has coverage on your rtk correction service? I have done this before, sent dummy positions that are outside the network coverage and nothing was returned, not even an error message, just no corrections.
Cheers,
Steve.

You need to add '\r\n' to the end of the NMEA string.
dummyNMEA = "$GPGGA,143741.356,7839.493,S,07627.626,W,0,00,,,M,,M,,*45\r\n"

You have to create an Header for the NMEA as well.
dummyHeader = \
"Ntrip-GGA: {}\r\n".format(dummyNMEA)
Then you should get an answer.

Python code for merchantos.com API use

I am trying to work through some code to connect to merchantos.com's rest API via Python.
With some research, I have managed to get the GET access working, using the following urllib2 code:
# NOTE: This api key has been made bogus
lcMOS_APIKey = '07203c82fab495xxxxxxxxxxxxxxxxxxxc2a499c'
# also bogus...
lcMOS_Acct = '98765'
lcBaseURL = 'https://api.merchantos.com/API/Account/' + lcMOS_Acct + '/'
# create a password manager
password_mgr = urllib2.HTTPPasswordMgrWithDefaultRealm()
password_mgr.add_password(None, lcBaseURL, lcMOS_APIKey, 'apikey')
# create "opener" (OpenerDirector instance)
handler = urllib2.HTTPBasicAuthHandler(password_mgr)
opener = urllib2.build_opener(handler)
urllib2.install_opener(opener)
# use the opener to fetch a URL
#loReturn = opener.open(lcBaseURL + lcURLEnd)
loReturn = opener.open(lcBaseURL + 'Customer.xml?firstName=Alex')
lcResponse = loReturn.read()
So, the above successfully pulls data back. I get an XML of the customer record.
Now, what I need to do is change the method so that I can do a PUT (for an update) and a POST (for a create/new).
MerchantOS requires the following for an update:
UPDATE / HTTP PUT
To update an existing record/object you do an HTTP PUT request. The put/post data should be an XML block defining the updates to the object. For example to update an Item you would PUT to API/Account/1/Item/2 with an block (1 is the account number and 2 the itemID in this example).
So, for example, I want to do a PUT to update customer ID = 2
I would provide a data reference to an XML block for the
<Customer>
..contents omitted here...
</Customer>
And, I am to point it to theURL.
The problems I am facing here are..
I do not know where/how to change the method to PUT
I need top know how to attach my data block and post it
So, can someone please show me how to adapt the above code for a GET to make a PUT .. as well as a POST (for creating a new record)
Thanks, in advance, for any assistance in this regard.
Scott.

You might try cURL instead of urllib. cURL is extremely flexible and addresses your needs:
http://pycurl.sourceforge.net/
Here are two of the options you can set with cURL:
CURLOPT_POST: A parameter set to 1 tells the library to do a regular HTTP post...
CURLOPT_POSTFIELDS: The full data to post in a HTTP POST operation...

Using python sockets to receive large http requests

I am using python sockets to receive web style and soap requests. The code I have is
import socket
svrsocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
host = socket.gethostname();
svrsocket.bind((host,8091))
svrsocket.listen(1)
clientSocket, clientAddress = svrsocket.accept()
message = clientSocket.recv(4096)
Some of the soap requests I receive, however, are huge. 650k huge, and this could become several Mb. Instead of the single recv I tried
message = ''
while True:
data = clientSocket.recv(4096)
if len(data) == 0:
break;
message = message + data
but I never receive a 0 byte data chunk with firefox or safari, although the python socket how to says I should.
What can I do to get round this?

Unfortunately you can't solve this on the TCP level - HTTP defines its own connection management, see RFC 2616. This basically means you need to parse the stream (at least the headers) to figure out when a connection could be closed.
See related questions here - https://stackoverflow.com/search?q=http+connection

Hiya
Firstly I want to reinforce what the previous answer said
Unfortunately you can't solve this on the TCP level
Which is true, you can't. However you can implement an http parser on top of your tcp sockets. And that's what I want to explore here.
Let's get started
Problem and Desired Outcome
Right now we are struggling to find the end to a datastream. We expected our stream to end with a fixed ending but now we know that HTTP does not define any message suffix
And yet, we move forward.
There is one question we can now ask, "Can we ever know the length of the message in advance?" and the answer to that is YES! Sometimes...
You see HTTP/1.1 defines a header called Content-Length and as you'd expect it has exactly what we want, the content length; but there is something else in the shadows: Transfer-Encoding: chunked. unless you really want to learn about it, we'll stay away from it for now.
Solution
Here is a solution. You're not gonna know what some of these functions are at first, but if you stick with me, I'll explain. Alright... Take a deep breath.
Assuming conn is a socket connection to the desired HTTP server
...
rawheaders = recvheaders(conn,end=CRLF)
headers = dict_headers(io.StringIO(rawheaders))
l_content = headers['Content-Length']
#okay. we've got content length by magic
buffersize = 4096
while True:
if l_content <= 0: break
data = clientSocket.recv(buffersize)
message += data
l_content -= len(data)
...
As you can see, we enter the loop already knowing the Content-Length as l_content
While we iterate we keep track of the remaining content by subtracting the length of clientSocket.recv(buff) from l_content.
When we've read at least as much data as l_content, we are done
if l_content <= 0: break
Frustration
Note: For some these next bits I'm gonna give psuedo code because the code can be a bit dense
So now you're asking, what is rawheaders = recvheaders(conn), what is headers = dict_headers(io.StringIO(rawheaders)),
and HOW did we get headers['Content-Length']?!
For starters, recvheaders. The HTTP/1.1 spec doesn't define a message suffix, but it does define something useful: a suffix for the http headers! And that suffix is CRLF aka \r\n.That means we know when we've recieved the headers when we read CRLF. So we can write a function like
def recvheaders(sock):
rawheaders = ''
until we read crlf:
rawheaders = sock.recv()
return rawheaders
Next, parsing the headers.
def dict_header(ioheaders:io.StringIO):
"""
parses an http response into the status-line and headers
"""
#here I expect ioheaders to be io.StringIO
#the status line is always the first line
status = ioheaders.readline().strip()
headers = {}
for line in ioheaders:
item = line.strip()
if not item:
break
//headers look like this
//'Header-Name' : 'Value'
item = item.split(':', 1)
if len(item) == 2:
key, value = item
headers[key] = value
return status, headers
Here we read the status line then we continue to iterate over every remaining line
and build [key,value] pairs from Header: Value with
item = line.strip()
item = item.split(':', 1)
# We do split(':',1) to avoid cases like
# 'Header' : 'foo:bar' -> ['Header','foo','bar']
# when we want ---------> ['Header','foo:bar']
then we take that list and add it to the headers dict
#unpacking
#key = item[0], value = item[1]
key, value = item
header[key] = value
BAM, we've created a map of headers
From there headers['Content-Length'] falls right out.
So,
This structure will work as long as you can guarantee that you will always recieve Content-Length
If you've made it this far WOW, thanks for taking the time and I hope this helped you out!
TLDR; if you want to know the length of an http message with sockets, write an http parser