Hope you're doing great !
The usecase
I'm trying to PoC something on AWS, the use case is that we need to be able to check on all our infrastructure that all instance are reachable through AWS Session Manager.
In order to do that, I will use a Lambda in Python 3.7, I make my PoC locally currently. I'm able to open the websocket, send the Token Payload and get an output that contains a shell.
The problem is that the byte output contains character that the python decode function can't decode in a lot of tested character encoding, every time something block.
The output
Here is the output I have after sending the payload :
print(event)
b'\x00\x00\x00toutput_stream_data \x00\x00\x00\x01\x00\x00\x01m\x1a\x1b\x9b\x15\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\xb1\x0b?\x19\x99A\xfc\xae%\xb2b\xab\xfd\x02A\xd7C\xcd\xd8}L\xa8\xb2J\xad\x12\xe3\x94\n\xed\xb81\xfa\xb6\x11\x18\xc2\xecR\xf66&4\x18\xf6\xbdd\x00\x00\x00\x01\x00\x00\x00\x10\x1b[?1034hsh-4.2$ '
What I already tried
I researched a lot on stackoverflow, tried to decode with ascii, cp1252, cp1251, cp1250, iso8859-1, utf-16, utf-8, utf_16_be, but everytime, it doesn't decode anything or it leads to an error because a character is unknown.
I also already tried to use chardet.detect, but the returned encoding is not working and also the probability result is really low. And also tried to strip the \x00 but strip doesn't work that time.
I already know that shell output can sometimes contains coloring character and some things that make it looks like garbled, but here, I tried to pass colorama on it, tried to match some ANSI character with some regex, nothing successfully decode this bytes response.
The code
Here is the code for my PoC, feel free to use it to try, you just have to change the target instance id (your instance needs to have the latest amazon-ssm-agent running on it).
import boto3
import uuid
import json
from websocket import create_connection
# Setting the boto3 client and the target
client = boto3.client('ssm','eu-west-1')
target = 'i-012345678910'
# Starting a session, this return a WebSocket URL and a Token for the Payload
response = client.start_session(Target=target)
# Creating a session with websocket.create_connection()
ws = create_connection(response['StreamUrl'])
# Building the Payload with the Token
payload = {
"MessageSchemaVersion": "1.0",
"RequestId": str(uuid.uuid4()),
"TokenValue": response['TokenValue']
}
# Sending the Payload
ws.send(json.dumps(payload))
# Receiving, printing and measuring the received message
event = ws.recv()
print(event)
print(len(event))
# Sending pwd, that should output /usr/bin
ws.send('pwd')
# Checking the result of the received message after the pwd
event = ws.recv()
print(event)
print(len(event))
Expected output
In the final solution, I expect to be able to do something like a curl http://169.254.169.254/latest/meta-data/instance-id through the websocket, and compare the instance-id of the command output against the target, to validate that instance is reachable. But I need to be able to decode the websocket output before achieving that.
Thank you in advance for any help on this.
Enjoy the rest of your day !
As per my reading of the amazon-ssm-agent code, the payload exchanged via the websocket connection and managed by the session-manager channels follow a specific structure called the AgentMessage.
You will have to comply with this structure to use session-manager with the remote agent through the MGS Service, which means serializing messages and deserializing responses.
The fields of the above struct are also broken down into models via additional structs.
It shouldn't be too long to re-implement that in python. Good luck!
Related
My problem: I try to send logs from python3 project via logging module to fluentd.
log = '{"#timestamp":"2020-06-18T11:52:37.391","severity":"INFO", "message":"Processing request started"}'
logging.error(json.dumps(log))
At fluentd I get such error:
pattern not matched data="<14>{"#timestamp":"2020-06-18T11:52:37.391","severity":"INFO", "message":"Processing request started"}\x00"
I see strange symbols, <14> and \x00. When I try to send same string via bash console --everything works well
echo -n '{"#timestamp":"2020-06-18T11:52:37.391","severity":"INFO", "message":"Processing request started"}' > /dev/udp/HOST/PORT
Looks like there are some problems with encoding, but I can't recognise how to fix this error in python.
i am dealing with cTrader Trading platform.
My project is written in python 3 on tornado.
And have issue in decoding the prtobuf message from report API Events.
Below will list everything what i achieved and where have the problem.
First cTrader have Rest API for Report
so i got the .proto file and generated it for python 3
proto file is called : cTraderReportingMessages5_9_pb2
from rest Report API getting the protobuf message and able to decode in the following way because i know which descriptor to pass for decoding
from models import cTraderReportingMessages5_9_pb2
from protobuf_to_dict import protobuf_to_dict
raw_response = yield async_client.fetch(base_url, method=method, body=form_data, headers=headers)
decoded_response = cTraderReportingMessages5_9_pb2._reflection.ParseMessage(descriptors[endpoint]['decode'], raw_response.body)
descriptors[endpoint]['decode'] = is my descriptor know exactly which descriptor to pass to decode my message
my content from cTraderReportingMessages5_9_pb2
# here is .proto file generated for python 3 is too big cant paste content here
https://ufile.io/2p2d6
So until here using rest api and know exactly which descriptor to pass, i am able to decode protobuf message and go forward.
2. Now the issue i face
Connecting with python 3 to the tunnel on 127.0.0.:5672
i am listening for events and receiving this kind of data back
b'\x08\x00\x12\x88\x01\x08\xda\xc9\x06\x10\xb6\xc9\x03\x18\xa1\x8b\xb8\x01 \x00*\x00:\x00B\x00J\x00R\x00Z\x00b\x00j\x00r\x00z\x00\x80\x01\xe9\x9b\x8c\xb5\x99-\x90\x01d\x98\x01\xea\x9b\x8c\xb5\x99-\xa2\x01\x00\xaa\x01\x00\xb0\x01\x00\xb8\x01\x01\xc0\x0
1\x00\xd1\x01\x00\x00\x00\x00\x00\x00\x00\x00\xd9\x01\x00\x00\x00\x00\x00\x00\x00\x00\xe1\x01\x00\x00\x00\x00\x00\x00\x00\x00\xea\x01\x00\xf0\x01\x01\xf8\x01\x00\x80\x02\x00\x88\x02\x00\x90\x02\x00\x98\x02\x00\xa8\x02\x00\xb0\x02\x00\xb8\x02\x90N\xc0\x02\x00\xc8\x0
2\x00
as recommendation i got, i need to use same .proto file generated for python that i did in step 1 and decode the message but without any success because i don't know the descriptor need to be passed.
so in 1 step was doing and working perfect this way
decoded_response = cTraderReportingMessages5_9_pb2._reflection.ParseMessage(descriptors[endpoint]['decode'], raw_response.body)
but in second step can not decode the message using in the same way, what i am missing or how to decode the message using same .proto file?
Finally found a workaround by my self, maybe is a primitive way but only this worked for me.
By the answer got from providers need to use same .proto file for both situations
SOLUTION:
1. Did list with all the descriptors from .proto file
here is .proto file generated for python 3 is too big cant paste content here
https://ufile.io/2p2d6
descriptors = [cTraderReportingMessages5_9_pb2.descriptor_1, cTraderReportingMessages5_9_pb2.descriptor_2]
2. Loop throw list and pass one by one
for d in descriptors:
decoded_response = cTraderReportingMessages5_9_pb2._reflection.ParseMessage(d, raw_response.body)
3. Check if decoded_response is not blank
if decoded_response:
# descriptor was found
# response is decoded
else:
# no descriptor
4. After decoded response we go parse it into dict:
from protobuf_to_dict import protobuf_to_dict
decoded_response_to_dict = protobuf_to_dict(decoded_response)
This solution that spent weeks on it finally worked.
Hi I am working on a simple program that takes data from a json file (input through an html form with flask handling the data) and uses this data to make calls to an API.
So I have some JSON like this:
[{"id": "ßLÙ", "server": "NA"}]
and I want to send the id to an api call like this example:
http://apicallnamewhatever+id=ßLÙ
however when i load the json file into my app.py with the following command
ids = json.load(open('../names.json'))
json.load seems to alter the id from 'ßLÙ' to 'ßLÙ'
im not sure why this happens during json.load, but i need to find a way to get 'ßLÙ' into the api call instead of the deformed 'ßLÙ'
It looks as if your names.json is encoded in "utf-8", but you are opening it as "windows-1252" [*] or something like that. Try
json.load(open('names.json', encoding="utf-8"))
and you probably should also URL-encode the id instead of concatenating it directly with that server address, something along these lines:
urllib2.quote(idExtractedFromJson.encode("utf-8")
[*] Thanks #jDo for pointing that out, I initially guessed the wrong codepage.
I'm new to Python and trying to create a email sending script via socket communication but can't seem to sign it with the dkimpy lib. I tried a couple of examples on the web but all returned the same error when running dkim.sign:
File "C:\Python34\lib\re.py", line 196, in split return _compile(pattern,flags).split(string, maxsplit)
TypeError: expected string or buffer
Near as I can tell, the first variable in the dkim.sign function should be a string so I tried readlines () and even .as_string() just to be sure. I have checked the message and it seems RFC822 compliant. But I'll double check if anyone thinks that might be the problem. Without the dkim.sign it works perfectly ( minus any security like SPF/DKIM )
This is a snippet of the code I'm using:
f=open('mail.htm','r')
text=MIMEText(f.read(),'html')
headers = Parser().parse(open('mail.htm', 'r'))
sender=headers['From']
receiver=headers['To']
subj=headers['Subject']
f.close()
private_key = open('default.pem').read()
headers = ['To', 'From', 'Subject']
sig = dkim.sign(text, 'default', '<mydomain.here>', private_key, include_headers=headers)
The parsed headers are also used as input to the socket sending script. I do have a dkim key for test purposes, but I do not think it even reaches that point.
Any insight?
EDIT: Ok, I just tried parsing the string ( instead of signing it ) with dkim.rfc822_parse from dkimpy lib and I get the following error:
return _compile(pattern, flags).split(string, maxsplit)
TypeError: can't use a bytes pattern on a string-like object
Am I reading this write or does it seem that the code is expecting a string but the pattern is in bytes?
FIXED: Odly enough I did not think to check the private_key. I manually created the key in Win so unbeknownst to me, windows added an invisible linebreak character that even vim or nano could not see. After removing it with MCEdit the program worked without a hitch.
Thanks for the help :)
if I remember correctly, dkim.sign expects the full message source as parameter, but you are passing a MIMEText object.
try passing text.as_string() instead
sig = dkim.sign(text.as_string(), .... )
python3 enforces a strong difference between processing bytes and string.
the easiest way I found to avoid conversion when using the dkim module is to stay in bytes, here is what I use:
from email.parser import BytesParser
import dkim
mail = BytesParser().parse (open('mail.eml', 'rb'))
print(dkim.verify( mail.as_bytes () ) )))
The "rb" is for opening the file in bytes mode.
Give it a try.
So, I am using python and BOTO to access my AWS SQS. I have some messages in the SQS which I can see from the AWS dashboard. However, when I try to get these messages through python, the characters that come through are just gibberish. Any idea what is going on here?
conn = boto.sqs.connect_to_region("us-east-1")
q = conn.get_queue('my-worker-queue')
print q
#read from message queue
message = q.read(60)
print message
print message.get_body()
Given the code above, I get the following:
Queue(https://queue.amazonaws.com/247124526695/my-worker-queue)
<boto.sqs.message.Message instance at 0x16f31b8>
??e??b?+??-
The text in the message queue is the following:
hello this is a test
I guess the reason is base64 decoding issue, as boto uses base64 for messages encoding and decoding. You can try to use get_body_encoded method:
print message.get_body_encoded()
Other option is convert to RawMessage:
from boto.sqs.message import RawMessage
q.set_message_class(RawMessage)
Update
Yes it is, it became clear with your test case:
>>> print 'hello this is a test'.decode('base64')
??e??b?+??-