We Keep Coding

Scapy - persistent RandIP

I am trying to simulate a TCP communication between two hosts with scapy.
My problem is, that I can't save the random IP addresses scapy generates for me.
This code
src_IP = RandIP()
print(src_IP)
print(src_IP)
print(src_IP)
gives me an output like this
234.200.98.20
147.3.56.17
135.102.142.49
So every time I access src_IP it has a new value.
Is there a way to save a random IP from scapy? So I could generated 2 IPs at the beginning of my function and use them as source and destination for my TCP communication.
I could generate the IPs myself, but I thought there had to be a more elegant solution to it.
BTW. non of the packets are going to be sent, they will be written into a PCAP file. Therefor I have to create both sides of the communication.

The accepted answer is a hack; it is correct (as in "it works and do what was asked"), but it is not the correct way to do that in Scapy.
The method you are looking for is ._fix(). It will work with any volatile value type.
src_IP = RandIP()._fix()
If you also need for example a random source port, you could do:
src_port = RandShort()._fix()

I found an answer.
RandIP() creates an instance of an object, and every time this object gets accessed, to print or do send a packet or something else, it generates a new IP.
So my solution is to cast it to a string
src_IP = str(RandIP())
print(src_IP)
print(src_IP)
print(src_IP)
And the output is just as intended
232.119.133.38
232.119.133.38
232.119.133.38

python printing out a array of bytes via serial port

I am kind a new to python and I am trying to print out an array of bytes which I am receiving via a serial interface.
The code Looks like this
print('Teststring')
ref.ser.write('outx=0\n'.encode())
ref.ser.flush()
s = self.ser.read(size=10)
print(s)
I am expecting the printout to look like this
Teststring
INPUT DATA (Input of my bytearray s)
So the behaviour is like I have an external device which I want to send an reply to my write/flush command and receive the data via the serial input and print out the received information on my computer.
Right now the issue is that the computer is not printing out the expected input data.
I guess my problem is that I need to convert the data s to a string or something like that?

Read pcap using scapy (DNS queries)

dnsPackets = rdpcap(dnsPcap)
domain = 'stackoverflow.com'
for packet in dnsPackets:
if packet.haslayer(DNSQR):
query = packet[DNSQR].qname
if domain in query:
outFile.write(query + '\n')
This code gives me a nice list of DNS queries containing the domain stackoverflow.com.
However, comparing the results in wireshark i can see that i have 1 more query, and that query is in a malformed packet (wireshark reads it fine). How would i go extracting that as well?
Thank you.

This was solved. Above code was correct but bug in code later on.

How to read JSON from socket in python? (Incremental parsing of JSON)

I have a socket opened and I'd like to read some json data from it. The problem is that the json module from standard library can only parse from strings (load only reads the whole file and calls loads inside) It even looks that all the way inside the module it all depends on the parameter being string.
This is a real problem with sockets since you can never read it all to string and you don't know how many bytes to read before you actually parse it.
So my questions are: Is there a (simple and elegant) workaround? Is there another json library that can parse data incrementally? Is it worth writing it myself?
Edit: It is XBMC jsonrpc api. There are no message envelopes, and I have no control over the format. Each message may be on a single line or on several lines.
I could write some simple parser that needs only getc function in some form and feed it using s.recv(1), but this doesn't as a very pythonic solution and I'm a little lazy to do that :-)

Edit: given that you aren't defining the protocol, this isn't useful, but it might be useful in other contexts.
Assuming it's a stream (TCP) socket, you need to implement your own message framing mechanism (or use an existing higher level protocol that does so). One straightforward way is to define each message as a 32-bit integer length field, followed by that many bytes of data.
Sender: take the length of the JSON packet, pack it into 4 bytes with the struct module, send it on the socket, then send the JSON packet.
Receiver: Repeatedly read from the socket until you have at least 4 bytes of data, use struct.unpack to unpack the length. Read from the socket until you have at least that much data and that's your JSON packet; anything left over is the length for the next message.
If at some point you're going to want to send messages that consist of something other than JSON over the same socket, you may want to send a message type code between the length and the data payload; congratulations, you've invented yet another protocol.
Another, slightly more standard, method is DJB's Netstrings protocol; it's very similar to the system proposed above, but with text-encoded lengths instead of binary; it's directly supported by frameworks such as Twisted.

If you're getting the JSON from an HTTP stream, use the Content-Length header to get the length of the JSON data. For example:
import httplib
import json
h = httplib.HTTPConnection('graph.facebook.com')
h.request('GET', '/19292868552')
response = h.getresponse()
content_length = int(response.getheader('Content-Length','0'))
# Read data until we've read Content-Length bytes or the socket is closed
data = ''
while len(data) < content_length or content_length == 0:
s = response.read(content_length - len(data))
if not s:
break
data += s
# We now have the full data -- decode it
j = json.loads(data)
print j

What you want(ed) is ijson, an incremental json parser.
It is available here: https://pypi.python.org/pypi/ijson/ . The usage should be simple as (copying from that page):
import ijson.backends.python as ijson
for item in ijson.items(file_obj):
# ...
(for those who prefer something self-contained - in the sense that it relies only on the standard library: I wrote yesterday a small wrapper around json - but just because I didn't know about ijson. It is probably much less efficient.)
EDIT: since I found out that in fact (a cythonized version of) my approach was much more efficient than ijson, I have packaged it as an independent library - see here also for some rough benchmarks: http://pietrobattiston.it/jsaone

Do you have control over the json? Try writing each object as a single line. Then do a readline call on the socket as described here.
infile = sock.makefile()
while True:
line = infile.readline()
if not line: break
# ...
result = json.loads(line)

Skimming the XBMC JSON RPC docs, I think you want an existing JSON-RPC library - you could take a look at:
http://www.freenet.org.nz/dojo/pyjson/
If that's not suitable for whatever reason, it looks to me like each request and response is contained in a JSON object (rather than a loose JSON primitive that might be a string, array, or number), so the envelope you're looking for is the '{ ... }' that defines a JSON object.
I would, therefore, try something like (pseudocode):
while not dead:
read from the socket and append it to a string buffer
set a depth counter to zero
walk each character in the string buffer:
if you encounter a '{':
increment depth
if you encounter a '}':
decrement depth
if depth is zero:
remove what you have read so far from the buffer
pass that to json.loads()

You may find JSON-RPC useful for this situation. It is a remote procedure call protocol that should allow you to call the methods exposed by the XBMC JSON-RPC. You can find the specification on Trac.

res = str(s.recv(4096), 'utf-8') # Getting a response as string
res_lines = res.splitlines() # Split the string to an array
last_line = res_lines[-1] # Normally, the last one is the json data
pair = json.loads(last_line)
https://github.com/A1vinSmith/arbitrary-python/blob/master/sockets/loopHost.py

How do I dump the TCP client's buffer in order to accept more data?

I've got a simple TCP server and client. The client receives data:
received = sock.recv(1024)
It seems trivial, but I can't figure out how to recieve data larger than the buffer. I tried chunking my data and sending it multiple times from the server (worked for UDP), but it just told me that my pipe was broken.
Suggestions?

If you have no idea how much data is going to pour over the socket, and you simply want to read everything until the socket closes, then you need to put socket.recv() in a loop:
# Assumes a blocking socket.
while True:
data = sock.recv(4096)
if not data:
break
# Do something with `data` here.

Mike's answer is the one you're looking for, but that's not a situation you want to find yourself in. You should develop an over-the-wire protocol that uses a fixed-length field that describes how much data is going to be sent. It's a Type-Length-Value protocol, which you'll find again and again and again in network protocols. It future-proofs your protocol against unforeseen requirements and helps isolate network transmission problems from programmatic ones.
The sending side becomes something like:
socket.write(struct.pack("B", type) #send a one-byte msg type
socket.write(struct.pack("H", len(data)) #send a two-byte size field
socket.write(data)
And the receiving side something like:
type = socket.read(1) # get the type of msg
dataToRead = struct.unpack("H", socket.read(2))[0] # get the len of the msg
data = socket.read(dataToRead) # read the msg
if TYPE_FOO == type:
handleFoo(data)
elif TYPE_BAR == type:
handleBar(data)
else:
raise UnknownTypeException(type)
You end up with an over-the-wire message format that looks like:
struct {
unsigned char type;
unsigned short length;
void *data;
}

Keep in mind that:
Your operating system has it's own idea of what it's TCP/IP socket buffer size is.
TCP/IP packet maximum size (generally is 1500 bytes)
pydoc for socket suggests that 4096 is a good buffer size
With that said, it'd really be helpful to see the code around that one line. There are a few things that could play into this, if you're using select or just polling, is the socket non-blocking, etc.
It also matters how you're sending the data, if your remote end disconnects. More details.