We Keep Coding

How to create a custom stream like stdin in python

I was wondering if there is a "stream"(I don't know if that is what you guys call it) like stdin/stdout/stderr that can be referenced later using open(). I was doing this for temporary data that is rather large and wanted to put it to a stream instead of a variable.
Would I use subprocess?

There are two possible ways:
tempfile is a module that can create temporary files for you with or without name and store them on disk and in memory which is similar to the next solution:
pipes is a module the allows you to create pipes which can be read and written to. These are both viable solutions

Is there a way to load variables directly from a binary file in python? [duplicate]

I understood that Python pickling is a way to 'store' a Python Object in a way that does respect Object programming - different from an output written in txt file or DB.
Do you have more details or references on the following points:
where are pickled objects 'stored'?
why is pickling preserving object representation more than, say, storing in DB?
can I retrieve pickled objects from one Python shell session to another?
do you have significant examples when serialization is useful?
does serialization with pickle imply data 'compression'?
In other words, I am looking for a doc on pickling - Python.doc explains how to implement pickle but seems not dive into details about use and necessity of serialization.

Pickling is a way to convert a python object (list, dict, etc.) into a character stream. The idea is that this character stream contains all the information necessary to reconstruct the object in another python script.
As for where the pickled information is stored, usually one would do:
with open('filename', 'wb') as f:
var = {1 : 'a' , 2 : 'b'}
pickle.dump(var, f)
That would store the pickled version of our var dict in the 'filename' file. Then, in another script, you could load from this file into a variable and the dictionary would be recreated:
with open('filename','rb') as f:
var = pickle.load(f)
Another use for pickling is if you need to transmit this dictionary over a network (perhaps with sockets or something.) You first need to convert it into a character stream, then you can send it over a socket connection.
Also, there is no "compression" to speak of here...it's just a way to convert from one representation (in RAM) to another (in "text").
About.com has a nice introduction of pickling here.

Pickling is absolutely necessary for distributed and parallel computing.
Say you wanted to do a parallel map-reduce with multiprocessing (or across cluster nodes with pyina), then you need to make sure the function you want to have mapped across the parallel resources will pickle. If it doesn't pickle, you can't send it to the other resources on another process, computer, etc. Also see here for a good example.
To do this, I use dill, which can serialize almost anything in python. Dill also has some good tools for helping you understand what is causing your pickling to fail when your code fails.
And, yes, people use picking to save the state of a calculation, or your ipython session, or whatever. You can also extend pickle's Pickler and UnPickler to do compression with bz2 or gzip if you'd like.

I find it to be particularly useful with large and complex custom classes. In a particular example I'm thinking of, "Gathering" the information (from a database) to create the class was already half the battle. Then that information stored in the class might be altered at runtime by the user.
You could have another group of tables in the database and write another function to go through everything stored and write it to the new database tables. Then you would need to write another function to be able to load something saved by reading all of that info back in.
Alternatively, you could pickle the whole class as is and then store that to a single field in the database. Then when you go to load it back, it will all load back in at once as it was before. This can end up saving a lot of time and code when saving and retrieving complicated classes.

it is kind of serialization. use cPickle it is much faster than pickle.
import pickle
##make Pickle File
with open('pickles/corups.pickle', 'wb') as handle:
pickle.dump(corpus, handle)
#read pickle file
with open('pickles/corups.pickle', 'rb') as handle:
corpus = pickle.load(handle)

Python function requires path but I have an image stored in memory

I have a python function (using the Pythonista app) to show an image in the console. I have the image saved in a BytesIO object but the function requires a file path.
Is there any way to give it a path to the bytesIO or somehow give it the image without needing to save it as a file?
The specific function is console.show_image(image_path)

The general answer is that if the function you call expects a filesystem path and cannot handle a file-like object instead then your only solution is to write your data to a file (and ask the function's author to add support for file-like object, or if it's OSS implement it by yourself and send a merge request).

Create a file object from raw binary information in Python

Question
What is a clean way to create a file object from raw binary information in Python?
More Info
The reason I need to do this is because I have the raw binary information comprising a jpeg image stored in ram. I need to put it inside some kind of file object so that I can resize the image using Python's Pillow library.
According to the pillow documentation, the file object needs to implement the read(), seek(), and tell() methods.
The file object must implement read(), seek(), and tell() methods, and be opened in binary mode.

I was able to find a mention of how to handle this situation under the documentation for PIL.Image.frombytes:
...If you have an entire image in a string, wrap it in a BytesIO object,
and use open() to load it.
This is what I ended up with that worked using BytesIO:
import io
import PIL
from PIL.Image import Image
file_body = <binary image data>
t_file = io.BytesIO(file_body)
img = PIL.Image.open(t_file)
Note: The comments mention tempfile.SpooledTemporaryFile. This seems like it should have worked, but it did not for some reason.

Python: what is the correct way to handle gzipped json?

I've found this snippet, which seems to do the job, but I can't understand why it uses StringIO. Isn't f already a file-like object? What is the need to read it, then make it look like a file again, only to read it again? I've tested it (well, a slightly modified version of it), and it doesn't work without StringIO.

Seems to be a flaw in python standard library which is fixed in Python 3.2.
see http://www.enricozini.org/2011/cazzeggio/python-gzip/
urllib and urllib2 file objects do not provide a method tell() as requested by gzip.

It's possible that the gunzip code needs a file-like object that has a seek method, which a HTTP library is very unlikely to provide. What does "doesn't work" mean? Error message?
If efficiency is your real concern, slightly modify the code so that it uses cStringIO, not StringIO.

The way I read the relevant part of the code says:
Open an url
Download it completely into memory (with the read method)
Store the content in a StringIO object, so that it's available as a file-like object
Do the gzip and json stuff with it.