Is this possible to pickle and unpickle data without creating a file for it? In the question that was suggested as duplicate I don't see how to unpickle this.
Pickle.dump to variable
I want to do this remotely so I can't create new files on the fly.
All examples with pickling and unpickling show use of pickle.dump, and pickle.load.
I read the docs, and the file argument can be a BytesIO object but when I try to use load() function, I get.
EOFError: Ran out of input
Can somebody give me some examples of how to do this?
What I currently have is:
a = A("some_random_string")
bio = BytesIO(b"some_bytes_data")
d = pickle.dump(a, bio)
f = pickle.Unpickler(bio).load()
The load gives me the above error. What am I doing wrong?
I was able to resolve it after all. Using dumps instead of dump made it possible for me.
An example of how it works in my case now:
from io import BytesIO
import pickle
a = b"asdf"
f = pickle.dumps(a)
file = BytesIO(f)
unpickled = pickle.load(file) # It is equal to initial value.
Maybe it helps someone else too.
Related
I'm currently working on a simple proof of concept for a pdf-editor application. The example is supposed to be a simplified python script showcasing how we could use the pdfrw library to edit PDF files with forms in them.
So, here's the issue. I'm not interested in writing the edited PDF to a file.
The idea is that file opening and closing is going to most likely be handled by external code and so I want all the edits in my files to be done in memory. I don't want to write the edited filestream to a local file.
Let me specify what I mean by this. I currently have a piece of code like this:
class FormFiller:
def __fill_pdf__(input_pdf_filestream : bytes, data_dict : dict):
template_pdf : pdfrw.PdfReader = pdfrw.PdfReader(input_pdf_filestream)
# <some editing magic here>
return template_pdf
def fillForm(self,mapper : FieldMapper):
value_mapping : dict = mapper.getValues()
filled_pdf : pdfrw.PdfReader = self.__fill_pdf__(self.filesteam, value_mapping)
#<this point is crucial>
def __init__(self, filestream : bytes):
self.filesteam : bytes = filestream
So, as you see the FormFiller constructor receives an array of bytes. In fact, it's an io.BytesIO object. The template_pdf variable uses a PdfReader object from the pdfrw library. Now, when we get to the #<this point is crucial> marker, I have a filled_pdf variable which is a PdfReader object. I would like to convert it to a filestream (a bytes array, or an io.BytesIO object if you will), and return it in that form. I don't want to write it to a file. However, the writer class provided by pdfrw (pdfrw.PdfWriter) does not allow for such an operation. It only provides a write(<filename>) method, which saves the PdfReader object to a pdf output file.
How should I approach this? Do you recommend a workaround? Or perhaps I should use a completely different library to accomplish this?
Please help :-(
To save your altered PDF to memory in an object that can be passed around (instead of writing to a file), simply create an empty instance of io.BytesIO:
from io import BytesIO
new_bytes_object = BytesIO()
Then, use pdfrw's PdfWriter.write() method to write your data to the empty BytesIO object:
pdfrw.PdfWriter.write(new_bytes_object, filled_pdf)
# I'm not sure about the syntax, I haven't used this lib before
This works because io.BytesIO objects act like a file object, also known as a file-like object. It and related classes like io.StringIO behave like files in memory, such as the object f created with the built-in function open below:
with open("output.txt", "a") as f:
f.write(some_data)
Before you attempt to read from new_bytes_object, don't forget to seek(0) back to the beginning, or rewind it. Otherwise, the object seems empty.
new_bytes_object.seek(0)
am new to Python and working a bit on pickle files.
I have already a pickle file called training.pickle and a txt file called danish.txt
I would like to import the danish.txt to the training.pickle file but i don't know how to do ?
I have tried some thing but am sure its wrong :-)
import pickle
file1=open('danish.txt','r')
file2=open('training.pickle','r')
obj=[file1.read(), file2.read()]
outfile.write("obj,training.pickle")
I don't know much about pickle but if you're just trying to add the data from "danish.txt" to the pickle file you should be able to just open the .txt, store the data in a variable, and then write the data in the pickle.
To demonstrate my thinking:
f = open("danish.txt", "r+")
data = f.read()
output = data
f.close() #this reads the .txt file
and then afterward you'd write "output" into the pickle file via whatever method you use to write a string variable to a pickle file.
P.S. as I said I don't know much about pickle, but if it works anything like writing to a .txt you'd have to change the r to a w because r means opening it in read mode. If its just reading it can't write, or atleast that's how it works with .txts. Also, if there's no particular reason why you're using a pickle to store data, why not just use a .txt?
I'm not sure how to word my question exactly, and I have seen some similar questions asked but not exactly what I'm trying to do. If there already is a solution please direct me to it.
Here is what I'm trying to do:
At my work, we have a few pkgs we've built to handle various data types. One I am working with is reading in a csv file into a std_io object (std_io is our all-purpose object class that reads in any type of data file).
I am trying to connect this to another pkg I am writing, so I can make an object in the new pkg, and covert it to a std_io object.
The problem is, the std_io object is meant to read an actual file, not take in an object. To get around this, I can basically write my data to temp.csv file then read it into a std_io object.
I am wondering if there is a way to eliminate this step of writing the temp.csv file.
Here is my code:
x #my object
df = x.to_df() #object class method to convert to a pandas dataframe
df.to_csv('temp.csv') #write data to a csv file
std_io_obj = std_read('temp.csv') #read csv file into a std_io object
Is there a way to basically pass what the output of writing the csv file would be directly into std_read? Does this make sense?
The only reason I want to do this is to avoid having to code additional functionality into either of the pkgs to directly accept an object as input.
Hope this was clear, and thanks to anyone who contributes.
For those interested, or who may have this same kind of issue/objective, here's what I did to solve this problem.
I basically just created a temporary named file, linked a .csv filename to this temp file, then passed it into my std_read function which requires a csv filename as an input.
This basically tricks the function into thinking it's taking the name of a real file as an input, and it just opens it as usual and uses csvreader to parse it up.
This is the code:
import tempfile
import os
x #my object I want to convert to a std_io object
text = x.to_df().to_csv() #object class method to convert to a pandas dataframe then generate the 'text' of a csv file
filename = 'temp.csv'
with tempfile.NamedTemporaryFile(dir = os.path.dirname('.')) as f:
f.write(text.encode())
os.link(f.name, filename)
stdio_obj = std_read(filename)
os.unlink(filename)
del f
FYI - the std_read function essentially just opens the file the usual way, and passes it into csvreader:
with open(filename, 'r') as f:
rdr = csv.reader(f)
I am figuring out the differences between the pickle.load() and pickle.loads(). Somebody said what kind of object that pickle.load() process is "file_like_object", however, pickle.loads() corresponds to "file object".
Your choice of which function to use depends on the object from whence you are loading the pickled data:
pickle.loads is used to load pickled data from a bytes string. The "s" in loads refers to the fact that in Python 2, the data was loaded from a string.
For example:
import pickle
with open("myobj.pickle", "rb") as f:
rawdata = f.read()
myobj = pickle.loads(rawdata)
pickle.load is used to load pickled data from a file-like object. This is any object that acts like a file - in this case, meaning it has a read() method that returns bytes.
For example:
import pickle
with open("myobj.pickle", "rb") as f:
myobj = pickle.load(f)
This same convention applies to the dump/dumps functions in the pickle library, as well as the json module and others.
I'm trying to use AjaxUpload with Python:
http://valums.com/ajax-upload/
I would like to know how to access the uploaded file with Python. On the web site, it says:
* PHP: $_FILES['userfile']
* Rails: params[:userfile]
What is the Syntax for Python?
request.params['userfile'] doesn't seem to work.
Thanks in advance! Here is my current code (using PIL imported as Image)
im = Image.open(request.params['myFile'].file)
import cgi
#This will give you the data of the file,
# but won't give you the filename, unfortunately.
# For that you have to do some other trick.
file_data = cgi.FieldStorage.getfirst('file')
#<IGNORE if you're not using mod_python>
#(If you're using mod_python you can also get the Request object
# by passing 'req' to the relevant function in 'index.py', like "def func(req):"
# Then you access it with req.form.getfirst('file') instead. NOTE that the
# first method will work even when using mod_python, but the first FieldStorage
# object called is the only one with relevant data, so if you pass 'req' to the
# function you have to use the method that uses 'req'.)
#</IGNORE>
#Then you can write it to a file like so...
file = open('example_filename.wtvr','w')#'w' is for 'write'
file.write(file_data)
file.close()
#Then access it like so...
file = open('example_filename.wtvr','r')#'r' is for 'read'
#And use file.read() or whatever else to do what you want.
I'm working with Pyramid, and I was trying to do the same thing. After some time I came up with this solution.
from cStringIO import StringIO
from cgi import FieldStorage
fs = FieldStorage(fp=request['wsgi.input'], environ=request)
f = StringIO(fs.value)
im = Image.open(f)
I'm not sure if it's the "right" one, but it seems to work.
in django, you can use:
request.FILES['file']
instead of:
request.POST['file']
i did not know how to do in pylons...maybe it is the same concept..