I am trying to write a dictionary to a txt file. Then read the dict values by typing the keys with raw_input. I feel like I am just missing one step but I have been looking for a while now.
I get this error
File "name.py", line 24, in reading
print whip[name]
TypeError: string indices must be integers, not str
My code:
#!/usr/bin/env python
from sys import exit
class Person(object):
def __init__(self):
self.name = ""
self.address = ""
self.phone = ""
self.age = ""
self.whip = {}
def writing(self):
self.whip[p.name] = p.age, p.address, p.phone
target = open('deed.txt', 'a')
target.write(str(self.whip))
print self.whip
def reading(self):
self.whip = open('deed.txt', 'r').read()
name = raw_input("> ")
if name in self.whip:
print self.whip[name]
p = Person()
while True:
print "Type:\n\t*read to read data base\n\t*write to write to data base\n\t*exit to exit"
action = raw_input("\n> ")
if "write" in action:
p.name = raw_input("Name?\n> ")
p.phone = raw_input("Phone Number?\n> ")
p.age = raw_input("Age?\n> ")
p.address = raw_input("Address?\n>")
p.writing()
elif "read" in action:
p.reading()
elif "exit" in action:
exit(0)
Have you tried the json module? JSON format is very similar to python dictionary. And it's human readable/writable:
>>> import json
>>> d = {"one":1, "two":2}
>>> json.dump(d, open("text.txt",'w'))
This code dumps to a text file
$ cat text.txt
{"two": 2, "one": 1}
Also you can load from a JSON file:
>>> d2 = json.load(open("text.txt"))
>>> print d2
{u'two': 2, u'one': 1}
Your code is almost right! You are right, you are just missing one step. When you read in the file, you are reading it as a string; but you want to turn the string back into a dictionary.
The error message you saw was because self.whip was a string, not a dictionary. So you need to convert the string to a dictionary.
Example
Here is the simplest way: feed the string into eval(). Like so:
def reading(self):
s = open('deed.txt', 'r').read()
self.whip = eval(s)
You can do it in one line, but I think it looks messy this way:
def reading(self):
self.whip = eval(open('deed.txt', 'r').read())
But eval() is sometimes not recommended. The problem is that eval() will evaluate any string, and if someone tricked you into running a really tricky string, something bad might happen. In this case, you are just running eval() on your own file, so it should be okay.
But because eval() is useful, someone made an alternative to it that is safer. This is called literal_eval and you get it from a Python module called ast.
import ast
def reading(self):
s = open('deed.txt', 'r').read()
self.whip = ast.literal_eval(s)
ast.literal_eval() will only evaluate strings that turn into the basic Python types, so there is no way that a tricky string can do something bad on your computer.
EDIT
Actually, best practice in Python is to use a with statement to make sure the file gets properly closed. Rewriting the above to use a with statement:
import ast
def reading(self):
with open('deed.txt', 'r') as f:
s = f.read()
self.whip = ast.literal_eval(s)
In the most popular Python, known as "CPython", you usually don't need the with statement as the built-in "garbage collection" features will figure out that you are done with the file and will close it for you. But other Python implementations, like "Jython" (Python for the Java VM) or "PyPy" (a really cool experimental system with just-in-time code optimization) might not figure out to close the file for you. It's good to get in the habit of using with, and I think it makes the code pretty easy to understand.
To store Python objects in files, use the pickle module:
import pickle
a = {
'a': 1,
'b': 2
}
with open('file.txt', 'wb') as handle:
pickle.dump(a, handle)
with open('file.txt', 'rb') as handle:
b = pickle.loads(handle.read())
print a == b # True
Notice that I never set b = a, but instead pickled a to a file and then unpickled it into b.
As for your error:
self.whip = open('deed.txt', 'r').read()
self.whip was a dictionary object. deed.txt contains text, so when you load the contents of deed.txt into self.whip, self.whip becomes the string representation of itself.
You'd probably want to evaluate the string back into a Python object:
self.whip = eval(open('deed.txt', 'r').read())
Notice how eval sounds like evil. That's intentional. Use the pickle module instead.
Hi there is a way to write and read the dictionary to file you can turn your dictionary to JSON format and read and write quickly just do this :
To write your date:
import json
your_dictionary = {"some_date" : "date"}
f = open('destFile.txt', 'w+')
f.write(json.dumps(your_dictionary))
and to read your data:
import json
f = open('destFile.txt', 'r')
your_dictionary = json.loads(f.read())
I created my own functions which work really nicely:
def writeDict(dict, filename, sep):
with open(filename, "a") as f:
for i in dict.keys():
f.write(i + " " + sep.join([str(x) for x in dict[i]]) + "\n")
It will store the keyname first, followed by all values. Note that in this case my dict contains integers so that's why it converts to int. This is most likely the part you need to change for your situation.
def readDict(filename, sep):
with open(filename, "r") as f:
dict = {}
for line in f:
values = line.split(sep)
dict[values[0]] = {int(x) for x in values[1:len(values)]}
return(dict)
You can iterate through the key-value pair and write it into file
pair = {'name': name,'location': location}
with open('F:\\twitter.json', 'a') as f:
f.writelines('{}:{}'.format(k,v) for k, v in pair.items())
f.write('\n')
Related
I need to read some JSON data for processing. I have a single line file that has multiple JSON objects how can I parse this?
I want the output to be a file with a single line per object.
I have tried a brute force method that will use json.loads recursively to check if the json is valid but I'm getting different results every time I run the program
import json
with open('sample.json') as inp:
s = inp.read()
jsons = []
start, end = s.find('{'), s.find('}')
while True:
try:
jsons.append(json.loads(s[start:end + 1]))
print(jsons)
except ValueError:
end = end + 1 + s[end + 1:].find('}')
else:
s = s[end + 1:]
if not s:
break
start, end = s.find('{'), s.find('}')
for x in jsons:
writeToFilee(x)
The json format can be seen here
https://pastebin.com/DgbyjAG9
why not just use the pos attribute of the JSONDecodeError to tell you where to delimit things?
something like:
import json
def json_load_all(buf):
while True:
try:
yield json.loads(buf)
except json.JSONDecodeError as err:
yield json.loads(buf[:err.pos])
buf = buf[err.pos:]
else:
break
works with your demo data as:
with open('data.json') as fd:
arr = list(json_load_all(fd.read()))
gives me exactly two elements, but I presume you have more?
to complete this using the standard library, writing out would look something like:
with open('data.json') as inp, open('out.json', 'w') as out:
for obj in json_load_all(inp.read()):
json.dump(obj, out)
print(file=out)
otherwise the jsonlines package is good for dealing with this data format
The code below worked for me:
import json
with open(input_file_path) as f_in:
file_data = f_in.read()
file_data = file_data.replace("}{", "},{")
file_data = "[" + file_data + "]"
data = json.loads(file_data)
Following #Chris A's comment, I've prepared this snippet which should work just fine:
with open('my_jsons.file') as file:
json_string = file.read()
json_objects = re.sub('}\s*{', '}|!|{', json_string).split('|!|')
# replace |!| with whatever suits you best
for json_object in json_objects:
print(json.loads(obj))
This example, however, will become worthless as soon as '}{' string appears in some value inside your JSON, so I strongly recommend using #Sam Mason's solution
I need to save a dictionary and then be able to read the dictionary after it's been saved.
This is what I have and it should work (i think), but i keep getting the following error when it comes to the read_dict function:
return dict(line.split() for line in x)
ValueError: dictionary update sequence element #0 has length 1; 2 is required
Any advice?
def save_dict(dict1):
with open('save.txt', 'w') as fh:
for key in dict1.keys():
fh.write(key + '' + dictionary1[key] + '\n')
def readDB():
with open('save.txt', 'r') as fh:
return dict(new.split() for new in fh)
Unless you actually need a line-by-line list in the file, use something like json or pickle to save the dict. These formats deal with things like spaces in the key name, non-string values, non-ascii characters and such.
import json
dict1 = {'test':123}
with open('save.txt', 'w') as fh:
json.dump(dict1, fh)
with open('save.txt', 'r') as fh:
dict2 = json.load(fh)
Use space instead of empty string, otherwise str.split will return a single item list which is going to raise an error when passed to dict().
fh.write(key + ' ' + dictionary1[key] + '\n')
Or better use string formatting:
for key, val in dict1.items():
fh.write('{} {}\n'.format(key, val))
Demo:
>>> s = 'k' + '' + 'v' #WRONG
>>> s
'kv'
>>> s.split()
['kv']
>>> s = 'k' + ' ' + 'v' #RIGHT
>>> s
'k v'
>>> s.split()
['k', 'v']
You probably need to use pickle module man!
Check out this example :
## Importing
from pickle import dump
## You make the dictionary
my_dict = {'a':1 , 'b':2 , 'c':3}
## You dump the dictionary's items to a binary (.txt file for windows)
with open('the path you want to save','wb') as da_file:
dump(my_dict , da_file)
save that file as "something0.py"
## Importing
from pickle import load
## You getting the data back from file
## the variable that will get the result of load module
## will be the same type with the variable that "dumped"
## the items to that file!
with open('the file path which you will get the items from' , 'rb') as da_file:
my_dict = load(da_file)
## Print out the results
from pprint import pprint
pprint(my_dict)
save that file as "something1.py"
Now run the two modules with the same file on "with" statement,
first 0 then 1 .
And 1 will print you the same results that the 0 gave to the file!
As mentioned you should use pickle, but as a more simplified way
FileTowriteto = open("foo.txt", "wb")
import pickle
DumpingDict = {"Foo":"Foo"}
pickle.dump(DumpingDict, FileTowriteto)
Then when you want to read it you can do this
OldDict = open("foo.txt", "rb")
OldDictRecover = pickle.load(OldDict)
This should work, and if the output is binary run the str() function on it.
I have problem with changing a dict value and saving the dict to a text file (the format must be same), I only want to change the member_phone field.
My text file is the following format:
memberID:member_name:member_email:member_phone
and I split the text file with:
mdict={}
for line in file:
x=line.split(':')
a=x[0]
b=x[1]
c=x[2]
d=x[3]
e=b+':'+c+':'+d
mdict[a]=e
When I try change the member_phone stored in d, the value has changed not flow by the key,
def change(mdict,b,c,d,e):
a=input('ID')
if a in mdict:
d= str(input('phone'))
mdict[a]=b+':'+c+':'+d
else:
print('not')
and how to save the dict to a text file with same format?
Python has the pickle module just for this kind of thing.
These functions are all that you need for saving and loading almost any object:
import pickle
with open('saved_dictionary.pkl', 'wb') as f:
pickle.dump(dictionary, f)
with open('saved_dictionary.pkl', 'rb') as f:
loaded_dict = pickle.load(f)
In order to save collections of Python there is the shelve module.
Pickle is probably the best option, but in case anyone wonders how to save and load a dictionary to a file using NumPy:
import numpy as np
# Save
dictionary = {'hello':'world'}
np.save('my_file.npy', dictionary)
# Load
read_dictionary = np.load('my_file.npy',allow_pickle='TRUE').item()
print(read_dictionary['hello']) # displays "world"
FYI: NPY file viewer
We can also use the json module in the case when dictionaries or some other data can be easily mapped to JSON format.
import json
# Serialize data into file:
json.dump( data, open( "file_name.json", 'w' ) )
# Read data from file:
data = json.load( open( "file_name.json" ) )
This solution brings many benefits, eg works for Python 2.x and Python 3.x in an unchanged form and in addition, data saved in JSON format can be easily transferred between many different platforms or programs. This data are also human-readable.
Save and load dict to file:
def save_dict_to_file(dic):
f = open('dict.txt','w')
f.write(str(dic))
f.close()
def load_dict_from_file():
f = open('dict.txt','r')
data=f.read()
f.close()
return eval(data)
As Pickle has some security concerns and is slow (source), I would go for JSON, as it is fast, built-in, human-readable, and interchangeable:
import json
data = {'another_dict': {'a': 0, 'b': 1}, 'a_list': [0, 1, 2, 3]}
# e.g. file = './data.json'
with open(file, 'w') as f:
json.dump(data, f)
Reading is similar easy:
with open(file, 'r') as f:
data = json.load(f)
This is similar to this answer, but implements the file handling correctly.
If the performance improvement is still not enough, I highly recommend orjson, fast, correct JSON library for Python build upon Rust.
I'm not sure what your first question is, but if you want to save a dictionary to file you should use the json library. Look up the documentation of the loads and puts functions.
I would suggest saving your data using the JSON format instead of pickle format as JSON's files are human-readable which makes your debugging easier since your data is small. JSON files are also used by other programs to read and write data. You can read more about it here
You'll need to install the JSON module, you can do so with pip:
pip install json
# To save the dictionary into a file:
json.dump( data, open( "myfile.json", 'w' ) )
This creates a json file with the name myfile.
# To read data from file:
data = json.load( open( "myfile.json" ) )
This reads and stores the myfile.json data in a data object.
For a dictionary of strings such as the one you're dealing with, it could be done using only Python's built-in text processing capabilities.
(Note this wouldn't work if the values are something else.)
with open('members.txt') as file:
mdict={}
for line in file:
a, b, c, d = line.strip().split(':')
mdict[a] = b + ':' + c + ':' + d
a = input('ID: ')
if a not in mdict:
print('ID {} not found'.format(a))
else:
b, c, d = mdict[a].split(':')
d = input('phone: ')
mdict[a] = b + ':' + c + ':' + d # update entry
with open('members.txt', 'w') as file: # rewrite file
for id, values in mdict.items():
file.write(':'.join([id] + values.split(':')) + '\n')
I like using the pretty print module to store the dict in a very user-friendly readable form:
import pprint
def store_dict(fname, dic):
with open(fname, "w") as f:
f.write(pprint.pformat(dic, indent=4, sort_dicts=False))
# note some of the defaults are: indent=1, sort_dicts=True
Then, when recovering, read in the text file and eval() it to turn the string back into a dict:
def load_file(fname):
try:
with open(fname, "r") as f:
dic = eval(f.read())
except:
dic = {}
return dic
Unless you really want to keep the dictionary, I think the best solution is to use the csv Python module to read the file.
Then, you get rows of data and you can change member_phone or whatever you want ;
finally, you can use the csv module again to save the file in the same format
as you opened it.
Code for reading:
import csv
with open("my_input_file.txt", "r") as f:
reader = csv.reader(f, delimiter=":")
lines = list(reader)
Code for writing:
with open("my_output_file.txt", "w") as f:
writer = csv.writer(f, delimiter=":")
writer.writerows(lines)
Of course, you need to adapt your change() function:
def change(lines):
a = input('ID')
for line in lines:
if line[0] == a:
d=str(input("phone"))
line[3]=d
break
else:
print "not"
I haven't timed it but I bet h5 is faster than pickle; the filesize with compression is almost certainly smaller.
import deepdish as dd
dd.io.save(filename, {'dict1': dict1, 'dict2': dict2}, compression=('blosc', 9))
file_name = open("data.json", "w")
json.dump(test_response, file_name)
file_name.close()
or use context manager, which is better:
with open("data.json", "w") as file_name:
json.dump(test_response, file_name)
I have an input file in the below format:
<ftnt>
<p><su>1</su> aaaaaaaaaaa </p>
</ftnt>
...........
...........
...........
... the <su>1</su> is availabe in the .........
I need to convert this to the below format by replacing the value and deleting the whole data in ftnt tags:
"""...
...
... the aaaaaaaaaaa is available in the ..........."""
Please find the code which i have written. Initially i saved the keys & values in dictionary and tried to replace the value based on the key using grouping.
import re
dict = {}
in_file = open("in.txt", "r")
outfile = open("out.txt", "w")
File1 = in_file.read()
infile1 = File1.replace("\n", " ")
for mo in re.finditer(r'<p><su>(\d+)</su>(.*?)</p>',infile1):
dict[mo.group(1)] = mo.group(2)
subval = re.sub(r'<p><su>(\d+)</su>(.*?)</p>','',infile1)
subval = re.sub('<su>(\d+)</su>',dict[\\1], subval)
outfile.write(subval)
I tried to use dictionary in re.sub but I am getting a KeyError. I don't know why this happens could you please tell me how to use. I'd appreciate any help here.
Try using a lambda for the second argument to re.sub, rather than a string with backreferences:
subval = re.sub('<su>(\d+)</su>',lambda m:dict[m.group(1)], subval)
First off, don't name dictionaries dict or you'll destroy the dict function. Second, \\1 doesn't work outside of a string hence the syntax error. I think the best bet is to take advantage of str.format
import re
# store the substitutions
subs = {}
# read the data
in_file = open("in.txt", "r")
contents = in_file.read().replace("\n", " ")
in_file.close()
# save some regexes for later
ftnt_tag = re.compile(r'<ftnt>.*</ftnt>')
var_tag = re.compile(r'<p><su>(\d+)</su>(.*?)</p>')
# pull the ftnt tag out
ftnt = ftnt_tag.findall(contents)[0]
contents = ftnt_tag.sub('', contents)
# pull the su
for match in var_tag.finditer(ftnt):
# added s so they aren't numbers, useful for format
subs["s" + match.group(1)] = match.group(2)
# replace <su>1</su> with {s1}
contents = re.sub(r"<su>(\d+)</su>", r"{s\1}", contents)
# now that the <su> are the keys, we can just use str.format
out_file = open("out.txt", "w")
out_file.write( contents.format(**subs) )
out_file.close()
I've been tasked with password-protecting a Java app with minimal concern for real security. So it seemed sensible to store username/password pairs in a text file and then encrypt it. For the encryption, it seemed appropriate to use XOR ciphers because they're easy and fast (remember--it just has to discourage the casual user, not be bulletproof).
I wrote all the appropriate Java, and then realized I needed a way to encrypt the config file. I wrote an additional method, but it was clunky to use more than once or twice (and seemed to only work for some inputs), so I decided it was best to write something in Python, to be played with at a REPL.
This is what I ended up with:
from itertools import izip, cycle
KEY = "stackoverflow"
def encrypt(text):
return ''.join(chr(ord(x) ^ ord(y)) for (x,y) in izip(text,cycle(KEY)))
def decrypt(text):
return encrypt(text)
def export(users, file):
with open(file, "w") as f:
for user, password in users.items():
f.write(encrypt('"%s" "%s"'%(user, password)) + "\n")
def import_data(file):
with open(file) as f:
return [decrypt(i) for i in f.readlines()]
On the surface, it works:
>>> x = encrypt("Hello world!")
>>> x
';\x11\r\x0f\x04O\x01\n\x00\n\x08N'
>>> decrypt(x)
'Hello world!'
But then things start to fall apart:
>>> export({"foo" : "bar", "baz" : "quux", "spam" : "eggs"}, "users.dat")
>>> import_data("users.dat")
['"foo" "bar"e', '"baz" "quux"}', '"spam" "eggs"y']
And here's how vim reads it -
And then:
>>> export({"what" : "not", "this" : "that", "admin_istrator" : "quux"}, "users2.dat")
>>> import_data("users2.dat")
['"thi', "k97$ma{~'l", '"what" "not"}', '"admin_istrator" "quux', '7~']
Vim:
It occurred to me that I might be having a problem with a character's encrypted form being a newline, but as far as I see that doesn't explain the wacky behavior in the first example or all of the wacky behavior in the second one.
Regarding the newlines, my Plan B is to encrypt the entire file--newlines and all--and then slurp it back up, decrypt it, split it on "\n", and proceed with my line-based parsing.
Thanks in advance.
Update: Here's my implementation of Plan B (described two paragraphs ago).
def import2(file):
with open(file) as f:
return decrypt(f.read())
and then:
>>> export({"foo" : "bar", "this" : "that", "admin_istrator" : "letmein"}, "users2.dat")
>>> import2("users2.dat")
'"this" "that"y%smg&91uux!}"admin_istrator" "letmein"y'
Update Two: Binary.
[Code is the same as above, except that all opens are open(file, "rb") or open(file, "wb").]
>>> export({"foo" : "bar", "this" : "that", "admin_istrator" : "letmein"}, "users2.dat")
>>> import2("users2.dat")
'"this" "that"y%smg&91uux!}"admin_istrator" "letmein"y'
>>> import_data("users2.dat")
['"t', "k97$ma{~'", '"foo" "bar"', '"admin_istrator" "letmein"']
Final update: Base 64, other shenanigans.
def import2(file):
with open(file, "rb") as f:
return filter(str.strip, [decrypt(i) for i in f.readlines()])
where encrypt and decrypt encode in/decode base 64.
You are trying to store binary in text mode file. Use open(file, "wb") for writing and open(file, "rb") for reading to open file in binary mode and fix an issue.
In text mode every "\r", "\n" and "\r\n" sequences are treated as newlines so they are converted into your local OS line ending convention ("\r\n" for Windows, "\n" for Unix, "\r" for old Macs). In case you read them from text file, you will always get "\n" instead, in case you write them, I don't remember actual behavior, but you will surely also get mess instead of your data :)
And with XOR encryption it's very likely you'll run into this kind of stuff :)
If you are forced not to use binary files, try base64 encoding (e.g. "some\0te\n\nxt with bi\x01naries".encode('base64')). To decode use .decode (thanks, Captain Obvious!).
The problem is you are not reading the same data you codified (you add a '\n' after the encryption), just do a rstrip() of the data you read:
def import_data(file):
with open(file) as f:
return [decrypt(i.rstrip()) for i in f.readlines()]
You can fix it by encrypting the newlines and not resetting the key between lines
from itertools import izip, cycle
KEY = "stackoverflow"
def encrypt(text):
return ''.join(chr(ord(x) ^ ord(y)) for (x,y) in izip(text,key))
def decrypt(text):
return encrypt(text)
def export(users, file):
with open(file, "w") as f:
for user, password in users.items():
f.write(encrypt('"%s" "%s"\n'%(user, password)))
def import_data(file):
with open(file) as f:
return [decrypt(i) for i in f]
key = cycle(KEY)
export({"foo" : "bar", "baz" : "quux", "spam" : "eggs"}, "users.dat")
key = cycle(KEY)
for row in import_data("users.dat"):
print row
This should be turned into a class, and key would be an instance variable instead of a global as it is here