We Keep Coding

Merge files created with JSON arrays into a single file

just to get you started, I want to merge json array files into a single file, with (comma) appended to the end of the array.
MemoryError now in my code, please help me!
in my code >
import os, sys
path = "censored"
dirs = os.listdir(path)
save_list = []
s = ""
for file in dirs:
save_list.append(file)
for i in range(len(save_list)):
f = open(path + save_list[i], 'r')
s += f.read()
s += s.replace("]", "],")
f.close()
ff = open("a", 'w')
ff.write(s)
ff.close()
print("done")
Error >
Traceback (most recent call last):
File "test.py", line 15, in <module>
s += s.replace("]", "],")
MemoryError
Want result
file "a" in substance
[{a:b}]
file "b" in substance
[{c:d}]
file "c" want result substance
[{a:b}], [{c:d}]

Based on your code, you're opening every single file one after another, and only closing the last one. You should be opening and closing each file after you are done with it to make it more memory efficient.
You can actually progressively write to the output file instead of storing it as a string in memory and writing it at one shot.
There's actually no need to save all files from dir into save_list and reaccess it in another loop. So you can omit save_list.
Putting everything together, you'll get the following code snippet:
# everything above as follows
for file in dirs:
curr_file_path = path + file
curr_file_string = ""
# using this would close the file automatically
with open(curr_file_path, 'r') as f:
raw_file = f.read()
curr_file_string = raw_file.replace("]", "],")
# open the output file and set the mode to append ('a') to batch write
# similarly, this will close the output file after every write
with open("output file", "a") as out_f:
out_f.write(curr_file_string)
print("done")

Python CSV Error

Hello all I keep getting this error while making a small program to sort large CSV files out, below is my code and error, what am I doing wrong?
if selection:
for stuff in stuffs:
try:
textFile = open("output.txt",'w')
mycsv = csv.reader(open(stuff))
d_reader = csv.DictReader(mycsv)
headers = d_reader.fieldnames <-- Error happens here
if selection in headers:
placeInList = headers.index(selection)
#placeInList = selection.index(selection)
for selection in tqdm(mycsv, desc='Extracting column values...', leave = True):
textFile.write(str(selection[int(placeInList)])+'\n')
print 'Done!'
textFile.close()
sys.exit()
except IOError:
print 'No CSV file present in directory'
sys.exit()
else:
sys.exit()
And the error:
Traceback (most recent call last):
File "postcodeExtractor.py", line 27, in <module> headers = d_reader.fieldnames
File "C:\Python27\lib\csv.py", line 90, in fieldnames self._fieldnames = self.reader.next()
TypeError: expected string or Unicode object, list found

instead of
mycsv = csv.reader(open(stuff))
d_reader = csv.DictReader(mycsv)
you want
d_reader = csv.DictReader(open(stuff))
the first line is the problem.

using FIFOs for input and output in python

To communicate with a shell, which is started once and runs in a separate process, I used Popen from subprocess.
import os
from subprocess import Popen, PIPE
def server():
FIFO_PATH = '/tmp/my_fifo'
FIFO_PATH2 = '/tmp/in_fifo'
if os.path.exists(FIFO_PATH):
os.unlink(FIFO_PATH)
if os.path.exists(FIFO_PATH2):
os.unlink(FIFO_PATH2)
if not os.path.exists(FIFO_PATH2):
os.mkfifo(FIFO_PATH2)
in_fifo = open(FIFO_PATH2, 'rw+')
print "in_fifo:", in_fifo
if not os.path.exists(FIFO_PATH):
os.mkfifo(FIFO_PATH)
my_fifo = open(FIFO_PATH, 'rw+')
print "my_fifo:", my_fifo
p = Popen(['python', '-u', 'shell.py'], shell=False, stdin=in_fifo, stdout=my_fifo)
def read():
FIFO_PATH = '/tmp/my_fifo'
i=0
while i < 10:
++i
print i, open(FIFO_PATH, 'r').readline()
def write(input):
FIFO_PATH2 = '/tmp/in_fifo'
pipe = open(FIFO_PATH2, 'w+')
pipe.write(input+'\n')
def test():
server()
write('test')
read()
and the shell.py
Input = ' '
print 'shell called'
while Input!= 'z':
Input=raw_input()
print 'input ', Input
if Input != '':
if Input == 'test':
print 'Yeehhaaaaa it works'
so calling test() give the following result
in_fifo: <open file '/tmp/in_fifo', mode 'rw+' at 0x7f0a4e17ed20>
my_fifo: <open file '/tmp/my_fifo', mode 'rw+' at 0x7f0a4e17edb0>
0 shell called
0 input test
Questions
Why is only the first line printed? How to print all lines?
Also I'm not sure about the proper use of FIFOs. Maybe there are better ways to get this done. I'm open for any suggestions.
Using p to call p.stdin.write() and p.stdout.readline() is no solution for me because I have to call the functions from javascript without having the instance p.

From the man page for mkfifo:
Opening a FIFO for reading normally blocks until some other process opens the same FIFO for writing, and vice versa. See fifo(7) for nonblocking handling of FIFO special files.
So the second time you open the FIFO for reading, the call blocks. This can be seen in the traceback after pressing Ctrl+C:
^CTraceback (most recent call last):
0
Traceback (most recent call last):
File "shell_fifo.py", line 51, in <module>
File "shell.py", line 4, in <module>
test()
File "shell_fifo.py", line 48, in test
read()
File "shell_fifo.py", line 29, in read
print i, open(FIFO_PATH, 'r').readline() # read() is blocked here
KeyboardInterrupt
Input=raw_input()
KeyboardInterrupt
Change your read function so that it only opens the FIFO once:
def read():
FIFO_PATH = '/tmp/my_fifo'
i = 0
with open(FIFO_PATH, 'r') as read_fifo:
while i < 10:
i += 1
print i, read_fifo.readline().rstrip()
You should see output like this:
in_fifo: <open file '/tmp/in_fifo', mode 'rw+' at 0x7f1ba655b5d0>
my_fifo: <open file '/tmp/my_fifo', mode 'rw+' at 0x7f1ba655b540>
1 shell called
2 input test
3 Yeehhaaaaa it works

Read a list of hostnames and resolve to IP addresses

i'm attempting to read a plain text file and resolve each IP address and (for now) just spit them back out on-screen.
import socket
f = open("test.txt")
num_line = sum(1 for line in f)
f.close()
with open("test.txt", "r") as ins:
array = []
for line in ins:
array.append(line)
for i in range(0,num_line):
x = array[i]
print x
data = socket.gethostbyname_ex(x)
print data
Currently I'm getting the following:
me#v:/home/# python resolve-list2.py
test.com
Traceback (most recent call last):
File "resolve-list2.py", line 15, in <module>
data = socket.gethostbyname_ex(x)
socket.gaierror: [Errno -2] Name or service not known
Googling that error doesn't seem to help me...
The text file only contains one line at the moment (test.com) but i get the same error even with multiple lines/different hosts.
Any suggestions?
Thanks!

import socket
with open("test.txt", "r") as ins:
for line in ins:
print socket.gethostbyname(line.strip())

I wrote a parser for a calendar, but can not read it

import codecs
fileIN ="calendar.ics "
fileOUT="newCalendar.ics "
fd =codecs.open(fileIN,"r ","utf −8")
line=fd.readline()
head=""
#set the headers of the file in var head
while(line != "BEGIN:VEVENT\ r \n") :
head+=line
line=fd.readline()
#set all the events in a list of events
while (line!="END:VCALENDAR\ r \n") :
newEvent=line
line=fd.readline()
while(line != "END:VEVENT\ r \n") :
newEvent+=line
line=fd.readline()
newEvent+=line
line=fd.readline()
# last line
tail=line
fd.close()
#print calendar with only first event on newCalendarFile
file =codecs.open(fileOUT ,"w","utf −8")
file.write(head)
file.write(newEvent)
file.write( tail )
file.close()
I created a parser to select my programation cours and my practical course from all the groups!
But I have a problem to read the script, there is this message error:
Traceback (most recent call last):
File "C:\Users\\Desktop\projet3.py", line 5, in <module>
fd =codecs.open(fileIN,"r ","utf −8")
File "C:\Users\\AppData\Local\Programs\Python\Python35-32\lib\codecs.py", line 895, in open
file = builtins.open(filename, mode, buffering)
ValueError: invalid mode: 'r b'