We Keep Coding

How do I use python or jq to merge multiple JSON files with uniform columns into one CSV?

I am trying to make Primary Source American Law more American with Disabilities Act (ADA), more compliant for my Study.
The technical question is in the topic.
The source files I am using are found here: https://www.courtlistener.com/api/bulk-data/opinions/wash.tar.gz
A sample JSON file within that archive (which contains 52,565 Washington State Supreme Court Judicial Opinions),
987095.json is an example with the following "elements" (I want to carry the element names over as column names while providing a new id for record count; rather than source database id column):
Using Dadroit Viewer 1.5 Build 1935.appimage to view the JSON data easiest for me.
I extracted the following data (the bottom opinions_cited has [] brackets with 0 - 28 in them; and I expect each JSON to have different numbers of opinions_cited ; anyone from 0 to 1000 or more perhaps); I don't know the most cases cited in a Judicial Opinion.
Extracted from JSON:
resource_url
id
absolute_url
cluster
author
joined_by
author_str
per_curiam
date_created
date_modified
type
sha1
page_count
download_url
local_path
plain_text
html
html_lawbox
html_columbia
html_with_citations
extracted_by_ocr
opinions_cited (sub nodes) [0] - [28]
I need assistance in merging the directory (I tried various solutions found on google and stackoverflow for merging; none that I put against my cpu, memory or time has proven to work without taking for hours to error or produce nothing in the end).
How to I create 1 large CSV for every folder of JSONs (containing tens of thousands; in this example 54k) 1 JSON for every row in the CSV using JSON element names to create column's.
The StackOverFlow script I was using (code can be seen here to the error I posted in comments):
import glob
import json
file_names = glob.glob('*.json')
json_list = []
for curr_f_name in file_names:
with open(curr_f_name) as curr_f_obj:
json_list.append(json.load(curr_f_obj))
with open('json_merge_out.json', 'w') as out_file:
json.dump(json_list, out_file, indent=4)
The above attempt remedy outputs the following error:
/Wash/json# ./json.merge.python.1.py
./json.merge.python.1.py: line 4: syntax error near unexpected token `('
./json.merge.python.1.py: line 4: `file_names = glob.glob('*.json')'

Here is a fast solution using jq. It requires hardly any memory, but there is general caveat and one potential complication.
The caveat is that it has been assumed that the "joined_by" array can be flattened into a string using "; " as a separator.
The complication is that some CSV parsers require a "rectangular" structure. In the first potential solution below, the header that is generated will only take into account the length of the
"opinions_cited" array in the first file that is presented. If later JSON files contain more
"opinions_cited", the rows will still be correct, but the header will not take them into account.
If the header is an issue, then you might wish to consider a two-pass solution to avoid memory issues. As shown in Part 2 of this answer, the first pass would be responsible for
determining the maximum array length,
and in the second pass, this value would be used to determine the appropriate headers.
You might also wish to consider a post-processing "fixup".
For testing, I'd suggest using a truncated version of topKeys, e.g.
def topKeys: {resource_url, opinions_cited};
Part 1 - a one-pass approach
The following has been tested with this invocation of jq in the directory in which all the .json files reside:
jq -Mnrf combine.jq *.json > wash.csv
combine.jq
def topKeys: {
resource_url,
id,
absolute_url,
cluster,
author,
joined_by,
author_str,
per_curiam,
date_created,
date_modified,
type,
sha1,
page_count,
download_url,
local_path,
plain_text,
html,
html_lawbox,
html_columbia,
html_with_citations,
extracted_by_ocr
}
| .joined_by |= (if type == "array" then join("; ") else . end)
;
def header:
(topKeys|keys_unsorted) + [range(1; 1 + (.opinions_cited | length)) ];
def row:
[topKeys[]] + .opinions_cited;
# s should be a stream of arrays
# For each item in the stream, a counter (starting at 1) is inserted before the other items
def insert_counter(s): foreach s as $x (0; .+1; [.] + $x);
# Read the first file for the headers
input
| (["record"] + header),
insert_counter( (., inputs) | row )
| #csv
Part 2 - a two-pass solution
(a) Change the def of header above to:
def header:
(topKeys|keys_unsorted) + [range(1; 1 + $n) ];
(b) Use or adapt the following script:
#!/bin/bash
n=$(jq 'def max(s): reduce s as $x (null; if . == null or $x > . then $x else . end); max(.opinions_cited|length)' *.json )
jq -Mnr --argjson n $n -f combine.jq *.json > wash.csv
For the record...
There are many post-processing alternatives. One is to use csvq, e.g.:
csvq --allow-uneven-fields -f CSV 'select *' < wash.csv | sponge wash.csv

How to modify this script to search for multiple keywords?

I am trying to modify a script. It is being difficult for me so I came for help. This script is supposed to extract data from some .out files and then write them in a .txtfile. The problem is that I have two different keywords to look for. So, I provide the script, the things I am not able to modify and then two examples of input files.
#!/usr/bin/env python
# -*- coding: utf-8
#~ Data analysis
import glob, subprocess, shutil, os, math
from funciones import *
for namefile in glob.glob("*.mol2"):
lstmol2 = []
lstG=[]
os.chdir("some_directory")
searchprocess="grep -i -H 'CURRENT VALUE OF HEAT OF FORMATION =' *.out | sort -k 4 > firstfile.txt"
#~I need also to look for 'CURRENT BEST VALUE OF HEAT OF FORMATION ='
os.system(searchprocess)
fileout=open("results.txt","w")
filein=open("firstfile.txt", "r")
#~ write data in results.txt
fileout.write('\t %s \n' %(" HOF"))
for line in filein:
linediv=line.split()
HOF=float(linediv[8])
#~or [10] (for the keyword in need to add) but in both cases I need the float. I need both data of the keywords be included on this file.
lstG.append(HOF)
fileout.close()
filein.close()
Input data, type 1:
foofoofooofoofoofoofoofoo
foofoofooofoofoofoofoofoov
foofoofooofoofoofoofoofoo
CURRENT VALUE OF HEAT OF FORMATION = 1928
foofoofooofoofoofoofoofoo
foofoofooofoofoofoofoofoov
Input data, type 2:
foofoofooofoofoofoofoofoo
foofoofooofoofoofoofoofoov
foofoofooofoofoofoofoofoo
CURRENT BEST VALUE OF HEAT OF FORMATION = 1930
foofoofooofoofoofoofoofoo
foofoofooofoofoofoofoofoov

You should update your grep command to look for the optional word with the ? operator. Use the -E flag to enable exteneded regular expressions so you don't have to escape your regex operators. Always use single quotes around your pattern:
searchprocess="grep -E -i -H 'CURRENT( BEST)? VALUE OF HEAT OF FORMATION =' *.out | sort -k 4 > firstfile.txt"
#PrestonHager is correct that you should change linediv[8] to linediv[-1], since in the cases where BEST is present, the number will be in the linediv[9] position, but in both cases linediv[-1] will give you the desired result.

Using Makefile bash to save the contents of a python file

For those who are curious as to why I'm doing this: I need specific files in a tar ball - no more, no less. I have to write unit tests for make check, but since I'm constrained to having "no more" files, I have to write the check within the make check. In this way, I have to write bash(but I don't want to).
I dislike using bash for unit testing(sorry to all those who like bash. I just dislike it so much that I would rather go with an extremely hacky approach than to write many lines of bash code), so I wrote a python file. I later learned that I have to use bash because of some unknown strict rule. I figured that there was a way to cache the entire content of the python file into a single string in the bash file, so I could take the string literal in bash and write to a python file and then execute it.
I tried the following attempt (in the following script and result, I used another python file that's not unit_test.py, so don't worry if it doesn't actually look like a unit test):
toStr.py:
import re
with open("unit_test.py", 'r+') as f:
s = f.read()
s = s.replace("\n", "\\n")
print(s)
And then I piped the results out using:
python toStr.py > temp.txt
It looked something like:
#!/usr/bin/env python\n\nimport os\nimport sys\n\n#create number of bytes as specified in the args:\nif len(sys.argv) != 3:\n print("We need a correct number of args : 2 [NUM_BYTES][FILE_NAME].")\n exit(1)\nn = -1\ntry:\n n = int(sys.argv[1])\nexcept:\n print("Error casting number : " + sys.argv[1])\n exit(1)\n\nrand_string = os.urandom(n)\n\nwith open(sys.argv[2], 'wb+') as f:\n f.write(rand_string)\n f.flush()\n f.close()\n\n
I tried taking this as a string literal and echoing it into a new file and see whether I could run it as a python file but it failed.
echo '{insert that giant string above here}' > new_unit_test.py
I wanted to take this statement above and copy it into my "bash unit test" file so I can just execute the python file within the bash script.
The resulting file looked exactly like {insert giant string here}. What am I doing wrong in my attempt? Are there other, much easier ways where I can hold a python file as a string literal in a bash script?

the easiest way is to only use double-quotes in your python code, then, in your bash script, wrap all of your python code in one pair of single-quotes, e.g.,
#!/bin/bash
python -c 'import os
import sys
#create number of bytes as specified in the args:
if len(sys.argv) != 3:
print("We need a correct number of args : 2 [NUM_BYTES][FILE_NAME].")
exit(1)
n = -1
try:
n = int(sys.argv[1])
except:
print("Error casting number : " + sys.argv[1])
exit(1)
rand_string = os.urandom(n)
# i changed ""s to ''s below -webb
with open(sys.argv[2], "wb+") as f:
f.write(rand_string)
f.flush()
f.close()'

Using Python to insert multiple rows into a Hive table

Hive is a data warehouse designed for querying and aggregating large datasets that reside on HDFS.
The standard INSERT INTO syntax performs poorly because:
Each statement required a Map/Reduce process to be executed.
Each statement will result in a new file being added to HDFS - over time this will lead to very poor performance when reading from the table.
With that said, there is now a Streaming API for Hive / HCatalog, as detailed here.
I am faced with the need to insert data at velocity into Hive, using Python. I am aware of the pyhive and pyhs2 libraries, but neither of them appears to make use of the Streaming API.
Has anyone successfully managed to get Python to insert many rows into Hive using the Streaming API, and how was this done?
I look forward to your insights!

Hive user can stream table through script to transform that data:
ADD FILE replace-nan-with-zeros.py;
SELECT
TRANSFORM (...)
USING 'python replace-nan-with-zeros.py'
AS (...)
FROM some_table;
Here a simple Python script:
#!/usr/bin/env python
import sys
kFirstColumns= 7
def main(argv):
for line in sys.stdin:
line = line.strip();
inputs = line.split('\t')
# replace NaNs with zeros
outputs = [ ]
columnIndex = 1;
for value in inputs:
newValue = value
if columnIndex > kFirstColumns:
newValue = value.replace('NaN','0.0')
outputs.append(newValue)
columnIndex = columnIndex + 1
print '\t'.join(outputs)
if __name__ == "__main__":
main(sys.argv[1:])
Hive and Python
Python can be used as a UDF from Hive through the HiveQL TRANSFORM statement. For example, the following HiveQL invokes a Python script stored in the streaming.py file.
Linux-based HDInsight
add file wasb:///streaming.py;
SELECT TRANSFORM (clientid, devicemake, devicemodel)
USING 'streaming.py' AS
(clientid string, phoneLable string, phoneHash string)
FROM hivesampletable
ORDER BY clientid LIMIT 50;
Windows Based HDInsight
add file wasb:///streaming.py;
SELECT TRANSFORM (clientid, devicemake, devicemodel)
USING 'D:\Python27\python.exe streaming.py' AS
(clientid string, phoneLable string, phoneHash string)
FROM hivesampletable
ORDER BY clientid LIMIT 50;
Here's what this example does:
1.The add file statement at the beginning of the file adds the streaming.py file to the distributed cache, so it's accessible by all nodes in the cluster.
2.The SELECT TRANSFORM ... USING statement selects data from the hivesampletable, and passes clientid, devicemake, and devicemodel to the streaming.py script.
3.The AS clause describes the fields returned from streaming.py
Here's the streaming.py file used by the HiveQL example.
#!/usr/bin/env python
import sys
import string
import hashlib
while True:
line = sys.stdin.readline()
if not line:
break
line = string.strip(line, "\n ")
clientid, devicemake, devicemodel = string.split(line, "\t")
phone_label = devicemake + ' ' + devicemodel
print "\t".join([clientid, phone_label, hashlib.md5(phone_label).hexdigest()])
Since we are using streaming, this script has to do the following:
1.Read data from STDIN. This is accomplished by using sys.stdin.readline() in this example.
2.The trailing newline character is removed using string.strip(line, "\n "), since we just want the text data and not the end of line indicator.
3.When doing stream processing, a single line contains all the values with a tab character between each value. So string.split(line, "\t") can be used to split the input at each tab, returning just the fields.
4.When processing is complete, the output must be written to STDOUT as a single line, with a tab between each field. This is accomplished by using print "\t".join([clientid, phone_label, hashlib.md5(phone_label).hexdigest()]).
5.This all occurs within a while loop, that will repeat until no line is read, at which point break exits the loop and the script terminates.
Beyond that, the script just concatenates the input values for devicemake and devicemodel, and calculates a hash of the concatenated value. Pretty simple, but it describes the basics of how any Python script invoked from Hive should function: Loop, read input until there is no more, break each line of input apart at the tabs, process, write a single line of tab delimited output.

Script to compare a string in two different files

I am brand new to stackoverflow and to scripting. I was looking for help to get started in a script, not necessarily looking for someone to write it.
Here's what I have:
File1.csv - contains some information, I am only interested in MAC addresses.
File2.csv - has some different information, but also contains MAC address.
I need a script that parses the MAC addresses from file1.csv and logs a report if any MAC address shows up in file2.csv.
The questions:
Any tips on the language I use, preferably perl, python or bash?
Can anyone suggest some structure for the logic needed (even if just in psuedo-code)?
update
Using #Adam Wagner's approach, I am really close!
import csv
#Need to strip out NUL values from .csv file to make python happy
class FilteredFile(file):
def next(self):
return file.next(self).replace('\x00','').replace('\xff\xfe','')
reader = csv.reader(FilteredFile('wifi_clients.csv', 'rb'), delimiter=',', quotechar='|')
s1 = set(rec[0] for rec in reader)
inventory = csv.reader(FilteredFile('inventory.csv','rb'),delimiter=',')
s2 = set(rec[6] for rec in inventory)
shared_items = s1.intersection(s2)
print shared_items
This always outputs:(even if I doctor the .csv files to have matching MAC addresses)
set([])
Contents of the csv files
wifi_clients.csv
macNames, First time seen, Last time seen,Power, # packets, BSSID, Probed ESSIDs
inventory.csv
Name,Manufacturer,Device Type,Model,Serial Number,IP Address,MAC Address,...

Here's the approach I'd take:
Iterate over each csv file (python has a handy csv module for accomplishing this), capturing the mac-address and placing it in a set (one per file). And once again, python has a great builtin set type. Here's a good example of using the csv module and of-course, the docs.
Next, you can get the intersection of set1 (file1) and set2 (file2). This will show you mac-addresses that exist in both files one and two.
Example (in python):
s1 = set([1,2,3]) # You can add things incrementally with "s1.add(value)"
s2 = set([2,3,4])
shared_items = s1.intersection(s2)
print shared_items
Which outputs:
set([2, 3])
Logging these shared items could be done with anything from printing (then redirecting output to a file), to using the logging module, to saving directly to a file.
I'm not sure how in-depth of an answer you were looking for, but this should get you started.
Update: CSV/Set usage example
Assuming you have a file "foo.csv", that looks something like this:
bob,123,127.0.0.1,mac-address-1
fred,124,127.0.0.1,mac-address-2
The simplest way to build the set, would be something like this:
import csv
set1 = set()
for record in csv.reader(open('foo.csv', 'rb')):
user, machine_id, ip_address, mac_address = record
set1.add(mac_address)
# or simply "set1.add(record[3])", if you don't need the other fields.
Obviously, you'd need something like this for each file, so you may want to put this in a function to make life easier.
Finally, if you want to go the less-verbose-but-cooler-python-way, you could also build the set like this:
csvfile = csv.reader(open('foo.csv', 'rb'))
set1 = set(rec[3] for rec in csvfile) # Assuming mac-address is the 4th column.

I strongly recommend python to do this.
'Cause you didn't give the structure of the csv file, I can only show a framework:
def get_MAC_from_file1():
... parse the file to get MAC
return a_MAC_list
def get_MAC_from_file2():
... parse the file to get MAC
return a_MAC_list
def log_MACs():
MAC_list1, MAC_list2 = get_MAC_from_file1(), get_MAC_from_file2()
for a_MAC in MAC_list1:
if a_MAC in MAC_list2:
...write your logs
if the data set is large, use a dict or set instead of the list and the intersect operation. But as it's MAC address, I guess your dataset is not that large. So keeping the script easy to read is the most important thing.

Awk is perfect for this
{
mac = $1 # assuming the mac addresses are in the first column
do_grep = "grep " mac " otherfilename" # we'll use grep to check if the mac address is in the other file
do_grep | getline mac_in_other_file # pipe the output of the grep command into a new variable
close(do_grep) # close the pipe
if(mac_in_other_file != ""){ # if grep found the mac address in the other file
print mac > "naughty_macs.log" # append the mac address to the log file
}
}
Then you'd run that on the first file:
awk -f logging_script.awk mac_list.txt
(this code is untested and I'm not the greatest awk hacker, but it should give the general idea)

For the example purpose generate 2 files that that look like yours.
File1:
for i in `seq 100`; do
echo -e "user$i\tmachine$i\t192.168.0.$i\tmac$i";
done > file1.csv
File2 (contains random entries of "mac addresses" numbered from 1-200)
for j in `seq 100`; do
i=$(($RANDOM % 200)) ;
echo -e "mac$i\tmachine$i\tuser$i";
done > file2.csv
Simplest approach would be to use join command and do a join on the appropriate field. This approach has the advantage that fields from both files would be available in the output.
Based on the example files above, the command would look like this:
join -1 4 -2 1 <(sort -k4 file1.csv) <(sort -k1 file2.csv)
join needs the input to be sorted by the field you are matching, that's why the sort is there (-k tells which column to use)
The command above matches rows from file1.csv with rows from file2.csv if column 4 in the first file is equal with column 1 from the second file.
If you only need specific fields, you can specify the output format to the join command:
join -1 4 -2 1 -o1.4 1.2 <(sort -k4 file1.csv) <(sort -k1 file2.csv)
This would print only the mac address and the machine field from the first file.
If you only need a list of matching mac addresses, you can use uniq or sort -u. Since the join output will be sorted by mac, uniq is faster. But if you need a unique list of another field, sort -u is better.
If you only need the mac addresses that match, grep can accept patterns from a file, and you can use cut to extract only the forth field.
fgrep -f<(cut -f4 file1.csv) file2.csv
The above would list all the lines in file2.csv that contain a mac address from file1
Note that I'm using fgrep which doesn't do pattern matching. Also, if file1 is big, this may be slower than the first approach. Also, it assumes that the mac is present only in the field1 of file2 and the other fields don't contain mac addresses.
If you only need the mac, you can either use -o option on fgrep but there are grep variants that don't have it, or you can pipe the output trough cut and then sort -u
fgrep -f<(cut -f4 file1.csv) file2.csv | cut -f1 | sort -u
This would be the bash way.
Python and awk hints have been shown above, I will take a stab at perl:
#!/usr/bin/perl -w
use strict;
open F1, $ARGV[0];
my %searched_mac_addresses = map {chomp; (split /\t/)[3] => 1 } <F1>;
close F1;
open F2, $ARGV[1];
while (<F2>) {
print if $searched_mac_addresses{(split "\t")[0]}
}
close F2
First you create a dictionary containing all the mac addresses from the first file:
my %searched_mac_addresses = map {chomp; (split /\t/)[3] => 1 } <F1>;
reads all the lines from the file1
chomp removes the end of line
split splits the line based on tab, you can use a more complex regexp if needed
() around split force an array context
[3] selects the forth field
map runs a piece of code for all elements of the array
=> generates a dictionary (hash in perl's terminology) element instead of an array
Then you read line by line the second file, and check if the mac exists in the above dictionary:
while (<F2>) {
print if $searched_mac_addresses{(split "\t")[0]}
}
while () will read the file F2, and put each line in the $_ variable
print without any parameters prints the default variable $_
if can postfix a instruction
dictionary elements can be accessed via {}
split by default splits the $_ default variable