python iterating over JSON object and writing to csv - python

I am trying to iterate over a JSON object, and write to a new CSV file.
Anyhow I am getting an error when trying this code:
def flat_attr(thisAttr):
if type(thisAttr) is bytes:
thisAttr = (thisAttr.decode('utf-8'))[:1500]
else:
try:
thisAttr = str(thisAttr)[:1500]
except:
thsAttr = thisAttr
return thisAttr
thisDate = (datetime.today().date())
thisFile = 'sim_' + thisDate.strftime('%Y%m%d') + '.csv'
with open('/tmp/' + thisFile, 'w') as csvfile:
writer = csv.DictWriter(csvfile, fieldnames = ['sim_id', 'data'], delimiter = '\t', lineterminator = '\n')
counter = 0
for issue in results.issues:
counter += 1
print('Writer written line ' + str(counter) + ' issue_id: ' + issue.main_id)
print('Writer written line ' + str(counter) + ' issue_id: ' + issue.labels)
writer.writerow({
'sim_id': issue.main_id,
'data': json.dumps({
for a in dir(issue):
if a in attr_list:
a: flat_attr(getattr(issue, a))
print(a)
})
})
The Error is this one:
E for a in dir(issue):
E ^
E SyntaxError: invalid syntax
When I change that writerow() for loop to the following code, it works:
writer.writerow({
'sim_id': issue.main_id,
'data': json.dumps({
a: flat_attr(getattr(issue, a)) for a in dir(issue) if a in attr_list
})
})
I want to debut, that's why I am trying to print 'a'.
How come the loop works, when the for loop and if-clause are after the a: flat_attr(getattr(issue, a)) and it doesn't when the for and if are before that line? How can I print 'a' to debug the code?
Thanks!

If you want to debug what data has been passing in a particular line then you go for an IDE like Pycharm. Using Pycharm you can keep breakpoints and there will be option for debug an application at run time and now you can easily debug your program.
Just try..

Ok, do one thing ZeleIB, append the value of 'a' to a list and return the list for a testing purpose.
Example,
for a in dir(issue):
debug_a = []
if a in attr_list:
a: flat_attr(getattr(issue, a))
debug_a.append(a)
return {'test': debug_a}

Related

Printing Values to CSV (Python)

I am trying to send the failed values to a CSV file but it's only giving me the last failed value in the list.
print(("Folder\t"+ "Expected\t"+ "Actual\t"+"Result").expandtabs(20))
for key in expected:
expectedCount = str(expected[key])
actualCount = "0"
if key in newDictionary:
actualCount = str(newDictionary[key])
elif expectedCount == actualCount:
result = "Pass"
else:
result = "Fail"
with open('XML Output.csv', 'w',encoding='utf-8', newline="") as csvfile:
header = ['Folder', 'Expected', 'Actual','Result']
my_writer = csv.writer(csvfile)
my_writer.writerow(header)
my_writer.writerow([key, expectedCount, actualCount, result])
csvfile.close()
print((key + "\t"+expectedCount+ "\t"+actualCount+ "\t"+result).expandtabs(20))
print("======================== Data Exported to CSV file ========================")
Output:
Folder Expected Actual Result
D 2 1 Fail
Here is what the output should be:
Folder Expected Actual Result
A 2 1 Fail
B 2 1 Fail
C 2 1 Fail
D 2 1 Fail
This is because each iteration of with open using w is overwriting the file, leaving only the last iteration at the end of it. You could use a for append.
A better method may be to create a data structure to hold the failures and write to the file simultaneously. Give the below a try. I couldn't test it without initial data, but I think you'll get what I was going for.
print(("Folder\t"+ "Expected\t"+ "Actual\t"+"Result").expandtabs(20))
failures = []
for key in expected:
expectedCount = str(expected[key])
actualCount = "0"
if key in newDictionary:
actualCount = str(newDictionary[key])
elif expectedCount == actualCount:
result = "Pass"
else:
result = "Fail"
csv_row = {
"Folder":key,
"Expected":expectedCount,
"Actual":actualCount,
"Result":"Fail"
}
failures.append(csv_row)
print((key + "\t"+expectedCount+ "\t"+actualCount+ "\t"+result).expandtabs(20))
try:
with open('XML Output.csv', 'w') as csvfile:
writer = csv.DictWriter(csvfile, fieldnames=failures[0].keys())
writer.writeheader()
for data in failures:
writer.writerow(data)
except IOError:
print('I/O Error on CSV export')
print("======================== Data Exported to CSV file ========================")
Edit:
Wanted to add a note that if you want to use dictionaries to write to CSV, DictWriter is an apt choice for this.
https://docs.python.org/3/library/csv.html#csv.DictWriter
you're recreating the csv file upon every iteration that hits the else branch. You need to move the with statement out of your loop:
import csv
expected = {"a": 1, "b": 2}
newDictionary = {"a": 2, "c": 2}
with open('XML Output2.csv', 'w', encoding='utf-8', newline="") as csvfile:
header = ['Folder', 'Expected', 'Actual', 'Result']
my_writer = csv.writer(csvfile)
my_writer.writerow(header)
for key in expected:
expectedCount = str(expected[key])
actualCount = "0"
if key in newDictionary:
actualCount = str(newDictionary[key])
if expectedCount == actualCount:
result = "Pass"
else:
result = "Fail"
my_writer.writerow([key, expectedCount, actualCount, result])
print((key + "\t" + expectedCount + "\t" + actualCount + "\t" + result).expandtabs(20))
print("======================== Data Exported to CSV file ========================")
Also, note that you do not need to invoke close() explicitly on a file that was created using a context manager (with). See this article for more on the matter: https://realpython.com/python-with-statement/

Python replace str in list with new value

I’m writing a program that makes music albums into files that you can search for, and for that i need a str in the file that have a specific value that is made after the list is complete. Can you go back in that list and change a blank str with a new value?
I have searched online and found something called words.replace, but it doesn’t work, i get a Attribute error.
def create_album():
global idnumber, current_information
file_information = []
if current_information[0] != 'N/A':
save()
file_information.append(idnumber)
idnumber += 1
print('Type c at any point to abort creation')
for i in creation_list:
value = input('\t' + i)
if value.upper == 'C':
menu()
else:
-1file_information.append('')
file_information.append(value)
file_information.append('Album created - ' + file_information[2] +'\nSongs:')
-2file_information = [w.replace(file_information[1], str(file_information[0]) + '-' + file_information[2]) for w in file_information]
current_information = file_information
save_name = open(save_path + str(file_information[0]) + '-' + str(file_information[2]) + '.txt', 'w')
for i in file_information:
save_name.write(str(i) + '\n')
current_files_ = open(information_file + 'files.txt', 'w')
filenames.append(file_information[0])
for i in filenames:
current_files_.write(str(i) + '\n')
id_file = open(information_file + 'albumid.txt', 'w')
id_file.write(str(idnumber))
-1 is where i have put aside a blank row
-2 is the where i try to replace row 1 in the list with the value of row 0 and row 2.
The error message I receive is ‘int’ object has no attribute ‘replace’
Did you try this?
-2file_information = [w.replace(str(file_information[1]), str(file_information[0]) + '-' + file_information[2]) for w in file_information]

concatinating multiple strings from dictionary and save in file using python

I able to write hostname in the /tmp/filter.log but any hint how can i write all three values[hostname, owner, seats] in the file?
def list_hosts(nc):
resp = nc.send_service_request('ListHosts', json.dumps({}))
result = resp['result']
l = []
f=open("/tmp/filter.log", "w+")
for r in result:
if "team-prod" in r['owner']:
print r['owner'], r['hostname'], r['seats']
f.write(r['hostname'] + "\n")
f.close()
l.append(r['hostname'])
return l
nc = create_client('zone', 'team_PROD_USERNAME', 'team_PROD_PASSWORD')
l = list_hosts(nc)
print l
The file should have entries as below:
team-prod\*, np-team-052, [u'123123123-18d1-483d-9af8-169ac66b26e4']
Current entry is:
np-team-052
f.write(str(r['owner']) + ', ' + str(r['hostname']) + ', ' + str(r['seats']) + '\n')

TypeError: cannot concatenate 'str' and 'NoneType' objects?

I have this large script ( I will post the whole thing if I have to but it is very big) which starts off okay when I run it but it immediatly gives me 'TypeError: cannot concatenate 'str' and 'NoneType' objects' when it comes to this last bit of the code:
with open("self.txt", "a+") as f:
f = open("self.txt", "a+")
text = f.readlines()
text_model = markovify.Text(text)
for i in range(1):
tool = grammar_check.LanguageTool('en-GB')
lin = (text_model.make_sentence(tries=800))
word = ('' + lin)
matches = tool.check (word)
correct = grammar_check.correct (word, matches)
print ">",
print correct
print ' '
f = open("self.txt", "a+")
f.write(correct + "\n")
I have searched everywhere but gotten nowhere. It seems to have something to do with: word = ('' + lin). but no matter what I do I can't fix it. What am I doing wrong?
I'm not sure how I did it but with a bit of fiddling and google I came up with a solution, the corrected code is here (if you're interested):
with open("self.txt", "a+") as f:
f = open("self.txt", "a+")
text = f.readlines()
text_model = markovify.Text(text)
for i in range(1):
tool = grammar_check.LanguageTool ('en-GB')
lin = (text_model.make_sentence(tries=200))
matches = tool.check (lin)
correct = grammar_check.correct (lin, matches)
lowcor = (correct.lower())
print ">",
print str (lowcor)
print ' '
f = open("self.txt", "a+")
f.write(lowcor + "\n")
Thanks for all the replies, they had me thinking and that's how I fixed it!
You can't concatenate a string and a NoneType object. In your code, it appears your variable lin is not getting assigned the value you think it is. You might try an if block that starts like this:
if type(lin) == str:
some code
else:
raise Exception('lin is not the correct datatype')
to verify that lin is the correct datatype before printing.

Python: File Writing Adding Unintentional Newlines on Linux Only

I am using Python 2.7.9. I'm working on a program that is supposed to produce the following output in a .csv file per loop:
URL,number
Here's the main loop of the code I'm using:
csvlist = open(listfile,'w')
f = open(list, "r")
def hasQuality(item):
for quality in qualities:
if quality in item:
return True
return False
for line in f:
line = line.split('\n')
line = line[0]
# print line
itemname = urllib.unquote(line).decode('utf8')
# print itemhash
if hasQuality(itemname):
try:
looptime = time.time()
url = baseUrl + line
results = json.loads(urlopen(url).read())
# status = results.status_code
content = results
if 'median_price' in content:
medianstr = str(content['median_price']).replace('$','')
medianstr = medianstr.replace('.','')
median = float(medianstr)
volume = content['volume']
print url+'\n'+itemname
print 'Median: $'+medianstr
print 'Volume: '+str(volume)
if (median > minprice) and (volume > minvol):
csvlist.write(line + ',' + medianstr + '\n')
print '+ADDED TO LIST'
else:
print 'No median price given for '+itemname+'.\nGiving up on item.'
print "Finished loop in " + str(round(time.time() - looptime,3)) + " seconds."
except ValueError:
print "we blacklisted fool?? cause we skippin beats"
else:
print itemname+'is a commodity.\nGiving up on item.'
csvlist.close()
f.close()
print "Finished script in " + str(round(time.time() - runtime, 3)) + " seconds."
It should be generating a list that looks like this:
AWP%20%7C%20Asiimov%20%28Field-Tested%29,3911
M4A1-S%20%7C%20Hyper%20Beast%20%28Field-Tested%29,4202
But it's actually generating a list that looks like this:
AWP%20%7C%20Asiimov%20%28Field-Tested%29
,3911
M4A1-S%20%7C%20Hyper%20Beast%20%28Field-Tested%29
,4202
Whenever it is ran on a Windows machine, I have no issue. Whenever I run it on my EC2 instance, however, it adds that extra newline. Any ideas why? Running commands on the file like
awk 'NR%2{printf $0" ";next;}1' output.csv
do not do anything. I have transferred it to my Windows machine and it still reads the same. However, when I paste the output into Steam's chat client it concatenates it in the way that I want.
Thanks in advance!
This is where the problem occurs
code:
csvlist.write(line + ',' + medianstr + '\n')
This can be cleared is you strip the space
modified code:
csvlist.write(line.strip() + ',' + medianstr + '\n')
Problem:
The problem is due to the fact you are reading raw lines from the input file
Raw_lines contain \n to indicate there is a new line for every line which is not the last and for the last line it just ends with the given character .
for more details:
Just type print(repr(line)) before writing and see the output

Categories