Creating a script that will eventually allow users to enter a hardware/asset id and query an API to get back a bunch of information, currently in the process of simply printing out the entire JSON file and printing it, however it currently prints it all to one line and makes it near impossible to read, ideally it looks like the following:
name: x
owner: x
team: x
and so on and so forth however it prints
name: x owner: x team: x
Code is as follows:
import json
from json import dumps, loads
def json_format():
json.dumps(skipkeys=False, ensure_ascii=True, check_circular=True, allow_nan=True, cls=None, indent=None, separators=None, default=None, sort_keys=False)
def main():
hardware_id = "x"
asset_id = "x"
host = get_information(asset_id='x')
json_object = dumps(host)
print(dumps(json_object, indent=4))
Tried following some pretty print guides however the above didn't fix the issue I'm facing, any ideas thank you! :D
I am trying to test this demo program from lynda using Python 3. I am using Pycharm as my IDE. I already added and installed the request package, but when I run the program, it runs cleanly and shows a message "Process finished with exit code 0", but does not show any output from print statement. Where am I going wrong ?
import urllib.request # instead of urllib2 like in Python 2.7
import json
def printResults(data):
# Use the json module to load the string data into a dictionary
theJSON = json.loads(data)
# now we can access the contents of the JSON like any other Python object
if "title" in theJSON["metadata"]:
print(theJSON["metadata"]["title"])
# output the number of events, plus the magnitude and each event name
count = theJSON["metadata"]["count"];
print(str(count) + " events recorded")
# for each event, print the place where it occurred
for i in theJSON["features"]:
print(i["properties"]["place"])
# print the events that only have a magnitude greater than 4
for i in theJSON["features"]:
if i["properties"]["mag"] >= 4.0:
print("%2.1f" % i["properties"]["mag"], i["properties"]["place"])
# print only the events where at least 1 person reported feeling something
print("Events that were felt:")
for i in theJSON["features"]:
feltReports = i["properties"]["felt"]
if feltReports != None:
if feltReports > 0:
print("%2.1f" % i["properties"]["mag"], i["properties"]["place"], " reported " + str(feltReports) + " times")
# Open the URL and read the data
urlData = "http://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/2.5_day.geojson"
webUrl = urllib.request.urlopen(urlData)
print(webUrl.getcode())
if webUrl.getcode() == 200:
data = webUrl.read()
data = data.decode("utf-8") # in Python 3.x we need to explicitly decode the response to a string
# print out our customized results
printResults(data)
else:
print("Received an error from server, cannot retrieve results " + str(webUrl.getcode()))
Not sure if you left this out on purpose, but this script isn't actually executing any code beyond the imports and function definition. Assuming you didn't leave it out on purpose, you would need the following at the end of your file.
if __name__ == '__main__':
data = "" # your data
printResults(data)
The check on __name__ equaling "__main__" is just so your code is only executing when the file is explicitly run. To always run your printResults(data) function when the file is accessed (like, say, if its imported into another module) you could just call it at the bottom of your file like so:
data = "" # your data
printResults(data)
I had to restart the IDE after installing the module. I just realized and tried it now with "Run as Admin". Strangely seems to work now.But not sure if it was a temp error, since even without restart, it was able to detect the module and its methods.
Your comments re: having to restart your IDE makes me think that pycharm might not automatically detect newly installed python packages. This SO answer seems to offer a solution.
SO answer
Overall Goal: I am trying to read some progress data from a python exe to update the progress of the exe in another application
I have a python exe that is going to do some stuff, I want to be able to communicate the progress to another program. Based on several other Q&A here I have been able to have my running application send progress data to a named pipe using the following code
import win32pipe
import win32file
import glob
test_files = glob.glob('J:\\someDirectory\\*.htm')
# test_files has two items a.htm and b.htm
p = win32pipe.CreateNamedPipe(r'\\.\pipe\wfsr_pipe',
win32pipe.PIPE_ACCESS_DUPLEX,
win32pipe.PIPE_TYPE_MESSAGE | win32pipe.PIPE_WAIT,
1,65536,65536,300,None)
# the following line is the server-side function for accepting a connection
# see the following SO question and answer
""" http://stackoverflow.com/questions/1749001/named-pipes-between-c-sharp-and-python
"""
win32pipe.ConnectNamedPipe(p, None)
for each in testFiles:
win32file.WriteFile(p,each + '\n')
#send final message
win32file.WriteFile(p,'Process Complete')
# close the connection
p.close()
In short the example code writes the path of the each file that was globbed to the NamedPipe - this is useful and can be easily extended to more logging type events. However, the problem is trying to figure out how to read the content of the named pipe without knowing the size of each possible message. For example the first file could be named J:\someDirectory\a.htm, but the second could have 300 characters in the name.
So far the code I am using to read the contents of the pipe requires that I specify a buffer size
First establish the connection
file_handle = win32file.CreateFile("\\\\.\\pipe\\wfsr_pipe",
win32file.GENERIC_READ | win32file.GENERIC_WRITE,
0, None,
win32file.OPEN_EXISTING,
0, None)
and then I have been playing around with reading from the file
data = win32file.ReadFile(file_handle,128)
This generally works but I really want to read until I hit a newline character, do something with the content between when I started reading and the newline character and then repeat the process until I get to a line that has Process Complete in the line
I have been struggling with how to read only until I find a newline character (\n). I basically want to read the file by lines and based on the content of the line do something (either display the line or shift the application focus).
Based on the suggestion provided by #meuh I am updating this because I think there is a dearth of examples, guidance in how to use pipes
My server code
import win32pipe
import win32file
import glob
import os
p = win32pipe.CreateNamedPipe(r'\\.\pipe\wfsr_pipe',
win32pipe.PIPE_ACCESS_DUPLEX,
win32pipe.PIPE_TYPE_MESSAGE | win32pipe.PIPE_WAIT,
1,65536,65536,300,None)
# the following line is the server-side function for accepting a connection
# see the following SO question and answer
""" http://stackoverflow.com/questions/1749001/named-pipes-between-c-sharp-and-python
"""
win32pipe.ConnectNamedPipe(p, None)
for file_id in glob.glob('J:\\level1\\level2\\level3\\*'):
for filer_id in glob.glob(file_id + os.sep + '*'):
win32file.WriteFile(p,filer_id)
#send final message
win32file.WriteFile(p,'Process Complete')
# close the connection
p.close() #still not sure if this should be here, I need more testing
# I think the client can close p
The Client code
import win32pipe
import win32file
file_handle = win32file.CreateFile("\\\\.\\pipe\\wfsr_pipe",
win32file.GENERIC_READ |
win32file.GENERIC_WRITE,
0, None,win32file.OPEN_EXISTING,0, None)
# this is the key, setting readmode to MESSAGE
win32pipe.SetNamedPipeHandleState(file_handle,
win32pipe.PIPE_READMODE_MESSAGE, None, None)
# for testing purposes I am just going to write the messages to a file
out_ref = open('e:\\testpipe.txt','w')
dstring = '' # need some way to know that the messages are complete
while dstring != 'Process Complete':
# setting the blocksize at 4096 to make sure it can handle any message I
# might anticipate
data = win32file.ReadFile(file_handle,4096)
# data is a tuple, the first position seems to always be 0 but need to find
# the docs to help understand what determines the value, the second is the
# message
dstring = data[1]
out_ref.write(dstring + '\n')
out_ref.close() # got here so close my testfile
file_handle.close() # close the file_handle
I don't have windows but looking through the api it seems you should convert
your client to message mode by adding after the CreateFile() the call:
win32pipe.SetNamedPipeHandleState(file_handle,
win32pipe.PIPE_READMODE_MESSAGE, None, None)
then each sufficiently long read will return a single message, ie what the other wrote in a single write. You already set PIPE_TYPE_MESSAGE when you created the pipe.
You could simply use an implementation of io.IOBase that would wrap the NamedPipe.
class PipeIO(io.RawIOBase):
def __init__(self, handle):
self.handle = handle
def read(self, n):
if (n == 0): return ""
elif n == -1: return self.readall()
data = win32file.ReadFile(self.file_handle,n)
return data
def readinto(self, b):
data = self.read(len(b))
for i in range(len(data)):
b[i] = data[i]
return len(data)
def readall(self):
data = ""
while True:
chunk = win32file.ReadFile(self.file_handle,10240)
if (len(chunk) == 0): return data
data += chunk
BEWARE : untested, but it should work after fixing the eventual typos.
You could then do:
with PipeIO(file_handle) as fd:
for line in fd:
# process a line
You could use the msvcrt module and open to turn the pipe into a file object.
Sending code
import win32pipe
import os
import msvcrt
from io import open
pipe = win32pipe.CreateNamedPipe(r'\\.\pipe\wfsr_pipe',
win32pipe.PIPE_ACCESS_OUTBOUND,
win32pipe.PIPE_TYPE_MESSAGE | win32pipe.PIPE_WAIT,
1,65536,65536,300,None)
# wait for another process to connect
win32pipe.ConnectNamedPipe(pipe, None)
# get a file descriptor to write to
write_fd = msvcrt.open_osfhandle(pipe, os.O_WRONLY)
with open(write_fd, "w") as writer:
# now we have a file object that we can write to in a standard way
for i in range(0, 10):
# create "a\n" in the first iteration, "bb\n" in the second and so on
text = chr(ord("a") + i) * (i + 1) + "\n"
writer.write(text)
Receiving code
import win32file
import os
import msvcrt
from io import open
handle = win32file.CreateFile(r"\\.\pipe\wfsr_pipe",
win32file.GENERIC_READ,
0, None,
win32file.OPEN_EXISTING,
0, None)
read_fd = msvcrt.open_osfhandle(handle, os.O_RDONLY)
with open(read_fd, "r") as reader:
# now we have a file object with the readlines and other file api methods
lines = reader.readlines()
print(lines)
Some notes.
I've only tested this with python 3.4, but I believe you may be using python 2.x.
Python seems to get weird if you try to close both the file object and the pipe..., so I've only used the file object (by using the with block)
I've only created the file objects to read on one end and write on the other. You can of course make the file objects duplex by
Creating the file descriptors (read_fd and write_fd) with the os.O_RDWR flag
Creating the file objects in in "r+" mode rather than "r" or "w"
Going back to creating the pipe with the win32pipe.PIPE_ACCESS_DUPLEX flag
Going back to creating the file handle object with the win32file.GENERIC_READ | win32file.GENERIC_WRITE flags.
I haave a file and I want to read a specific part of it. This is the file.
.....
.....
Admin server interface begins
...
....
....
....
....
Admin server interface ends
....
....
I want to read the part of file between 'admin server interface begins' till 'admin server interface ends'. I found a way of doing it in perl but can't find a way in python.
in perl
while (<INP>)
{
print $_ if(/^AdminServer interface definitions begins/ .. /^AdminServer interface definitions ends/);
}
Could anyonle please help.
You can read the file line by line and gather what is in between your markers.
def dispatch(inputfile):
# if the separator lines must be included, set to True
need_separator = True
new = False
rec = []
with open(inputfile) as f:
for line in f:
if "Admin server interface begins" in line:
new = True
if need_separator:
rec = [line]
else:
rec = []
elif "Admin server interface ends" in line:
if need_separator:
rec.append(line)
new = False
# if you do not need to process further, uncomment the following line
#return ''.join(rec)
elif new:
rec.append(line)
return ''.join(rec)
The code above will successfully return data even if the input file does not contain the ending separator (Admin server interface ends). You can amend the last return with a condition if you want to catch such files:
if new:
# handle the case where there is no end separator
print("Error in input file: no ending separator")
return ''
else:
return ''.join(rec)
If the file isn't very big and you are not concerned about memory consumption, you can write this simple solution:
from os.path import isfile
def collect_admin_server_interface_info(filename):
""" Collects admin server interface information from specified file. """
if isfile(filename):
contents = ''
with open(filename, 'r') as f:
contents = file.read()
beg_str = 'Admin server interface begins'
end_str = 'Admin server interface ends'
beg_index = contents.find(beg_str + len(beg_str))
end_index = contents.find(end_str)
if beg_index == -1 or end_index == -1:
raise("Admin server interface not found.")
return contents[beg_index : end_index]
else:
raise("File doesn't exist.")
This method will try to return a single string containing administrator server interface information.
Background:
I found this code from Alex Hanna (https://github.com/alexhanna) and I've been trying to adapt it to my situation, but haven't been as successful as I would like, and was hoping for help.
This code takes JSON output from the Twitter API and extracts user mentions from it to create a mention network. Currently, the code is executed as follows:
cat file.json | mentionMapper.py
Desires:
I would like to change it so that the python code reads in the JSON file. Previously I've used the code below, but haven't been able to figure out how to adapt it for this instance.
for line in in_file:
try:
tweet = json.loads(line)
except:
pass
I would also like the code to write the output to a CSV file instead of printing it to the screen or having to manually direct the output using > after the code.
Ultimate Goal:
Edgelist of Twitter users who mention one another.
Code:
import json
import sys
def main():
for line in sys.stdin:
line = line.strip()
data = ''
try:
data = json.loads(line)
except ValueError as detail:
continue
if not (isinstance(data, dict)):
## not a dictionary, skip
pass
elif 'delete' in data:
## a delete element, skip for now.
pass
elif 'user' not in data:
## bizarre userless edge case
pass
else:
if 'entities' in data and len(data['entities']['user_mentions']) > 0:
user = data['user']
user_mentions = data['entities']['user_mentions']
for u2 in user_mentions:
print ",".join([
user['screen_name'],
u2['screen_name'],
"1"
])
if __name__ == '__main__':
main()