How do I use ParaView's CSVReader in a Python Script? An example would be appreciated.
If you have a .csv file that looks like this:
x,y,z,attribute
0,0,0,0
1,0,0,1
0,1,0,2
1,1,0,3
0,0,1,4
1,0,1,5
0,1,1,6
1,1,1,7
then you can import it with a command that looks like this:
myReader = CSVReader(FileName='C:\foo.csv', guiName='foo.csv')
Also, if you don't add that guiName parameter, you can change the name later using the RenameSource command like this:
RenameSource(proxy = myReader, newName = 'MySuperNewName'
Credit for the renaming part of this answer to Sebastien Jourdain.
Unfortunately, I don't know Paraview at all. But I found "... simply record your work in the desktop application in the form of a python script ..." at their site. If you import a CSV like that, it might give you a hint.
Improving the #GregNash's answer. If you want to include only a single file (called foo.csv):
outcsv = CSVReader(FileName= 'foo.csv')
Or if you want to include all files with certain pattern use glob. For example if files start with string foo (aka foo.csv.0, foo.csv.1, foo.csv.2):
myreader = CSVReader(FileName=glob.glob('foo*'))
To use glob is neccesary import glob in the preamble. In general in Filename you could work with strings generated with python which could contain more complex pattern files and file's path.
Related
I did my research on the internet and it seems, that LOAD XML INFILE could not save nested childs with same names or simply with different names.
imported XML sample here
But is there any option, which could be used to keep whole content in parent as plaintext? Its not problem for me after that to parse that content line by line.
Please do not tell me I need to parse it with PHP, it fails in case of speed and I have many XMLs I need to load, so terminal is best solution for me.
So if there is for example some kind of shell or python script (in case that its not possible to import it as plain).
Thanks in advance
Thank you all for correcting grammar mistakes, its very useful and you should earn another badge for helping to community.
Since nobody came up with solution, I did following, which helped me:
1) create file script.py with this contents
#!/usr/bin/python3
# coding: utf-8
import os
import sys
import fileinput
replacements = {'<Image>':'', '</Image>':';',' ':'','\n':''}
with open('/var/www/html/XX/data/xml/products.xml') as infile, open('/var/www/html/XXX/data/xml/products_clean.xml', 'w') as outfile:
for line in infile:
for src, target in replacements.iteritems():
line = line.replace(src, target)
outfile.write(line)
2) run it through terminal
python /var/www/html/script.py
3) then you load XML infile that XML to your mysql as usual, or you can transform that column into json for better use
I'm trying to create a Python Script but I'm a bit stuck and can't find what I'm looking for on a Google search as it's quite specific.
I need to run a script on two .log files (auth.log and access.log) to view the following information:
Find how many attempts were made with the bin account
So how many attempts the bin account made to try and get into the server.
The logs are based off being hacked and needing to identify how and who is responsible.
Would anyone be able to give me some help in how I go about doing this? I can provide more information if needed.
Thanks in advance.
Edit:
I've managed to print all the times 'bin' appears in the log which is one way of doing it. Does anyone know if I can count how many times 'bin' appears as well?
with open("auth.log") as f:
for line in f:
if "bin" in line:
print line
Given that you work with system logs and their format is known and stable, my approach would be something like:
identify a set of keywords (either common, or one per log)
for each log, iterate line by line
once keywords match, add the relevant information from each line in e.g. a dictionary
You could use shell tools (like grep, cut and/or awk) to pre-process the log and extract relevant lines from the log (I assume you only need e.g. error entries).
You can use something like this as a starting point.
If you want ot use tool then you can use ELK(Elastic,Logstash and kibana).
if no then you have to read first log file then apply regex according to your requirment.
In case you might be interested in extracting some data and save it to a .txt file, the following sample code might be helpful:
import re
import sys
import os.path
expDate = '2018-11-27'
expTime = '11-21-09'
infile = r"/home/xenial/Datasets/CIVIT/Nov_27/rover/NMND17420010S_"+expDate+"_"+expTime+".LOG"
keep_phrases = ["FINESTEERING"]
with open(infile) as f:
f = f.readlines()
with open('/home/xenial/Datasets/CIVIT/Nov_27/rover/GPS_'+expDate+'_'+expTime+'.txt', 'w') as file:
file.write("gpsWeek,gpsSOW\n")
for line in f:
for phrase in keep_phrases:
if phrase in line:
resFind = re.findall('\.*?FINESTEERING,(\d+).*?,(\d+\.\d*)',line)[0]
gpsWeek = re.findall('\.*?FINESTEERING,(\d+)',line)[0]
gpsWeekStr = str(gpsWeek)
gpsSOW = re.findall('\.*?FINESTEERING,'+ gpsWeekStr + ',(\d+\.\d*)',line)[0]
gpsSOWStr = str(gpsSOW)
file.write(gpsWeekStr+','+gpsSOWStr+'\n')
break
print ("------------------------------------")
In my case, FINESTEERING was an interesting keyword in my .log file to extract numbers, including GPS_Week and GPS_Seconds_of_Weeks. You may modify this code to suit your own application.
I am trying to edit a .reg file in python to replace strings in a file. I can do this for any other file type such as .txt.
Here is the python code:
with open ("C:/Users/UKa51070/Desktop/regFile.reg", "r") as myfile:
data=myfile.read()
print data
It returns an empty string
I am not sure why you are not seeing any output, perhaps you could try:
print len(data)
Depending on your version of Windows, your REG file will be saved using UTF-16 encoding, unless you specifically export it using the Win9x/NT4 format.
You could try using the following script:
import codecs
with codecs.open("C:/Users/UKa51070/Desktop/regFile.reg", encoding='utf-16') as myfile:
data = myfile.read()
print data
It's probably not a good idea to edit .reg files manually. My suggestion is to search for a Python package that handles it for you. I think the _winreg Python built-in library is what you are looking for.
I have a file with the format
VarName=Value
.
.
I want to read it into a hash such that H("VarName") will return the value.
What would be a quick way? (read a set of strings, split all of them where the equality sign is, and then put it into a hash?
I am working with python.
The oneliner answer:
H = dict(line.strip().split('=') for line in open('filename.txt'))
(optionally use .split() with maxsplit=1 if the values could also contain the "=" character)
Maybe ConfigParser can help you.
d = {}
with open('filename') as f:
for line in f:
key, value = line.split('=')
d[key] = value
Edit:
As suggested by foret, you could change it to
for line in f:
tokens = line.split('=')
d[tokens[0]] = '='.join(tokens[1:])
which would handle the case where equals signs were allowed in the value, but would still fail if the name could have equals signs as well -- for that you would need a true parser.
Taking #Steven's answer doesn't account comments and newlines in the properties file, this one does:
H = dict(line.strip().split('=') for line in open('file.properties') if not line.startswith('#') and not line.startswith('\n'))
Or ConfigObj
The csv module will let you do this easily enough:
import csv
H = dict([(row[0], row[1]) for row in csv.reader(open('the_file', 'r'), delimiter='=' )])
this may be a stupid answer but who know maybe it can help you :)
change the extension of your file to .py, and do necessary change like this:
file.py
VarName="Value" # if it's a string
VarName_2=1
# and you can also assign a dict a list to a var, how cool is that ?
and put it in your package tree or in sys.path, and now you can call it like this in the script when you want to use it:
>>> import file
>>> file.VarName
'Value'
why i'm writing this answer it's because ,what the hell is this file ? i never see a conf file like this , no section no nothing ? why you want to create a config file like this ? it look like a bad config file that should look like the Django settings, and i prefer using a django setting-like config file when ever i can.
Now you can put your -1 in the left :)
For python2 there is a jproperties https://pypi.python.org/pypi/jproperties/1.0.1
For python2/3 there is javaproperties http://javaproperties.readthedocs.io/en/v0.1.0/
as simple as:
import os, javaproperties
with open(file, 'rb') as f:
properties_dict = javaproperties.load(f)
OK nobody else in the answers has mentioned it, so I guess I'm going to. If you're writing Python, and have control over your interpreter, maybe you can force the use of the Jython interpreter.
Jython is a Python interpreter implemented entirely in Java. You have all the Python standard libraries at your fingertips, with the additional advantage of all your Java SE libraries available.
I haven't actually executed any of the following (think of it more like psudeo-code without exception handling), but you can mix and match Python and Java libraries, and your code might end up looking something like:
from java.util import Properties
from java.io import File, FileInputStream
import os
javasPropertyObject = Properties()
pathToPropFile = os.path.join('path', 'to', 'property', 'file.properties')
if os.path.isfile(pathToPropFile):
#this is java.io.File, not Python's file descriptor
propFile = File(pathToPropFile )
javasFileInputStreamObject = FileInputStream(propFile)
javasPropertyObject.load(javasFileInputStreamObject)
#now we can pull out Java properties as defined by the .property file grammar
myProp = javasPropertyObject.getProperty('myPropName')
where a file like this will be valid, that wouldn't in the simple split on '=' solutions:
myPropName1:value
myPropName2=value
myPropName3=\
value
#this is a = comment
myPropName4:my \
value
myPropNameWithUnicode=\u0009
The downside, is that you lose your ability to be portable among varying Python interpreters and now you're locked into Jython. You would be locked into a library if you attempt that approach as well. The reason why I like Jython is that your added flexibility with having all of the Java SE libraries available.
If you need to read all values from a section in properties file in a simple manner:
Your config.properties file layout :
[SECTION_NAME]
key1 = value1
key2 = value2
You code:
import configparser
config = configparser.RawConfigParser()
config.read('path_to_config.properties file')
details_dict = dict(config.items('SECTION_NAME'))
This will give you a dictionary where keys are same as in config file and their corresponding values.
details_dict becomes
{'key1':'value1', 'key2':'value2'}
Now to get key1's value :
value_1 = details_dict['key1']
Putting it all in a method which reads that section from config file only once(the first time the method is called during a program run).
def get_config_dict():
if not hasattr(get_config_dict, 'config_dict'):
get_config_dict.config_dict = dict(config.items('SECTION_NAME'))
return get_config_dict.config_dict
Now call the above function and get the required key's value :
config_details = get_config_dict()
key_1_value = config_details['key1']
I know I can include Python code from a common file using import MyModuleName - but how do I go about importing just a dict?
The problem I'm trying to solve is I have a dict that needs to be in a file in an editable location, while the actual script is in another file. The dict might also be edited by hand, by a non-programmer.
script.py
airportName = 'BRISTOL'
myAirportCode = airportCode[airportName]
myDict.py
airportCode = {'ABERDEEN': 'ABZ', 'BELFAST INTERNATIONAL': 'BFS', 'BIRMINGHAM INTERNATIONAL': 'BHX', 'BIRMINGHAM INTL': 'BHX', 'BOURNMOUTH': 'BOH', 'BRISTOL': 'BRS'}
How do I access the airportCode dict from within script.py?
Just import it
import myDict
print myDict.airportCode
or, better
from myDict import airportCode
print airportCode
Just be careful to put both scripts on the same directory (or make a python package, a subdir with __init__.py file; or put the path to script.py on the PYTHONPATH; but these are "advanced options", just put it on the same directory and it'll be fine).
Assuming your import myDict works, you need to do the following:
from myDict import airportCode
Well, it doesn't need to be a .py file. You could just do:
eval(open("myDict").read())
It's a gaping security hole, though.
Another module you might want to look at is csv for importing CSV files. Then your users could edit it with a spreadsheet and you don't have to teach them Python syntax.
When you perform an import in python you are really just pulling in names into your current namespace. It does not really matter what those names refer to so:
from myDict import airportCode
Will work regardless of whether airportCode is a function, class or just a field as in your case.
If your dict has to be hand-editable by a non-programmer, perhaps it might make more sense using a CSV file for this. Then you editor can even use Excel.
So you can use:
import csv
csvfile = csv.reader(open("airports.csv"))
airportCode = dict(csvfile)
to read a CSV file like
"ABERDEEN","ABZ"
"BELFAST INTERNATIONAL","BFS"
"BIRMINGHAM INTERNATIONAL","BHX"
"BIRMINGHAM INTL","BHX"
"BOURNMOUTH","BOH"
"BRISTOL","BRS"
Careful: If an airport were in that list twice, the last occurrence would silently "overwrite" any previous one(s).
Use csv. Stick import csv with the rest of your module imports,
and then you can do as follows:
f = open('somefile.csv')
reader = csv.DictReader(f, (airport, iatacode))
for row in reader:
print row
which should give you a list of dictionaries:
airport | iatacode
__________________
Aberdeen| ABZ
to create the csv file:
f = open('somefile.csv', 'w')
writer = csv.DictWriter(f, (airport, iatacode))
for row in airportcode:
writer.writerow()
f.close()
which will create a csv file with airports and IATA TLAs in two columns with airport and iatacode as the headers.
You can also skip the dicts and just have strings by using Reader and Writer rather than DictReader and DictWriter.
By default, the csv module produces excel-style csv, but you can set whatever dialect you like as a kwarg.
from myDict import airportCode
airportNode = 'BRISTOL'
myAirportCode = airportCode[airportName]
If myDict should get accessed from a Python module in a different directory, you have to provide a __init__.py module.
For more Information about this topic have a look at the module chapter of the Python documentation.