Open and display a file within a python script in Linux - python

I'd like to open/display an excel file that I'm saving as part of python processing within the python script.
The save portion of the code works fine (ie, it saves successfully, and I'm open and view from a Nautilus window), but attempting to open programmatically throws errors no matter how I attempt it.
I've been using the Popen method within the subprocess package:
from subprocess import Popen
Popen('./temp/testtest.xlsx')
Gives:
PermissionError: [Errno 13] Permission denied: './temp/testtest.xlsx'
I subsequently tried changing file permissions:
import os
from subprocess import Popen
os.chmod('./temp/testtest.xlsx',0o777)
Popen('./temp/testtest.xlsx')
Which gave:
Out[127]: <subprocess.Popen at 0x7faeb22a4b00>invalid file (bad magic number): Exec format error
And against better judgement tried running as shell:
from subprocess import Popen
Popen('./temp/testtest.xlsx', shell=True)
Which gave:
invalid file (bad magic number): Exec format error
Out[129]: <subprocess.Popen at 0x7faeb22a46a0>
I also tried it with the file saved in a different directory with similar errors. If it matters, I'm using the openpyxl module to create and save the excel file, but I have the same issues even if it's an excel file I created manually.

The argument to Popen() needs to be a shell command to open the file. If you're using LibreOffice, the program is localc; if you're using OpenOffice it's oocalc.
from subprocess import Popen
import os
f = os.path.join('temp', filename)
Popen(['localc', f])

You have Many options here. You could install libreoffice, this is an open source office suite and is fairly decent you should be able to open that file directly with ooffice —-calc “filename”. If you really want to stay with python You could save the data to a .csv file, and pythons anaconda distribution has pandas library and you could read the .csv into a data frame fairly easily. import pandas as pd Then
pd.read_csv(“File_name.csv”) returns to you a dataframe, but remember to import os and os.chdir(r“/path/to/data”).
From that point pandas lets you easily access the data for plotting or manipulation.
Here is all the functionality of a data frame and see if it meets your fancy.
Python Pandas DataFrame

Related

How to fix "'bool' object is not iterable" error when running a UNIX command on a directory of files in python

I am trying to run a list of files in a directory through a UNIX executable using a python. I would the output of the executable for each file written to a different directory but retaining the original filename.
I am using python 2.7 so using the subprocess.call method. I am getting an error that says "'bool' object is not iterable" which I am guessing is due to the part where I am trying to write the output files as when I run the following script through the console I get an expected output specific to the executable within the console window:
import subprocess
import os
for inp in os.listdir('/path/to/input/directory/'):
subprocess.call(['/path/to/UNIX/executable', inp])
My code is currently this:
import subprocess
import os
for inp in os.listdir('/path/to/input/directory/'):
out = ['/path/to/output/directory/%s' % inp]
subprocess.call(['/path/to/UNIX/executable', inp] > out)
However, this second lot of code returns the "'bool' is not iterable" error.
I'm guessing the solution is pretty trivial as it is not a complicated task however, as a beginner, I do not know where to start!
SOLVED: following #barak-itkin's answer, for those who may stumble across this issue in the future, the code ran successfully using the following:
import subprocess
import os
for inp in os.listdir('/path/to/input/directory/'):
with open('/path/to/output/directory/%s' % inp, 'w') as out_file:
subprocess.call(['/path/to/UNIX/executable', inp], stdout=out_file)
To write the output of a subprocess.call to a file, you would need to either use the > path/to/out as part of the command itself, or to do it "properly" by specifying the file to which the output should go:
# Option 1:
# Specify that the command is using a "shell" syntax, meaning that
# things like output redirection (such as with ">") should be handled
# by the shell that will evaluate the command
subprocess.call('my_command arg1 arg2 > /path/to/out', shell=True)
# Option 2:
# Open the file to which you want to write the output, and then specify
# the `stdout` parameter to be that file
with open('/path/to/out', 'w') as out_file:
subprocess.call(['my_command', 'arg1', 'arg2'], stdout=out_file)
Does this work for you?

Running python code in Apache Nifi ExecuteStreamCommand

I'm trying to run python code in Nifi ExecuteStreamCommand processor.
The code includes non pure python modules like Pandas and Numpy so to use Nifi executeScript is not an option.
The issue is around reading in flow file and modifying flow file content.
Apparently it is possible to read incoming flow file with STDIN and to write out with STDOUT, see this SO question:
Python Script using ExecuteStreamCommand
But I have not been able to get this working.
1.
Tried simply reading in a CSV from STDIN and modifying it, but when sent to putFile processor the file is the same.
import sys
import pandas as pd
import io
df = pd.read_csv(io.StringIO(sys.stdin.read(1)))
df2 = pd.DataFrame([[5, 6], [7, 8]], columns=list('AB'))
df2 = df.append(df2)
2.
Tried wrapping some other code in a function and returning in assumption that function output would go to STDOUT, but same outcome.
def convert_csv_dataframe():
a = pd.read_csv(io.StringIO(sys.stdin.read(1)))
a.replace(["ABC", "AB"], "A", inplace=True)
return a
convert_csv_dataframe()
If anybody can help it would be most appreciated.
EDIT:
This code works. The issue was in Nifi. I was reading from "original" relationship instead of "output flow" relationship. Note that stdin is reading one line but don't think that should make a difference. The only question I have is: Can I reference a flow file itself (not it's contents) from executeStreamCommand ?
import sys
a = sys.stdin.readline()
a = a.upper()
sys.stdout.write(a)
I think you need to write to STDOUT somewhere in your script. I don't know much Python, but both examples look like you read from STDIN and then modify data in memory, but never write it back out.

how to modify txt file properties with python

I am trying to make a python program that creates and writes in a txt file.
the program works, but I want it to cross the "hidden" thing in the txt file's properties, so that the txt can't be seen without using the python program I made. I have no clues how to do that, please understand I am a beginner in python.
I'm not 100% sure but I don't think you can do this in Python. I'd suggest finding a simple Visual Basic script and running it from your Python file.
Assuming you mean the file-properties, where you can set a file as "hidden". Like in Windows as seen in screenshot below:
Use operating-system's command-line from Python
For example in Windows command-line attrib +h Secret_File.txt to hide a file in CMD.
import subprocess
subprocess.run(["attrib", "+h", "Secret_File.txt"])
See also:
How to execute a program or call a system command?
Directly call OS functions (Windows)
import ctypes
path = "my_hidden_file.txt"
ctypes.windll.kernel32.SetFileAttributesW(path, 2)
See also:
Hide Folders/ File with Python
Rename the file (Linux)
import os
filename = "my_hidden_file.txt"
os.rename(filename, '.'+filename) # the prefix dot means hidden in Linux
See also:
How to rename a file using Python

How to start external program and pass in CSV file?

I am trying to start a program and write a saved CSV file to it.
I have the following code to start the program, that works fine:
os.startfile('C:\Program Files\Files\CSV_Reader')
But I'm not sure how to open the CSV File. I tried a few other options like:
os.system('start CSV_Reader.exe "{0}C:\Program Files\Files\card_kingdom.csv"'.format(sys.path[0], ))
But I get the following error message:
C:\Users\JJ\AppData\Local\Programs\Python\Python36\python.exe C:/Users/JJ/PycharmProjects/Buylist/CK_file_test
The system cannot find the file CSV_Reader.exe.
I know that the directory is right, because when I call the os.startfile function it opens the program.
Your help would be much appreciated!
You can use popen like it:
import subprocess
subprocess.Popen(["C:\Program Files\Files\CSV_Reader.exe" , "C:\Program Files\Files\card_kingdom.csv"])

Editing temporary file with Vim in Python subprocess not working as expected on Mac OS

My initial goal was to get user input via a command-line text-editor from within a Python script. More specifically, my plan was to create a temporary file and populate it with some pre-written text, open the file with a text-editor and allow the user to modify the file content, read the data from the file after the user exits the editor, then finally delete the file after it's all over.
I seem to have found a way to do this that is working for me, but along the way I tried a couple of approaches that did not work and I'd like to understand exactly why.
Consider the following Python script (a slightly modified version of the script taken from this post):
#!/usr/bin/env python2
# -*- encoding: ascii -*-
"""callvim.py
Demonstrates calling a text-editor (e.g. Vim) from within a Python script,
including passing input to the editor and reading output from the editor.
"""
import tempfile
import os
from subprocess import call
# Get the text editor from the shell, otherwise default to Vim
EDITOR = os.environ.get('EDITOR','vim')
# Set initial input with which to populate the buffer
initial_message = "Hello world!"
# Open a temporary file to communicate through
with tempfile.NamedTemporaryFile(suffix=".tmp") as tf:
# Write the initial content to the file I/O buffer
tf.write(initial_message)
# Flush the I/O buffer to make sure the data is written to the file
tf.flush()
# Open the file with the text editor
call([EDITOR, tf.name])
# Rewind the file offset to the beginning of the file
tf.seek(0)
# Read the file data into a variable
edited_message = tf.read()
# Output the data
print(edited_message)
I've tried running this script in two different environments so far: on a macOS computer (running macOS 10.12) and on a Debian computer (running Debian 8.8). Both computers have the same (minor) version of Vim installed (Vim 7.4).
When I run this script with EDITOR=vim on my Debian 8 (Jessie) machine it works as expected. I'm prompted with Vim and a buffer containing the string "Hello world!". After editing the buffer to contain the string "Goodbye world!", saving the file, and exiting Vim, I see the string "Goodbye world!" printed to the console.
When I run the same script on my macOS 10.12 (Sierra) machine it does not seem to work. The same procedure results in "Hello world!" being displayed on-screen - as if the file is being read before it is edited.
However if run the script on my Mac with EDITOR=nano then once again everything seems to work as expected.
I tried a few variations on this script using different methods from the tempfile module (e.g. using tempfile.TemporaryFile() and tempfile.mkstemp()) with the same results.
Now consider the following alternative script:
#!/usr/bin/env python2
# -*- encoding: ascii -*-
"""callvim.py
Demonstrates calling a text-editor (e.g. Vim) from within a Python script,
including passing input to the editor and reading output from the editor.
"""
import subprocess
import os
# Create a temporary file and write some default text
file_path = "tempfile"
file_handle = open(file_path, "w")
file_handle.write("Hello world!")
file_handle.close()
# Open the file with Vim
subprocess.call(["vim", file_path])
# Rewind to the beginning of the file
file_handle = open(file_path, 'r')
# Read the data from the file
data = file_handle.read()
# Close the temporary file
file_handle.close()
# Delete the temporary file
os.remove(file_path)
# Print the data
print(data)
This script, which avoids using the tempfile module, appears to be working consistently across both platforms.
So it seems that this script may be failing for some reason having to do with how Vim and the tempfile Python module interact on macOS. What's going on here?
This is happening because your second script closes the file handle before invoking vim, then opens a new one afterwards, whereas the first script doesn't. It has nothing to do with the tempfile module per se. This code works as expected:
import tempfile, os
from subprocess import call
initial_message = "Hello world!"
tf = tempfile.NamedTemporaryFile(suffix=".tmp", delete=False)
tf.write(initial_message)
tf.close()
call(['vim', tf.name])
tf = open(tf.name, 'r')
edited_message = tf.read()
tf.close()
os.unlink(tf.name)
print(edited_message)
Note the delete=False in the call to NamedTemporaryFile, which ensures that the file isn't deleted when we close it the first time (we have to delete it manually with os.unlink later).
I think what's going on here is that vim isn't editing your file in-place, it's writing to a swap file. When you save and quit, vim replaces the original file with the edited one, leaving your file handle pointing to the old, unedited one. There are ways to prevent vim from using swap files (see, e.g. here), but I wouldn't recommend them. Just remember to update your file handles after vim has done its business.

Categories