We are not ready to automatically format the whole source code with black.
But from time to time I would like to execute black -S on a region via PyCharm.
There is a hint in the docs how to run black (or black -S (what I like)) on the whole file. But ...
How to run black only on a selected region?
Using Python Black on a code region in the PyCharm IDE can be done by implementing it as an external tool. Currently Black has two main options to choose the code to format
Run Black on the whole module specifying it on the CLI as the [SRC]...
Passing the code region as a string on the CLI using the -c, --code TEXT option.
The following implementation shows how to do this using the 2nd option. The reason is that applying Black to the whole module is likely to change the number of lines thus making the job of selecting the code region by choosing start and end line numbers more complicated.
Implementing the 1st option can be done, but it would require mapping the initial code region to the final code region after Black formats the entire module.
Lets take as example the following code that has a number of obvious PEP-8 violations (missing white-spaces and empty lines):
"""
long multi-line
comment
"""
def foo(token:int=None)->None:
a=token+1
class bar:
foo:int=None
def the_simple_test():
"""the_simple_test"""
pass
Step 1.
Using Black as an external tool in the IDE can be configured by going to File > Tools > External Tools and clicking the Add or Edit icons.
What is of interesst is passing the right Macros - (see point 3 "Parameter with macros") from the PyCharm IDE to the custom script that calls Black and does the necessary processing. Namely you'll need the Macros
FilePath - File Path
SelectionStartLine - Selected text start line number
SelectionEndLine - Select text end line number
PyInterpreterDirectory - The directory containing the Python interpreter selected for the project
But from time to time I would like to execute black -S on a region via PyCharm.
Any additional Black CLI options you want to pass as arguments are best placed at the end of the parameter list.
Since you may have Black installed on a specific venv, the example also uses the PyInterpreterDirectory macro.
The screenshot illustrates the above:
Step 2.
You'll need to implement a script to call Black and interface with the IDE. The following is a working example. It should be noted:
Four lines are OS/shell specific as commented (it should be trivial to adapt them to your environment).
Some details could be further tweaked, for purpose of example the implementation makes simplistic choices.
import os
import pathlib
import tempfile
import subprocess
import sys
def region_to_str(file_path: pathlib.Path, start_line: int, end_line: int) -> str:
file = open(file_path)
str_build = list()
for line_number, line in enumerate(file, start=1):
if line_number > end_line:
break
elif line_number < start_line:
continue
else:
str_build.append(line)
return "".join(str_build)
def black_to_clipboard(py_interpeter, black_cli_options, code_region_str):
py_interpreter_path = pathlib.Path(py_interpeter) / "python.exe" # OS specific, .exe for Windows.
proc = subprocess.Popen([py_interpreter_path, "-m", "black", *black_cli_options,
"-c", code_region_str], stdout=subprocess.PIPE)
try:
outs, errs = proc.communicate(timeout=15)
except TimeoutExpired:
proc.kill()
outs, errs = proc.communicate()
# By default Black outputs binary, decodes to default Python module utf-8 encoding.
result = outs.decode('utf-8').replace('\r','') # OS specific, remove \r from \n\r Windows new-line.
tmp_dir_name = tempfile.gettempdir()
tmp_file = tempfile.gettempdir() + "\\__run_black_tmp.txt" # OS specific, escaped path separator.
with open(tmp_file, mode='w+', encoding='utf-8', errors='strict') as out_file:
out_file.write(result + '\n')
command = 'clip < ' + str(tmp_file) # OS specific, send result to clipboard for copy-paste.
os.system(command)
def main(argv: list[str] = sys.argv[1:]) -> int:
"""External tool script to run black on a code region.
Args:
argv[0] (str): Path to module containing code region.
argv[1] (str): Code region start line.
argv[2] (str): Code region end line.
argv[3] (str): Path to venv /Scripts directory.
argv[4:] (str): Black CLI options.
"""
# print(argv)
lines_as_str = region_to_str(argv[0], int(argv[1]), int(argv[2]))
black_to_clipboard(argv[3], argv[4:], lines_as_str)
if __name__ == "__main__":
main(sys.argv[1:])
Step 3.
The hard part is done. Lets use the new functionality.
Normally select the lines you want as your code region in the editor. This has to be emphasized because the previous SelectionStartLine and SelectionEndLine macros need the selection to work. (See the next screenshot).
Step 4.
Run the external tool previously implemented. This can be done by right clicking in the editor and choosing External Tools > the_name_of_your_external_tool.
Step 5.
Simply paste (the screenshot shows the result after running the external tool and pressing Ctrl + v). The implementation in Step 2 copies Black's output to your OS's clipboard, this seemed like the preferable solution since this way you change the file inside the editor thus Undo Ctrl + z will also work. Changing the file by overwrite it programmatically outside the editor would be less seamless and might require refreshing it inside the editor.
Step 6.
You can record a macro of the previous steps and associate it with a keyboard shortcut to have the above functionality in one keystroke (similar to copy-paste Ctrl + c + Ctrl + v).
End Notes.
If you need to debug the functionality in Step 2 a Run Configuration can also be configured using the same macros the external tool configuration did.
It's important to notice when using the clipboard that character encodings can change across the layers. I decided to use clip and read into it directly from a temporary file, this was to avoid passing the code string to Black on the command line because the CMD Windows encoding is not UTF-8 by default. (For Linux users this should be simpler but can depend on your system settings.)
One important note is that you can choose a code region without the broader context of its indentation level. Meaning, for example, if you only choose 2 methods inside a class they will be passed to Black and formatted with the indentation level of module level functions. This shouldn't be a problem if you are careful to select code regions with their proper scope. This could also easily be solved by passing the additional macro SelectionStartColumn - Select text start column number from Step 1 and prepending that number of whitespaces to each line in the Step 2 script. (Ideally such functionality would be implemented by Black as a CLI option.) In any case, if needed, using Tab to put the region in its proper indentation level is easy enough.
The main topic of the question is how to integrating Black with the PyCharm IDE for a code region, so demonstrating the 2nd option should be enough to address the problem because the 1st option would, for the most part, only add implementation specific complexity. (The answer is long enough as it is. The specifics of implementing the 1st option would make a good Feature/Pull Request for the Black project.)
I have researched about this because it actually looks interesting, and I've came to the conclusion that you can maybe use:
black -S and_your_file_path
or:
black -c and_a_string
to format the code passed in as a string.
I will also follow this thread because it looks interesting.
And I'm also going to do more research on this and if I find something I will let you know.
Related
In my Windows 10 scenario, I want to print arbitrary python console output (print(), sys.exit(), etc) both to console and a log file. I don't have control over some parts of the code (external python packages), so I cannot use some dedicated logging mechanism.
After some research I found the tool tee.exe in UnxUtils which does this task almost the way I want.
My problem is to preserve color as generated by python's colorama package. Is there any way to accomplish that? Currently, tee.exe strips away the color.
The answer I am looking does not have to rely on tee.exe, it's just the closest I got to a real solution. What I am looking for should do the following:
any command line output appears both in the command line and the log file (both STOUT and STDERR)
the output appears on the command line in real time. Bonus points if this is also true for the log file.
color is preserved on the command line. Bonus points if the log file does not contain any color-related artifacts.
What I have so far is this:
Python file teetest.py:
import sys
import colorama
print("Test")
print("2nd Test")
colorama.init(autoreset=True)
print(colorama.Fore.RED + 'Color Test')
colorama.deinit(autoreset=True)
sys.exit("Error_Test")
Batch file teetest.bat:
#echo off
python teetest.py 2>&1 | tee log.txt
pause
My output looks like this (command line and log file are identical, no color):
Test
2nd Test
Color Test
Error_Test
The solution I am looking for will print the above to the command line so the words Color Test are red.
Edit:
It seems that tee.exe is not at fault. Instead, colorama strips away the ANSI characters for color control by design, so the color is lost when passed through tee.exe.
From the colorama manual:
Colorama makes this work on Windows, too, by wrapping stdout, stripping ANSI sequences it finds (which would appear as gobbledygook in the output), and converting them into the appropriate win32 calls to modify the state of the terminal.
Colorama's init() function offers the parameter strip, which if False, does cause colorama to not strip away the ANSI characters. This in turn allows to write a custom tee.py that does the same as tee.exe, as outlined by user #martineau below. In it, we can call colorama and handle the color properly.
This might be a workable solution, but it still has the downside that I would have to replace all colorama init() calls with init(strip=False) in my original python code and that in turn would cause ANSI characters to appear in the output if the code was called without redirecting through tee.py.
This might actually be the closest we can get to a proper solution here. If anyone can offer other ideas, I'm all ears but I fear chances are slim.
I don't know how it will work with respect to colorama, but after being unsatisfied with several tee utilities for Windows that I found online, I ended up writing my own in Python (3.x).
You may have to modify it to suit your own needs, but it should be a good start.
mytee.py:
"""
Copies stdin to stdout (screen) *and* the specified file.
"""
import fileinput
import os
from pathlib import Path
import sys
SHOW_FULL_PATH = False
DIVIDER = True
DIV_CH = ' '
if len(sys.argv) != 2:
raise SystemExit('Usage: mytee <filepath>')
try:
inp = fileinput.input(()) # Read from stdin.
path = Path(sys.argv[1])
stdout_write = sys.stdout.write
stdout_flush = sys.stdout.flush
# Assumes .py in same dir as output file.
script = (f'{path.parent/path.stem}{path.suffix}' if SHOW_FULL_PATH else
f'{path.stem}{path.suffix}')
with open(path, 'w') as outp: # Write to specified file.
outp_write = outp.write
outp_flush = outp.flush
def write(line):
stdout_write(line)
outp_write(line)
def writeln(line):
write(line + '\n')
banner = f'"{script}"' if ' ' in script else f'-[{script}]-'
writeln(f'{banner}')
if DIVIDER:
writeln(f'{DIV_CH * len(banner)}')
for line in inp:
write(line)
if DIVIDER:
writeln(f'{DIV_CH * len(banner)}')
writeln('-[done]-')
finally:
inp.close() # Not sure this is really necessary.
sys.exit(0)
I'm working inside a system that has Jython2.5 but I need to be able to call some of Google's apis so I wrote an offline script that I wanted to call from my Jython environment and return to me small pieces of data. Like a JobID or a sheet URL or something from Google.
I've tried a number of things but I always get an error back from Windows, saying that it cannot find the file specified.
Path is done in two ways.
The first way using a string
stringPath = r"C:\GooglePipes\Scripts\filetobq.py C:\GooglePipes\Keys\DEV-BigQueryKey.json nofile C:\GooglePipes\BQ_Downtime\TESTFILE.CSV dataset1 table1"
And the second way, as a sequence (per the docs, using shell=false supply a sequence)
seqPath = [r"C:\GooglePipes\Scripts\filetobq.py",r"C:\GooglePipes\Keys\DEV-BigQueryKey.json","nofile",r"C:\GooglePipes\BQ_Downtime\TESTFILE.CSV","dataset1","table1"]
Called with
data, err = Popen(seqPath, shell=True, stderr=PIPE, stdout=PIPE).communicate()
#Read values back in
print data
print err
Replacing seqPath with stringPath to try it either way.
I've been at this all weekend, every time I run it I get from Windows
The system cannot find the path specified.
from the err print. I've been unable to debug much further than this. I'm not really sure what's happening. When I paste the stringPath variable directly into my computer's command window it executes.
I've also called subprocess.list2cmdline(seqPath) to see what it's outputting. It's giving me a ? in front of the string, but I haven't been able to figure out what that means. I can paste the rest of the string, starting after the question mark into the command window and it executes.
?C:\GooglePipes\Scripts\filetobq.py C:\GooglePipes...
I've tried a number of different combinations of true and false on shell, passing different args into Popen, double slashes, and I have no less than 30 tabs open from stack overflow and other help forums. I just have no idea what to do at this point and any help is appreciated.
Edit
The ? at the start of the sting is actually a NULL character when I did some additional logging. This seems to be the root of my problem. I can't figure out why it shows up, but it was present in my copy pastes. I started manually typing, and I got it working. When I feed the path with my Jython program it is present again.
Ultimately the error was the ?/NULL character.
I went back to the source value where the program was grabbing the path and it was present there. After I hand-re keyed it in, everything started working.
If you copy and paste what I put in the question, you can see the NULL character in the string if you run it through a string->ASCII converter.
>C:
>NULL 67 58
What a bunch of bullsh***.
I have written piece of code which runs sextractor from python, however I only know how to do this for one file, and i need to loop it over 62 files. Im not sure how i would go about doing this. I have attached my code bellow:
#!/usr/bin/env python
# build a catalog using sextractor on des image here
sys.path.append('/home/fitsfiles') #not sure if this does anything/is correct
def sex(image, output, sexdir='/home/sextractor-2.5.0', check_img=None,config=None, l=None) :
'''Construct a sextractor command and run it.'''
#creates a sextractor line e.g sex img.fits -catalog_name -checkimage_name
q="/home/fitsfiles/"+ "01" +".fits"
com = [ "sex ", q, " -CATALOG_NAME " + output]
s0=''
com = s0.join(com)
res = os.system(com)
return res
img_name=sys.argv[0]
output=img_name[0:1]+'_star_catalog.fits'
t=sex(img_name,output)
print '----done !---'
so this code produces a command in my main terminal of, sex /home/fitsfiles/01.fits -CATALOG_NAME g_star_catalog.fits
which successfully produces a star catalogue as I want.
However I want my code to to this for 62 fits files and change the name of star_catalog.fits depending upon which fitsfile is being used. any help would be appreciated.
There are many ways you could approach this. Let's assume you want to run your script as something like
python extract_stars.py /home/fitsfiles/*.fits
Then, you could try something like this:
for arg in len(sys.argv):
filename = arg.split('/')[-1].strip('.fits')
t = sex(arg, filename +'_star_catalog.fits')
# Whatever else
This assumes that you remove the line in sex that reformats the input filename. Also, you do not need to append the fits directory to your path.
The alternative approach is, if you do not plan to do anything else in python, you could write a bash script which would really simplify the task.
And, as a side note, you if you had asked this question more generally (ie, I wish to apply a function I wrote to a number of input files) and without reference to a rather uncommonly used application, you would have likely received an answer much more quickly.
The community has now developed some python wrappers which allow you to run sextractor as if it was a python command. These are: pysex, sewpy and astromatic_wrapper.
The good thing about sextractor wrappers is that allow you to write much cleaner code without the need of defining extra functions, invoking os commands or having the configuration files and the outputfiles on your machine. Moreover, the output can be an astropy table, a pandas dataframe or a numpy array.
For your specific case, you could use pysex and do:
import pysex
import glob
filelist = glob.glob('/directory/*.fits')
for fitsfile in filelist:
cat = pysex.run(fitsfile, params=['X_IMAGE', 'Y_IMAGE', 'FLUX_APER'],
conf_args={'PHOT_APERTURES':5})
print cat['FLUX_APER']
Is there any easy way to handle multiple lines user input in command-line Python application?
I was looking for an answer without any result, because I don't want to:
read data from a file (I know, it's the easiest way);
create any GUI (let's stay with just a command line, OK?);
load text line by line (it should pasted at once, not typed and not pasted line by line);
work with each of lines separately (I'd like to have whole text as a string).
What I would like to achieve is to allow user pasting whole text (containing multiple lines) and capture the input as one string in entirely command-line tool. Is it possible in Python?
It would be great, if the solution worked both in Linux and Windows environments (I've heard that e.g. some solutions may cause problems due to the way cmd.exe works).
import sys
text = sys.stdin.read()
After pasting, you have to tell python that there is no more input by sending an end-of-file control character (ctrl+D in Linux, ctrl+Z followed by enter in Windows).
This method also works with pipes. If the above script is called paste.py, you can do
$ echo "hello" | python paste.py
and text will be equal to "hello\n". It's the same in windows:
C:\Python27>dir | python paste.py
The above command will save the output of dir to the text variable. There is no need to manually type an end-of-file character when the input is provided using pipes -- python will be notified automatically when the program creating the input has completed.
You could get the text from clipboard without any additional actions which raw_input() requires from a user to paste the multiline text:
import Tkinter
root = Tkinter.Tk()
root.withdraw()
text = root.clipboard_get()
root.destroy()
See also How do I copy a string to the clipboard on Windows using Python?
Use :
input = raw_input("Enter text")
These gets in input as a string all the input. So if you paste a whole text, all of it will be in the input variable.
EDIT: Apparently, this works only with Python Shell on Windows.
In the actual window where I right code is there a way to insert part of the code into everyline that I already have. Like insert a comma into all lines at the first spot>?
You need a file editor, not python.
Install the appropriate VIM variant for your operating system
Open the file you want to modify using VIM
Type: :%s/^/,/
Type: :wq
If you are in UNIX environment, open up a terminal, cd to the directory your file is in and use the sed command. I think this may work:
sed "s/\n/\n,/" your_filename.py > new_filename.py
What this says is to replace all \n (newline character) to \n, (newline character + comma character) in your_filename.py and to output the result into new_filename.py.
UPDATE: This is much better:
sed "s/^/,/" your_filename.py > new_filename.py
This is very similar to the previous example, however we use the regular expression token ^ which matches the beginning of each line (and $ is the symbol for end).
There are chances this doesn't work or that it doesn't even apply to you because you didn't really provide that much information in your question (and I would have just commented on it, but I can't because I don't have enough reputation or something). Good luck.
Are you talking about the interactive shell? (a.k.a. opening up a prompt and typing python)? You can't go back and edit what those previous commands did (as they have been executed), but you can hit the up arrow to flip through those commands to edit and reexecute them.
If you're doing anything very long, the best bet is to write your program into your text editor of choice, save that file, then launch it.
Adding a comma to the start of every line with Python:
import sys
src = open(sys.argv[1])
dest = open('withcommas-' + sys.argv[1],'w')
for line in src:
dest.write(',' + line)
src.close()
dest.close()
Call like so: C:\Scripts>python commaz.py cc.py. This is a bizzare thing to do, but who am I to argue.
Code is data. You could do this like you would with any other text file. Open the file, read the line, stick a comma on the front of it, then write it back to file.
Also, most modern IDEs/text editors have the ability to define macros. You could post a question asking for specific help for your editor. For example, in Emacs I would use C-x ( to start defining a macro, then ',' to write a comma, then C-b C-n to go back a character and down a line, then C-x ) to end my macro. I could then run this macro with C-x e, pressing e to execute it an additional time.