Emacs: Altering python.el indentation in derived mode - python

I am attempting to derive a new emacs mode from python.el (the current official gnu one) for Boo and I am having trouble with altering the indentation. Does anyone have any suggestions about how to best handle this? I do not need to change anything drastically, just add some new block forms and stuff.
For example, since this is for Boo, the try/except syntax uses "ensure" instead of "finally". I can change this easily enough in python.el by changing the block-start def of python-rx-constituents. However, I can't seem to be able to override this in a derived mode because python-rx-constituents is being then used by a macro, python-rx, and I guess once those two things are defined when python.el loads (as it has to, since I am deriving from it), I can no longer override it after-load or in a hook? Because I've definitely changed it in memory and in a hook after python.el loads and in an after-load statement and none of them work. While directly altering python.el works fine.
Here is the code in question from python.el:
(eval-when-compile
(defconst python-rx-constituents
`((block-start . ,(rx symbol-start
(or "def" "class" "if" "elif" "else" "try"
"except" "finally" "for" "while" "with"
)
symbol-end))
(decorator . ,(rx line-start (* space) ?# (any letter ?_)
(* (any word ?_))))
(defun . ,(rx symbol-start (or "def" "class") symbol-end))
(if-name-main . ,(rx line-start "if" (+ space) "__name__"
(+ space) "==" (+ space)
(any ?' ?\") "__main__" (any ?' ?\")
(* space) ?:))
(symbol-name . ,(rx (any letter ?_) (* (any word ?_))))
(open-paren . ,(rx (or "{" "[" "(")))
(close-paren . ,(rx (or "}" "]" ")")))
(simple-operator . ,(rx (any ?+ ?- ?/ ?& ?^ ?~ ?| ?* ?< ?> ?= ?%)))
;; FIXME: rx should support (not simple-operator).
(not-simple-operator . ,(rx
(not
(any ?+ ?- ?/ ?& ?^ ?~ ?| ?* ?< ?> ?= ?%))))
;; FIXME: Use regexp-opt.
(operator . ,(rx (or "+" "-" "/" "&" "^" "~" "|" "*" "<" ">"
"=" "%" "**" "//" "<<" ">>" "<=" "!="
"==" ">=" "is" "not")))
;; FIXME: Use regexp-opt.
(assignment-operator . ,(rx (or "=" "+=" "-=" "*=" "/=" "//=" "%=" "**="
">>=" "<<=" "&=" "^=" "|=")))
(string-delimiter . ,(rx (and
;; Match even number of backslashes.
(or (not (any ?\\ ?\' ?\")) point
;; Quotes might be preceded by a escaped quote.
(and (or (not (any ?\\)) point) ?\\
(* ?\\ ?\\) (any ?\' ?\")))
(* ?\\ ?\\)
;; Match single or triple quotes of any kind.
(group (or "\"" "\"\"\"" "'" "'''"))))))
"Additional Python specific sexps for `python-rx'")
(defmacro python-rx (&rest regexps)
"Python mode specialized rx macro.
This variant of `rx' supports common python named REGEXPS."
(let ((rx-constituents (append python-rx-constituents rx-constituents)))
(cond ((null regexps)
(error "No regexp"))
((cdr regexps)
(rx-to-string `(and ,#regexps) t))
(t
(rx-to-string (car regexps) t))))))
I would like to change python-rx-constituents so that block-start includes "ensure" instead of finally.

As commented already, employing derived-mode is not suitable here: you can't back-change a macro. Also re-defining it isn't recommendable: the order of loading/evaluation than will decide which one is in effect - at a larger scale that means running into a mess.
Copy the file, store as boo.el, replace the prefix by "boo-", reload and edit the stuff which needs changing.
Your concern expressed IMO isn't justified, as permitting copying, changing and re-release of the changed code is the core of the GPL.

Copying the file is a bad idea, indeed, since it makes it painful to track the evolution of python.el.
Deriving is not that great an idea either, since Boo isn't really an extension or a variation of Python, so you probably want to reuse some of its indentation machinery and some of its code to understand the significant-indentation, but not much more than that.
You might like to contact python.el's author (with Cc to emacs-devel) to see if python.el could be adjusted to make your life easier. e.g. maybe extract the common code to an auxiliary file that can be shared between the two. Ideally, this file might be usable for cofferscript-mode and maybe Haskell-mode as well.

Related

Python wont open a file named PRN.anything [duplicate]

I know that / is illegal in Linux, and the following are illegal in Windows
(I think) * . " / \ [ ] : ; | ,
What else am I missing?
I need a comprehensive guide, however, and one that takes into account
double-byte characters. Linking to outside resources is fine with me.
I need to first create a directory on the filesystem using a name that may
contain forbidden characters, so I plan to replace those characters with
underscores. I then need to write this directory and its contents to a zip file
(using Java), so any additional advice concerning the names of zip directories
would be appreciated.
The forbidden printable ASCII characters are:
Linux/Unix:
/ (forward slash)
Windows:
< (less than)
> (greater than)
: (colon - sometimes works, but is actually NTFS Alternate Data Streams)
" (double quote)
/ (forward slash)
\ (backslash)
| (vertical bar or pipe)
? (question mark)
* (asterisk)
Non-printable characters
If your data comes from a source that would permit non-printable characters then there is more to check for.
Linux/Unix:
0 (NULL byte)
Windows:
0-31 (ASCII control characters)
Note: While it is legal under Linux/Unix file systems to create files with control characters in the filename, it might be a nightmare for the users to deal with such files.
Reserved file names
The following filenames are reserved:
Windows:
CON, PRN, AUX, NUL
COM1, COM2, COM3, COM4, COM5, COM6, COM7, COM8, COM9
LPT1, LPT2, LPT3, LPT4, LPT5, LPT6, LPT7, LPT8, LPT9
(both on their own and with arbitrary file extensions, e.g. LPT1.txt).
Other rules
Windows:
Filenames cannot end in a space or dot.
macOS:
You didn't ask for it, but just in case: Colon : and forward slash / depending on context are not permitted (e.g. Finder supports slashes, terminal supports colons). (More details)
A “comprehensive guide” of forbidden filename characters is not going to work on Windows because it reserves filenames as well as characters. Yes, characters like
* " ? and others are forbidden, but there are a infinite number of names composed only of valid characters that are forbidden. For example, spaces and dots are valid filename characters, but names composed only of those characters are forbidden.
Windows does not distinguish between upper-case and lower-case characters, so you cannot create a folder named A if one named a already exists. Worse, seemingly-allowed names like PRN and CON, and many others, are reserved and not allowed. Windows also has several length restrictions; a filename valid in one folder may become invalid if moved to another folder. The rules for
naming files and folders
are on the Microsoft docs.
You cannot, in general, use user-generated text to create Windows directory names. If you want to allow users to name anything they want, you have to create safe names like A, AB, A2 et al., store user-generated names and their path equivalents in an application data file, and perform path mapping in your application.
If you absolutely must allow user-generated folder names, the only way to tell if they are invalid is to catch exceptions and assume the name is invalid. Even that is fraught with peril, as the exceptions thrown for denied access, offline drives, and out of drive space overlap with those that can be thrown for invalid names. You are opening up one huge can of hurt.
Under Linux and other Unix-related systems, there were traditionally only two characters that could not appear in the name of a file or directory, and those are NUL '\0' and slash '/'. The slash, of course, can appear in a pathname, separating directory components.
Rumour1 has it that Steven Bourne (of 'shell' fame) had a directory containing 254 files, one for every single letter (character code) that can appear in a file name (excluding /, '\0'; the name . was the current directory, of course). It was used to test the Bourne shell and routinely wrought havoc on unwary programs such as backup programs.
Other people have covered the rules for Windows filenames, with links to Microsoft and Wikipedia on the topic.
Note that MacOS X has a case-insensitive file system. Current versions of it appear to allow colon : in file names, though historically that was not necessarily always the case:
$ echo a:b > a:b
$ ls -l a:b
-rw-r--r-- 1 jonathanleffler staff 4 Nov 12 07:38 a:b
$
However, at least with macOS Big Sur 11.7, the file system does not allow file names that are not valid UTF-8 strings. That means the file name cannot consist of the bytes that are always invalid in UTF-8 (0xC0, 0xC1, 0xF5-0xFF), and you can't use the continuation bytes 0x80..0xBF as the only byte in a file name. The error given is 92 Illegal byte sequence.
POSIX defines a Portable Filename Character Set consisting of:
A B C D E F G H I J K L M N O P Q R S T U V W X Y Z
a b c d e f g h i j k l m n o p q r s t u v w x y z
0 1 2 3 4 5 6 7 8 9 . _ -
Sticking with names formed solely from those characters avoids most of the problems, though Windows still adds some complications.
1 It was Kernighan & Pike in ['The Practice of Programming'](http://www.cs.princeton.edu/~bwk/tpop.webpage/) who said as much in Chapter 6, Testing, §6.5 Stress Tests:
When Steve Bourne was writing his Unix shell (which came to be known as the Bourne shell), he made a directory of 254 files with one-character names, one for each byte value except '\0' and slash, the two characters that cannot appear in Unix file names. He used that directory for all manner of tests of pattern-matching and tokenization. (The test directory was of course created by a program.) For years afterwards, that directory was the bane of file-tree-walking programs; it tested them to destruction.
Note that the directory must have contained entries . and .., so it was arguably 253 files (and 2 directories), or 255 name entries, rather than 254 files. This doesn't affect the effectiveness of the anecdote, or the careful testing it describes.
TPOP was previously at
http://plan9.bell-labs.com/cm/cs/tpop and
http://cm.bell-labs.com/cm/cs/tpop but both are now (2021-11-12) broken.
See also Wikipedia on TPOP.
Instead of creating a blacklist of characters, you could use a whitelist. All things considered, the range of characters that make sense in a file or directory name context is quite short, and unless you have some very specific naming requirements your users will not hold it against your application if they cannot use the whole ASCII table.
It does not solve the problem of reserved names in the target file system, but with a whitelist it is easier to mitigate the risks at the source.
In that spirit, this is a range of characters that can be considered safe:
Letters (a-z A-Z) - Unicode characters as well, if needed
Digits (0-9)
Underscore (_)
Hyphen (-)
Space
Dot (.)
And any additional safe characters you wish to allow. Beyond this, you just have to enforce some additional rules regarding spaces and dots. This is usually sufficient:
Name must contain at least one letter or number (to avoid only dots/spaces)
Name must start with a letter or number (to avoid leading dots/spaces)
Name may not end with a dot or space (simply trim those if present, like Explorer does)
This already allows quite complex and nonsensical names. For example, these names would be possible with these rules, and be valid file names in Windows/Linux:
A...........ext
B -.- .ext
In essence, even with so few whitelisted characters you should still decide what actually makes sense, and validate/adjust the name accordingly. In one of my applications, I used the same rules as above but stripped any duplicate dots and spaces.
The easy way to get Windows to tell you the answer is to attempt to rename a file via Explorer and type in a backslash, /, for the new name. Windows will popup a message box telling you the list of illegal characters.
A filename cannot contain any of the following characters:
\ / : * ? " < > |
Microsoft Docs - Naming Files, Paths, and Namespaces - Naming Conventions
Well, if only for research purposes, then your best bet is to look at this Wikipedia entry on Filenames.
If you want to write a portable function to validate user input and create filenames based on that, the short answer is don't. Take a look at a portable module like Perl's File::Spec to have a glimpse to all the hops needed to accomplish such a "simple" task.
Discussing different possible approaches
Difficulties with defining, what's legal and not were already adressed and whitelists were suggested. But not only Windows, but also many unixoid OSes support more-than-8-bit characters such as Unicode. You could here also talk about encodings such as UTF-8. You can consider Jonathan Leffler's comment, where he gives info about modern Linux and describes details for MacOS. Wikipedia states, that (for example) the
modifier letter colon [(See 7. below) is] sometimes used in Windows filenames as it is identical to the colon in the Segoe UI font used for filenames. The [inherited ASCII] colon itself is not permitted.
Therefore, I want to present a much more liberal approach using Unicode Homoglyph characters to replace the "illegal" ones. I found the result in my comparable use-case by far more readable and it's only limited by the used font, which is very broad, 3903 characters for Windows default. Plus you can even restore the original content from the replacements.
Possible choices and research notes
To keep things organized, I will always give the character, it's name and the hexadecimal number representation. The latter is is not case sensitive and leading zeroes can be added or ommitted freely, so for example U+002A and u+2a are equivalent. If available, I'll try to point to more info or alternatives - feel free to show me more or better ones.
Instead of * (U+2A * ASTERISK), you can use one of the many listed, for example U+2217 ∗ (ASTERISK OPERATOR) or the Full Width Asterisk U+FF0A *. u+20f0 ⃰ combining asterisk above from combining diacritical marks for symbols might also be a valid choice. You can read 4. for more info about the combining characters.
Instead of . (U+2E . full stop), one of these could be a good option, for example ⋅ U+22C5 dot operator.
Instead of " (U+22 " quotation mark), you can use “ U+201C english leftdoublequotemark, more alternatives see here. I also included some of the good suggestions of Wally Brockway's answer, in this case u+2036 ‶ reversed double prime and u+2033 ″ double prime - I will from now on denote ideas from that source by ¹³.
Instead of / (U+2F / SOLIDUS), you can use ∕ DIVISION SLASH U+2215 (others here), ̸ U+0338 COMBINING LONG SOLIDUS OVERLAY, ̷ COMBINING SHORT SOLIDUS OVERLAY U+0337 or u+2044 ⁄ fraction slash¹³. Be aware about spacing for some characters, including the combining or overlay ones, as they have no width and can produce something like -> ̸th̷is which is ̸th̷is. With added spaces you get -> ̸ th ̷ is, which is ̸ th ̷ is. The second one (COMBINING SHORT SOLIDUS OVERLAY) looks bad in the stackoverflow-font.
Instead of \ (U+5C Reverse solidus), you can use ⧵ U+29F5 Reverse solidus operator (more) or u+20E5 ⃥ combining reverse solidus overlay¹³.
To replace [ (U+5B [ Left square bracket) and ] (U+005D ] Right square bracket), you can use for example U+FF3B[ FULLWIDTH LEFT SQUARE BRACKET and U+FF3D ]FULLWIDTH RIGHT SQUARE BRACKET (from here, more possibilities here).
Instead of : (u+3a : colon), you can use U+2236 ∶ RATIO (for mathematical usage) or U+A789 ꞉ MODIFIER LETTER COLON, (see colon (letter), sometimes used in Windows filenames as it is identical to the colon in the Segoe UI font used for filenames. The colon itself is not permitted ... source and more replacements see here). Another alternative is this one: u+1361 ፡ ethiopic wordspace¹³.
Instead of ; (u+3b ; semicolon), you can use U+037E ; GREEK QUESTION MARK (see here).
For | (u+7c | vertical line), there are some good substitutes such as: U+2223 ∣ DIVIDES, U+0964 । DEVANAGARI DANDA, U+01C0 ǀ LATIN LETTER DENTAL CLICK (the last ones from Wikipedia) or U+2D4F ⵏ Tifinagh Letter Yan. Also the box drawing characters contain various other options.
Instead of , (, U+002C COMMA), you can use for example ‚ U+201A SINGLE LOW-9 QUOTATION MARK (see here).
For ? (U+003F ? QUESTION MARK), these are good candidates: U+FF1F ? FULLWIDTH QUESTION MARK or U+FE56 ﹖ SMALL QUESTION MARK (from here and here). There are also two more from the Dingbats Block (search for "question") and the u+203d ‽ interrobang¹³.
While my machine seems to accept it unchanged, I still want to include > (u+3e greater-than sign) and < (u+3c less-than sign) for the sake of completeness. The best replacement here is probably also from the quotation block, such as u+203a › single right-pointing angle quotation mark and u+2039 ‹ single left-pointing angle quotation mark respectively. The tifinagh block only contains ⵦ (u+2D66)¹³ to replace <. The last notion is ⋖ less-than with dot u+22D6 and ⋗ greater-than with dot u+22D7.
For additional ideas, you can also look for example into this block. You still want more ideas? You can try to draw your desired character and look at the suggestions here.
How do you type these characters
Say you want to type ⵏ (Tifinagh Letter Yan). To get all of its information, you can always search for this character (ⵏ) on a suited platform such as this Unicode Lookup (please add 0x when you search for hex) or that Unicode Table (that only allows to search for the name, in this case "Tifinagh Letter Yan"). You should obtain its Unicode number U+2D4F and the HTML-code ⵏ (note that 2D4F is hexadecimal for 11599). With this knowledge, you have several options to produce these special characters including the use of
code points to unicode converter or again the Unicode Lookup to reversely convert the numerical representation into the unicode character (remember to set the code point base below to decimal or hexadecimal respectively)
a one-liner makro in Autohotkey: :?*:altpipe::{U+2D4F} to type ⵏ instead of the string altpipe - this is the way I input those special characters, my Autohotkey script can be shared if there is common interest
Alt Characters or alt-codes by pressing and holding alt, followed by the decimal number for the desired character (more info for example here, look at a table here or there). For the example, that would be Alt+11599. Be aware, that many programs do not fully support this windows feature for all of unicode (as of time writing). Microsoft Office is an exception where it usually works, some other OSes provide similar functionality. Typing these chars with Alt-combinations into MS Word is also the way Wally Brockway suggests in his answer¹³ that was already mentionted - if you don't want to transfer all the hexadecimal values to the decimal asc, you can find some of them there¹³.
in MS Office, you can also use ALT + X as described in this MS article to produce the chars
if you rarely need it, you can of course still just copy-paste the special character of your choice instead of typing it
For Windows you can check it using PowerShell
$PathInvalidChars = [System.IO.Path]::GetInvalidPathChars() #36 chars
To display UTF-8 codes you can convert
$enc = [system.Text.Encoding]::UTF8
$PathInvalidChars | foreach { $enc.GetBytes($_) }
$FileNameInvalidChars = [System.IO.Path]::GetInvalidFileNameChars() #41 chars
$FileOnlyInvalidChars = #(':', '*', '?', '\', '/') #5 chars - as a difference
For anyone looking for a regex:
const BLACKLIST = /[<>:"\/\\|?*]/g;
In Windows 10 (2019), the following characters are forbidden by an error when you try to type them:
A file name can't contain any of the following characters:
\ / : * ? " < > |
Here's a c# implementation for windows based on Christopher Oezbek's answer
It was made more complex by the containsFolder boolean, but hopefully covers everything
/// <summary>
/// This will replace invalid chars with underscores, there are also some reserved words that it adds underscore to
/// </summary>
/// <remarks>
/// https://stackoverflow.com/questions/1976007/what-characters-are-forbidden-in-windows-and-linux-directory-names
/// </remarks>
/// <param name="containsFolder">Pass in true if filename represents a folder\file (passing true will allow slash)</param>
public static string EscapeFilename_Windows(string filename, bool containsFolder = false)
{
StringBuilder builder = new StringBuilder(filename.Length + 12);
int index = 0;
// Allow colon if it's part of the drive letter
if (containsFolder)
{
Match match = Regex.Match(filename, #"^\s*[A-Z]:\\", RegexOptions.IgnoreCase);
if (match.Success)
{
builder.Append(match.Value);
index = match.Length;
}
}
// Character substitutions
for (int cntr = index; cntr < filename.Length; cntr++)
{
char c = filename[cntr];
switch (c)
{
case '\u0000':
case '\u0001':
case '\u0002':
case '\u0003':
case '\u0004':
case '\u0005':
case '\u0006':
case '\u0007':
case '\u0008':
case '\u0009':
case '\u000A':
case '\u000B':
case '\u000C':
case '\u000D':
case '\u000E':
case '\u000F':
case '\u0010':
case '\u0011':
case '\u0012':
case '\u0013':
case '\u0014':
case '\u0015':
case '\u0016':
case '\u0017':
case '\u0018':
case '\u0019':
case '\u001A':
case '\u001B':
case '\u001C':
case '\u001D':
case '\u001E':
case '\u001F':
case '<':
case '>':
case ':':
case '"':
case '/':
case '|':
case '?':
case '*':
builder.Append('_');
break;
case '\\':
builder.Append(containsFolder ? c : '_');
break;
default:
builder.Append(c);
break;
}
}
string built = builder.ToString();
if (built == "")
{
return "_";
}
if (built.EndsWith(" ") || built.EndsWith("."))
{
built = built.Substring(0, built.Length - 1) + "_";
}
// These are reserved names, in either the folder or file name, but they are fine if following a dot
// CON, PRN, AUX, NUL, COM0 .. COM9, LPT0 .. LPT9
builder = new StringBuilder(built.Length + 12);
index = 0;
foreach (Match match in Regex.Matches(built, #"(^|\\)\s*(?<bad>CON|PRN|AUX|NUL|COM\d|LPT\d)\s*(\.|\\|$)", RegexOptions.IgnoreCase))
{
Group group = match.Groups["bad"];
if (group.Index > index)
{
builder.Append(built.Substring(index, match.Index - index + 1));
}
builder.Append(group.Value);
builder.Append("_"); // putting an underscore after this keyword is enough to make it acceptable
index = group.Index + group.Length;
}
if (index == 0)
{
return built;
}
if (index < built.Length - 1)
{
builder.Append(built.Substring(index));
}
return builder.ToString();
}
Though the only illegal Unix chars might be / and NULL, although some consideration for command line interpretation should be included.
For example, while it might be legal to name a file 1>&2 or 2>&1 in Unix, file names such as this might be misinterpreted when used on a command line.
Similarly it might be possible to name a file $PATH, but when trying to access it from the command line, the shell will translate $PATH to its variable value.
The .NET Framework System.IO provides the following functions for invalid file system characters:
Path.GetInvalidFileNameChars
Path.GetInvalidPathChars
Those functions should return appropriate results depending on the platform the .NET runtime is running in. That said, the Remarks in the documentation pages for those functions say:
The array returned from this method is not guaranteed to contain the
complete set of characters that are invalid in file and directory
names. The full set of invalid characters can vary by file system.
I always assumed that banned characters in Windows filenames meant that all exotic characters would also be outlawed. The inability to use ?, / and : in particular irked me. One day I discovered that it was virtually only those chars which were banned. Other Unicode characters may be used. So the nearest Unicode characters to the banned ones I could find were identified and MS Word macros were made for them as Alt+?, Alt+: etc. Now I form the filename in Word, using the substitute chars, and copy it to the Windows filename. So far I have had no problems.
Here are the substitute chars (Alt + the decimal Unicode) :
⃰ ⇔ Alt8432
⁄ ⇔ Alt8260
⃥ ⇔ Alt8421
∣ ⇔ Alt8739
ⵦ ⇔ Alt11622
⮚ ⇔ Alt11162
‽ ⇔ Alt8253
፡ ⇔ Alt4961
‶ ⇔ Alt8246
″ ⇔ Alt8243
As a test I formed a filename using all of those chars and Windows accepted it.
This is good enough for me in Python:
def fix_filename(name, max_length=255):
"""
Replace invalid characters on Linux/Windows/MacOS with underscores.
List from https://stackoverflow.com/a/31976060/819417
Trailing spaces & periods are ignored on Windows.
>>> fix_filename(" COM1 ")
'_ COM1 _'
>>> fix_filename("COM10")
'COM10'
>>> fix_filename("COM1,")
'COM1,'
>>> fix_filename("COM1.txt")
'_.txt'
>>> all('_' == fix_filename(chr(i)) for i in list(range(32)))
True
"""
return re.sub(r'[/\\:|<>"?*\0-\x1f]|^(AUX|COM[1-9]|CON|LPT[1-9]|NUL|PRN)(?![^.])|^\s|[\s.]$', "_", name[:max_length], flags=re.IGNORECASE)
See also this outdated list for additional legacy stuff like = in FAT32.
As of 18/04/2017, no simple black or white list of characters and filenames is evident among the answers to this topic - and there are many replies.
The best suggestion I could come up with was to let the user name the file however he likes. Using an error handler when the application tries to save the file, catch any exceptions, assume the filename is to blame (obviously after making sure the save path was ok as well), and prompt the user for a new file name. For best results, place this checking procedure within a loop that continues until either the user gets it right or gives up. Worked best for me (at least in VBA).
In Unix shells, you can quote almost every character in single quotes '. Except the single quote itself, and you can't express control characters, because \ is not expanded. Accessing the single quote itself from within a quoted string is possible, because you can concatenate strings with single and double quotes, like 'I'"'"'m' which can be used to access a file called "I'm" (double quote also possible here).
So you should avoid all control characters, because they are too difficult to enter in the shell. The rest still is funny, especially files starting with a dash, because most commands read those as options unless you have two dashes -- before, or you specify them with ./, which also hides the starting -.
If you want to be nice, don't use any of the characters the shell and typical commands use as syntactical elements, sometimes position dependent, so e.g. you can still use -, but not as first character; same with ., you can use it as first character only when you mean it ("hidden file"). When you are mean, your file names are VT100 escape sequences ;-), so that an ls garbles the output.
When creating internet shortcuts in Windows, to create the file name, it skips illegal characters, except for forward slash, which is converted to minus.
I had the same need and was looking for recommendation or standard references and came across this thread. My current blacklist of characters that should be avoided in file and directory names are:
$CharactersInvalidForFileName = {
"pound" -> "#",
"left angle bracket" -> "<",
"dollar sign" -> "$",
"plus sign" -> "+",
"percent" -> "%",
"right angle bracket" -> ">",
"exclamation point" -> "!",
"backtick" -> "`",
"ampersand" -> "&",
"asterisk" -> "*",
"single quotes" -> "“",
"pipe" -> "|",
"left bracket" -> "{",
"question mark" -> "?",
"double quotes" -> "”",
"equal sign" -> "=",
"right bracket" -> "}",
"forward slash" -> "/",
"colon" -> ":",
"back slash" -> "\\",
"lank spaces" -> "b",
"at sign" -> "#"
};

What does a semicolon do?

I got a function online to help me with my current project and it had semicolons on some of the lines. I was wondering why? Is it to break the function?
def containsAny(self, strings=[]):
alphabet = 'abcdefghijklmnopqrstuvwxyz0123456789'
for string in strings:
for char in string:
if char in alphabet: return 1;
return 0;
The function I got online with little modification:
for string in strings:
for char in string:
if char in alphabet: return 1;
Is the above saying the following?
if char in alphabet:
return 1
break
The semicolon does nothing in the code you show.
I suspect this is someone who programs in another language (C, Java, ...) that requires semicolons at the end of statements and it's just a habit (happens to me sometimes too).
If you want to put several Python statements on the same line, you can use a semi-colon to separate them, see this Python Doc:
A suite is a group of statements controlled by a clause. A suite can
be one or more semicolon-separated simple statements on the same line
as the header, following the header’s colon, or it can be one or more
indented statements on subsequent lines
The semicolon here does not do anything. People who come from C/C++/Java/(many other language) backgrounds tend to use the semicolon out of habit.
In general the semicolon does nothing. But if you are using the Jupyter Notebook (depending on your version), you might get a figure plotted twice. The semicolon at the end of your plot command prevents this:
df.plot();
Programmers of C, C++, and Java are habituated of using a semicolon to tell the compiler that this is the end of a statement, but for Python this is not the case.
The reason is that in Python, newlines are an unambiguous way of separating code lines; this is by design, and the way this works has been thoroughly thought through. As a result, Python code is perfectly readable and unambiguous without any special end-of-statement markers (apart from the newline).
As other answers point out, the semicolon does nothing there. It's a separator (e.g. print 1;print 2). But it does not work like this: def func():print 1;print 2;;print'Defined!' (;; is a syntax error). Out of habit, people tend to use it (as it is required in languages such as C/Java...).

Emacs Python.el, Syntax Highlighting Quirks

I'm using python.el version 0.23.1 for Emacs right now. The syntax highlighting seems to be a bit off -- any variable name containing an underscore followed by a keyword will result in the keyword being highlighted. Example, "foo_list" will result in "list" being highlighted.
More for my own understanding of Emacs-Lisp than anything (it's not a big deal) how do I go about fixing that. Here is where I think the relevant code is in "python.el", lines 312-318. I suspect the problem is with the definition of "symbol-start" but I have no idea what that name refers to.
(defvar python-font-lock-keywords
;; Keywords
`(,(rx symbol-start
(or "and" "del" "from" "not" "while" "as" "elif" "global" "or" "with"
"assert" "else" "if" "pass" "yield" "break" "except" "import"
"print" "class" "exec" "in" "raise" "continue" "finally" "is"
"return" "def" "for" "lambda" "try" "self")
symbol-end)
One thing I've struggled with in Emacs-Lisp so far is that I am finding it difficult in situations like these to follow names back to their definitions.
Thanks in advance!
When you say you're using python-mode 0.23.1, do you mean the one that comes bundled with Emacs or this one: http://launchpad.net/python-mode ? (which seems to be on version 6.something)
The reason I'm asking is that I can't reproduce what you're seeing. In an empty python buffer, I inserted
def x ():
a_list =3
and "list" is only highlighted when I delete the "a_". I'm using the version bundled with Emacs, with a snapshot version of Emacs so this might be the difference?
Incidentally, the font lock rule that you quote looks right to me: maybe the problem is that in your version "_" isn't set to have symbol syntax? You can check by typing
M-: (string (char-syntax ?_))
when in a python buffer. You should get "_" (which means symbol). If you get "." (punctuation) or something else weird, that probably explains what's gone wrong.

How to fold long docstrings in python source code in VIM?

Does anybody know of a method, or perhaps a plugin, that will
automatically fold long docstrings in Python? I have docstrings in my
code that span several pages, so it is troublesome to keep paging
through them. The other tricky part is that there is embedded python
testing code in the docstrings, so that might make parsing them
difficult. Note that I only need to automatically fold the entire
docstring, regardless of what is in it.
This is a bit of a dirty hack, but you can go through the python syntax file (:sp $VIMRUNTIME/syntax/python.vim) and find all the syntax regions for triple-quoted strings (search for ''' and """) and add the fold keyword to the end of those statements. Then just set foldmethod=syntax for python files and the comments should be folded.
I'm not sure about a plugin or automation, but if you type zf/ you can then search for something and it will fold up to the next instance of it. So in a document like the following (where [] is the cursor):
def foo():
"""[]
Some long docstring
that takes up many
lines
"""
pass
Look at edit2 first for the updated search string!
If you use the command zf/"""[ENTER], it should fold everything from the current line (the beginning of the docstring) to the next occurrence of """ which should be the end of the docstring.
I know this isn't automation, but perhaps it will help in the interim, or lead you down the right path to automating it. See edit2 for a better search function, although I still don't know how to automate.
Hope this helps.
Edit: in a corollary, you can search for any docstring with /"""\_.\{-}""", although this will also return the code within the docstring. To search for a function definition followed by a docstring, you can use /def\_.\{-}"""\_.\{-}""", although this breaks on a def inside the docstring.
Edit2: Actually, some more playing with regexs led me to this: /def.\{-}):\_s*"""\_.\{-}""" which should find any function followed by a docstring. It searches for def followed by any characters, then ): followed by a newline and/or whitespace followed by """ followed by any number of lines than the next """, but always ensures the 2nd triple quote is the one immediately following the first.
In your .vimrc add:
" folding
set foldmethod=indent
This will auto-fold at every indentation, which, in python, translates to docstrings. It works VERY VERY nice. Give it a try. The above answer is correct, but requires a bunch of keystrokes (blah!)
I think I've found a slight improvement to the answer of #too_much_php by trying to figure out how to do the same thing without root access. Copying the syntax region definitions from $VIMRUNTIME/syntax/python.vim to ~/.vim/after/syntax/python.vim and editing them solved the problem for me. Here's my code for ~/.vim/after/syntax/python.vim:
syn region docString1
\ start=+[uU]\=\z('''\|"""\)+ end="\z1" keepend transparent fold
syn region docString2
\ start=+[uU]\=[rR]\z('''\|"""\)+ end="\z1" keepend transparent fold
Now I can run :set foldmethod=syntax to fold docstrings.
PS: Credits to #too_much_php for the initial idea
You can do this with :set foldmethod=marker foldmarker=""",""", I think. I haven't tested it, but that should do the trick. The arguments to foldmarker are start and end markers.
I wrote a vimscript plugin many years ago to do exactly this, but never got around to publishing it. I've put it up as a gist, but have also pasted the source below. I've also replicated its functionality for Sublime Text here.
"
" Fold multi-line Python comments into one line.
"
" Also maps the "-" key to toggle expansion and <C-f> to toggle all folding.
"
setlocal foldmethod=syntax
setlocal foldtext=FoldText()
setlocal fillchars=
map <buffer> - za
map <buffer> <C-f> :call ToggleFold()<CR>
let b:folded = 1
hi Folded gui=bold cterm=bold guifg=cyan ctermfg=cyan guibg=NONE ctermbg=NONE
function! ToggleFold()
if b:folded == 0
exec "normal! zM"
let b:folded = 1
else
exec "normal! zR"
let b:folded = 0
endif
endfunction
function! s:Strip(string)
return substitute(a:string, '^[[:space:][:return:][:cntrl:]]\+\|[[:space:][:return:][:cntrl:]]\+$', '', '')
endfunction
" Extract the first line of a multi-line comment to use as the fold snippet
function! FoldText()
let l:snippet = getline(v:foldstart)
if len(s:Strip(l:snippet)) == 3
let l:snippet = strpart(l:snippet, 1) . s:Strip(getline(v:foldstart + 1))
endif
return '+' . l:snippet . ' ...'
endfunction
My solution involves using SimpylFold. After installing it using Vundle, I put this in my vimrc file:
autocmd FileType python setlocal foldlevel=2
Which makes all docstrings folded by default when you open a python file, which is super awesome.

Why allow concatenation of string literals?

I was recently bitten by a subtle bug.
char ** int2str = {
"zero", // 0
"one", // 1
"two" // 2
"three",// 3
nullptr };
assert( int2str[1] == std::string("one") ); // passes
assert( int2str[2] == std::string("two") ); // fails
If you have godlike code review powers you'll notice I forgot the , after "two".
After the considerable effort to find that bug I've got to ask why would anyone ever want this behavior?
I can see how this might be useful for macro magic, but then why is this a "feature" in a modern language like python?
Have you ever used string literal concatenation in production code?
Sure, it's the easy way to make your code look good:
char *someGlobalString = "very long "
"so broken "
"onto multiple "
"lines";
The best reason, though, is for weird printf formats, like type forcing:
uint64_t num = 5;
printf("Here is a number: %"PRIX64", what do you think of that?", num);
There are a bunch of those defined, and they can come in handy if you have type size requirements. Check them all out at this link. A few examples:
PRIo8 PRIoLEAST16 PRIoFAST32 PRIoMAX PRIoPTR
It's a great feature that allows you to combine preprocessor strings with your strings.
// Here we define the correct printf modifier for time_t
#ifdef TIME_T_LONG
#define TIME_T_MOD "l"
#elif defined(TIME_T_LONG_LONG)
#define TIME_T_MOD "ll"
#else
#define TIME_T_MOD ""
#endif
// And he we merge the modifier into the rest of our format string
printf("time is %" TIME_T_MOD "u\n", time(0));
I see several C and C++ answers but none of the really answer why or really what was the rationale for this feature? In C++ this is feature comes from C99 and we can find the rationale for this feature by going to Rationale for International Standard—Programming Languages—C section 6.4.5 String literals which says (emphasis mine):
A string can be continued across multiple lines by using the backslash–newline line continuation, but this requires that the continuation of the string start in the first position of the next line. To permit more flexible layout, and to solve some preprocessing problems (see §6.10.3), the C89 Committee introduced string literal concatenation. Two string literals in a row are pasted together, with no null character in the middle, to make one combined string literal. This addition to the C language allows a programmer to extend a string literal beyond the end of a physical line without having to use the backslash–newline mechanism and thereby destroying the indentation scheme of the program. An explicit concatenation operator was not introduced because the concatenation is a lexical construct rather than a run-time operation.
Python which seems to have the same reason, this reduces the need for ugly \ to continue long string literals. Which is covered in section 2.4.2 String literal concatenation of the
The Python Language Reference.
Cases where this can be useful:
Generating strings including components defined by the preprocessor (this is perhaps the largest use case in C, and it's one I see very, very frequently).
Splitting string constants over multiple lines
To provide a more concrete example for the former:
// in version.h
#define MYPROG_NAME "FOO"
#define MYPROG_VERSION "0.1.2"
// in main.c
puts("Welcome to " MYPROG_NAME " version " MYPROG_VERSION ".");
I'm not sure about other programming languages, but for example C# doesn't allow you to do this (and I think this is a good thing). As far as I can tell, most of the examples that show why this is useful in C++ would still work if you could use some special operator for string concatenation:
string someGlobalString = "very long " +
"so broken " +
"onto multiple " +
"lines";
This may not be as comfortable, but it is certainly safer. In your motivating example, the code would be invalid unless you added either , to separate elements or + to concatenate strings...
From the python lexical analysis reference, section 2.4.2:
This feature can be used to reduce the
number of backslashes needed, to split
long strings conveniently across long
lines, or even to add comments to
parts of strings
http://docs.python.org/reference/lexical_analysis.html
For rationale, expanding and simplifying Shafik Yaghmour’s answer: string literal concatenation originated in C (hence inherited by C++), as did the term, for two reasons (references are from Rationale for the ANSI C Programming Language):
For formatting: to allow long string literals to span multiple lines with proper indentation – in contrast to line continuation, which destroys the indentation scheme (3.1.4 String literals); and
For macro magic: to allow the construction of string literals by macros (via stringizing) (3.8.3.2 The # operator).
It is included in the modern languages Python and D because they copied it from C, though in both of these it has been proposed for deprecation, as it is bug-prone (as you note) and unnecessary (since one can just have a concatenation operator and constant folding for compile-time evaluation; you can’t do this in C because strings are pointers, and so you can’t add them).
It’s not simple to remove because that breaks compatibility, and you have to be careful about precedence (implicit concatenation happens during lexing, prior to operators, but replacing this with an operator means you need to be careful about precedence), hence why it’s still present.
Yes, it is in used production code. Google Python Style Guide: Line length specifies:
When a literal string won't fit on a single line, use parentheses for implicit line joining.
x = ('This will build a very long long '
'long long long long long long string')
See “String literal concatenation” at Wikipedia for more details and references.
So that you can split long string literals across lines.
And yes, I've seen it in production code.
While people have taken the words out of my mouth about the practical uses of the feature, nobody has so far tried to defend the choice of syntax.
For all I know, the typo that can slip through as a result was probably just overlooked. After all, it seems robustness against typos wasn't at the front of Dennis's mind, as shown further by:
if (a = b);
{
printf("%d", a);
}
Furthermore, there's the possible view that it wasn't worth using up an extra symbol for concatenation of string literals - after all, there isn't much else that can be done with two of them, and having a symbol there might create temptation to try to use it for runtime string concatenation, which is above the level of C's built-in features.
Some modern, higher-level languages based on C syntax have discarded this notation presumably because it is typo-prone. But these languages have an operator for string concatenation, such as + (JS, C#), . (Perl, PHP), ~ (D, though this has also kept C's juxtaposition syntax), and constant folding (in compiled languages, anyway) means that there is no runtime performance overhead.
Another sneaky error I've seen in the wild is people presuming that two single quotes are a way to escape the quote (as it is commonly used for double quotes in CSV files, for example), so they'll write things like the following in python:
print('Beggars can''t be choosers')
which outputs Beggars cant be choosers instead of the Beggars can't be choosers the coder desired.
As for the original "why" question: why is this a "feature" in a modern language like python? - in my opinion, I concur with the OP, it shouldn't be.

Categories