Fix a Python 2 crash

This commit is contained in:
Kodi Arfer 2019-01-20 11:59:49 -05:00
parent ada524e887
commit f1e693c96b
2 changed files with 7 additions and 0 deletions

View File

@ -35,6 +35,8 @@ Bug Fixes
module docstrings caused by automatic importing of Hy builtins.
* Fixed crash in `mangle` for some pathological inputs
* Fixed incorrect mangling of some characters at low code points
* Fixed a crash on certain versions of Python 2 due to changes
in the standard module `tokenize`
0.15.0
==============================

View File

@ -47,6 +47,11 @@ def isidentifier(x):
tokens = list(T.generate_tokens(StringIO(x).readline))
except (T.TokenError, IndentationError):
return False
# Some versions of Python 2.7 (including one that made it into
# Ubuntu 18.10) have a Python 3 backport that adds a NEWLINE
# token. Remove it if it's present.
# https://bugs.python.org/issue33899
tokens = [t for t in tokens if t[0] != T.NEWLINE]
return len(tokens) == 2 and tokens[0][0] == T.NAME
try: