diff --git a/NEWS.rst b/NEWS.rst index a1b503f..c8b5923 100644 --- a/NEWS.rst +++ b/NEWS.rst @@ -5,10 +5,20 @@ Unreleased Other Breaking Changes ------------------------------ +* Mangling rules have been overhauled, such that mangled names + are always legal Python identifiers +* `_` and `-` are now equivalent even as single-character names + + * The REPL history variable `_` is now `*1` + * Non-shadow unary `=`, `is`, `<`, etc. now evaluate their argument instead of ignoring it. This change increases consistency a bit and makes accidental unary uses easier to notice. +New Features +------------------------------ +* Added `mangle` and `unmangle` as core functions + Bug Fixes ------------------------------ * Fix `(return)` so it works correctly to exit a Python 2 generator diff --git a/docs/language/api.rst b/docs/language/api.rst index 2c4a0ac..1fe0f83 100644 --- a/docs/language/api.rst +++ b/docs/language/api.rst @@ -1,142 +1,6 @@ ================= -Hy (the language) -================= - -.. warning:: - This is incomplete; please consider contributing to the documentation - effort. - - -Theory of Hy -============ - -Hy maintains, over everything else, 100% compatibility in both directions -with Python itself. All Hy code follows a few simple rules. Memorize -this, as it's going to come in handy. - -These rules help ensure that Hy code is idiomatic and interfaceable in both -languages. - - - * Symbols in earmuffs will be translated to the upper-cased version of that - string. For example, ``foo`` will become ``FOO``. - - * UTF-8 entities will be encoded using - `punycode `_ and prefixed with - ``hy_``. For instance, ``⚘`` will become ``hy_w7h``, ``♥`` will become - ``hy_g6h``, and ``i♥u`` will become ``hy_iu_t0x``. - - * Symbols that contain dashes will have them replaced with underscores. For - example, ``render-template`` will become ``render_template``. This means - that symbols with dashes will shadow their underscore equivalents, and vice - versa. - -Notes on Syntax -=============== - -numeric literals ----------------- - -In addition to regular numbers, standard notation from Python 3 for non-base 10 -integers is used. ``0x`` for Hex, ``0o`` for Octal, ``0b`` for Binary. - -.. code-block:: clj - - (print 0x80 0b11101 0o102 30) - -Underscores and commas can appear anywhere in a numeric literal except the very -beginning. They have no effect on the value of the literal, but they're useful -for visually separating digits. - -.. code-block:: clj - - (print 10,000,000,000 10_000_000_000) - -Unlike Python, Hy provides literal forms for NaN and infinity: ``NaN``, -``Inf``, and ``-Inf``. - -string literals ---------------- - -Hy allows double-quoted strings (e.g., ``"hello"``), but not single-quoted -strings like Python. The single-quote character ``'`` is reserved for -preventing the evaluation of a form (e.g., ``'(+ 1 1)``), as in most Lisps. - -Python's so-called triple-quoted strings (e.g., ``'''hello'''`` and -``"""hello"""``) aren't supported. However, in Hy, unlike Python, any string -literal can contain newlines. Furthermore, Hy supports an alternative form of -string literal called a "bracket string" similar to Lua's long brackets. -Bracket strings have customizable delimiters, like the here-documents of other -languages. A bracket string begins with ``#[FOO[`` and ends with ``]FOO]``, -where ``FOO`` is any string not containing ``[`` or ``]``, including the empty -string. For example:: - - => (print #[["That's very kind of yuo [sic]" Tom wrote back.]]) - "That's very kind of yuo [sic]" Tom wrote back. - => (print #[==[1 + 1 = 2]==]) - 1 + 1 = 2 - -A bracket string can contain newlines, but if it begins with one, the newline -is removed, so you can begin the content of a bracket string on the line -following the opening delimiter with no effect on the content. Any leading -newlines past the first are preserved. - -Plain string literals support :ref:`a variety of backslash escapes -`. To create a "raw string" that interprets all backslashes -literally, prefix the string with ``r``, as in ``r"slash\not"``. Bracket -strings are always raw strings and don't allow the ``r`` prefix. - -Whether running under Python 2 or Python 3, Hy treats all string literals as -sequences of Unicode characters by default, and allows you to prefix a plain -string literal (but not a bracket string) with ``b`` to treat it as a sequence -of bytes. So when running under Python 3, Hy translates ``"foo"`` and -``b"foo"`` to the identical Python code, but when running under Python 2, -``"foo"`` is translated to ``u"foo"`` and ``b"foo"`` is translated to -``"foo"``. - -.. _syntax-keywords: - -keywords --------- - -An identifier headed by a colon, such as ``:foo``, is a keyword. Keywords -evaluate to a string preceded by the Unicode non-character code point U+FDD0, -like ``"\ufdd0:foo"``, so ``:foo`` and ``":foo"`` aren't equal. However, if a -literal keyword appears in a function call, it's used to indicate a keyword -argument rather than passed in as a value. For example, ``(f :foo 3)`` calls -the function ``f`` with the keyword argument named ``foo`` set to ``3``. Hence, -trying to call a function on a literal keyword may fail: ``(f :foo)`` yields -the error ``Keyword argument :foo needs a value``. To avoid this, you can quote -the keyword, as in ``(f ':foo)``, or use it as the value of another keyword -argument, as in ``(f :arg :foo)``. - -discard prefix --------------- - -Hy supports the Extensible Data Notation discard prefix, like Clojure. -Any form prefixed with ``#_`` is discarded instead of compiled. -This completely removes the form so it doesn't evaluate to anything, -not even None. -It's often more useful than linewise comments for commenting out a -form, because it respects code structure even when part of another -form is on the same line. For example: - -.. code-block:: clj - - => (print "Hy" "cruel" "World!") - Hy cruel World! - => (print "Hy" #_"cruel" "World!") - Hy World! - => (+ 1 1 (print "Math is hard!")) - Math is hard! - Traceback (most recent call last): - ... - TypeError: unsupported operand type(s) for +: 'int' and 'NoneType' - => (+ 1 1 #_(print "Math is hard!")) - 2 - Built-Ins -========= +================= Hy features a number of special forms that are used to help generate correct Python AST. The following are "special" forms, which may have diff --git a/docs/language/core.rst b/docs/language/core.rst index 8838153..3b089ea 100644 --- a/docs/language/core.rst +++ b/docs/language/core.rst @@ -699,6 +699,20 @@ Returns the single step macro expansion of *form*. HySymbol('e'), HySymbol('f')])]) +.. _mangle-fn: + +mangle +------ + +Usage: ``(mangle x)`` + +Stringify the input and translate it according to :ref:`Hy's mangling rules +`. + +.. code-block:: hylang + + => (mangle "foo-bar") + 'foo_bar' .. _merge-with-fn: @@ -1431,6 +1445,22 @@ Returns an iterator from *coll* as long as *pred* returns ``True``. => (list (take-while neg? [ 1 2 3 -4 5])) [] +.. _unmangle-fn: + +unmangle +-------- + +Usage: ``(unmangle x)`` + +Stringify the input and return a string that would :ref:`mangle ` to +it. Note that this isn't a one-to-one operation, and nor is ``mangle``, so +``mangle`` and ``unmangle`` don't always round-trip. + +.. code-block:: hylang + + => (unmangle "foo_bar") + 'foo-bar' + Included itertools ================== diff --git a/docs/language/index.rst b/docs/language/index.rst index 672063a..1a73bd2 100644 --- a/docs/language/index.rst +++ b/docs/language/index.rst @@ -9,6 +9,7 @@ Contents: cli interop + syntax api core internals diff --git a/docs/language/internals.rst b/docs/language/internals.rst index 48a4985..155ab0a 100644 --- a/docs/language/internals.rst +++ b/docs/language/internals.rst @@ -157,17 +157,8 @@ HySymbol ``hy.models.HySymbol`` is the model used to represent symbols in the Hy language. It inherits :ref:`HyString`. -``HySymbol`` objects are mangled in the parsing phase, to help Python -interoperability: - - - Symbols surrounded by asterisks (``*``) are turned into uppercase; - - Dashes (``-``) are turned into underscores (``_``); - - One trailing question mark (``?``) is turned into a leading ``is_``. - -Caveat: as the mangling is done during the parsing phase, it is possible -to programmatically generate HySymbols that can't be generated with Hy -source code. Such a mechanism is used by :ref:`gensym` to generate -"uninterned" symbols. +Symbols are :ref:`mangled ` when they are compiled +to Python variable names. .. _hykeyword: @@ -340,7 +331,7 @@ Since they have no "value" to Python, this makes working in Hy hard, since doing something like ``(print (if True True False))`` is not just common, it's expected. -As a result, we auto-mangle things using a ``Result`` object, where we offer +As a result, we reconfigure things using a ``Result`` object, where we offer up any ``ast.stmt`` that need to get run, and a single ``ast.expr`` that can be used to get the value of whatever was just run. Hy does this by forcing assignment to things while running. @@ -352,11 +343,11 @@ As example, the Hy:: Will turn into:: if True: - _mangled_name_here = True + _temp_name_here = True else: - _mangled_name_here = False + _temp_name_here = False - print _mangled_name_here + print _temp_name_here OK, that was a bit of a lie, since we actually turn that statement diff --git a/docs/language/interop.rst b/docs/language/interop.rst index df34016..34d61ea 100644 --- a/docs/language/interop.rst +++ b/docs/language/interop.rst @@ -8,6 +8,12 @@ Hy <-> Python interop Despite being a Lisp, Hy aims to be fully compatible with Python. That means every Python module or package can be imported in Hy code, and vice versa. +:ref:`Mangling ` allows variable names to be spelled differently in +Hy and Python. For example, Python's ``str.format_map`` can be written +``str.format-map`` in Hy, and a Hy function named ``valid?`` would be called +``is_valid`` in Python. In Python, you can import Hy's core functions +``mangle`` and ``unmangle`` directly from the ``hy`` package. + Using Python from Hy ==================== @@ -27,41 +33,6 @@ You can use it in Hy: You can also import ``.pyc`` bytecode files, of course. -A quick note about mangling --------- - -In Python, snake_case is used by convention. Lisp dialects tend to use dashes -instead of underscores, so Hy does some magic to give you more pleasant names. - -In the same way, ``UPPERCASE_NAMES`` from Python can be used ``*with-earmuffs*`` -instead. - -You can use either the original names or the new ones. - -Imagine ``example.py``:: - - def function_with_a_long_name(): - print(42) - - FOO = "bar" - -Then, in Hy: - -.. code-block:: clj - - (import example) - (.function-with-a-long-name example) ; prints "42" - (.function_with_a_long_name example) ; also prints "42" - - (print (. example *foo*)) ; prints "bar" - (print (. example FOO)) ; also prints "bar" - -.. warning:: - Mangling isn’t that simple; there is more to discuss about it, yet it doesn’t - belong in this section. -.. TODO: link to mangling section, when it is done - - Using Hy from Python ==================== diff --git a/docs/language/syntax.rst b/docs/language/syntax.rst new file mode 100644 index 0000000..deed2cb --- /dev/null +++ b/docs/language/syntax.rst @@ -0,0 +1,156 @@ +============== +Syntax +============== + +identifiers +----------- + +An identifier consists of a nonempty sequence of Unicode characters that are not whitespace nor any of the following: ``( ) [ ] { } ' "``. Hy first tries to parse each identifier into a numeric literal, then into a keyword if that fails, and finally into a symbol if that fails. + +numeric literals +---------------- + +In addition to regular numbers, standard notation from Python 3 for non-base 10 +integers is used. ``0x`` for Hex, ``0o`` for Octal, ``0b`` for Binary. + +.. code-block:: clj + + (print 0x80 0b11101 0o102 30) + +Underscores and commas can appear anywhere in a numeric literal except the very +beginning. They have no effect on the value of the literal, but they're useful +for visually separating digits. + +.. code-block:: clj + + (print 10,000,000,000 10_000_000_000) + +Unlike Python, Hy provides literal forms for NaN and infinity: ``NaN``, +``Inf``, and ``-Inf``. + +string literals +--------------- + +Hy allows double-quoted strings (e.g., ``"hello"``), but not single-quoted +strings like Python. The single-quote character ``'`` is reserved for +preventing the evaluation of a form (e.g., ``'(+ 1 1)``), as in most Lisps. + +Python's so-called triple-quoted strings (e.g., ``'''hello'''`` and +``"""hello"""``) aren't supported. However, in Hy, unlike Python, any string +literal can contain newlines. Furthermore, Hy supports an alternative form of +string literal called a "bracket string" similar to Lua's long brackets. +Bracket strings have customizable delimiters, like the here-documents of other +languages. A bracket string begins with ``#[FOO[`` and ends with ``]FOO]``, +where ``FOO`` is any string not containing ``[`` or ``]``, including the empty +string. For example:: + + => (print #[["That's very kind of yuo [sic]" Tom wrote back.]]) + "That's very kind of yuo [sic]" Tom wrote back. + => (print #[==[1 + 1 = 2]==]) + 1 + 1 = 2 + +A bracket string can contain newlines, but if it begins with one, the newline +is removed, so you can begin the content of a bracket string on the line +following the opening delimiter with no effect on the content. Any leading +newlines past the first are preserved. + +Plain string literals support :ref:`a variety of backslash escapes +`. To create a "raw string" that interprets all backslashes +literally, prefix the string with ``r``, as in ``r"slash\not"``. Bracket +strings are always raw strings and don't allow the ``r`` prefix. + +Whether running under Python 2 or Python 3, Hy treats all string literals as +sequences of Unicode characters by default, and allows you to prefix a plain +string literal (but not a bracket string) with ``b`` to treat it as a sequence +of bytes. So when running under Python 3, Hy translates ``"foo"`` and +``b"foo"`` to the identical Python code, but when running under Python 2, +``"foo"`` is translated to ``u"foo"`` and ``b"foo"`` is translated to +``"foo"``. + +.. _syntax-keywords: + +keywords +-------- + +An identifier headed by a colon, such as ``:foo``, is a keyword. Keywords +evaluate to a string preceded by the Unicode non-character code point U+FDD0, +like ``"\ufdd0:foo"``, so ``:foo`` and ``":foo"`` aren't equal. However, if a +literal keyword appears in a function call, it's used to indicate a keyword +argument rather than passed in as a value. For example, ``(f :foo 3)`` calls +the function ``f`` with the keyword argument named ``foo`` set to ``3``. Hence, +trying to call a function on a literal keyword may fail: ``(f :foo)`` yields +the error ``Keyword argument :foo needs a value``. To avoid this, you can quote +the keyword, as in ``(f ':foo)``, or use it as the value of another keyword +argument, as in ``(f :arg :foo)``. + +.. _mangling: + +symbols +------- + +Symbols are identifiers that are neither legal numeric literals nor legal +keywords. In most contexts, symbols are compiled to Python variable names. Some +example symbols are ``hello``, ``+++``, ``3fiddy``, ``$40``, ``just✈wrong``, +and ``🦑``. + +Since the rules for Hy symbols are much more permissive than the rules for +Python identifiers, Hy uses a mangling algorithm to convert its own names to +Python-legal names. The rules are: + +- Convert all hyphens (``-``) to underscores (``_``). Thus, ``foo-bar`` becomes + ``foo_bar``. +- If the name ends with ``?``, remove it and prepend ``is``. Thus, ``tasty?`` + becomes ``is_tasty``. +- If the name still isn't Python-legal, make the following changes. A name + could be Python-illegal because it contains a character that's never legal in + a Python name, it contains a character that's illegal in that position, or + it's equal to a Python reserved word. + + - Prepend ``hyx_`` to the name. + - Replace each illegal character with ``ΔfooΔ`` (or on Python 2, ``XfooX``), + where ``foo`` is the the Unicode character name in lowercase, with spaces + replaced by underscores and hyphens replaced by ``H``. Replace ``Δ`` itself + (or on Python 2, ``X``) the same way. If the character doesn't have a name, + use ``U`` followed by its code point in lowercase hexadecimal. + + Thus, ``green☘`` becomes ``hyx_greenΔshamrockΔ`` and ``if`` becomes + ``hyx_if``. + +- Finally, any added ``hyx_`` or ``is_`` is added after any leading + underscores, because leading underscores have special significance to Python. + Thus, ``_tasty?`` becomes ``_is_tasty`` instead of ``is__tasty``. + +Mangling isn't something you should have to think about often, but you may see +mangled names in error messages, the output of ``hy2py``, etc. A catch to be +aware of is that mangling, as well as the inverse "unmangling" operation +offered by the ``unmangle`` function, isn't one-to-one. Two different symbols +can mangle to the same string and hence compile to the same Python variable. +The chief practical consequence of this is that ``-`` and ``_`` are +interchangeable in all symbol names, so you shouldn't assign to the +one-character name ``_`` , or else you'll interfere with certain uses of +subtraction. + +discard prefix +-------------- + +Hy supports the Extensible Data Notation discard prefix, like Clojure. +Any form prefixed with ``#_`` is discarded instead of compiled. +This completely removes the form so it doesn't evaluate to anything, +not even None. +It's often more useful than linewise comments for commenting out a +form, because it respects code structure even when part of another +form is on the same line. For example: + +.. code-block:: clj + + => (print "Hy" "cruel" "World!") + Hy cruel World! + => (print "Hy" #_"cruel" "World!") + Hy World! + => (+ 1 1 (print "Math is hard!")) + Math is hard! + Traceback (most recent call last): + ... + TypeError: unsupported operand type(s) for +: 'int' and 'NoneType' + => (+ 1 1 #_(print "Math is hard!")) + 2 diff --git a/hy/__init__.py b/hy/__init__.py index 6b986d1..42d3133 100644 --- a/hy/__init__.py +++ b/hy/__init__.py @@ -12,5 +12,5 @@ import hy.importer # NOQA # we import for side-effects. -from hy.core.language import read, read_str # NOQA +from hy.core.language import read, read_str, mangle, unmangle # NOQA from hy.importer import hy_eval as eval # NOQA diff --git a/hy/_compat.py b/hy/_compat.py index a22bb13..3c5b7d9 100644 --- a/hy/_compat.py +++ b/hy/_compat.py @@ -18,7 +18,7 @@ except ImportError: (x >> 8) & 0xff, (x >> 16) & 0xff, (x >> 24) & 0xff])) -import sys +import sys, keyword PY3 = sys.version_info[0] >= 3 PY35 = sys.version_info >= (3, 5) @@ -35,3 +35,24 @@ if PY3: else: def raise_empty(t, *args): raise t(*args) + +def isidentifier(x): + if x in ('True', 'False', 'None', 'print'): + # `print` is special-cased here because Python 2's + # keyword.iskeyword will count it as a keyword, but we + # use the __future__ feature print_function, which makes + # it a non-keyword. + return True + if keyword.iskeyword(x): + return False + if PY3: + return x.isidentifier() + if x.rstrip() != x: + return False + import tokenize as T + from io import StringIO + try: + tokens = list(T.generate_tokens(StringIO(x).readline)) + except T.TokenError: + return False + return len(tokens) == 2 and tokens[0][0] == T.NAME diff --git a/hy/cmdline.py b/hy/cmdline.py index a7126a9..75fe09a 100644 --- a/hy/cmdline.py +++ b/hy/cmdline.py @@ -16,7 +16,7 @@ import astor.code_gen import hy from hy.lex import LexException, PrematureEndOfInput -from hy.lex.parser import hy_symbol_mangle +from hy.lex.parser import mangle from hy.compiler import HyTypeError from hy.importer import (hy_eval, import_buffer_to_module, import_file_to_ast, import_file_to_hst, @@ -63,12 +63,12 @@ class HyREPL(code.InteractiveConsole): elif callable(output_fn): self.output_fn = output_fn else: - f = hy_symbol_mangle(output_fn) if "." in output_fn: - module, f = f.rsplit(".", 1) + parts = [mangle(x) for x in output_fn.split(".")] + module, f = '.'.join(parts[:-1]), parts[-1] self.output_fn = getattr(importlib.import_module(module), f) else: - self.output_fn = __builtins__[f] + self.output_fn = __builtins__[mangle(output_fn)] code.InteractiveConsole.__init__(self, locals=locals, filename=filename) @@ -112,8 +112,8 @@ class HyREPL(code.InteractiveConsole): if value is not None: # Make the last non-None value available to - # the user as `_`. - self.locals['_'] = value + # the user as `*1`. + self.locals[mangle("*1")] = value # Print the value. try: output = self.output_fn(value) diff --git a/hy/compiler.py b/hy/compiler.py index 1132b65..bd19739 100755 --- a/hy/compiler.py +++ b/hy/compiler.py @@ -8,7 +8,7 @@ from hy.models import (HyObject, HyExpression, HyKeyword, HyInteger, HyComplex, HyDict, HyCons, wrap_value) from hy.errors import HyCompileError, HyTypeError -from hy.lex.parser import hy_symbol_mangle +from hy.lex.parser import mangle import hy.macros from hy._compat import ( @@ -53,7 +53,7 @@ def load_stdlib(): import hy.core for module in hy.core.STDLIB: mod = importlib.import_module(module) - for e in mod.EXPORTS: + for e in map(ast_str, mod.EXPORTS): if getattr(mod, e) is not getattr(builtins, e, ''): # Don't bother putting a name in _stdlib if it # points to a builtin with the same name. This @@ -61,38 +61,17 @@ def load_stdlib(): _stdlib[e] = module -# True, False and None included here since they -# are assignable in Python 2.* but become -# keywords in Python 3.* -def _is_hy_builtin(name, module_name): - extras = ['True', 'False', 'None'] - if name in extras or keyword.iskeyword(name): - return True - # for non-Hy modules, check for pre-existing name in - # _compile_table - if not module_name.startswith("hy."): - return name in _compile_table - return False - - _compile_table = {} _decoratables = (ast.FunctionDef, ast.ClassDef) if PY35: _decoratables += (ast.AsyncFunctionDef,) -def ast_str(foobar): - if PY3: - return str(foobar) - - try: - return str(foobar) - except UnicodeEncodeError: - pass - - enc = codecs.getencoder('punycode') - foobar, _ = enc(foobar) - return "hy_%s" % (str(foobar).replace("-", "_")) +def ast_str(x, piecewise=False): + if piecewise: + return ".".join(ast_str(s) if s else "" for s in x.split(".")) + x = mangle(x) + return x if PY3 else x.encode('UTF8') def builds(*types, **kwargs): @@ -104,6 +83,8 @@ def builds(*types, **kwargs): def _dec(fn): for t in types: + if isinstance(t, string_types): + t = ast_str(t) _compile_table[t] = fn return fn return _dec @@ -379,7 +360,7 @@ def is_unpack(kind, x): return (isinstance(x, HyExpression) and len(x) > 0 and isinstance(x[0], HySymbol) - and x[0] == "unpack_" + kind) + and x[0] == "unpack-" + kind) def ends_with_else(expr): @@ -393,7 +374,6 @@ def ends_with_else(expr): class HyASTCompiler(object): def __init__(self, module_name): - self.allow_builtins = module_name.startswith("hy.core") self.anon_var_count = 0 self.imports = defaultdict(set) self.module_name = module_name @@ -437,6 +417,8 @@ class HyASTCompiler(object): return ret.stmts def compile_atom(self, atom_type, atom): + if isinstance(atom_type, string_types): + atom_type = ast_str(atom_type) if atom_type in _compile_table: # _compile_table[atom_type] is a method for compiling this # type of atom, so call it. If it has an extra parameter, @@ -525,10 +507,11 @@ class HyASTCompiler(object): compiled_value = self.compile(value) ret += compiled_value - # no unicode for py2 in ast names - keyword = str(expr[2:]) - if "-" in keyword and keyword != "-": - keyword = keyword.replace("-", "_") + keyword = expr[2:] + if not keyword: + raise HyTypeError(expr, "Can't call a function with the " + "empty keyword") + keyword = ast_str(keyword) keywords.append(asty.keyword( expr, arg=keyword, value=compiled_value.force_expr)) @@ -699,17 +682,17 @@ class HyASTCompiler(object): """ if level == 0: if isinstance(form, HyExpression): - if form and form[0] in ("unquote", "unquote_splice"): + if form and form[0] in ("unquote", "unquote-splice"): if len(form) != 2: raise HyTypeError(form, ("`%s' needs 1 argument, got %s" % form[0], len(form) - 1)) - return set(), form[1], (form[0] == "unquote_splice") + return set(), form[1], (form[0] == "unquote-splice") if isinstance(form, HyExpression): if form and form[0] == "quasiquote": level += 1 - if form and form[0] in ("unquote", "unquote_splice"): + if form and form[0] in ("unquote", "unquote-splice"): level -= 1 name = form.__class__.__name__ @@ -783,12 +766,12 @@ class HyASTCompiler(object): ret.add_imports("hy", imports) return ret - @builds("unquote", "unquote_splicing") + @builds("unquote", "unquote-splicing") def compile_unquote(self, expr): raise HyTypeError(expr, "`%s' can't be used at the top-level" % expr[0]) - @builds("unpack_iterable") + @builds("unpack-iterable") @checkargs(exact=1) def compile_unpack_iterable(self, expr): if not PY3: @@ -797,7 +780,7 @@ class HyASTCompiler(object): ret += asty.Starred(expr, value=ret.force_expr, ctx=ast.Load()) return ret - @builds("unpack_mapping") + @builds("unpack-mapping") @checkargs(exact=1) def compile_unpack_mapping(self, expr): raise HyTypeError(expr, "`unpack-mapping` isn't allowed here") @@ -1143,12 +1126,12 @@ class HyASTCompiler(object): ret += self.compile(expr[1]) return ret + asty.Yield(expr, value=ret.force_expr) - @builds("yield_from", iff=PY3) + @builds("yield-from", iff=PY3) @builds("await", iff=PY35) @checkargs(1) def compile_yield_from_or_await_expression(self, expr): ret = Result() + self.compile(expr[1]) - node = asty.YieldFrom if expr[0] == "yield_from" else asty.Await + node = asty.YieldFrom if expr[0] == "yield-from" else asty.Await return ret + node(expr, value=ret.force_expr) @builds("import") @@ -1156,19 +1139,16 @@ class HyASTCompiler(object): expr = copy.deepcopy(expr) def _compile_import(expr, module, names=None, importer=asty.Import): if not names: - names = [ast.alias(name=ast_str(module), asname=None)] + names = [ast.alias(name=ast_str(module, piecewise=True), asname=None)] - ast_module = ast_str(module) + ast_module = ast_str(module, piecewise=True) module = ast_module.lstrip(".") level = len(ast_module) - len(module) if not module: module = None - ret = importer(expr, - module=module, - names=names, - level=level) - return Result() + ret + return Result() + importer( + expr, module=module, names=names, level=level) expr.pop(0) # index rimports = Result() @@ -1196,7 +1176,7 @@ class HyASTCompiler(object): "garbage after aliased import") iexpr.pop(0) # :as alias = iexpr.pop(0) - names = [ast.alias(name=ast_str(module), + names = [ast.alias(name=ast_str(module, piecewise=True), asname=ast_str(alias))] rimports += _compile_import(expr, ast_str(module), names) continue @@ -1210,7 +1190,7 @@ class HyASTCompiler(object): alias = ast_str(entry.pop(0)) else: alias = None - names.append(ast.alias(name=ast_str(sym), + names.append(ast.alias(name=(str(sym) if sym == "*" else ast_str(sym)), asname=alias)) rimports += _compile_import(expr, module, @@ -1307,7 +1287,7 @@ class HyASTCompiler(object): slice=ast.Slice(lower=nodes[1], upper=nodes[2], step=nodes[3]), ctx=ast.Load()) - @builds("with_decorator") + @builds("with-decorator") @checkargs(min=1) def compile_decorate_expression(self, expr): expr.pop(0) # with-decorator @@ -1403,7 +1383,7 @@ class HyASTCompiler(object): return gen_res + cond, gen - @builds("list_comp", "set_comp", "genexpr") + @builds("list-comp", "set-comp", "genexpr") @checkargs(min=2, max=3) def compile_comprehension(self, expr): # (list-comp expr (target iter) cond?) @@ -1421,13 +1401,13 @@ class HyASTCompiler(object): ret = self.compile(expression) node_class = ( - asty.ListComp if form == "list_comp" else - asty.SetComp if form == "set_comp" else + asty.ListComp if form == "list-comp" else + asty.SetComp if form == "set-comp" else asty.GeneratorExp) return ret + gen_res + node_class( expr, elt=ret.force_expr, generators=gen) - @builds("dict_comp") + @builds("dict-comp") @checkargs(min=3, max=4) def compile_dict_comprehension(self, expr): expr.pop(0) # dict-comp @@ -1554,15 +1534,16 @@ class HyASTCompiler(object): values=[value.force_expr for value in values]) return ret - def _compile_compare_op_expression(self, expression): - ops = {"=": ast.Eq, "!=": ast.NotEq, - "<": ast.Lt, "<=": ast.LtE, - ">": ast.Gt, ">=": ast.GtE, - "is": ast.Is, "is_not": ast.IsNot, - "in": ast.In, "not_in": ast.NotIn} + ops = {"=": ast.Eq, "!=": ast.NotEq, + "<": ast.Lt, "<=": ast.LtE, + ">": ast.Gt, ">=": ast.GtE, + "is": ast.Is, "is-not": ast.IsNot, + "in": ast.In, "not-in": ast.NotIn} + ops = {ast_str(k): v for k, v in ops.items()} - inv = expression.pop(0) - ops = [ops[inv]() for _ in range(len(expression) - 1)] + def _compile_compare_op_expression(self, expression): + inv = ast_str(expression.pop(0)) + ops = [self.ops[inv]() for _ in range(len(expression) - 1)] e = expression[0] exprs, ret, _ = self._compile_collect(expression) @@ -1578,12 +1559,12 @@ class HyASTCompiler(object): asty.Name(expression, id="True", ctx=ast.Load())) return self._compile_compare_op_expression(expression) - @builds("!=", "is_not") + @builds("!=", "is-not") @checkargs(min=2) def compile_compare_op_expression_coll(self, expression): return self._compile_compare_op_expression(expression) - @builds("in", "not_in") + @builds("in", "not-in") @checkargs(2) def compile_compare_op_expression_binary(self, expression): return self._compile_compare_op_expression(expression) @@ -1680,7 +1661,7 @@ class HyASTCompiler(object): def compile_maths_expression_sub(self, expression): return self._compile_maths_expression_additive(expression) - @builds("+=", "/=", "//=", "*=", "_=", "%=", "**=", "<<=", ">>=", "|=", + @builds("+=", "/=", "//=", "*=", "-=", "%=", "**=", "<<=", ">>=", "|=", "^=", "&=") @builds("@=", iff=PY35) @checkargs(2) @@ -1689,7 +1670,7 @@ class HyASTCompiler(object): "/=": ast.Div, "//=": ast.FloorDiv, "*=": ast.Mult, - "_=": ast.Sub, + "-=": ast.Sub, "%=": ast.Mod, "**=": ast.Pow, "<<=": ast.LShift, @@ -1732,7 +1713,7 @@ class HyASTCompiler(object): if isinstance(fn, HySymbol): # First check if `fn` is a special form, unless it has an - # `unpack_iterable` in it, since Python's operators (`+`, + # `unpack-iterable` in it, since Python's operators (`+`, # etc.) can't unpack. An exception to this exception is that # tuple literals (`,`) can unpack. if fn == "," or not ( @@ -1785,7 +1766,7 @@ class HyASTCompiler(object): # An exception for pulling together keyword args is if we're doing # a typecheck, eg (type :foo) with_kwargs = fn not in ( - "type", "HyKeyword", "keyword", "name", "is_keyword") + "type", "HyKeyword", "keyword", "name", "keyword?") args, ret, keywords, oldpy_star, oldpy_kw = self._compile_collect( expression[1:], with_kwargs, oldpy_unpack=True) @@ -1813,10 +1794,11 @@ class HyASTCompiler(object): def _compile_assign(self, name, result): str_name = "%s" % name - if (_is_hy_builtin(str_name, self.module_name) and - not self.allow_builtins): + if str_name in (["None"] + (["True", "False"] if PY3 else [])): + # Python 2 allows assigning to True and False, although + # this is rarely wise. raise HyTypeError(name, - "Can't assign to a builtin: `%s'" % str_name) + "Can't assign to `%s'" % str_name) result = self.compile(result) ld_name = self.compile(name) @@ -2057,7 +2039,7 @@ class HyASTCompiler(object): pairs = expr[1:] while len(pairs) > 0: k, v = (pairs.pop(0), pairs.pop(0)) - if k == HySymbol("__init__"): + if ast_str(k) == "__init__": v.append(HySymbol("None")) new_args.append(k) new_args.append(v) @@ -2092,8 +2074,6 @@ class HyASTCompiler(object): body += self._compile_assign(symb, docstring) body += body.expr_as_stmt() - allow_builtins = self.allow_builtins - self.allow_builtins = True if expressions and isinstance(expressions[0], HyList) \ and not isinstance(expressions[0], HyExpression): expr = expressions.pop(0) @@ -2105,8 +2085,6 @@ class HyASTCompiler(object): for expression in expressions: body += self.compile(rewire_init(macroexpand(expression, self))) - self.allow_builtins = allow_builtins - if not body.stmts: body += asty.Pass(expressions) @@ -2120,7 +2098,7 @@ class HyASTCompiler(object): bases=bases_expr, body=body.stmts) - @builds("dispatch_tag_macro") + @builds("dispatch-tag-macro") @checkargs(exact=2) def compile_dispatch_tag_macro(self, expression): expression.pop(0) # dispatch-tag-macro @@ -2131,11 +2109,11 @@ class HyASTCompiler(object): "Trying to expand a tag macro using `{0}' instead " "of string".format(type(tag).__name__), ) - tag = HyString(hy_symbol_mangle(str(tag))).replace(tag) + tag = HyString(mangle(tag)).replace(tag) expr = tag_macroexpand(tag, expression.pop(0), self) return self.compile(expr) - @builds("eval_and_compile", "eval_when_compile") + @builds("eval-and-compile", "eval-when-compile") def compile_eval_and_compile(self, expression, building): expression[0] = HySymbol("do") hy.importer.hy_eval(expression, @@ -2198,8 +2176,8 @@ class HyASTCompiler(object): attr=ast_str(local), ctx=ast.Load()) - if symbol in _stdlib: - self.imports[_stdlib[symbol]].add(symbol) + if ast_str(symbol) in _stdlib: + self.imports[_stdlib[ast_str(symbol)]].add(ast_str(symbol)) return asty.Name(symbol, id=ast_str(symbol), ctx=ast.Load()) diff --git a/hy/contrib/hy_repr.hy b/hy/contrib/hy_repr.hy index 627649a..ce2e83b 100644 --- a/hy/contrib/hy_repr.hy +++ b/hy/contrib/hy_repr.hy @@ -75,9 +75,9 @@ 'quote "'" 'quasiquote "`" 'unquote "~" - 'unquote_splice "~@" - 'unpack_iterable "#* " - 'unpack_mapping "#** "}) + 'unquote-splice "~@" + 'unpack-iterable "#* " + 'unpack-mapping "#** "}) (if (and x (symbol? (first x)) (in (first x) syntax)) (+ (get syntax (first x)) (hy-repr (second x))) (+ "(" (-cat x) ")")))) diff --git a/hy/contrib/walk.hy b/hy/contrib/walk.hy index d61bb62..7ce171d 100644 --- a/hy/contrib/walk.hy +++ b/hy/contrib/walk.hy @@ -257,7 +257,7 @@ Arguments without a header are under None. (= head 'defclass) (self.handle-defclass) (= head 'quasiquote) (self.+quote) ;; must be checked last! - (in head special-forms) (self.handle-special-form) + (in (mangle head) special-forms) (self.handle-special-form) ;; Not a special form. Traverse it like a coll (self.handle-coll))) diff --git a/hy/core/language.hy b/hy/core/language.hy index cda8d0a..75fe799 100644 --- a/hy/core/language.hy +++ b/hy/core/language.hy @@ -18,6 +18,7 @@ (import [hy._compat [long-type]]) ; long for python2, int for python3 (import [hy.models [HyCons HySymbol HyKeyword]]) (import [hy.lex [LexException PrematureEndOfInput tokenize]]) +(import [hy.lex.parser [mangle unmangle]]) (import [hy.compiler [HyASTCompiler spoof-positions]]) (import [hy.importer [hy-eval :as eval]]) @@ -87,7 +88,7 @@ If the second argument `codegen` is true, generate python code instead." "Return a generator from the original collection `coll` with no duplicates." (setv seen (set) citer (iter coll)) (for* [val citer] - (if (not_in val seen) + (if (not-in val seen) (do (yield val) (.add seen val))))) @@ -453,20 +454,16 @@ as EOF (defaults to an empty string)." "Reads and tokenizes first line of `input`." (read :from-file (StringIO input))) -(defn hyify [text] - "Convert `text` to match hy identifier." - (.replace (string text) "_" "-")) - (defn keyword [value] "Create a keyword from `value`. Strings numbers and even objects with the __name__ magic will work." (if (and (string? value) (value.startswith HyKeyword.PREFIX)) - (hyify value) + (unmangle value) (if (string? value) - (HyKeyword (+ ":" (hyify value))) + (HyKeyword (+ ":" (unmangle value))) (try - (hyify (.__name__ value)) + (unmangle (.__name__ value)) (except [] (HyKeyword (+ ":" (string value)))))))) (defn name [value] @@ -475,11 +472,11 @@ Strings numbers and even objects with the __name__ magic will work." Keyword special character will be stripped. String will be used as is. Even objects with the __name__ magic will work." (if (and (string? value) (value.startswith HyKeyword.PREFIX)) - (hyify (cut value 2)) + (unmangle (cut value 2)) (if (string? value) - (hyify value) + (unmangle value) (try - (hyify (. value __name__)) + (unmangle (. value __name__)) (except [] (string value)))))) (defn xor [a b] @@ -488,14 +485,14 @@ Even objects with the __name__ magic will work." False (or a b))) -(setv *exports* +(setv EXPORTS '[*map accumulate butlast calling-module-name chain coll? combinations comp complement compress cons cons? constantly count cycle dec distinct disassemble drop drop-last drop-while empty? eval even? every? exec first filter flatten float? fraction gensym group-by identity inc input instance? integer integer? integer-char? interleave interpose islice iterable? iterate iterator? juxt keyword keyword? last list* macroexpand - macroexpand-1 map merge-with multicombinations name neg? none? nth + macroexpand-1 mangle map merge-with multicombinations name neg? none? nth numeric? odd? partition permutations pos? product range read read-str remove repeat repeatedly rest reduce second some string string? symbol? - take take-nth take-while xor tee zero? zip zip-longest]) + take take-nth take-while unmangle xor tee zero? zip zip-longest]) diff --git a/hy/core/shadow.hy b/hy/core/shadow.hy index 7471edb..65acb46 100644 --- a/hy/core/shadow.hy +++ b/hy/core/shadow.hy @@ -163,7 +163,7 @@ (setv coll (get coll k))) coll) -(setv *exports* [ +(setv EXPORTS [ '+ '- '* '** '/ '// '% '@ '<< '>> '& '| '^ '~ '< '> '<= '>= '= '!= @@ -171,4 +171,4 @@ 'is 'is-not 'in 'not-in 'get]) (if (not PY35) - (.remove *exports* '@)) + (.remove EXPORTS '@)) diff --git a/hy/extra/reserved.hy b/hy/extra/reserved.hy index c245224..d7ae23c 100644 --- a/hy/extra/reserved.hy +++ b/hy/extra/reserved.hy @@ -13,10 +13,9 @@ The result of the first call is cached." (global _cache) (if (is _cache None) (do - (setv unmangle (. sys.modules ["hy.lex.parser"] hy_symbol_unmangle)) (setv _cache (frozenset (map unmangle (+ - hy.core.language.*exports* - hy.core.shadow.*exports* + hy.core.language.EXPORTS + hy.core.shadow.EXPORTS (list (.keys (get hy.macros._hy_macros None))) keyword.kwlist (list-comp k [k (.keys hy.compiler.-compile-table)] diff --git a/hy/lex/parser.py b/hy/lex/parser.py index c5e3c43..238f57a 100755 --- a/hy/lex/parser.py +++ b/hy/lex/parser.py @@ -1,3 +1,4 @@ +# -*- encoding: utf-8 -*- # Copyright 2018 the authors. # This file is part of Hy, which is free software licensed under the Expat # license. See the LICENSE. @@ -5,10 +6,11 @@ from __future__ import unicode_literals from functools import wraps +import string, re, unicodedata from rply import ParserGenerator -from hy._compat import str_type +from hy._compat import PY3, str_type, isidentifier from hy.models import (HyBytes, HyComplex, HyCons, HyDict, HyExpression, HyFloat, HyInteger, HyKeyword, HyList, HySet, HyString, HySymbol) @@ -21,43 +23,65 @@ pg = ParserGenerator( cache_id="hy_parser" ) +mangle_delim = 'Δ' if PY3 else 'X' -def hy_symbol_mangle(p): - if p.startswith("*") and p.endswith("*") and p not in ("*", "**"): - p = p[1:-1].upper() +def mangle(s): + """Stringify the argument and convert it to a valid Python identifier + according to Hy's mangling rules.""" - if "-" in p and p != "-": - p = p.replace("-", "_") + assert s - if p.endswith("?") and p != "?": - p = "is_%s" % (p[:-1]) + s = str_type(s) - if p.endswith("!") and p != "!": - p = "%s_bang" % (p[:-1]) + s = s.replace("-", "_") + s2 = s.lstrip('_') + leading_underscores = '_' * (len(s) - len(s2)) + s = s2 - return p + if s.endswith("?"): + s = 'is_' + s[:-1] + if not isidentifier(leading_underscores + s): + # Replace illegal characters with their Unicode character + # names, or hexadecimal if they don't have one. + s = 'hyx_' + ''.join( + c + if c != mangle_delim and isidentifier('S' + c) + # We prepend the "S" because some characters aren't + # allowed at the start of an identifier. + else '{0}{1}{0}'.format(mangle_delim, + unicodedata.name(c, '').lower().replace('-', 'H').replace(' ', '_') + or 'U{:x}'.format(ord(c))) + for c in s) + + s = leading_underscores + s + assert isidentifier(s) + return s -def hy_symbol_unmangle(p): - # hy_symbol_mangle is one-way, so this can't be perfect. - # But it can be useful till we have a way to get the original - # symbol (https://github.com/hylang/hy/issues/360). - p = str_type(p) +def unmangle(s): + """Stringify the argument and try to convert it to a pretty unmangled + form. This may not round-trip, because different Hy symbol names can + mangle to the same Python identifier.""" - if p.endswith("_bang") and p != "_bang": - p = p[:-len("_bang")] + "!" + s = str_type(s) - if p.startswith("is_") and p != "is_": - p = p[len("is_"):] + "?" + s2 = s.lstrip('_') + leading_underscores = len(s) - len(s2) + s = s2 - if "_" in p and p != "_": - p = p.replace("_", "-") + if s.startswith('hyx_'): + s = re.sub('{0}(U)?([_a-z0-9H]+?){0}'.format(mangle_delim), + lambda mo: + chr(int(mo.group(2), base=16)) + if mo.group(1) + else unicodedata.lookup( + mo.group(2).replace('_', ' ').replace('H', '-').upper()), + s[len('hyx_'):]) + if s.startswith('is_'): + s = s[len("is_"):] + "?" + s = s.replace('_', '-') - if (all([c.isalpha() and c.isupper() or c == '_' for c in p]) and - any([c.isalpha() for c in p])): - p = '*' + p.lower() + '*' - - return p + return '-' * leading_underscores + s def set_boundaries(fun): @@ -201,7 +225,7 @@ def term_unquote(p): @pg.production("term : UNQUOTESPLICE term") @set_quote_boundaries def term_unquote_splice(p): - return HyExpression([HySymbol("unquote_splice"), p[1]]) + return HyExpression([HySymbol("unquote-splice"), p[1]]) @pg.production("term : HASHSTARS term") @@ -209,9 +233,9 @@ def term_unquote_splice(p): def term_hashstars(p): n_stars = len(p[0].getstr()[1:]) if n_stars == 1: - sym = "unpack_iterable" + sym = "unpack-iterable" elif n_stars == 2: - sym = "unpack_mapping" + sym = "unpack-mapping" else: raise LexException( "Too many stars in `#*` construct (if you want to unpack a symbol " @@ -227,7 +251,7 @@ def hash_other(p): st = p[0].getstr()[1:] str_object = HyString(st) expr = p[1] - return HyExpression([HySymbol("dispatch_tag_macro"), str_object, expr]) + return HyExpression([HySymbol("dispatch-tag-macro"), str_object, expr]) @pg.production("set : HLCURLY list_contents RCURLY") @@ -307,7 +331,7 @@ def t_identifier(p): '`(. )` or `(. )`)', p[0].source_pos.lineno, p[0].source_pos.colno) - return HySymbol(".".join(hy_symbol_mangle(x) for x in obj.split("."))) + return HySymbol(obj) def symbol_like(obj): diff --git a/hy/macros.py b/hy/macros.py index 0613711..110c37d 100644 --- a/hy/macros.py +++ b/hy/macros.py @@ -2,8 +2,12 @@ # This file is part of Hy, which is free software licensed under the Expat # license. See the LICENSE. +from hy._compat import PY3 import hy.inspect from hy.models import replace_hy_obj, HyExpression, HySymbol +from hy.lex.parser import mangle +from hy._compat import str_type + from hy.errors import HyTypeError, HyMacroExpansionError from collections import defaultdict @@ -32,6 +36,7 @@ def macro(name): This function is called from the `defmacro` special form in the compiler. """ + name = mangle(name) def _(fn): fn.__name__ = '({})'.format(name) try: @@ -62,11 +67,14 @@ def tag(name): """ def _(fn): - fn.__name__ = '#{}'.format(name) + _name = mangle('#{}'.format(name)) + if not PY3: + _name = _name.encode('UTF-8') + fn.__name__ = _name module_name = fn.__module__ if module_name.startswith("hy.core"): module_name = None - _hy_tag[module_name][name] = fn + _hy_tag[module_name][mangle(name)] = fn return fn return _ @@ -89,14 +97,15 @@ def require(source_module, target_module, seen_names = set() if prefix: prefix += "." + assignments = {mangle(str_type(k)): v for k, v in assignments.items()} for d in _hy_macros, _hy_tag: for name, macro in d[source_module].items(): seen_names.add(name) if all_macros: - d[target_module][prefix + name] = macro + d[target_module][mangle(prefix + name)] = macro elif name in assignments: - d[target_module][prefix + assignments[name]] = macro + d[target_module][mangle(prefix + assignments[name])] = macro if not all_macros: unseen = frozenset(assignments.keys()).difference(seen_names) @@ -178,6 +187,7 @@ def macroexpand_1(tree, compiler): opts = {} if isinstance(fn, HySymbol): + fn = mangle(str_type(fn)) m = _hy_macros[compiler.module_name].get(fn) if m is None: m = _hy_macros[None].get(fn) diff --git a/hy/models.py b/hy/models.py index f071ba2..35ff55a 100644 --- a/hy/models.py +++ b/hy/models.py @@ -338,7 +338,7 @@ class HyCons(HyObject): # Keep unquotes in the cdr of conses if type(cdr) == HyExpression: if len(cdr) > 0 and type(cdr[0]) == HySymbol: - if cdr[0] in ("unquote", "unquote_splice"): + if cdr[0] in ("unquote", "unquote-splice"): return super(HyCons, cls).__new__(cls) return cdr.__class__([wrap_value(car)] + cdr) diff --git a/tests/compilers/test_ast.py b/tests/compilers/test_ast.py index 0ee88cc..ead657a 100644 --- a/tests/compilers/test_ast.py +++ b/tests/compilers/test_ast.py @@ -596,13 +596,11 @@ def test_invalid_list_comprehension(): def test_bad_setv(): """Ensure setv handles error cases""" - cant_compile("(setv if* 1)") cant_compile("(setv (a b) [1 2])") def test_defn(): """Ensure that defn works correctly in various corner cases""" - cant_compile("(defn if* [] 1)") cant_compile("(defn \"hy\" [] 1)") cant_compile("(defn :hy [] 1)") can_compile("(defn &hy [] 1)") @@ -611,7 +609,6 @@ def test_defn(): def test_setv_builtins(): """Ensure that assigning to a builtin fails, unless in a class""" cant_compile("(setv None 42)") - cant_compile("(defn get [&rest args] 42)") can_compile("(defclass A [] (defn get [self] 42))") can_compile(""" (defclass A [] diff --git a/tests/native_tests/language.hy b/tests/native_tests/language.hy index ea146c9..8f99b1d 100644 --- a/tests/native_tests/language.hy +++ b/tests/native_tests/language.hy @@ -65,19 +65,19 @@ (defn test-setv-builtin [] - "NATIVE: test that setv doesn't work on builtins" - (try (eval '(setv False 1)) - (except [e [TypeError]] (assert (in "Can't assign to a builtin" (str e))))) - (try (eval '(setv True 0)) - (except [e [TypeError]] (assert (in "Can't assign to a builtin" (str e))))) + "NATIVE: test that setv doesn't work on names Python can't assign to + and that we can't mangle" (try (eval '(setv None 1)) - (except [e [TypeError]] (assert (in "Can't assign to a builtin" (str e))))) - (try (eval '(defn defclass [] (print "hello"))) - (except [e [TypeError]] (assert (in "Can't assign to a builtin" (str e))))) - (try (eval '(defn get [] (print "hello"))) - (except [e [TypeError]] (assert (in "Can't assign to a builtin" (str e))))) - (try (eval '(defn fn [] (print "hello"))) - (except [e [TypeError]] (assert (in "Can't assign to a builtin" (str e)))))) + (except [e [TypeError]] (assert (in "Can't assign to" (str e))))) + (try (eval '(defn None [] (print "hello"))) + (except [e [TypeError]] (assert (in "Can't assign to" (str e))))) + (when PY3 + (try (eval '(setv False 1)) + (except [e [TypeError]] (assert (in "Can't assign to" (str e))))) + (try (eval '(setv True 0)) + (except [e [TypeError]] (assert (in "Can't assign to" (str e))))) + (try (eval '(defn True [] (print "hello"))) + (except [e [TypeError]] (assert (in "Can't assign to" (str e))))))) (defn test-setv-pairs [] @@ -187,14 +187,6 @@ (assert (in "takes a parameter list as second" (str e)))))) -(defn test-alias-names-in-errors [] - "NATIVE: tests that native aliases show the correct names in errors" - (try (eval '(list-comp 1 2 3 4)) - (except [e [Exception]] (assert (in "list_comp" (str e))))) - (try (eval '(set-comp 1 2 3 4)) - (except [e [Exception]] (assert (in "set_comp" (str e)))))) - - (defn test-for-loop [] "NATIVE: test for loops" (setv count1 0 count2 0) @@ -223,14 +215,14 @@ ; don't be fooled by constructs that look like else (setv s "") - (setv (get (globals) "else") True) + (setv else True) (for [x "abcde"] (+= s x) [else (+= s "_")]) (assert (= s "a_b_c_d_e_")) (setv s "") - (setv (get (globals) "else") True) + (setv else True) (with [(pytest.raises TypeError)] (for [x "abcde"] (+= s x) @@ -329,7 +321,7 @@ ; don't be fooled by constructs that look like else clauses (setv x 2) (setv a []) - (setv (get (globals) "else") True) + (setv else True) (while x (.append a x) (-= x 1) @@ -738,13 +730,6 @@ (assert (= x 2))) -(defn test-earmuffs [] - "NATIVE: Test earmuffs" - (setv *foo* "2") - (setv foo "3") - (assert (= *foo* FOO)) - (assert (!= *foo* foo))) - (defn test-threading [] "NATIVE: test threading macro" @@ -1112,27 +1097,6 @@ (assert (= ((fn [] (-> 2 (+ 1 1) (* 1 2)))) 8))) -(defn test-symbol-utf-8 [] - "NATIVE: test symbol encoded" - (setv ♥ "love" - ⚘ "flower") - (assert (= (+ ⚘ ♥) "flowerlove"))) - - -(defn test-symbol-dash [] - "NATIVE: test symbol encoded" - (setv ♥-♥ "doublelove" - -_- "what?") - (assert (= ♥-♥ "doublelove")) - (assert (= -_- "what?"))) - - -(defn test-symbol-question-mark [] - "NATIVE: test foo? -> is_foo behavior" - (setv foo? "nachos") - (assert (= is_foo "nachos"))) - - (defn test-and [] "NATIVE: test the and function" @@ -1260,11 +1224,7 @@ (assert (= : :)) (assert (keyword? :)) (assert (!= : ":")) - (assert (= (name :) "")) - - (defn f [&kwargs kwargs] - (list (.items kwargs))) - (assert (= (f : 3) [(, "" 3)]))) + (assert (= (name :) ""))) (defn test-nested-if [] @@ -1816,4 +1776,4 @@ macros() (defn test-relative-import [] "Make sure relative imports work properly" (import [..resources [tlib]]) - (assert (= tlib.*secret-message* "Hello World"))) + (assert (= tlib.SECRET-MESSAGE "Hello World"))) diff --git a/tests/native_tests/mangling.hy b/tests/native_tests/mangling.hy new file mode 100644 index 0000000..032e205 --- /dev/null +++ b/tests/native_tests/mangling.hy @@ -0,0 +1,189 @@ +;; Copyright 2018 the authors. +;; This file is part of Hy, which is free software licensed under the Expat +;; license. See the LICENSE. + + +(import [hy._compat [PY3]]) + + +(defn test-hyphen [] + (setv a-b 1) + (assert (= a-b 1)) + (assert (= a_b 1)) + (setv -a-_b- 2) + (assert (= -a-_b- 2)) + (assert (= -a--b- 2)) + (assert (= -a__b- 2)) + (setv -_- 3) + (assert (= -_- 3)) + (assert (= --- 3)) + (assert (= ___ 3))) + + +(defn test-underscore-number [] + (setv _42 3) + (assert (= _42 3)) + (assert (!= _42 -42)) + (assert (not (in "_hyx_42" (locals))))) + + +(defn test-question-mark [] + (setv foo? "nachos") + (assert (= foo? "nachos")) + (assert (= is_foo "nachos")) + (setv ___ab_cd? "tacos") + (assert (= ___ab_cd? "tacos")) + (assert (= ___is_ab_cd "tacos"))) + + +(defn test-py-forbidden-ascii [] + + (setv # "no comment") + (assert (= # "no comment")) + (if PY3 + (assert (= hyx_Δnumber_signΔ "no comment")) + (assert (= hyx_Xnumber_signX "no comment"))) + + (setv $ "dosh") + (assert (= $ "dosh")) + (if PY3 + (assert (= hyx_Δdollar_signΔ "dosh")) + (assert (= hyx_Xdollar_signX "dosh")))) + + +(defn test-basic-multilingual-plane [] + (setv ♥ "love" + ⚘ab "flower") + (assert (= (+ ⚘ab ♥) "flowerlove")) + (if PY3 + (assert (= (+ hyx_ΔflowerΔab hyx_Δblack_heart_suitΔ) "flowerlove")) + (assert (= (+ hyx_XflowerXab hyx_Xblack_heart_suitX) "flowerlove"))) + (setv ⚘-⚘ "doubleflower") + (assert (= ⚘-⚘ "doubleflower")) + (if PY3 + (assert (= hyx_ΔflowerΔ_ΔflowerΔ "doubleflower")) + (assert (= hyx_XflowerX_XflowerX "doubleflower"))) + (setv ⚘? "mystery") + (assert (= ⚘? "mystery")) + (if PY3 + (assert (= hyx_is_ΔflowerΔ "mystery")) + (assert (= hyx_is_XflowerX "mystery")))) + + +(defn test-higher-unicode [] + (setv 😂 "emoji") + (assert (= 😂 "emoji")) + (if PY3 + (assert (= hyx_Δface_with_tears_of_joyΔ "emoji")) + (assert (= hyx_XU1f602X "emoji")))) + + +(defn test-nameless-unicode [] + (setv  "private use") + (assert (=  "private use")) + (if PY3 + (assert (= hyx_ΔUe000Δ "private use")) + (assert (= hyx_XUe000X "private use")))) + + +(defn test-charname-with-hyphen [] + (setv a