Merge pull request #1517 from Kodiologist/mangling-makeover

Mangling makeover

NEWS.rst

@@ -5,10 +5,20 @@ Unreleased
 
 Other Breaking Changes
 ------------------------------
+* Mangling rules have been overhauled, such that mangled names
+  are always legal Python identifiers
+* `_` and `-` are now equivalent even as single-character names
+
+  * The REPL history variable `_` is now `*1`
+
 * Non-shadow unary `=`, `is`, `<`, etc. now evaluate their argument
   instead of ignoring it. This change increases consistency a bit
   and makes accidental unary uses easier to notice.
 
+New Features
+------------------------------
+* Added `mangle` and `unmangle` as core functions
+
 Bug Fixes
 ------------------------------
 * Fix `(return)` so it works correctly to exit a Python 2 generator

docs/language/api.rst

@@ -1,142 +1,6 @@
 =================
-Hy (the language)
-=================
-
-.. warning::
-    This is incomplete; please consider contributing to the documentation
-    effort.
-
-
-Theory of Hy
-============
-
-Hy maintains, over everything else, 100% compatibility in both directions
-with Python itself. All Hy code follows a few simple rules. Memorize
-this, as it's going to come in handy.
-
-These rules help ensure that Hy code is idiomatic and interfaceable in both
-languages.
-
-
-  * Symbols in earmuffs will be translated to the upper-cased version of that
-    string. For example, ``foo`` will become ``FOO``.
-
-  * UTF-8 entities will be encoded using
-    `punycode <https://en.wikipedia.org/wiki/Punycode>`_ and prefixed with
-    ``hy_``. For instance, ``⚘`` will become ``hy_w7h``, ``♥`` will become
-    ``hy_g6h``, and ``i♥u`` will become ``hy_iu_t0x``.
-
-  * Symbols that contain dashes will have them replaced with underscores. For
-    example, ``render-template`` will become ``render_template``. This means
-    that symbols with dashes will shadow their underscore equivalents, and vice
-    versa.
-
-Notes on Syntax
-===============
-
-numeric literals
-----------------
-
-In addition to regular numbers, standard notation from Python 3 for non-base 10
-integers is used. ``0x`` for Hex, ``0o`` for Octal, ``0b`` for Binary.
-
-.. code-block:: clj
-    
-    (print 0x80 0b11101 0o102 30)
-
-Underscores and commas can appear anywhere in a numeric literal except the very
-beginning. They have no effect on the value of the literal, but they're useful
-for visually separating digits.
-
-.. code-block:: clj
-
-    (print 10,000,000,000 10_000_000_000)
-
-Unlike Python, Hy provides literal forms for NaN and infinity: ``NaN``,
-``Inf``, and ``-Inf``.
-
-string literals
----------------
-
-Hy allows double-quoted strings (e.g., ``"hello"``), but not single-quoted
-strings like Python. The single-quote character ``'`` is reserved for
-preventing the evaluation of a form (e.g., ``'(+ 1 1)``), as in most Lisps.
-
-Python's so-called triple-quoted strings (e.g., ``'''hello'''`` and
-``"""hello"""``) aren't supported. However, in Hy, unlike Python, any string
-literal can contain newlines. Furthermore, Hy supports an alternative form of
-string literal called a "bracket string" similar to Lua's long brackets.
-Bracket strings have customizable delimiters, like the here-documents of other
-languages. A bracket string begins with ``#[FOO[`` and ends with ``]FOO]``,
-where ``FOO`` is any string not containing ``[`` or ``]``, including the empty
-string. For example::
-
-   => (print #[["That's very kind of yuo [sic]" Tom wrote back.]])
-   "That's very kind of yuo [sic]" Tom wrote back.
-   => (print #[==[1 + 1 = 2]==])
-   1 + 1 = 2
-
-A bracket string can contain newlines, but if it begins with one, the newline
-is removed, so you can begin the content of a bracket string on the line
-following the opening delimiter with no effect on the content. Any leading
-newlines past the first are preserved.
-
-Plain string literals support :ref:`a variety of backslash escapes
-<py:strings>`. To create a "raw string" that interprets all backslashes
-literally, prefix the string with ``r``, as in ``r"slash\not"``. Bracket
-strings are always raw strings and don't allow the ``r`` prefix.
-
-Whether running under Python 2 or Python 3, Hy treats all string literals as
-sequences of Unicode characters by default, and allows you to prefix a plain
-string literal (but not a bracket string) with ``b`` to treat it as a sequence
-of bytes. So when running under Python 3, Hy translates ``"foo"`` and
-``b"foo"`` to the identical Python code, but when running under Python 2,
-``"foo"`` is translated to ``u"foo"`` and ``b"foo"`` is translated to
-``"foo"``.
-
-.. _syntax-keywords:
-
-keywords
---------
-
-An identifier headed by a colon, such as ``:foo``, is a keyword. Keywords
-evaluate to a string preceded by the Unicode non-character code point U+FDD0,
-like ``"\ufdd0:foo"``, so ``:foo`` and ``":foo"`` aren't equal. However, if a
-literal keyword appears in a function call, it's used to indicate a keyword
-argument rather than passed in as a value. For example, ``(f :foo 3)`` calls
-the function ``f`` with the keyword argument named ``foo`` set to ``3``. Hence,
-trying to call a function on a literal keyword may fail: ``(f :foo)`` yields
-the error ``Keyword argument :foo needs a value``. To avoid this, you can quote
-the keyword, as in ``(f ':foo)``, or use it as the value of another keyword
-argument, as in ``(f :arg :foo)``.
-
-discard prefix
---------------
-
-Hy supports the Extensible Data Notation discard prefix, like Clojure.
-Any form prefixed with ``#_`` is discarded instead of compiled.
-This completely removes the form so it doesn't evaluate to anything,
-not even None.
-It's often more useful than linewise comments for commenting out a
-form, because it respects code structure even when part of another
-form is on the same line. For example:
-
-.. code-block:: clj
-
-   => (print "Hy" "cruel" "World!")
-   Hy cruel World!
-   => (print "Hy" #_"cruel" "World!")
-   Hy World!
-   => (+ 1 1 (print "Math is hard!"))
-   Math is hard!
-   Traceback (most recent call last):
-      ...
-   TypeError: unsupported operand type(s) for +: 'int' and 'NoneType'
-   => (+ 1 1 #_(print "Math is hard!"))
-   2
-
 Built-Ins
-=========
+=================
 
 Hy features a number of special forms that are used to help generate
 correct Python AST. The following are "special" forms, which may have

docs/language/core.rst

@@ -699,6 +699,20 @@ Returns the single step macro expansion of *form*.
         HySymbol('e'),
         HySymbol('f')])])
 
+.. _mangle-fn:
+
+mangle
+------
+
+Usage: ``(mangle x)``
+
+Stringify the input and translate it according to :ref:`Hy's mangling rules
+<mangling>`.
+
+.. code-block:: hylang
+
+    => (mangle "foo-bar")
+    'foo_bar'
 
 .. _merge-with-fn:
 
@@ -1431,6 +1445,22 @@ Returns an iterator from *coll* as long as *pred* returns ``True``.
    => (list (take-while neg? [ 1 2 3 -4 5]))
    []
 
+.. _unmangle-fn:
+
+unmangle
+--------
+
+Usage: ``(unmangle x)``
+
+Stringify the input and return a string that would :ref:`mangle <mangling>` to
+it. Note that this isn't a one-to-one operation, and nor is ``mangle``, so
+``mangle`` and ``unmangle`` don't always round-trip.
+
+.. code-block:: hylang
+
+    => (unmangle "foo_bar")
+    'foo-bar'
+
 Included itertools
 ==================
 

docs/language/index.rst

@@ -9,6 +9,7 @@ Contents:
 
    cli
    interop
+   syntax
    api
    core
    internals

docs/language/internals.rst

@@ -157,17 +157,8 @@ HySymbol
 ``hy.models.HySymbol`` is the model used to represent symbols
 in the Hy language. It inherits :ref:`HyString`.
 
-``HySymbol`` objects are mangled in the parsing phase, to help Python
+Symbols are :ref:`mangled <mangling>` when they are compiled
-interoperability:
+to Python variable names.
-
- - Symbols surrounded by asterisks (``*``) are turned into uppercase;
- - Dashes (``-``) are turned into underscores (``_``);
- - One trailing question mark (``?``) is turned into a leading ``is_``.
-
-Caveat: as the mangling is done during the parsing phase, it is possible
-to programmatically generate HySymbols that can't be generated with Hy
-source code. Such a mechanism is used by :ref:`gensym` to generate
-"uninterned" symbols.
 
 .. _hykeyword:
 
@@ -340,7 +331,7 @@ Since they have no "value" to Python, this makes working in Hy hard, since
 doing something like ``(print (if True True False))`` is not just common, it's
 expected.
 
-As a result, we auto-mangle things using a ``Result`` object, where we offer
+As a result, we reconfigure things using a ``Result`` object, where we offer
 up any ``ast.stmt`` that need to get run, and a single ``ast.expr`` that can
 be used to get the value of whatever was just run. Hy does this by forcing
 assignment to things while running.
@@ -352,11 +343,11 @@ As example, the Hy::
 Will turn into::
 
     if True:
-        _mangled_name_here = True
+        _temp_name_here = True
     else:
-        _mangled_name_here = False
+        _temp_name_here = False
 
-    print _mangled_name_here
+    print _temp_name_here
 
 
 OK, that was a bit of a lie, since we actually turn that statement

docs/language/interop.rst

@@ -8,6 +8,12 @@ Hy <-> Python interop
 Despite being a Lisp, Hy aims to be fully compatible with Python. That means
 every Python module or package can be imported in Hy code, and vice versa.
 
+:ref:`Mangling <mangling>` allows variable names to be spelled differently in
+Hy and Python. For example, Python's ``str.format_map`` can be written
+``str.format-map`` in Hy, and a Hy function named ``valid?`` would be called
+``is_valid`` in Python. In Python, you can import Hy's core functions
+``mangle`` and ``unmangle`` directly from the ``hy`` package.
+
 Using Python from Hy
 ====================
 
@@ -27,41 +33,6 @@ You can use it in Hy:
 
 You can also import ``.pyc`` bytecode files, of course.
 
-A quick note about mangling
---------
-
-In Python, snake_case is used by convention. Lisp dialects tend to use dashes
-instead of underscores, so Hy does some magic to give you more pleasant names.
-
-In the same way, ``UPPERCASE_NAMES`` from Python can be used ``*with-earmuffs*``
-instead.
-
-You can use either the original names or the new ones.
-
-Imagine ``example.py``::
-
-    def function_with_a_long_name():
-        print(42)
-
-    FOO = "bar"
-
-Then, in Hy:
-
-.. code-block:: clj
-
-    (import example)
-    (.function-with-a-long-name example) ; prints "42"
-    (.function_with_a_long_name example) ; also prints "42"
-
-    (print (. example *foo*)) ; prints "bar"
-    (print (. example FOO))   ; also prints "bar"
-
-.. warning::
-   Mangling isn’t that simple; there is more to discuss about it, yet it doesn’t
-   belong in this section.
-.. TODO: link to mangling section, when it is done
-
-
 Using Hy from Python
 ====================
 

docs/language/syntax.rst

@@ -0,0 +1,156 @@
+==============
+Syntax
+==============
+
+identifiers
+-----------
+
+An identifier consists of a nonempty sequence of Unicode characters that are not whitespace nor any of the following: ``( ) [ ] { } ' "``. Hy first tries to parse each identifier into a numeric literal, then into a keyword if that fails, and finally into a symbol if that fails.
+
+numeric literals
+----------------
+
+In addition to regular numbers, standard notation from Python 3 for non-base 10
+integers is used. ``0x`` for Hex, ``0o`` for Octal, ``0b`` for Binary.
+
+.. code-block:: clj
+    
+    (print 0x80 0b11101 0o102 30)
+
+Underscores and commas can appear anywhere in a numeric literal except the very
+beginning. They have no effect on the value of the literal, but they're useful
+for visually separating digits.
+
+.. code-block:: clj
+
+    (print 10,000,000,000 10_000_000_000)
+
+Unlike Python, Hy provides literal forms for NaN and infinity: ``NaN``,
+``Inf``, and ``-Inf``.
+
+string literals
+---------------
+
+Hy allows double-quoted strings (e.g., ``"hello"``), but not single-quoted
+strings like Python. The single-quote character ``'`` is reserved for
+preventing the evaluation of a form (e.g., ``'(+ 1 1)``), as in most Lisps.
+
+Python's so-called triple-quoted strings (e.g., ``'''hello'''`` and
+``"""hello"""``) aren't supported. However, in Hy, unlike Python, any string
+literal can contain newlines. Furthermore, Hy supports an alternative form of
+string literal called a "bracket string" similar to Lua's long brackets.
+Bracket strings have customizable delimiters, like the here-documents of other
+languages. A bracket string begins with ``#[FOO[`` and ends with ``]FOO]``,
+where ``FOO`` is any string not containing ``[`` or ``]``, including the empty
+string. For example::
+
+   => (print #[["That's very kind of yuo [sic]" Tom wrote back.]])
+   "That's very kind of yuo [sic]" Tom wrote back.
+   => (print #[==[1 + 1 = 2]==])
+   1 + 1 = 2
+
+A bracket string can contain newlines, but if it begins with one, the newline
+is removed, so you can begin the content of a bracket string on the line
+following the opening delimiter with no effect on the content. Any leading
+newlines past the first are preserved.
+
+Plain string literals support :ref:`a variety of backslash escapes
+<py:strings>`. To create a "raw string" that interprets all backslashes
+literally, prefix the string with ``r``, as in ``r"slash\not"``. Bracket
+strings are always raw strings and don't allow the ``r`` prefix.
+
+Whether running under Python 2 or Python 3, Hy treats all string literals as
+sequences of Unicode characters by default, and allows you to prefix a plain
+string literal (but not a bracket string) with ``b`` to treat it as a sequence
+of bytes. So when running under Python 3, Hy translates ``"foo"`` and
+``b"foo"`` to the identical Python code, but when running under Python 2,
+``"foo"`` is translated to ``u"foo"`` and ``b"foo"`` is translated to
+``"foo"``.
+
+.. _syntax-keywords:
+
+keywords
+--------
+
+An identifier headed by a colon, such as ``:foo``, is a keyword. Keywords
+evaluate to a string preceded by the Unicode non-character code point U+FDD0,
+like ``"\ufdd0:foo"``, so ``:foo`` and ``":foo"`` aren't equal. However, if a
+literal keyword appears in a function call, it's used to indicate a keyword
+argument rather than passed in as a value. For example, ``(f :foo 3)`` calls
+the function ``f`` with the keyword argument named ``foo`` set to ``3``. Hence,
+trying to call a function on a literal keyword may fail: ``(f :foo)`` yields
+the error ``Keyword argument :foo needs a value``. To avoid this, you can quote
+the keyword, as in ``(f ':foo)``, or use it as the value of another keyword
+argument, as in ``(f :arg :foo)``.
+
+.. _mangling:
+
+symbols
+-------
+
+Symbols are identifiers that are neither legal numeric literals nor legal
+keywords. In most contexts, symbols are compiled to Python variable names. Some
+example symbols are ``hello``, ``+++``, ``3fiddy``, ``$40``, ``just✈wrong``,
+and ``🦑``.
+
+Since the rules for Hy symbols are much more permissive than the rules for
+Python identifiers, Hy uses a mangling algorithm to convert its own names to
+Python-legal names. The rules are:
+
+- Convert all hyphens (``-``) to underscores (``_``). Thus, ``foo-bar`` becomes
+  ``foo_bar``.
+- If the name ends with ``?``, remove it and prepend ``is``. Thus, ``tasty?``
+  becomes ``is_tasty``.
+- If the name still isn't Python-legal, make the following changes. A name
+  could be Python-illegal because it contains a character that's never legal in
+  a Python name, it contains a character that's illegal in that position, or
+  it's equal to a Python reserved word.
+
+  - Prepend ``hyx_`` to the name.
+  - Replace each illegal character with ``ΔfooΔ`` (or on Python 2, ``XfooX``),
+    where ``foo`` is the the Unicode character name in lowercase, with spaces
+    replaced by underscores and hyphens replaced by ``H``. Replace ``Δ`` itself
+    (or on Python 2, ``X``) the same way. If the character doesn't have a name,
+    use ``U`` followed by its code point in lowercase hexadecimal.
+
+  Thus, ``green☘`` becomes ``hyx_greenΔshamrockΔ`` and ``if`` becomes
+  ``hyx_if``.
+
+- Finally, any added ``hyx_`` or ``is_`` is added after any leading
+  underscores, because leading underscores have special significance to Python.
+  Thus, ``_tasty?`` becomes ``_is_tasty`` instead of ``is__tasty``.
+
+Mangling isn't something you should have to think about often, but you may see
+mangled names in error messages, the output of ``hy2py``, etc. A catch to be
+aware of is that mangling, as well as the inverse "unmangling" operation
+offered by the ``unmangle`` function, isn't one-to-one. Two different symbols
+can mangle to the same string and hence compile to the same Python variable.
+The chief practical consequence of this is that ``-`` and ``_`` are
+interchangeable in all symbol names, so you shouldn't assign to the
+one-character name ``_`` , or else you'll interfere with certain uses of
+subtraction.
+
+discard prefix
+--------------
+
+Hy supports the Extensible Data Notation discard prefix, like Clojure.
+Any form prefixed with ``#_`` is discarded instead of compiled.
+This completely removes the form so it doesn't evaluate to anything,
+not even None.
+It's often more useful than linewise comments for commenting out a
+form, because it respects code structure even when part of another
+form is on the same line. For example:
+
+.. code-block:: clj
+
+   => (print "Hy" "cruel" "World!")
+   Hy cruel World!
+   => (print "Hy" #_"cruel" "World!")
+   Hy World!
+   => (+ 1 1 (print "Math is hard!"))
+   Math is hard!
+   Traceback (most recent call last):
+      ...
+   TypeError: unsupported operand type(s) for +: 'int' and 'NoneType'
+   => (+ 1 1 #_(print "Math is hard!"))
+   2

hy/__init__.py

@@ -12,5 +12,5 @@ import hy.importer  # NOQA
 # we import for side-effects.
 
 
-from hy.core.language import read, read_str  # NOQA
+from hy.core.language import read, read_str, mangle, unmangle  # NOQA
 from hy.importer import hy_eval as eval  # NOQA

hy/_compat.py

@@ -18,7 +18,7 @@ except ImportError:
                        (x >> 8) & 0xff,
                        (x >> 16) & 0xff,
                        (x >> 24) & 0xff]))
-import sys
+import sys, keyword
 
 PY3 = sys.version_info[0] >= 3
 PY35 = sys.version_info >= (3, 5)
@@ -35,3 +35,24 @@ if PY3:
 else:
     def raise_empty(t, *args):
         raise t(*args)
+
+def isidentifier(x):
+    if x in ('True', 'False', 'None', 'print'):
+        # `print` is special-cased here because Python 2's
+        # keyword.iskeyword will count it as a keyword, but we
+        # use the __future__ feature print_function, which makes
+        # it a non-keyword.
+        return True
+    if keyword.iskeyword(x):
+        return False
+    if PY3:
+        return x.isidentifier()
+    if x.rstrip() != x:
+        return False
+    import tokenize as T
+    from io import StringIO
+    try:
+        tokens = list(T.generate_tokens(StringIO(x).readline))
+    except T.TokenError:
+        return False
+    return len(tokens) == 2 and tokens[0][0] == T.NAME

hy/cmdline.py

@@ -16,7 +16,7 @@ import astor.code_gen
 import hy
 
 from hy.lex import LexException, PrematureEndOfInput
-from hy.lex.parser import hy_symbol_mangle
+from hy.lex.parser import mangle
 from hy.compiler import HyTypeError
 from hy.importer import (hy_eval, import_buffer_to_module,
                          import_file_to_ast, import_file_to_hst,
@@ -63,12 +63,12 @@ class HyREPL(code.InteractiveConsole):
         elif callable(output_fn):
             self.output_fn = output_fn
         else:
-            f = hy_symbol_mangle(output_fn)
             if "." in output_fn:
-                module, f = f.rsplit(".", 1)
+                parts = [mangle(x) for x in output_fn.split(".")]
+                module, f = '.'.join(parts[:-1]), parts[-1]
                 self.output_fn = getattr(importlib.import_module(module), f)
             else:
-                self.output_fn = __builtins__[f]
+                self.output_fn = __builtins__[mangle(output_fn)]
 
         code.InteractiveConsole.__init__(self, locals=locals,
                                          filename=filename)
@@ -112,8 +112,8 @@ class HyREPL(code.InteractiveConsole):
 
         if value is not None:
             # Make the last non-None value available to
-            # the user as `_`.
+            # the user as `*1`.
-            self.locals['_'] = value
+            self.locals[mangle("*1")] = value
             # Print the value.
             try:
                 output = self.output_fn(value)

hy/compiler.py

@@ -8,7 +8,7 @@ from hy.models import (HyObject, HyExpression, HyKeyword, HyInteger, HyComplex,
                        HyDict, HyCons, wrap_value)
 from hy.errors import HyCompileError, HyTypeError
 
-from hy.lex.parser import hy_symbol_mangle
+from hy.lex.parser import mangle
 
 import hy.macros
 from hy._compat import (
@@ -53,7 +53,7 @@ def load_stdlib():
     import hy.core
     for module in hy.core.STDLIB:
         mod = importlib.import_module(module)
-        for e in mod.EXPORTS:
+        for e in map(ast_str, mod.EXPORTS):
             if getattr(mod, e) is not getattr(builtins, e, ''):
                 # Don't bother putting a name in _stdlib if it
                 # points to a builtin with the same name. This
@@ -61,38 +61,17 @@ def load_stdlib():
                 _stdlib[e] = module
 
 
-# True, False and None included here since they
-# are assignable in Python 2.* but become
-# keywords in Python 3.*
-def _is_hy_builtin(name, module_name):
-    extras = ['True', 'False', 'None']
-    if name in extras or keyword.iskeyword(name):
-        return True
-    # for non-Hy modules, check for pre-existing name in
-    # _compile_table
-    if not module_name.startswith("hy."):
-        return name in _compile_table
-    return False
-
-
 _compile_table = {}
 _decoratables = (ast.FunctionDef, ast.ClassDef)
 if PY35:
     _decoratables += (ast.AsyncFunctionDef,)
 
 
-def ast_str(foobar):
+def ast_str(x, piecewise=False):
-    if PY3:
+    if piecewise:
-        return str(foobar)
+        return ".".join(ast_str(s) if s else "" for s in x.split("."))
-
+    x = mangle(x)
-    try:
+    return x if PY3 else x.encode('UTF8')
-        return str(foobar)
-    except UnicodeEncodeError:
-        pass
-
-    enc = codecs.getencoder('punycode')
-    foobar, _ = enc(foobar)
-    return "hy_%s" % (str(foobar).replace("-", "_"))
 
 
 def builds(*types, **kwargs):
@@ -104,6 +83,8 @@ def builds(*types, **kwargs):
 
     def _dec(fn):
         for t in types:
+            if isinstance(t, string_types):
+                t = ast_str(t)
             _compile_table[t] = fn
         return fn
     return _dec
@@ -379,7 +360,7 @@ def is_unpack(kind, x):
     return (isinstance(x, HyExpression)
             and len(x) > 0
             and isinstance(x[0], HySymbol)
-            and x[0] == "unpack_" + kind)
+            and x[0] == "unpack-" + kind)
 
 
 def ends_with_else(expr):
@@ -393,7 +374,6 @@ def ends_with_else(expr):
 class HyASTCompiler(object):
 
     def __init__(self, module_name):
-        self.allow_builtins = module_name.startswith("hy.core")
         self.anon_var_count = 0
         self.imports = defaultdict(set)
         self.module_name = module_name
@@ -437,6 +417,8 @@ class HyASTCompiler(object):
         return ret.stmts
 
     def compile_atom(self, atom_type, atom):
+        if isinstance(atom_type, string_types):
+            atom_type = ast_str(atom_type)
         if atom_type in _compile_table:
             # _compile_table[atom_type] is a method for compiling this
             # type of atom, so call it. If it has an extra parameter,
@@ -525,10 +507,11 @@ class HyASTCompiler(object):
                 compiled_value = self.compile(value)
                 ret += compiled_value
 
-                # no unicode for py2 in ast names
+                keyword = expr[2:]
-                keyword = str(expr[2:])
+                if not keyword:
-                if "-" in keyword and keyword != "-":
+                    raise HyTypeError(expr, "Can't call a function with the "
-                    keyword = keyword.replace("-", "_")
+                                            "empty keyword")
+                keyword = ast_str(keyword)
 
                 keywords.append(asty.keyword(
                     expr, arg=keyword, value=compiled_value.force_expr))
@@ -699,17 +682,17 @@ class HyASTCompiler(object):
         """
         if level == 0:
             if isinstance(form, HyExpression):
-                if form and form[0] in ("unquote", "unquote_splice"):
+                if form and form[0] in ("unquote", "unquote-splice"):
                     if len(form) != 2:
                         raise HyTypeError(form,
                                           ("`%s' needs 1 argument, got %s" %
                                            form[0], len(form) - 1))
-                    return set(), form[1], (form[0] == "unquote_splice")
+                    return set(), form[1], (form[0] == "unquote-splice")
 
         if isinstance(form, HyExpression):
             if form and form[0] == "quasiquote":
                 level += 1
-            if form and form[0] in ("unquote", "unquote_splice"):
+            if form and form[0] in ("unquote", "unquote-splice"):
                 level -= 1
 
         name = form.__class__.__name__
@@ -783,12 +766,12 @@ class HyASTCompiler(object):
         ret.add_imports("hy", imports)
         return ret
 
-    @builds("unquote", "unquote_splicing")
+    @builds("unquote", "unquote-splicing")
     def compile_unquote(self, expr):
         raise HyTypeError(expr,
                           "`%s' can't be used at the top-level" % expr[0])
 
-    @builds("unpack_iterable")
+    @builds("unpack-iterable")
     @checkargs(exact=1)
     def compile_unpack_iterable(self, expr):
         if not PY3:
@@ -797,7 +780,7 @@ class HyASTCompiler(object):
         ret += asty.Starred(expr, value=ret.force_expr, ctx=ast.Load())
         return ret
 
-    @builds("unpack_mapping")
+    @builds("unpack-mapping")
     @checkargs(exact=1)
     def compile_unpack_mapping(self, expr):
         raise HyTypeError(expr, "`unpack-mapping` isn't allowed here")
@@ -1143,12 +1126,12 @@ class HyASTCompiler(object):
             ret += self.compile(expr[1])
         return ret + asty.Yield(expr, value=ret.force_expr)
 
-    @builds("yield_from", iff=PY3)
+    @builds("yield-from", iff=PY3)
     @builds("await", iff=PY35)
     @checkargs(1)
     def compile_yield_from_or_await_expression(self, expr):
         ret = Result() + self.compile(expr[1])
-        node = asty.YieldFrom if expr[0] == "yield_from" else asty.Await
+        node = asty.YieldFrom if expr[0] == "yield-from" else asty.Await
         return ret + node(expr, value=ret.force_expr)
 
     @builds("import")
@@ -1156,19 +1139,16 @@ class HyASTCompiler(object):
         expr = copy.deepcopy(expr)
         def _compile_import(expr, module, names=None, importer=asty.Import):
             if not names:
-                names = [ast.alias(name=ast_str(module), asname=None)]
+                names = [ast.alias(name=ast_str(module, piecewise=True), asname=None)]
 
-            ast_module = ast_str(module)
+            ast_module = ast_str(module, piecewise=True)
             module = ast_module.lstrip(".")
             level = len(ast_module) - len(module)
             if not module:
                 module = None
 
-            ret = importer(expr,
+            return Result() + importer(
-                           module=module,
+                expr, module=module, names=names, level=level)
-                           names=names,
-                           level=level)
-            return Result() + ret
 
         expr.pop(0)  # index
         rimports = Result()
@@ -1196,7 +1176,7 @@ class HyASTCompiler(object):
                                           "garbage after aliased import")
                     iexpr.pop(0)  # :as
                     alias = iexpr.pop(0)
-                    names = [ast.alias(name=ast_str(module),
+                    names = [ast.alias(name=ast_str(module, piecewise=True),
                                        asname=ast_str(alias))]
                     rimports += _compile_import(expr, ast_str(module), names)
                     continue
@@ -1210,7 +1190,7 @@ class HyASTCompiler(object):
                             alias = ast_str(entry.pop(0))
                         else:
                             alias = None
-                        names.append(ast.alias(name=ast_str(sym),
+                        names.append(ast.alias(name=(str(sym) if sym == "*" else ast_str(sym)),
                                                asname=alias))
 
                     rimports += _compile_import(expr, module,
@@ -1307,7 +1287,7 @@ class HyASTCompiler(object):
             slice=ast.Slice(lower=nodes[1], upper=nodes[2], step=nodes[3]),
             ctx=ast.Load())
 
-    @builds("with_decorator")
+    @builds("with-decorator")
     @checkargs(min=1)
     def compile_decorate_expression(self, expr):
         expr.pop(0)  # with-decorator
@@ -1403,7 +1383,7 @@ class HyASTCompiler(object):
 
         return gen_res + cond, gen
 
-    @builds("list_comp", "set_comp", "genexpr")
+    @builds("list-comp", "set-comp", "genexpr")
     @checkargs(min=2, max=3)
     def compile_comprehension(self, expr):
         # (list-comp expr (target iter) cond?)
@@ -1421,13 +1401,13 @@ class HyASTCompiler(object):
 
         ret = self.compile(expression)
         node_class = (
-            asty.ListComp if form == "list_comp" else
+            asty.ListComp if form == "list-comp" else
-            asty.SetComp if form == "set_comp" else
+            asty.SetComp if form == "set-comp" else
             asty.GeneratorExp)
         return ret + gen_res + node_class(
             expr, elt=ret.force_expr, generators=gen)
 
-    @builds("dict_comp")
+    @builds("dict-comp")
     @checkargs(min=3, max=4)
     def compile_dict_comprehension(self, expr):
         expr.pop(0)  # dict-comp
@@ -1554,15 +1534,16 @@ class HyASTCompiler(object):
                                values=[value.force_expr for value in values])
         return ret
 
-    def _compile_compare_op_expression(self, expression):
     ops = {"=": ast.Eq, "!=": ast.NotEq,
            "<": ast.Lt, "<=": ast.LtE,
            ">": ast.Gt, ">=": ast.GtE,
-               "is": ast.Is, "is_not": ast.IsNot,
+           "is": ast.Is, "is-not": ast.IsNot,
-               "in": ast.In, "not_in": ast.NotIn}
+           "in": ast.In, "not-in": ast.NotIn}
+    ops = {ast_str(k): v for k, v in ops.items()}
 
-        inv = expression.pop(0)
+    def _compile_compare_op_expression(self, expression):
-        ops = [ops[inv]() for _ in range(len(expression) - 1)]
+        inv = ast_str(expression.pop(0))
+        ops = [self.ops[inv]() for _ in range(len(expression) - 1)]
 
         e = expression[0]
         exprs, ret, _ = self._compile_collect(expression)
@@ -1578,12 +1559,12 @@ class HyASTCompiler(object):
                 asty.Name(expression, id="True", ctx=ast.Load()))
         return self._compile_compare_op_expression(expression)
 
-    @builds("!=", "is_not")
+    @builds("!=", "is-not")
     @checkargs(min=2)
     def compile_compare_op_expression_coll(self, expression):
         return self._compile_compare_op_expression(expression)
 
-    @builds("in", "not_in")
+    @builds("in", "not-in")
     @checkargs(2)
     def compile_compare_op_expression_binary(self, expression):
         return self._compile_compare_op_expression(expression)
@@ -1680,7 +1661,7 @@ class HyASTCompiler(object):
     def compile_maths_expression_sub(self, expression):
         return self._compile_maths_expression_additive(expression)
 
-    @builds("+=", "/=", "//=", "*=", "_=", "%=", "**=", "<<=", ">>=", "|=",
+    @builds("+=", "/=", "//=", "*=", "-=", "%=", "**=", "<<=", ">>=", "|=",
             "^=", "&=")
     @builds("@=", iff=PY35)
     @checkargs(2)
@@ -1689,7 +1670,7 @@ class HyASTCompiler(object):
                "/=": ast.Div,
                "//=": ast.FloorDiv,
                "*=": ast.Mult,
-               "_=": ast.Sub,
+               "-=": ast.Sub,
                "%=": ast.Mod,
                "**=": ast.Pow,
                "<<=": ast.LShift,
@@ -1732,7 +1713,7 @@ class HyASTCompiler(object):
 
         if isinstance(fn, HySymbol):
             # First check if `fn` is a special form, unless it has an
-            # `unpack_iterable` in it, since Python's operators (`+`,
+            # `unpack-iterable` in it, since Python's operators (`+`,
             # etc.) can't unpack. An exception to this exception is that
             # tuple literals (`,`) can unpack.
             if fn == "," or not (
@@ -1785,7 +1766,7 @@ class HyASTCompiler(object):
         # An exception for pulling together keyword args is if we're doing
         # a typecheck, eg (type :foo)
         with_kwargs = fn not in (
-            "type", "HyKeyword", "keyword", "name", "is_keyword")
+            "type", "HyKeyword", "keyword", "name", "keyword?")
         args, ret, keywords, oldpy_star, oldpy_kw = self._compile_collect(
             expression[1:], with_kwargs, oldpy_unpack=True)
 
@@ -1813,10 +1794,11 @@ class HyASTCompiler(object):
     def _compile_assign(self, name, result):
 
         str_name = "%s" % name
-        if (_is_hy_builtin(str_name, self.module_name) and
+        if str_name in (["None"] + (["True", "False"] if PY3 else [])):
-                not self.allow_builtins):
+            # Python 2 allows assigning to True and False, although
+            # this is rarely wise.
             raise HyTypeError(name,
-                              "Can't assign to a builtin: `%s'" % str_name)
+                              "Can't assign to `%s'" % str_name)
 
         result = self.compile(result)
         ld_name = self.compile(name)
@@ -2057,7 +2039,7 @@ class HyASTCompiler(object):
                 pairs = expr[1:]
                 while len(pairs) > 0:
                     k, v = (pairs.pop(0), pairs.pop(0))
-                    if k == HySymbol("__init__"):
+                    if ast_str(k) == "__init__":
                         v.append(HySymbol("None"))
                     new_args.append(k)
                     new_args.append(v)
@@ -2092,8 +2074,6 @@ class HyASTCompiler(object):
             body += self._compile_assign(symb, docstring)
             body += body.expr_as_stmt()
 
-        allow_builtins = self.allow_builtins
-        self.allow_builtins = True
         if expressions and isinstance(expressions[0], HyList) \
            and not isinstance(expressions[0], HyExpression):
             expr = expressions.pop(0)
@@ -2105,8 +2085,6 @@ class HyASTCompiler(object):
         for expression in expressions:
             body += self.compile(rewire_init(macroexpand(expression, self)))
 
-        self.allow_builtins = allow_builtins
-
         if not body.stmts:
             body += asty.Pass(expressions)
 
@@ -2120,7 +2098,7 @@ class HyASTCompiler(object):
             bases=bases_expr,
             body=body.stmts)
 
-    @builds("dispatch_tag_macro")
+    @builds("dispatch-tag-macro")
     @checkargs(exact=2)
     def compile_dispatch_tag_macro(self, expression):
         expression.pop(0)  # dispatch-tag-macro
@@ -2131,11 +2109,11 @@ class HyASTCompiler(object):
                 "Trying to expand a tag macro using `{0}' instead "
                 "of string".format(type(tag).__name__),
             )
-        tag = HyString(hy_symbol_mangle(str(tag))).replace(tag)
+        tag = HyString(mangle(tag)).replace(tag)
         expr = tag_macroexpand(tag, expression.pop(0), self)
         return self.compile(expr)
 
-    @builds("eval_and_compile", "eval_when_compile")
+    @builds("eval-and-compile", "eval-when-compile")
     def compile_eval_and_compile(self, expression, building):
         expression[0] = HySymbol("do")
         hy.importer.hy_eval(expression,
@@ -2198,8 +2176,8 @@ class HyASTCompiler(object):
                 attr=ast_str(local),
                 ctx=ast.Load())
 
-        if symbol in _stdlib:
+        if ast_str(symbol) in _stdlib:
-            self.imports[_stdlib[symbol]].add(symbol)
+            self.imports[_stdlib[ast_str(symbol)]].add(ast_str(symbol))
 
         return asty.Name(symbol, id=ast_str(symbol), ctx=ast.Load())
 

hy/contrib/hy_repr.hy

@@ -75,9 +75,9 @@
     'quote "'"
     'quasiquote "`"
     'unquote "~"
-    'unquote_splice "~@"
+    'unquote-splice "~@"
-    'unpack_iterable "#* "
+    'unpack-iterable "#* "
-    'unpack_mapping "#** "})
+    'unpack-mapping "#** "})
   (if (and x (symbol? (first x)) (in (first x) syntax))
     (+ (get syntax (first x)) (hy-repr (second x)))
     (+ "(" (-cat x) ")"))))

hy/contrib/walk.hy

@@ -257,7 +257,7 @@ Arguments without a header are under None.
         (= head 'defclass) (self.handle-defclass)
         (= head 'quasiquote) (self.+quote)
         ;; must be checked last!
-        (in head special-forms) (self.handle-special-form)
+        (in (mangle head) special-forms) (self.handle-special-form)
         ;; Not a special form. Traverse it like a coll
         (self.handle-coll)))
 

hy/core/language.hy

@@ -18,6 +18,7 @@
 (import [hy._compat [long-type]]) ; long for python2, int for python3
 (import [hy.models [HyCons HySymbol HyKeyword]])
 (import [hy.lex [LexException PrematureEndOfInput tokenize]])
+(import [hy.lex.parser [mangle unmangle]])
 (import [hy.compiler [HyASTCompiler spoof-positions]])
 (import [hy.importer [hy-eval :as eval]])
 
@@ -87,7 +88,7 @@ If the second argument `codegen` is true, generate python code instead."
   "Return a generator from the original collection `coll` with no duplicates."
   (setv seen (set) citer (iter coll))
   (for* [val citer]
-    (if (not_in val seen)
+    (if (not-in val seen)
       (do
        (yield val)
        (.add seen val)))))
@@ -453,20 +454,16 @@ as EOF (defaults to an empty string)."
   "Reads and tokenizes first line of `input`."
   (read :from-file (StringIO input)))
 
-(defn hyify [text]
-  "Convert `text` to match hy identifier."
-  (.replace (string text) "_" "-"))
-
 (defn keyword [value]
   "Create a keyword from `value`.
 
 Strings numbers and even objects with the __name__ magic will work."
   (if (and (string? value) (value.startswith HyKeyword.PREFIX))
-    (hyify value)
+    (unmangle value)
     (if (string? value)
-      (HyKeyword (+ ":" (hyify value)))
+      (HyKeyword (+ ":" (unmangle value)))
       (try
-        (hyify (.__name__ value))
+        (unmangle (.__name__ value))
         (except [] (HyKeyword (+ ":" (string value))))))))
 
 (defn name [value]
@@ -475,11 +472,11 @@ Strings numbers and even objects with the __name__ magic will work."
 Keyword special character will be stripped. String will be used as is.
 Even objects with the __name__ magic will work."
   (if (and (string? value) (value.startswith HyKeyword.PREFIX))
-    (hyify (cut value 2))
+    (unmangle (cut value 2))
     (if (string? value)
-      (hyify value)
+      (unmangle value)
       (try
-        (hyify (. value __name__))
+        (unmangle (. value __name__))
         (except [] (string value))))))
 
 (defn xor [a b]
@@ -488,14 +485,14 @@ Even objects with the __name__ magic will work."
     False
     (or a b)))
 
-(setv *exports*
+(setv EXPORTS
   '[*map accumulate butlast calling-module-name chain coll? combinations
     comp complement compress cons cons? constantly count cycle dec distinct
     disassemble drop drop-last drop-while empty? eval even? every? exec first
     filter flatten float? fraction gensym group-by identity inc input instance?
     integer integer? integer-char? interleave interpose islice iterable?
     iterate iterator? juxt keyword keyword? last list* macroexpand
-    macroexpand-1 map merge-with multicombinations name neg? none? nth
+    macroexpand-1 mangle map merge-with multicombinations name neg? none? nth
     numeric? odd? partition permutations pos? product range read read-str
     remove repeat repeatedly rest reduce second some string string? symbol?
-    take take-nth take-while xor tee zero? zip zip-longest])
+    take take-nth take-while unmangle xor tee zero? zip zip-longest])

hy/core/shadow.hy

@@ -163,7 +163,7 @@
     (setv coll (get coll k)))
   coll)
 
-(setv *exports* [
+(setv EXPORTS [
   '+ '- '* '** '/ '// '% '@
   '<< '>> '& '| '^ '~
   '< '> '<= '>= '= '!=
@@ -171,4 +171,4 @@
   'is 'is-not 'in 'not-in
   'get])
 (if (not PY35)
-  (.remove *exports* '@))
+  (.remove EXPORTS '@))

hy/extra/reserved.hy

@@ -13,10 +13,9 @@
   The result of the first call is cached."
   (global _cache)
   (if (is _cache None) (do
-    (setv unmangle (. sys.modules ["hy.lex.parser"] hy_symbol_unmangle))
     (setv _cache (frozenset (map unmangle (+
-      hy.core.language.*exports*
+      hy.core.language.EXPORTS
-      hy.core.shadow.*exports*
+      hy.core.shadow.EXPORTS
       (list (.keys (get hy.macros._hy_macros None)))
       keyword.kwlist
       (list-comp k [k (.keys hy.compiler.-compile-table)]

hy/lex/parser.py

@@ -1,3 +1,4 @@
+# -*- encoding: utf-8 -*-
 # Copyright 2018 the authors.
 # This file is part of Hy, which is free software licensed under the Expat
 # license. See the LICENSE.
@@ -5,10 +6,11 @@
 from __future__ import unicode_literals
 
 from functools import wraps
+import string, re, unicodedata
 
 from rply import ParserGenerator
 
-from hy._compat import str_type
+from hy._compat import PY3, str_type, isidentifier
 from hy.models import (HyBytes, HyComplex, HyCons, HyDict, HyExpression,
                        HyFloat, HyInteger, HyKeyword, HyList, HySet, HyString,
                        HySymbol)
@@ -21,43 +23,65 @@ pg = ParserGenerator(
     cache_id="hy_parser"
 )
 
+mangle_delim = 'Δ' if PY3 else 'X'
 
-def hy_symbol_mangle(p):
+def mangle(s):
-    if p.startswith("*") and p.endswith("*") and p not in ("*", "**"):
+    """Stringify the argument and convert it to a valid Python identifier
-        p = p[1:-1].upper()
+    according to Hy's mangling rules."""
 
-    if "-" in p and p != "-":
+    assert s
-        p = p.replace("-", "_")
 
-    if p.endswith("?") and p != "?":
+    s = str_type(s)
-        p = "is_%s" % (p[:-1])
 
-    if p.endswith("!") and p != "!":
+    s = s.replace("-", "_")
-        p = "%s_bang" % (p[:-1])
+    s2 = s.lstrip('_')
+    leading_underscores = '_' * (len(s) - len(s2))
+    s = s2
 
-    return p
+    if s.endswith("?"):
+        s = 'is_' + s[:-1]
+    if not isidentifier(leading_underscores + s):
+        # Replace illegal characters with their Unicode character
+        # names, or hexadecimal if they don't have one.
+        s = 'hyx_' + ''.join(
+            c
+               if c != mangle_delim and isidentifier('S' + c)
+                 # We prepend the "S" because some characters aren't
+                 # allowed at the start of an identifier.
+               else '{0}{1}{0}'.format(mangle_delim,
+                   unicodedata.name(c, '').lower().replace('-', 'H').replace(' ', '_')
+                   or 'U{:x}'.format(ord(c)))
+            for c in s)
+
+    s = leading_underscores + s
+    assert isidentifier(s)
+    return s
 
 
-def hy_symbol_unmangle(p):
+def unmangle(s):
-    # hy_symbol_mangle is one-way, so this can't be perfect.
+    """Stringify the argument and try to convert it to a pretty unmangled
-    # But it can be useful till we have a way to get the original
+    form. This may not round-trip, because different Hy symbol names can
-    # symbol (https://github.com/hylang/hy/issues/360).
+    mangle to the same Python identifier."""
-    p = str_type(p)
 
-    if p.endswith("_bang") and p != "_bang":
+    s = str_type(s)
-        p = p[:-len("_bang")] + "!"
 
-    if p.startswith("is_") and p != "is_":
+    s2 = s.lstrip('_')
-        p = p[len("is_"):] + "?"
+    leading_underscores = len(s) - len(s2)
+    s = s2
 
-    if "_" in p and p != "_":
+    if s.startswith('hyx_'):
-        p = p.replace("_", "-")
+        s = re.sub('{0}(U)?([_a-z0-9H]+?){0}'.format(mangle_delim),
+            lambda mo:
+               chr(int(mo.group(2), base=16))
+               if mo.group(1)
+               else unicodedata.lookup(
+                   mo.group(2).replace('_', ' ').replace('H', '-').upper()),
+            s[len('hyx_'):])
+    if s.startswith('is_'):
+        s = s[len("is_"):] + "?"
+    s = s.replace('_', '-')
 
-    if (all([c.isalpha() and c.isupper() or c == '_' for c in p]) and
+    return '-' * leading_underscores + s
-            any([c.isalpha() for c in p])):
-        p = '*' + p.lower() + '*'
-
-    return p
 
 
 def set_boundaries(fun):
@@ -201,7 +225,7 @@ def term_unquote(p):
 @pg.production("term : UNQUOTESPLICE term")
 @set_quote_boundaries
 def term_unquote_splice(p):
-    return HyExpression([HySymbol("unquote_splice"), p[1]])
+    return HyExpression([HySymbol("unquote-splice"), p[1]])
 
 
 @pg.production("term : HASHSTARS term")
@@ -209,9 +233,9 @@ def term_unquote_splice(p):
 def term_hashstars(p):
     n_stars = len(p[0].getstr()[1:])
     if n_stars == 1:
-        sym = "unpack_iterable"
+        sym = "unpack-iterable"
     elif n_stars == 2:
-        sym = "unpack_mapping"
+        sym = "unpack-mapping"
     else:
         raise LexException(
             "Too many stars in `#*` construct (if you want to unpack a symbol "
@@ -227,7 +251,7 @@ def hash_other(p):
     st = p[0].getstr()[1:]
     str_object = HyString(st)
     expr = p[1]
-    return HyExpression([HySymbol("dispatch_tag_macro"), str_object, expr])
+    return HyExpression([HySymbol("dispatch-tag-macro"), str_object, expr])
 
 
 @pg.production("set : HLCURLY list_contents RCURLY")
@@ -307,7 +331,7 @@ def t_identifier(p):
             '`(. <expression> <attr>)` or `(.<attr> <expression>)`)',
             p[0].source_pos.lineno, p[0].source_pos.colno)
 
-    return HySymbol(".".join(hy_symbol_mangle(x) for x in obj.split(".")))
+    return HySymbol(obj)
 
 
 def symbol_like(obj):

hy/macros.py

@@ -2,8 +2,12 @@
 # This file is part of Hy, which is free software licensed under the Expat
 # license. See the LICENSE.
 
+from hy._compat import PY3
 import hy.inspect
 from hy.models import replace_hy_obj, HyExpression, HySymbol
+from hy.lex.parser import mangle
+from hy._compat import str_type
+
 from hy.errors import HyTypeError, HyMacroExpansionError
 
 from collections import defaultdict
@@ -32,6 +36,7 @@ def macro(name):
     This function is called from the `defmacro` special form in the compiler.
 
     """
+    name = mangle(name)
     def _(fn):
         fn.__name__ = '({})'.format(name)
         try:
@@ -62,11 +67,14 @@ def tag(name):
 
     """
     def _(fn):
-        fn.__name__ = '#{}'.format(name)
+        _name = mangle('#{}'.format(name))
+        if not PY3:
+            _name = _name.encode('UTF-8')
+        fn.__name__ = _name
         module_name = fn.__module__
         if module_name.startswith("hy.core"):
             module_name = None
-        _hy_tag[module_name][name] = fn
+        _hy_tag[module_name][mangle(name)] = fn
 
         return fn
     return _
@@ -89,14 +97,15 @@ def require(source_module, target_module,
     seen_names = set()
     if prefix:
         prefix += "."
+    assignments = {mangle(str_type(k)): v for k, v in assignments.items()}
 
     for d in _hy_macros, _hy_tag:
         for name, macro in d[source_module].items():
             seen_names.add(name)
             if all_macros:
-                d[target_module][prefix + name] = macro
+                d[target_module][mangle(prefix + name)] = macro
             elif name in assignments:
-                d[target_module][prefix + assignments[name]] = macro
+                d[target_module][mangle(prefix + assignments[name])] = macro
 
     if not all_macros:
         unseen = frozenset(assignments.keys()).difference(seen_names)
@@ -178,6 +187,7 @@ def macroexpand_1(tree, compiler):
         opts = {}
 
         if isinstance(fn, HySymbol):
+            fn = mangle(str_type(fn))
             m = _hy_macros[compiler.module_name].get(fn)
             if m is None:
                 m = _hy_macros[None].get(fn)

hy/models.py

@@ -338,7 +338,7 @@ class HyCons(HyObject):
             # Keep unquotes in the cdr of conses
             if type(cdr) == HyExpression:
                 if len(cdr) > 0 and type(cdr[0]) == HySymbol:
-                    if cdr[0] in ("unquote", "unquote_splice"):
+                    if cdr[0] in ("unquote", "unquote-splice"):
                         return super(HyCons, cls).__new__(cls)
 
             return cdr.__class__([wrap_value(car)] + cdr)

tests/compilers/test_ast.py

@@ -596,13 +596,11 @@ def test_invalid_list_comprehension():
 
 def test_bad_setv():
     """Ensure setv handles error cases"""
-    cant_compile("(setv if* 1)")
     cant_compile("(setv (a b) [1 2])")
 
 
 def test_defn():
     """Ensure that defn works correctly in various corner cases"""
-    cant_compile("(defn if* [] 1)")
     cant_compile("(defn \"hy\" [] 1)")
     cant_compile("(defn :hy [] 1)")
     can_compile("(defn &hy [] 1)")
@@ -611,7 +609,6 @@ def test_defn():
 def test_setv_builtins():
     """Ensure that assigning to a builtin fails, unless in a class"""
     cant_compile("(setv None 42)")
-    cant_compile("(defn get [&rest args] 42)")
     can_compile("(defclass A [] (defn get [self] 42))")
     can_compile("""
     (defclass A []

tests/native_tests/language.hy

@@ -65,19 +65,19 @@
 
 
 (defn test-setv-builtin []
-  "NATIVE: test that setv doesn't work on builtins"
+  "NATIVE: test that setv doesn't work on names Python can't assign to
-  (try (eval '(setv False 1))
+  and that we can't mangle"
-       (except [e [TypeError]] (assert (in "Can't assign to a builtin" (str e)))))
-  (try (eval '(setv True 0))
-       (except [e [TypeError]] (assert (in "Can't assign to a builtin" (str e)))))
   (try (eval '(setv None 1))
-       (except [e [TypeError]] (assert (in "Can't assign to a builtin" (str e)))))
+       (except [e [TypeError]] (assert (in "Can't assign to" (str e)))))
-  (try (eval '(defn defclass [] (print "hello")))
+  (try (eval '(defn None [] (print "hello")))
-       (except [e [TypeError]] (assert (in "Can't assign to a builtin" (str e)))))
+       (except [e [TypeError]] (assert (in "Can't assign to" (str e)))))
-  (try (eval '(defn get [] (print "hello")))
+  (when PY3
-       (except [e [TypeError]] (assert (in "Can't assign to a builtin" (str e)))))
+    (try (eval '(setv False 1))
-  (try (eval '(defn fn [] (print "hello")))
+         (except [e [TypeError]] (assert (in "Can't assign to" (str e)))))
-       (except [e [TypeError]] (assert (in "Can't assign to a builtin" (str e))))))
+    (try (eval '(setv True 0))
+         (except [e [TypeError]] (assert (in "Can't assign to" (str e)))))
+    (try (eval '(defn True [] (print "hello")))
+         (except [e [TypeError]] (assert (in "Can't assign to" (str e)))))))
 
 
 (defn test-setv-pairs []
@@ -187,14 +187,6 @@
          (assert (in "takes a parameter list as second" (str e))))))
 
 
-(defn test-alias-names-in-errors []
-  "NATIVE: tests that native aliases show the correct names in errors"
-  (try (eval '(list-comp 1 2 3 4))
-       (except [e [Exception]] (assert (in "list_comp" (str e)))))
-  (try (eval '(set-comp 1 2 3 4))
-       (except [e [Exception]] (assert (in "set_comp" (str e))))))
-
-
 (defn test-for-loop []
   "NATIVE: test for loops"
   (setv count1 0 count2 0)
@@ -223,14 +215,14 @@
 
   ; don't be fooled by constructs that look like else
   (setv s "")
-  (setv (get (globals) "else") True)
+  (setv else True)
   (for [x "abcde"]
     (+= s x)
     [else (+= s "_")])
   (assert (= s "a_b_c_d_e_"))
 
   (setv s "")
-  (setv (get (globals) "else") True)
+  (setv else True)
   (with [(pytest.raises TypeError)]
     (for [x "abcde"]
       (+= s x)
@@ -329,7 +321,7 @@
   ; don't be fooled by constructs that look like else clauses
   (setv x 2)
   (setv a [])
-  (setv (get (globals) "else") True)
+  (setv else True)
   (while x
     (.append a x)
     (-= x 1)
@@ -738,13 +730,6 @@
 
   (assert (= x 2)))
 
-(defn test-earmuffs []
-  "NATIVE: Test earmuffs"
-  (setv *foo* "2")
-  (setv foo "3")
-  (assert (= *foo* FOO))
-  (assert (!= *foo* foo)))
-
 
 (defn test-threading []
   "NATIVE: test threading macro"
@@ -1112,27 +1097,6 @@
   (assert (= ((fn [] (-> 2 (+ 1 1) (* 1 2)))) 8)))
 
 
-(defn test-symbol-utf-8 []
-  "NATIVE: test symbol encoded"
-  (setv ♥ "love"
-        ⚘ "flower")
-  (assert (= (+ ⚘ ♥) "flowerlove")))
-
-
-(defn test-symbol-dash []
-  "NATIVE: test symbol encoded"
-  (setv ♥-♥ "doublelove"
-        -_- "what?")
-  (assert (= ♥-♥ "doublelove"))
-  (assert (= -_- "what?")))
-
-
-(defn test-symbol-question-mark []
-  "NATIVE: test foo? -> is_foo behavior"
-  (setv foo? "nachos")
-  (assert (= is_foo "nachos")))
-
-
 (defn test-and []
   "NATIVE: test the and function"
 
@@ -1260,11 +1224,7 @@
   (assert (= : :))
   (assert (keyword? :))
   (assert (!= : ":"))
-  (assert (= (name :) ""))
+  (assert (= (name :) "")))
-
-  (defn f [&kwargs kwargs]
-    (list (.items kwargs)))
-  (assert (= (f : 3) [(, "" 3)])))
 
 
 (defn test-nested-if []
@@ -1816,4 +1776,4 @@ macros()
 (defn test-relative-import []
   "Make sure relative imports work properly"
   (import [..resources [tlib]])
-  (assert (= tlib.*secret-message* "Hello World")))
+  (assert (= tlib.SECRET-MESSAGE "Hello World")))

tests/native_tests/mangling.hy

@@ -0,0 +1,189 @@
+;; Copyright 2018 the authors.
+;; This file is part of Hy, which is free software licensed under the Expat
+;; license. See the LICENSE.
+
+
+(import [hy._compat [PY3]])
+
+
+(defn test-hyphen []
+  (setv a-b 1)
+  (assert (= a-b 1))
+  (assert (= a_b 1))
+  (setv -a-_b- 2)
+  (assert (= -a-_b- 2))
+  (assert (= -a--b- 2))
+  (assert (= -a__b- 2))
+  (setv -_- 3)
+  (assert (= -_- 3))
+  (assert (= --- 3))
+  (assert (= ___ 3)))
+
+
+(defn test-underscore-number []
+  (setv _42 3)
+  (assert (= _42 3))
+  (assert (!= _42 -42))
+  (assert (not (in "_hyx_42" (locals)))))
+
+
+(defn test-question-mark []
+  (setv foo? "nachos")
+  (assert (= foo? "nachos"))
+  (assert (= is_foo "nachos"))
+  (setv ___ab_cd? "tacos")
+  (assert (= ___ab_cd? "tacos"))
+  (assert (= ___is_ab_cd "tacos")))
+
+
+(defn test-py-forbidden-ascii []
+
+  (setv # "no comment")
+  (assert (= # "no comment"))
+  (if PY3
+    (assert (= hyx_Δnumber_signΔ "no comment"))
+    (assert (= hyx_Xnumber_signX "no comment")))
+
+  (setv $ "dosh")
+  (assert (= $ "dosh"))
+  (if PY3
+    (assert (= hyx_Δdollar_signΔ "dosh"))
+    (assert (= hyx_Xdollar_signX "dosh"))))
+
+
+(defn test-basic-multilingual-plane []
+  (setv ♥ "love"
+        ⚘ab "flower")
+  (assert (= (+ ⚘ab ♥) "flowerlove"))
+  (if PY3
+    (assert (= (+ hyx_ΔflowerΔab hyx_Δblack_heart_suitΔ) "flowerlove"))
+    (assert (= (+ hyx_XflowerXab hyx_Xblack_heart_suitX) "flowerlove")))
+  (setv ⚘-⚘ "doubleflower")
+  (assert (= ⚘-⚘ "doubleflower"))
+  (if PY3
+    (assert (= hyx_ΔflowerΔ_ΔflowerΔ "doubleflower"))
+    (assert (= hyx_XflowerX_XflowerX "doubleflower")))
+  (setv ⚘? "mystery")
+  (assert (= ⚘? "mystery"))
+  (if PY3
+    (assert (= hyx_is_ΔflowerΔ "mystery"))
+    (assert (= hyx_is_XflowerX "mystery"))))
+
+
+(defn test-higher-unicode []
+  (setv 😂 "emoji")
+  (assert (= 😂 "emoji"))
+  (if PY3
+    (assert (= hyx_Δface_with_tears_of_joyΔ "emoji"))
+    (assert (= hyx_XU1f602X "emoji"))))
+
+
+(defn test-nameless-unicode []
+  (setv  "private use")
+  (assert (=  "private use"))
+  (if PY3
+    (assert (= hyx_ΔUe000Δ "private use"))
+    (assert (= hyx_XUe000X "private use"))))
+
+
+(defn test-charname-with-hyphen []
+  (setv a<b "little")
+  (assert (= a<b "little"))
+  (if PY3
+    (assert (= hyx_aΔlessHthan_signΔb "little"))
+    (assert (= hyx_aXlessHthan_signXb "little"))))
+
+
+(defn test-delimiters []
+  (setv Δ✈ "Delta Air Lines")
+  (assert (= Δ✈ "Delta Air Lines"))
+  (if PY3
+    (assert (= hyx_Δgreek_capital_letter_deltaΔΔairplaneΔ "Delta Air Lines"))
+    (assert (= hyx_Xgreek_capital_letter_deltaXXairplaneX "Delta Air Lines")))
+  (setv X☠ "treasure")
+  (if PY3
+    (assert (= hyx_XΔskull_and_crossbonesΔ "treasure"))
+    (assert (= hyx_Xlatin_capital_letter_xXXskull_and_crossbonesX "treasure"))))
+
+
+(defmacro m---x [form]
+  [form form])
+(defn test-macro []
+  (setv x "")
+  (assert (= (m---x (do (+= x "a") 1)) [1 1]))
+  (assert (= (m___x (do (+= x "b") 2)) [2 2]))
+  (assert (= x "aabb")))
+
+
+(deftag tm---x [form]
+  [form form])
+(defn test-tag-macro []
+  (setv x "")
+  (assert (= #tm---x (do (+= x "a") 1) [1 1]))
+  (assert (= #tm___x (do (+= x "b") 2) [2 2]))
+  (assert (= x "aabb")))
+
+
+(defn test-special-form []
+  (setv not-in 1)
+  ; We set the variable to make sure that if this test works, it's
+  ; because we're calling the special form instead of the shadow
+  ; function.
+  (assert (is (not-in 2 [1 2 3]) False))
+  (assert (is (not_in 2 [1 2 3]) False)))
+
+
+(defn test-python-keyword []
+  (setv if 3)
+  (assert (= if 3))
+  (assert (= hyx_if 3)))
+
+
+(defn test-operator []
+  (setv + 3)
+  (assert (= + 3))
+  (if PY3
+    (assert (= hyx_Δplus_signΔ 3))
+    (assert (= hyx_Xplus_signX 3))))
+
+
+(defn test-keyword-args []
+
+  (defn f [a a-b foo? ☘]
+    [a a-b foo? ☘])
+  (assert (= (f :foo? 3 :☘ 4 :a 1 :a-b 2) [1 2 3 4]))
+  (if PY3
+    (assert (= (f :is_foo 3 :hyx_ΔshamrockΔ 4 :a 1 :a_b 2) [1 2 3 4]))
+    (assert (= (f :is_foo 3 :hyx_XshamrockX 4 :a 1 :a_b 2) [1 2 3 4])))
+
+  (defn g [&kwargs x]
+    x)
+  (setv sk (.format "hyx_{0}shamrock{0}" (if PY3 "Δ" "X")))
+  (assert (= (g :foo? 3 :☘ 4 :a 1 :a-b 2)
+    {"a" 1  "a_b" 2  "is_foo" 3  sk 4}))
+  (if PY3
+    (assert (= (g :is_foo 3 :hyx_ΔshamrockΔ 4 :a 1 :a_b 2)
+      {"a" 1  "a_b" 2  "is_foo" 3  sk 4}))
+    (assert (= (g :is_foo 3 :hyx_XshamrockX 4 :a 1 :a_b 2)
+      {"a" 1  "a_b" 2  "is_foo" 3  sk 4}))))
+
+
+(defn test-late-mangling []
+  ; Mangling should only happen during compilation.
+  (assert (!= 'foo? 'is_foo))
+  (setv sym 'foo?)
+  (assert (= sym "foo?"))
+  (assert (!= sym "is_foo"))
+  (setv out (eval `(do
+    (setv ~sym 10)
+    [foo? is_foo])))
+  (assert (= out [10 10])))
+
+
+(defn test-functions []
+  (for [[a b] [
+      ["---ab-cd?" "___is_ab_cd"]
+      ["if" "hyx_if"]
+      ["⚘-⚘" (if PY3 "hyx_ΔflowerΔ_ΔflowerΔ" "hyx_XflowerX_XflowerX")]]]
+    (assert (= (mangle a) b))
+    (assert (= (unmangle b) a))))

tests/native_tests/native_macros.hy

@@ -162,9 +162,9 @@
   (setv _ast2 (import_buffer_to_ast macro1 "foo"))
   (setv s1 (to_source _ast1))
   (setv s2 (to_source _ast2))
-  ;; and make sure there is something new that starts with :G_
+  ;; and make sure there is something new that starts with _;G|
-  (assert (in "_;G|" s1))
+  (assert (in (mangle "_;G|") s1))
-  (assert (in "_;G|" s2))
+  (assert (in (mangle "_;G|") s2))
   ;; but make sure the two don't match each other
   (assert (not (= s1 s2))))
 
@@ -188,8 +188,8 @@
   (setv _ast2 (import_buffer_to_ast macro1 "foo"))
   (setv s1 (to_source _ast1))
   (setv s2 (to_source _ast2))
-  (assert (in "_;a|" s1))
+  (assert (in (mangle "_;a|") s1))
-  (assert (in "_;a|" s2))
+  (assert (in (mangle "_;a|") s2))
   (assert (not (= s1 s2))))
 
 (defn test-defmacro-g! []

tests/native_tests/operators.hy

@@ -247,9 +247,9 @@
   (forbid (f))
   (forbid (f "hello"))
   (defclass C)
-  (setv x (get {"is_not" (C) "!=" 0} f-name))
+  (setv x (get {"is-not" (C) "!=" 0} f-name))
-  (setv y (get {"is_not" (C) "!=" 1} f-name))
+  (setv y (get {"is-not" (C) "!=" 1} f-name))
-  (setv z (get {"is_not" (C) "!=" 2} f-name))
+  (setv z (get {"is-not" (C) "!=" 2} f-name))
   (assert (is (f x x) False))
   (assert (is (f y y) False))
   (assert (is (f x y) True))

tests/test_bin.py

@@ -74,6 +74,11 @@ def test_bin_hy_stdin_multiline():
     assert "'abcd'" in output
 
 
+def test_bin_hy_history():
+    output, _ = run_cmd("hy", '(+ "a" "b")\n(+ *1 "y" "z")')
+    assert "'abyz'" in output
+
+
 def test_bin_hy_stdin_comments():
     _, err_empty = run_cmd("hy", '')
 

tests/test_lex.py

@@ -121,8 +121,8 @@ def test_lex_nan_and_inf():
     assert tokenize("INF") == [HySymbol("INF")]
 
     assert tokenize("-Inf") == [HyFloat(float("-inf"))]
-    assert tokenize("-inf") == [HySymbol("_inf")]
+    assert tokenize("-inf") == [HySymbol("-inf")]
-    assert tokenize("-INF") == [HySymbol("_INF")]
+    assert tokenize("-INF") == [HySymbol("-INF")]
 
 
 def test_lex_expression_complex():
@@ -140,7 +140,7 @@ def test_lex_expression_complex():
     assert t("nanj") == f(HySymbol("nanj"))
     assert t("Inf+Infj") == f(HyComplex(complex(float("inf"), float("inf"))))
     assert t("Inf-Infj") == f(HyComplex(complex(float("inf"), float("-inf"))))
-    assert t("Inf-INFj") == f(HySymbol("Inf_INFj"))
+    assert t("Inf-INFj") == f(HySymbol("Inf-INFj"))
 
 
 def test_lex_digit_separators():
@@ -332,7 +332,7 @@ def test_complex():
 def test_tag_macro():
     """Ensure tag macros are handled properly"""
     entry = tokenize("#^()")
-    assert entry[0][0] == HySymbol("dispatch_tag_macro")
+    assert entry[0][0] == HySymbol("dispatch-tag-macro")
     assert entry[0][1] == HyString("^")
     assert len(entry[0]) == 3
 
@@ -343,78 +343,6 @@ def test_lex_comment_382():
     assert entry == [HySymbol("foo")]
 
 
-def test_lex_mangling_star():
-    """Ensure that mangling starred identifiers works according to plan"""
-    entry = tokenize("*foo*")
-    assert entry == [HySymbol("FOO")]
-    entry = tokenize("*")
-    assert entry == [HySymbol("*")]
-    entry = tokenize("*foo")
-    assert entry == [HySymbol("*foo")]
-
-
-def test_lex_mangling_hyphen():
-    """Ensure that hyphens get translated to underscores during mangling"""
-    entry = tokenize("foo-bar")
-    assert entry == [HySymbol("foo_bar")]
-    entry = tokenize("-")
-    assert entry == [HySymbol("-")]
-
-
-def test_lex_mangling_qmark():
-    """Ensure that identifiers ending with a question mark get mangled ok"""
-    entry = tokenize("foo?")
-    assert entry == [HySymbol("is_foo")]
-    entry = tokenize("?")
-    assert entry == [HySymbol("?")]
-    entry = tokenize("im?foo")
-    assert entry == [HySymbol("im?foo")]
-    entry = tokenize(".foo?")
-    assert entry == [HySymbol(".is_foo")]
-    entry = tokenize("foo.bar?")
-    assert entry == [HySymbol("foo.is_bar")]
-    entry = tokenize("foo?.bar")
-    assert entry == [HySymbol("is_foo.bar")]
-    entry = tokenize(".foo?.bar.baz?")
-    assert entry == [HySymbol(".is_foo.bar.is_baz")]
-
-
-def test_lex_mangling_bang():
-    """Ensure that identifiers ending with a bang get mangled ok"""
-    entry = tokenize("foo!")
-    assert entry == [HySymbol("foo_bang")]
-    entry = tokenize("!")
-    assert entry == [HySymbol("!")]
-    entry = tokenize("im!foo")
-    assert entry == [HySymbol("im!foo")]
-    entry = tokenize(".foo!")
-    assert entry == [HySymbol(".foo_bang")]
-    entry = tokenize("foo.bar!")
-    assert entry == [HySymbol("foo.bar_bang")]
-    entry = tokenize("foo!.bar")
-    assert entry == [HySymbol("foo_bang.bar")]
-    entry = tokenize(".foo!.bar.baz!")
-    assert entry == [HySymbol(".foo_bang.bar.baz_bang")]
-
-
-def test_unmangle():
-    import sys
-    f = sys.modules["hy.lex.parser"].hy_symbol_unmangle
-
-    assert f("FOO") == "*foo*"
-    assert f("<") == "<"
-    assert f("FOOa") == "FOOa"
-
-    assert f("foo_bar") == "foo-bar"
-    assert f("_") == "_"
-
-    assert f("is_foo") == "foo?"
-    assert f("is_") == "is-"
-
-    assert f("foo_bang") == "foo!"
-    assert f("_bang") == "-bang"
-
-
 def test_simple_cons():
     """Check that cons gets tokenized correctly"""
     entry = tokenize("(a . b)")[0]