Give require the same features as import (#1142)

Give `require` the same features as `import`

You can now do (require foo), (require [foo [a b c]]), (require [foo [*]]), and (require [foo :as bar]). The first and last forms get you macros named foo.a, foo.b, etc. or bar.a, bar.b, etc., respectively. The second form only gets the macros in the list.

Implements #1118 and perhaps partly addresses #277.

N.B. The new meaning of (require foo) will cause all existing code that uses macros to break. Simply replace these forms with (require [foo [*]]) to get your code working again.

There's a bit of a hack involved in the forms (require foo) or (require [foo :as bar]). When you call (foo.a ...) or (bar.a ...), Hy doesn't actually look inside modules. Instead, these (require ...) forms give the macros names that have periods in them, which happens to work fine with the way Hy finds and interprets macro calls.

* Make `require` syntax stricter and add tests

* Update documentation for `require`

* Documentation wording improvements

* Allow :as in `require` name lists

docs/contrib/anaphoric.rst

@@ -15,7 +15,7 @@ concise and easy to read.
 
 To use these macros you need to require the hy.contrib.anaphoric module like so:
 
-``(require hy.contrib.anaphoric)``
+``(require [hy.contrib.anaphoric [*]])``
 
 .. _ap-if:
 

docs/contrib/flow.rst

@@ -25,7 +25,7 @@ Example:
 
 .. code-block:: hy
 
-    (require hy.contrib.flow)
+    (require [hy.contrib.flow [case]])
 
     (defn temp-commenter [temp]
         (case temp
@@ -48,7 +48,7 @@ Example:
 
 .. code-block:: hy
 
-    (require hy.contrib.flow)
+    (require [hy.contrib.flow [switch]])
 
     (defn temp-commenter [temp]
         (switch temp

docs/contrib/loop.rst

@@ -45,7 +45,7 @@ Example:
 
 .. code-block:: hy
 
-    (require hy.contrib.loop)
+    (require [hy.contrib.loop [loop]])
 
     (defn factorial [n]
       (loop [[i n] [acc 1]]

docs/contrib/multi.rst

@@ -9,7 +9,7 @@ args and/or kwargs. Inspired by Clojure's take on ``defn``.
 
 .. code-block:: clj
 
-    => (require hy.contrib.multi)
+    => (require [hy.contrib.multi [defmulti]])
     =>   (defmulti fun
     ...     ([a] "a")
     ...     ([a b] "a b")

docs/language/api.rst

@@ -1210,16 +1210,84 @@ alternatively be written using the apostrophe (``'``) symbol.
 require
 -------
 
-``require`` is used to import macros from a given module. It takes at least one
+``require`` is used to import macros from one or more given modules. It allows
-parameter specifying the module which macros should be imported. Multiple
+parameters in all the same formats as ``import``. The ``require`` form itself
-modules can be imported with a single ``require``.
+produces no code in the final program: its effect is purely at compile-time, for
+the benefit of macro expansion. Specifically, ``require`` imports each named
+module and then makes each requested macro available in the current module.
 
-The following example will import macros from ``module-1`` and ``module-2``:
+The following are all equivalent ways to call a macro named ``foo`` in the module ``mymodule``:
 
 .. code-block:: clj
 
-    (require module-1 module-2)
+    (require mymodule)
+    (mymodule.foo 1)
 
+    (require [mymodule :as M])
+    (M.foo 1)
+
+    (require [mymodule [foo]])
+    (foo 1)
+
+    (require [mymodule [*]])
+    (foo 1)
+
+    (require [mymodule [foo :as bar]])
+    (bar 1)
+
+Macros that call macros
+~~~~~~~~~~~~~~~~~~~~~~~
+
+One aspect of ``require`` that may be surprising is what happens when one
+macro's expansion calls another macro. Suppose ``mymodule.hy`` looks like this:
+
+.. code-block:: clj
+
+    (defmacro repexpr [n expr]
+      ; Evaluate the expression n times
+      ; and collect the results in a list.
+      `(list (map (fn [_] ~expr) (range ~n))))
+
+    (defmacro foo [n]
+      `(repexpr ~n (input "Gimme some input: ")))
+
+And then, in your main program, you write:
+
+.. code-block:: clj
+
+    (require [mymodule [foo]])
+
+    (print (mymodule.foo 3))
+
+Running this raises ``NameError: name 'repexpr' is not defined``, even though
+writing ``(print (foo 3))`` in ``mymodule`` works fine. The trouble is that your
+main program doesn't have the macro ``repexpr`` available, since it wasn't
+imported (and imported under exactly that name, as opposed to a qualified name).
+You could do ``(require [mymodule [*]])`` or ``(require [mymodule [foo
+repexpr]])``, but a less error-prone approach is to change the definition of
+``foo`` to require whatever sub-macros it needs:
+
+.. code-block:: clj
+
+    (defmacro foo [n]
+      `(do
+        (require mymodule)
+        (mymodule.repexpr ~n (raw-input "Gimme some input: "))))
+
+It's wise to use ``(require mymodule)`` here rather than ``(require [mymodule
+[repexpr]])`` to avoid accidentally shadowing a function named ``repexpr`` in
+the main program.
+
+Qualified macro names
+~~~~~~~~~~~~~~~~~~~~~
+
+Note that in the current implementation, there's a trick in qualified macro
+names, like ``mymodule.foo`` and ``M.foo`` in the above example. These names
+aren't actually attributes of module objects; they're just identifiers with
+periods in them. In fact, ``mymodule`` and ``M`` aren't defined by these
+``require`` forms, even at compile-time. None of this will hurt you unless try
+to do introspection of the current module's set of defined macros, which isn't
+really supported anyway.
 
 rest / cdr
 ----------

docs/tutorial.rst

@@ -568,15 +568,18 @@ Macros are useful when one wishes to extend Hy or write their own
 language on top of that. Many features of Hy are macros, like ``when``,
 ``cond`` and ``->``.
 
-To use macros defined in a different module, it is not enough to
+What if you want to use a macro that's defined in a different
-``import`` the module, because importing happens at run-time, while we
+module? The special form ``import`` won't help, because it merely
-would need macros at compile-time. Instead of importing the module
+translates to a Python ``import`` statement that's executed at
-with macros, ``require`` must be used:
+run-time, and macros are expanded at compile-time, that is,
+during the translate from Hy to Python. Instead, use ``require``,
+which imports the module and makes macros available at
+compile-time. ``require`` uses the same syntax as ``import``.
 
 .. code-block:: clj
 
    => (require tutorial.macros)
-   => (rev (1 2 3 +))
+   => (tutorial.macros.rev (1 2 3 +))
    6
 
 Hy <-> Python interop

hy/cmdline.py

@@ -173,8 +173,8 @@ def ideas_macro():
 
 """)])
 
-require("hy.cmdline", "__console__")
+require("hy.cmdline", "__console__", all_macros=True)
-require("hy.cmdline", "__main__")
+require("hy.cmdline", "__main__", all_macros=True)
 
 SIMPLE_TRACEBACKS = True
 

hy/compiler.py

@@ -1737,10 +1737,45 @@ class HyASTCompiler(object):
         "unimport" it after we've completed `thing' so that we don't pollute
         other envs.
         """
-        expression.pop(0)
+        for entry in expression[1:]:
-        for entry in expression:
+            if isinstance(entry, HySymbol):
-            __import__(entry)  # Import it fo' them macros.
+                # e.g., (require foo)
-            require(entry, self.module_name)
+                __import__(entry)
+                require(entry, self.module_name, all_macros=True,
+                        prefix=entry)
+            elif isinstance(entry, HyList) and len(entry) == 2:
+                # e.g., (require [foo [bar baz :as MyBaz bing]])
+                # or (require [foo [*]])
+                module, names = entry
+                if not isinstance(names, HyList):
+                    raise HyTypeError(names,
+                                      "(require) name lists should be HyLists")
+                __import__(module)
+                if '*' in names:
+                    if len(names) != 1:
+                        raise HyTypeError(names, "* in a (require) name list "
+                                                 "must be on its own")
+                    require(module, self.module_name, all_macros=True)
+                else:
+                    assignments = {}
+                    while names:
+                        if len(names) > 1 and names[1] == HyKeyword(":as"):
+                            k, _, v = names[:3]
+                            del names[:3]
+                            assignments[k] = v
+                        else:
+                            symbol = names.pop(0)
+                            assignments[symbol] = symbol
+                    require(module, self.module_name, assignments=assignments)
+            elif (isinstance(entry, HyList) and len(entry) == 3
+                    and entry[1] == HyKeyword(":as")):
+                # e.g., (require [foo :as bar])
+                module, _, prefix = entry
+                __import__(module)
+                require(module, self.module_name, all_macros=True,
+                        prefix=prefix)
+            else:
+                raise HyTypeError(entry, "unrecognized (require) syntax")
         return Result()
 
     @builds("and")

hy/contrib/curry.hy

@@ -17,4 +17,5 @@
 
 
 (defmacro defnc [name args &rest body]
-  `(def ~name (fnc [~@args] ~@body)))
+  `(do (require hy.contrib.curry)
+       (def ~name (hy.contrib.curry.fnc [~@args] ~@body))))

hy/contrib/loop.hy

@@ -70,7 +70,8 @@
 (defmacro defnr [name lambda-list &rest body]
   (if (not (= (type name) HySymbol))
     (macro-error name "defnr takes a name as first argument"))
-  `(setv ~name (fnr ~lambda-list ~@body)))
+  `(do (require hy.contrib.loop)
+       (setv ~name (hy.contrib.loop.fnr ~lambda-list ~@body))))
 
 
 (defmacro/g! loop [bindings &rest body]
@@ -87,5 +88,6 @@
   ;; and erroring if not is a giant TODO.
   (let [fnargs (map (fn [x] (first x)) bindings)
         initargs (map second bindings)]
-    `(do (defnr ~g!recur-fn [~@fnargs] ~@body)
+    `(do (require hy.contrib.loop)
+         (hy.contrib.loop.defnr ~g!recur-fn [~@fnargs] ~@body)
          (~g!recur-fn ~@initargs))))

hy/contrib/meth.hy

@@ -9,19 +9,23 @@
        (defn ~name ~params
          (do ~@code)))))
 
+(defn rwm [name path method params code]
+  `(do (require hy.contrib.meth)
+       (hy.contrib.meth.route-with-methods ~name ~path ~method ~params ~@code)))
+
 ;; Some macro examples
 (defmacro route [name path params &rest code]
   "Get request"
-  `(route-with-methods ~name ~path ["GET"] ~params ~@code))
+  (rwm name path ["GET"] params code))
 
 (defmacro post-route [name path params &rest code]
   "Post request"
-  `(route-with-methods ~name ~path ["POST"] ~params ~@code))
+  (rwm name path ["POST"] params code))
 
 (defmacro put-route [name path params &rest code]
   "Put request"
-  `(route-with-methods ~name ~path ["PUT"] ~params ~@code))
+  (rwm name path ["PUT"] params code))
 
 (defmacro delete-route [name path params &rest code]
   "Delete request"
-  `(route-with-methods ~name ~path ["DELETE"] ~params ~@code))
+  (rwm name path ["DELETE"] params code))

hy/macros.py

@@ -84,22 +84,36 @@ def reader(name):
     return _
 
 
-def require(source_module, target_module):
+def require(source_module, target_module,
-    """Load the macros from `source_module` in the namespace of
+            all_macros=False, assignments={}, prefix=""):
-    `target_module`.
+    """Load macros from `source_module` in the namespace of
+    `target_module`. `assignments` maps old names to new names, but is
+    ignored if `all_macros` is true. If `prefix` is nonempty, it is
+    prepended to the name of each imported macro. (This means you get
+    macros named things like "mymacromodule.mymacro", which looks like
+    an attribute of a module, although it's actually just a symbol
+    with a period in its name.)
 
     This function is called from the `require` special form in the compiler.
 
     """
-    macros = _hy_macros[source_module]
-    refs = _hy_macros[target_module]
-    for name, macro in macros.items():
-        refs[name] = macro
 
-    readers = _hy_reader[source_module]
+    seen_names = set()
-    reader_refs = _hy_reader[target_module]
+    if prefix:
-    for name, reader in readers.items():
+        prefix += "."
-        reader_refs[name] = reader
+
+    for d in _hy_macros, _hy_reader:
+        for name, macro in d[source_module].items():
+            seen_names.add(name)
+            if all_macros:
+                d[target_module][prefix + name] = macro
+            elif name in assignments:
+                d[target_module][prefix + assignments[name]] = macro
+
+    if not all_macros:
+        unseen = frozenset(assignments.keys()).difference(seen_names)
+        if unseen:
+            raise ImportError("cannot require names: " + repr(list(unseen)))
 
 
 def load_macros(module_name):

tests/compilers/test_ast.py

@@ -249,6 +249,20 @@ def test_ast_good_import_from():
     can_compile("(import [x [y]])")
 
 
+def test_ast_require():
+    "Make sure AST respects (require) syntax"
+    can_compile("(require tests.resources.tlib)")
+    can_compile("(require [tests.resources.tlib [qplah parald]])")
+    can_compile("(require [tests.resources.tlib [*]])")
+    can_compile("(require [tests.resources.tlib :as foobar])")
+    can_compile("(require [tests.resources.tlib [qplah :as quiz]])")
+    can_compile("(require [tests.resources.tlib [qplah :as quiz parald]])")
+    cant_compile("(require [tests.resources.tlib])")
+    cant_compile("(require [tests.resources.tlib [* qplah]])")
+    cant_compile("(require [tests.resources.tlib [qplah *]])")
+    cant_compile("(require [tests.resources.tlib [* *]])")
+
+
 def test_ast_good_get():
     "Make sure AST can compile valid get"
     can_compile("(get x y)")

tests/native_tests/contrib/alias.hy

@@ -1,4 +1,4 @@
-(require hy.contrib.alias)
+(require [hy.contrib.alias [defn-alias]])
 
 (defn test-defn-alias []
   (defn-alias [tda-main tda-a1 tda-a2] [] :bazinga)

tests/native_tests/contrib/anaphoric.hy

@@ -19,7 +19,7 @@
 ;; DEALINGS IN THE SOFTWARE.
 
 (import [hy.errors [HyMacroExpansionError]])
-(require hy.contrib.anaphoric)
+(require [hy.contrib.anaphoric [*]])
 
 ;;;; some simple helpers
 

tests/native_tests/contrib/botsbuildbots.hy

@@ -1,5 +1,4 @@
-(import [hy.contrib.botsbuildbots [*]])
+(require [hy.contrib.botsbuildbots [Botsbuildbots]])
-(require hy.contrib.botsbuildbots)
 
 (defn test-botsbuildbots []
   (assert (> (len (first (Botsbuildbots))) 50)))

tests/native_tests/contrib/curry.hy

@@ -1,4 +1,4 @@
-(require hy.contrib.curry)
+(require [hy.contrib.curry [defnc]])
 
 
 (defnc s [x y z] ((x z) (y z)))  ; λxyz.xz(yz)

tests/native_tests/contrib/loop.hy

@@ -1,4 +1,4 @@
-(require hy.contrib.loop)
+(require [hy.contrib.loop [loop]])
 (import sys)
 
 (defn tco-sum [x y]

tests/native_tests/contrib/meth.hy

@@ -1,4 +1,4 @@
-(require hy.contrib.meth)
+(require [hy.contrib.meth [route post-route put-route delete-route]])
 
 (defclass FakeMeth []
   "Mocking decorator class"

tests/native_tests/contrib/multi.hy

@@ -18,7 +18,7 @@
 ;; FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
 ;; DEALINGS IN THE SOFTWARE.
 
-(require hy.contrib.multi)
+(require [hy.contrib.multi [defmulti]])
 
 
 (defn test-basic-multi []

tests/native_tests/language.hy

@@ -1101,11 +1101,34 @@
 
 (defn test-require []
   "NATIVE: test requiring macros from python code"
-  (try
+  (try (qplah 1 2 3 4)
-    (assert (= "this won't happen" (qplah 1 2 3 4)))
+       (except [NameError] True)
-  (except [NameError]))
+       (else (assert False)))
+  (try (parald 1 2 3 4)
+       (except [NameError] True)
+       (else (assert False)))
+  (require [tests.resources.tlib [qplah]])
+  (assert (= (qplah 1 2 3) [8 1 2 3]))
+  (try (parald 1 2 3 4)
+       (except [NameError] True)
+       (else (assert False)))
   (require tests.resources.tlib)
-  (assert (= [1 2 3] (qplah 1 2 3))))
+  (assert (= (tests.resources.tlib.parald 1 2 3) [9 1 2 3]))
+  (try (parald 1 2 3 4)
+       (except [NameError] True)
+       (else (assert False)))
+  (require [tests.resources.tlib :as T])
+  (assert (= (T.parald 1 2 3) [9 1 2 3]))
+  (try (parald 1 2 3 4)
+       (except [NameError] True)
+       (else (assert False)))
+  (require [tests.resources.tlib [parald :as p]])
+  (assert (= (p 1 2 3) [9 1 2 3]))
+  (try (parald 1 2 3 4)
+       (except [NameError] True)
+       (else (assert False)))
+  (require [tests.resources.tlib [*]])
+  (assert (= (parald 1 2 3) [9 1 2 3])))
 
 
 (defn test-require-native []
@@ -1125,7 +1148,7 @@
                   (assert (= x [3 2 1]))
                   "success")
               (except [NameError] "failure"))))
-  (require tests.native_tests.native_macros)
+  (require [tests.native_tests.native_macros [rev]])
   (assert (= "success"
              (try
               (do (setv x [])

tests/resources/tlib.py

@@ -1,7 +1,12 @@
 from hy.macros import macro
-from hy import HyList
+from hy import HyList, HyInteger
 
 
 @macro("qplah")
 def tmac(*tree):
-    return HyList(tree)
+    return HyList((HyInteger(8), ) + tree)
+
+
+@macro("parald")
+def tmac2(*tree):
+    return HyList((HyInteger(9), ) + tree)