Update NEWS and docs for the new comprehensions

NEWS.rst

@@ -14,6 +14,7 @@ Removals
 * Macros `ap-pipe` and `ap-compose` have been removed.
   Anaphoric macros do not work well with point-free style programming,
   in which case both threading macros and `comp` are more adequate.
+* `for/a` has been removed. Use `(for [:async ...] ...)` instead.
 
 Other Breaking Changes
 ------------------------------
@@ -30,6 +31,10 @@ Other Breaking Changes
 * Non-shadow unary `=`, `is`, `<`, etc. now evaluate their argument
   instead of ignoring it. This change increases consistency a bit
   and makes accidental unary uses easier to notice.
+* `list-comp`, `set-comp`, `dict-comp`, and `genexpr` have been replaced
+  by `lfor`, `sfor`, `dfor`, and `gfor`, respectively, which use a new
+  syntax and have additional features. All Python comprehensions can now
+  be written in Hy.
 * `hy-repr` uses registered functions instead of methods
 * `HyKeyword` no longer inherits from the string type and has been
   made into its own object type.
@@ -47,6 +52,7 @@ New Features
   keyword arguments
 * Added a command-line option `-E` per CPython
 * `while` and `for` are allowed to have empty bodies
+* `for` supports the various new clause types offered by `lfor`
 * Added a new module ``hy.model_patterns``
 
 Bug Fixes

docs/extra/reserved.rst

@@ -10,7 +10,7 @@ Usage: ``(names)``
 This function can be used to get a list (actually, a ``frozenset``) of the
 names of Hy's built-in functions, macros, and special forms. The output
 also includes all Python reserved words. All names are in unmangled form
-(e.g., ``list-comp`` rather than ``list_comp``).
+(e.g., ``not-in`` rather than ``not_in``).
 
 .. code-block:: hy
 

docs/language/api.rst

@@ -321,48 +321,17 @@ is only called on every other value in the list.
       (side-effect2 x))
 
 
-dict-comp
----------
-
-``dict-comp`` is used to create dictionaries. It takes three or four parameters.
-The first two parameters are for controlling the return value (key-value pair)
-while the third is used to select items from a sequence. The fourth and optional
-parameter can be used to filter out some of the items in the sequence based on a
-conditional expression.
-
-.. code-block:: hy
-
-    => (dict-comp x (* x 2) [x (range 10)] (odd? x))
-    {1: 2, 3: 6, 9: 18, 5: 10, 7: 14}
-
-
 do
 ----------
 
-``do`` is used to evaluate each of its arguments and return the
+``do`` (called ``progn`` in some Lisps) takes any number of forms,
-last one. Return values from every other than the last argument are discarded.
+evaluates them, and returns the value of the last one, or ``None`` if no
-It can be used in ``list-comp`` to perform more complex logic as shown in one
+forms were provided.
-of the following examples.
 
-Some example usage:
+::
 
-.. code-block:: clj
+    => (+ 1 (do (setv x (+ 1 1)) x))
-
+    3
-    => (if True
-    ...  (do (print "Side effects rock!")
-    ...      (print "Yeah, really!")))
-    Side effects rock!
-    Yeah, really!
-
-    ;; assuming that (side-effect) is a function that we want to call for each
-    ;; and every value in the list, but whose return value we do not care about
-    => (list-comp (do (side-effect x)
-    ...               (if (< x 5) (* 2 x)
-    ...                   (* 4 x)))
-    ...           (x (range 10)))
-    [0, 2, 4, 6, 8, 20, 24, 28, 32, 36]
-
-``do`` can accept any number of arguments, from 1 to n.
 
 
 doc / #doc
@@ -400,6 +369,20 @@ Gets help for macros or tag macros, respectively.
     Gets help for a tag macro function available in this module.
 
 
+dfor
+----
+
+``dfor`` creates a :ref:`dictionary comprehension <py:dict>`. Its syntax
+is the same as that of `lfor`_ except that the final value form must be
+a literal list of two elements, the first of which becomes each key and
+the second of which becomes each value.
+
+.. code-block:: hy
+
+    => (dfor x (range 5) [x (* x 10)])
+    {0: 0, 1: 10, 2: 20, 3: 30, 4: 40}
+
+
 setv
 ----
 
@@ -524,8 +507,8 @@ Parameters may have the following keywords in front of them:
     .. code-block:: clj
 
         => (defn zig-zag-sum [&rest numbers]
-             (setv odd-numbers (list-comp x [x numbers] (odd? x))
+             (setv odd-numbers (lfor x numbers :if (odd? x) x)
-                   even-numbers (list-comp x [x numbers] (even? x)))
+                   even-numbers (lfor x numbers :if (even? x) x))
              (- (sum odd-numbers) (sum even-numbers)))
 
         => (zig-zag-sum)
@@ -850,24 +833,39 @@ raising an exception.
 for
 ---
 
-``for`` is used to call a function for each element in a list or vector.
+``for`` is used to evaluate some forms for each element in an iterable
-The results of each call are discarded and the ``for`` expression returns
+object, such as a list. The return values of the forms are discarded and
-``None`` instead. The example code iterates over *collection* and for each
+the ``for`` form returns ``None``.
-*element* in *collection* calls the ``side-effect`` function with *element*
-as its argument:
 
-.. code-block:: clj
+::
 
-    ;; assuming that (side-effect) is a function that takes a single parameter
+    => (for [x [1 2 3]]
-    (for [element collection] (side-effect element))
+    ...  (print "iterating")
+    ...  (print x))
+    iterating
+    1
+    iterating
+    2
+    iterating
+    3
 
-    ;; for can have an optional else block
+In its square-bracketed first argument, ``for`` allows the same types of
-    (for [element collection] (side-effect element)
+clauses as lfor_.
-         (else (side-effect-2)))
 
-The optional ``else`` block is only executed if the ``for`` loop terminates
+::
-normally. If the execution is halted with ``break``, the ``else`` block does
+
-not execute.
+   => (for [x [1 2 3]  :if (!= x 2)  y [7 8]]
+   ...  (print x y))
+   1 7
+   1 8
+   3 7
+   3 8
+
+Furthermore, the last argument of ``for`` can be an ``(else …)`` form.
+This form is executed after the last iteration of the ``for``\'s
+outermost iteration clause, but only if that outermost loop terminates
+normally. If it's jumped out of with e.g. ``break``, the ``else`` is
+ignored.
 
 .. code-block:: clj
 
@@ -888,43 +886,6 @@ not execute.
     loop finished
 
 
-for/a
------
-
-``for/a`` behaves like ``for`` but is used to call a function for each
-element generated by an asynchronous generator expression. The results
-of each call are discarded and the ``for/a`` expression returns
-``None`` instead.
-
-.. code-block:: clj
-
-    ;; assuming that (side-effect) is a function that takes a single parameter
-    (for/a [element (agen)] (side-effect element))
-
-    ;; for/a can have an optional else block
-    (for/a [element (agen)] (side-effect element)
-         (else (side-effect-2)))
-
-
-genexpr
--------
-
-``genexpr`` is used to create generator expressions. It takes two or three
-parameters. The first parameter is the expression controlling the return value,
-while the second is used to select items from a list. The third and optional
-parameter can be used to filter out some of the items in the list based on a
-conditional expression. ``genexpr`` is similar to ``list-comp``, except it
-returns an iterable that evaluates values one by one instead of evaluating them
-immediately.
-
-.. code-block:: hy
-
-    => (setv collection (range 10))
-    => (setv filtered (genexpr x [x collection] (even? x)))
-    => (list filtered)
-    [0, 2, 4, 6, 8]
-
-
 .. _gensym:
 
 gensym
@@ -977,6 +938,24 @@ successive elements in a nested structure. Example usage:
           index that is out of bounds.
 
 
+gfor
+----
+
+``gfor`` creates a :ref:`generator expression <py:genexpr>`. Its syntax
+is the same as that of `lfor`_. The difference is that ``gfor`` returns
+an iterator, which evaluates and yields values one at a time.
+
+::
+
+    => (setv accum [])
+    => (list (take-while
+    ...  (fn [x] (< x 5))
+    ...  (gfor x (count) :do (.append accum x) x)))
+    [0, 1, 2, 3, 4]
+    => accum
+    [0, 1, 2, 3, 4, 5]
+
+
 global
 ------
 
@@ -1190,27 +1169,69 @@ last
     6
 
 
-list-comp
+lfor
----------
+----
 
-``list-comp`` performs list comprehensions. It takes two or three parameters.
+The comprehension forms ``lfor``, `sfor`_, `dfor`_, `gfor`_, and `for`_
-The first parameter is the expression controlling the return value, while
+are used to produce various kinds of loops, including Python-style
-the second is used to select items from a list. The third and optional
+:ref:`comprehensions <py:comprehensions>`. ``lfor`` in particular
-parameter can be used to filter out some of the items in the list based on a
+creates a list comprehension. A simple use of ``lfor`` is::
-conditional expression. Some examples:
 
-.. code-block:: clj
+    => (lfor x (range 5) (* 2 x))
-
-    => (setv collection (range 10))
-    => (list-comp x [x collection])
-    [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
-
-    => (list-comp (* x 2) [x collection])
-    [0, 2, 4, 6, 8, 10, 12, 14, 16, 18]
-
-    => (list-comp (* x 2) [x collection] (< x 5))
     [0, 2, 4, 6, 8]
 
+``x`` is the name of a new variable, which is bound to each element of
+``(range 5)``. Each such element in turn is used to evaluate the value
+form ``(* 2 x)``, and the results are accumulated into a list.
+
+Here's a more complex example::
+
+    => (lfor
+    ...  x (range 3)
+    ...  y (range 3)
+    ...  :if (!= x y)
+    ...  :setv total (+ x y)
+    ...  [x y total])
+    [[0, 1, 1], [0, 2, 2], [1, 0, 1], [1, 2, 3], [2, 0, 2], [2, 1, 3]]
+
+When there are several iteration clauses (here, the pairs of forms ``x
+(range 3)`` and ``y (range 3)``), the result works like a nested loop or
+Cartesian product: all combinations are considered in lexicographic
+order.
+
+The general form of ``lfor`` is::
+
+    (lfor CLAUSES VALUE)
+
+where the ``VALUE`` is an arbitrary form that is evaluated to produce
+each element of the result list, and ``CLAUSES`` is any number of
+clauses. There are several types of clauses:
+
+- Iteration clauses, which look like ``LVALUE ITERABLE``. The ``LVALUE``
+  is usually just a symbol, but could be something more complicated,
+  like ``[x y]``.
+- ``:async LVALUE ITERABLE``, which is an
+  :ref:`asynchronous <py:async for>` form of iteration clause.
+- ``:do FORM``, which simply evaluates the ``FORM``. If you use
+  ``(continue)`` or ``(break)`` here, they will apply to the innermost
+  iteration clause before the ``:do``.
+- ``:setv LVALUE RVALUE``, which is equivalent to ``:do (setv LVALUE
+  RVALUE)``.
+- ``:if CONDITION``, which is equivalent to ``:do (unless CONDITION
+  (continue))``.
+
+For ``lfor``, ``sfor``, ``gfor``, and ``dfor``, variables are scoped as
+if the comprehension form were its own function, so variables defined by
+an iteration clause or ``:setv`` are not visible outside the form. In
+fact, these forms are implemented as generator functions whenever they
+contain Python statements, with the attendant consequences for calling
+``return``. By contrast, ``for`` shares the caller's scope.
+
+.. note:: An exception to the above scoping rules occurs on Python 2 for
+   ``lfor`` specifically (and not ``sfor``, ``gfor``, or ``dfor``) when
+   Hy can implement the ``lfor`` as a Python list comprehension. Then,
+   variables will leak to the surrounding scope.
+
 
 nonlocal
 --------
@@ -1465,20 +1486,12 @@ the end of a function, put ``None`` there yourself.
     => (print (f 4))
     None
 
-set-comp
---------
 
-``set-comp`` is used to create sets. It takes two or three parameters.
+sfor
-The first parameter is for controlling the return value, while the second is
+----
-used to select items from a sequence. The third and optional parameter can be
-used to filter out some of the items in the sequence based on a conditional
-expression.
 
-.. code-block:: hy
+``sfor`` creates a set comprehension. ``(sfor CLAUSES VALUE)`` is
-
+equivalent to ``(set (lfor CLAUSES VALUE))``. See `lfor`_.
-    => (setv data [1 2 3 4 5 2 3 4 5 3 4 5])
-    => (set-comp x [x data] (odd? x))
-    {1, 3, 5}
 
 
 cut
@@ -1944,13 +1957,13 @@ infinite series without consuming infinite amount of memory.
     => (multiply (range 5) (range 5))
     <generator object multiply at 0x978d8ec>
 
-    => (list-comp value [value (multiply (range 10) (range 10))])
+    => (list (multiply (range 10) (range 10)))
     [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
 
     => (import random)
     => (defn random-numbers [low high]
     ...  (while True (yield (.randint random low high))))
-    => (list-comp x [x (take 15 (random-numbers 1 50))])
+    => (list (take 15 (random-numbers 1 50)))
     [7, 41, 6, 22, 32, 17, 5, 38, 18, 38, 17, 14, 23, 23, 19]
 
 

docs/tutorial.rst

@@ -378,21 +378,20 @@ In Hy, you could do these like:
 .. code-block:: clj
 
   (setv odds-squared
-    (list-comp
+    (lfor
-      (pow num 2)
+      num (range 100)
-      (num (range 100))
+      :if (= (% num 2) 1)
-      (= (% num 2) 1)))
+      (pow num 2)))
-
 
 .. code-block:: clj
 
   ; And, an example stolen shamelessly from a Clojure page:
   ; Let's list all the blocks of a Chessboard:
 
-  (list-comp
+  (lfor
-    (, x y)
+    x (range 8)
-    (x (range 8)
+    y "ABCDEFGH"
-     y "ABCDEFGH"))
+    (, x y))
 
   ; [(0, 'A'), (0, 'B'), (0, 'C'), (0, 'D'), (0, 'E'), (0, 'F'), (0, 'G'), (0, 'H'),
   ;  (1, 'A'), (1, 'B'), (1, 'C'), (1, 'D'), (1, 'E'), (1, 'F'), (1, 'G'), (1, 'H'),