Issue LOOP-INITFORM-ENVIRONMENT Writeup

Issue:        LOOP-INITFORM-ENVIRONMENT

Forum:	      Editorial

References:   LOOP (pages 8-85, 

	      Draft 8.81 

Category:     CLARIFICATION

Edit history: 05-Mar-91, Version 1 by Pitman

	      15-Mar-91, Version 2 by Pitman

Status:	      For X3J13 consideration


Problem Description:


  The Symbolics implementation of X3J13's LOOP spec surprised some users 

  by being incompatible with the old Symbolics Genera LOOP on the 

  following test case:


    (let ((list '(1 2 3)))

      (loop for list = list then (cdr list)

	 until (null list)

	 do (princ (car list))))


  The Symbolics Genera implementation prints nothing, but old Symbolics 

  Genera LOOP prints 123. In double-checking the implementation against the

  working draft specification to determine if the implementation was in

  error, more than one Symbolics implementors did not feel this behavior 

  was clearly enough spelled out and worried that implementations might not

  agree.


  Further investigation showed that Lucid Common Lisp had the same behavior.

  

  The following are the only references which appeared relevant:


   From page 8-85:

 

    The LOOP macro translates the given <form> into code and returns the

    expanded <form>.   The expanded <form> is one or more <lambda

    expressions> for the local <binding> of loop variables and a block

    and a tagbody that express a looping control structure. The variables

    established in LOOP are bound as if by LET or LAMBDA.

    Implementations can interleave the setting of initial values with

    the bindings.  However, the assignment of the initial values is

    always calculated in othe order specified by the user.  A variable is 

    thus sometimes bound to a meaningless value of the correct type

    and then later in the prologue is set to the true initial value by

    using SETQ.

   

   Later, on page 8-86:

   

    The FOR and AS constructs provide iteration control clauses that

    establish a variable to be initialized.  FOR and AS clauses can be

    combined with the LOOP keyword AND to get parallel initialization and

    stepping.  Otherwise, the initialization and stepping are sequential.

   

  The need for this information is so fundamental that it should be very

  plainly stated.  These extremely vague phrases don't really say much

  about the environment, and since they don't say what goes into the LET

  or the LAMBDA, or even how many LET or LAMBDA forms are involved, they

  don't really say much at all.


  Further, the vague statement on p8-85 about how LET+SETQ might be used to

  implement binding leaves an unusually large amount of latitude to 

  implementations.


Proposal (LOOP-INITFORM-ENVIRONMENT:INITFORM-BEFORE-BINDING)


  Clarify that the initforms in a FOR-AS clause

  (each being variously called the <form1>, the <hash-table>, or the

  <package> in the `bnf' on pages 8-82..8-83)

  are all evaluated (in left to right order) prior to establishing

  the binding for any of the the <vars> in the same clause.


  Clarify that the initforms in a WITH clause (each being

  called a <form1> in the `bnf' on pages 8-82..8-83)

  are all evaluated (in left to right order) prior to establishing

  the binding for any <varN> in the same clause.


Test Case:


    (let ((list '(1 2 3)))

      (loop for list = list then (cdr list)

	 until (null list)

	 collect (car list)))

    => (1 2 3)


Rationale:


  This is what most users who make an analogy to LET or DO will expect.


Current Practice:


  Symbolics Genera's FUTURE-COMMON-LISP:LOOP (which purports to conform

  to the draft specification) returns NIL.


  Symbolics Genera's LISP:LOOP (which purports to conform to CLtL)

  returns (1 2 3) for the test case.


Cost to Implementors:


  Hopefully small.


Cost to Users:


  Hopefully most users will consider this the status quo.


Cost of Non-Adoption:


  The specification might be ambiguous on an important point.


Benefits:


  Cost of non-adoption is avoided.


Aesthetics:


  Clarity improves aesthetics.


Discussion:


  Pitman and Moon support this proposal.

  JonL opposes this proposal.


  A long-winded discussion ensued, excerpts of which follow.


  Moon writes:

   ``It's intentional that implementations should have latitude in 

     the expansion of LOOP, but not to the extent that the meaning of

     the program changes!  The problem is that the only thing this

     version of the LOOP specification says about the scope of LOOP

     variables is that they are lexical unless proclaimed special and

     their scope is the loop.  This isn't specific enough.


   ``Also in CLtL2 p.722 there is an example expansion which could be

     taken as an argument against the INITFORM-BEFORE-BINDING proposal.

     However I don't think that example was intended to talk about

     variable scoping.  This example does not appear in the draft ANSI

     CL spec, which is good in my opinion.''


  JonL writes:

   ``I don't see this as a clarification but as a change;  in particular,

     it isn't consistent with the section of CLtL/II that you cite:

     [... page 8-85 ...]

     since it forbids the aforementioned "interleaving".  As Moon said, 

     it is "intentional that implementations should have latitude in

     the expansion of LOOP, ...".  So the question is just how

     importantant is the case of loop variables "shadowing" local

     lexical variables?  That is, I agree that the current spec is

     ambiguous on just when the binding of 'list' occurs in

	  (let ((list '(1 2 3)))

	    (loop for list = list then (cdr list)

	    . . . ))

     but I'm more inclined to say SO WHAT?  Is the world going into

     terminal meltdown just because this case isn't completely portable?

     is LOOP completely unusable *** just because of this lexical

     shadowing ambiguity?    


   ``The point I'm trying to make is not that this case isn't a problem, 

     but that it isn't a *big* problem; and even if it remained

     ambiguous (as was the intention for the draft proposal -- reasons

     cited below), it is nowhere near the magnitude of problem that we 

     face in other areas.


   ``... Lucid's LOOP (unchanged for many years now) was modeled after

     GSB's MIT code, and it has the same behaviour as Genera.  Even if

     the "original designers" intended something here, I think Glenn's

     execution of it might have led people to think otherwise. ...''


  Moon writes:


   ``I don't see this particular issue as some technical issue of angels

     dancing on the heads of pins that is only of real interest to

     specialists.  It's not uncommon for user programs to use the same

     variable name more than once.  I hear that users very often find it

     frustrating to try to use LOOP because there are implementations around

     that do unintuitive things and documentation around that they can't

     decode.  Those problems are so unnecessary.


   ``... I just now loaded up the MIT LOOP, unchanged for nearly five       

     years (unless someone's been changing it without updating the edit

     history at the front) from REAGAN.AI.MIT.EDU, and tried the test case

     from the cleanup issue.  It prints 123, so it conforms to the cleanup.

     This makes me suspect that the MIT LOOP has always had the intuitive

     behavior, but programmers at Lucid and also at Symbolics broke it in the

     process of improving it.  I don't think it's true that Glenn implemented

     it differently from what the cleanup says.

     

   ``So then I tried the CMU implementation referenced in Scott Fahlman's

     recent message.  It, too, conforms to the cleanup, although it seems to

     have its share of bugs (several compiler warnings and errors while

     loading that I fixed up and proceeded past).


   ``... We can always specify a language feature as doing something

     unpredictable, but is that really necessary?''


  JonL writes:


   ``I claim this ambiguity is purposeful -- the "interleaving" of binding

     and initialization clauses, which in some implementations has 

     performance consequences  -- and that the case that brought this 

     ambiguity to public attention is not of great consequence to the

     language.  


   ``Of course, I agree that we must specify *exactly* the behaviour

     of LET when local variables are being shadowed, such as in:

       (let ((x (list x))) . . .)

     because renaming of bound variables is an important concept (not just

     "pinheads dancing on angels").  But LOOP bindings are not documented to

     be *exactly* like LET, and they aren't the primariy means to achieve

     lexical "shadows"; thus I strongly disagree that seeking out and "fixing"

     documented ambiguities in the binding-order for LOOP "shadows" is of high

     priority.''