2012-12-16 21:42:22 8 Comments
How do you access other class variables from a list comprehension within the class definition? The following works in Python 2 but fails in Python 3:
class Foo:
x = 5
y = [x for i in range(1)]
Python 3.2 gives the error:
NameError: global name 'x' is not defined
Trying Foo.x
doesn't work either. Any ideas on how to do this in Python 3?
A slightly more complicated motivating example:
from collections import namedtuple
class StateDatabase:
State = namedtuple('State', ['name', 'capital'])
db = [State(*args) for args in [
['Alabama', 'Montgomery'],
['Alaska', 'Juneau'],
# ...
]]
In this example, apply()
would have been a decent workaround, but it is sadly removed from Python 3.
Related Questions
Sponsored Content
19 Answered Questions
8 Answered Questions
[SOLVED] Generator Expressions vs. List Comprehension
- 2008-09-06 20:07:59
- Readonly
- 122255 View
- 361 Score
- 8 Answer
- Tags: python list-comprehension generator
14 Answered Questions
[SOLVED] list comprehension vs. lambda + filter
- 2010-06-10 10:14:00
- Agos
- 585046 View
- 701 Score
- 14 Answer
- Tags: python list functional-programming filter lambda
12 Answered Questions
[SOLVED] How do I access environment variables from Python?
- 2011-02-05 13:03:35
- Amit Yadav
- 1261748 View
- 1493 Score
- 12 Answer
- Tags: python environment-variables
17 Answered Questions
[SOLVED] Are static class variables possible?
- 2008-09-16 01:46:36
- Andrew Walker
- 1032753 View
- 1643 Score
- 17 Answer
- Tags: python class static class-variables
11 Answered Questions
[SOLVED] Create a dictionary with list comprehension in Python
- 2009-11-17 10:07:53
- flybywire
- 686375 View
- 1116 Score
- 11 Answer
- Tags: python dictionary list-comprehension language-features dict-comprehension
9 Answered Questions
[SOLVED] List comprehension vs map
- 2009-08-07 23:43:31
- TimothyAWiseman
- 160199 View
- 600 Score
- 9 Answer
- Tags: python list-comprehension map-function
15 Answered Questions
5 Answered Questions
[SOLVED] if/else in Python's list comprehension?
- 2010-11-23 19:56:44
- AP257
- 481886 View
- 610 Score
- 5 Answer
- Tags: python list-comprehension
8 Answered Questions
[SOLVED] if else in a list comprehension
- 2010-12-10 06:45:51
- user225312
- 343536 View
- 317 Score
- 8 Answer
- Tags: python list list-comprehension if-statement
4 comments
@bzip2 2018-12-30 12:00:12
This is a bug in Python. Comprehensions are advertised as being equivalent to for loops, but this is not true in classes. At least up to Python 3.6.6, in a comprehension used in a class, only one variable from outside the comprehension is accessible inside the comprehension, and it must be used as the outermost iterator. In a function, this scope limitation does not apply.
To illustrate why this is a bug, let's return to the original example. This fails:
But this works:
The limitation is stated at the end of this section in the reference guide.
@Martijn Pieters 2012-12-17 12:11:56
Class scope and list, set or dictionary comprehensions, as well as generator expressions do not mix.
The why; or, the official word on this
In Python 3, list comprehensions were given a proper scope (local namespace) of their own, to prevent their local variables bleeding over into the surrounding scope (see Python list comprehension rebind names even after scope of comprehension. Is this right?). That's great when using such a list comprehension in a module or in a function, but in classes, scoping is a little, uhm, strange.
This is documented in pep 227:
and in the
class
compound statement documentation:Emphasis mine; the execution frame is the temporary scope.
Because the scope is repurposed as the attributes on a class object, allowing it to be used as a nonlocal scope as well leads to undefined behaviour; what would happen if a class method referred to
x
as a nested scope variable, then manipulatesFoo.x
as well, for example? More importantly, what would that mean for subclasses ofFoo
? Python has to treat a class scope differently as it is very different from a function scope.Last, but definitely not least, the linked Naming and binding section in the Execution model documentation mentions class scopes explicitly:
So, to summarize: you cannot access the class scope from functions, list comprehensions or generator expressions enclosed in that scope; they act as if that scope does not exist. In Python 2, list comprehensions were implemented using a shortcut, but in Python 3 they got their own function scope (as they should have had all along) and thus your example breaks. Other comprehension types have their own scope regardless of Python version, so a similar example with a set or dict comprehension would break in Python 2.
The (small) exception; or, why one part may still work
There's one part of a comprehension or generator expression that executes in the surrounding scope, regardless of Python version. That would be the expression for the outermost iterable. In your example, it's the
range(1)
:Thus, using
x
in that expression would not throw an error:This only applies to the outermost iterable; if a comprehension has multiple
for
clauses, the iterables for innerfor
clauses are evaluated in the comprehension's scope:This design decision was made in order to throw an error at genexp creation time instead of iteration time when creating the outermost iterable of a generator expression throws an error, or when the outermost iterable turns out not to be iterable. Comprehensions share this behavior for consistency.
Looking under the hood; or, way more detail than you ever wanted
You can see this all in action using the
dis
module. I'm using Python 3.3 in the following examples, because it adds qualified names that neatly identify the code objects we want to inspect. The bytecode produced is otherwise functionally identical to Python 3.2.To create a class, Python essentially takes the whole suite that makes up the class body (so everything indented one level deeper than the
class <name>:
line), and executes that as if it were a function:The first
LOAD_CONST
there loads a code object for theFoo
class body, then makes that into a function, and calls it. The result of that call is then used to create the namespace of the class, its__dict__
. So far so good.The thing to note here is that the bytecode contains a nested code object; in Python, class definitions, functions, comprehensions and generators all are represented as code objects that contain not only bytecode, but also structures that represent local variables, constants, variables taken from globals, and variables taken from the nested scope. The compiled bytecode refers to those structures and the python interpreter knows how to access those given the bytecodes presented.
The important thing to remember here is that Python creates these structures at compile time; the
class
suite is a code object (<code object Foo at 0x10a436030, file "<stdin>", line 2>
) that is already compiled.Let's inspect that code object that creates the class body itself; code objects have a
co_consts
structure:The above bytecode creates the class body. The function is executed and the resulting
locals()
namespace, containingx
andy
is used to create the class (except that it doesn't work becausex
isn't defined as a global). Note that after storing5
inx
, it loads another code object; that's the list comprehension; it is wrapped in a function object just like the class body was; the created function takes a positional argument, therange(1)
iterable to use for its looping code, cast to an iterator. As shown in the bytecode,range(1)
is evaluated in the class scope.From this you can see that the only difference between a code object for a function or a generator, and a code object for a comprehension is that the latter is executed immediately when the parent code object is executed; the bytecode simply creates a function on the fly and executes it in a few small steps.
Python 2.x uses inline bytecode there instead, here is output from Python 2.7:
No code object is loaded, instead a
FOR_ITER
loop is run inline. So in Python 3.x, the list generator was given a proper code object of its own, which means it has its own scope.However, the comprehension was compiled together with the rest of the python source code when the module or script was first loaded by the interpreter, and the compiler does not consider a class suite a valid scope. Any referenced variables in a list comprehension must look in the scope surrounding the class definition, recursively. If the variable wasn't found by the compiler, it marks it as a global. Disassembly of the list comprehension code object shows that
x
is indeed loaded as a global:This chunk of bytecode loads the first argument passed in (the
range(1)
iterator), and just like the Python 2.x version usesFOR_ITER
to loop over it and create its output.Had we defined
x
in thefoo
function instead,x
would be a cell variable (cells refer to nested scopes):The
LOAD_DEREF
will indirectly loadx
from the code object cell objects:The actual referencing looks the value up from the current frame data structures, which were initialized from a function object's
.__closure__
attribute. Since the function created for the comprehension code object is discarded again, we do not get to inspect that function's closure. To see a closure in action, we'd have to inspect a nested function instead:So, to summarize:
A workaround; or, what to do about it
If you were to create an explicit scope for the
x
variable, like in a function, you can use class-scope variables for a list comprehension:The 'temporary'
y
function can be called directly; we replace it when we do with its return value. Its scope is considered when resolvingx
:Of course, people reading your code will scratch their heads over this a little; you may want to put a big fat comment in there explaining why you are doing this.
The best work-around is to just use
__init__
to create an instance variable instead:and avoid all the head-scratching, and questions to explain yourself. For your own concrete example, I would not even store the
namedtuple
on the class; either use the output directly (don't store the generated class at all), or use a global:@ecatmur 2012-12-17 13:48:39
You can also use a lambda to fix the binding:
y = (lambda x=x: [x for i in range(1)])()
@Martijn Pieters 2012-12-17 14:03:12
@ecatmur: Exactly,
lambda
are just anonymous functions, after all.@Martijn Pieters 2013-03-15 14:14:02
@eryksun: Right; I kept it simple on purpose there; I've reworded it a little to make it clear Python looks at closures indirectly.
@Neal Young 2014-08-02 18:30:30
For the record, the work-around that uses a default argument (to a lambda or a function) to pass in the class variable has a gotcha. Namely, it passes the current value of the variable. So, if the variable changes later, and then the lambda or function is called, the lambda or function will be using the old value. This behavior differs from the behavior of a closure (which would capture a reference to the variable, rather than its value), so may be unexpected.
@Jonathan 2015-01-24 22:28:04
If it requires a page of technical information to explain why something doesn't work intuitively, I call that a bug.
@Lutz Prechelt 2016-11-23 09:05:18
@JonathanLeaders: Don't call it a bug, call it a tradeoff. If you want A and B, but can get only one of them, then no matter how you decide, in some situations you will dislike the result. That's life.
@Lutz Prechelt 2016-11-23 09:07:58
@MartijnPieters: +1 for your conscientious use of really helpful subheadings. And another for your extraordinary effort.
@FMc 2018-08-05 02:53:29
The accepted answer provides excellent information, but there appear to be a few other wrinkles here -- differences between list comprehension and generator expressions. A demo that I played around with:
@Jonathan 2015-01-24 22:25:15
Basically it's a problem in Python 3. I hope they change it.
Bugged (works in 2.7):
To workaround it (works in 3+):
@Martijn Pieters 2015-07-16 13:20:40
The problem is present in Python 2 as well, when using generator expressions, as well as with set and dictionary comprehensions. It is not a bug, it is a consequence of how class namespaces work. It'll not change.
@Martijn Pieters 2017-04-14 11:31:57
And I note that your workaround does exactly what my answer already states: create a new scope (a lambda is no different here from using
def
to create a function).@jsbueno 2017-11-06 01:16:45
yep. While it is nice to have an answer with the work-around at a glance, this one incorrecly states the behavior as a bug, when it is a side-effect of the way the language works (and therefore, won't be changed)
@Riaz Rizvi 2018-07-28 01:07:21
This is a different problem, that actually isn't a problem in Python 3. It only occurs in IPython when you call it in embed mode using say
python -c "import IPython;IPython.embed()"
. Run IPython directly using sayipython
and the problem will disappear.