By dwstein


2012-08-24 00:55:21 8 Comments

Toward the end of a program I'm looking to load a specific variable from all the instances of a class into a dictionary.

For example:

class Foo():
    __init__(self):
    x = {}

foo1 = Foo()
foo2 = Foo()
foo...etc.

Let's say the number of instances will vary and I want the x dict from each instance of Foo() loaded into a new dict. How would I do that?

The examples I've seen in SO assume one already has the list of instances.

6 comments

@sfinkens 2018-01-18 18:35:22

You can also solve this problem using a metaclass:

  1. When a class is created (__init__ method of metaclass), add a new instance registry
  2. When a new instance of this class is created (__call__ method of metaclass), add it to the instance registry.

The advantage of this approach is that each class has a registry - even if no instance exists. In contrast, when overriding __new__ (as in Blckknght's answer), the registry is added when the first instance is created.

class MetaInstanceRegistry(type):
    """Metaclass providing an instance registry"""

    def __init__(cls, name, bases, attrs):
        # Create class
        super(MetaInstanceRegistry, cls).__init__(name, bases, attrs)

        # Initialize fresh instance storage
        cls._instances = weakref.WeakSet()

    def __call__(cls, *args, **kwargs):
        # Create instance (calls __init__ and __new__ methods)
        inst = super(MetaInstanceRegistry, cls).__call__(*args, **kwargs)

        # Store weak reference to instance. WeakSet will automatically remove
        # references to objects that have been garbage collected
        cls._instances.add(inst)

        return inst

    def _get_instances(cls, recursive=False):
        """Get all instances of this class in the registry. If recursive=True
        search subclasses recursively"""
        instances = list(cls._instances)
        if recursive:
            for Child in cls.__subclasses__():
                instances += Child._get_instances(recursive=recursive)

        # Remove duplicates from multiple inheritance.
        return list(set(instances))

Usage: Create a registry and subclass it.

class Registry(object):
    __metaclass__ = MetaInstanceRegistry


class Base(Registry):
    def __init__(self, x):
        self.x = x


class A(Base):
    pass


class B(Base):
    pass


class C(B):
    pass


a = A(x=1)
a2 = A(2)
b = B(x=3)
c = C(4)

for cls in [Base, A, B, C]:
    print cls.__name__
    print cls._get_instances()
    print cls._get_instances(recursive=True)
    print

del c
print C._get_instances()

If using abstract base classes from the abc module, just subclass abc.ABCMeta to avoid metaclass conflicts:

from abc import ABCMeta, abstractmethod


class ABCMetaInstanceRegistry(MetaInstanceRegistry, ABCMeta):
    pass


class ABCRegistry(object):
    __metaclass__ = ABCMetaInstanceRegistry


class ABCBase(ABCRegistry):
    __metaclass__ = ABCMeta

    @abstractmethod
    def f(self):
        pass


class E(ABCBase):
    def __init__(self, x):
        self.x = x

    def f(self):
        return self.x

e = E(x=5)
print E._get_instances()

@OrenR61 2013-08-19 19:46:51

You would probably want to use weak references to your instances. Otherwise the class could likely end up keeping track of instances that were meant to have been deleted. A weakref.WeakSet will automatically remove any dead instances from its set.

One way to keep track of instances is with a class variable:

import weakref
class A(object):
    instances = weakref.WeakSet()

    def __init__(self, foo):
        self.foo = foo
        A.instances.add(self)

    @classmethod
    def get_instances(cls):
        return list(A.instances) #Returns list of all current instances

At the end of the program, you can create your dict like this:

foo_vars = {id(instance): instance.foo for instance in A.instances} There is only one list:

>>> a = A(1)
>>> b = A(2)
>>> A.get_instances()
[<inst.A object at 0x100587290>, <inst.A object at 0x100587250>]
>>> id(A.instances)
4299861712
>>> id(a.instances)
4299861712
>>> id(b.instances)
4299861712
>>> a = A(3) #original a will be dereferenced and replaced with new instance
>>> A.get_instances()
[<inst.A object at 0x100587290>, <inst.A object at 0x1005872d0>]   

@omegacore 2016-01-06 19:46:53

Would it be possible to use a dictionary of sorts instead of the WeakSet to allow looking up an instance by key?

@omegacore 2016-01-07 21:53:03

Answering my own question here, yes it is possible. I used the weakvaluedictionary. Seems to work perfectly.

@zezollo 2018-02-04 08:01:17

This is interesting but not completely reliable: when a reference is deleted (del a), it may not be out of the instances' set at next line, especially if an exception has been handled in the mean time. See the question I asked here for more details.

@Walter Mundt 2012-08-24 07:32:05

Another option for quick low-level hacks and debugging is to filter the list of objects returned by gc.get_objects() and generate the dictionary on the fly that way. In CPython that function will return you a (generally huge) list of everything the garbage collector knows about, so it will definitely contain all of the instances of any particular user-defined class.

Note that this is digging a bit into the internals of the interpreter, so it may or may not work (or work well) with the likes of Jython, PyPy, IronPython, etc. I haven't checked. It's also likely to be really slow regardless. Use with caution/YMMV/etc.

However, I imagine that some people running into this question might eventually want to do this sort of thing as a one-off to figure out what's going on with the runtime state of some slice of code that's behaving strangely. This method has the benefit of not affecting the instances or their construction at all, which might be useful if the code in question is coming out of a third-party library or something.

@Blckknght 2012-08-24 03:03:44

@JoelCornett's answer covers the basics perfectly. This is a slightly more complicated version, which might help with a few subtle issues.

If you want to be able to access all the "live" instances of a given class, subclass the following (or include equivalent code in your own base class):

from weakref import WeakSet

class base(object):
    def __new__(cls, *args, **kwargs):
        instance = object.__new__(cls, *args, **kwargs)
        if "instances" not in cls.__dict__:
            cls.instances = WeakSet()
        cls.instances.add(instance)
        return instance

This addresses two possible issues with the simpler implementation that @JoelCornett presented:

  1. Each subclass of base will keep track of its own instances separately. You won't get subclass instances in a parent class's instance list, and one subclass will never stumble over instances of a sibling subclass. This might be undesirable, depending on your use case, but it's probably easier to merge the sets back together than it is to split them apart.

  2. The instances set uses weak references to the class's instances, so if you del or reassign all the other references to an instance elsewhere in your code, the bookkeeping code will not prevent it from being garbage collected. Again, this might not be desirable for some use cases, but it is easy enough to use regular sets (or lists) instead of a weakset if you really want every instance to last forever.

Some handy-dandy test output (with the instances sets always being passed to list only because they don't print out nicely):

>>> b = base()
>>> list(base.instances)
[<__main__.base object at 0x00000000026067F0>]
>>> class foo(base):
...     pass
... 
>>> f = foo()
>>> list(foo.instances)
[<__main__.foo object at 0x0000000002606898>]
>>> list(base.instances)
[<__main__.base object at 0x00000000026067F0>]
>>> del f
>>> list(foo.instances)
[]

@Kevin 2016-05-15 00:06:28

WeakSet, unfortunately, will use standard hashing semantics instead of identity semantics, which means if the OP's base class wants to override __eq__, it will error out without a corresponding __hash__ override, and even with the override, it will still misbehave since it will coalesce objects that are equal.

@Blckknght 2016-05-15 08:44:43

Hmm, that's a good point, we don't really need or want the set semantics that come with WeakSet. I just picked it because it's the only non-mapping "weak" container defined in the weakref module. I guess a list of weakref.ref objects would be better, but it's a bit less convenient to work with.

@dwstein 2012-08-24 03:01:18

Using the answer from @Joel Cornett I've come up with the following, which seems to work. i.e. i'm able to total up object variables.

import os

os.system("clear")

class Foo():
    instances = []
    def __init__(self):
        Foo.instances.append(self)
        self.x = 5

class Bar():
    def __init__(self):
        pass

    def testy(self):
        self.foo1 = Foo()
        self.foo2 = Foo()
        self.foo3 = Foo()

foo = Foo()
print Foo.instances
bar = Bar()
bar.testy()
print Foo.instances

x_tot = 0
for inst in Foo.instances:
    x_tot += inst.x
    print x_tot

output:

[<__main__.Foo instance at 0x108e334d0>]
[<__main__.Foo instance at 0x108e334d0>, <__main__.Foo instance at 0x108e33560>, <__main__.Foo instance at 0x108e335a8>, <__main__.Foo instance at 0x108e335f0>]
5
10
15
20

@Joel Cornett 2012-08-24 01:30:10

One way to keep track of instances is with a class variable:

class A(object):
    instances = []

    def __init__(self, foo):
        self.foo = foo
        A.instances.append(self)

At the end of the program, you can create your dict like this:

foo_vars = {id(instance): instance.foo for instance in A.instances}

There is only one list:

>>> a = A(1)
>>> b = A(2)
>>> A.instances
[<__main__.A object at 0x1004d44d0>, <__main__.A object at 0x1004d4510>]
>>> id(A.instances)
4299683456
>>> id(a.instances)
4299683456    
>>> id(b.instances)
4299683456    

@dwstein 2012-08-24 01:45:24

thanks! But, won't this create a separate copy of 'instances' in each instance of A? Wouldn't A.instances always have one item in the list?

@Joel Cornett 2012-08-24 01:50:30

@dwstein: No, see edit. instances is a class variable. Here is a related concept: “Least Astonishment” in Python: The Mutable Default Argument

@dwstein 2012-08-24 01:59:30

Thanks for the detailed answer and the info. It's gonna take me a while to absorb. Also, will 'instances' include instances of derived classes? Or, will that be a separate list?

@Joel Cornett 2012-08-24 02:08:17

They will be the same list. This behavior can be confusing, but it shouldn't be (if you look at it correctly). In python, all variables are references to objects. Only assignment operations change what a variable is pointing to (thus instances = [] will cause this variable to point to a new list object). There is only one assignment operation defined here. All the rest of the operations (A.instances.append() for example) operate on the actual object -- they don't reassign the variable name. Classes don't operate any differently.

@Blckknght 2012-08-24 02:30:11

You could make it create separate instance lists for subclasses if you overrode the __new__ method instead of __init__. One of its arguments is the class of the object being created, so you can assign to it in the right place (though you'll need to go explictly via cls.__dict__[instances] to avoid the instances dict being inherited). Hmm, perhaps I should write that up as my own answer....

@dwstein 2012-08-24 03:44:16

@JoelCornett I've been playing with this for a couple hours and it's exactly what I'm looking for. Wherever I am in the code I have access to a list of instances and the information in them. Thanks again.

@Joel Cornett 2012-08-24 18:02:28

@dwstein: Thanks for that link. This is a great post.

@atm 2017-03-22 15:47:46

This answer seems to offer a different perspective. Can anyone confirm? stackoverflow.com/questions/4831307/…

Related Questions

Sponsored Content

38 Answered Questions

[SOLVED] How do I check whether a file exists without exceptions?

8 Answered Questions

[SOLVED] Getting the class name of an instance?

21 Answered Questions

[SOLVED] How do I list all files of a directory?

  • 2010-07-08 19:31:22
  • duhhunjonn
  • 4098907 View
  • 3473 Score
  • 21 Answer
  • Tags:   python directory

34 Answered Questions

[SOLVED] How do I sort a dictionary by value?

42 Answered Questions

[SOLVED] How do I efficiently iterate over each entry in a Java Map?

42 Answered Questions

[SOLVED] How do I merge two dictionaries in a single expression?

25 Answered Questions

[SOLVED] How can I safely create a nested directory?

27 Answered Questions

[SOLVED] How do I check if a list is empty?

  • 2008-09-10 06:20:11
  • Ray Vega
  • 2725123 View
  • 3234 Score
  • 27 Answer
  • Tags:   python list

18 Answered Questions

[SOLVED] Are static class variables possible in Python?

Sponsored Content