Coverage for /home/martinb/.local/share/virtualenvs/camcops/lib/python3.6/site-packages/sqlalchemy/orm/collections.py : 46%

Hot-keys on this page
r m x p toggle line displays
j k next/prev highlighted chunk
0 (zero) top of page
1 (one) first highlighted chunk
1# orm/collections.py
2# Copyright (C) 2005-2020 the SQLAlchemy authors and contributors
3# <see AUTHORS file>
4#
5# This module is part of SQLAlchemy and is released under
6# the MIT License: http://www.opensource.org/licenses/mit-license.php
8"""Support for collections of mapped entities.
10The collections package supplies the machinery used to inform the ORM of
11collection membership changes. An instrumentation via decoration approach is
12used, allowing arbitrary types (including built-ins) to be used as entity
13collections without requiring inheritance from a base class.
15Instrumentation decoration relays membership change events to the
16:class:`.CollectionAttributeImpl` that is currently managing the collection.
17The decorators observe function call arguments and return values, tracking
18entities entering or leaving the collection. Two decorator approaches are
19provided. One is a bundle of generic decorators that map function arguments
20and return values to events::
22 from sqlalchemy.orm.collections import collection
23 class MyClass(object):
24 # ...
26 @collection.adds(1)
27 def store(self, item):
28 self.data.append(item)
30 @collection.removes_return()
31 def pop(self):
32 return self.data.pop()
35The second approach is a bundle of targeted decorators that wrap appropriate
36append and remove notifiers around the mutation methods present in the
37standard Python ``list``, ``set`` and ``dict`` interfaces. These could be
38specified in terms of generic decorator recipes, but are instead hand-tooled
39for increased efficiency. The targeted decorators occasionally implement
40adapter-like behavior, such as mapping bulk-set methods (``extend``,
41``update``, ``__setslice__``, etc.) into the series of atomic mutation events
42that the ORM requires.
44The targeted decorators are used internally for automatic instrumentation of
45entity collection classes. Every collection class goes through a
46transformation process roughly like so:
481. If the class is a built-in, substitute a trivial sub-class
492. Is this class already instrumented?
503. Add in generic decorators
514. Sniff out the collection interface through duck-typing
525. Add targeted decoration to any undecorated interface method
54This process modifies the class at runtime, decorating methods and adding some
55bookkeeping properties. This isn't possible (or desirable) for built-in
56classes like ``list``, so trivial sub-classes are substituted to hold
57decoration::
59 class InstrumentedList(list):
60 pass
62Collection classes can be specified in ``relationship(collection_class=)`` as
63types or a function that returns an instance. Collection classes are
64inspected and instrumented during the mapper compilation phase. The
65collection_class callable will be executed once to produce a specimen
66instance, and the type of that specimen will be instrumented. Functions that
67return built-in types like ``lists`` will be adapted to produce instrumented
68instances.
70When extending a known type like ``list``, additional decorations are not
71generally not needed. Odds are, the extension method will delegate to a
72method that's already instrumented. For example::
74 class QueueIsh(list):
75 def push(self, item):
76 self.append(item)
77 def shift(self):
78 return self.pop(0)
80There's no need to decorate these methods. ``append`` and ``pop`` are already
81instrumented as part of the ``list`` interface. Decorating them would fire
82duplicate events, which should be avoided.
84The targeted decoration tries not to rely on other methods in the underlying
85collection class, but some are unavoidable. Many depend on 'read' methods
86being present to properly instrument a 'write', for example, ``__setitem__``
87needs ``__getitem__``. "Bulk" methods like ``update`` and ``extend`` may also
88reimplemented in terms of atomic appends and removes, so the ``extend``
89decoration will actually perform many ``append`` operations and not call the
90underlying method at all.
92Tight control over bulk operation and the firing of events is also possible by
93implementing the instrumentation internally in your methods. The basic
94instrumentation package works under the general assumption that collection
95mutation will not raise unusual exceptions. If you want to closely
96orchestrate append and remove events with exception management, internal
97instrumentation may be the answer. Within your method,
98``collection_adapter(self)`` will retrieve an object that you can use for
99explicit control over triggering append and remove events.
101The owning object and :class:`.CollectionAttributeImpl` are also reachable
102through the adapter, allowing for some very sophisticated behavior.
104"""
106import operator
107import weakref
109from sqlalchemy.util.compat import inspect_getfullargspec
110from . import base
111from .. import exc as sa_exc
112from .. import util
113from ..sql import expression
116__all__ = [
117 "collection",
118 "collection_adapter",
119 "mapped_collection",
120 "column_mapped_collection",
121 "attribute_mapped_collection",
122]
124__instrumentation_mutex = util.threading.Lock()
127class _PlainColumnGetter(object):
128 """Plain column getter, stores collection of Column objects
129 directly.
131 Serializes to a :class:`._SerializableColumnGetterV2`
132 which has more expensive __call__() performance
133 and some rare caveats.
135 """
137 def __init__(self, cols):
138 self.cols = cols
139 self.composite = len(cols) > 1
141 def __reduce__(self):
142 return _SerializableColumnGetterV2._reduce_from_cols(self.cols)
144 def _cols(self, mapper):
145 return self.cols
147 def __call__(self, value):
148 state = base.instance_state(value)
149 m = base._state_mapper(state)
151 key = [
152 m._get_state_attr_by_column(state, state.dict, col)
153 for col in self._cols(m)
154 ]
156 if self.composite:
157 return tuple(key)
158 else:
159 return key[0]
162class _SerializableColumnGetter(object):
163 """Column-based getter used in version 0.7.6 only.
165 Remains here for pickle compatibility with 0.7.6.
167 """
169 def __init__(self, colkeys):
170 self.colkeys = colkeys
171 self.composite = len(colkeys) > 1
173 def __reduce__(self):
174 return _SerializableColumnGetter, (self.colkeys,)
176 def __call__(self, value):
177 state = base.instance_state(value)
178 m = base._state_mapper(state)
179 key = [
180 m._get_state_attr_by_column(
181 state, state.dict, m.mapped_table.columns[k]
182 )
183 for k in self.colkeys
184 ]
185 if self.composite:
186 return tuple(key)
187 else:
188 return key[0]
191class _SerializableColumnGetterV2(_PlainColumnGetter):
192 """Updated serializable getter which deals with
193 multi-table mapped classes.
195 Two extremely unusual cases are not supported.
196 Mappings which have tables across multiple metadata
197 objects, or which are mapped to non-Table selectables
198 linked across inheriting mappers may fail to function
199 here.
201 """
203 def __init__(self, colkeys):
204 self.colkeys = colkeys
205 self.composite = len(colkeys) > 1
207 def __reduce__(self):
208 return self.__class__, (self.colkeys,)
210 @classmethod
211 def _reduce_from_cols(cls, cols):
212 def _table_key(c):
213 if not isinstance(c.table, expression.TableClause):
214 return None
215 else:
216 return c.table.key
218 colkeys = [(c.key, _table_key(c)) for c in cols]
219 return _SerializableColumnGetterV2, (colkeys,)
221 def _cols(self, mapper):
222 cols = []
223 metadata = getattr(mapper.local_table, "metadata", None)
224 for (ckey, tkey) in self.colkeys:
225 if tkey is None or metadata is None or tkey not in metadata:
226 cols.append(mapper.local_table.c[ckey])
227 else:
228 cols.append(metadata.tables[tkey].c[ckey])
229 return cols
232def column_mapped_collection(mapping_spec):
233 """A dictionary-based collection type with column-based keying.
235 Returns a :class:`.MappedCollection` factory with a keying function
236 generated from mapping_spec, which may be a Column or a sequence
237 of Columns.
239 The key value must be immutable for the lifetime of the object. You
240 can not, for example, map on foreign key values if those key values will
241 change during the session, i.e. from None to a database-assigned integer
242 after a session flush.
244 """
245 cols = [
246 expression._only_column_elements(q, "mapping_spec")
247 for q in util.to_list(mapping_spec)
248 ]
249 keyfunc = _PlainColumnGetter(cols)
250 return lambda: MappedCollection(keyfunc)
253class _SerializableAttrGetter(object):
254 def __init__(self, name):
255 self.name = name
256 self.getter = operator.attrgetter(name)
258 def __call__(self, target):
259 return self.getter(target)
261 def __reduce__(self):
262 return _SerializableAttrGetter, (self.name,)
265def attribute_mapped_collection(attr_name):
266 """A dictionary-based collection type with attribute-based keying.
268 Returns a :class:`.MappedCollection` factory with a keying based on the
269 'attr_name' attribute of entities in the collection, where ``attr_name``
270 is the string name of the attribute.
272 The key value must be immutable for the lifetime of the object. You
273 can not, for example, map on foreign key values if those key values will
274 change during the session, i.e. from None to a database-assigned integer
275 after a session flush.
277 """
278 getter = _SerializableAttrGetter(attr_name)
279 return lambda: MappedCollection(getter)
282def mapped_collection(keyfunc):
283 """A dictionary-based collection type with arbitrary keying.
285 Returns a :class:`.MappedCollection` factory with a keying function
286 generated from keyfunc, a callable that takes an entity and returns a
287 key value.
289 The key value must be immutable for the lifetime of the object. You
290 can not, for example, map on foreign key values if those key values will
291 change during the session, i.e. from None to a database-assigned integer
292 after a session flush.
294 """
295 return lambda: MappedCollection(keyfunc)
298class collection(object):
299 """Decorators for entity collection classes.
301 The decorators fall into two groups: annotations and interception recipes.
303 The annotating decorators (appender, remover, iterator, linker, converter,
304 internally_instrumented) indicate the method's purpose and take no
305 arguments. They are not written with parens::
307 @collection.appender
308 def append(self, append): ...
310 The recipe decorators all require parens, even those that take no
311 arguments::
313 @collection.adds('entity')
314 def insert(self, position, entity): ...
316 @collection.removes_return()
317 def popitem(self): ...
319 """
321 # Bundled as a class solely for ease of use: packaging, doc strings,
322 # importability.
324 @staticmethod
325 def appender(fn):
326 """Tag the method as the collection appender.
328 The appender method is called with one positional argument: the value
329 to append. The method will be automatically decorated with 'adds(1)'
330 if not already decorated::
332 @collection.appender
333 def add(self, append): ...
335 # or, equivalently
336 @collection.appender
337 @collection.adds(1)
338 def add(self, append): ...
340 # for mapping type, an 'append' may kick out a previous value
341 # that occupies that slot. consider d['a'] = 'foo'- any previous
342 # value in d['a'] is discarded.
343 @collection.appender
344 @collection.replaces(1)
345 def add(self, entity):
346 key = some_key_func(entity)
347 previous = None
348 if key in self:
349 previous = self[key]
350 self[key] = entity
351 return previous
353 If the value to append is not allowed in the collection, you may
354 raise an exception. Something to remember is that the appender
355 will be called for each object mapped by a database query. If the
356 database contains rows that violate your collection semantics, you
357 will need to get creative to fix the problem, as access via the
358 collection will not work.
360 If the appender method is internally instrumented, you must also
361 receive the keyword argument '_sa_initiator' and ensure its
362 promulgation to collection events.
364 """
365 fn._sa_instrument_role = "appender"
366 return fn
368 @staticmethod
369 def remover(fn):
370 """Tag the method as the collection remover.
372 The remover method is called with one positional argument: the value
373 to remove. The method will be automatically decorated with
374 :meth:`removes_return` if not already decorated::
376 @collection.remover
377 def zap(self, entity): ...
379 # or, equivalently
380 @collection.remover
381 @collection.removes_return()
382 def zap(self, ): ...
384 If the value to remove is not present in the collection, you may
385 raise an exception or return None to ignore the error.
387 If the remove method is internally instrumented, you must also
388 receive the keyword argument '_sa_initiator' and ensure its
389 promulgation to collection events.
391 """
392 fn._sa_instrument_role = "remover"
393 return fn
395 @staticmethod
396 def iterator(fn):
397 """Tag the method as the collection remover.
399 The iterator method is called with no arguments. It is expected to
400 return an iterator over all collection members::
402 @collection.iterator
403 def __iter__(self): ...
405 """
406 fn._sa_instrument_role = "iterator"
407 return fn
409 @staticmethod
410 def internally_instrumented(fn):
411 """Tag the method as instrumented.
413 This tag will prevent any decoration from being applied to the
414 method. Use this if you are orchestrating your own calls to
415 :func:`.collection_adapter` in one of the basic SQLAlchemy
416 interface methods, or to prevent an automatic ABC method
417 decoration from wrapping your implementation::
419 # normally an 'extend' method on a list-like class would be
420 # automatically intercepted and re-implemented in terms of
421 # SQLAlchemy events and append(). your implementation will
422 # never be called, unless:
423 @collection.internally_instrumented
424 def extend(self, items): ...
426 """
427 fn._sa_instrumented = True
428 return fn
430 @staticmethod
431 @util.deprecated(
432 "1.0",
433 "The :meth:`.collection.linker` handler is deprecated and will "
434 "be removed in a future release. Please refer to the "
435 ":meth:`.AttributeEvents.init_collection` "
436 "and :meth:`.AttributeEvents.dispose_collection` event handlers. ",
437 )
438 def linker(fn):
439 """Tag the method as a "linked to attribute" event handler.
441 This optional event handler will be called when the collection class
442 is linked to or unlinked from the InstrumentedAttribute. It is
443 invoked immediately after the '_sa_adapter' property is set on
444 the instance. A single argument is passed: the collection adapter
445 that has been linked, or None if unlinking.
448 """
449 fn._sa_instrument_role = "linker"
450 return fn
452 link = linker
453 """Synonym for :meth:`.collection.linker`.
455 .. deprecated:: 1.0 - :meth:`.collection.link` is deprecated and will be
456 removed in a future release.
458 """
460 @staticmethod
461 @util.deprecated(
462 "1.3",
463 "The :meth:`.collection.converter` handler is deprecated and will "
464 "be removed in a future release. Please refer to the "
465 ":class:`.AttributeEvents.bulk_replace` listener interface in "
466 "conjunction with the :func:`.event.listen` function.",
467 )
468 def converter(fn):
469 """Tag the method as the collection converter.
471 This optional method will be called when a collection is being
472 replaced entirely, as in::
474 myobj.acollection = [newvalue1, newvalue2]
476 The converter method will receive the object being assigned and should
477 return an iterable of values suitable for use by the ``appender``
478 method. A converter must not assign values or mutate the collection,
479 its sole job is to adapt the value the user provides into an iterable
480 of values for the ORM's use.
482 The default converter implementation will use duck-typing to do the
483 conversion. A dict-like collection will be convert into an iterable
484 of dictionary values, and other types will simply be iterated::
486 @collection.converter
487 def convert(self, other): ...
489 If the duck-typing of the object does not match the type of this
490 collection, a TypeError is raised.
492 Supply an implementation of this method if you want to expand the
493 range of possible types that can be assigned in bulk or perform
494 validation on the values about to be assigned.
496 """
497 fn._sa_instrument_role = "converter"
498 return fn
500 @staticmethod
501 def adds(arg):
502 """Mark the method as adding an entity to the collection.
504 Adds "add to collection" handling to the method. The decorator
505 argument indicates which method argument holds the SQLAlchemy-relevant
506 value. Arguments can be specified positionally (i.e. integer) or by
507 name::
509 @collection.adds(1)
510 def push(self, item): ...
512 @collection.adds('entity')
513 def do_stuff(self, thing, entity=None): ...
515 """
517 def decorator(fn):
518 fn._sa_instrument_before = ("fire_append_event", arg)
519 return fn
521 return decorator
523 @staticmethod
524 def replaces(arg):
525 """Mark the method as replacing an entity in the collection.
527 Adds "add to collection" and "remove from collection" handling to
528 the method. The decorator argument indicates which method argument
529 holds the SQLAlchemy-relevant value to be added, and return value, if
530 any will be considered the value to remove.
532 Arguments can be specified positionally (i.e. integer) or by name::
534 @collection.replaces(2)
535 def __setitem__(self, index, item): ...
537 """
539 def decorator(fn):
540 fn._sa_instrument_before = ("fire_append_event", arg)
541 fn._sa_instrument_after = "fire_remove_event"
542 return fn
544 return decorator
546 @staticmethod
547 def removes(arg):
548 """Mark the method as removing an entity in the collection.
550 Adds "remove from collection" handling to the method. The decorator
551 argument indicates which method argument holds the SQLAlchemy-relevant
552 value to be removed. Arguments can be specified positionally (i.e.
553 integer) or by name::
555 @collection.removes(1)
556 def zap(self, item): ...
558 For methods where the value to remove is not known at call-time, use
559 collection.removes_return.
561 """
563 def decorator(fn):
564 fn._sa_instrument_before = ("fire_remove_event", arg)
565 return fn
567 return decorator
569 @staticmethod
570 def removes_return():
571 """Mark the method as removing an entity in the collection.
573 Adds "remove from collection" handling to the method. The return
574 value of the method, if any, is considered the value to remove. The
575 method arguments are not inspected::
577 @collection.removes_return()
578 def pop(self): ...
580 For methods where the value to remove is known at call-time, use
581 collection.remove.
583 """
585 def decorator(fn):
586 fn._sa_instrument_after = "fire_remove_event"
587 return fn
589 return decorator
592collection_adapter = operator.attrgetter("_sa_adapter")
593"""Fetch the :class:`.CollectionAdapter` for a collection."""
596class CollectionAdapter(object):
597 """Bridges between the ORM and arbitrary Python collections.
599 Proxies base-level collection operations (append, remove, iterate)
600 to the underlying Python collection, and emits add/remove events for
601 entities entering or leaving the collection.
603 The ORM uses :class:`.CollectionAdapter` exclusively for interaction with
604 entity collections.
607 """
609 __slots__ = (
610 "attr",
611 "_key",
612 "_data",
613 "owner_state",
614 "_converter",
615 "invalidated",
616 )
618 def __init__(self, attr, owner_state, data):
619 self.attr = attr
620 self._key = attr.key
621 self._data = weakref.ref(data)
622 self.owner_state = owner_state
623 data._sa_adapter = self
624 self._converter = data._sa_converter
625 self.invalidated = False
627 def _warn_invalidated(self):
628 util.warn("This collection has been invalidated.")
630 @property
631 def data(self):
632 "The entity collection being adapted."
633 return self._data()
635 @property
636 def _referenced_by_owner(self):
637 """return True if the owner state still refers to this collection.
639 This will return False within a bulk replace operation,
640 where this collection is the one being replaced.
642 """
643 return self.owner_state.dict[self._key] is self._data()
645 def bulk_appender(self):
646 return self._data()._sa_appender
648 def append_with_event(self, item, initiator=None):
649 """Add an entity to the collection, firing mutation events."""
651 self._data()._sa_appender(item, _sa_initiator=initiator)
653 def append_without_event(self, item):
654 """Add or restore an entity to the collection, firing no events."""
655 self._data()._sa_appender(item, _sa_initiator=False)
657 def append_multiple_without_event(self, items):
658 """Add or restore an entity to the collection, firing no events."""
659 appender = self._data()._sa_appender
660 for item in items:
661 appender(item, _sa_initiator=False)
663 def bulk_remover(self):
664 return self._data()._sa_remover
666 def remove_with_event(self, item, initiator=None):
667 """Remove an entity from the collection, firing mutation events."""
668 self._data()._sa_remover(item, _sa_initiator=initiator)
670 def remove_without_event(self, item):
671 """Remove an entity from the collection, firing no events."""
672 self._data()._sa_remover(item, _sa_initiator=False)
674 def clear_with_event(self, initiator=None):
675 """Empty the collection, firing a mutation event for each entity."""
677 remover = self._data()._sa_remover
678 for item in list(self):
679 remover(item, _sa_initiator=initiator)
681 def clear_without_event(self):
682 """Empty the collection, firing no events."""
684 remover = self._data()._sa_remover
685 for item in list(self):
686 remover(item, _sa_initiator=False)
688 def __iter__(self):
689 """Iterate over entities in the collection."""
691 return iter(self._data()._sa_iterator())
693 def __len__(self):
694 """Count entities in the collection."""
695 return len(list(self._data()._sa_iterator()))
697 def __bool__(self):
698 return True
700 __nonzero__ = __bool__
702 def fire_append_event(self, item, initiator=None):
703 """Notify that a entity has entered the collection.
705 Initiator is a token owned by the InstrumentedAttribute that
706 initiated the membership mutation, and should be left as None
707 unless you are passing along an initiator value from a chained
708 operation.
710 """
711 if initiator is not False:
712 if self.invalidated:
713 self._warn_invalidated()
714 return self.attr.fire_append_event(
715 self.owner_state, self.owner_state.dict, item, initiator
716 )
717 else:
718 return item
720 def fire_remove_event(self, item, initiator=None):
721 """Notify that a entity has been removed from the collection.
723 Initiator is the InstrumentedAttribute that initiated the membership
724 mutation, and should be left as None unless you are passing along
725 an initiator value from a chained operation.
727 """
728 if initiator is not False:
729 if self.invalidated:
730 self._warn_invalidated()
731 self.attr.fire_remove_event(
732 self.owner_state, self.owner_state.dict, item, initiator
733 )
735 def fire_pre_remove_event(self, initiator=None):
736 """Notify that an entity is about to be removed from the collection.
738 Only called if the entity cannot be removed after calling
739 fire_remove_event().
741 """
742 if self.invalidated:
743 self._warn_invalidated()
744 self.attr.fire_pre_remove_event(
745 self.owner_state, self.owner_state.dict, initiator=initiator
746 )
748 def __getstate__(self):
749 return {
750 "key": self._key,
751 "owner_state": self.owner_state,
752 "owner_cls": self.owner_state.class_,
753 "data": self.data,
754 "invalidated": self.invalidated,
755 }
757 def __setstate__(self, d):
758 self._key = d["key"]
759 self.owner_state = d["owner_state"]
760 self._data = weakref.ref(d["data"])
761 self._converter = d["data"]._sa_converter
762 d["data"]._sa_adapter = self
763 self.invalidated = d["invalidated"]
764 self.attr = getattr(d["owner_cls"], self._key).impl
767def bulk_replace(values, existing_adapter, new_adapter, initiator=None):
768 """Load a new collection, firing events based on prior like membership.
770 Appends instances in ``values`` onto the ``new_adapter``. Events will be
771 fired for any instance not present in the ``existing_adapter``. Any
772 instances in ``existing_adapter`` not present in ``values`` will have
773 remove events fired upon them.
775 :param values: An iterable of collection member instances
777 :param existing_adapter: A :class:`.CollectionAdapter` of
778 instances to be replaced
780 :param new_adapter: An empty :class:`.CollectionAdapter`
781 to load with ``values``
784 """
786 assert isinstance(values, list)
788 idset = util.IdentitySet
789 existing_idset = idset(existing_adapter or ())
790 constants = existing_idset.intersection(values or ())
791 additions = idset(values or ()).difference(constants)
792 removals = existing_idset.difference(constants)
794 appender = new_adapter.bulk_appender()
796 for member in values or ():
797 if member in additions:
798 appender(member, _sa_initiator=initiator)
799 elif member in constants:
800 appender(member, _sa_initiator=False)
802 if existing_adapter:
803 for member in removals:
804 existing_adapter.fire_remove_event(member, initiator=initiator)
807def prepare_instrumentation(factory):
808 """Prepare a callable for future use as a collection class factory.
810 Given a collection class factory (either a type or no-arg callable),
811 return another factory that will produce compatible instances when
812 called.
814 This function is responsible for converting collection_class=list
815 into the run-time behavior of collection_class=InstrumentedList.
817 """
818 # Convert a builtin to 'Instrumented*'
819 if factory in __canned_instrumentation:
820 factory = __canned_instrumentation[factory]
822 # Create a specimen
823 cls = type(factory())
825 # Did factory callable return a builtin?
826 if cls in __canned_instrumentation:
827 # Wrap it so that it returns our 'Instrumented*'
828 factory = __converting_factory(cls, factory)
829 cls = factory()
831 # Instrument the class if needed.
832 if __instrumentation_mutex.acquire():
833 try:
834 if getattr(cls, "_sa_instrumented", None) != id(cls):
835 _instrument_class(cls)
836 finally:
837 __instrumentation_mutex.release()
839 return factory
842def __converting_factory(specimen_cls, original_factory):
843 """Return a wrapper that converts a "canned" collection like
844 set, dict, list into the Instrumented* version.
846 """
848 instrumented_cls = __canned_instrumentation[specimen_cls]
850 def wrapper():
851 collection = original_factory()
852 return instrumented_cls(collection)
854 # often flawed but better than nothing
855 wrapper.__name__ = "%sWrapper" % original_factory.__name__
856 wrapper.__doc__ = original_factory.__doc__
858 return wrapper
861def _instrument_class(cls):
862 """Modify methods in a class and install instrumentation."""
864 # In the normal call flow, a request for any of the 3 basic collection
865 # types is transformed into one of our trivial subclasses
866 # (e.g. InstrumentedList). Catch anything else that sneaks in here...
867 if cls.__module__ == "__builtin__":
868 raise sa_exc.ArgumentError(
869 "Can not instrument a built-in type. Use a "
870 "subclass, even a trivial one."
871 )
873 roles, methods = _locate_roles_and_methods(cls)
875 _setup_canned_roles(cls, roles, methods)
877 _assert_required_roles(cls, roles, methods)
879 _set_collection_attributes(cls, roles, methods)
882def _locate_roles_and_methods(cls):
883 """search for _sa_instrument_role-decorated methods in
884 method resolution order, assign to roles.
886 """
888 roles = {}
889 methods = {}
891 for supercls in cls.__mro__:
892 for name, method in vars(supercls).items():
893 if not util.callable(method):
894 continue
896 # note role declarations
897 if hasattr(method, "_sa_instrument_role"):
898 role = method._sa_instrument_role
899 assert role in (
900 "appender",
901 "remover",
902 "iterator",
903 "linker",
904 "converter",
905 )
906 roles.setdefault(role, name)
908 # transfer instrumentation requests from decorated function
909 # to the combined queue
910 before, after = None, None
911 if hasattr(method, "_sa_instrument_before"):
912 op, argument = method._sa_instrument_before
913 assert op in ("fire_append_event", "fire_remove_event")
914 before = op, argument
915 if hasattr(method, "_sa_instrument_after"):
916 op = method._sa_instrument_after
917 assert op in ("fire_append_event", "fire_remove_event")
918 after = op
919 if before:
920 methods[name] = before + (after,)
921 elif after:
922 methods[name] = None, None, after
923 return roles, methods
926def _setup_canned_roles(cls, roles, methods):
927 """see if this class has "canned" roles based on a known
928 collection type (dict, set, list). Apply those roles
929 as needed to the "roles" dictionary, and also
930 prepare "decorator" methods
932 """
933 collection_type = util.duck_type_collection(cls)
934 if collection_type in __interfaces:
935 canned_roles, decorators = __interfaces[collection_type]
936 for role, name in canned_roles.items():
937 roles.setdefault(role, name)
939 # apply ABC auto-decoration to methods that need it
940 for method, decorator in decorators.items():
941 fn = getattr(cls, method, None)
942 if (
943 fn
944 and method not in methods
945 and not hasattr(fn, "_sa_instrumented")
946 ):
947 setattr(cls, method, decorator(fn))
950def _assert_required_roles(cls, roles, methods):
951 """ensure all roles are present, and apply implicit instrumentation if
952 needed
954 """
955 if "appender" not in roles or not hasattr(cls, roles["appender"]):
956 raise sa_exc.ArgumentError(
957 "Type %s must elect an appender method to be "
958 "a collection class" % cls.__name__
959 )
960 elif roles["appender"] not in methods and not hasattr(
961 getattr(cls, roles["appender"]), "_sa_instrumented"
962 ):
963 methods[roles["appender"]] = ("fire_append_event", 1, None)
965 if "remover" not in roles or not hasattr(cls, roles["remover"]):
966 raise sa_exc.ArgumentError(
967 "Type %s must elect a remover method to be "
968 "a collection class" % cls.__name__
969 )
970 elif roles["remover"] not in methods and not hasattr(
971 getattr(cls, roles["remover"]), "_sa_instrumented"
972 ):
973 methods[roles["remover"]] = ("fire_remove_event", 1, None)
975 if "iterator" not in roles or not hasattr(cls, roles["iterator"]):
976 raise sa_exc.ArgumentError(
977 "Type %s must elect an iterator method to be "
978 "a collection class" % cls.__name__
979 )
982def _set_collection_attributes(cls, roles, methods):
983 """apply ad-hoc instrumentation from decorators, class-level defaults
984 and implicit role declarations
986 """
987 for method_name, (before, argument, after) in methods.items():
988 setattr(
989 cls,
990 method_name,
991 _instrument_membership_mutator(
992 getattr(cls, method_name), before, argument, after
993 ),
994 )
995 # intern the role map
996 for role, method_name in roles.items():
997 setattr(cls, "_sa_%s" % role, getattr(cls, method_name))
999 cls._sa_adapter = None
1001 if not hasattr(cls, "_sa_converter"):
1002 cls._sa_converter = None
1003 cls._sa_instrumented = id(cls)
1006def _instrument_membership_mutator(method, before, argument, after):
1007 """Route method args and/or return value through the collection
1008 adapter."""
1009 # This isn't smart enough to handle @adds(1) for 'def fn(self, (a, b))'
1010 if before:
1011 fn_args = list(
1012 util.flatten_iterator(inspect_getfullargspec(method)[0])
1013 )
1014 if isinstance(argument, int):
1015 pos_arg = argument
1016 named_arg = len(fn_args) > argument and fn_args[argument] or None
1017 else:
1018 if argument in fn_args:
1019 pos_arg = fn_args.index(argument)
1020 else:
1021 pos_arg = None
1022 named_arg = argument
1023 del fn_args
1025 def wrapper(*args, **kw):
1026 if before:
1027 if pos_arg is None:
1028 if named_arg not in kw:
1029 raise sa_exc.ArgumentError(
1030 "Missing argument %s" % argument
1031 )
1032 value = kw[named_arg]
1033 else:
1034 if len(args) > pos_arg:
1035 value = args[pos_arg]
1036 elif named_arg in kw:
1037 value = kw[named_arg]
1038 else:
1039 raise sa_exc.ArgumentError(
1040 "Missing argument %s" % argument
1041 )
1043 initiator = kw.pop("_sa_initiator", None)
1044 if initiator is False:
1045 executor = None
1046 else:
1047 executor = args[0]._sa_adapter
1049 if before and executor:
1050 getattr(executor, before)(value, initiator)
1052 if not after or not executor:
1053 return method(*args, **kw)
1054 else:
1055 res = method(*args, **kw)
1056 if res is not None:
1057 getattr(executor, after)(res, initiator)
1058 return res
1060 wrapper._sa_instrumented = True
1061 if hasattr(method, "_sa_instrument_role"):
1062 wrapper._sa_instrument_role = method._sa_instrument_role
1063 wrapper.__name__ = method.__name__
1064 wrapper.__doc__ = method.__doc__
1065 return wrapper
1068def __set(collection, item, _sa_initiator=None):
1069 """Run set events.
1071 This event always occurs before the collection is actually mutated.
1073 """
1075 if _sa_initiator is not False:
1076 executor = collection._sa_adapter
1077 if executor:
1078 item = executor.fire_append_event(item, _sa_initiator)
1079 return item
1082def __del(collection, item, _sa_initiator=None):
1083 """Run del events.
1085 This event occurs before the collection is actually mutated, *except*
1086 in the case of a pop operation, in which case it occurs afterwards.
1087 For pop operations, the __before_pop hook is called before the
1088 operation occurs.
1090 """
1091 if _sa_initiator is not False:
1092 executor = collection._sa_adapter
1093 if executor:
1094 executor.fire_remove_event(item, _sa_initiator)
1097def __before_pop(collection, _sa_initiator=None):
1098 """An event which occurs on a before a pop() operation occurs."""
1099 executor = collection._sa_adapter
1100 if executor:
1101 executor.fire_pre_remove_event(_sa_initiator)
1104def _list_decorators():
1105 """Tailored instrumentation wrappers for any list-like class."""
1107 def _tidy(fn):
1108 fn._sa_instrumented = True
1109 fn.__doc__ = getattr(list, fn.__name__).__doc__
1111 def append(fn):
1112 def append(self, item, _sa_initiator=None):
1113 item = __set(self, item, _sa_initiator)
1114 fn(self, item)
1116 _tidy(append)
1117 return append
1119 def remove(fn):
1120 def remove(self, value, _sa_initiator=None):
1121 __del(self, value, _sa_initiator)
1122 # testlib.pragma exempt:__eq__
1123 fn(self, value)
1125 _tidy(remove)
1126 return remove
1128 def insert(fn):
1129 def insert(self, index, value):
1130 value = __set(self, value)
1131 fn(self, index, value)
1133 _tidy(insert)
1134 return insert
1136 def __setitem__(fn):
1137 def __setitem__(self, index, value):
1138 if not isinstance(index, slice):
1139 existing = self[index]
1140 if existing is not None:
1141 __del(self, existing)
1142 value = __set(self, value)
1143 fn(self, index, value)
1144 else:
1145 # slice assignment requires __delitem__, insert, __len__
1146 step = index.step or 1
1147 start = index.start or 0
1148 if start < 0:
1149 start += len(self)
1150 if index.stop is not None:
1151 stop = index.stop
1152 else:
1153 stop = len(self)
1154 if stop < 0:
1155 stop += len(self)
1157 if step == 1:
1158 if value is self:
1159 return
1160 for i in range(start, stop, step):
1161 if len(self) > start:
1162 del self[start]
1164 for i, item in enumerate(value):
1165 self.insert(i + start, item)
1166 else:
1167 rng = list(range(start, stop, step))
1168 if len(value) != len(rng):
1169 raise ValueError(
1170 "attempt to assign sequence of size %s to "
1171 "extended slice of size %s"
1172 % (len(value), len(rng))
1173 )
1174 for i, item in zip(rng, value):
1175 self.__setitem__(i, item)
1177 _tidy(__setitem__)
1178 return __setitem__
1180 def __delitem__(fn):
1181 def __delitem__(self, index):
1182 if not isinstance(index, slice):
1183 item = self[index]
1184 __del(self, item)
1185 fn(self, index)
1186 else:
1187 # slice deletion requires __getslice__ and a slice-groking
1188 # __getitem__ for stepped deletion
1189 # note: not breaking this into atomic dels
1190 for item in self[index]:
1191 __del(self, item)
1192 fn(self, index)
1194 _tidy(__delitem__)
1195 return __delitem__
1197 if util.py2k:
1199 def __setslice__(fn):
1200 def __setslice__(self, start, end, values):
1201 for value in self[start:end]:
1202 __del(self, value)
1203 values = [__set(self, value) for value in values]
1204 fn(self, start, end, values)
1206 _tidy(__setslice__)
1207 return __setslice__
1209 def __delslice__(fn):
1210 def __delslice__(self, start, end):
1211 for value in self[start:end]:
1212 __del(self, value)
1213 fn(self, start, end)
1215 _tidy(__delslice__)
1216 return __delslice__
1218 def extend(fn):
1219 def extend(self, iterable):
1220 for value in iterable:
1221 self.append(value)
1223 _tidy(extend)
1224 return extend
1226 def __iadd__(fn):
1227 def __iadd__(self, iterable):
1228 # list.__iadd__ takes any iterable and seems to let TypeError
1229 # raise as-is instead of returning NotImplemented
1230 for value in iterable:
1231 self.append(value)
1232 return self
1234 _tidy(__iadd__)
1235 return __iadd__
1237 def pop(fn):
1238 def pop(self, index=-1):
1239 __before_pop(self)
1240 item = fn(self, index)
1241 __del(self, item)
1242 return item
1244 _tidy(pop)
1245 return pop
1247 if not util.py2k:
1249 def clear(fn):
1250 def clear(self, index=-1):
1251 for item in self:
1252 __del(self, item)
1253 fn(self)
1255 _tidy(clear)
1256 return clear
1258 # __imul__ : not wrapping this. all members of the collection are already
1259 # present, so no need to fire appends... wrapping it with an explicit
1260 # decorator is still possible, so events on *= can be had if they're
1261 # desired. hard to imagine a use case for __imul__, though.
1263 l = locals().copy()
1264 l.pop("_tidy")
1265 return l
1268def _dict_decorators():
1269 """Tailored instrumentation wrappers for any dict-like mapping class."""
1271 def _tidy(fn):
1272 fn._sa_instrumented = True
1273 fn.__doc__ = getattr(dict, fn.__name__).__doc__
1275 Unspecified = util.symbol("Unspecified")
1277 def __setitem__(fn):
1278 def __setitem__(self, key, value, _sa_initiator=None):
1279 if key in self:
1280 __del(self, self[key], _sa_initiator)
1281 value = __set(self, value, _sa_initiator)
1282 fn(self, key, value)
1284 _tidy(__setitem__)
1285 return __setitem__
1287 def __delitem__(fn):
1288 def __delitem__(self, key, _sa_initiator=None):
1289 if key in self:
1290 __del(self, self[key], _sa_initiator)
1291 fn(self, key)
1293 _tidy(__delitem__)
1294 return __delitem__
1296 def clear(fn):
1297 def clear(self):
1298 for key in self:
1299 __del(self, self[key])
1300 fn(self)
1302 _tidy(clear)
1303 return clear
1305 def pop(fn):
1306 def pop(self, key, default=Unspecified):
1307 __before_pop(self)
1308 _to_del = key in self
1309 if default is Unspecified:
1310 item = fn(self, key)
1311 else:
1312 item = fn(self, key, default)
1313 if _to_del:
1314 __del(self, item)
1315 return item
1317 _tidy(pop)
1318 return pop
1320 def popitem(fn):
1321 def popitem(self):
1322 __before_pop(self)
1323 item = fn(self)
1324 __del(self, item[1])
1325 return item
1327 _tidy(popitem)
1328 return popitem
1330 def setdefault(fn):
1331 def setdefault(self, key, default=None):
1332 if key not in self:
1333 self.__setitem__(key, default)
1334 return default
1335 else:
1336 return self.__getitem__(key)
1338 _tidy(setdefault)
1339 return setdefault
1341 def update(fn):
1342 def update(self, __other=Unspecified, **kw):
1343 if __other is not Unspecified:
1344 if hasattr(__other, "keys"):
1345 for key in list(__other):
1346 if key not in self or self[key] is not __other[key]:
1347 self[key] = __other[key]
1348 else:
1349 for key, value in __other:
1350 if key not in self or self[key] is not value:
1351 self[key] = value
1352 for key in kw:
1353 if key not in self or self[key] is not kw[key]:
1354 self[key] = kw[key]
1356 _tidy(update)
1357 return update
1359 l = locals().copy()
1360 l.pop("_tidy")
1361 l.pop("Unspecified")
1362 return l
1365_set_binop_bases = (set, frozenset)
1368def _set_binops_check_strict(self, obj):
1369 """Allow only set, frozenset and self.__class__-derived
1370 objects in binops."""
1371 return isinstance(obj, _set_binop_bases + (self.__class__,))
1374def _set_binops_check_loose(self, obj):
1375 """Allow anything set-like to participate in set binops."""
1376 return (
1377 isinstance(obj, _set_binop_bases + (self.__class__,))
1378 or util.duck_type_collection(obj) == set
1379 )
1382def _set_decorators():
1383 """Tailored instrumentation wrappers for any set-like class."""
1385 def _tidy(fn):
1386 fn._sa_instrumented = True
1387 fn.__doc__ = getattr(set, fn.__name__).__doc__
1389 Unspecified = util.symbol("Unspecified")
1391 def add(fn):
1392 def add(self, value, _sa_initiator=None):
1393 if value not in self:
1394 value = __set(self, value, _sa_initiator)
1395 # testlib.pragma exempt:__hash__
1396 fn(self, value)
1398 _tidy(add)
1399 return add
1401 def discard(fn):
1402 def discard(self, value, _sa_initiator=None):
1403 # testlib.pragma exempt:__hash__
1404 if value in self:
1405 __del(self, value, _sa_initiator)
1406 # testlib.pragma exempt:__hash__
1407 fn(self, value)
1409 _tidy(discard)
1410 return discard
1412 def remove(fn):
1413 def remove(self, value, _sa_initiator=None):
1414 # testlib.pragma exempt:__hash__
1415 if value in self:
1416 __del(self, value, _sa_initiator)
1417 # testlib.pragma exempt:__hash__
1418 fn(self, value)
1420 _tidy(remove)
1421 return remove
1423 def pop(fn):
1424 def pop(self):
1425 __before_pop(self)
1426 item = fn(self)
1427 # for set in particular, we have no way to access the item
1428 # that will be popped before pop is called.
1429 __del(self, item)
1430 return item
1432 _tidy(pop)
1433 return pop
1435 def clear(fn):
1436 def clear(self):
1437 for item in list(self):
1438 self.remove(item)
1440 _tidy(clear)
1441 return clear
1443 def update(fn):
1444 def update(self, value):
1445 for item in value:
1446 self.add(item)
1448 _tidy(update)
1449 return update
1451 def __ior__(fn):
1452 def __ior__(self, value):
1453 if not _set_binops_check_strict(self, value):
1454 return NotImplemented
1455 for item in value:
1456 self.add(item)
1457 return self
1459 _tidy(__ior__)
1460 return __ior__
1462 def difference_update(fn):
1463 def difference_update(self, value):
1464 for item in value:
1465 self.discard(item)
1467 _tidy(difference_update)
1468 return difference_update
1470 def __isub__(fn):
1471 def __isub__(self, value):
1472 if not _set_binops_check_strict(self, value):
1473 return NotImplemented
1474 for item in value:
1475 self.discard(item)
1476 return self
1478 _tidy(__isub__)
1479 return __isub__
1481 def intersection_update(fn):
1482 def intersection_update(self, other):
1483 want, have = self.intersection(other), set(self)
1484 remove, add = have - want, want - have
1486 for item in remove:
1487 self.remove(item)
1488 for item in add:
1489 self.add(item)
1491 _tidy(intersection_update)
1492 return intersection_update
1494 def __iand__(fn):
1495 def __iand__(self, other):
1496 if not _set_binops_check_strict(self, other):
1497 return NotImplemented
1498 want, have = self.intersection(other), set(self)
1499 remove, add = have - want, want - have
1501 for item in remove:
1502 self.remove(item)
1503 for item in add:
1504 self.add(item)
1505 return self
1507 _tidy(__iand__)
1508 return __iand__
1510 def symmetric_difference_update(fn):
1511 def symmetric_difference_update(self, other):
1512 want, have = self.symmetric_difference(other), set(self)
1513 remove, add = have - want, want - have
1515 for item in remove:
1516 self.remove(item)
1517 for item in add:
1518 self.add(item)
1520 _tidy(symmetric_difference_update)
1521 return symmetric_difference_update
1523 def __ixor__(fn):
1524 def __ixor__(self, other):
1525 if not _set_binops_check_strict(self, other):
1526 return NotImplemented
1527 want, have = self.symmetric_difference(other), set(self)
1528 remove, add = have - want, want - have
1530 for item in remove:
1531 self.remove(item)
1532 for item in add:
1533 self.add(item)
1534 return self
1536 _tidy(__ixor__)
1537 return __ixor__
1539 l = locals().copy()
1540 l.pop("_tidy")
1541 l.pop("Unspecified")
1542 return l
1545class InstrumentedList(list):
1546 """An instrumented version of the built-in list."""
1549class InstrumentedSet(set):
1550 """An instrumented version of the built-in set."""
1553class InstrumentedDict(dict):
1554 """An instrumented version of the built-in dict."""
1557__canned_instrumentation = {
1558 list: InstrumentedList,
1559 set: InstrumentedSet,
1560 dict: InstrumentedDict,
1561}
1563__interfaces = {
1564 list: (
1565 {"appender": "append", "remover": "remove", "iterator": "__iter__"},
1566 _list_decorators(),
1567 ),
1568 set: (
1569 {"appender": "add", "remover": "remove", "iterator": "__iter__"},
1570 _set_decorators(),
1571 ),
1572 # decorators are required for dicts and object collections.
1573 dict: ({"iterator": "values"}, _dict_decorators())
1574 if util.py3k
1575 else ({"iterator": "itervalues"}, _dict_decorators()),
1576}
1579class MappedCollection(dict):
1580 """A basic dictionary-based collection class.
1582 Extends dict with the minimal bag semantics that collection
1583 classes require. ``set`` and ``remove`` are implemented in terms
1584 of a keying function: any callable that takes an object and
1585 returns an object for use as a dictionary key.
1587 """
1589 def __init__(self, keyfunc):
1590 """Create a new collection with keying provided by keyfunc.
1592 keyfunc may be any callable that takes an object and returns an object
1593 for use as a dictionary key.
1595 The keyfunc will be called every time the ORM needs to add a member by
1596 value-only (such as when loading instances from the database) or
1597 remove a member. The usual cautions about dictionary keying apply-
1598 ``keyfunc(object)`` should return the same output for the life of the
1599 collection. Keying based on mutable properties can result in
1600 unreachable instances "lost" in the collection.
1602 """
1603 self.keyfunc = keyfunc
1605 @collection.appender
1606 @collection.internally_instrumented
1607 def set(self, value, _sa_initiator=None):
1608 """Add an item by value, consulting the keyfunc for the key."""
1610 key = self.keyfunc(value)
1611 self.__setitem__(key, value, _sa_initiator)
1613 @collection.remover
1614 @collection.internally_instrumented
1615 def remove(self, value, _sa_initiator=None):
1616 """Remove an item by value, consulting the keyfunc for the key."""
1618 key = self.keyfunc(value)
1619 # Let self[key] raise if key is not in this collection
1620 # testlib.pragma exempt:__ne__
1621 if self[key] != value:
1622 raise sa_exc.InvalidRequestError(
1623 "Can not remove '%s': collection holds '%s' for key '%s'. "
1624 "Possible cause: is the MappedCollection key function "
1625 "based on mutable properties or properties that only obtain "
1626 "values after flush?" % (value, self[key], key)
1627 )
1628 self.__delitem__(key, _sa_initiator)
1631# ensure instrumentation is associated with
1632# these built-in classes; if a user-defined class
1633# subclasses these and uses @internally_instrumented,
1634# the superclass is otherwise not instrumented.
1635# see [ticket:2406].
1636_instrument_class(MappedCollection)
1637_instrument_class(InstrumentedList)
1638_instrument_class(InstrumentedSet)