grscheller.fp.iterables

Library of iterator related functions and enumerations.

  • iterables are not necessarily iterators
  • at all times iterator protocol is assumed to be followed, that is
    • all iterators are assumed to be iterable
    • for all iterators foo we assume iter(foo) is foo

Concatenating and merging iterables:

  • function concat: sequentially chain iterables
  • function exhaust: shuffle together iterables until all are exhausted
  • function merge: shuffle together iterables until one is exhausted

Dropping and taking values from an iterable:

  • function drop: drop first n values from iterable
  • function dropWhile: drop values from iterable while predicate holds
  • function take: take up to n initial values from iterable
  • function takeWhile: take values from iterable while predicate holds

Reducing and accumulating an iterable:

  • function accumulate: take iterable & function, return iterator of accumulated values
  • function foldL: fold iterable from the left with a function
  • function foldR: fold reversible iterable from the right with a function
  • function foldLsc: fold iterable from left with function and a premature stop condition
  • function foldRsc: fold iterable from right with function and a premature start condition

  • function itargs: function reterning an iterator of its arguments

  1# Copyright 2023-2024 Geoffrey R. Scheller
  2#
  3# Licensed under the Apache License, Version 2.0 (the "License");
  4# you may not use this file except in compliance with the License.
  5# You may obtain a copy of the License at
  6#
  7#     http://www.apache.org/licenses/LICENSE-2.0
  8#
  9# Unless required by applicable law or agreed to in writing, software
 10# distributed under the License is distributed on an "AS IS" BASIS,
 11# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 12# See the License for the specific language governing permissions and
 13# limitations under the License.
 14
 15"""### Module fp.iterables - Iterator related tools
 16
 17Library of iterator related functions and enumerations.
 18
 19* iterables are not necessarily iterators
 20* at all times iterator protocol is assumed to be followed, that is
 21  * all iterators are assumed to be iterable
 22  * for all iterators `foo` we assume `iter(foo) is foo`
 23
 24#### Concatenating and merging iterables:
 25
 26* **function concat:** sequentially chain iterables
 27* **function exhaust:** shuffle together iterables until all are exhausted
 28* **function merge:** shuffle together iterables until one is exhausted
 29
 30---
 31
 32#### Dropping and taking values from an iterable:
 33
 34* **function drop:** drop first `n` values from iterable
 35* **function dropWhile:** drop values from iterable while predicate holds
 36* **function take:** take up to `n` initial values from iterable
 37* **function takeWhile:** take values from iterable while predicate holds
 38
 39---
 40
 41#### Reducing and accumulating an iterable:
 42
 43* **function accumulate:** take iterable & function, return iterator of accumulated values
 44* **function foldL:** fold iterable from the left with a function
 45* **function foldR:** fold reversible iterable from the right with a function
 46* **function foldLsc:** fold iterable from left with function and a premature stop condition
 47* **function foldRsc:** fold iterable from right with function and a premature start condition
 48
 49---
 50
 51#### Iterator related utility functions
 52
 53* **function itargs:** function reterning an iterator of its arguments
 54
 55---
 56
 57"""
 58from __future__ import annotations
 59from collections.abc import Callable, Iterator, Iterable, Reversible
 60from enum import auto, Enum
 61from typing import cast
 62from .err_handling import MB
 63
 64__all__ = [ 'FM', 'concat', 'merge', 'exhaust',
 65            'drop', 'dropWhile', 'take', 'takeWhile',
 66            'accumulate', 'foldL', 'foldR', 'foldLsc', 'foldRsc',
 67            'itargs' ]
 68
 69## Iterate over multiple Iterables
 70
 71class FM(Enum):
 72    CONCAT = auto()
 73    MERGE = auto()
 74    EXHAUST = auto()
 75
 76def concat[D](*iterables: Iterable[D]) -> Iterator[D]:
 77    """Sequentially concatenate multiple iterables together.
 78
 79    * pure Python version of standard library's `itertools.chain`
 80    * iterator sequentially yields each iterable until all are exhausted
 81    * an infinite iterable will prevent subsequent iterables from yielding any values
 82    * performant to `itertools.chain`
 83
 84    """
 85    for iterator in map(lambda x: iter(x), iterables):
 86        while True:
 87            try:
 88                value = next(iterator)
 89                yield value
 90            except StopIteration:
 91                break
 92
 93def exhaust[D](*iterables: Iterable[D]) -> Iterator[D]:
 94    """Shuffle together multiple iterables until all are exhausted.
 95
 96    * iterator yields until all iterables are exhausted
 97
 98    """
 99    iterList = list(map(lambda x: iter(x), iterables))
100    if (numIters := len(iterList)) > 0:
101        ii = 0
102        values = []
103        while True:
104            try:
105                while ii < numIters:
106                    values.append(next(iterList[ii]))
107                    ii += 1
108                for value in values:
109                    yield value
110                ii = 0
111                values.clear()
112            except StopIteration:
113                numIters -= 1
114                if numIters < 1:
115                    break
116                del iterList[ii]
117        for value in values:
118            yield value
119
120def merge[D](*iterables: Iterable[D], yield_partials: bool=False) -> Iterator[D]:
121    """Shuffle together the `iterables` until one is exhausted.
122
123    * iterator yields until one of the iterables is exhausted
124    * if `yield_partials` is true,
125      * yield any unmatched yielded values from other iterables
126      * prevents data lose
127        * if any of the iterables are iterators with external references
128
129    """
130    iterList = list(map(lambda x: iter(x), iterables))
131    values = []
132    if (numIters := len(iterList)) > 0:
133        while True:
134            try:
135                for ii in range(numIters):
136                    values.append(next(iterList[ii]))
137                for value in values:
138                    yield value
139                values.clear()
140            except StopIteration:
141                break
142        if yield_partials:
143            for value in values:
144                yield value
145
146## dropping and taking
147
148def drop[D](iterable: Iterable[D], n: int) -> Iterator[D]:
149    """Drop the next `n` values from `iterable`."""
150    it = iter(iterable)
151    for _ in range(n):
152        try:
153            value = next(it)
154        except StopIteration:
155            break
156    return it
157
158def dropWhile[D](iterable: Iterable[D], pred: Callable[[D], bool]) -> Iterator[D]:
159    """Drop initial values from `iterable` while predicate is true."""
160    it = iter(iterable)
161    try:
162        value = next(it)
163    except:
164        return it
165
166    while True:
167        try:
168            if not pred(value):
169                break
170            value = next(it)
171        except StopIteration:
172            break
173    return concat((value,), it)
174
175def take[D](iterable: Iterable[D], n: int) -> Iterator[D]:
176    """Take up to `n` values from `iterable`."""
177    it = iter(iterable)
178    for _ in range(n):
179        try:
180            value = next(it)
181            yield value
182        except StopIteration:
183            break
184
185def takeWhile[D](iterable: Iterable[D], pred: Callable[[D], bool]) -> Iterator[D]:
186    """Yield values from `iterable` while predicate is true.
187
188    **Warning**, potential value loss if iterable is iterator with multiple
189    references.
190    """
191    it = iter(iterable)
192    while True:
193        try:
194            value = next(it)
195            if pred(value):
196                yield value
197            else:
198                break
199        except StopIteration:
200            break
201
202## reducing and accumulating
203
204def accumulate[D,L](iterable: Iterable[D], f: Callable[[L, D], L],
205                  initial: L|None=None) -> Iterator[L]:
206    """Returns an iterator of accumulated values.
207
208    * pure Python version of standard library's `itertools.accumulate`
209    * function `f` does not default to addition (for typing flexibility)
210    * begins accumulation with an optional `initial` value
211    * `itertools.accumulate` had mypy issues
212
213    """
214    it = iter(iterable)
215    try:
216        it0 = next(it)
217    except StopIteration:
218        if initial is None:
219            return
220        else:
221            yield initial
222    else:
223        if initial is not None:
224            yield initial
225            acc = f(initial, it0)
226            for ii in it:
227                yield acc
228                acc = f(acc, ii)
229            yield acc
230        else:
231            acc = cast(L, it0)  # in this case L = D
232            for ii in it:
233                yield acc
234                acc = f(acc, ii)
235            yield acc
236
237def foldL[D,L](iterable: Iterable[D],
238          f: Callable[[L, D], L],
239          initial: L|None=None) -> MB[L]:
240    """Folds an iterable left with optional initial value.
241
242    * traditional FP type order given for function `f`
243    * when an initial value is not given then `~L = ~D`
244    * if iterable empty and no `initial` value given, return empty `MB()`
245    * never returns if `iterable` generates an infinite iterator
246
247    """
248    acc: L
249    it = iter(iterable)
250
251    if initial is None:
252        try:
253            acc = cast(L, next(it))  # in this case L = D
254        except StopIteration:
255            return MB()
256    else:
257        acc = initial
258
259    for v in it:
260        acc = f(acc, v)
261
262    return MB(acc)
263
264def foldR[D,R](iterable: Reversible[D],
265          f: Callable[[D, R], R],
266          initial: R|None=None) -> MB[R]:
267    """Folds a reversible `iterable` right with an optional `initial` value.
268
269    * `iterable` needs to be reversible
270    * traditional FP type order given for function `f`
271    * when initial value is not given then `~R = ~D`
272    * if iterable empty and no `initial` value given, return an empty `MB()`
273
274    """
275    acc: R
276    it = reversed(iterable)
277
278    if initial is None:
279        try:
280            acc = cast(R, next(it))  # in this case R = D
281        except StopIteration:
282            return MB()
283    else:
284        acc = initial
285
286    for v in it:
287        acc = f(v, acc)
288
289    return MB(acc)
290
291def foldLsc[D,L,S](iterable: Iterable[D],
292            f: Callable[[L, D], L],
293            initial: L|None=None,
294            stopfold: Callable[[D, S], MB[S]]=lambda d, s: MB(s),
295            istate: S|None=None) -> MB[L]:
296    """Short circuit version of foldL.
297
298    * Callable `stopfold` purpose is to prematurely stop the fold before end
299      * useful for infinite iterables
300
301    """
302    state = cast(MB[S], MB(istate))
303
304    it = iter(iterable)
305
306    if initial is None:
307        try:
308            acc = cast(L, next(it))  # in this case L = D
309        except StopIteration:
310            return MB()
311    else:
312        acc = initial
313
314    for d in it:
315        if (state := stopfold(d, state.get())):
316            acc = f(acc, d)
317        else:
318            break
319
320    return MB(acc)
321
322def foldRsc[D,R,S](iterable: Iterable[D],
323            f: Callable[[D, R], R],
324            initial: R|None=None,
325            startfold: Callable[[D, S], MB[S]]=lambda d, s: MB(s),
326            istate: S|None=None) -> MB[R]:
327    """Short circuit version of foldR.
328
329    * Callable `startfold` purpose is to start fold before reaching the end
330      * does NOT start fold at end and prematurely stop
331      * useful for infinite and non-reversible iterables
332
333    """
334    state = cast(MB[S], MB(istate))
335
336    it = iter(iterable)
337
338    acc: R
339
340    ds: list[D] = []
341    for d in it:
342        if (state := startfold(d, state.get())):
343            ds.append(d)
344        else:
345            break
346
347    if initial is None:
348        if len(ds) == 0:
349            return MB()
350        else:
351            acc = cast(R, ds.pop())  # in this case R = D
352    else:
353        acc = initial
354
355    while ds:
356        acc = f(ds.pop(), acc)
357
358    return MB(acc)
359
360## Iterator related utility functions
361# TODO: move to fp.experimental.args
362
363def itargs[A](*args: A) -> Iterator[A]:
364    """Function returning an iterators of its arguments.
365
366    Useful when an API only provides a constructor taking a single iterable.
367    """
368    for arg in args:
369        yield arg
class FM(enum.Enum):
72class FM(Enum):
73    CONCAT = auto()
74    MERGE = auto()
75    EXHAUST = auto()
CONCAT = <FM.CONCAT: 1>
MERGE = <FM.MERGE: 2>
EXHAUST = <FM.EXHAUST: 3>
def concat(*iterables: 'Iterable[D]') -> 'Iterator[D]':
77def concat[D](*iterables: Iterable[D]) -> Iterator[D]:
78    """Sequentially concatenate multiple iterables together.
79
80    * pure Python version of standard library's `itertools.chain`
81    * iterator sequentially yields each iterable until all are exhausted
82    * an infinite iterable will prevent subsequent iterables from yielding any values
83    * performant to `itertools.chain`
84
85    """
86    for iterator in map(lambda x: iter(x), iterables):
87        while True:
88            try:
89                value = next(iterator)
90                yield value
91            except StopIteration:
92                break

Sequentially concatenate multiple iterables together.

  • pure Python version of standard library's itertools.chain
  • iterator sequentially yields each iterable until all are exhausted
  • an infinite iterable will prevent subsequent iterables from yielding any values
  • performant to itertools.chain
def merge(*iterables: 'Iterable[D]', yield_partials: bool = False) -> 'Iterator[D]':
121def merge[D](*iterables: Iterable[D], yield_partials: bool=False) -> Iterator[D]:
122    """Shuffle together the `iterables` until one is exhausted.
123
124    * iterator yields until one of the iterables is exhausted
125    * if `yield_partials` is true,
126      * yield any unmatched yielded values from other iterables
127      * prevents data lose
128        * if any of the iterables are iterators with external references
129
130    """
131    iterList = list(map(lambda x: iter(x), iterables))
132    values = []
133    if (numIters := len(iterList)) > 0:
134        while True:
135            try:
136                for ii in range(numIters):
137                    values.append(next(iterList[ii]))
138                for value in values:
139                    yield value
140                values.clear()
141            except StopIteration:
142                break
143        if yield_partials:
144            for value in values:
145                yield value

Shuffle together the iterables until one is exhausted.

  • iterator yields until one of the iterables is exhausted
  • if yield_partials is true,
    • yield any unmatched yielded values from other iterables
    • prevents data lose
      • if any of the iterables are iterators with external references
def exhaust(*iterables: 'Iterable[D]') -> 'Iterator[D]':
 94def exhaust[D](*iterables: Iterable[D]) -> Iterator[D]:
 95    """Shuffle together multiple iterables until all are exhausted.
 96
 97    * iterator yields until all iterables are exhausted
 98
 99    """
100    iterList = list(map(lambda x: iter(x), iterables))
101    if (numIters := len(iterList)) > 0:
102        ii = 0
103        values = []
104        while True:
105            try:
106                while ii < numIters:
107                    values.append(next(iterList[ii]))
108                    ii += 1
109                for value in values:
110                    yield value
111                ii = 0
112                values.clear()
113            except StopIteration:
114                numIters -= 1
115                if numIters < 1:
116                    break
117                del iterList[ii]
118        for value in values:
119            yield value

Shuffle together multiple iterables until all are exhausted.

  • iterator yields until all iterables are exhausted
def drop(iterable: 'Iterable[D]', n: int) -> 'Iterator[D]':
149def drop[D](iterable: Iterable[D], n: int) -> Iterator[D]:
150    """Drop the next `n` values from `iterable`."""
151    it = iter(iterable)
152    for _ in range(n):
153        try:
154            value = next(it)
155        except StopIteration:
156            break
157    return it

Drop the next n values from iterable.

def dropWhile(iterable: 'Iterable[D]', pred: 'Callable[[D], bool]') -> 'Iterator[D]':
159def dropWhile[D](iterable: Iterable[D], pred: Callable[[D], bool]) -> Iterator[D]:
160    """Drop initial values from `iterable` while predicate is true."""
161    it = iter(iterable)
162    try:
163        value = next(it)
164    except:
165        return it
166
167    while True:
168        try:
169            if not pred(value):
170                break
171            value = next(it)
172        except StopIteration:
173            break
174    return concat((value,), it)

Drop initial values from iterable while predicate is true.

def take(iterable: 'Iterable[D]', n: int) -> 'Iterator[D]':
176def take[D](iterable: Iterable[D], n: int) -> Iterator[D]:
177    """Take up to `n` values from `iterable`."""
178    it = iter(iterable)
179    for _ in range(n):
180        try:
181            value = next(it)
182            yield value
183        except StopIteration:
184            break

Take up to n values from iterable.

def takeWhile(iterable: 'Iterable[D]', pred: 'Callable[[D], bool]') -> 'Iterator[D]':
186def takeWhile[D](iterable: Iterable[D], pred: Callable[[D], bool]) -> Iterator[D]:
187    """Yield values from `iterable` while predicate is true.
188
189    **Warning**, potential value loss if iterable is iterator with multiple
190    references.
191    """
192    it = iter(iterable)
193    while True:
194        try:
195            value = next(it)
196            if pred(value):
197                yield value
198            else:
199                break
200        except StopIteration:
201            break

Yield values from iterable while predicate is true.

Warning, potential value loss if iterable is iterator with multiple references.

def accumulate( iterable: 'Iterable[D]', f: 'Callable[[L, D], L]', initial: 'L | None' = None) -> 'Iterator[L]':
205def accumulate[D,L](iterable: Iterable[D], f: Callable[[L, D], L],
206                  initial: L|None=None) -> Iterator[L]:
207    """Returns an iterator of accumulated values.
208
209    * pure Python version of standard library's `itertools.accumulate`
210    * function `f` does not default to addition (for typing flexibility)
211    * begins accumulation with an optional `initial` value
212    * `itertools.accumulate` had mypy issues
213
214    """
215    it = iter(iterable)
216    try:
217        it0 = next(it)
218    except StopIteration:
219        if initial is None:
220            return
221        else:
222            yield initial
223    else:
224        if initial is not None:
225            yield initial
226            acc = f(initial, it0)
227            for ii in it:
228                yield acc
229                acc = f(acc, ii)
230            yield acc
231        else:
232            acc = cast(L, it0)  # in this case L = D
233            for ii in it:
234                yield acc
235                acc = f(acc, ii)
236            yield acc

Returns an iterator of accumulated values.

  • pure Python version of standard library's itertools.accumulate
  • function f does not default to addition (for typing flexibility)
  • begins accumulation with an optional initial value
  • itertools.accumulate had mypy issues
def foldL( iterable: 'Iterable[D]', f: 'Callable[[L, D], L]', initial: 'L | None' = None) -> 'MB[L]':
238def foldL[D,L](iterable: Iterable[D],
239          f: Callable[[L, D], L],
240          initial: L|None=None) -> MB[L]:
241    """Folds an iterable left with optional initial value.
242
243    * traditional FP type order given for function `f`
244    * when an initial value is not given then `~L = ~D`
245    * if iterable empty and no `initial` value given, return empty `MB()`
246    * never returns if `iterable` generates an infinite iterator
247
248    """
249    acc: L
250    it = iter(iterable)
251
252    if initial is None:
253        try:
254            acc = cast(L, next(it))  # in this case L = D
255        except StopIteration:
256            return MB()
257    else:
258        acc = initial
259
260    for v in it:
261        acc = f(acc, v)
262
263    return MB(acc)

Folds an iterable left with optional initial value.

  • traditional FP type order given for function f
  • when an initial value is not given then ~L = ~D
  • if iterable empty and no initial value given, return empty MB()
  • never returns if iterable generates an infinite iterator
def foldR( iterable: 'Reversible[D]', f: 'Callable[[D, R], R]', initial: 'R | None' = None) -> 'MB[R]':
265def foldR[D,R](iterable: Reversible[D],
266          f: Callable[[D, R], R],
267          initial: R|None=None) -> MB[R]:
268    """Folds a reversible `iterable` right with an optional `initial` value.
269
270    * `iterable` needs to be reversible
271    * traditional FP type order given for function `f`
272    * when initial value is not given then `~R = ~D`
273    * if iterable empty and no `initial` value given, return an empty `MB()`
274
275    """
276    acc: R
277    it = reversed(iterable)
278
279    if initial is None:
280        try:
281            acc = cast(R, next(it))  # in this case R = D
282        except StopIteration:
283            return MB()
284    else:
285        acc = initial
286
287    for v in it:
288        acc = f(v, acc)
289
290    return MB(acc)

Folds a reversible iterable right with an optional initial value.

  • iterable needs to be reversible
  • traditional FP type order given for function f
  • when initial value is not given then ~R = ~D
  • if iterable empty and no initial value given, return an empty MB()
def foldLsc( iterable: 'Iterable[D]', f: 'Callable[[L, D], L]', initial: 'L | None' = None, stopfold: 'Callable[[D, S], MB[S]]' = <function <lambda>>, istate: 'S | None' = None) -> 'MB[L]':
292def foldLsc[D,L,S](iterable: Iterable[D],
293            f: Callable[[L, D], L],
294            initial: L|None=None,
295            stopfold: Callable[[D, S], MB[S]]=lambda d, s: MB(s),
296            istate: S|None=None) -> MB[L]:
297    """Short circuit version of foldL.
298
299    * Callable `stopfold` purpose is to prematurely stop the fold before end
300      * useful for infinite iterables
301
302    """
303    state = cast(MB[S], MB(istate))
304
305    it = iter(iterable)
306
307    if initial is None:
308        try:
309            acc = cast(L, next(it))  # in this case L = D
310        except StopIteration:
311            return MB()
312    else:
313        acc = initial
314
315    for d in it:
316        if (state := stopfold(d, state.get())):
317            acc = f(acc, d)
318        else:
319            break
320
321    return MB(acc)

Short circuit version of foldL.

  • Callable stopfold purpose is to prematurely stop the fold before end
    • useful for infinite iterables
def foldRsc( iterable: 'Iterable[D]', f: 'Callable[[D, R], R]', initial: 'R | None' = None, startfold: 'Callable[[D, S], MB[S]]' = <function <lambda>>, istate: 'S | None' = None) -> 'MB[R]':
323def foldRsc[D,R,S](iterable: Iterable[D],
324            f: Callable[[D, R], R],
325            initial: R|None=None,
326            startfold: Callable[[D, S], MB[S]]=lambda d, s: MB(s),
327            istate: S|None=None) -> MB[R]:
328    """Short circuit version of foldR.
329
330    * Callable `startfold` purpose is to start fold before reaching the end
331      * does NOT start fold at end and prematurely stop
332      * useful for infinite and non-reversible iterables
333
334    """
335    state = cast(MB[S], MB(istate))
336
337    it = iter(iterable)
338
339    acc: R
340
341    ds: list[D] = []
342    for d in it:
343        if (state := startfold(d, state.get())):
344            ds.append(d)
345        else:
346            break
347
348    if initial is None:
349        if len(ds) == 0:
350            return MB()
351        else:
352            acc = cast(R, ds.pop())  # in this case R = D
353    else:
354        acc = initial
355
356    while ds:
357        acc = f(ds.pop(), acc)
358
359    return MB(acc)

Short circuit version of foldR.

  • Callable startfold purpose is to start fold before reaching the end
    • does NOT start fold at end and prematurely stop
    • useful for infinite and non-reversible iterables
def itargs(*args: 'A') -> 'Iterator[A]':
364def itargs[A](*args: A) -> Iterator[A]:
365    """Function returning an iterators of its arguments.
366
367    Useful when an API only provides a constructor taking a single iterable.
368    """
369    for arg in args:
370        yield arg

Function returning an iterators of its arguments.

Useful when an API only provides a constructor taking a single iterable.