grscheller.fp.iterables
Module fp.iterables - Iterator related tools
Library of iterator related functions and enumerations.
- iterables are not necessarily iterators
- at all times iterator protocol is assumed to be followed, that is
- all iterators are assumed to be iterable
- for all iterators
foo
we assumeiter(foo) is foo
Concatenating and merging iterables:
- function concat: sequentially chain iterables
- function exhaust: shuffle together iterables until all are exhausted
- function merge: shuffle together iterables until one is exhausted
Dropping and taking values from an iterable:
- function drop: drop first
n
values from iterable - function drop_while: drop values from iterable while predicate holds
- function take: take up to
n
initial values from iterable - function take_split: splitting out initial
n
initial values of iterable * function take_while: take values from iterable while predicate holds - function take_while_split: splitting an iterable while predicate holds
Reducing and accumulating an iterable:
- function accumulate: take iterable & function, return iterator of accumulated values
- function foldL0: fold iterable left with a function
- raises
StopIteration
exception if iterable is empty
- raises
- function foldL1: fold iterable left with a function and initial value
- function mbFoldL: fold iterable left with an optional initial value
- wraps result in a
MB
monad
- wraps result in a
1# Copyright 2023-2025 Geoffrey R. Scheller 2# 3# Licensed under the Apache License, Version 2.0 (the "License"); 4# you may not use this file except in compliance with the License. 5# You may obtain a copy of the License at 6# 7# http://www.apache.org/licenses/LICENSE-2.0 8# 9# Unless required by applicable law or agreed to in writing, software 10# distributed under the License is distributed on an "AS IS" BASIS, 11# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12# See the License for the specific language governing permissions and 13# limitations under the License. 14 15"""### Module fp.iterables - Iterator related tools 16 17Library of iterator related functions and enumerations. 18 19* iterables are not necessarily iterators 20* at all times iterator protocol is assumed to be followed, that is 21 * all iterators are assumed to be iterable 22 * for all iterators `foo` we assume `iter(foo) is foo` 23 24#### Concatenating and merging iterables: 25 26* function **concat**: sequentially chain iterables 27* function **exhaust**: shuffle together iterables until all are exhausted 28* function **merge**: shuffle together iterables until one is exhausted 29 30--- 31 32#### Dropping and taking values from an iterable: 33 34* function **drop**: drop first `n` values from iterable 35* function **drop_while**: drop values from iterable while predicate holds 36* function **take**: take up to `n` initial values from iterable 37* function **take_split**: splitting out initial `n` initial values of iterable * function **take_while**: take values from iterable while predicate holds 38* function **take_while_split**: splitting an iterable while predicate holds 39 40--- 41 42#### Reducing and accumulating an iterable: 43 44* function **accumulate**: take iterable & function, return iterator of accumulated values 45* function **foldL0**: fold iterable left with a function 46 * raises `StopIteration` exception if iterable is empty 47* function **foldL1**: fold iterable left with a function and initial value 48* function **mbFoldL**: fold iterable left with an optional initial value 49 * wraps result in a `MB` monad 50 51""" 52from __future__ import annotations 53from collections.abc import Callable, Iterator, Iterable, Reversible 54from enum import auto, Enum 55from typing import cast, Never 56from .err_handling import MB 57from .function import swap 58from .singletons import NoValue 59 60__all__ = [ 'FM', 'concat', 'merge', 'exhaust', 61 'drop', 'drop_while', 62 'take', 'take_while', 63 'take_split', 'take_while_split', 64 'accumulate', 'foldL0', 'foldL1', 'mbFoldL' ] #, 65 # 'scFoldL', 'scFoldR' ] 66 67## Iterate over multiple Iterables 68 69class FM(Enum): 70 CONCAT = auto() 71 MERGE = auto() 72 EXHAUST = auto() 73 74def concat[D](*iterables: Iterable[D]) -> Iterator[D]: 75 """Sequentially concatenate multiple iterables together. 76 77 * pure Python version of standard library's `itertools.chain` 78 * iterator sequentially yields each iterable until all are exhausted 79 * an infinite iterable will prevent subsequent iterables from yielding any values 80 * performant to `itertools.chain` 81 82 """ 83 for iterator in map(lambda x: iter(x), iterables): 84 while True: 85 try: 86 value = next(iterator) 87 yield value 88 except StopIteration: 89 break 90 91def exhaust[D](*iterables: Iterable[D]) -> Iterator[D]: 92 """Shuffle together multiple iterables until all are exhausted. 93 94 * iterator yields until all iterables are exhausted 95 96 """ 97 iterList = list(map(lambda x: iter(x), iterables)) 98 if (numIters := len(iterList)) > 0: 99 ii = 0 100 values = [] 101 while True: 102 try: 103 while ii < numIters: 104 values.append(next(iterList[ii])) 105 ii += 1 106 for value in values: 107 yield value 108 ii = 0 109 values.clear() 110 except StopIteration: 111 numIters -= 1 112 if numIters < 1: 113 break 114 del iterList[ii] 115 for value in values: 116 yield value 117 118def merge[D](*iterables: Iterable[D], yield_partials: bool=False) -> Iterator[D]: 119 """Shuffle together the `iterables` until one is exhausted. 120 121 * iterator yields until one of the iterables is exhausted 122 * if `yield_partials` is true, 123 * yield any unmatched yielded values from other iterables 124 * prevents data lose 125 * if any of the iterables are iterators with external references 126 127 """ 128 iterList = list(map(lambda x: iter(x), iterables)) 129 values = [] 130 if (numIters := len(iterList)) > 0: 131 while True: 132 try: 133 for ii in range(numIters): 134 values.append(next(iterList[ii])) 135 for value in values: 136 yield value 137 values.clear() 138 except StopIteration: 139 break 140 if yield_partials: 141 for value in values: 142 yield value 143 144## dropping and taking 145 146def drop[D]( 147 iterable: Iterable[D], 148 n: int, / 149 ) -> Iterator[D]: 150 """Drop the next `n` values from `iterable`.""" 151 it = iter(iterable) 152 for _ in range(n): 153 try: 154 next(it) 155 except StopIteration: 156 break 157 return it 158 159def drop_while[D]( 160 iterable: Iterable[D], 161 predicate: Callable[[D], bool], / 162 ) -> Iterator[D]: 163 """Drop initial values from `iterable` while predicate is true.""" 164 it = iter(iterable) 165 while True: 166 try: 167 value = next(it) 168 if not predicate(value): 169 it = concat((value,), it) 170 break 171 except StopIteration: 172 break 173 return it 174 175def take[D]( 176 iterable: Iterable[D], 177 n: int, / 178 ) -> Iterator[D]: 179 """Return an iterator of up to `n` initial values of an iterable""" 180 it = iter(iterable) 181 for _ in range(n): 182 try: 183 value = next(it) 184 yield value 185 except StopIteration: 186 break 187 188def take_split[D]( 189 iterable: Iterable[D], 190 n: int, / 191 ) -> tuple[Iterator[D], Iterator[D]]: 192 """Same as take except also return an iterator of the remaining values. 193 194 * return a tuple of 195 * an iterator of up to `n` initial values 196 * an iterator of the remaining vales of the `iterable` 197 * best practice is not to access second iterator until first is exhausted 198 199 """ 200 it = iter(iterable) 201 itn = take(it, n) 202 203 return itn, it 204 205def take_while[D]( 206 iterable: Iterable[D], 207 pred: Callable[[D], bool], / 208 ) -> Iterator[D]: 209 """Yield values from `iterable` while predicate is true. 210 211 **Warning:** risk of potential value loss if iterable is iterator with 212 multiple references. 213 """ 214 it = iter(iterable) 215 while True: 216 try: 217 value = next(it) 218 if pred(value): 219 yield value 220 else: 221 break 222 except StopIteration: 223 break 224 225def take_while_split[D]( 226 iterable: Iterable[D], 227 predicate: Callable[[D], bool], / 228 ) -> tuple[Iterator[D], Iterator[D]]: 229 """Yield values from `iterable` while `predicate` is true. 230 231 * return a tuple of two iterators 232 * first of initial values where predicate is true, followed by first to fail 233 * second of the remaining values of the iterable after first failed value 234 * best practice is not to access second iterator until first is exhausted 235 236 """ 237 def _take_while(it: Iterator[D], pred: Callable[[D], bool], val: list[D]) -> Iterator[D]: 238 while True: 239 try: 240 if val: 241 val[0] = next(it) 242 else: 243 val.append(next(it)) 244 if pred(val[0]): 245 yield val[0] 246 val.pop() 247 else: 248 break 249 except StopIteration: 250 break 251 252 it = iter(iterable) 253 value: list[D] = [] 254 it_pred = _take_while(it, predicate, value) 255 256 return (it_pred, concat(cast(list[D], value), it)) 257 258## reducing and accumulating 259 260def accumulate[D,L]( 261 iterable: Iterable[D], 262 f: Callable[[L, D], L], 263 initial: L|NoValue=NoValue(), / 264 ) -> Iterator[L]: 265 """Returns an iterator of accumulated values. 266 267 * pure Python version of standard library's `itertools.accumulate` 268 * function `f` does not default to addition (for typing flexibility) 269 * begins accumulation with an optional `initial` value 270 271 """ 272 it = iter(iterable) 273 try: 274 it0 = next(it) 275 except StopIteration: 276 if initial is NoValue(): 277 return 278 else: 279 yield cast(L, initial) 280 else: 281 if initial is not NoValue(): 282 init = cast(L, initial) 283 yield init 284 acc = f(init, it0) 285 for ii in it: 286 yield acc 287 acc = f(acc, ii) 288 yield acc 289 else: 290 acc = cast(L, it0) # in this case L = D 291 for ii in it: 292 yield acc 293 acc = f(acc, ii) 294 yield acc 295 296def foldL0[D]( 297 iterable: Iterable[D], 298 f: Callable[[D, D], D], / 299 ) -> D|Never: 300 """Folds an iterable left with optional initial value. 301 302 * traditional FP type order given for function `f` 303 * if iterable empty raises StopIteration exception 304 * does not catch any exception `f` raises 305 * never returns if `iterable` generates an infinite iterator 306 307 """ 308 it = iter(iterable) 309 try: 310 acc = next(it) 311 except StopIteration: 312 msg = "Attemped to left fold an empty iterable." 313 raise StopIteration(msg) 314 315 for v in it: 316 acc = f(acc, v) 317 318 return acc 319 320def foldL1[D, L]( 321 iterable: Iterable[D], 322 f: Callable[[L, D], L], 323 initial: L, / 324 ) -> L|Never: 325 """Folds an iterable left with optional initial value. 326 327 * traditional FP type order given for function `f` 328 * does not catch any exception `f` raises 329 * never returns if `iterable` generates an infinite iterator 330 331 """ 332 acc = initial 333 for v in iterable: 334 acc = f(acc, v) 335 return acc 336 337def mbFoldL[L, D]( 338 iterable: Iterable[D], 339 f: Callable[[L, D], L], 340 initial: L|NoValue=NoValue() 341 ) -> MB[L]: 342 """Folds an iterable left with optional initial value. 343 344 * traditional FP type order given for function `f` 345 * when an initial value is not given then `~L = ~D` 346 * if iterable empty and no `initial` value given, return `MB()` 347 * never returns if iterable generates an infinite iterator 348 349 """ 350 acc: L 351 it = iter(iterable) 352 if initial is NoValue(): 353 try: 354 acc = cast(L, next(it)) # in this case L = D 355 except StopIteration: 356 return MB() 357 else: 358 acc = cast(L, initial) 359 360 for v in it: 361 try: 362 acc = f(acc, v) 363 except Exception: 364 return MB() 365 366 return MB(acc) 367 368#def scFoldL[D, L](iterable: Iterable[D], 369# f: Callable[[L, D], L], 370# initial: L|NoValue=NoValue(), /, 371# start_folding: Callable[[D], bool]=lambda d: True, 372# stop_folding: Callable[[D], bool]=lambda d: False, 373# include_start: bool=True, 374# propagate_failed: bool=True) -> tuple[MB[L], Iterable[D]]: 375# """Short circuit version of a left fold. Useful for infinite or 376# non-reversible iterables. 377# 378# * Behavior for default arguments will 379# * left fold finite iterable 380# * start folding immediately 381# * continue folding until end (of a possibly infinite iterable) 382# * Callable `start_folding` delays starting a left fold 383# * Callable `stop_folding` is to prematurely stop the folding left 384# * Returns an XOR of either the folded value or error string 385# 386# """ 387# 388#def scFoldR[D, R](iterable: Iterable[D], 389# f: Callable[[D, R], R], 390# initial: R|NoValue=NoValue(), /, 391# start_folding: Callable[[D], bool]=lambda d: False, 392# stop_folding: Callable[[D], bool]=lambda d: False, 393# include_start: bool=True, 394# include_stop: bool=True) -> tuple[MB[R], Iterable[D]]: 395# """Short circuit version of a right fold. Useful for infinite or 396# non-reversible iterables. 397# 398# * Behavior for default arguments will 399# * right fold finite iterable 400# * start folding at end (of a possibly infinite iterable) 401# * continue folding right until beginning 402# * Callable `start_folding` prematurely starts a right fold 403# * Callable `stop_folding` is to prematurely stops a right fold 404# * Returns an XOR of either the folded value or error string 405# * best practice is not to access second iterator until first is exhausted 406# 407# """ 408#
75def concat[D](*iterables: Iterable[D]) -> Iterator[D]: 76 """Sequentially concatenate multiple iterables together. 77 78 * pure Python version of standard library's `itertools.chain` 79 * iterator sequentially yields each iterable until all are exhausted 80 * an infinite iterable will prevent subsequent iterables from yielding any values 81 * performant to `itertools.chain` 82 83 """ 84 for iterator in map(lambda x: iter(x), iterables): 85 while True: 86 try: 87 value = next(iterator) 88 yield value 89 except StopIteration: 90 break
Sequentially concatenate multiple iterables together.
- pure Python version of standard library's
itertools.chain
- iterator sequentially yields each iterable until all are exhausted
- an infinite iterable will prevent subsequent iterables from yielding any values
- performant to
itertools.chain
119def merge[D](*iterables: Iterable[D], yield_partials: bool=False) -> Iterator[D]: 120 """Shuffle together the `iterables` until one is exhausted. 121 122 * iterator yields until one of the iterables is exhausted 123 * if `yield_partials` is true, 124 * yield any unmatched yielded values from other iterables 125 * prevents data lose 126 * if any of the iterables are iterators with external references 127 128 """ 129 iterList = list(map(lambda x: iter(x), iterables)) 130 values = [] 131 if (numIters := len(iterList)) > 0: 132 while True: 133 try: 134 for ii in range(numIters): 135 values.append(next(iterList[ii])) 136 for value in values: 137 yield value 138 values.clear() 139 except StopIteration: 140 break 141 if yield_partials: 142 for value in values: 143 yield value
Shuffle together the iterables
until one is exhausted.
- iterator yields until one of the iterables is exhausted
- if
yield_partials
is true,- yield any unmatched yielded values from other iterables
- prevents data lose
- if any of the iterables are iterators with external references
92def exhaust[D](*iterables: Iterable[D]) -> Iterator[D]: 93 """Shuffle together multiple iterables until all are exhausted. 94 95 * iterator yields until all iterables are exhausted 96 97 """ 98 iterList = list(map(lambda x: iter(x), iterables)) 99 if (numIters := len(iterList)) > 0: 100 ii = 0 101 values = [] 102 while True: 103 try: 104 while ii < numIters: 105 values.append(next(iterList[ii])) 106 ii += 1 107 for value in values: 108 yield value 109 ii = 0 110 values.clear() 111 except StopIteration: 112 numIters -= 1 113 if numIters < 1: 114 break 115 del iterList[ii] 116 for value in values: 117 yield value
Shuffle together multiple iterables until all are exhausted.
- iterator yields until all iterables are exhausted
147def drop[D]( 148 iterable: Iterable[D], 149 n: int, / 150 ) -> Iterator[D]: 151 """Drop the next `n` values from `iterable`.""" 152 it = iter(iterable) 153 for _ in range(n): 154 try: 155 next(it) 156 except StopIteration: 157 break 158 return it
Drop the next n
values from iterable
.
160def drop_while[D]( 161 iterable: Iterable[D], 162 predicate: Callable[[D], bool], / 163 ) -> Iterator[D]: 164 """Drop initial values from `iterable` while predicate is true.""" 165 it = iter(iterable) 166 while True: 167 try: 168 value = next(it) 169 if not predicate(value): 170 it = concat((value,), it) 171 break 172 except StopIteration: 173 break 174 return it
Drop initial values from iterable
while predicate is true.
176def take[D]( 177 iterable: Iterable[D], 178 n: int, / 179 ) -> Iterator[D]: 180 """Return an iterator of up to `n` initial values of an iterable""" 181 it = iter(iterable) 182 for _ in range(n): 183 try: 184 value = next(it) 185 yield value 186 except StopIteration: 187 break
Return an iterator of up to n
initial values of an iterable
206def take_while[D]( 207 iterable: Iterable[D], 208 pred: Callable[[D], bool], / 209 ) -> Iterator[D]: 210 """Yield values from `iterable` while predicate is true. 211 212 **Warning:** risk of potential value loss if iterable is iterator with 213 multiple references. 214 """ 215 it = iter(iterable) 216 while True: 217 try: 218 value = next(it) 219 if pred(value): 220 yield value 221 else: 222 break 223 except StopIteration: 224 break
Yield values from iterable
while predicate is true.
Warning: risk of potential value loss if iterable is iterator with multiple references.
189def take_split[D]( 190 iterable: Iterable[D], 191 n: int, / 192 ) -> tuple[Iterator[D], Iterator[D]]: 193 """Same as take except also return an iterator of the remaining values. 194 195 * return a tuple of 196 * an iterator of up to `n` initial values 197 * an iterator of the remaining vales of the `iterable` 198 * best practice is not to access second iterator until first is exhausted 199 200 """ 201 it = iter(iterable) 202 itn = take(it, n) 203 204 return itn, it
Same as take except also return an iterator of the remaining values.
- return a tuple of
- an iterator of up to
n
initial values - an iterator of the remaining vales of the
iterable
- an iterator of up to
- best practice is not to access second iterator until first is exhausted
226def take_while_split[D]( 227 iterable: Iterable[D], 228 predicate: Callable[[D], bool], / 229 ) -> tuple[Iterator[D], Iterator[D]]: 230 """Yield values from `iterable` while `predicate` is true. 231 232 * return a tuple of two iterators 233 * first of initial values where predicate is true, followed by first to fail 234 * second of the remaining values of the iterable after first failed value 235 * best practice is not to access second iterator until first is exhausted 236 237 """ 238 def _take_while(it: Iterator[D], pred: Callable[[D], bool], val: list[D]) -> Iterator[D]: 239 while True: 240 try: 241 if val: 242 val[0] = next(it) 243 else: 244 val.append(next(it)) 245 if pred(val[0]): 246 yield val[0] 247 val.pop() 248 else: 249 break 250 except StopIteration: 251 break 252 253 it = iter(iterable) 254 value: list[D] = [] 255 it_pred = _take_while(it, predicate, value) 256 257 return (it_pred, concat(cast(list[D], value), it))
Yield values from iterable
while predicate
is true.
- return a tuple of two iterators
- first of initial values where predicate is true, followed by first to fail
- second of the remaining values of the iterable after first failed value
- best practice is not to access second iterator until first is exhausted
261def accumulate[D,L]( 262 iterable: Iterable[D], 263 f: Callable[[L, D], L], 264 initial: L|NoValue=NoValue(), / 265 ) -> Iterator[L]: 266 """Returns an iterator of accumulated values. 267 268 * pure Python version of standard library's `itertools.accumulate` 269 * function `f` does not default to addition (for typing flexibility) 270 * begins accumulation with an optional `initial` value 271 272 """ 273 it = iter(iterable) 274 try: 275 it0 = next(it) 276 except StopIteration: 277 if initial is NoValue(): 278 return 279 else: 280 yield cast(L, initial) 281 else: 282 if initial is not NoValue(): 283 init = cast(L, initial) 284 yield init 285 acc = f(init, it0) 286 for ii in it: 287 yield acc 288 acc = f(acc, ii) 289 yield acc 290 else: 291 acc = cast(L, it0) # in this case L = D 292 for ii in it: 293 yield acc 294 acc = f(acc, ii) 295 yield acc
Returns an iterator of accumulated values.
- pure Python version of standard library's
itertools.accumulate
- function
f
does not default to addition (for typing flexibility) - begins accumulation with an optional
initial
value
297def foldL0[D]( 298 iterable: Iterable[D], 299 f: Callable[[D, D], D], / 300 ) -> D|Never: 301 """Folds an iterable left with optional initial value. 302 303 * traditional FP type order given for function `f` 304 * if iterable empty raises StopIteration exception 305 * does not catch any exception `f` raises 306 * never returns if `iterable` generates an infinite iterator 307 308 """ 309 it = iter(iterable) 310 try: 311 acc = next(it) 312 except StopIteration: 313 msg = "Attemped to left fold an empty iterable." 314 raise StopIteration(msg) 315 316 for v in it: 317 acc = f(acc, v) 318 319 return acc
Folds an iterable left with optional initial value.
- traditional FP type order given for function
f
- if iterable empty raises StopIteration exception
- does not catch any exception
f
raises - never returns if
iterable
generates an infinite iterator
321def foldL1[D, L]( 322 iterable: Iterable[D], 323 f: Callable[[L, D], L], 324 initial: L, / 325 ) -> L|Never: 326 """Folds an iterable left with optional initial value. 327 328 * traditional FP type order given for function `f` 329 * does not catch any exception `f` raises 330 * never returns if `iterable` generates an infinite iterator 331 332 """ 333 acc = initial 334 for v in iterable: 335 acc = f(acc, v) 336 return acc
Folds an iterable left with optional initial value.
- traditional FP type order given for function
f
- does not catch any exception
f
raises - never returns if
iterable
generates an infinite iterator
338def mbFoldL[L, D]( 339 iterable: Iterable[D], 340 f: Callable[[L, D], L], 341 initial: L|NoValue=NoValue() 342 ) -> MB[L]: 343 """Folds an iterable left with optional initial value. 344 345 * traditional FP type order given for function `f` 346 * when an initial value is not given then `~L = ~D` 347 * if iterable empty and no `initial` value given, return `MB()` 348 * never returns if iterable generates an infinite iterator 349 350 """ 351 acc: L 352 it = iter(iterable) 353 if initial is NoValue(): 354 try: 355 acc = cast(L, next(it)) # in this case L = D 356 except StopIteration: 357 return MB() 358 else: 359 acc = cast(L, initial) 360 361 for v in it: 362 try: 363 acc = f(acc, v) 364 except Exception: 365 return MB() 366 367 return MB(acc)
Folds an iterable left with optional initial value.
- traditional FP type order given for function
f
- when an initial value is not given then
~L = ~D
- if iterable empty and no
initial
value given, returnMB()
- never returns if iterable generates an infinite iterator