grscheller.fp.iterables
Module fp.iterables - Iterator related tools
Library of iterator related functions and enumerations.
- iterables are not necessarily iterators
- at all times iterator protocol is assumed to be followed, that is
- all iterators are assumed to be iterable
- for all iterators
foo
we assumeiter(foo) is foo
Concatenating and merging iterables:
- function concat: sequentially chain iterables
- function exhaust: shuffle together iterables until all are exhausted
- function merge: shuffle together iterables until one is exhausted
Dropping and taking values from an iterable:
- function drop: drop first
n
values from iterable - function dropWhile: drop values from iterable while predicate holds
- function take: take up to
n
initial values from iterable - function takeWhile: take values from iterable while predicate holds
Reducing and accumulating an iterable:
- function accumulate: take iterable & function, return iterator of accumulated values
- function foldL: fold iterable from the left with a function
- function foldR: fold reversible iterable from the right with a function
- function foldLsc: fold iterable from left with function and a premature stop condition
- function foldRsc: fold iterable from right with function and a premature start condition
Iterator related utility functions
- function itargs: function reterning an iterator of its arguments
1# Copyright 2023-2024 Geoffrey R. Scheller 2# 3# Licensed under the Apache License, Version 2.0 (the "License"); 4# you may not use this file except in compliance with the License. 5# You may obtain a copy of the License at 6# 7# http://www.apache.org/licenses/LICENSE-2.0 8# 9# Unless required by applicable law or agreed to in writing, software 10# distributed under the License is distributed on an "AS IS" BASIS, 11# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12# See the License for the specific language governing permissions and 13# limitations under the License. 14 15"""### Module fp.iterables - Iterator related tools 16 17Library of iterator related functions and enumerations. 18 19* iterables are not necessarily iterators 20* at all times iterator protocol is assumed to be followed, that is 21 * all iterators are assumed to be iterable 22 * for all iterators `foo` we assume `iter(foo) is foo` 23 24#### Concatenating and merging iterables: 25 26* **function concat:** sequentially chain iterables 27* **function exhaust:** shuffle together iterables until all are exhausted 28* **function merge:** shuffle together iterables until one is exhausted 29 30--- 31 32#### Dropping and taking values from an iterable: 33 34* **function drop:** drop first `n` values from iterable 35* **function dropWhile:** drop values from iterable while predicate holds 36* **function take:** take up to `n` initial values from iterable 37* **function takeWhile:** take values from iterable while predicate holds 38 39--- 40 41#### Reducing and accumulating an iterable: 42 43* **function accumulate:** take iterable & function, return iterator of accumulated values 44* **function foldL:** fold iterable from the left with a function 45* **function foldR:** fold reversible iterable from the right with a function 46* **function foldLsc:** fold iterable from left with function and a premature stop condition 47* **function foldRsc:** fold iterable from right with function and a premature start condition 48 49--- 50 51#### Iterator related utility functions 52 53* **function itargs:** function reterning an iterator of its arguments 54 55--- 56 57""" 58from __future__ import annotations 59from collections.abc import Callable, Iterator, Iterable, Reversible 60from enum import auto, Enum 61from typing import cast 62from .err_handling import MB 63 64__all__ = [ 'FM', 'concat', 'merge', 'exhaust', 65 'drop', 'dropWhile', 'take', 'takeWhile', 66 'accumulate', 'foldL', 'foldR', 'foldLsc', 'foldRsc', 67 'itargs' ] 68 69## Iterate over multiple Iterables 70 71class FM(Enum): 72 CONCAT = auto() 73 MERGE = auto() 74 EXHAUST = auto() 75 76def concat[D](*iterables: Iterable[D]) -> Iterator[D]: 77 """Sequentially concatenate multiple iterables together. 78 79 * pure Python version of standard library's `itertools.chain` 80 * iterator sequentially yields each iterable until all are exhausted 81 * an infinite iterable will prevent subsequent iterables from yielding any values 82 * performant to `itertools.chain` 83 84 """ 85 for iterator in map(lambda x: iter(x), iterables): 86 while True: 87 try: 88 value = next(iterator) 89 yield value 90 except StopIteration: 91 break 92 93def exhaust[D](*iterables: Iterable[D]) -> Iterator[D]: 94 """Shuffle together multiple iterables until all are exhausted. 95 96 * iterator yields until all iterables are exhausted 97 98 """ 99 iterList = list(map(lambda x: iter(x), iterables)) 100 if (numIters := len(iterList)) > 0: 101 ii = 0 102 values = [] 103 while True: 104 try: 105 while ii < numIters: 106 values.append(next(iterList[ii])) 107 ii += 1 108 for value in values: 109 yield value 110 ii = 0 111 values.clear() 112 except StopIteration: 113 numIters -= 1 114 if numIters < 1: 115 break 116 del iterList[ii] 117 for value in values: 118 yield value 119 120def merge[D](*iterables: Iterable[D], yield_partials: bool=False) -> Iterator[D]: 121 """Shuffle together the `iterables` until one is exhausted. 122 123 * iterator yields until one of the iterables is exhausted 124 * if `yield_partials` is true, 125 * yield any unmatched yielded values from other iterables 126 * prevents data lose 127 * if any of the iterables are iterators with external references 128 129 """ 130 iterList = list(map(lambda x: iter(x), iterables)) 131 values = [] 132 if (numIters := len(iterList)) > 0: 133 while True: 134 try: 135 for ii in range(numIters): 136 values.append(next(iterList[ii])) 137 for value in values: 138 yield value 139 values.clear() 140 except StopIteration: 141 break 142 if yield_partials: 143 for value in values: 144 yield value 145 146## dropping and taking 147 148def drop[D](iterable: Iterable[D], n: int) -> Iterator[D]: 149 """Drop the next `n` values from `iterable`.""" 150 it = iter(iterable) 151 for _ in range(n): 152 try: 153 value = next(it) 154 except StopIteration: 155 break 156 return it 157 158def dropWhile[D](iterable: Iterable[D], pred: Callable[[D], bool]) -> Iterator[D]: 159 """Drop initial values from `iterable` while predicate is true.""" 160 it = iter(iterable) 161 try: 162 value = next(it) 163 except: 164 return it 165 166 while True: 167 try: 168 if not pred(value): 169 break 170 value = next(it) 171 except StopIteration: 172 break 173 return concat((value,), it) 174 175def take[D](iterable: Iterable[D], n: int) -> Iterator[D]: 176 """Take up to `n` values from `iterable`.""" 177 it = iter(iterable) 178 for _ in range(n): 179 try: 180 value = next(it) 181 yield value 182 except StopIteration: 183 break 184 185def takeWhile[D](iterable: Iterable[D], pred: Callable[[D], bool]) -> Iterator[D]: 186 """Yield values from `iterable` while predicate is true. 187 188 **Warning**, potential value loss if iterable is iterator with multiple 189 references. 190 """ 191 it = iter(iterable) 192 while True: 193 try: 194 value = next(it) 195 if pred(value): 196 yield value 197 else: 198 break 199 except StopIteration: 200 break 201 202## reducing and accumulating 203 204def accumulate[D,L](iterable: Iterable[D], f: Callable[[L, D], L], 205 initial: L|None=None) -> Iterator[L]: 206 """Returns an iterator of accumulated values. 207 208 * pure Python version of standard library's `itertools.accumulate` 209 * function `f` does not default to addition (for typing flexibility) 210 * begins accumulation with an optional `initial` value 211 * `itertools.accumulate` had mypy issues 212 213 """ 214 it = iter(iterable) 215 try: 216 it0 = next(it) 217 except StopIteration: 218 if initial is None: 219 return 220 else: 221 yield initial 222 else: 223 if initial is not None: 224 yield initial 225 acc = f(initial, it0) 226 for ii in it: 227 yield acc 228 acc = f(acc, ii) 229 yield acc 230 else: 231 acc = cast(L, it0) # in this case L = D 232 for ii in it: 233 yield acc 234 acc = f(acc, ii) 235 yield acc 236 237def foldL[D,L](iterable: Iterable[D], 238 f: Callable[[L, D], L], 239 initial: L|None=None) -> MB[L]: 240 """Folds an iterable left with optional initial value. 241 242 * traditional FP type order given for function `f` 243 * when an initial value is not given then `~L = ~D` 244 * if iterable empty and no `initial` value given, return empty `MB()` 245 * never returns if `iterable` generates an infinite iterator 246 247 """ 248 acc: L 249 it = iter(iterable) 250 251 if initial is None: 252 try: 253 acc = cast(L, next(it)) # in this case L = D 254 except StopIteration: 255 return MB() 256 else: 257 acc = initial 258 259 for v in it: 260 acc = f(acc, v) 261 262 return MB(acc) 263 264def foldR[D,R](iterable: Reversible[D], 265 f: Callable[[D, R], R], 266 initial: R|None=None) -> MB[R]: 267 """Folds a reversible `iterable` right with an optional `initial` value. 268 269 * `iterable` needs to be reversible 270 * traditional FP type order given for function `f` 271 * when initial value is not given then `~R = ~D` 272 * if iterable empty and no `initial` value given, return an empty `MB()` 273 274 """ 275 acc: R 276 it = reversed(iterable) 277 278 if initial is None: 279 try: 280 acc = cast(R, next(it)) # in this case R = D 281 except StopIteration: 282 return MB() 283 else: 284 acc = initial 285 286 for v in it: 287 acc = f(v, acc) 288 289 return MB(acc) 290 291def foldLsc[D,L,S](iterable: Iterable[D], 292 f: Callable[[L, D], L], 293 initial: L|None=None, 294 stopfold: Callable[[D, S], MB[S]]=lambda d, s: MB(s), 295 istate: S|None=None) -> MB[L]: 296 """Short circuit version of foldL. 297 298 * Callable `stopfold` purpose is to prematurely stop the fold before end 299 * useful for infinite iterables 300 301 """ 302 state = cast(MB[S], MB(istate)) 303 304 it = iter(iterable) 305 306 if initial is None: 307 try: 308 acc = cast(L, next(it)) # in this case L = D 309 except StopIteration: 310 return MB() 311 else: 312 acc = initial 313 314 for d in it: 315 if (state := stopfold(d, state.get())): 316 acc = f(acc, d) 317 else: 318 break 319 320 return MB(acc) 321 322def foldRsc[D,R,S](iterable: Iterable[D], 323 f: Callable[[D, R], R], 324 initial: R|None=None, 325 startfold: Callable[[D, S], MB[S]]=lambda d, s: MB(s), 326 istate: S|None=None) -> MB[R]: 327 """Short circuit version of foldR. 328 329 * Callable `startfold` purpose is to start fold before reaching the end 330 * does NOT start fold at end and prematurely stop 331 * useful for infinite and non-reversible iterables 332 333 """ 334 state = cast(MB[S], MB(istate)) 335 336 it = iter(iterable) 337 338 acc: R 339 340 ds: list[D] = [] 341 for d in it: 342 if (state := startfold(d, state.get())): 343 ds.append(d) 344 else: 345 break 346 347 if initial is None: 348 if len(ds) == 0: 349 return MB() 350 else: 351 acc = cast(R, ds.pop()) # in this case R = D 352 else: 353 acc = initial 354 355 while ds: 356 acc = f(ds.pop(), acc) 357 358 return MB(acc) 359 360## Iterator related utility functions 361# TODO: move to fp.experimental.args 362 363def itargs[A](*args: A) -> Iterator[A]: 364 """Function returning an iterators of its arguments. 365 366 Useful when an API only provides a constructor taking a single iterable. 367 """ 368 for arg in args: 369 yield arg
77def concat[D](*iterables: Iterable[D]) -> Iterator[D]: 78 """Sequentially concatenate multiple iterables together. 79 80 * pure Python version of standard library's `itertools.chain` 81 * iterator sequentially yields each iterable until all are exhausted 82 * an infinite iterable will prevent subsequent iterables from yielding any values 83 * performant to `itertools.chain` 84 85 """ 86 for iterator in map(lambda x: iter(x), iterables): 87 while True: 88 try: 89 value = next(iterator) 90 yield value 91 except StopIteration: 92 break
Sequentially concatenate multiple iterables together.
- pure Python version of standard library's
itertools.chain
- iterator sequentially yields each iterable until all are exhausted
- an infinite iterable will prevent subsequent iterables from yielding any values
- performant to
itertools.chain
121def merge[D](*iterables: Iterable[D], yield_partials: bool=False) -> Iterator[D]: 122 """Shuffle together the `iterables` until one is exhausted. 123 124 * iterator yields until one of the iterables is exhausted 125 * if `yield_partials` is true, 126 * yield any unmatched yielded values from other iterables 127 * prevents data lose 128 * if any of the iterables are iterators with external references 129 130 """ 131 iterList = list(map(lambda x: iter(x), iterables)) 132 values = [] 133 if (numIters := len(iterList)) > 0: 134 while True: 135 try: 136 for ii in range(numIters): 137 values.append(next(iterList[ii])) 138 for value in values: 139 yield value 140 values.clear() 141 except StopIteration: 142 break 143 if yield_partials: 144 for value in values: 145 yield value
Shuffle together the iterables
until one is exhausted.
- iterator yields until one of the iterables is exhausted
- if
yield_partials
is true,- yield any unmatched yielded values from other iterables
- prevents data lose
- if any of the iterables are iterators with external references
94def exhaust[D](*iterables: Iterable[D]) -> Iterator[D]: 95 """Shuffle together multiple iterables until all are exhausted. 96 97 * iterator yields until all iterables are exhausted 98 99 """ 100 iterList = list(map(lambda x: iter(x), iterables)) 101 if (numIters := len(iterList)) > 0: 102 ii = 0 103 values = [] 104 while True: 105 try: 106 while ii < numIters: 107 values.append(next(iterList[ii])) 108 ii += 1 109 for value in values: 110 yield value 111 ii = 0 112 values.clear() 113 except StopIteration: 114 numIters -= 1 115 if numIters < 1: 116 break 117 del iterList[ii] 118 for value in values: 119 yield value
Shuffle together multiple iterables until all are exhausted.
- iterator yields until all iterables are exhausted
149def drop[D](iterable: Iterable[D], n: int) -> Iterator[D]: 150 """Drop the next `n` values from `iterable`.""" 151 it = iter(iterable) 152 for _ in range(n): 153 try: 154 value = next(it) 155 except StopIteration: 156 break 157 return it
Drop the next n
values from iterable
.
159def dropWhile[D](iterable: Iterable[D], pred: Callable[[D], bool]) -> Iterator[D]: 160 """Drop initial values from `iterable` while predicate is true.""" 161 it = iter(iterable) 162 try: 163 value = next(it) 164 except: 165 return it 166 167 while True: 168 try: 169 if not pred(value): 170 break 171 value = next(it) 172 except StopIteration: 173 break 174 return concat((value,), it)
Drop initial values from iterable
while predicate is true.
176def take[D](iterable: Iterable[D], n: int) -> Iterator[D]: 177 """Take up to `n` values from `iterable`.""" 178 it = iter(iterable) 179 for _ in range(n): 180 try: 181 value = next(it) 182 yield value 183 except StopIteration: 184 break
Take up to n
values from iterable
.
186def takeWhile[D](iterable: Iterable[D], pred: Callable[[D], bool]) -> Iterator[D]: 187 """Yield values from `iterable` while predicate is true. 188 189 **Warning**, potential value loss if iterable is iterator with multiple 190 references. 191 """ 192 it = iter(iterable) 193 while True: 194 try: 195 value = next(it) 196 if pred(value): 197 yield value 198 else: 199 break 200 except StopIteration: 201 break
Yield values from iterable
while predicate is true.
Warning, potential value loss if iterable is iterator with multiple references.
205def accumulate[D,L](iterable: Iterable[D], f: Callable[[L, D], L], 206 initial: L|None=None) -> Iterator[L]: 207 """Returns an iterator of accumulated values. 208 209 * pure Python version of standard library's `itertools.accumulate` 210 * function `f` does not default to addition (for typing flexibility) 211 * begins accumulation with an optional `initial` value 212 * `itertools.accumulate` had mypy issues 213 214 """ 215 it = iter(iterable) 216 try: 217 it0 = next(it) 218 except StopIteration: 219 if initial is None: 220 return 221 else: 222 yield initial 223 else: 224 if initial is not None: 225 yield initial 226 acc = f(initial, it0) 227 for ii in it: 228 yield acc 229 acc = f(acc, ii) 230 yield acc 231 else: 232 acc = cast(L, it0) # in this case L = D 233 for ii in it: 234 yield acc 235 acc = f(acc, ii) 236 yield acc
Returns an iterator of accumulated values.
- pure Python version of standard library's
itertools.accumulate
- function
f
does not default to addition (for typing flexibility) - begins accumulation with an optional
initial
value itertools.accumulate
had mypy issues
238def foldL[D,L](iterable: Iterable[D], 239 f: Callable[[L, D], L], 240 initial: L|None=None) -> MB[L]: 241 """Folds an iterable left with optional initial value. 242 243 * traditional FP type order given for function `f` 244 * when an initial value is not given then `~L = ~D` 245 * if iterable empty and no `initial` value given, return empty `MB()` 246 * never returns if `iterable` generates an infinite iterator 247 248 """ 249 acc: L 250 it = iter(iterable) 251 252 if initial is None: 253 try: 254 acc = cast(L, next(it)) # in this case L = D 255 except StopIteration: 256 return MB() 257 else: 258 acc = initial 259 260 for v in it: 261 acc = f(acc, v) 262 263 return MB(acc)
Folds an iterable left with optional initial value.
- traditional FP type order given for function
f
- when an initial value is not given then
~L = ~D
- if iterable empty and no
initial
value given, return emptyMB()
- never returns if
iterable
generates an infinite iterator
265def foldR[D,R](iterable: Reversible[D], 266 f: Callable[[D, R], R], 267 initial: R|None=None) -> MB[R]: 268 """Folds a reversible `iterable` right with an optional `initial` value. 269 270 * `iterable` needs to be reversible 271 * traditional FP type order given for function `f` 272 * when initial value is not given then `~R = ~D` 273 * if iterable empty and no `initial` value given, return an empty `MB()` 274 275 """ 276 acc: R 277 it = reversed(iterable) 278 279 if initial is None: 280 try: 281 acc = cast(R, next(it)) # in this case R = D 282 except StopIteration: 283 return MB() 284 else: 285 acc = initial 286 287 for v in it: 288 acc = f(v, acc) 289 290 return MB(acc)
Folds a reversible iterable
right with an optional initial
value.
iterable
needs to be reversible- traditional FP type order given for function
f
- when initial value is not given then
~R = ~D
- if iterable empty and no
initial
value given, return an emptyMB()
292def foldLsc[D,L,S](iterable: Iterable[D], 293 f: Callable[[L, D], L], 294 initial: L|None=None, 295 stopfold: Callable[[D, S], MB[S]]=lambda d, s: MB(s), 296 istate: S|None=None) -> MB[L]: 297 """Short circuit version of foldL. 298 299 * Callable `stopfold` purpose is to prematurely stop the fold before end 300 * useful for infinite iterables 301 302 """ 303 state = cast(MB[S], MB(istate)) 304 305 it = iter(iterable) 306 307 if initial is None: 308 try: 309 acc = cast(L, next(it)) # in this case L = D 310 except StopIteration: 311 return MB() 312 else: 313 acc = initial 314 315 for d in it: 316 if (state := stopfold(d, state.get())): 317 acc = f(acc, d) 318 else: 319 break 320 321 return MB(acc)
Short circuit version of foldL.
- Callable
stopfold
purpose is to prematurely stop the fold before end- useful for infinite iterables
323def foldRsc[D,R,S](iterable: Iterable[D], 324 f: Callable[[D, R], R], 325 initial: R|None=None, 326 startfold: Callable[[D, S], MB[S]]=lambda d, s: MB(s), 327 istate: S|None=None) -> MB[R]: 328 """Short circuit version of foldR. 329 330 * Callable `startfold` purpose is to start fold before reaching the end 331 * does NOT start fold at end and prematurely stop 332 * useful for infinite and non-reversible iterables 333 334 """ 335 state = cast(MB[S], MB(istate)) 336 337 it = iter(iterable) 338 339 acc: R 340 341 ds: list[D] = [] 342 for d in it: 343 if (state := startfold(d, state.get())): 344 ds.append(d) 345 else: 346 break 347 348 if initial is None: 349 if len(ds) == 0: 350 return MB() 351 else: 352 acc = cast(R, ds.pop()) # in this case R = D 353 else: 354 acc = initial 355 356 while ds: 357 acc = f(ds.pop(), acc) 358 359 return MB(acc)
Short circuit version of foldR.
- Callable
startfold
purpose is to start fold before reaching the end- does NOT start fold at end and prematurely stop
- useful for infinite and non-reversible iterables
364def itargs[A](*args: A) -> Iterator[A]: 365 """Function returning an iterators of its arguments. 366 367 Useful when an API only provides a constructor taking a single iterable. 368 """ 369 for arg in args: 370 yield arg
Function returning an iterators of its arguments.
Useful when an API only provides a constructor taking a single iterable.