Class and Function Documentation

Send emails for each item in the input iterable using SMTP.

This function processes a list of items and sends an email for each one, using the specified
fields for subject and body content. It supports both HTML and plain text email formats.

Args:
    subject_field (str): Field name in the item to use as email subject
    body_fields (list[str]): List of field names to include in email body
    sender_email (str, optional): Sender's email address. If None, uses config value
    recipient_email (str, optional): Recipient's email address. If None, uses config value
    smtp_server (str, optional): SMTP server address. Defaults to 'smtp.gmail.com'
    port (int, optional): SMTP server port. Defaults to 587

Yields:
    item: Returns each processed item after sending its corresponding email

Raises:
    AssertionError: If subject_field or body_fields are None
    ValueError: If required fields are missing in items

Example:
    >>> items = [{'title': 'Hello', 'content': 'World'}]
    >>> for item in sendEmail(items, 'title', ['content'], 'sender@email.com', 'recipient@email.com'):
    ...     print(f"Processed {item}")

Notes:
    - Requires valid SMTP credentials in config
    - Supports HTML formatting in email body
    - Uses TLS encryption for email transmission

Read and extract text from Microsoft Word (.docx) files.

This function takes an iterable of file paths to .docx documents and yields the
extracted text content from each document, with paragraphs joined by spaces.

Yields:
    str: The full text content of each document with paragraphs joined by spaces

Raises:
    Exception: If there are issues reading the .docx files

Example:
    >>> paths = ['doc1.docx', 'doc2.docx']
    >>> for text in readdocx(paths):
    ...     print(text)

Reads text files from given file paths and yields their contents.

Args:
    file_paths (Iterable[str]): An iterable containing paths to text files to be read.

Yields:
    str: The contents of each text file.

Raises:
    FileNotFoundError: If a file path does not exist.
    IOError: If there is an error reading any of the files.

Example:
    >>> files = ['file1.txt', 'file2.txt']
    >>> for content in readtxt(files):
    ...     print(content)

Download a URL segment and return its content.

This function is a wrapper around downloadURL that specifically handles URL segments.
It attempts to download content from the specified URL with configurable error handling
and timeout settings.

Args:
    fail_on_error (bool, optional): If True, raises exceptions on download errors.
        If False, returns None on errors. Defaults to True.
    timeout (int, optional): The timeout in seconds for the download request. 
        Defaults to 10 seconds.

Returns:
    bytes|None: The downloaded content as bytes if successful, None if fail_on_error
        is False and an error occurs.

Raises:
    Various exceptions from downloadURL function when fail_on_error is True and
    an error occurs during download.

Converts HTML content to text segment.

This function takes HTML content and converts it to plain text format.
If cleanText is enabled, the resulting text will also be cleaned so it 
tries to retain only the main body content.

Args:
    raw (str): The raw HTML content to be converted
    cleanText (bool, optional): Whether to clean and normalize the output text. Defaults to True.

Returns:
    str: The extracted text content from the HTML

See Also:
    htmlToText: The underlying function used for HTML to text conversion

Generator function that monitors and yields new entries from an RSS feed.

This function continuously monitors an RSS feed at the specified URL and yields new entries
as they become available. It uses a SQLite database to keep track of previously seen entries
to avoid duplicates.

Args:
    url (str): The URL of the RSS feed to monitor.  If None, the URL is read from the config using
        the key "RSS_URL"
    db_path (str, optional): Path to the SQLite database file for storing entry history.
        Defaults to ':memory:' for an in-memory database.
    poll_interval_minutes (int, optional): Number of minutes to wait between polling
        the RSS feed for updates. Defaults to 10 minutes.

Yields:
    dict: New entries from the RSS feed, containing feed item data.

Example:
    >>> for entry in rss_source("http://example.com/feed.xml"):
    ...     print(entry["title"])

Filter items using a Bloom Filter to yield only distinct elements based on specified fields.

A Bloom Filter is a space-efficient probabilistic data structure used to test whether 
an element is a member of a set. False positive matches are possible, but false 
negatives are not.

Args:
    items (iterable): Input items to filter.
    capacity (int): Expected number of items to be added to the Bloom Filter.
    error_rate (float): Acceptable false positive probability (between 0 and 1).
    field_list (str, optional): Dot-separated string of nested fields to use for 
        distinctness check. Defaults to "_" which uses the entire item.

Yields:
    item: Items that have not been seen before according to the Bloom Filter.

Example:
    >>> items = [{"id": 1, "name": "John"}, {"id": 2, "name": "John"}]
    >>> list(distinctBloomFilter(items, 1000, 0.01, "name"))
    [{'id': 1, 'name': 'John'}]  # Only first item with name "John" is yielded

Note:
    Due to the probabilistic nature of Bloom Filters, there is a small chance
    of false positives (items incorrectly identified as duplicates) based on
    the specified error_rate.

Links the input stream to a threaded queue system.

This segment takes an input stream and links it to a threaded queue system.
It starts the queue system and then starts yielding from the queue.  That way
the upstream units don't have to wait for the downstream segments to draw 
from them.

Fills null (None) values in a sequence of dictionaries with specified defaults.

This generator function processes dictionaries by replacing None values with either
a general default value or specific values for named fields.

Args:
    items: An iterable of dictionaries to process.
    default (str, optional): The default value to use for any None values not 
        specified in kwargs. Defaults to ''.
    **kwargs: Field-specific default values. Each keyword argument specifies a
        field name and the default value to use for that field.

Yields:
    dict: The processed dictionary with None values replaced by defaults.

Raises:
    AssertionError: If any item in the input is not a dictionary.
    TypeError: If any item doesn't support item assignment using square brackets.

Examples:
    >>> data = [{'a': None, 'b': 1}, {'a': 2, 'b': None}]
    >>> list(fill_null(data, default='N/A'))
    [{'a': 'N/A', 'b': 1}, {'a': 2, 'b': 'N/A'}]
    
    >>> list(fill_null(data, b='EMPTY'))
    [{'a': None, 'b': 1}, {'a': 2, 'b': 'EMPTY'}]

Transform items by applying regex pattern replacement.

This segment transforms items by applying a regex pattern replacement to either
the entire item (if field="_") or a specific field of the item.

Args:
    items (Iterable): Input items to transform.
    pattern (str): Regular expression pattern to match.
    replacement (str): Replacement string for matched patterns.
    field (str, optional): Field to apply transformation to. Use "_" for entire item. Defaults to "_".

Yields:
    Union[str, dict]: Transformed items. Returns string if field="_", otherwise returns modified item dict.

Raises:
    TypeError: If extracted value is not a string or if item is not subscriptable when field != "_".

Examples:
    >>> list(regex_replace(["hello world"], r"world", "everyone"))
    ['hello everyone']
    
    >>> list(regex_replace([{"text": "hello world"}], r"world", "everyone", field="text"))
    [{'text': 'hello everyone'}]

Appends the specified fields to the input item.

Equivalent to toDict except that that item is modified with the new key/value pairs 
rather than a new dictionary returned.

Assumes that the input item can has items assigned using bracket notation ([]).

Call a function on each item in the input stream.

Args:
    func (callable): The function to call on each item

Concatenates specified fields from each item with a delimiter.

    Args:
        items: Iterable of input items to process
        fields: String specifying fields to extract and concatenate
        delimiter (str, optional): String to insert between concatenated fields. Defaults to "

"
        append_as (str, optional): If specified, adds concatenated result as new field with this name. 
                                Defaults to None.

    Yields:
        If append_as is specified, yields the original item with concatenated result added as new field.
        Otherwise, yields just the concatenated string.
    

Yields every nth item from the input stream.

Args:
    items: Iterable of items to process
    n: Number of items to skip between each yield

Yields:
    Every nth item from the input stream.

Execute a command and yields each line passed to stdout as an item.

Fill a template string with values from the input item.

Args:
    item: The input item containing values to fill the template
    template (str): The template string with placeholders for values

Returns:
    str: The filled template string

Flattens a nested list of items.

Args:
    items: Iterable of items to flatten

Yields:
    Flattened list of items

Filters items based on whether a field contains a specified value.

Args:
    items: Iterable of items to filter
    field: Field name to check for value
    value: Value to check for in the field

Yields:
    Items where the specified field contains the specified value.

Filters items based on whether a field does not contain a specified value.

Args:
    field: Field name to check for value
    value: Value to check for in the field

Yields:
    Items where the specified field does not contain the specified value.

Slices a sequence using start and end indices.

This function takes a sequence and a range string in the format "start:end" to slice the sequence.
Both start and end indices are optional.

Args:
    item: Any sequence that supports slicing (e.g., list, string, tuple)
    range (str, optional): String in format "start:end" where both start and end are optional.
        For example: "2:5", ":3", "4:", ":" are all valid. Defaults to None.

Returns:
    The sliced sequence containing elements from start to end index.
    If range is None, returns a full copy of the sequence.

Examples:
    >>> slice([1,2,3,4,5], "1:3")
    [2, 3]
    >>> slice("hello", ":3")
    "hel"
    >>> slice([1,2,3,4,5], "2:")
    [3, 4, 5]

Drains the input stream and dumps each item as a jsonl string.
    

A source that generates input from a string.

This source will generate input from a string, splitting it on a delimiter.

Reads each item from the input stream, interpreting it as a jsonl string. 
    
    

Reads each item from the input stream as a path to a jsonl file. Loads each line of
each file as a json object and yields each individually.

Drains the input stream into a list and then writes the list as a pickle file.

Args:
    fname (str): The name of the file to write.
    first_only (bool): If True, the segment will write only the first item in the input stream,
        throwing an exception if there is more than one.
        If False, the segment will write the entire input stream.

Generate a range of integers between lower and upper

Filter items where a specified field's value is greater than a number.

Takes an iterable of items and yields only those where the value of the specified field
is greater than the given number n.

Args:
    items: Iterable of items to filter
    field: String representing the field/property to compare
    n: Number to compare against

Yields:
    Items where the specified field's value is greater than n

Raises:
    AttributeError: If the specified field is missing from any item

Example:
    >>> list(gt([{'value': 5}, {'value': 2}, {'value': 8}], 'value', 4))
    [{'value': 5}, {'value': 8}]

Filters items based on a field value being less than a specified number.

Yields items where the specified field value is less than the given number n.

Args:
    items (iterable): An iterable of items to filter
    field (str): The field/property name to check against
    n (numeric): The number to compare against

Yields:
    item: Items where the specified field value is less than n

Raises:
    AttributeError: If the specified field does not exist on an item (due to fail_on_missing=True)

Example:
    >>> items = [{'val': 1}, {'val': 5}, {'val': 3}]
    >>> list(lt(items, 'val', 4))
    [{'val': 1}, {'val': 3}]

Generate n random integers between lower and upper.

Scale each item in the input stream by the multiplier.