Here you can find exact details regarding app’s internals.

Main Interface

This is the entry point ot the application.

main.run(*, domain: str, reports_path: str, reports_list: list[dict[str, str]] = [], summary_filepath: str = '', log_path: str = '', report: str = '', path: str = '', threads: int = 0, stdout_loglevel: str = 'WARNING', file_loglevel: str = 'WARNING', verbose: bool = False) → None

Main function of the program.

Parameters:

• domain (str) – SalesForce domain of your organization -> “https://corp.my.salesforce.com/”

• reports_path (str) – Path to reports.csv file, template -> Template

• reports_list (list[dict[str, str]]) – List of the reports as dictionaries -> [{'name': 'RaportName', 'id': '00O1V00000999GHES', 'path': WindowsPath('C:/downloads')}]

• summary_filepath (str) – File path to summary report -> C:/downloads/summary.csv

• log_path (str) – Path to log file -> C:/downloads/logs/

• report (str) – Single report mode -> RaportName,00O1V00000999GHES,C:/downloads

• path (str) – Save location path override -> C:/new_downloads

• threads (int) – Number of threads to use. (Default: half of available threads of the machine)

• stdout_loglevel (str) – Log level for stdout logging -> ['critical'|'error'|'warn'|'warning'|'info'|'debug'] (Default: WARNING)

• file_loglevel (str) – Log level for file logging -> ['critical'|'error'|'warn'|'warning'|'info'|'debug'] (Default: WARNING)

• verbose (bool) – Toggles between Progress Bar and stdout logging (Default: False)

Usage:

import sfrout
sfrout.run(domain="https://corp.my.salesforce.com/", reports_path="C:/path/to/reports.csv")

cli

SFrout is a scalable, asynchronous SalesForce report downloader for SAML/SSO clients. The app allows you to download reports based on their ID using your personal SFDC account. Supports asynchronous requests, threaded processing of the files, logging to rotating file and stdout, produces summary report for the session.

Usage:

$ sfrout "https://corp.my.salesforce.com/" "C:\path\to\reports.csv"

cli [OPTIONS] DOMAIN [REPORTS_PATH]

Options

-s, --summary_filepath <summary_filepath>

Path to the summary report -> c:/summary_report.csv

-l, --log_path <log_path>

Path to the log file -> c:/log

-r, --report <report>

Run single report -> “name,id,path,optional_report_params”

-p, --path <path>

Override save location of the reports

-t, --threads <threads>

Number of threads to spawn

Default:

0

-ls, --stdout_loglevel <stdout_loglevel>

STDOUT logging level -> [DEBUG | INFO | WARN |WARNING | ERROR | CRITICAL]

Default:

WARNING

-lf, --file_loglevel <file_loglevel>

File logging level -> [DEBUG | INFO | WARN| WARNING | ERROR | CRITICAL]

Default:

INFO

-v, --verbose

Turn off progress bar

Default:

False

Arguments

DOMAIN

Required argument

REPORTS_PATH

Optional argument

Components

Config

class components.config.ConfigProtocol(*args, **kwargs)

Protocol class for config object.

Parameters:

• reports_list_path (str) – CLI argument for input report list path.

• report (str) – CLI argument for single report params.

• path (str) – CLI argument for save location path override.

• threads (int) – CLI argument for number of threads to use.

class components.config.Config(*, domain: str, reports_csv_path: str, reports_list: list[dict[str, Any]] = [], summary_filepath: str | None = None, log_path: str | None = None, report: str = '', path: str = '', threads: int = 0, stdout_loglevel: str = 'WARNNING', file_loglevel: str = 'INFO', verbose=False)

Concrete class representing Config object. Contains entire configuration required for a program.

Parameters:

• reports_list_path (str) – CLI argument for input report list path.

• report (str) – CLI argument for single report params.

• path (str) – CLI argument for save location path override.

• threads (int) – CLI argument for number of threads to use.

_define_number_of_threads()

Defines number of threads. By default number of threads is set to half of available threads. If threads value is not available number of threds will be set to 2. If threads number has been defined in CLI configuration threads will be equal to this number. If CLI report is filled (single report mode) then number of threads will be automatically set to 1

_input_report_path_cast(object_kwargs: list[dict[str, Any]]) → list[dict[str, str | os.PathLike]]

Casts value of path key into Path object.

Parameters:

object_kwargs (list[dict[str, Any]]) – Colection of object parameters

Returns:

Collection of object parameters with path casted to Path object

Return type:

list[dict[str, str | PathLike]]

_input_report_single_mode_override() → list[dict[str, str]]

Reads parameters taken from CLI. Returns parsed parameters into object kwargs.

Returns:

collection of single object kwargs (parameters) based on CLI argument

Return type:

list[dict[str, str]]

_input_report_csv_standard_file_mode() → list[dict[str, str]]

Reads parameteres taken from input CSV. Returns parsed parameters into objects kwargs.

Returns:

collection of objects kwargs (parameters) based on input CSV

Return type:

list[dict[str, str]]

_input_report_path_override(object_kwargs: list[dict[str, str]]) → list[dict[str, str]]

Replaces value of path kwarg parameter of the object with path value from CLI argument.

Parameters:

object_kwargs (list[dict[str, str]]) – Colection of object parameters

Returns:

Collection of object parameters with path replaced with value of path CLI argument

Return type:

list[dict[str, str]]

_parse_input_report() → list[dict[str, Any]]

Orchestrating function for parsing parameters for input reports.

Returns:

Collection of ready to use object kwargs.

Return type:

list[dict[str, Any]]

Connectors

class components.connectors.ConnectorProtocol(*args, **kwargs)

Protocol class for connector object.

Parameters:

• queue (Queue) – Shared queue object.

• timeout (int) – Request’s timeout value in seconds.

• headers (dict[str, str]) – Headers required to establish the connection.

check_connection() → bool

Checks connection with given domain.

Returns:

Flag, True if connection is established, False otherwise.

Return type:

bool

async report_gathering(reports: list[components.containers.ReportProtocol], session: ClientSession) → None

Collects asynchronous responses from the servers.

Parameters:

• reports (list[ReportProt]) – Collection of ReportProtocol objects.

• session (aiohttp.ClientSession) – HTTP client session object to handle request in transaction.

class components.connectors.SfdcConnector(queue: Queue, *, domain: str, verbose: bool = False, timeout: int = 900, headers: dict[str, str] = {'Content-Type': 'application/csv', 'X-PrettyPrint': '1'})

Concrete class representing Connector object for SFDC

Parameters:

• queue (Queue) – Shared queue object.

• verbose – CLI parameter used as switch between progress bar and logging to stdout on INFO level. Defaults to False.

• timeout (int) – Request’s timeout value in seconds. Defaults to 900.

• headers (dict[str, str]) – Headers required to establish the connection. Defaults to {‘Content-Type’: ‘application/csv’, ‘X-PrettyPrint’: ‘1’}.

• export_params (str) – Default parameters required by SFDC. Defaults to ‘?export=csv&enc=UTF-8&isdtp=p1’.

_convert_domain_for_cookies_lookup() → str

Converts domain as key in cookier for sid lookup.

Returns:

Converted url complaiant with cookies keys.

Return type:

str

_parse_headers() → None

Parses headers for request.

_intercept_sid() → str

Intercepts sid from MS Edge’s CookieJar.

Returns:

Intercepted sid or empty string if sid doesn’t exist.

Return type:

str

_open_sfdc_site() → None

Opens SFDC website on given domain url if sid is not present or not valid.

_sid_check() → bool

Checks SID valididty for given SFDC domain.

Returns:

Flag, True if SID was valid, False when wasn’t.

Return type:

bool

check_connection() → bool

Checks the connection with given domain.

Returns:

Flag, True if connection was successful, False wasn’t.

Return type:

bool

_parse_report_url(report: ReportProtocol) → str

Parses report object url.

Parameters:

report (ReportProtocol) – Instance of ReportProtocol.

Returns:

Parsed url.

Return type:

str

async _request_report(report: ReportProtocol, session: ClientSession) → None

Sends asynchronous request to given domain with given parameters within shared session. Checks response status: - 200: response is saved in ReportProtocol.response, ReportProtocol.valid set to True, ReportProtocol is being put to the queue. - 404: error in response, ReportProtocol.valid set to False, no retries. - 500: request timeour, ReportProtocol.valid set to False, another attempt. - *: unknown error, ReportProtocol.valid set to False, another attempt.

Parameters:

• report (ReportProtocol) – Instance of ReportProtocol.

• session (aiohttp.ClientSession) – Shared session object.

async _toggle_progress_bar(tasks: list[_asyncio.Task]) → None

Toggles between showing progress bar and logging on INFO level.

Parameters:

tasks (list[asyncio.Task]) – Collection of asynchronous request tasks.

_create_async_tasks(reports: list[components.containers.ReportProtocol], session: ClientSession) → list[_asyncio.Task]

Creates collection of asynchronous request tasks.

Parameters:

• reports (list[ReportProtocol]) – Collection of ReportsProtocol instances.

• session (aiohttp.ClientSession) – Shared, asynchronous session.

Returns:

Collection of asynchronous request tasks.

Return type:

list[asyncio.Task]

async _report_request_all(reports: list[components.containers.ReportProtocol], session: ClientSession) → None

Orchestrates entire process of processing tasks.

Parameters:

• reports (list[ReportProtocol]) – Collection of ReportProtocol instances.

• session (aiohttp.ClientSession) – Shared asyncio session.

async handle_requests(reports: list[components.containers.ReportProtocol]) → None

Creates session and process asynchronous tasks.

Parameters:

reports (list[ReportProtocol]) – Collection of ReportProtocol instances.

Containers

class components.containers.ReportProtocol(*args, **kwargs)

Protocol class for report object.

Parameters:

• name (str) – Report name, propagated to report file name

• id (str) – Report id, identification number of the report in SFDC

• path (PathLike) – Report path, save location for the report in form of Path object

• type (str) – Report type, allowed options [‘SFDC’], type drives connector and report objects selection

• export_params (str) – Default parameters required by SFDC. Defaults to ‘?export=csv&enc=UTF-8&isdtp=p1’.

• downloaded (bool) – Flag indicating whether the reports has been succesfully downloaded or not

• valid (bool) – Flag indicating whether the response has been succesfully retrieved or not

• created_date (datetime) – Report save completition date

• pull_date (timedelta) – Report response completition date

• processing_time – The time it took to process the report in seconds

• attempt_count (int) – Number of attempts to process the report

• size (float) – Size of saved report file in Mb

• response (str) – Container for request response

• content (DataFrame) – Pandas DataFrame based on response

class components.containers.ReportsContainerProtocol(*args, **kwargs)

Protocol class for report container object.

Parameters:

• report_params_list (list[dict[str, Any]]) – Collection of dicts with parameters for object crafting.

• summary_report_path (PathLike) – Path to save location of summary report.

create_reports() → list[components.containers.ReportProtocol]

Orchestrating method to handle report objects factory

Returns:

Collection of Reports

Return type:

list[ReportProtocol]

create_summary_report() → None

Creates summary report which consist of all important details regarding Report objects. Summary report is generated once all the reports are completed.

class components.containers.SfdcReport(name: str, id: str, path: ~os.PathLike, type: str = 'SFDC', export_params: str = '?export=csv&enc=UTF-8&isdtp=p1', downloaded: bool = False, valid: bool = False, created_date: ~datetime.datetime = datetime.datetime(2023, 4, 5, 8, 48, 11, 310604), pull_date: ~datetime.datetime = datetime.datetime(2023, 4, 5, 8, 48, 11, 310604), processing_time: ~datetime.timedelta = datetime.timedelta(0), attempt_count: int = 0, size: float = 0.0, response: str = '', content: ~pandas.core.frame.DataFrame = <factory>)

Concrete class representing Report object from SFDC.

Parameters:

• name (str) – Report name, propagated to report file name.

• id (str) – Report id, identification number of the report in SFDC.

• path (PathLike) – Report path, save location for the report in form of Path object.

• type (str) – Report type, allowed options [‘SFDC’], type drives connector and report objects selection. Defaults to ‘SFDC’.

• export_params (str) – Default parameters required by SFDC. Defaults to ‘?export=csv&enc=UTF-8&isdtp=p1’.

• downloaded (bool) – Flag indicating whether the reports has been succesfully downloaded or not. Defaults to False.

• valid (bool) – Flag indicating whether the response has been succesfully retrieved or not. Defaults to False.

• created_date (datetime) – Report save completition date. Defaults to current datetime.

• pull_date (timedelta) – Report response completition date. Defaults to current datetime.

• processing_time – The time it took to process the report in seconds. Defaults to 0 microseconds.

• attempt_count (int) – Number of attempts to process the report. Defaults to 0 .

• size (float) – Size of saved report file in Mb. Defaults to 0.0 .

• response (str) – Container for request response. Defaults to empty string.

• content (DataFrame) – Pandas DataFrame based on response. Defaults to empty Pandas DataFrame.

class components.containers.ReportsContainer(reports_params_list: list[dict[str, Any]], summary_path: PathLike | None)

Concrete class representing ReportContainer object.

_create_sfdc_reports() → Generator[SfdcReport, None, None]

SFDC Report objects factory

Returns:

Generator with SFDC Reeport objects

Return type:

Generator[SfdcReport, None, None]

Yield:

SFDC Report instance based on parsed report parameters

Return type:

SfdcReport

create_reports() → list[components.containers.ReportProtocol]

Orchestrating method to handle report objects factory

Returns:

Collection of Reports

Return type:

list[ReportProtocol]

create_summary_report() → None

Creates summary report which consist of all important details regarding reports. Report is generated once all the reports are completed.

print_summary_table() → None

Prints summary report which consist of all important details regarding reports. Report is generated once all the reports are completed.

Handlers

class components.handlers.WorkerFactoryProtocol(*args, **kwargs)

Protocol class for worker factory objects.

Parameters:

• queue (Queue) – Shared, thread-safe queue.

• threads (int) – Number of threads, equal to number of Workers to be deployed.

create_workers() → None

Creates workers on independent threads

static active_workers() → int

Counts active works in current time.

Returns:

Number of active workers.

Return type:

int

class components.handlers.WorkerProtocol(*args, **kwargs)

Protocol class for worker factory objects.

Parameters:

queue (Queue) – Shared, thread-safe queue.

_read_stream(report: ReportProtocol) → None

Reads the stream of data kept in Report object via Pandas read method. Deletes response content from the object.

Parameters:

report (ReportProtocol) – Instance of the ReportProtocol object.

_save_to_csv(report: ReportProtocol) → None

Saves readed data to CSV file using Pandas save method.

Parameters:

report (ReportProtocol) – Instance of the ReportProtocol object.

_erase_report(report: ReportProtocol) → None

Erases the report data.

Parameters:

report (ReportProtocol) – Instance of the ReportProtocol object.

report_processing(report: ReportProtocol) → None

Orchiestrates the report processing.

Parameters:

report (ReportProtocol) – Instance of the ReportProtocol object.

run() → NoReturn

Starts listner process on sepearet thread, awaits objects in the queue.

Returns:

Method never returns.

Return type:

NoReturn

class components.handlers.WorkerFactory(queue: Queue, *, threads: int = 1)

Concrete class representing WorkerFactory object.

create_workers() → None

Deploys given number of workers.

static active_workers() → int

Returns number of currently active workers.

Returns:

Number of workers.

Return type:

int

class components.handlers.Worker(queue: Queue)

Concrete class representing Worker object.

_read_stream(report: ReportProtocol) → None

Reads report’s response and save it as content atribute. Erases saved response.

Parameters:

report (ReportProtocol) – Instance of the ReportProtocol object.

_parse_save_path(report: ReportProtocol) → PathLike

Parses path to save location.

Parameters:

report (ReportProtocol) – Instance of the ReportProtocol object.

Returns:

Path to save location

Return type:

os.PathLike

_save_to_csv(report: ReportProtocol) → None

Saves report content to CSV file. Sets object flags.

Parameters:

report (ReportProtocol) – Instance of the ReportProtocol object.

_erase_report(report: ReportProtocol) → None

Deletes report content in ReportProtocol object.

Parameters:

report (ReportProtocol) – Instance of the ReportProtocol object.

process_report(report: ReportProtocol) → None

Orchiestrates entire process of downloading the report.

Parameters:

report (ReportProtocol) – Instance of the ReportProtocol object.

run() → NoReturn

begins to listen to the queue. Starts processing once will get item from the queue. Sends signal to the queue once task is done.

Returns:

Function never returns.

Return type:

NoReturn

Excpetions

exception components.exceptions.OutdatedSIDError(message: str = 'Your SID is outdate, please provide recent SID')

Exception raised for errors in the SID value.

…

message

explanation of the error

Type:

str

exception components.exceptions.EnvFileNotPresent(message: str = '.env file not present in main directory')

Exception raised if the .env file is not present in main directory.

…

message

explanation of the error

Type:

str