Here you can find exact details regarding app’s components.

Components

Config

class components.config.ConfigProtocol(*args, **kwargs)[source]

Protocol class for config object.

Parameters:
  • domain (str) – argument for SFDC domain

  • reports_csv_path (str) – argument for input report csv path.

  • reports_list (list[dict[str, Any]]) – argument for list of reports as dictionary

  • summary_filepath (str) – argument for summary file path

  • log_filepath (str) – argument for log file path

  • report (str) – argument for single report parameters

  • path (str) – argument for save location path override

  • threads (int) – argument for number of threads to use

  • stdout_loglevel (str) – argument for stdout log level

  • file_loglevel (str) – argument for stdout log level

  • verbose (bool) – Flag, toggles progress bar/stdout

class components.config.Config(*, domain: str, reports_csv_path: str, reports_list: list[dict[str, Any]] = [], summary_filepath: str | None = None, log_path: str | None = None, report: str = '', path: str = '', threads: int = 0, stdout_loglevel: str = 'WARNNING', file_loglevel: str = 'INFO', verbose=False)[source]

Concrete class representing Config object. Contains entire configuration required for a program.

Parameters:
  • domain (str) – argument for SFDC domain

  • reports_csv_path (str) – argument for input report csv path.

  • reports_list (list[dict[str, Any]], optional) – argument for list of reports as dictionary, defaults to []

  • summary_filepath (str, optional) – argument for summary file path, defaults to None

  • log_filepath (str, optional) – argument for log file path, defaults to None

  • report (str, optional) – argument for single report parameters, defaults to “”

  • path (str, optional) – argument for save location path override, defaults to “”

  • threads (int, optional) – argument for number of threads to use, defaults to 0

  • stdout_loglevel (str, optional) – argument for stdout log level, defaults to “WARNING”

  • file_loglevel (str, optional) – argument for stdout log level, defaults to “INFO”

  • verbose (bool, optional) – Flag, toggles progress bar/stdout, defaults to False

_define_number_of_threads()[source]

Defines number of threads. By default number of threads is set to half of available threads. If threads value is not available number of threds will be set to 2. If report parameter is filled (single report mode) then number of threads will be automatically set to 1.

_input_report_path_cast(object_kwargs: list[dict[str, Any]]) list[dict[str, str | os.PathLike]][source]

Casts value of path into Path object.

Parameters:

object_kwargs (list[dict[str, Any]]) – colection of report parameters

Returns:

collection of object parameters with path casted to Path object

Return type:

list[dict[str, str | os.PathLike]]

_input_report_single_mode_override() list[dict[str, str]][source]

Creates report’s parameters in single report mode. Returns parsed parameters as object kwargs.

Returns:

collection of report’s parameters in single report mode

Return type:

list[dict[str, str]]

_input_report_csv_standard_file_mode() list[dict[str, str]][source]

Reads parameteres taken from input csv . Returns parsed parameters as objects kwargs.

Returns:

collection of report’s parameters in standard mode

Return type:

list[dict[str, str]]

_input_report_path_override(object_kwargs: list[dict[str, str]]) list[dict[str, str]][source]

Replaces value of path key in object_kwargs dict with path parameter value. path override.

Parameters:

object_kwargs (list[dict[str, str]]) – Colection of report parameters

Returns:

Collection of report’s parameters with replaced path

Return type:

list[dict[str, str]]

_parse_input_report() list[dict[str, Any]][source]

Orchestrating function which parses parameters for ReportProtocol .

Returns:

Collection of ready to use report kwargs.

Return type:

list[dict[str, Any]]

Connectors

class components.connectors.ConnectorProtocol(*args, **kwargs)[source]

Protocol class for connector object.

Parameters:
  • queue (Queue) – shared queue object

  • timeout (int) – request’s timeout value in seconds

  • headers (dict[str, str]) – headers required to establish the connection

check_connection() bool[source]

Checks connection with given domain.

Returns:

flag, True if connection is established, False otherwise.

Return type:

bool

async report_gathering(reports: list[components.containers.ReportProtocol], session: ClientSession) None[source]

Collects asynchronous responses from the servers.

Parameters:
  • reports (list[ReportProtocol]) – collection of ReportProtocol objects.

  • session (ClientSession) – HTTP client session object to handle request in transaction.

class components.connectors.SfdcConnector(*, queue: Queue, domain: str, verbose: bool = False, timeout: int = 900, headers: dict[str, str] = {'Content-Type': 'application/csv', 'X-PrettyPrint': '1'})[source]

Concrete class representing SFDC Connector object.

Parameters:
  • queue (Queue) – shared queue object

  • domain (str) – SalesForce domain of your organization -> “https://corp.my.salesforce.com/”

  • verbose – flag, parameter used as switch between progress bar and logging to stdout, defaults to False

  • timeout (int, optional) – Request’s timeout value in seconds, defaults to 900

  • headers (dict[str, str], optional) – headers required connection, defaults to {‘Content-Type’: ‘application/csv’, ‘X-PrettyPrint’: ‘1’}

  • export_params (str, optional) – GET request parameters required by SFDC, defaults to ‘?export=csv&enc=UTF-8&isdtp=p1’

_convert_domain_for_cookies_lookup() str[source]

Converts domain string according to sid CookieJar lookup requirements.

Returns:

converted url complaiant with CookieJar schema

Return type:

str

_parse_headers() None[source]

Parses headers for request.

_intercept_sid() str[source]

Intercepts sid from MS Edge’s CookieJar.

Returns:

Intercepted sid or empty string if sid doesn’t exist.

Return type:

str

_open_sfdc_site() None[source]

Opens SFDC website on given domain url if sid is not present or not valid.

_sid_check() bool[source]

Checks sid valididty for given SFDC domain.

Returns:

Flag, True if sid was valid, False when wasn’t.

Return type:

bool

check_connection() bool[source]

Checks the connection with given domain and sid validity.

Returns:

flag, True if connection was successful, False if wasn’t.

Return type:

bool

_parse_report_url(report: ReportProtocol) str[source]

Parses report object url.

Parameters:

report (ReportProtocol) – instance of ReportProtocol

Returns:

parsed url

Return type:

str

async _request_report(report: ReportProtocol, session: ClientSession) None[source]
Sends asynchronous request to given domain with given parameters within shared session. Checks response status:
200: response is saved in ReportProtocol.response() , ReportProtocol.valid() set to True , ReportProtocol is being put to the queue .
404: error in response, ReportProtocol.valid() set to False , no retries.
500: request timeour, ReportProtocol.valid() set to False , another attempt.
*: unknown error, ReportProtocol.valid() set to False, another attempt.
Parameters:
  • report (ReportProtocol) – instance of ReportProtocol

  • session (aiohttp.ClientSession) – shared session object

async _toggle_progress_bar(tasks: list[_asyncio.Task]) None[source]

Toggles between showing progress bar or stdout logging.

Parameters:

tasks (list[asyncio.Task]) – collection of asynchronous request tasks

_create_async_tasks(reports: list[components.containers.ReportProtocol], session: ClientSession) list[_asyncio.Task][source]

Creates collection of asynchronous request tasks.

Parameters:
  • reports (list[ReportProtocol]) – collection of ReportProtocol instances

  • session (aiohttp.ClientSession) – shared, asynchronous session

Returns:

collection of asynchronous request tasks

Return type:

list[asyncio.Task]

async _report_request_all(reports: list[components.containers.ReportProtocol], session: ClientSession) None[source]

Orchestrates entire process of processing tasks.

Parameters:
  • reports (list[ReportProtocol]) – collection of ReportProtocol instances

  • session (aiohttp.ClientSession) – shared, asynchronous session

async handle_requests(reports: list[components.containers.ReportProtocol]) None[source]

Creates session and processes asynchronous tasks.

Parameters:

reports (list[ReportProtocol]) – collection of ReportProtocol instances

Containers

class components.containers.ReportProtocol(*args, **kwargs)[source]

Protocol class for report object.

Parameters:
  • name (str) – report name, propagated to report file name

  • id (str) – report id

  • path (os.PathLike) – report path, save location for the report in form of pathlib.Path object

  • type (str) – report type, drives connector selection

  • export_params (str) – default GET parameters

  • downloaded (bool) – flag indicating whether the reports has been succesfully downloaded or not

  • valid (bool) – flag indicating whether the response has been succesfully retrieved or not

  • created_date (datetime) – report save completition date

  • pull_date (timedelta) – report response completition date

  • processing_time – the time it took to process the report in seconds

  • attempt_count (int) – number of attempts to process the report

  • size (float) – size of saved report file in Mb

  • response (str) – container for request response

  • content (pd.DataFrame) – Pandas pd.DataFrame based on response

class components.containers.ReportsContainerProtocol(*args, **kwargs)[source]

Protocol class for report container object.

Parameters:
  • report_params_list (list[dict[str, Any]]) – collection of dictionaries with parameters for object crafting

  • summary_report_path (os.PathLike) – path to save location of summary report

create_reports() list[components.containers.ReportProtocol][source]

Orchestrating method to handle report objects factory

Returns:

collection of ReportProtocol objects

Return type:

list[ReportProtocol]

create_summary_report() None[source]

Creates summary report which consist of all important details regarding ReportProtocol objects. Summary report is generated once all the reports are completed.

class components.containers.SfdcReport(name: str, id: str, path: ~os.PathLike, type: str = 'SFDC', export_params: str = '?export=csv&enc=UTF-8&isdtp=p1', downloaded: bool = False, valid: bool = False, created_date: ~datetime.datetime = datetime.datetime(2023, 4, 7, 3, 40, 58, 22998), pull_date: ~datetime.datetime = datetime.datetime(2023, 4, 7, 3, 40, 58, 22998), processing_time: ~datetime.timedelta = datetime.timedelta(0), attempt_count: int = 0, size: float = 0.0, response: str = '', content: ~pandas.core.frame.DataFrame = <factory>)[source]

Concrete class representing SFDC Report object.

Parameters:
  • name (str) – report name, propagated to report file name

  • id (str) – report id, identification number of the report in SFDC

  • path (os.PathLike) – report path, save location for the report in form of pathlib.Path object

  • type (str, optional) – report type, type drives connector and report objects selectiond, defaults to ‘SFDC’

  • export_params (str, optional) – default GET parameters required by SFDC, defaults to ‘?export=csv&enc=UTF-8&isdtp=p1’

  • downloaded (bool, optional) – flag indicating whether the reports has been succesfully downloaded or not, defaults to False

  • valid (bool, optional) – flag indicating whether the response has been succesfully retrieved or not, defaults to False

  • created_date (datetime, optional) – report save completition date, defaults to current datetime

  • pull_date (timedelta, optional) – report response completition date, defaults to current datetime

  • processing_time – the time it took to process the report in seconds, defaults to 0 microseconds

  • attempt_count (int, optional) – number of attempts to process the report, defaults to 0

  • size (float, optional) – size of saved report file in Mb, defaults to 0.0

  • response (str, optional) – container for request response, defaults “”

  • content (pd.DataFrame , optional) – Pandas DataFrame based on response, defaults to empty pd.DataFrame

class components.containers.ReportsContainer(reports_params_list: list[dict[str, Any]], summary_path: PathLike | None)[source]

Concrete class representing ReportContainer object.

_create_sfdc_reports() Generator[SfdcReport, None, None][source]

Creates SfdcReport objects.

Returns:

generator with SfdcReport objects

Return type:

Generator[SfdcReport, None, None]

Yield:

SfdcReport instance based on parsed report parameters

Return type:

SfdcReport

create_reports() list[components.containers.ReportProtocol][source]

Orchestrating method to handle report objects crafting

Returns:

collection of ReportProtocol objects

Return type:

list[ReportProtocol]

_create_summary_folder_if_not_exist()[source]

Creates folder for summary report if doesn’t exist.

create_summary_report() None[source]

Creates summary report which consist of all important details regarding ReportProtocol objects. Summary is generated once all the ReportProtocol are completed.

print_summary_table() None[source]

Prints summary table which consist of all important details regarding ReportProtocol objects. Table is generated once all the ReportProtocol are completed.

Handlers

class components.handlers.WorkerFactoryProtocol(*args, **kwargs)[source]

Protocol class for worker factory objects.

Parameters:
  • queue (Queue) – shared, thread-safe queue object

  • threads (int) – number of threads, equal to number of WorkerProtocol to be deployed

create_workers() None[source]

Creates WorkerProtocol on independent threads

static active_workers() int[source]

Counts active WorkerProtocol in current time.

Returns:

number of active WorkerProtocol objects

Return type:

int

class components.handlers.WorkerProtocol(*args, **kwargs)[source]

Protocol class for worker objects.

Parameters:

queue (Queue) – shared, thread-safe queue object

_read_stream(report: ReportProtocol) None[source]

Reads the stream of data kept in ReportProtocol object via Pandas read method. Deletes response content from the object.

Parameters:

report (ReportProtocol) – instance of the ReportProtocol object

_save_to_csv(report: ReportProtocol) None[source]

Saves readed data to csv file using Pandas save method.

Parameters:

report (ReportProtocol) – instance of the ReportProtocol object

_erase_report(report: ReportProtocol) None[source]

Erases the report data.

Parameters:

report (ReportProtocol) – instance of the ReportProtocol object

report_processing(report: ReportProtocol) None[source]

Orchiestrates the ReportProtocol processing.

Parameters:

report (ReportProtocol) – instance of the ReportProtocol object

run() NoReturn[source]

Starts listner process on separate thread, awaits objects in the queue.

Returns:

method never returns

Return type:

NoReturn

class components.handlers.WorkerFactory(*, queue: Queue, threads: int = 1)[source]

Concrete class representing WorkerFactory object.

Parameters:
  • queue (Queue) – shared, thread-safe queue object

  • threads (int, optional) – number of threads, equal to number of WorkerProtocol to be deployed, defaults to 1

create_workers() None[source]

Deploys given number of WorkerProtocol.

static active_workers() int[source]

Returns number of currently active WorkerProtocol.

Returns:

number of WorkerProtocol

Return type:

int

class components.handlers.Worker(queue: Queue)[source]

Concrete class representing Worker object.

_read_stream(report: ReportProtocol) None[source]

Reads ReportProtocol response and save it as content atribute. Erases saved response.

Parameters:

report (ReportProtocol) – instance of the ReportProtocol object

_parse_save_path(report: ReportProtocol) PathLike[source]

Parses path to save location.

Parameters:

report (ReportProtocol) – instance of the ReportProtocol object.

Returns:

Path to save location

Return type:

os.PathLike

_save_to_csv(report: ReportProtocol) None[source]

Saves ReportProtocol content to csv file. Sets object flags.

Parameters:

report (ReportProtocol) – instance of the ReportProtocol object

_erase_report(report: ReportProtocol) None[source]

Deletes ReportProtocol object content .

Parameters:

report (ReportProtocol) – instance of the ReportProtocol object

process_report(report: ReportProtocol) None[source]

Orchiestrates entire process of downloading the report.

Parameters:

report (ReportProtocol) – instance of the ReportProtocol object

run() NoReturn[source]

Starts to listen to the queue. Starts processing once will get item from the queue. Sends signal to the queue once task is done.

Returns:

function never returns

Return type:

NoReturn

Excpetions

exception components.exceptions.OutdatedSIDError(message: str = 'Your SID is outdate, please provide recent SID')[source]

Exception raised for errors in the SID value.

message

explanation of the error

Type:

str

exception components.exceptions.EnvFileNotPresent(message: str = '.env file not present in main directory')[source]

Exception raised if the .env file is not present in main directory.

message

explanation of the error

Type:

str