kedro.io.SQLTableDataSet

class kedro.io.SQLTableDataSet(table_name, credentials, load_args=None, save_args=None)[source]

Bases: kedro.io.core.AbstractDataSet, kedro.io.core.ExistsMixin

SQLTableDataSet loads data from a SQL table and saves a pandas dataframe to a table. It uses pandas.DataFrame internally, so it supports all allowed pandas options on read_sql_table and to_sql methods. However, it modifies the save parameters and stores the data with no index. This is designed to make load and save methods symmetric.

Example:

from kedro.io import SQLTableDataSet
import pandas as pd

data = pd.DataFrame({'col1': [1, 2], 'col2': [4, 5],
                     'col3': [5, 6]})
table_name="table_a"
credentials = {
         con: "postgresql://scott:tiger@localhost/test"
}
data_set = SQLTableDataSet(table_name=table_name,
                           credentials=credentials)

data_set.save(data)
reloaded = data_set.load()

assert data.equals(reloaded)

Methods

SQLTableDataSet.__init__(table_name, credentials) Creates a new SQLTableDataSet.
SQLTableDataSet.exists() Checks whether a data set’s output already exists by calling the provided _exists() method.
SQLTableDataSet.from_config(name, config[, …]) Create a data set instance using the configuration provided.
SQLTableDataSet.load() Loads data by delegation to the provided load method.
SQLTableDataSet.save(data) Saves data by delegation to the provided save method.
__init__(table_name, credentials, load_args=None, save_args=None)[source]

Creates a new SQLTableDataSet.

Parameters:
  • table_name (str) – The table name to load or save data to. It overwrites name in save_args and table_name parameters in load_args.
  • credentials (Dict[str, Any]) – A dictionary with a SQLAlchemy connection string. Users are supposed to provide the connection string ‘con’ through credentials. It overwrites con parameter in load_args and save_args in case it is provided.
  • load_args (Optional[Dict[str, Any]]) – Provided to underlying pandas read_sql_table function along with the connection string. To find all supported arguments, see here: https://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_sql_table.html
  • save_args (Optional[Dict[str, Any]]) – Provided to underlying pandas to_sql function along with the connection string. To find all supported arguments, see here: https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.to_sql.html It has index=False in the default parameters.
Raises:

DataSetError – When either table_name or con is empty.

Return type:

None

exists()

Checks whether a data set’s output already exists by calling the provided _exists() method.

Return type:bool
Returns:Flag indicating whether the output already exists.
Raises:DataSetError – when underlying exists method raises error.
classmethod from_config(name, config, load_version=None, save_version=None)

Create a data set instance using the configuration provided.

Parameters:
  • name (str) – Data set name.
  • config (Dict[str, Any]) – Data set config dictionary.
  • load_version (Optional[str]) – Version string to be used for load operation if the data set is versioned. Has no effect on the data set if versioning was not enabled.
  • save_version (Optional[str]) – Version string to be used for save operation if the data set is versioned. Has no effect on the data set if versioning was not enabled.
Return type:

AbstractDataSet

Returns:

An instance of an AbstractDataSet subclass.

Raises:

DataSetError – When the function fails to create the data set from its config.

load()

Loads data by delegation to the provided load method.

Return type:Any
Returns:Data returned by the provided load method.
Raises:DataSetError – When underlying load method raises error.
save(data)

Saves data by delegation to the provided save method.

Parameters:data (Any) – the value to be saved by provided save method.
Raises:DataSetError – when underlying save method raises error.
Return type:None