Welcome to Intake DAL (data access layer) plugin¶
This Intake plugin helps abstract a dataset over heterogeneous storage systems. It also provides a helper method to specialize a hierarchical catalog to a default storage system.
user_events:
driver: dal
args:
default: 'local'
storage:
local: 'csv://{{ CATALOG_DIR }}/data/user_events.csv'
serving: 'in-memory-kv://foo'
batch: 'parquet://{{ CATALOG_DIR }}/data/user_events.parquet'
# Specialize the catalog dal default storage mode datasources
# to be "serving".
cat = DalCatalog(path, storage_mode="serving")
# reads from the serving storage system
# using the in-memory-kv Intake plugin
df = cat.user_events.read()