CrumblPy: Examples and Tutorials¶

Table of Contents¶

  • Installation
  • Import
  • SnowflakeToolKit
    • Snowflake Connection
    • Fetch Data
    • Write Data
    • Execute Query (DML)
  • SlackToolKit
    • Slack Connection
    • Post Message
    • Post Reply
    • Post File
    • Push Notification
  • Gmail

Installation¶

In [ ]:
pip install crumblpy

Import the package¶

In [2]:
from CrumblPy import SnowflakeToolKit, SlackToolKit, send_gmail

SnowflakeToolKit¶

Snowflake Connection¶

In [3]:
# Configure the Snowflake connection

# Option 1: Use environment variables (os.environ)
sf = SnowflakeToolKit()

# Option 2: Hard-code the credentials (not recommended)
# sf = SnowflakeToolKit(user='your_username', password='your_password', role='your_role')

# # Option 3: Prefect Secrets (deployment only)
# sf = SnowflakeToolKit(prefect=True)

Fetch Data¶

In [4]:
# Fetch data from Snowflake
sf.fetch_data(
    """
    SELECT * FROM STORES limit 10;
    """
)
Out[4]:
STORE_ID BRAND_ID WHEN_I_WORK_LOCATION_ID PUSH_OPERATIONS_ID SQUARE_LOCATION_ID UBER_EATS_STORE_ID GRUB_HUB_MERCHANT_ID DOORDASH_NAME NAME STATE ... WEEKLY_MINI_PRICE MINI_PRICE MAXIMUM_COOKIES_PER_DAY MINIMUM_LEAD_TIME_IN_HOURS CATERING_COOKIE_IDS SQUARE_ID EVENT_NAME PROCESSED_AT UPDATED_AT CREATED_AT
0 e0628b10-74f8-11ee-8d9f-f38f32e60e86:Store b92e9010-74e6-11ed-9518-ab265dbfb691:Brand NaN 25639 e0628b10-74f8-11ee-8d9f-f38f32e60e86:Store 52f62611-38f2-5fcf-8518-a672693c1602 None Crumbl Canada Nepean Ontario ... 299.0 319.0 NaN NaN None None MODIFY 2024-10-04 19:05:53.570 2024-10-04 19:05:49 2023-10-27 18:44:36
1 d6621dc0-e696-11ec-b408-e37cce0087a4:Store b92e9010-74e6-11ed-9518-ab265dbfb691:Brand NaN None None None None None Garden City New York ... NaN NaN NaN NaN None None MODIFY 2024-10-04 19:05:59.799 2024-10-04 19:05:49 2022-06-07 19:20:05
2 2b963950-a8d1-11ed-86d0-f7fefe7d0de0:Store b92e9010-74e6-11ed-9518-ab265dbfb691:Brand NaN None None None None None Wheat Ridge Colorado ... NaN NaN NaN NaN None None MODIFY 2024-10-23 17:00:59.028 2024-10-23 17:00:56 2023-02-09 23:26:25
3 6a7e55e0-7fd8-11eb-84bd-ab163f932224:Store b92e9010-74e6-11ed-9518-ab265dbfb691:Brand 5112727.0 None 6a7e55e0-7fd8-11eb-84bd-ab163f932224:Store a01f9a90-83e8-5b1e-b479-4c700fc20877 None CRUMBL (HILLSBORO) Hillsboro Oregon ... 249.0 299.0 200.0 48.0 None None MODIFY 2024-10-06 06:01:52.087 2024-10-06 06:00:51 2021-03-08 06:35:04
4 3ff382f0-4c1d-11ee-9d49-6d7f8c129d9e:Store b92e9010-74e6-11ed-9518-ab265dbfb691:Brand NaN None None None None None Edgewater New Jersey ... NaN NaN NaN NaN None None MODIFY 2024-10-04 19:06:00.012 2024-10-04 19:05:49 2023-09-05 18:51:40
5 625b8e70-0c39-11ed-98e1-291a683e694a:Store b92e9010-74e6-11ed-9518-ab265dbfb691:Brand 5446711.0 None 625b8e70-0c39-11ed-98e1-291a683e694a:Store ff077d6f-e754-5c00-9db7-1275d60a4c42 None Crumbl (Norwalk) Norwalk Connecticut ... 249.0 299.0 1000.0 36.0 None None MODIFY 2024-10-04 19:05:57.452 2024-10-04 19:05:49 2022-07-25 16:46:52
6 113ac8a0-50e6-11ef-897e-39d4e6071072:Store be2c5570-74e6-11ed-a722-fb8c9888066e:Brand NaN None 113ac8a0-50e6-11ef-897e-39d4e6071072:Store None None None Preston Idaho ... NaN NaN NaN NaN None None MODIFY 2024-10-04 19:06:00.781 2024-10-04 19:05:49 2024-08-02 15:44:13
7 71d48e70-ae22-11ed-97e0-b3effe7779ef:Store b92e9010-74e6-11ed-9518-ab265dbfb691:Brand NaN None None None None None Grove St. Station New Jersey ... NaN NaN NaN NaN None None MODIFY 2024-10-16 16:12:48.937 2024-10-16 16:12:45 2023-02-16 17:50:48
8 16f5f4d0-513f-11ec-aaed-ab35c35632a6:Store b92e9010-74e6-11ed-9518-ab265dbfb691:Brand NaN None None None None None Northgate Washington ... NaN NaN NaN NaN None None MODIFY 2024-10-04 19:05:57.836 2024-10-04 19:05:49 2021-11-29 18:06:35
9 6291c400-b219-11ed-9811-e5c137e4de84:Store b92e9010-74e6-11ed-9518-ab265dbfb691:Brand NaN None None None None None West Orange New Jersey ... NaN NaN NaN NaN None None MODIFY 2024-10-04 19:05:52.839 2024-10-04 19:05:49 2023-02-21 18:56:01

10 rows × 66 columns

Write Data¶

In [5]:
table_to_write = sf.fetch_data(
    """
    SELECT COOKIE_ID, NAME FROM COOKIES limit 10;
    """
)
table_to_write
Out[5]:
COOKIE_ID NAME
0 9b2ba8a0-9875-11ef-a16e-abc1c2cf6cf5:Cookies Chocolate Peanut Butter Sandwich
1 e72f42b0-c4f1-11ed-a893-2158bb08e2f1:Cookies Apple Pie
2 dfbe4a70-e2ed-11ed-b65e-4bac356c9d69:Cookies Chocolate Chip Test - V1-3
3 37b52280-8fbe-11ee-961e-e1a804ad307a:Cookies Apple Sweet Pie
4 8e90dd00-a0a1-11ec-adcf-a5314440ec47:Cookie Dirt Cake ft. OTTO
5 77eab600-84c1-11ef-b69d-eb08e677efd8:Cookies Holiday Spiced Butter Cake
6 ed449980-72bf-11ee-a2b7-319903869c9c:Cookies Buckeye Brownie*
7 c93898e0-63fd-11ef-acc7-8b24d42a682d:Cookies Cookie Dough Bits
8 854a3ee0-7e6b-11ee-bd83-59656f65a976:Cookies Chocolate Sprinkle Blondie
9 d3c9b120-27e8-11ed-a43a-21c73e66cda7:Cookies R&D Recipe Builder
In [6]:
# Write data to Snowflake
sf.insert_data(
    df = table_to_write,
    table_name = 'CRUMBLPY_DEV',
    auto_create_table=True # May not the best practice in production. The table better be created manually. 
)
In [ ]:
# Check the data we just inserted
sf.fetch_data(
    """
    SELECT * FROM DATA_SCIENCE.CRUMBLPY_DEV;
    """
)
Out[ ]:
COOKIE_ID NAME
0 e72f42b0-c4f1-11ed-a893-2158bb08e2f1:Cookies Apple Pie
1 dfbe4a70-e2ed-11ed-b65e-4bac356c9d69:Cookies Chocolate Chip Test - V1-3
2 37b52280-8fbe-11ee-961e-e1a804ad307a:Cookies Apple Sweet Pie
3 8e90dd00-a0a1-11ec-adcf-a5314440ec47:Cookie Dirt Cake ft. OTTO
4 77eab600-84c1-11ef-b69d-eb08e677efd8:Cookies Holiday Spiced Butter Cake
5 ed449980-72bf-11ee-a2b7-319903869c9c:Cookies Buckeye Brownie*
6 c93898e0-63fd-11ef-acc7-8b24d42a682d:Cookies Cookie Dough Bits
7 854a3ee0-7e6b-11ee-bd83-59656f65a976:Cookies Chocolate Sprinkle Blondie
8 d3c9b120-27e8-11ed-a43a-21c73e66cda7:Cookies R&D Recipe Builder
9 ea84d370-8d22-11eb-b017-13424779f04a:Cookie Chocolate Wafer Bar 
notebook controller is DISPOSED. 

View Jupyter <a href='command:jupyter.viewOutput'>log</a> for further details.

Execute DML Query¶

In [7]:
# Drop the table
sf.execute_query(
    """
    DROP TABLE IF EXISTS ANALYTICS.DATA_SCIENCE.CRUMBLPY_DEV;
    """
)
In [8]:
# # Check it again. This will return an error because the table no longer exists. 
# sf.fetch_data(
#     """
#     SELECT * FROM DATA_SCIENCE.CRUMBLPY_DEV;
#     """
# )

SlackToolKit¶

Slack Connection¶

In [9]:
# Configure the Slack connection

# Channel ID
channel = 'C07ULKY3VD2' # CrumblPy Dev channel

# Option 1: Use environment variables (os.environ)
slack = SlackToolKit(default_channel=channel)

# Option 2: Hard-code the credentials (not recommended)
# slack = SlackToolKit(token='data_bot_token', default_channel=channel)

# # Option 3: Prefect Secrets (deployment only)
# sf = SnowflakeToolKit(prefect=True)

Post Message¶

In [10]:
slack.post_message("Hello from CrumblPy!")

Alt text

Post Reply¶

In [11]:
latest_thread_id = slack.get_thread_id(channel=channel)
slack.post_message("Reply from CrumblPy!", thread_id=latest_thread_id)

Alt text

Post File¶

In [12]:
# create an example file
sf.fetch_data(
    """
    SELECT NAME, OPENED_AT FROM STORES limit 10;
    """
).to_csv('example.csv', index=False)
In [13]:
slack.post_file('example.csv', 'Here is an example file from CrumblPy!', thread_id=latest_thread_id)
# The file will be deleted locally after being uploaded to Slack.

Alt text

Push Notification¶

In [14]:
# Success message. Usually at the end of the script
slack.push_notification(project="CrumblPy")

Alt text

In [15]:
# Error message
try:
    1/0 # This will raise an error because you can't divide by zero
    slack.push_notification(project="CrumblPy")
except Exception as e:
    slack.push_notification(project="CrumblPy", e=e)

Alt text

Gmail¶

In [16]:
# create an example csv
sf.fetch_data(
    """
    SELECT NAME, OPENED_AT FROM STORES limit 10;
    """
).to_csv('example.csv', index=False)
In [17]:
send_gmail(
    sender = 'donation-reporting@crumbl.com',
    recipient = 'steven.wang@crumbl.com',
    subject = 'Subject from CrumblPy',
    body = 'Body from CrumblPy',
    token_path = 'token.json', # This token is not in Doppler or Prefect yet.
    image_paths = ['slack_message.png','slack_reply.png'],
    attachment_paths = ['example.csv']
    )

Alt text