EnergyID Webhook V2 Demo¶

This notebook demonstrates the new EnergyID Webhook V2 API, which includes device provisioning, claiming, and the new data format for sending measurements to EnergyID.

The new webhook implementation offers several advantages:

  • Simplified device registration through a claiming process
  • Token-based authentication with automatic refresh
  • Standardized metric types for common energy measurements
  • Support for batch uploads and data aggregation
  • More efficient data format for transmitting measurements

Summary¶

This notebook demonstrates the EnergyID Webhook V2 API with the following features:

  1. Device Provisioning: Setting up a device with a unique identifier
  2. Claiming Process: Allowing users to claim a device and link it to their EnergyID account
  3. Token-based Authentication: Automatic token refresh and handling of expired tokens
  4. Simplified Data Format: Using standardized keys like el, pv, gas for common energy metrics
  5. Batch Data Uploads: Sending multiple metrics in a single request
  6. Custom Timestamps: Attaching specific timestamps to data
  7. Prefixed Metrics: Using prefixes to handle multiple metrics of the same type
  8. Sensor Objects: Managing sensor state and synchronization
  9. Automatic Synchronization: Periodically sending updates without manual intervention

The new Webhook V2 API provides a more efficient and user-friendly way to integrate devices with EnergyID.

1. Setup & Dependencies¶

First, let's install the required dependencies if needed and import the necessary libraries:

In [17]:
# Uncomment to install dependencies if needed
# !pip install -q aiohttp nest-asyncio python-dotenv

import asyncio
import datetime as dt
import json
import logging
import os
import random
import uuid

import nest_asyncio
from dotenv import load_dotenv

# Configure logging for better output
logging.basicConfig(
    level=logging.WARNING,
    format="%(asctime)s - %(name)s - %(levelname)s - %(message)s"
)
logger = logging.getLogger("energyid-demo")

# Apply nest_asyncio to make asyncio work in Jupyter
nest_asyncio.apply()

# Load environment variables
load_dotenv()
Out[17]:
True

2. WebhookClient Implementation¶

We'll implement a client for the EnergyID Webhook V2 API. This client handles the device provisioning flow, authentication, and sending data to EnergyID.

For brevity, we'll import our implementation from energyid_webhooks.client module. Let's look at how to use it:

In [18]:
# Import the WebhookClient module
from energyid_webhooks.client_v2 import WebhookClient

# We'll also create a helper function to run async functions in the notebook
def run_async(coroutine):
    """Run an async coroutine."""
    return asyncio.get_event_loop().run_until_complete(coroutine)

3. Device Provisioning and Claiming¶

The first step in using the EnergyID Webhook V2 API is device provisioning. This process involves:

  1. Creating a unique device identifier
  2. Authenticating with EnergyID using provisioning credentials
  3. Claiming the device through the EnergyID web interface

Let's set up the client with credentials from environment variables:

NOTE: you can also use existing device by specifying it in the .env. To not have to add a new device.

In [19]:
# Get credentials from environment variables
client_id = os.getenv("CLIENT_ID")
client_secret = os.getenv("CLIENT_SECRET")
device_id = os.getenv("ENERGYID_DEVICE_ID")
device_name = os.getenv("ENERGYID_DEVICE_NAME", "Jupyter Demo Device")

if not client_id or not client_secret:
    raise ValueError("Please set CLIENT_ID and CLIENT_SECRET environment variables")


# Generate a device ID if not provided in environment
if not device_id:
    device_id = f"jupyter_demo_{uuid.uuid4().hex[:8]}"
    print(f"Generated new device ID: {device_id}")
else:
    print(f"Using existing device ID from environment: {device_id}")

# Create the webhook client
client = WebhookClient(
    client_id=client_id,
    client_secret=client_secret,
    device_id=device_id,
    device_name=device_name,
    firmware_version="1.0.0",
)
# Display device info for future reference
print("\nDevice info for future reference:")
print(f"device_id: {client.device_id}")
print(f"device_name: {client.device_name}")
print(f"firmware_version: {client.firmware_version}")
print("To reuse this device in future runs, set these environment variables:")
print(f"ENERGYID_DEVICE_ID={client.device_id}")
print(f"ENERGYID_DEVICE_NAME={client.device_name}")

Using existing device ID from environment: jupyter_demo_c6157c76

Device info for future reference:
device_id: jupyter_demo_c6157c76
device_name: Jupyter Demo Device
firmware_version: 1.0.0
To reuse this device in future runs, set these environment variables:
ENERGYID_DEVICE_ID=jupyter_demo_c6157c76
ENERGYID_DEVICE_NAME=Jupyter Demo Device

Now let's authenticate and check if the device is already claimed:

In [20]:
# Authenticate with EnergyID
is_claimed = run_async(client.authenticate())

if is_claimed:
    print("✅ Device is already claimed and ready to send data!")
    print(f"Webhook URL: {client.webhook_url}")
    print(f"Auth valid until: {client.auth_valid_until}")
    print(f"\nWebhook policy: {json.dumps(client.webhook_policy, indent=2)}")
else:
    # Device needs to be claimed
    claim_info = client.get_claim_info()
    print("⚠️ Device needs to be claimed before sending data!")
    print(f"\nClaim Code: {claim_info['claim_code']}")
    print(f"Claim URL: {claim_info['claim_url']}")
    print(f"Valid until: {claim_info['valid_until']}")
    print("\n1. Visit the claim URL above in your browser")
    print("2. Log in to your EnergyID account if needed")
    print("3. Enter the claim code shown above")
    print("4. Once claimed, re-run this cell to continue or run the next cell")

2025-02-28 20:10:56,959 - energyid_webhooks.client_v2 - INFO - Webhook allows interval: PT1M
2025-02-28 20:10:56,961 - energyid_webhooks.client_v2 - INFO - Webhook allows metrics: ['*']

✅ Device is already claimed and ready to send data!
Webhook URL: https://hooks.energyid.eu/webhook-in
Auth valid until: 2025-03-01 19:10:56.959704+00:00

Webhook policy: {
  "allowedInterval": "PT1M",
  "allowedMetrics": [
    "*"
  ]
}

If the device isn't claimed yet, follow the instructions above to claim it through the EnergyID web interface. Then re-run the cell or the next one to verify it's been claimed successfully.

In [21]:
# Authenticate with EnergyID
is_claimed = run_async(client.authenticate())

if is_claimed:
    print("✅ Device is already claimed and ready to send data!")
    print(f"Webhook URL: {client.webhook_url}")
    # print(f"client info: {client.client_info}")
    print(f"Auth valid until: {client.auth_valid_until}")
    print(f"\nWebhook policy: {json.dumps(client.webhook_policy, indent=2)}")
else:
    raise ValueError("Device is not claimed yet, please claim it first")

2025-02-28 20:10:57,045 - energyid_webhooks.client_v2 - INFO - Webhook allows interval: PT1M
2025-02-28 20:10:57,046 - energyid_webhooks.client_v2 - INFO - Webhook allows metrics: ['*']

✅ Device is already claimed and ready to send data!
Webhook URL: https://hooks.energyid.eu/webhook-in
Auth valid until: 2025-03-01 19:10:57.045628+00:00

Webhook policy: {
  "allowedInterval": "PT1M",
  "allowedMetrics": [
    "*"
  ]
}

3.5 Testing reauthentication¶

In [22]:
# Test reauthentication by simulating an expired token
async def test_reauthentication(client):
    """Test that the client properly handles a 401 Unauthorized response."""
    if not client.is_claimed:
        print("⚠️ Device must be claimed to test reauthentication")
        return

    print("Testing reauthentication flow:")

    # Create a copy of the original headers to restore later
    original_headers = client.headers.copy() if client.headers else None

    try:
        # 1. Temporarily remove the authorization header to force a 401
        if client.headers and "authorization" in client.headers:
            # Save the auth header value
            auth_value = client.headers.pop("authorization")
            print("1. Removed authorization header to simulate expired token")
        else:
            print("1. Cannot test - no authorization header found")
            return

        # 2. Try to send data without the authorization header
        data = {"el": 9999.9}  # Use a distinctive value
        print("2. Sending data without authorization header...")

        response = await client.send_data(data)
        print("   ✅ Client successfully reauthorized and sent data")

        # 3. Verify token refresh worked
        print("3. Checking authentication status...")
        if client.auth_valid_until:
            print(f"   ✅ Token refreshed, valid until: {client.auth_valid_until}")
        else:
            print("   ❌ Token refresh might have failed, no valid_until time set")

        return True
    finally:
        # Restore the original headers if needed
        if original_headers:
            client.headers = original_headers

# Run the reauthentication test
if client.is_claimed:
    run_async(test_reauthentication(client))

2025-02-28 20:10:57,060 - energyid_webhooks.client_v2 - INFO - Proactively refreshing token based on reauth_interval of 24 hours
2025-02-28 20:10:57,127 - energyid_webhooks.client_v2 - INFO - Webhook allows interval: PT1M
2025-02-28 20:10:57,128 - energyid_webhooks.client_v2 - INFO - Webhook allows metrics: ['*']

Testing reauthentication flow:
1. Removed authorization header to simulate expired token
2. Sending data without authorization header...
   ✅ Client successfully reauthorized and sent data
3. Checking authentication status...
   ✅ Token refreshed, valid until: 2025-03-01 19:10:57.127488+00:00

4. Sending Data in the New Format¶

Once the device is claimed, we can start sending data. The new V2 API uses a simpler key-value structure with standardized metric keys:

In [23]:
# Check if device is claimed before proceeding
if not client.is_claimed:
    print("⚠️ Please claim the device first!")
else:
    # Create a simple data point with current timestamp
    data = {
        # ts is automatically added if not provided
        "el": 1250.5,  # Electricity consumption in kWh
        "pv": 3560.2   # Solar production in kWh
    }

    # Send the data
    response = run_async(client.send_data(data))
    print("✅ Data sent successfully!")
    print(f"Response: {response}")

2025-02-28 20:10:57,273 - energyid_webhooks.client_v2 - INFO - Proactively refreshing token based on reauth_interval of 24 hours
2025-02-28 20:10:57,321 - energyid_webhooks.client_v2 - INFO - Webhook allows interval: PT1M
2025-02-28 20:10:57,323 - energyid_webhooks.client_v2 - INFO - Webhook allows metrics: ['*']

✅ Data sent successfully!
Response:

5. Sending Multiple Metrics in a Single Request¶

The new format allows sending multiple metrics in a single request, which is more efficient:

In [24]:
# Check if device is claimed before proceeding
if not client.is_claimed:
    print("⚠️ Please claim the device first!")
else:
    # Create a batch of metrics
    metrics = {
        "el": 1255.7,        # Electricity consumption
        "el-i": 150.3,       # Electricity injection
        "pv": 3570.8,        # Solar production
        "gas": 450.2,        # Gas consumption
        "temperature": 21.5  # Custom metric for temperature
    }

    # Send all metrics in a single request
    response = run_async(client.send_batch_data(metrics))
    print("✅ Batch data sent successfully!")
    print(f"Response: {response}")

2025-02-28 20:10:57,507 - energyid_webhooks.client_v2 - INFO - Proactively refreshing token based on reauth_interval of 24 hours
2025-02-28 20:10:57,578 - energyid_webhooks.client_v2 - INFO - Webhook allows interval: PT1M
2025-02-28 20:10:57,579 - energyid_webhooks.client_v2 - INFO - Webhook allows metrics: ['*']

✅ Batch data sent successfully!
Response:

6. Sending Data with Specific Timestamp¶

We can also send data with a specific timestamp instead of using the current time:

In [25]:
# Check if device is claimed before proceeding
if not client.is_claimed:
    print("⚠️ Please claim the device first!")
else:
    # Create data for a specific time (yesterday)
    yesterday = dt.datetime.now(dt.timezone.utc) - dt.timedelta(days=1)
    metrics = {
        "el": 1240.3,
        "pv": 3550.1
    }

    # Send with specific timestamp
    response = run_async(client.send_batch_data(metrics, timestamp=yesterday))
    print(f"✅ Data sent with timestamp {yesterday.isoformat()}")
    print(f"Response: {response}")

2025-02-28 20:10:57,723 - energyid_webhooks.client_v2 - INFO - Proactively refreshing token based on reauth_interval of 24 hours
2025-02-28 20:10:57,783 - energyid_webhooks.client_v2 - INFO - Webhook allows interval: PT1M
2025-02-28 20:10:57,785 - energyid_webhooks.client_v2 - INFO - Webhook allows metrics: ['*']

✅ Data sent with timestamp 2025-02-27T19:10:57.723076+00:00
Response:

7. Using Prefixed Metrics¶

The new API also supports prefixed metrics to handle multiple data points of the same type:

In [26]:
# Check if device is claimed before proceeding
if not client.is_claimed:
    print("⚠️ Please claim the device first!")
else:
    # Create data with prefixed metrics
    metrics = {
        # Electricity with time-of-use tariffs
        "el.t1": 850.5,  # Day tariff
        "el.t2": 410.2,  # Night tariff

        # Electricity injection with time-of-use tariffs
        "el-i.t1": 120.3,  # Day tariff
        "el-i.t2": 35.7,   # Night tariff

        # Custom sensors
        "temperature.living": 21.5,
        "temperature.outside": 15.2
    }

    # Send with prefixed metrics
    response = run_async(client.send_batch_data(metrics))
    print("✅ Prefixed metrics sent successfully!")
    print(f"Response: {response}")

2025-02-28 20:10:57,966 - energyid_webhooks.client_v2 - INFO - Proactively refreshing token based on reauth_interval of 24 hours
2025-02-28 20:10:58,025 - energyid_webhooks.client_v2 - INFO - Webhook allows interval: PT1M
2025-02-28 20:10:58,026 - energyid_webhooks.client_v2 - INFO - Webhook allows metrics: ['*']

✅ Prefixed metrics sent successfully!
Response:

8. Using Sensor Objects¶

The WebhookClient provides a Sensor class to manage each sensor's state and synchronization:

In [27]:
# Check if device is claimed before proceeding
if not client.is_claimed:
    print("⚠️ Please claim the device first!")
else:
    # Set up some sensors
    electricity = client.add_sensor("el")
    electricity_injection = client.add_sensor("el-i")
    pv = client.add_sensor("pv")
    gas = client.add_sensor("gas")

    # Update sensor values
    electricity.update(1260.0)
    electricity_injection.update(155.2)
    pv.update(3580.0)
    gas.update(455.3)

    # Synchronize the updated sensors
    response = run_async(client.synchronize_sensors())
    print("✅ Sensors synchronized successfully!")
    print(f"Response: {response}")

2025-02-28 20:10:58,079 - energyid_webhooks.client_v2 - INFO - Proactively refreshing token based on reauth_interval of 24 hours
2025-02-28 20:10:58,129 - energyid_webhooks.client_v2 - INFO - Webhook allows interval: PT1M
2025-02-28 20:10:58,130 - energyid_webhooks.client_v2 - INFO - Webhook allows metrics: ['*']

✅ Sensors synchronized successfully!
Response:

9. Simulating Continuous Data Updates¶

Let's simulate how you might continuously update data over time:

In [28]:
async def simulate_updates(client, updates=5, interval=2):
    """Simulate sensor updates over time."""
    if not client.is_claimed:
        print("⚠️ Please claim the device first!")
        return

    # Set up sensors if they don't exist
    electricity = client.add_sensor("el")
    pv = client.add_sensor("pv")
    temperature = client.add_sensor("temperature.living")

    # Starting values
    el_value = 1260.0
    pv_value = 3580.0
    temp_value = 21.0

    print(f"Simulating {updates} updates at {interval} second intervals...")

    for i in range(updates):
        # Update values with small random changes
        el_value += random.uniform(0.1, 0.5)  # Small increase in consumption
        pv_value += random.uniform(0.2, 1.0)  # Larger increase in production
        temp_value = max(18, min(25, temp_value + random.uniform(-0.5, 0.5)))

        # Update sensors
        electricity.update(el_value)
        pv.update(pv_value)
        temperature.update(temp_value)

        now = dt.datetime.now()
        print(f"[{now.isoformat()}] Update {i+1}/{updates}: el={el_value:.2f}, "
              f"pv={pv_value:.2f}, temp={temp_value:.1f}")

        # Synchronize after each update (in a real application, you'd do this less frequently)
        if i % 2 == 1:  # Sync every other update
            print("   Synchronizing...")
            await client.synchronize_sensors()

        # Wait between updates
        if i < updates - 1:  # Don't wait after the last update
            await asyncio.sleep(interval)

    print("\n✅ Simulation complete!")

# Run the simulation
run_async(simulate_updates(client, updates=5, interval=2))

Simulating 5 updates at 2 second intervals...
[2025-02-28T20:10:58.249343] Update 1/5: el=1260.32, pv=3580.64, temp=21.4

2025-02-28 20:11:00,252 - energyid_webhooks.client_v2 - INFO - Proactively refreshing token based on reauth_interval of 24 hours
2025-02-28 20:11:00,361 - energyid_webhooks.client_v2 - INFO - Webhook allows interval: PT1M
2025-02-28 20:11:00,363 - energyid_webhooks.client_v2 - INFO - Webhook allows metrics: ['*']

[2025-02-28T20:11:00.252475] Update 2/5: el=1260.45, pv=3581.44, temp=21.5
   Synchronizing...
[2025-02-28T20:11:02.463947] Update 3/5: el=1260.69, pv=3582.07, temp=21.5

2025-02-28 20:11:04,467 - energyid_webhooks.client_v2 - INFO - Proactively refreshing token based on reauth_interval of 24 hours
2025-02-28 20:11:04,518 - energyid_webhooks.client_v2 - INFO - Webhook allows interval: PT1M
2025-02-28 20:11:04,520 - energyid_webhooks.client_v2 - INFO - Webhook allows metrics: ['*']

[2025-02-28T20:11:04.466995] Update 4/5: el=1260.94, pv=3583.05, temp=21.4
   Synchronizing...
[2025-02-28T20:11:06.617455] Update 5/5: el=1261.19, pv=3583.97, temp=21.1

✅ Simulation complete!

10. Auto-synchronization¶

The WebhookClient supports automatic synchronization at a specified interval:

In [ ]:
# Start auto-sync with a 30-second interval
client.start_auto_sync(30)
print("✅ Auto-sync started with 30-second interval")
print("   Now you can update sensors and they will be synchronized automatically")

✅ Auto-sync started with 30-second interval
   Now you can update sensors and they will be synchronized automatically

2025-02-28 20:11:06,630 - energyid_webhooks.client_v2 - INFO - Proactively refreshing token based on reauth_interval of 24 hours

Let's update some sensors and rely on auto-sync to send the data:

In [30]:
# Update sensors without manually synchronizing
client.update_sensor("el", 1265.5)
client.update_sensor("pv", 3590.2)
client.update_sensor("temperature.living", 22.0)

print("✅ Sensors updated, they will be synchronized automatically")
print("   Wait for the auto-sync interval to see the data being sent")

✅ Sensors updated, they will be synchronized automatically
   Wait for the auto-sync interval to see the data being sent

11. Clean Up¶

Finally, let's properly close the client to clean up resources:

In [31]:
# Close the client
run_async(client.close())
print("✅ Client closed and resources cleaned up!")

✅ Client closed and resources cleaned up!