hass.tibber_prices/docs/developer/docs/recorder-optimization.md

Recorder History Optimization

Status: ✅ IMPLEMENTED Last Updated: 2025-12-07

Overview

This document describes the implementation of _unrecorded_attributes for Tibber Prices entities to prevent Home Assistant Recorder database bloat by excluding non-essential attributes from historical data storage.

Reference: HA Developer Docs - Excluding State Attributes

Implementation

Both TibberPricesSensor and TibberPricesBinarySensor implement _unrecorded_attributes as a class-level frozenset to exclude attributes that don't provide value in historical data analysis.

Pattern

class TibberPricesSensor(TibberPricesEntity, SensorEntity):
    """tibber_prices Sensor class."""
    
    _unrecorded_attributes = frozenset(
        {
            "description",
            "usage_tips",
            # ... more attributes
        }
    )

Key Points:

• Must be a class attribute (not instance attribute)
• Use frozenset for immutability and performance
• Applied automatically by Home Assistant's Recorder component

Categories of Excluded Attributes

1. Descriptions/Help Text

Attributes: description, usage_tips

Reason: Static, large text strings (100-500 chars each) that:

• Never change or change very rarely
• Don't provide analytical value in history
• Consume significant database space when recorded every state change
• Can be retrieved from translation files when needed

Impact: ~500-1000 bytes saved per state change

2. Large Nested Structures

Attributes:

• periods (binary_sensor) - Array of all period summaries
• data (chart_data_export) - Complete price data arrays
• trend_attributes - Detailed trend analysis
• current_trend_attributes - Current trend details
• trend_change_attributes - Trend change analysis
• volatility_attributes - Detailed volatility breakdown

Reason: Complex nested data structures that are:

• Serialized to JSON for storage (expensive)
• Create large database rows (2-20 KB each)
• Slow down history queries
• Provide limited value in historical analysis (current state usually sufficient)

Impact: ~10-30 KB saved per state change for affected sensors

Example - periods array:

{
  "periods": [
    {
      "start": "2025-12-07T06:00:00+01:00",
      "end": "2025-12-07T08:00:00+01:00",
      "duration_minutes": 120,
      "price_avg": 18.5,
      "price_min": 17.2,
      "price_max": 19.8,
      // ... 10+ more attributes × 10-20 periods
    }
  ]
}

3. Frequently Changing Diagnostics

Attributes: icon_color, cache_age, cache_validity, data_completeness, data_status

Reason:

• Change every update cycle (every 15 minutes or more frequently)
• Don't provide long-term analytical value
• Create state changes even when core values haven't changed
• Clutter history with cosmetic changes
• Can be reconstructed from other attributes if needed

Impact: Prevents unnecessary state writes when only cosmetic attributes change

Example: icon_color changes from #00ff00 to #ffff00 but price hasn't changed → No state write needed

4. Static/Rarely Changing Configuration

Attributes: tomorrow_expected_after, level_value, rating_value, level_id, rating_id, currency, resolution, yaxis_min, yaxis_max

Reason:

• Configuration values that rarely change
• Wastes space when recorded repeatedly
• Can be derived from other attributes or from entity state

Impact: ~100-200 bytes saved per state change

5. Temporary/Time-Bound Data

Attributes: next_api_poll, next_midnight_turnover, last_api_fetch, last_cache_update, last_turnover, last_error, error

Reason:

• Only relevant at moment of reading
• Won't be valid after some time
• Similar to entity_picture in HA core image entities
• Superseded by next update

Impact: ~200-400 bytes saved per state change

Example: next_api_poll: "2025-12-07T14:30:00" stored at 14:15 is useless when viewing history at 15:00

6. Relaxation Details

Attributes: relaxation_level, relaxation_threshold_original_%, relaxation_threshold_applied_%

Reason:

• Detailed technical information not needed for historical analysis
• Only useful for debugging during active development
• Boolean relaxation_active is kept for high-level analysis

Impact: ~50-100 bytes saved per state change

7. Redundant/Derived Data

Attributes: price_spread, volatility, diff_%, rating_difference_%, period_price_diff_from_daily_min, period_price_diff_from_daily_min_%, periods_total, periods_remaining

Reason:

• Can be calculated from other attributes
• Redundant information
• Doesn't add analytical value to history

Impact: ~100-200 bytes saved per state change

Example: price_spread = price_max - price_min (both are recorded, so spread can be calculated)

Attributes That ARE Recorded

These attributes remain in history because they provide essential analytical value:

Time-Series Core

• timestamp - Critical for time-series analysis (ALWAYS FIRST)
• All price values - Core sensor states

Diagnostics & Tracking

• cache_age_minutes - Numeric value for diagnostics tracking over time
• updates_today - Tracking API usage patterns

Data Completeness

• interval_count, intervals_available - Data completeness metrics
• yesterday_available, today_available, tomorrow_available - Boolean status

Period Data

• start, end, duration_minutes - Core period timing
• price_avg, price_min, price_max - Core price statistics

High-Level Status

• relaxation_active - Whether relaxation was used (boolean, useful for analyzing when periods needed relaxation)

Expected Database Impact

Space Savings

Per state change:

• Before: ~3-8 KB average
• After: ~0.5-1.5 KB average
• Reduction: 60-85%

Daily per sensor:

Sensor Type	Updates/Day	Before	After	Savings
High-frequency (15min)	96	~290 KB	~140 KB	50%
Low-frequency (6h)	4	~32 KB	~6 KB	80%

Most Impactful Exclusions

1. periods array (binary_sensor) - Saves 2-5 KB per state
2. data (chart_data_export) - Saves 5-20 KB per state
3. trend_attributes - Saves 1-2 KB per state
4. description/usage_tips - Saves 500-1000 bytes per state
5. icon_color - Prevents unnecessary state changes

Real-World Impact

For a typical installation with:

• 80+ sensors
• Updates every 15 minutes
• ~10 sensors updating every minute

Before: ~1.5 GB per month After: ~400-500 MB per month Savings: ~1 GB per month (~66% reduction)

Implementation Files

• Sensor Platform: custom_components/tibber_prices/sensor/core.py

  • Class: TibberPricesSensor
  • 47 attributes excluded

• Binary Sensor Platform: custom_components/tibber_prices/binary_sensor/core.py

  • Class: TibberPricesBinarySensor
  • 30 attributes excluded

When to Update _unrecorded_attributes

Add to Exclusion List When:

✅ Adding new description/help text attributes ✅ Adding large nested structures (arrays, complex objects) ✅ Adding frequently changing diagnostic info (colors, formatted strings) ✅ Adding temporary/time-bound data (timestamps that become stale) ✅ Adding redundant/derived calculations

Keep in History When:

✅ Core price/timing data needed for analysis ✅ Boolean status flags that show state transitions ✅ Numeric counters useful for tracking patterns ✅ Data that helps understand system behavior over time

Decision Framework

When adding a new attribute, ask:

1. Will this be useful in history queries 1 week from now?

   • No → Exclude
   • Yes → Keep

2. Can this be calculated from other recorded attributes?

   • Yes → Exclude
   • No → Keep

3. Is this primarily for current UI display?

   • Yes → Exclude
   • No → Keep

4. Does this change frequently without indicating state change?

   • Yes → Exclude
   • No → Keep

5. Is this larger than 100 bytes and not essential for analysis?

   • Yes → Exclude
   • No → Keep

Testing

After modifying _unrecorded_attributes:

1. Restart Home Assistant to apply changes
2. Check Recorder database size before/after
3. Verify essential attributes still appear in history
4. Confirm excluded attributes don't appear in new state writes

SQL Query to check attribute presence:

SELECT 
    state_id,
    attributes
FROM states 
WHERE entity_id = 'sensor.tibber_home_current_interval_price'
ORDER BY last_updated DESC 
LIMIT 5;

Maintenance Notes

• ✅ Must be a class attribute (instance attributes are ignored)
• ✅ Use frozenset for immutability
• ✅ Only affects new state writes (doesn't purge existing history)
• ✅ Attributes still available via entity.attributes in templates/automations
• ✅ Only prevents storage in Recorder, not runtime availability

References

• HA Developer Docs - Excluding State Attributes
• Implementation PR: [Link when merged]
• Related Issue: [Link if applicable]