diff --git a/docs/developer/docs/recorder-optimization.md b/docs/developer/docs/recorder-optimization.md new file mode 100644 index 0000000..4f21cda --- /dev/null +++ b/docs/developer/docs/recorder-optimization.md @@ -0,0 +1,289 @@ +# Recorder History Optimization + +**Status**: ✅ IMPLEMENTED +**Last Updated**: 2025-12-07 + +## Overview + +This document describes the implementation of `_unrecorded_attributes` for Tibber Prices entities to prevent Home Assistant Recorder database bloat by excluding non-essential attributes from historical data storage. + +**Reference**: [HA Developer Docs - Excluding State Attributes](https://developers.home-assistant.io/docs/core/entity/#excluding-state-attributes-from-recorder-history) + +## Implementation + +Both `TibberPricesSensor` and `TibberPricesBinarySensor` implement `_unrecorded_attributes` as a class-level `frozenset` to exclude attributes that don't provide value in historical data analysis. + +### Pattern + +```python +class TibberPricesSensor(TibberPricesEntity, SensorEntity): + """tibber_prices Sensor class.""" + + _unrecorded_attributes = frozenset( + { + "description", + "usage_tips", + # ... more attributes + } + ) +``` + +**Key Points:** +- Must be a **class attribute** (not instance attribute) +- Use `frozenset` for immutability and performance +- Applied automatically by Home Assistant's Recorder component + +## Categories of Excluded Attributes + +### 1. Descriptions/Help Text + +**Attributes:** `description`, `usage_tips` + +**Reason:** Static, large text strings (100-500 chars each) that: +- Never change or change very rarely +- Don't provide analytical value in history +- Consume significant database space when recorded every state change +- Can be retrieved from translation files when needed + +**Impact:** ~500-1000 bytes saved per state change + +### 2. Large Nested Structures + +**Attributes:** +- `periods` (binary_sensor) - Array of all period summaries +- `data` (chart_data_export) - Complete price data arrays +- `trend_attributes` - Detailed trend analysis +- `current_trend_attributes` - Current trend details +- `trend_change_attributes` - Trend change analysis +- `volatility_attributes` - Detailed volatility breakdown + +**Reason:** Complex nested data structures that are: +- Serialized to JSON for storage (expensive) +- Create large database rows (2-20 KB each) +- Slow down history queries +- Provide limited value in historical analysis (current state usually sufficient) + +**Impact:** ~10-30 KB saved per state change for affected sensors + +**Example - periods array:** +```json +{ + "periods": [ + { + "start": "2025-12-07T06:00:00+01:00", + "end": "2025-12-07T08:00:00+01:00", + "duration_minutes": 120, + "price_avg": 18.5, + "price_min": 17.2, + "price_max": 19.8, + // ... 10+ more attributes × 10-20 periods + } + ] +} +``` + +### 3. Frequently Changing Diagnostics + +**Attributes:** `icon_color`, `cache_age`, `cache_validity`, `data_completeness`, `data_status` + +**Reason:** +- Change every update cycle (every 15 minutes or more frequently) +- Don't provide long-term analytical value +- Create state changes even when core values haven't changed +- Clutter history with cosmetic changes +- Can be reconstructed from other attributes if needed + +**Impact:** Prevents unnecessary state writes when only cosmetic attributes change + +**Example:** `icon_color` changes from `#00ff00` to `#ffff00` but price hasn't changed → No state write needed + +### 4. Static/Rarely Changing Configuration + +**Attributes:** `tomorrow_expected_after`, `level_value`, `rating_value`, `level_id`, `rating_id`, `currency`, `resolution`, `yaxis_min`, `yaxis_max` + +**Reason:** +- Configuration values that rarely change +- Wastes space when recorded repeatedly +- Can be derived from other attributes or from entity state + +**Impact:** ~100-200 bytes saved per state change + +### 5. Temporary/Time-Bound Data + +**Attributes:** `next_api_poll`, `next_midnight_turnover`, `last_api_fetch`, `last_cache_update`, `last_turnover`, `last_error`, `error` + +**Reason:** +- Only relevant at moment of reading +- Won't be valid after some time +- Similar to `entity_picture` in HA core image entities +- Superseded by next update + +**Impact:** ~200-400 bytes saved per state change + +**Example:** `next_api_poll: "2025-12-07T14:30:00"` stored at 14:15 is useless when viewing history at 15:00 + +### 6. Relaxation Details + +**Attributes:** `relaxation_level`, `relaxation_threshold_original_%`, `relaxation_threshold_applied_%` + +**Reason:** +- Detailed technical information not needed for historical analysis +- Only useful for debugging during active development +- Boolean `relaxation_active` is kept for high-level analysis + +**Impact:** ~50-100 bytes saved per state change + +### 7. Redundant/Derived Data + +**Attributes:** `price_spread`, `volatility`, `diff_%`, `rating_difference_%`, `period_price_diff_from_daily_min`, `period_price_diff_from_daily_min_%`, `periods_total`, `periods_remaining` + +**Reason:** +- Can be calculated from other attributes +- Redundant information +- Doesn't add analytical value to history + +**Impact:** ~100-200 bytes saved per state change + +**Example:** `price_spread = price_max - price_min` (both are recorded, so spread can be calculated) + +## Attributes That ARE Recorded + +These attributes **remain in history** because they provide essential analytical value: + +### Time-Series Core +- `timestamp` - Critical for time-series analysis (ALWAYS FIRST) +- All price values - Core sensor states + +### Diagnostics & Tracking +- `cache_age_minutes` - Numeric value for diagnostics tracking over time +- `updates_today` - Tracking API usage patterns + +### Data Completeness +- `interval_count`, `intervals_available` - Data completeness metrics +- `yesterday_available`, `today_available`, `tomorrow_available` - Boolean status + +### Period Data +- `start`, `end`, `duration_minutes` - Core period timing +- `price_avg`, `price_min`, `price_max` - Core price statistics + +### High-Level Status +- `relaxation_active` - Whether relaxation was used (boolean, useful for analyzing when periods needed relaxation) + +## Expected Database Impact + +### Space Savings + +**Per state change:** +- Before: ~3-8 KB average +- After: ~0.5-1.5 KB average +- **Reduction: 60-85%** + +**Daily per sensor:** +| Sensor Type | Updates/Day | Before | After | Savings | +|------------|-------------|--------|-------|---------| +| High-frequency (15min) | 96 | ~290 KB | ~140 KB | 50% | +| Low-frequency (6h) | 4 | ~32 KB | ~6 KB | 80% | + +### Most Impactful Exclusions + +1. **`periods` array** (binary_sensor) - Saves 2-5 KB per state +2. **`data`** (chart_data_export) - Saves 5-20 KB per state +3. **`trend_attributes`** - Saves 1-2 KB per state +4. **`description`/`usage_tips`** - Saves 500-1000 bytes per state +5. **`icon_color`** - Prevents unnecessary state changes + +### Real-World Impact + +For a typical installation with: +- 80+ sensors +- Updates every 15 minutes +- ~10 sensors updating every minute + +**Before:** ~1.5 GB per month +**After:** ~400-500 MB per month +**Savings:** ~1 GB per month (~66% reduction) + +## Implementation Files + +- **Sensor Platform**: `custom_components/tibber_prices/sensor/core.py` + - Class: `TibberPricesSensor` + - 47 attributes excluded + +- **Binary Sensor Platform**: `custom_components/tibber_prices/binary_sensor/core.py` + - Class: `TibberPricesBinarySensor` + - 30 attributes excluded + +## When to Update _unrecorded_attributes + +### Add to Exclusion List When: + +✅ Adding new **description/help text** attributes +✅ Adding **large nested structures** (arrays, complex objects) +✅ Adding **frequently changing diagnostic info** (colors, formatted strings) +✅ Adding **temporary/time-bound data** (timestamps that become stale) +✅ Adding **redundant/derived calculations** + +### Keep in History When: + +✅ **Core price/timing data** needed for analysis +✅ **Boolean status flags** that show state transitions +✅ **Numeric counters** useful for tracking patterns +✅ **Data that helps understand system behavior** over time + +## Decision Framework + +When adding a new attribute, ask: + +1. **Will this be useful in history queries 1 week from now?** + - No → Exclude + - Yes → Keep + +2. **Can this be calculated from other recorded attributes?** + - Yes → Exclude + - No → Keep + +3. **Is this primarily for current UI display?** + - Yes → Exclude + - No → Keep + +4. **Does this change frequently without indicating state change?** + - Yes → Exclude + - No → Keep + +5. **Is this larger than 100 bytes and not essential for analysis?** + - Yes → Exclude + - No → Keep + +## Testing + +After modifying `_unrecorded_attributes`: + +1. **Restart Home Assistant** to apply changes +2. **Check Recorder database size** before/after +3. **Verify essential attributes** still appear in history +4. **Confirm excluded attributes** don't appear in new state writes + +**SQL Query to check attribute presence:** +```sql +SELECT + state_id, + attributes +FROM states +WHERE entity_id = 'sensor.tibber_home_current_interval_price' +ORDER BY last_updated DESC +LIMIT 5; +``` + +## Maintenance Notes + +- ✅ Must be a **class attribute** (instance attributes are ignored) +- ✅ Use `frozenset` for immutability +- ✅ Only affects **new** state writes (doesn't purge existing history) +- ✅ Attributes still available via `entity.attributes` in templates/automations +- ✅ Only prevents **storage** in Recorder, not runtime availability + +## References + +- [HA Developer Docs - Excluding State Attributes](https://developers.home-assistant.io/docs/core/entity/#excluding-state-attributes-from-recorder-history) +- Implementation PR: [Link when merged] +- Related Issue: [Link if applicable] diff --git a/docs/developer/sidebars.ts b/docs/developer/sidebars.ts index 85dd4b5..6eb975a 100644 --- a/docs/developer/sidebars.ts +++ b/docs/developer/sidebars.ts @@ -32,7 +32,7 @@ const sidebars: SidebarsConfig = { { type: 'category', label: '📐 Advanced Topics', - items: ['period-calculation-theory', 'refactoring-guide', 'performance'], + items: ['period-calculation-theory', 'refactoring-guide', 'performance', 'recorder-optimization'], collapsible: true, collapsed: false, },