mirror of
https://github.com/jpawlowski/hass.tibber_prices.git
synced 2026-03-30 05:13:40 +00:00
docs(developer): add recorder optimization guide
Add comprehensive documentation for _unrecorded_attributes implementation, categorizing all excluded attributes with reasoning, expected database impact, and decision framework for future attributes. Added to Developer Docs → Advanced Topics navigation. Content includes: - 7 exclusion categories with examples - Space savings calculations (60-85% reduction) - Decision framework for new attributes - Testing and validation guidelines - SQL queries for verification
This commit is contained in:
parent
763a6b76b9
commit
a9c04dc0ec
2 changed files with 290 additions and 1 deletions
289
docs/developer/docs/recorder-optimization.md
Normal file
289
docs/developer/docs/recorder-optimization.md
Normal file
|
|
@ -0,0 +1,289 @@
|
||||||
|
# Recorder History Optimization
|
||||||
|
|
||||||
|
**Status**: ✅ IMPLEMENTED
|
||||||
|
**Last Updated**: 2025-12-07
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
This document describes the implementation of `_unrecorded_attributes` for Tibber Prices entities to prevent Home Assistant Recorder database bloat by excluding non-essential attributes from historical data storage.
|
||||||
|
|
||||||
|
**Reference**: [HA Developer Docs - Excluding State Attributes](https://developers.home-assistant.io/docs/core/entity/#excluding-state-attributes-from-recorder-history)
|
||||||
|
|
||||||
|
## Implementation
|
||||||
|
|
||||||
|
Both `TibberPricesSensor` and `TibberPricesBinarySensor` implement `_unrecorded_attributes` as a class-level `frozenset` to exclude attributes that don't provide value in historical data analysis.
|
||||||
|
|
||||||
|
### Pattern
|
||||||
|
|
||||||
|
```python
|
||||||
|
class TibberPricesSensor(TibberPricesEntity, SensorEntity):
|
||||||
|
"""tibber_prices Sensor class."""
|
||||||
|
|
||||||
|
_unrecorded_attributes = frozenset(
|
||||||
|
{
|
||||||
|
"description",
|
||||||
|
"usage_tips",
|
||||||
|
# ... more attributes
|
||||||
|
}
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
**Key Points:**
|
||||||
|
- Must be a **class attribute** (not instance attribute)
|
||||||
|
- Use `frozenset` for immutability and performance
|
||||||
|
- Applied automatically by Home Assistant's Recorder component
|
||||||
|
|
||||||
|
## Categories of Excluded Attributes
|
||||||
|
|
||||||
|
### 1. Descriptions/Help Text
|
||||||
|
|
||||||
|
**Attributes:** `description`, `usage_tips`
|
||||||
|
|
||||||
|
**Reason:** Static, large text strings (100-500 chars each) that:
|
||||||
|
- Never change or change very rarely
|
||||||
|
- Don't provide analytical value in history
|
||||||
|
- Consume significant database space when recorded every state change
|
||||||
|
- Can be retrieved from translation files when needed
|
||||||
|
|
||||||
|
**Impact:** ~500-1000 bytes saved per state change
|
||||||
|
|
||||||
|
### 2. Large Nested Structures
|
||||||
|
|
||||||
|
**Attributes:**
|
||||||
|
- `periods` (binary_sensor) - Array of all period summaries
|
||||||
|
- `data` (chart_data_export) - Complete price data arrays
|
||||||
|
- `trend_attributes` - Detailed trend analysis
|
||||||
|
- `current_trend_attributes` - Current trend details
|
||||||
|
- `trend_change_attributes` - Trend change analysis
|
||||||
|
- `volatility_attributes` - Detailed volatility breakdown
|
||||||
|
|
||||||
|
**Reason:** Complex nested data structures that are:
|
||||||
|
- Serialized to JSON for storage (expensive)
|
||||||
|
- Create large database rows (2-20 KB each)
|
||||||
|
- Slow down history queries
|
||||||
|
- Provide limited value in historical analysis (current state usually sufficient)
|
||||||
|
|
||||||
|
**Impact:** ~10-30 KB saved per state change for affected sensors
|
||||||
|
|
||||||
|
**Example - periods array:**
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"periods": [
|
||||||
|
{
|
||||||
|
"start": "2025-12-07T06:00:00+01:00",
|
||||||
|
"end": "2025-12-07T08:00:00+01:00",
|
||||||
|
"duration_minutes": 120,
|
||||||
|
"price_avg": 18.5,
|
||||||
|
"price_min": 17.2,
|
||||||
|
"price_max": 19.8,
|
||||||
|
// ... 10+ more attributes × 10-20 periods
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3. Frequently Changing Diagnostics
|
||||||
|
|
||||||
|
**Attributes:** `icon_color`, `cache_age`, `cache_validity`, `data_completeness`, `data_status`
|
||||||
|
|
||||||
|
**Reason:**
|
||||||
|
- Change every update cycle (every 15 minutes or more frequently)
|
||||||
|
- Don't provide long-term analytical value
|
||||||
|
- Create state changes even when core values haven't changed
|
||||||
|
- Clutter history with cosmetic changes
|
||||||
|
- Can be reconstructed from other attributes if needed
|
||||||
|
|
||||||
|
**Impact:** Prevents unnecessary state writes when only cosmetic attributes change
|
||||||
|
|
||||||
|
**Example:** `icon_color` changes from `#00ff00` to `#ffff00` but price hasn't changed → No state write needed
|
||||||
|
|
||||||
|
### 4. Static/Rarely Changing Configuration
|
||||||
|
|
||||||
|
**Attributes:** `tomorrow_expected_after`, `level_value`, `rating_value`, `level_id`, `rating_id`, `currency`, `resolution`, `yaxis_min`, `yaxis_max`
|
||||||
|
|
||||||
|
**Reason:**
|
||||||
|
- Configuration values that rarely change
|
||||||
|
- Wastes space when recorded repeatedly
|
||||||
|
- Can be derived from other attributes or from entity state
|
||||||
|
|
||||||
|
**Impact:** ~100-200 bytes saved per state change
|
||||||
|
|
||||||
|
### 5. Temporary/Time-Bound Data
|
||||||
|
|
||||||
|
**Attributes:** `next_api_poll`, `next_midnight_turnover`, `last_api_fetch`, `last_cache_update`, `last_turnover`, `last_error`, `error`
|
||||||
|
|
||||||
|
**Reason:**
|
||||||
|
- Only relevant at moment of reading
|
||||||
|
- Won't be valid after some time
|
||||||
|
- Similar to `entity_picture` in HA core image entities
|
||||||
|
- Superseded by next update
|
||||||
|
|
||||||
|
**Impact:** ~200-400 bytes saved per state change
|
||||||
|
|
||||||
|
**Example:** `next_api_poll: "2025-12-07T14:30:00"` stored at 14:15 is useless when viewing history at 15:00
|
||||||
|
|
||||||
|
### 6. Relaxation Details
|
||||||
|
|
||||||
|
**Attributes:** `relaxation_level`, `relaxation_threshold_original_%`, `relaxation_threshold_applied_%`
|
||||||
|
|
||||||
|
**Reason:**
|
||||||
|
- Detailed technical information not needed for historical analysis
|
||||||
|
- Only useful for debugging during active development
|
||||||
|
- Boolean `relaxation_active` is kept for high-level analysis
|
||||||
|
|
||||||
|
**Impact:** ~50-100 bytes saved per state change
|
||||||
|
|
||||||
|
### 7. Redundant/Derived Data
|
||||||
|
|
||||||
|
**Attributes:** `price_spread`, `volatility`, `diff_%`, `rating_difference_%`, `period_price_diff_from_daily_min`, `period_price_diff_from_daily_min_%`, `periods_total`, `periods_remaining`
|
||||||
|
|
||||||
|
**Reason:**
|
||||||
|
- Can be calculated from other attributes
|
||||||
|
- Redundant information
|
||||||
|
- Doesn't add analytical value to history
|
||||||
|
|
||||||
|
**Impact:** ~100-200 bytes saved per state change
|
||||||
|
|
||||||
|
**Example:** `price_spread = price_max - price_min` (both are recorded, so spread can be calculated)
|
||||||
|
|
||||||
|
## Attributes That ARE Recorded
|
||||||
|
|
||||||
|
These attributes **remain in history** because they provide essential analytical value:
|
||||||
|
|
||||||
|
### Time-Series Core
|
||||||
|
- `timestamp` - Critical for time-series analysis (ALWAYS FIRST)
|
||||||
|
- All price values - Core sensor states
|
||||||
|
|
||||||
|
### Diagnostics & Tracking
|
||||||
|
- `cache_age_minutes` - Numeric value for diagnostics tracking over time
|
||||||
|
- `updates_today` - Tracking API usage patterns
|
||||||
|
|
||||||
|
### Data Completeness
|
||||||
|
- `interval_count`, `intervals_available` - Data completeness metrics
|
||||||
|
- `yesterday_available`, `today_available`, `tomorrow_available` - Boolean status
|
||||||
|
|
||||||
|
### Period Data
|
||||||
|
- `start`, `end`, `duration_minutes` - Core period timing
|
||||||
|
- `price_avg`, `price_min`, `price_max` - Core price statistics
|
||||||
|
|
||||||
|
### High-Level Status
|
||||||
|
- `relaxation_active` - Whether relaxation was used (boolean, useful for analyzing when periods needed relaxation)
|
||||||
|
|
||||||
|
## Expected Database Impact
|
||||||
|
|
||||||
|
### Space Savings
|
||||||
|
|
||||||
|
**Per state change:**
|
||||||
|
- Before: ~3-8 KB average
|
||||||
|
- After: ~0.5-1.5 KB average
|
||||||
|
- **Reduction: 60-85%**
|
||||||
|
|
||||||
|
**Daily per sensor:**
|
||||||
|
| Sensor Type | Updates/Day | Before | After | Savings |
|
||||||
|
|------------|-------------|--------|-------|---------|
|
||||||
|
| High-frequency (15min) | 96 | ~290 KB | ~140 KB | 50% |
|
||||||
|
| Low-frequency (6h) | 4 | ~32 KB | ~6 KB | 80% |
|
||||||
|
|
||||||
|
### Most Impactful Exclusions
|
||||||
|
|
||||||
|
1. **`periods` array** (binary_sensor) - Saves 2-5 KB per state
|
||||||
|
2. **`data`** (chart_data_export) - Saves 5-20 KB per state
|
||||||
|
3. **`trend_attributes`** - Saves 1-2 KB per state
|
||||||
|
4. **`description`/`usage_tips`** - Saves 500-1000 bytes per state
|
||||||
|
5. **`icon_color`** - Prevents unnecessary state changes
|
||||||
|
|
||||||
|
### Real-World Impact
|
||||||
|
|
||||||
|
For a typical installation with:
|
||||||
|
- 80+ sensors
|
||||||
|
- Updates every 15 minutes
|
||||||
|
- ~10 sensors updating every minute
|
||||||
|
|
||||||
|
**Before:** ~1.5 GB per month
|
||||||
|
**After:** ~400-500 MB per month
|
||||||
|
**Savings:** ~1 GB per month (~66% reduction)
|
||||||
|
|
||||||
|
## Implementation Files
|
||||||
|
|
||||||
|
- **Sensor Platform**: `custom_components/tibber_prices/sensor/core.py`
|
||||||
|
- Class: `TibberPricesSensor`
|
||||||
|
- 47 attributes excluded
|
||||||
|
|
||||||
|
- **Binary Sensor Platform**: `custom_components/tibber_prices/binary_sensor/core.py`
|
||||||
|
- Class: `TibberPricesBinarySensor`
|
||||||
|
- 30 attributes excluded
|
||||||
|
|
||||||
|
## When to Update _unrecorded_attributes
|
||||||
|
|
||||||
|
### Add to Exclusion List When:
|
||||||
|
|
||||||
|
✅ Adding new **description/help text** attributes
|
||||||
|
✅ Adding **large nested structures** (arrays, complex objects)
|
||||||
|
✅ Adding **frequently changing diagnostic info** (colors, formatted strings)
|
||||||
|
✅ Adding **temporary/time-bound data** (timestamps that become stale)
|
||||||
|
✅ Adding **redundant/derived calculations**
|
||||||
|
|
||||||
|
### Keep in History When:
|
||||||
|
|
||||||
|
✅ **Core price/timing data** needed for analysis
|
||||||
|
✅ **Boolean status flags** that show state transitions
|
||||||
|
✅ **Numeric counters** useful for tracking patterns
|
||||||
|
✅ **Data that helps understand system behavior** over time
|
||||||
|
|
||||||
|
## Decision Framework
|
||||||
|
|
||||||
|
When adding a new attribute, ask:
|
||||||
|
|
||||||
|
1. **Will this be useful in history queries 1 week from now?**
|
||||||
|
- No → Exclude
|
||||||
|
- Yes → Keep
|
||||||
|
|
||||||
|
2. **Can this be calculated from other recorded attributes?**
|
||||||
|
- Yes → Exclude
|
||||||
|
- No → Keep
|
||||||
|
|
||||||
|
3. **Is this primarily for current UI display?**
|
||||||
|
- Yes → Exclude
|
||||||
|
- No → Keep
|
||||||
|
|
||||||
|
4. **Does this change frequently without indicating state change?**
|
||||||
|
- Yes → Exclude
|
||||||
|
- No → Keep
|
||||||
|
|
||||||
|
5. **Is this larger than 100 bytes and not essential for analysis?**
|
||||||
|
- Yes → Exclude
|
||||||
|
- No → Keep
|
||||||
|
|
||||||
|
## Testing
|
||||||
|
|
||||||
|
After modifying `_unrecorded_attributes`:
|
||||||
|
|
||||||
|
1. **Restart Home Assistant** to apply changes
|
||||||
|
2. **Check Recorder database size** before/after
|
||||||
|
3. **Verify essential attributes** still appear in history
|
||||||
|
4. **Confirm excluded attributes** don't appear in new state writes
|
||||||
|
|
||||||
|
**SQL Query to check attribute presence:**
|
||||||
|
```sql
|
||||||
|
SELECT
|
||||||
|
state_id,
|
||||||
|
attributes
|
||||||
|
FROM states
|
||||||
|
WHERE entity_id = 'sensor.tibber_home_current_interval_price'
|
||||||
|
ORDER BY last_updated DESC
|
||||||
|
LIMIT 5;
|
||||||
|
```
|
||||||
|
|
||||||
|
## Maintenance Notes
|
||||||
|
|
||||||
|
- ✅ Must be a **class attribute** (instance attributes are ignored)
|
||||||
|
- ✅ Use `frozenset` for immutability
|
||||||
|
- ✅ Only affects **new** state writes (doesn't purge existing history)
|
||||||
|
- ✅ Attributes still available via `entity.attributes` in templates/automations
|
||||||
|
- ✅ Only prevents **storage** in Recorder, not runtime availability
|
||||||
|
|
||||||
|
## References
|
||||||
|
|
||||||
|
- [HA Developer Docs - Excluding State Attributes](https://developers.home-assistant.io/docs/core/entity/#excluding-state-attributes-from-recorder-history)
|
||||||
|
- Implementation PR: [Link when merged]
|
||||||
|
- Related Issue: [Link if applicable]
|
||||||
|
|
@ -32,7 +32,7 @@ const sidebars: SidebarsConfig = {
|
||||||
{
|
{
|
||||||
type: 'category',
|
type: 'category',
|
||||||
label: '📐 Advanced Topics',
|
label: '📐 Advanced Topics',
|
||||||
items: ['period-calculation-theory', 'refactoring-guide', 'performance'],
|
items: ['period-calculation-theory', 'refactoring-guide', 'performance', 'recorder-optimization'],
|
||||||
collapsible: true,
|
collapsible: true,
|
||||||
collapsed: false,
|
collapsed: false,
|
||||||
},
|
},
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue