From d7297174f90182284d3b5fade7d6d09f579e97f1 Mon Sep 17 00:00:00 2001 From: Julian Pawlowski Date: Mon, 6 Apr 2026 12:18:59 +0000 Subject: [PATCH] docs(period-calculation): document flat-day behavior and diagnostic attributes User docs (period-calculation.md): - New troubleshooting section "Fewer Periods Than Configured" (most common confusing scenario, added before "No Periods Found") - Extended "Understanding Sensor Attributes" section: all four diagnostic attributes documented with concrete YAML examples and prose explanations - Updated Table of Contents Developer docs (period-calculation-theory.md): - New section "Flat Day and Low-Price Adaptations" (between Flex Limits and Relaxation Strategy) documenting all three mechanisms with implementation details, scaling tables, and rationale for hardcoding - Two new scenarios: 2b (flat day with adaptive min_periods) and 2c (solar surplus / absolute low-price day) with expected logs and sensor attribute examples - Debugging checklist point 6: flat day / low-price checks with exact log string patterns to grep for - Diagnostic attribute reference table in the debugging section Impact: Users can self-diagnose "fewer periods than configured" without support. Contributors understand why the three thresholds are hardcoded and cannot be user-configured. --- .../docs/period-calculation-theory.md | 147 ++++++++++++++++++ docs/user/docs/period-calculation.md | 90 +++++++++++ 2 files changed, 237 insertions(+) diff --git a/docs/developer/docs/period-calculation-theory.md b/docs/developer/docs/period-calculation-theory.md index 66cd87b..5dcda87 100644 --- a/docs/developer/docs/period-calculation-theory.md +++ b/docs/developer/docs/period-calculation-theory.md @@ -384,6 +384,81 @@ Recommendation: Use 15-20% with relaxation enabled, or 25-35% without relaxation --- +## Flat Day and Low-Price Adaptations + +These three mechanisms handle pathological price situations where standard filters produce misleading or impossible results. All are hardcoded (not user-configurable) because they correct mathematical defects rather than expressing user preferences. + +### 1. Adaptive min_periods for Flat Days + +**File:** `coordinator/period_handlers/relaxation.py` → `_compute_day_effective_min()` + +**Trigger:** Coefficient of Variation (CV) of a day's prices ≤ `LOW_CV_FLAT_DAY_THRESHOLD` (10%) + +**Problem:** When all prices are nearly identical (e.g. 28–32 ct, CV=5.4%), requiring 2 distinct "best price" windows is geometrically impossible. Even after exhausting all 11 relaxation phases, only 1 period exists because there is no second cheap cluster. + +**Solution:** Before the baseline counting loop, compute per-day effective min_periods: +```python +if day_cv <= 10%: + day_effective_min[day] = 1 # Flat day: 1 period is enough +else: + day_effective_min[day] = min_periods # Normal day: respect config +``` + +**Scope:** Best Price only (`reverse_sort=False`). Peak Price always runs full relaxation because the genuinely most expensive window must be identified even on flat days. + +**Transparency:** The count of flat days is stored in `metadata["relaxation"]["flat_days_detected"]` and surfaced as a sensor attribute. + +### 2. min_distance Scaling on Absolute Low-Price Days + +**File:** `coordinator/period_handlers/level_filtering.py` → `check_interval_criteria()` + +**Trigger:** `criteria.avg_price < LOW_PRICE_AVG_THRESHOLD` (0.10 EUR) + +**Problem:** On solar surplus days (avg 2–5 ct/kWh), a percentage-based min_distance like 5% means only 0.1 ct absolute separation is required. The filter either accepts almost the entire day (if ref_price is 2 ct, 5% = 0.1 ct – nearly everything qualifies) or blocks everything (if the spread is within that 0.1 ct band). + +**Solution:** Linear scaling toward zero as avg_price approaches zero: +``` +scale_factor = avg_price / LOW_PRICE_AVG_THRESHOLD +adjusted_min_distance = original_min_distance × scale_factor +``` + +| avg_price | scale | Effect on 5% min_distance | +|---|---|---| +| ≥ 10 ct (0.10 EUR) | 100% | 5% (full distance) | +| 5 ct (0.05 EUR) | 50% | 2.5% | +| 2 ct (0.02 EUR) | 20% | 1% | +| 0 ct | 0% | 0% (disabled) | + +### 3. CV Quality Gate Bypass for Absolute Low-Price Periods + +**File:** `coordinator/period_handlers/relaxation.py` → `_check_period_quality()`, and `coordinator/period_handlers/period_overlap.py` → `_check_merge_quality_gate()` + +**Trigger:** Period mean price < `LOW_PRICE_QUALITY_BYPASS_THRESHOLD` (0.10 EUR) + +**Problem:** A period at 0.5–4 ct has high *relative* variation (CV ≈ 70–80%), but the absolute differences are fractions of a cent. The quality gate (CV ≤ `PERIOD_MAX_CV`) with a relative metric would wrongly reject this as a "heterogeneous" period. + +**Distinguishes from flat normal days:** A flat day at 33–36 ct also has low absolute range, but mean is 34.5 ct (>> 0.10 EUR threshold). The bypass only applies when the mean itself is below the threshold – i.e. the day is genuinely cheap in absolute terms. + +**Solution:** Short-circuit the quality gate check: +```python +period_mean = sum(period_prices) / len(period_prices) +if period_mean < LOW_PRICE_QUALITY_BYPASS_THRESHOLD: + return True, cv # Bypass: absolute price level is very low +passes = cv <= PERIOD_MAX_CV +``` + +The same bypass is applied in `_check_merge_quality_gate()` using the combined period's mean price. + +### Why These Are Hardcoded + +All three thresholds are internal correctness guards, not user preferences: + +1. **Currency-relative:** 0.10 EUR works across EUR, NOK, SEK etc. because Tibber prices in those currencies have similar nominal order of magnitude. A user setting this would need to understand currency-denominated physics. +2. **Mathematical necessity:** Disabling them reverts to pre-fix behavior (worst case: 0 periods on V-shape days). +3. **No valid configuration reason:** There is no scenario where a user benefits from "I want the CV gate to reject my 2 ct period" or "I want the full min_distance on a 3 ct average day". + +--- + ## Relaxation Strategy ### Purpose @@ -737,6 +812,59 @@ DEBUG: After build_periods: 1 raw period found DEBUG: Day 2025-11-11: Baseline insufficient (1 < 2), starting relaxation ``` +### Scenario 2b: Flat Day with Adaptive min_periods + +**Price Range:** 28 - 32 ct/kWh (CV ≈ 5.4%, very flat) +**Average:** 30 ct/kWh, Min: 28.5 ct/kWh +**Configuration:** `min_periods_best: 2`, relaxation enabled + +**Problem without adaptation:** +Relaxation would exhaust all 11 phases trying to find a second period. All prices are equally cheap – finding two distinct windows is geometrically impossible. + +**Implemented behavior:** +`_compute_day_effective_min()` detects CV ≤ 10% and sets `day_effective_min = 1` for this day. The result is accepted after finding the single cheapest cluster. + +**Expected Logs:** +``` +DEBUG: Day 2025-11-11: flat price profile (CV=5.4% ≤ 10.0%) → min_periods relaxed to 1 +INFO: Adaptive min_periods: 1 flat day(s) (CV ≤ 10%) need only 1 period instead of 2 +INFO: Day 2025-11-11: Baseline satisfied (1 period, effective minimum is 1) +``` + +**Sensor Attributes:** +```yaml +min_periods_configured: 2 # User's setting +periods_found_total: 1 # Actual result +flat_days_detected: 1 # Explains the difference +``` + +**Why not for Peak Price?** +Peak price always runs full relaxation. On a flat day, the integration still needs to identify the genuinely most expensive window (even if the absolute difference is small). Skipping relaxation would mean accepting any arbitrarily-chosen interval as "peak". + +### Scenario 2c: Absolute Low-Price Day (e.g. Solar Surplus) + +**Price Range:** 0.5 - 4.2 ct/kWh (V-shape from solar generation) +**Average:** 2.1 ct/kWh, Min: 0.5 ct/kWh +**Configuration:** `min_periods_best: 2`, 5% min_distance + +**Problems without fixes:** +1. **min_distance conflict:** 5% of 2.1 ct = 0.105 ct minimum distance. Only prices ≤ 1.995 ct qualify. The daily minimum is 0.5 ct – well within range. But the *relative* threshold becomes meaninglessly tiny: the entire day could qualify. +2. **CV quality gate:** Prices 0.5–4.2 ct show high relative variation (CV ≈ 70-80%), but the absolute differences are fractions of a cent. The quality gate would wrongly reject valid periods. + +**Implemented behavior:** + +*`LOW_PRICE_AVG_THRESHOLD = 0.10 EUR` (level_filtering.py):* +When `avg_price < 0.10 EUR`, min_distance is scaled linearly to 0. At avg=2.1 ct (0.021 EUR), scale ≈ 21% → min_distance effectively 1%. Prevents the distance filter from blocking the entire day or accepting the entire day. + +*`LOW_PRICE_QUALITY_BYPASS_THRESHOLD = 0.10 EUR` (relaxation.py):* +When period mean < 0.10 EUR, the CV quality gate is bypassed entirely. A period at 0.5–2 ct with CV=60% is practically homogeneous from a cost perspective. + +**Expected Logs:** +``` +DEBUG: Low-price day (avg=0.021 EUR < 0.10 threshold): min_distance scaled 5% → 1.1% +DEBUG: Period 02:00-05:00: mean=0.009 EUR < bypass threshold → quality gate bypassed +``` + ### Scenario 3: Extreme Day (High Volatility) **Price Range:** 5 - 40 ct/kWh (700% variation) @@ -816,6 +944,25 @@ When debugging period calculation issues: - Check `period_interval_smoothed_count` attribute - If no smoothing but periods fragmented: Not isolated spikes, but legitimate price levels +6. **Check Flat Day / Low-Price Adaptations** + - Sensor shows `flat_days_detected: N` → CV ≤ 10%, adaptive min_periods reduced target to 1 + - Sensor shows `relaxation_incomplete: true` without `flat_days_detected` → check filter settings + - Low absolute prices (avg < 10 ct): min_distance is auto-scaled, CV quality gate is bypassed + - To confirm: Enable DEBUG logging for `custom_components.tibber_prices.coordinator.period_handlers.relaxation.details` + - Look for: `"flat price profile (CV=X.X% ≤ 10.0%) → min_periods relaxed to 1"` + - Look for: `"Low-price day (avg=0.0XX EUR < 0.10 threshold): min_distance scaled X% → Y%"` + +**Diagnostic Sensor Attributes Summary:** + +| Attribute | Type | When shown | Meaning | +|---|---|---|---| +| `min_periods_configured` | int | Always | User's configured target per day | +| `periods_found_total` | int | Always | Actual periods found across all days | +| `flat_days_detected` | int | Only when > 0 | Days where CV ≤ 10% reduced target to 1 | +| `relaxation_incomplete` | bool | Only when true | Relaxation exhausted, target not reached | +| `relaxation_active` | bool | Only when true | This specific period needed relaxed filters | +| `relaxation_level` | string | Only when active | Flex% and filter combo that succeeded | + --- ## Future Enhancements diff --git a/docs/user/docs/period-calculation.md b/docs/user/docs/period-calculation.md index 3e63824..57480f0 100644 --- a/docs/user/docs/period-calculation.md +++ b/docs/user/docs/period-calculation.md @@ -12,10 +12,13 @@ Learn how Best Price and Peak Price periods work, and how to configure them for - [Understanding Relaxation](#understanding-relaxation) - [Common Scenarios](#common-scenarios) - [Troubleshooting](#troubleshooting) + - [Fewer Periods Than Configured](#fewer-periods-than-configured) - [No Periods Found](#no-periods-found) - [Periods Split Into Small Pieces](#periods-split-into-small-pieces) + - [Understanding Sensor Attributes](#understanding-sensor-attributes) - [Midnight Price Classification Changes](#midnight-price-classification-changes) - [Advanced Topics](#advanced-topics) +- [Advanced Topics](#advanced-topics) --- @@ -459,6 +462,40 @@ automation: ## Troubleshooting +### Fewer Periods Than Configured + +**Symptom:** You configured `min_periods_best: 2` but the sensor shows fewer periods on some days, and the attributes contain `flat_days_detected: 1` or `relaxation_incomplete: true`. + +**If `flat_days_detected` is present:** + +This is **expected behavior** on days with very uniform electricity prices. When prices vary by less than ~10% across the day (e.g. on sunny spring days with high solar generation), there is no meaningful second "cheap window" – all hours are equally cheap. The integration automatically reduces the target to 1 period for that day. + +```yaml +min_periods_configured: 2 +periods_found_total: 1 +flat_days_detected: 1 # Uniform prices today → 1 period is the right answer +``` + +You don't need to change anything. This is the integration protecting you from artificial periods. + +**If `relaxation_incomplete` is present (without `flat_days_detected`):** + +Relaxation tried all configured attempts but couldn't reach your target. Options: + +1. **Increase relaxation attempts** (tries more flexibility levels before giving up) + ```yaml + relaxation_attempts_best: 12 # Default: 11 + ``` + +2. **Reduce minimum period count** + ```yaml + min_periods_best: 1 # Only require 1 period per day + ``` + +3. **Check filter settings** – very strict `best_price_min_distance_from_avg` values block relaxation + +--- + ### No Periods Found **Symptom:** `binary_sensor._best_price_period` never turns "on" @@ -526,11 +563,64 @@ rating_level: "LOW" # All intervals have LOW rating relaxation_active: true # This day needed relaxation relaxation_level: "price_diff_18.0%+level_any" # Found at 18% flex, level filter removed +# Calculation summary (always shown – diagnostic overview of this calculation run): +min_periods_configured: 2 # What you configured as target +periods_found_total: 3 # What was actually found across all days + # Optional (only shown when relevant): period_interval_smoothed_count: 2 # Number of price spikes smoothed period_interval_level_gap_count: 1 # Number of "mediocre" intervals tolerated +flat_days_detected: 1 # Days where prices were so flat that 1 period is enough +relaxation_incomplete: true # Some days couldn't reach the configured target ``` +#### What the diagnostic attributes mean + +**`min_periods_configured` / `periods_found_total`** + +These two values together quickly show whether the calculation achieved its goal: + +```yaml +min_periods_configured: 2 # You asked for 2 periods per day +periods_found_total: 6 # 3 days × 2 periods = fully satisfied ✅ +``` + +```yaml +min_periods_configured: 2 +periods_found_total: 5 # 3 days, but one day got only 1 period +``` + +Note that `periods_found_total` counts **all periods across today and tomorrow** – so 4 on a two-day view means 2 per day on average. + +**`flat_days_detected`** + +This is the most important diagnostic for days with very uniform prices (e.g. sunny spring/summer days with high solar generation): + +```yaml +min_periods_configured: 2 +periods_found_total: 1 +flat_days_detected: 1 # ← This explains why you got 1 instead of 2 +``` + +When prices barely change across the day – typically a variation of less than 10% – the integration automatically reduces the target from your configured value to 1. There is no meaningful second "best price window" when all prices are essentially equal. + +**This is expected and correct behavior**, not a problem. It prevents the sensor from generating artificial periods that don't represent genuinely cheaper windows. + +**`relaxation_incomplete`** + +This flag appears when even after all relaxation attempts, at least one day could not reach the configured minimum number of periods: + +```yaml +min_periods_configured: 2 +periods_found_total: 1 +relaxation_incomplete: true # ← Relaxation tried everything, still short +``` + +This is most common on very flat days (see above) or with very strict filter settings. If you see this repeatedly on normal days, consider: +- Reducing `min_periods_best` to 1 +- Increasing `relaxation_attempts_best` +- Checking if your `best_price_min_distance_from_avg` is too high + ### Midnight Price Classification Changes **Symptom:** A Best Price period at 23:45 suddenly changes to Peak Price at 00:00 (or vice versa), even though the absolute price barely changed.