Major restructuring of the scripts/ directory with consistent output
formatting, improved organization, and stricter error handling.
Breaking Changes:
- Updated development environment to Home Assistant 2025.7+
- Removed Python 3.12 compatibility (HA 2025.7+ requires Python 3.13)
- Updated all HA core requirements from 2025.7 requirement files
- Added new dependencies: python-multipart, uv (for faster package management)
- Updated GitHub Actions workflows to use Python 3.13
Changes:
- Created centralized output library (scripts/.lib/output.sh)
- Unified color codes and Unicode symbols
- Consistent formatting functions (log_header, log_success, log_error, etc.)
- Support for embedded formatting codes (${BOLD}, ${GREEN}, etc.)
- Reorganized into logical subdirectories:
- scripts/setup/ - Setup and maintenance scripts
- bootstrap: Install/update dependencies (used in CI/CD)
- setup: Full DevContainer setup (pyright, copilot, HACS)
- reset: Reset config/ directory to fresh state (NEW)
- sync-hacs: Sync HACS integrations
- scripts/release/ - Release management scripts
- prepare: Version bump and tagging
- suggest-version: Semantic version suggestion
- generate-notes: Release notes generation
- check-if-released: Check release status
- hassfest: Local integration validation
- Updated all scripts with:
- set -euo pipefail for stricter error handling
- Consistent SCRIPT_DIR pattern for reliable sourcing
- Professional output with colors and emojis
- Unified styling across all 17 scripts
- Removed redundant scripts:
- scripts/update (was just wrapper around bootstrap)
- scripts/json_schemas/ (moved to schemas/json/)
- Enhanced clean script:
- Improved artifact cleanup
- Better handling of accidental package installations
- Hints for reset and deep clean options
- New reset script features:
- Standard mode: Keep configuration.yaml
- Full mode (--full): Reset configuration.yaml from git
- Automatic re-setup after reset
- Updated documentation:
- AGENTS.md: Updated script references and workflow guidance
- docs/development/: Updated all references to new script structure
Impact: Development environment now requires Python 3.13 and Home Assistant
2025.7+. Developers get consistent, professional script output with better
error handling and logical organization. Single source of truth for styling
makes future updates trivial.
- Introduced `get_intervals_for_day_offsets` helper to streamline access to price intervals for yesterday, today, and tomorrow.
- Updated various components to replace direct access to `priceInfo` with the new helper, ensuring a flat structure for price intervals.
- Adjusted calculations and data processing methods to accommodate the new data structure.
- Enhanced documentation to reflect changes in caching strategy and data structure.
Introduced TibberPricesMidnightHandler to prevent duplicate midnight
turnover when multiple timers fire simultaneously.
Problem: Timer #1 (API poll) and Timer #2 (quarter-hour refresh) both
wake at midnight, each detecting day change and triggering cache clear.
Race condition caused duplicate turnover operations.
Solution:
- Atomic flag coordination: First timer sets flag, subsequent timers skip
- Persistent state survives HA restart (cache stores last_turnover_time)
- Day-boundary detection: Compares current.date() vs last_check.date()
- 13 comprehensive tests covering race conditions and HA restart scenarios
Architecture:
- coordinator/midnight_handler.py: 165 lines, atomic coordination logic
- coordinator/core.py: Integrated handler in coordinator initialization
- coordinator/listeners.py: Delegate midnight check to handler
Impact: Eliminates duplicate cache clears at midnight. Single atomic
turnover operation regardless of how many timers fire simultaneously.
Periods can now naturally cross midnight boundaries, and new diagnostic
attributes help users understand price classification changes at midnight.
**New Features:**
1. Midnight-Crossing Period Support (relaxation.py):
- group_periods_by_day() assigns periods to ALL spanned days
- Periods crossing midnight appear in both yesterday and today
- Enables period formation across calendar day boundaries
- Ensures min_periods checking works correctly at midnight
2. Extended Price Data Window (relaxation.py):
- Period calculation now uses full 3-day data (yesterday+today+tomorrow)
- Enables natural period formation without artificial midnight cutoff
- Removed date filter that excluded yesterday's prices
3. Day Volatility Diagnostic Attributes (period_statistics.py, core.py):
- day_volatility_%: Daily price spread as percentage (span/avg × 100)
- day_price_min/max/span: Daily price range in minor currency (ct/øre)
- Helps detect when midnight classification changes are economically significant
- Uses period start day's reference prices for consistency
**Documentation:**
4. Design Principles (period-calculation-theory.md):
- Clarified per-day evaluation principle (always was the design)
- Added comprehensive section on midnight boundary handling
- Documented volatility threshold separation (sensor vs period filters)
- Explained market context for midnight price jumps (EPEX SPOT timing)
5. User Guides (period-calculation.md, automation-examples.md):
- Added \"Midnight Price Classification Changes\" troubleshooting section
- Provided automation examples using volatility attributes
- Explained why Best→Peak classification can change at midnight
- Documented level filter volatility threshold behavior
**Architecture:**
- Per-day evaluation: Each interval evaluated against its OWN day's min/max/avg
(not period start day) ensures mathematical correctness across midnight
- Period boundaries: Periods can naturally cross midnight but may split when
consecutive days differ significantly (intentional, mathematically correct)
- Volatility thresholds: Sensor thresholds (user-configurable) remain separate
from period filter thresholds (fixed internal) to prevent unexpected behavior
Impact: Periods crossing midnight are now consistently visible before and
after midnight turnover. Users can understand and handle edge cases where
price classification changes at midnight on low-volatility days.
BREAKING CHANGE: Period overlap resolution now merges adjacent/overlapping periods
instead of marking them as extensions. This simplifies automation logic and provides
clearer period boundaries for users.
Previous Behavior:
- Adjacent periods created by relaxation were marked with is_extension=true
- Multiple short periods instead of one continuous period
- Complex logic needed to determine actual period length in automations
New Behavior:
- Adjacent/overlapping periods are merged into single continuous periods
- Newer period's relaxation attributes override older period's
- Simpler automation: one period = one continuous time window
Changes:
- Period Overlap Resolution (new file: period_overlap.py):
* Added merge_adjacent_periods() to combine periods and preserve attributes
* Rewrote resolve_period_overlaps() with simplified merge logic
* Removed split_period_by_overlaps() (no longer needed)
* Removed is_extension marking logic
* Removed unused parameters: min_period_length, baseline_periods
- Relaxation Strategy (relaxation.py):
* Removed all is_extension filtering from period counting
* Simplified standalone counting to just len(periods)
* Changed from period_merging import to period_overlap import
* Added MAX_FLEX_HARD_LIMIT constant (0.50)
* Improved debug logging for merged periods
- Code Quality:
* Fixed all remaining linter errors (N806, PLR2004, PLR0912)
* Extracted magic values to module-level constants:
- FLEX_SCALING_THRESHOLD = 0.20
- SCALE_FACTOR_WARNING_THRESHOLD = 0.8
- MAX_FLEX_HARD_LIMIT = 0.50
* Added appropriate noqa comments for unavoidable patterns
- Configuration (from previous work in this session):
* Removed CONF_RELAXATION_STEP_BEST, CONF_RELAXATION_STEP_PEAK
* Hard-coded 3% relaxation increment for reliability
* Optimized defaults: RELAXATION_ATTEMPTS 8→11, ENABLE_MIN_PERIODS False→True,
MIN_PERIODS undefined→2
* Removed relaxation_step UI fields from config flow
* Updated all 5 translation files
- Documentation:
* Updated period_handlers/__init__.py: period_merging → period_overlap
* No user-facing docs changes needed (already described continuous periods)
Rationale - Period Merging:
User experience was complicated by fragmented periods:
- Automations had to check multiple adjacent periods
- Binary sensors showed ON/OFF transitions within same cheap time
- No clear way to determine actual continuous period length
With merging:
- One continuous cheap time = one period
- Binary sensor clearly ON during entire period
- Attributes show merge history via merged_from dict
- Relaxation info preserved from newest/highest flex period
Rationale - Hard-Coded Relaxation Increment:
The configurable relaxation_step parameter proved problematic:
- High base flex + high step → rapid explosion (40% base + 10% step → 100% in 6 steps)
- Users don't understand the multiplicative nature
- 3% increment provides optimal balance: 11 attempts to reach 50% hard cap
Impact:
- Existing installations: Periods may appear longer (merged instead of split)
- Automations benefit from simpler logic (no is_extension checks needed)
- Custom relaxation_step values will use new 3% increment
- Users may need to adjust relaxation_attempts if they relied on high step sizes
This commit completes multiple refactoring efforts and documentation improvements:
Code Structure Changes:
- Move round_to_nearest_quarter_hour() from sensor/helpers.py to average_utils.py
- Resolve circular import between price_utils.py and sensor/helpers.py
- Split api.py into api/ package (client.py, queries.py, exceptions.py, helpers.py)
- Split coordinator.py into coordinator/ package (core.py, cache.py, listeners.py, etc.)
- Move period_utils/ to coordinator/period_handlers/ for better organization
- All lint checks passing (no PLC0415 local import warnings)
Documentation Additions:
- Add docs/development/architecture.md with Mermaid diagrams (end-to-end flow, cache coordination)
- Add docs/development/timer-architecture.md (comprehensive 3-timer system documentation)
- Add docs/development/caching-strategy.md (4-layer cache system with invalidation logic)
- Update docs/development/README.md with cross-references
- Update AGENTS.md with new module structure and patterns
Smart Boundary Tolerance:
- Implement ±2 second tolerance for quarter-hour rounding
- Prevents premature interval switching during HA restarts (14:59:30 stays at 14:45)
- Enables boundary snapping for timer jitter (14:59:58 → 15:00)
Atomic Midnight Coordination:
- Add _check_midnight_turnover_needed() for race-free midnight handling
- Coordinate Timer #1 (HA DataUpdateCoordinator) with Timer #2 (quarter-hour refresh)
- Whoever runs first performs turnover, other skips gracefully
Timer Optimization:
- Change timer scheduling from second=1 to second=0 (absolute-time scheduling)
- Document load distribution rationale (unsynchronized API polling prevents thundering herd)
- Comprehensive explanation of 3 independent timers and their coordination
Impact: Cleaner code structure with resolved circular dependencies, comprehensive
documentation of timer and caching systems, and improved reliability during
boundary conditions and midnight turnovers. All changes are developer-facing
improvements with no user-visible behavior changes.
Home Assistant's hassfest validation requires config flows to be defined
in a file named config_flow.py (not a package directory).
Changes:
- Renamed custom_components/tibber_prices/config_flow/ → config_flow_handlers/
- Created config_flow.py as bridge file re-exporting from config_flow_handlers/
- Updated all import paths across 5 files (user_flow, options_flow, subentry_flow, etc.)
- Added ./scripts/hassfest for local validation (JSON/Python syntax, required files)
- Added ./scripts/clean with three modes (--minimal, normal, --deep)
- Refactored develop/lint/lint-check to use centralized cleanup (DRY principle)
- Updated documentation in AGENTS.md and docs/development/
Technical details:
- Bridge file uses __all__ exports to maintain clean public API
- hassfest script uses ast.parse() for syntax validation (no disk artifacts)
- clean --minimal removes .egg-info only (silent, for automated scripts)
- Dual pip/uv pip compatibility for package uninstallation
Impact: Integration now passes hassfest validation. Local validation available
via ./scripts/hassfest before pushing to GitHub. Cleanup logic centralized and
DRY across all development scripts.
- Introduced a comprehensive refactoring guide detailing when and how to plan major refactorings, including a structured planning process and real-world examples.
- Created a README for the planning directory, outlining its purpose, document lifecycle, and best practices for creating planning documents.
Created professional documentation structure:
**User Documentation (docs/user/):**
- README.md: Documentation hub with quick start guide
- Placeholder files for future content migration:
* installation.md, configuration.md, sensors.md
* services.md, automation-examples.md, troubleshooting.md
**Developer Documentation (docs/development/):**
- README.md: Comprehensive contributor guide with AI section
- setup.md: DevContainer and environment setup
- architecture.md: Code structure overview
- testing.md: Testing guidelines
- coding-guidelines.md: Style guide and critical patterns
- release-management.md: Complete release workflow documentation
**AI Development Disclosure:**
- README.md: "🤖 Development Note" section before license
* Honest disclosure about extensive AI assistance
* Quality assurance measures mentioned
* Invitation for bug reports with positive tone
- docs/development/README.md: Detailed AI section
* What AI handles (patterns, generation, refactoring)
* Benefits (rapid development, consistency)
* Limitations (edge cases, complex patterns)
* Quality assurance process
- CONTRIBUTING.md: Brief AI note with practical tip
**Updated:**
- README.md: Simplified to landing page with documentation links
- CONTRIBUTING.md: Modernized with new docs structure
- copilot-instructions.md: Added documentation organization section
Impact: Clear separation of user vs. developer documentation following
open-source best practices. Transparent about AI-assisted development
approach without being defensive. Scalable structure for future growth.
