[SECURITY] Automated DB backup/restore + data corruption detection - closes #62 #63

[Ibrahim-3d]

Summary

• [SECURITY] Create automated backup and restore procedures #62 — pt_backup.py: DatabaseBackupManager with SHA-256 verified SQLite backups, WAL checkpoint before backup, PRAGMA integrity_check, atomic point-in-time restore, configurable retention pruning, and background scheduler
• [SECURITY] Add comprehensive data validation layers #63 — pt_validation.py: DataIntegrityValidator added — NaN/Inf scanning, OHLCV cross-field consistency (high≥low, close in range), Z-score price spike detection, SHA-256 data checksums, batch integrity scanning

Changes

• New: app/pt_backup.py
• Modified: app/pt_validation.py — DataIntegrityValidator class added (no breaking changes)
• New: app/test_backup_validation.py — 35 tests

Usage

# Backup
from pt_backup import DatabaseBackupManager
mgr = DatabaseBackupManager("order_management.db", max_backups=30)
mgr.start_scheduler()  # auto-backup every 24h

# Manual + restore
record = mgr.create_backup()
mgr.restore(record.backup_id)

# Data integrity
from pt_validation import DataIntegrityValidator
violations = DataIntegrityValidator.check_ohlcv_consistency(candle)
result = DataIntegrityValidator.check_batch_integrity(records, ["price", "volume"])

Test plan

• 35 unit tests pass
• Backup checksum tamper detection tested
• Restore atomicity verified (tmp → rename)
• Retention pruning verified
• OHLCV violations all caught
• NaN/Inf detection across all numeric types

Closes #62
Closes #63

[sjackson0109]

Another outstanding contribution from Ibrahim.

[Ibrahim-3d]

Local CI Simulation Results

Ran all workflow steps locally against this branch prior to owner approval. Results:

Code Quality & Testing

Check	Tool	Result
Code formatting	black --check	✅ Pass — 0 files need reformatting
Import ordering	isort --check	✅ Pass
Linting (hard errors)	flake8 --select=E9,F63,F7,F82	✅ Pass — 0 syntax/undefined-name errors
Linting (extended)	flake8 --exit-zero	✅ Pass — 0 warnings in new files
Unit tests	pytest	✅ All tests pass (see table below)

Unit Test Results

File	Tests	Result
test_circuit_breaker.py	20	✅ All pass
test_credentials_rotation.py	30	✅ All pass
test_error_handler.py	22	✅ All pass
test_backup_validation.py	35	✅ All pass
test_database_manager.py	29	✅ All pass
test_security_logger.py	25	✅ All pass

CI/CD Pipeline

Workflow	Status	Reason
Code Quality & Testing	⏳ Awaiting owner approval	Fork PR — first-time contributor gate
PowerTrader AI+ CI/CD	⏳ Awaiting owner approval	Fork PR — first-time contributor gate
Project Management	⏳ Awaiting owner approval	Fork PR — first-time contributor gate

All three workflows are queued with action_required status. This is GitHub's security gate for PRs from fork contributors. Once approved by @sjackson0109, they will execute against the same code that was verified locally above.

[sjackson0109]

@Ibrahim-3d can you review the pipeline failures and refactor to re-try submission?

[Ibrahim-3d]

Pipeline failure investigated and fixed. The CI job was failing on Black formatting — specifically (the file that had a merge conflict during the rebase onto sjackson0109's Black-reformatted main). The conflict was resolved correctly for logic but the file was left unformatted. Fixed: ran Black on and committed. All other 97 files were already compliant — Black reported '1 file would be reformatted, 97 files would be left unchanged'.

[Ibrahim-3d]

Manual Testing Guide — PR #82: Automated DB Backup/Restore + Corruption Detection

Prerequisites

git fetch fork && git checkout feat/62-63-database-backup-validation
cd app

1. Run unit tests

python -m pytest test_backup_validation.py -v

Expected: All 35 tests pass.

2. CI Black formatting check (this was the pipeline failure)

uvx black --check app/pt_validation.py
# or: black --check app/pt_validation.py

Expected: 1 file would be left unchanged — no reformatting needed.

3. Smoke test — backup and restore

python -c "
import tempfile, os, sqlite3
from pt_backup_validation import DatabaseBackupManager  # adjust import if needed

with tempfile.TemporaryDirectory() as d:
    db = os.path.join(d, 'trade.db')
    backup = os.path.join(d, 'trade.db.bak')
    conn = sqlite3.connect(db)
    conn.execute('CREATE TABLE orders (id INTEGER PRIMARY KEY, symbol TEXT)')
    conn.execute(\"INSERT INTO orders VALUES (1, 'BTC-USD')\")
    conn.commit()
    conn.close()
    print('DB created. Run backup + restore manually via BackupManager.')
    print('Verify: backup file exists, restore reproduces the row.')
"

4. Verify data integrity checksum detects corruption

Refer to test_backup_validation.py::TestDataIntegrityValidator — run those tests individually to confirm hash mismatch detection works on your filesystem.

Rollback

git checkout main -- app/pt_validation.py

[Ibrahim-3d]

/gemini review

[Ibrahim-3d]

Hold this one for me please - I want to walk back some unintended changes first. While building the new validator I also stripped docstrings and inline comments off the existing InputValidator methods, which wasn't part of the issue scope and just adds noise to the review. Going to restore those and keep this PR limited to the actual backup + integrity work. Back shortly.

[Ibrahim-3d]

Done. I rebuilt pt_validation.py from main so every original docstring and inline comment in InputValidator is back the way it was, then re-applied only the additions this issue actually needs (DataCorruptionError, DataIntegrityValidator, plus the hashlib/math/Tuple imports it needs). Diff is now focused on just the new stuff. 35 tests passing.

[Ibrahim-3d]

@copilot review