# WeitmanHouse Homelab & Cloud Infrastructure Progress Journal

*Public Edition | Updated: October 13, 2026*  
*Baseline Document Revision: Private Master 0.11.1*  
*Archive Navigation: [Checkpoint Index](archive/index.md) | [Initial Release (2026-10-13)](archive/2026-10-13.md)*

---

## 1. Project Overview: What We Are Building and Why

The WeitmanHouse infrastructure project is a resilient, dual-tier personal computing and homelab environment designed to balance high-capacity on-premises media processing with reliable, independent off-site infrastructure monitoring.

Self-hosting complex workloads like media streaming, automated transcoding pipelines, and application services requires robust architecture. However, hosting everything within a single physical location introduces single points of failure: if local power, internet connectivity, or host hardware experiences an outage, internal monitoring tools go down along with the services they monitor.

To address this, our architecture decouples high-throughput local operations from cloud-based availability monitoring:

1. **On-Premises Infrastructure (Lenovo ThinkStation P520)**: A workstation-class server running Ubuntu Server LTS, equipped with an Intel Xeon processor, 128 GB of ECC memory, pooled multi-terabyte storage, and dual dedicated NVIDIA Quadro workstation GPUs. This host handles containerized media services, automated storage management, and hardware-accelerated video processing.
2. **External Cloud Monitoring (Multi-Cloud Free Tier)**: Independent, lightweight cloud compute nodes deployed across Google Cloud Platform (GCP) and Oracle Cloud Infrastructure (OCI). These instances run out-of-band telemetry (Uptime Kuma) and secure ingress tunnels, monitoring home services from the outside without relying on local network uptime.
3. **Hybrid Secure Access**: Zero-trust remote access built on Cloudflare Tunnels and Access for public telemetry endpoints, combined with an authenticated UniFi WireGuard VPN backbone for secure administrative access and private site-to-cloud telemetry.

---

## 2. Infrastructure Status Summary

To ensure operational transparency while maintaining rigorous standards of verification, all work across the environment is tracked under three explicit states:

- **Completed**: Fully implemented, tested, and validated against independent system probes and logs.
- **In Progress / Planned**: Actively architected and sequenced, but pending staging, testing, or final cutover.
- **Unverified / Paused**: Work suspended for architectural review, safety verification, or awaiting hardware validation.

| Workstream | Subsystem | Current State | Operational Summary |
|---|---|---|---|
| **Host System** | Compute & OS | **Completed** | Ubuntu Server LTS on Lenovo P520; 128 GB RAM; stable multi-day uptime. |
| **Storage** | Pooled Storage | **Completed** | Multi-drive ext4 pool managed via mergerfs with path-preserving allocation policy. |
| **GPU Compute** | Hardware Allocation | **Completed** | Dual Quadro P1000 GPUs partitioned: GPU 0 for media playback; GPU 1 dedicated to batch transcoding. |
| **Administration** | Management Workspace | **Completed** | Antigravity CLI environment configured with multi-pane tmux workspace and strict single-master operational context. |
| **Video Transcoding** | Tdarr Pilot | **Completed** | Isolated 5-file sample test completed with a verified ~30% space reduction and full stream integrity. |
| **Video Transcoding** | Production Pipeline | **Paused** | Transcoding suspended pending full Flow safety review, font attachment preservation audit, and hardlink protections. |
| **Cloud Monitoring** | GCP Uptime Kuma | **Completed** | Host recovered from memory pressure; application-consistent SQLite backup verified and secured off-site. |
| **Cloud Monitoring** | OCI Micro Instance | **Completed** | Always Free compute instance provisioned in Toronto; 2 GiB swap space configured and validated. |
| **Cloud Monitoring** | OCI Container Migration | **Planned / Next (Not Yet Completed)** | Docker runtime installation, pinned Uptime Kuma 1.23.17 deployment, and database restoration staged for execution. |
| **Secure Networking** | Hybrid Cloud Ingress | **Planned (Not Yet Completed)** | Cloudflare Tunnel and Access for dashboard; UniFi WireGuard tunnel for private telemetry probes. |

---

## 3. Completed Infrastructure Milestones

### Workstation Host & Storage Architecture
- **Operating System Stabilization**: Standardized the Lenovo P520 platform on Ubuntu Server LTS with long-term kernel support, ensuring full compatibility with NVIDIA container drivers and mergerfs pooling.
- **Storage Pool Implementation**: Configured multiple high-capacity ext4 storage disks unified through a mergerfs pool. Applied a path-preserving, free-space based (`pfrd`) creation policy, ensuring media series folders remain grouped on specific underlying disks while spreading newly added files across available volume capacity.
- **Dual Workstation GPU Segmentation**:
  - Assigned dedicated GPU 0 to real-time media playback streaming (Plex/Jellyfin), protecting interactive user streams from GPU starvation.
  - Isolated GPU 1 for background batch encoding workloads, ensuring long-running transcode tasks run entirely out-of-band from user-facing media sessions.

### Administration Workspace & Operational Governance
- **Operational Tooling Modernization**: Transitioned homelab management to the Antigravity CLI environment. Installed dedicated multi-pane tmux workspace helpers, verified rollbacks, and decommissioned legacy administrative tooling.
- **Canonical Master Documentation**: Established a single authoritative operational master context (`P520-Master.md`). Every change requires read-only inspection, written execution plans, explicit administrator review, and post-execution verification before documentation promotion.

### Isolated Transcoding Pilot
- **Sample Pilot Verification**: Deployed an isolated containerized Tdarr pilot to evaluate H.264 to H.265 (HEVC) compression efficiency on representative 1080p anime and television files.
- **Space & Quality Metrics**: Reclaimed approximately 30% storage space across the pilot batch (reducing file volume from ~20.1 GB to 14.0 GB) while verifying 100% video stream integrity, multiple audio track alignment, and chapter marker retention.

### Cloud Monitoring Recovery & Database Preservation
- **Google Cloud VM Recovery**: Successfully recovered the remote Uptime Kuma monitoring host (`weitman-watch-01`) following acute memory pressure and journal logging exhaustion.
- **Application-Consistent SQLite Backup**: Executed an online point-in-time database backup using SQLite's native backup API, capturing all configured health monitors, notification hooks, custom status pages, and over 172,000 historical heartbeat records without service interruption. Verified backup integrity via SHA-256 checksums.

### Oracle Cloud Provisioning & Swap Hardening
- **Always Free Instance Provisioning**: Secured an Always Free compute node (`VM.Standard.E2.1.Micro`) in Oracle Cloud's Toronto data center (`ca-toronto-1`) running Ubuntu 24.04 LTS to serve as the new primary out-of-band monitoring node.
- **Swap Space Implementation**: Configured and enabled a permanent 2 GiB swap file persisted in the system fstab. This eliminates the out-of-memory kernel panics historically experienced on 1 GB micro-tier instances during burst container activity.
- **Security Hardening**: Locked down Oracle Cloud Virtual Cloud Network (VCN) security lists, restricting remote administrative SSH ingress exclusively to verified operator IP addresses and disabling public exposure of internal container ports.

---

## 4. Problems Encountered and Lessons Learned

### 1. Memory Exhaustion on Micro Cloud Instances
- **Problem**: Micro-tier cloud virtual machines (typically provisioned with ~1 GB of physical RAM) are susceptible to kernel out-of-memory (OOM) kills when managing containerized services, automated log collection, and system upgrades simultaneously. The Google Cloud instance previously suffered journald instability when log volume spiked during high memory usage.
- **Lesson Learned**: Relying on physical RAM alone on free-tier micro instances is an operational anti-pattern. Every minimal cloud instance must be hardened with a properly sized swap file (2 GiB) and strict Docker log rotation policies (`max-size` and `max-file`) before deploying production workloads.

### 2. Media Transcoding: Font Attachments and Metadata Safety
- **Problem**: In media files containing stylized subtitles (such as anime MKV containers), subtitle tracks often rely on custom embedded font attachments (`.ttf`, `.otf`). During pipeline testing with automated multi-stage transcode flows, standard stream-reordering utilities stripped five embedded font attachments from temporary output files. Additionally, automated workflows attempted in-place metadata modifications on source files.
- **Lesson Learned**: Automated transcode flows must never write directly to source media in place. Production transcode pipelines require a strict preservation checklist:
  1. All subtitle streams, audio languages, and font attachments must be explicitly copied and verified post-transcode.
  2. Temporary files must reside on dedicated scratch volumes until final quality checks pass.
  3. Files actively linked to torrent clients for community seeding must be detected via hardlink link-count checks and protected from replacement to prevent corrupting seeding ratios.

### 3. SQLite Online Backups Under Write-Ahead Logging (WAL)
- **Problem**: Applications utilizing SQLite with Write-Ahead Logging active write transactions to separate `.wal` and `.shm` files. A simple file copy while the database is open can capture an inconsistent database snapshot or miss recent heartbeat transactions.
- **Lesson Learned**: Backups must be performed using the native SQLite online backup API (`sqlite3 .backup`), which ensures clean WAL checkpoint synchronization and delivers an immutable, transactionally consistent snapshot file.

### 4. Zero-Trust Access vs. Direct Container Port Exposure
- **Problem**: Exposing application web dashboards directly to the internet via port forwards creates unnecessary attack surfaces and exposes administrative login interfaces to brute-force scans.
- **Lesson Learned**: Cloud-based services should remain bound to local loopback interfaces or private container bridges. Ingress should be mediated exclusively through encrypted tunnels (Cloudflare Tunnel) coupled with identity provider policies (Cloudflare Access), ensuring that only authenticated devices can reach management consoles.

---

## 5. Architectural Decisions and Technical Reasoning

```mermaid
graph TD
    subgraph "External Cloud (Oracle Cloud / OCI)"
        OCI_Kuma["Uptime Kuma (Pinned v1.23.17)"]
        OCI_Swap["2 GiB Swap Hardening"]
        OCI_Tunnel["Cloudflare Tunnel (Zero-Trust)"]
        OCI_WG["WireGuard VPN Client"]
        OCI_Kuma --- OCI_Swap
        OCI_Tunnel --> OCI_Kuma
    end

    subgraph "Public Internet"
        CF["Cloudflare Edge / Access"]
        AdminUser["Authenticated Operator"]
        PublicUser["Public Status Viewer"]
        AdminUser --> CF
        PublicUser --> CF
        CF --> OCI_Tunnel
    end

    subgraph "WeitmanHouse Homelab (Lenovo P520)"
        UniFi["UniFi Gateway / WireGuard Server"]
        HostOS["Ubuntu Server LTS (128GB RAM)"]
        StoragePool["MergerFS Storage Pool (Multi-Drive)"]
        GPU0["Quadro P1000 #0 (Streaming)"]
        GPU1["Quadro P1000 #1 (Transcoding)"]
        MediaServices["Media Applications (Plex/Jellyfin)"]
        TranscodeService["Tdarr Pipeline (Paused for Flow Audit)"]
        
        HostOS --- StoragePool
        HostOS --- GPU0
        HostOS --- GPU1
        GPU0 --> MediaServices
        GPU1 -.-> TranscodeService
    end

    OCI_WG -.->|Encrypted VPN Tunnel| UniFi
    UniFi --> HostOS
```

### Why Decouple Monitoring from On-Premises Infrastructure?
Running Uptime Kuma outside the homelab guarantees out-of-band operational visibility. If an ISP outage, localized power cut, or hardware maintenance takes the home network offline, the external monitoring node remains live, alerts administrators via Discord and mobile push notifications, and presents an accurate public status page.

### Why Dual GPU Allocation?
Modern video codecs like H.265 (HEVC) demand significant compute. Software encoding places heavy loads on host CPU cores, while unconstrained GPU sharing can lead to transcoder starvation during concurrent playback streams. Dedicating one Quadro P1000 card exclusively to user playback and the other card exclusively to background batch tasks ensures real-time streaming quality is never compromised.

### Why MergerFS Instead of Traditional Striped RAID or ZFS?
The storage pool consists of heterogeneous hard disk drives. MergerFS pools separate ext4 filesystems under a unified mount without striping data across drives. This architectural choice delivers three distinct advantages:
1. **Fault Isolation**: If a physical disk fails, only the files residing on that specific drive are lost; all files on surviving drives remain completely intact and directly accessible on standard ext4 filesystems.
2. **Drive Expansion Flexibility**: Drives of varying capacities can be added to the pool without re-striping or rebuilding an array.
3. **Power Efficiency**: Only the specific drive containing the requested media spins up during playback, leaving idle drives in low-power standby.

---

## 6. Next Milestones & Upcoming Roadmap

The active engineering priorities for the upcoming cycle are structured in sequential phases:

### Phase 1: OCI Container Runtime & Image Deployment (Planned / Next Priority - Not Yet Completed)
- Update package repositories on the Oracle Cloud instance.
- Install Docker Engine and the Docker Compose plugin.
- Author an explicit Compose configuration pinning Uptime Kuma to version **1.23.17** (avoiding unvalidated major version jumps during migration).
- Validate container health and resource constraints.

### Phase 2: Database Restoration and Verification (Planned - Not Yet Completed)
- Transfer the verified SQLite backup archive to the Oracle Cloud instance.
- Restore the database file, ensuring correct container permissions and directory ownership.
- Start the container and verify all monitors, notification channels, and historical telemetry data match the baseline snapshot.

### Phase 3: Zero-Trust Ingress & Site-to-Cloud Telemetry (Planned - Not Yet Completed)
- Configure a Cloudflare Tunnel on the Oracle instance, exposing status dashboards via zero-trust access controls without opening inbound firewall ports.
- Establish an outbound WireGuard VPN connection from the Oracle instance to the home UniFi gateway, enabling secure private pings of internal LAN endpoints.
- Validate that Homepage dashboards receive clean, authenticated status JSON.

### Phase 4: Tdarr Flow Audit & Safe Production Pilot (Planned - Not Yet Completed)
- Conduct a line-by-line configuration audit of the 5-stage transcoding Flow.
- Implement explicit attachment preservation rules for anime subtitles and embedded font tracks.
- Enforce automated hardlink checks to prevent touching files actively seeding in torrent clients.
- Resume controlled batch transcoding tests on non-production test libraries.

---

## 7. Chronological Project Progress Log

| Date | Category | Milestone Description | Status |
|---|---|---|---|
| **2026-08-01** | Homelab | Initial restoration planning for Lenovo ThinkStation P520 on Ubuntu Server. | Completed |
| **2026-09-02** | Storage | Configured multi-drive ext4 mergerfs storage pool with path-preserving policy. | Completed |
| **2026-09-06** | GPU / Media | Configured dual NVIDIA Quadro P1000 cards; resolved driver fallback and container runtime. | Completed |
| **2026-09-30** | Web / Ingress | Deployed hardened portfolio web container with strict Nginx security headers and zero-privilege user. | Completed |
| **2026-10-07** | Operations | Upgraded administration workspace to Antigravity CLI; established canonical master governance. | Completed |
| **2026-10-07** | Transcoding | Successfully completed isolated Tdarr pilot on 5 sample files with 30% space reduction. | Completed |
| **2026-10-08** | Cloud | Documented external GCP monitoring deployment and began pursuit of Oracle Cloud Always Free instance. | Completed |
| **2026-10-13 (02:00 UTC)** | Transcoding | Suspended production Tdarr pipeline; removed GPU Compose reservations; initiated Flow safety audit. | Paused / In Review |
| **2026-10-13 (03:30 UTC)** | Cloud | Recovered Google Cloud Uptime Kuma host; verified application-consistent SQLite database backup. | Completed |
| **2026-10-13 (04:15 UTC)** | Cloud | Provisioned Oracle Cloud Micro instance in Toronto (`ca-toronto-1`); configured and verified 2 GiB swap. | Completed |
| **2026-10-13 (04:45 UTC)** | Documentation | Promoted private master to Revision 0.11.1; created initial public progress journal bundle. | Completed |

---

*This progress journal is maintained as part of the WeitmanHouse infrastructure documentation initiative. Historical snapshots are archived chronologically in the [Archive Index](archive/index.md).*