Homelab & Cloud.
Engineering in progress.

Project overview: What we are building and why

The WeitmanHouse infrastructure project is a resilient, dual-tier personal computing and homelab environment designed to balance high-capacity on-premises media processing with reliable, independent off-site infrastructure monitoring.

Self-hosting complex workloads like media streaming, automated transcoding pipelines, and application services requires robust architecture. However, hosting everything within a single physical location introduces single points of failure: if local power, internet connectivity, or host hardware experiences an outage, internal monitoring tools go down along with the services they monitor.

To address this, our architecture decouples high-throughput local operations from cloud-based availability monitoring:

• On-Premises Infrastructure (Lenovo ThinkStation P520): A workstation-class server running Ubuntu Server LTS, equipped with an Intel Xeon processor, 128 GB of ECC memory, pooled multi-terabyte storage, and dual dedicated NVIDIA Quadro workstation GPUs. This host handles containerized media services, automated storage management, and hardware-accelerated video processing.
• External Cloud Monitoring (Multi-Cloud Free Tier): Independent, lightweight cloud compute nodes deployed across Google Cloud Platform (GCP) and Oracle Cloud Infrastructure (OCI). These instances run out-of-band telemetry (Uptime Kuma) and secure ingress tunnels, monitoring home services from the outside without relying on local network uptime.
• Hybrid Secure Access: Zero-trust remote access built on Cloudflare Tunnels and Access for public telemetry endpoints, combined with an authenticated UniFi WireGuard VPN backbone for secure administrative access and private site-to-cloud telemetry.

Current status summary

All work across the environment is tracked under three explicit verification boundaries: Completed (fully validated against system probes and logs), In Progress / Planned (actively sequenced and staged), and Paused / Unverified (held for architectural review or safety verification).

• Compute & OS (Lenovo P520): Completed. Ubuntu Server LTS; 128 GB RAM; stable multi-day uptime.
• Pooled Storage (MergerFS): Completed. Multi-drive ext4 pool with path-preserving allocation policy.
• GPU Allocation: Completed. Dual Quadro P1000 GPUs partitioned for independent playback streaming and batch transcoding.
• Administration Workspace: Completed. Modern Antigravity CLI environment with tmux and single-master documentation governance.
• Transcoding Pilot: Completed. Verified ~30% space reduction with 100% stream integrity on 5 sample media files.
• Production Transcoding Pipeline: Paused. Suspended for full multi-stage Flow safety audit, font attachment verification, and hardlink protections.
• Cloud Monitoring (Google Cloud): Completed. Host recovered from memory pressure; application-consistent SQLite backup verified and archived.
• Cloud Infrastructure (Oracle Cloud): Completed. Always Free compute node provisioned in Toronto; 2 GiB swap space configured and validated.
• Cloud Service Migration (OCI): Planned / Next (Not Yet Completed). Docker runtime deployment, pinned Uptime Kuma 1.23.17 rollout, and database restoration staged for execution.
• Hybrid Ingress & Telemetry: Planned (Not Yet Completed). Cloudflare Tunnel zero-trust access and outbound UniFi WireGuard private telemetry.

Completed infrastructure milestones

Workstation Host & Storage Architecture: Standardized the Lenovo P520 platform on Ubuntu Server LTS with long-term kernel support. Configured multiple high-capacity ext4 storage disks unified through a mergerfs pool using a path-preserving, free-space based (pfrd) policy to keep media folders organized while balancing disk utilization.

Dual Workstation GPU Segmentation: Assigned dedicated GPU 0 to real-time media playback streaming (Plex/Jellyfin), protecting interactive user streams from GPU starvation. Dedicated GPU 1 to background batch encoding workloads, ensuring long-running transcode tasks run entirely out-of-band from user-facing media sessions.

Administration Workspace & Governance: Upgraded homelab administration to the Antigravity CLI environment with tmux multi-pane workspace helpers. Established a canonical master documentation standard (P520-Master.md) ensuring every change undergoes read-only pre-flight inspection, execution planning, and post-change validation.

Isolated Transcoding Pilot: Deployed an isolated containerized Tdarr pilot. Reclaimed approximately 30% storage space across the pilot batch (reducing file volume from ~20.1 GB to 14.0 GB) while verifying 100% video stream integrity, multiple audio track alignment, and chapter marker retention.

Cloud Monitoring Recovery & Backup: Recovered the remote Uptime Kuma monitoring host following acute memory pressure. Executed an online point-in-time database backup using SQLite's native backup API, capturing all configured health monitors, notification hooks, custom status pages, and over 172,000 historical heartbeat records without service interruption.

Oracle Cloud Provisioning & Swap Hardening: Secured an Always Free compute node in Oracle Cloud's Toronto data center (ca-toronto-1) running Ubuntu 24.04 LTS. Configured and enabled a permanent 2 GiB swap file persisted in the system fstab, eliminating out-of-memory kernel panics historically experienced on 1 GB micro-tier instances.

Problems encountered and lessons learned

Memory Exhaustion on Micro Cloud Instances: Micro-tier cloud virtual machines (~1 GB RAM) are prone to kernel out-of-memory kills during peak container activity or system updates. Relying on physical RAM alone on free-tier micro instances is an operational anti-pattern; every minimal cloud instance must be hardened with a properly sized swap file (2 GiB) and strict Docker log rotation policies.

Media Transcoding: Font Attachments and Metadata Safety: In stylized anime media, subtitle tracks rely on embedded font attachments (.ttf, .otf). During pipeline testing with automated multi-stage transcode flows, standard stream-reordering utilities stripped five embedded font attachments from temporary output files. Additionally, automated workflows attempted in-place metadata modifications on source files. Automated transcode flows must never write directly to source media in place, and must verify font attachment preservation and hardlink protections before touching files.

SQLite Online Backups Under Write-Ahead Logging (WAL): Applications utilizing SQLite with Write-Ahead Logging active write transactions to separate temporary files. Simple file copies while the database is open can capture inconsistent snapshots. Backups must be performed using the native SQLite online backup API (sqlite3 .backup) for true point-in-time consistency.

Zero-Trust Access vs. Direct Container Port Exposure: Exposing application web dashboards directly to the internet via port forwards creates unnecessary attack surfaces. Cloud-based services should remain bound to private container bridges or loopback, mediated exclusively through encrypted tunnels (Cloudflare Tunnel) coupled with identity provider policies (Cloudflare Access).

Architectural decisions and technical reasoning

Decoupling Monitoring from Homelab: Running Uptime Kuma outside the homelab guarantees out-of-band operational visibility. If an ISP outage, power interruption, or hardware maintenance takes the home network offline, the external monitoring node remains live, alerts administrators via Discord and mobile push notifications, and presents an accurate public status page.

Dual GPU Segmentation: Modern video codecs like H.265 (HEVC) demand significant compute. Dedicating one workstation GPU exclusively to user playback and the other card exclusively to background batch tasks ensures real-time streaming quality is never degraded by background maintenance.

MergerFS Over Striped RAID/ZFS: MergerFS pools separate ext4 filesystems under a unified mount without striping data across drives. This guarantees fault isolation (if one disk fails, data on remaining drives is completely readable), drive expansion flexibility (drives of mixed sizes can be added easily), and power efficiency (only the disk being accessed spins up).

Next milestones and upcoming roadmap (Planned / Not Yet Completed)

1. OCI Container Runtime & Image Deployment (Planned - Not Yet Completed): Update packages on the Oracle Cloud instance, install Docker and Compose, and author an explicit configuration pinning Uptime Kuma to version 1.23.17.
2. Database Restoration and Verification (Planned - Not Yet Completed): Transfer the verified SQLite backup archive to the Oracle Cloud instance, restore the database, and validate monitors and historical telemetry.
3. Zero-Trust Ingress & Site-to-Cloud Telemetry (Planned - Not Yet Completed): Configure Cloudflare Tunnel and outbound UniFi WireGuard VPN connection for private internal LAN health probes.
4. Tdarr Flow Audit & Safe Production Pilot (Planned - Not Yet Completed): Complete a configuration audit of the 5-stage transcoding Flow, enforce font attachment retention, and verify hardlink seed protections.

Chronological progress history

A full chronological log of historical checkpoints and decisions is maintained in the Archive Index. Key milestones include:

• 2026-08-01: Initial restoration planning for Lenovo ThinkStation P520 on Ubuntu Server.
• 2026-09-02: Configured multi-drive ext4 mergerfs storage pool with path-preserving policy.
• 2026-09-06: Configured dual NVIDIA Quadro P1000 cards for independent streaming and transcode compute.
• 2026-09-30: Deployed hardened portfolio web container with strict Nginx security headers.
• 2026-10-07: Upgraded administration workspace to Antigravity CLI; established canonical master governance.
• 2026-10-07: Completed isolated Tdarr pilot on 5 sample files with verified 30% space reduction.
• 2026-10-08: Documented external GCP monitoring deployment and began pursuit of Oracle Cloud instance.
• 2026-10-13 (02:00 UTC): Suspended production Tdarr pipeline; removed GPU Compose reservations; initiated Flow safety audit.
• 2026-10-13 (03:30 UTC): Recovered Google Cloud Uptime Kuma host; verified application-consistent SQLite database backup.
• 2026-10-13 (04:15 UTC): Provisioned Oracle Cloud Micro instance in Toronto; configured and verified 2 GiB swap space.
• 2026-10-13 (04:45 UTC): Promoted private master to Revision 0.11.1; created initial public progress journal bundle.