When I first built the OSINT lab, the goal was simple: a dedicated, network-isolated environment for open source intelligence work running inside a Proxmox LXC container. Maltego and SpiderFoot were up and running, the container was air-gapped from the rest of the homelab, and it did the job.

The problem was the dashboard. The original implementation used Node.js with Server-Sent Events to stream tool output to the browser, and it was fragile. SSE connections would drop unpredictably, tool output would get cut off mid-run, and the whole thing had a tendency to fall over under any kind of sustained load. It worked just well enough to be annoying rather than broken enough to force a fix — until I decided to fix it properly.

The Rewrite

The new backend is built in Python using FastAPI with genuine WebSocket streaming. The difference in reliability is significant. WebSockets maintain a persistent two-way connection between the browser and the server, so there is no reconnection dance, no dropped output, and no silent failures. Each tool streams its output in real time as it runs rather than buffering and dumping at the end.

The frontend stayed largely the same — a clean dashboard interface — but with the WebSocket connection underneath it actually behaves the way it should.

All 11 Tools Confirmed Working

The rewrite was also an opportunity to audit the full tool stack properly. Everything is now confirmed end to end:

  • Maigret — username reconnaissance across hundreds of platforms
  • Holehe — checks whether an email address is registered on various services
  • Sherlock — social media username search
  • theHarvester — email, subdomain, and hostname enumeration
  • Subfinder — passive subdomain discovery
  • Socialscan — username and email availability checks
  • Photon — web crawler for OSINT data extraction
  • Nuclei — vulnerability and misconfiguration scanning
  • Trufflehog — secret and credential detection in repositories
  • SpiderFoot — automated OSINT collection and correlation
  • ttyd — browser-based terminal for direct container access

Each one launches, runs, streams output, and exits cleanly. The only known outstanding issue is that cancelling a Subfinder run does not kill the child processes it spawns — something to address in a future iteration.

What This is Used For

The lab sits in CT200 on the homelab, network-isolated and only accessible via WireGuard. It is relevant to intelligence, security research, and investigative work — and having it as a functioning environment rather than a theoretical one is what makes it portfolio-worthy. Anyone can install these tools. Fewer people have built a working dashboard that streams their output cleanly in real time.

The full project page is on the portfolio site with more technical detail on the architecture.