🔍
PXE/iPXE boot into a full hardware inventory with SMART pre-flight checks and BMC certificate validation.
- A discovery agent (running inside the PXE-booted initrd image) inventories CPU, memory, disks, NICs, and TPM presence, then opens an outbound gRPC-mTLS control tunnel back to api-manager — it never accepts inbound connections.
- Every disk gets a SMART health check before it is trusted for a role.
- The BMC/iLO certificate is validated rather than blindly trusted; a mismatch is flagged, not silently accepted.
- Findings are auto-tagged onto the node (for example gpu:nvidia, cpu:vendor:intel, tpm:2.0) so Biomes with hardware requirements can only be scheduled where they will actually work.
🌐
Management, internal, and external traffic are each configured and secured independently, with a choice of control-plane frontend modes.
- Management baseline: out-of-band access to BMC/iLO/IPMI, SSH, and Prometheus scrape, typically on an isolated VLAN.
- Internal baseline: pod-to-pod and inter-node gRPC traffic, with Cilium (eBPF/XDP-accelerated) as the default CNI provider.
- External baseline: client-facing API and ingress traffic, isolated from internal cluster traffic.
- The Kubernetes control-plane frontend is selectable per cluster: kube-vip (default, L2 ARP failover in roughly 2–5 seconds) for smaller control planes, or an external load balancer for larger ones.
- Each baseline can run Gough’s built-in dnsmasq for bootstrap, or be migrated to Squawk DNS for HA DHCP/DNS with a non-disruptive cutover.
🔗
Nodes join the LXD cluster through a one-time-use token escrowed in Vault, with encrypted joiner secrets for bootstrap.
- Cloud-init configures the node’s network against the chosen baseline, installs LXD, and joins the cluster using a join token that is escrowed in Vault and consumed exactly once.
- Joiner secrets are node-scoped and encrypted, decrypted exactly once during deployment; reuse of a bootstrap token is treated as a security event.
- Root disks support LUKS encryption with a choice of sealing tiers — TPM2-only, TPM2+PIN, or network-bound.
🧬
Package any workload that runs as an LXC container or a KVM VM — cloud-init and resource sizing baked in — as a versioned Biome you deploy repeatably, the way a Kubernetes Deployment rolls out a service, but without being Kubernetes. A Kubernetes node, a Nextcloud server, a Windows AD domain controller, or your own image are all Biomes.
- Every Biome declares a workload_type of lxc or vm, so the instance Gough creates for it is a system container or a full virtual machine, plus cloud-init content and CPU/memory/disk sizing.
- Built-in catalog: k8s-primary (Kubernetes control plane), k8s-worker (join a worker), nest-agent (Nest-backed storage), and longhorn-agent (Longhorn-backed storage) — four Biomes among the kinds Gough supports (infrastructure, Kubernetes, monitoring, storage, and your own custom workloads).
- Because a Biome is just an LXC container or a KVM VM plus cloud-init, its payload can be almost any OS or appliance — a Windows Active Directory domain controller or a database VM, a Nextcloud or self-hosted app container — authored as a custom Biome. Kubernetes is one built-in kind, not a requirement.
- The four built-in Biomes ship pre-verified as part of the Gough release itself. Cosign signing and SBOM generation for a custom Biome you author is on the roadmap — today's API records and audits a signing request but does not yet execute a cryptographic signature, so treat custom Biomes as unverified until that ships.
- Full lifecycle from the CLI: gough biome new/validate to author, promote/rollback/diff to manage versions, and deploy/eligibility-check/upgrade to roll out.
- Custom biomes can be authored and deployed alongside the built-in set; distribution today is via git repositories.
🔐
HashiCorp Vault issues the platform’s certificate authority and secrets; every service holds a SPIRE-issued SPIFFE identity for mTLS.
- Vault manages the root CA, intermediate CA, and leaf certificates, plus joiner secrets, bootstrap tokens, and LXD join tokens.
- Every Gough service is issued a SPIFFE SVID by SPIRE and communicates with other services over mutual TLS using that identity — no long-lived static service credentials.
- Operator and API authentication is JWT/OIDC: api-manager is its own first-party OIDC issuer (ES256), and gough login uses an OIDC device-code flow with tokens stored in the OS keychain, never as CLI arguments or plaintext files.
- RBAC roles are operator, maintainer, and viewer.
- Every backend Helm chart runs rootless — runAsNonRoot, dropped Linux capabilities, read-only root filesystem — with one documented exception: worker-ipxe binds the privileged DHCP/TFTP ports via CAP_NET_BIND_SERVICE rather than root.
📜
Every provisioning, deployment, and administrative action is written to an append-only, hash-chained audit log with an offsite mirror.
- Each audit entry carries an entry_hash and prev_hash; a hash mismatch anywhere in the chain indicates tampering, corruption, or operator error.
- The chain replicates to an offsite mirror over NATS, with mirror-lag alerting if it falls behind.
- gough audit verify, gough audit log, and gough audit export cover integrity checking, review, and export from the CLI.
- A monthly compliance-audit routine (gough audit report) produces a report for a given date range.
☁️
The same provisioning API creates and manages machines across AWS, Azure, GCP, Vultr, LXD, and Canonical MaaS — LXD containers and VMs on your own hardware included, not just bare metal or cloud instances.
- Bare-metal PXE discovery does not require any cloud backend — it is the primary path, with MaaS as one integration among six, not a prerequisite.
- The LXD backend creates LXC containers or KVM virtual machines interchangeably (a per-request type choice), each with cloud-init user data and CPU/memory/disk sizing passed through at creation — the same unified request shape used for AWS, Azure, GCP, and Vultr instances.
- A cloud machine only counts toward a node allowance once Gough has provisioned it (tagged gough-managed=true); machines merely discovered during an inventory sync are never metered.
- Multi-cloud endpoints are controlled by a rollout feature flag, available on every license tier — the flag is a kill-switch, not a paid-tier gate.
🖥️
A React operator dashboard and the gough CLI both cover machines, biomes, deployments, storage, and audit, authenticated via OIDC.
- WebUI tabs: Machines, Biomes, Deployments, Storage, Audit, Teams, Users — including a Plan Disks view for reserving OS disks versus storage drives.
- The WebUI embeds an xterm.js terminal for node and pod shell access.
- The gough CLI mirrors the same operations for scripting and CI/CD, with a JSON output mode for the commands used in automation.
📊
Prometheus and Grafana visibility, 30 alert-driven runbooks, and quarterly DR drills with defined RTO/RPO targets.
- Every service exposes Prometheus metrics; Grafana ships dashboards for cluster overview and per-node resource usage.
- Thirty alert-driven runbooks cover critical failure modes (audit chain integrity, quorum loss, SPIRE identity rotation, biome signature verification) through warnings (MTU mismatch, certificate mismatch, SMART warnings) and external integration health.
- gough dr drill exercises the full recovery process — secondary connectivity, backup integrity, database replication lag, Vault state replication, SPIRE federation, and DNS update capability — without impacting production; RTO/RPO targets are documented per cluster size.
- Capacity forecasting is backed by WaddleAI and degrades gracefully if that service is unreachable, rather than blocking cluster operations.
💾
A documented decision guide matches fleet size to a storage backend — local-nvme, Nest, Longhorn, Ceph, iSCSI, or NFS.
- Nest and Longhorn deploy as Biomes (nest-agent, longhorn-agent) alongside Kubernetes workers; Ceph, iSCSI, and NFS integrate as external backends.
- Fleet-size guidance: 3–5 nodes favors Nest, 5–20 nodes favors Longhorn, 20+ nodes favors Ceph.
- gough storage status and kubectl get storageclasses report current backend health.
🔑
Node shell access is granted by short-lived, Vault-rooted SSH certificates instead of distributed static SSH keys.
- access-agent runs a reverse-SSH (rssh) server on each managed node and validates every connecting certificate against the CA’s public key before allowing a session.
- Certificates are scoped to specific principals and time-boxed; an expired certificate is rejected outright.
- Nodes enroll with access-agent automatically as part of Phase 2/3 provisioning — no separate manual step for normally provisioned nodes.
⎈
Gough’s own backend services ship as one Helm chart per service, deployable independently of gough cluster create for GitOps pipelines.
- Charts: api-manager (REST :8080, gRPC :50051), worker-ipxe (DHCP/TFTP boot orchestration), discovery-agent (packaging for the PXE initrd agent), access-agent (reverse-SSH), and webui.
- Deployed into a gough namespace — never environment-suffixed; the environment lives in the values file.
- Images are pulled from ghcr.io/penguintechinc/gough/<service>, pinned to an immutable tag or digest for any environment beyond alpha.
- api-manager’s OIDC signing key is provisioned once and shared across replicas via a Kubernetes Secret (Vault/External Secrets Operator preferred), since every replica both issues and validates tokens.