Reliability & Security

We design for "least privilege" and minimal attack surface across identity, network, and application layers.

• Identity & access: Centralised SSO/RBAC via ZITADEL; application sessions hardened with Better Auth; enforced MFA for all privileged accounts; keys and secrets stored in cloud KMS/secret stores with rotation.
• Network controls: Cloudflare WAF + firewall rules in front of all public endpoints; private networking for internal services; Kubernetes NetworkPolicies for east-west isolation; rate limiting and pagination on public APIs.
• Tenant isolation: Per-tenant scoping, row-level security, and service-to-service identity; no cross-tenant data access paths.
• Developer access: Principle of least privilege, regularly reviewed; unused tokens/SSH keys removed; audited admin paths; break-glass procedures tested.

Our stack is built for resilience with high availability and GitOps automation, hosted in Singapore.

• Cloud providers: Envoy CRM runs on Hetzner Cloud (Singapore, sin-dc1) and Volty runs on AWS (Singapore, ap-southeast-1), both fronted by Cloudflare for global Anycast, WAF, and caching.
• Orchestration: Kubernetes for self-healing, scaling, and isolation; deployments managed with ArgoCD (GitOps).
• Databases: CloudNativePG (CNPG) for high-availability Postgres with WAL streaming; pgBouncer for connection pooling; self-hosted Supabase components where applicable.
• Workflows: Windmill for background jobs, retries, and idempotent workflows.
• Observability: Metrics in Grafana, tracing/logs via SigNoz (OpenTelemetry); external uptime with Better Stack.
• Source & CI: Gitea hosts repositories and pipelines with signed images and policy checks.

We monitor from the outside-in and the inside-out to detect issues quickly.

• External HTTP and heartbeat checks from multiple regions via Better Stack.
• Transactional probes for critical user journeys (our "immune system").
• Provider-independent monitors so the watcher doesn't fail with the watched.
• Grafana SLO dashboards tracking latency, errors, and saturation.

When signals breach SLOs, we escalate rapidly with clear ownership and communication.

• 24/7 on-call rotations with multi-channel paging via Better Stack.
• Runbooks for common scenarios (db failover, queue backlog, auth outage, cache issues).
• Transparent user updates for material incidents and a public status page where applicable.
• Post-incident reviews with documented RCA and preventive actions.

We prefer safe rollbacks to last known-good over risky hot-patches in production.

• Restart workloads and autoscaled deployments first.
• Rollback with ArgoCD to a healthy Git revision (schema-safe).
• Escalate to extended maintenance only when necessary, with fixes tested in staging before release.
• No untested changes in prod—avoid cascading failures.

We minimise data exposure and secure data in transit and at rest.

• TLS everywhere; storage encryption; least-privilege data access.
• PII logging is off by default; redaction/masking in logs and traces.
• Per-tenant isolation (RLS and scoped tokens) to prevent cross-tenant access.
• Cloudflare in front of all public surfaces (WAF, bot management, TLS termination).

We instrument everything that matters, without capturing unnecessary personal data.

• Structured app and infra logs to SigNoz; metrics to Grafana; traces via OpenTelemetry.
• Business-event logs (e.g., verification emails, integration setup) for fast diagnosis.
• Windmill tracks workflow outcomes, retries, and idempotency.
• Provider logs (AWS CloudWatch and Hetzner) plus Cloudflare logs feed anomaly alerts.

Independent verifiers assert key outcomes and invariants, separate from app runtime.

• Separate repos/containers run checks (e.g., "no job stuck", "calc results valid").
• Failures open incidents automatically; checks are provider-diverse where possible.

We reduce coupling to avoid cascading failures and supply-chain risk.

• API, frontend, and background workers are isolated and rate-limited.
• Service calls authenticated via mTLS or signed tokens; strict egress controls.
• Environment segregation with separate VPC/VNet segments and namespaces.

We assume components—and sometimes providers—will fail, and design for graceful degradation.

• Stateless services behind Cloudflare + cloud load balancers for horizontal scaling.
• Kubernetes self-healing with PodDisruptionBudgets and node pools spread across fault domains where the provider supports it.
• CNPG with WAL streaming and read replicas; pgBouncer to smooth connection bursts.
• Self-hosted Supabase components externalize state and are replaceable.
• Provider-agnostic, GitOps-managed deployments (ArgoCD) so workloads can be rebuilt or relocated if a provider degrades.

We plan for the unexpected with clear objectives and practiced recovery.

• RTO (target): ≤ 6 hours for complete downtime.
• RTC (current): ~1–3 hours depending on incident scope.
• RPO (target): ≤ 12 hours.
• RPC (current): ~10 minutes–1 hour via frequent snapshots and WAL archives.
• Automated backups & PITR: CNPG continuous archiving; restore drills performed regularly.
• Backup storage: Encrypted backups written to AWS S3 in Singapore (single-region). Cross-region or cross-cloud replication is on the roadmap, not currently deployed.

We ship safely with automated tests, previews, and phased releases.

• GitOps with ArgoCD; every change lands via PR in Gitea with required checks and reviews.
• Unit, integration, and contract tests; coverage tracked with higher thresholds on critical paths.
• Ephemeral preview environments for realistic validation.
• Canary/blue-green rollouts; halt/rollback immediately on regressions.

We defend against volumetric and application-layer attacks without blocking legitimate users.

• Cloudflare WAF, bot management, and global CDN shielding origin infrastructure.
• Tiered caching and origin shields; surge queues and rate limits on dynamic endpoints.
• Challenge flows (e.g., captcha) for suspicious traffic while keeping UX smooth for real users.

• Clear incident communications and status updates for material events.
• Engineering runbooks and security practices documented for internal and customer reviews.
• Customer data remains the customer's; we practise data minimisation and purpose limitation.

For security questions or to report a potential vulnerability:

We take security reports seriously and aim to acknowledge verified vulnerabilities within 24 hours.