Engineering reference project

Environments that provision identically

Reusable Terraform for a multi-engine database estate, with remote locking state, secrets injected at apply time, and nothing applied without a plan and a deliberate gate.

Discuss a similar build
Fortis engineering reference project. Built and operated by Fortis as a working reference implementation, not delivered under contract to a client. No customer, program or employer material appears here. Figures describe measured behaviour of this build only.
Type
Engineering reference build
Engines covered
PostgreSQL, MySQL, SQL Server
Pattern
Reusable modules, gated apply
State
Remote backend with locking

Executive overview

Reusable Terraform for a multi-engine database estate, so that every environment provisions to the same baseline and a compliance posture becomes reproducible rather than reviewed once and hoped for.

Modules cover relational engines, networking, backup configuration and secret injection. State lives in a locking remote backend, never in version control, because state holds credentials in clear text. Nothing applies without passing a plan on a protected branch and a deliberate human gate.

The engineering interest is not that Terraform was used. It is the specific choices that make a module set safe to change: iteration over a keyed map rather than a positional index, lifecycle rules that stop provider drift restarting a fleet, and the components deliberately left outside automation because automating them causes an outage.

The problem

An estate that grew without a provisioning standard has no two environments alike.

Problem

  • Compliance cannot be reproduced. If each environment was configured by hand, a finding fixed in one place is still open in five others, and there is no way to demonstrate otherwise.
  • Change is unsafe. Without state and a plan, the only way to know what an apply will do is to run it.
  • Credentials proliferate. Connection strings end up in configuration files, then in version control, then in a pipeline log.

The requirement was one code path producing identical, compliant baselines across environments, with credentials injected at run time and a plan reviewable before anything changes.

Engineering approach

  • One code path, one variables file per environment. The same modules build development and production; only the inputs differ. A deliverable that hardcodes one environment is not reusable.
  • Iteration over a keyed map. Resources are addressed by a stable string key, so removing one entry affects exactly that resource.
  • Lifecycle rules where the provider drifts. Some provider attributes read back differently from how they were declared, which makes every plan propose changes and every apply restart the fleet. Those attributes are explicitly ignored, and the trade is documented.
  • Secrets injected, never stored. Modules take a secret reference and resolve it at apply time. No credential appears in a variables file or in state output.
  • Deliberate exclusions. One component is intentionally outside automation, because the tool that manages it runs the pipeline that manages the tool. That circular dependency is recorded as a decision, not left as an inconsistency.
Architecture

Architecture

The module, state and pipeline relationship, and where credentials enter.

Enlarge Terraform module, state and pipeline topology
Figure 1Module and state topology. Plan runs on every merge request, apply runs only after a deliberate gate, and the runner holds the credentials so no engineer workstation needs them. State sits in a locking remote backend because it contains secrets in clear text.

Key technologies

Provisioning
  • Terraform
  • Reusable modules over a keyed map
  • One variables file per environment
  • Version-pinned providers
Relational engines
  • PostgreSQL
  • MySQL
  • SQL Server
  • Managed and self-managed targets
State
  • Remote HTTP backend
  • State locking
  • State excluded from version control
Secrets
  • Managed secret store integration
  • Resolved at apply time
  • Encrypted vault for configuration secrets
Configuration
  • Ansible for in-guest state
  • Idempotent roles
  • Content comparison rather than state tracking
Delivery
  • Protected branch, merge request required
  • Plan on every change
  • Manual gate before apply

Security and operational considerations

  • State contains secrets in clear text. It goes in a locking remote backend and never in version control. This is verified against the tracked file list rather than assumed from the ignore file.
  • Credentials live in the pipeline; settings live in the repository. A pipeline variable that silently overrides a security-relevant default is invisible drift. If a setting matters it belongs in code where a merge request can review it.
  • Version skew breaks state. State written by a newer binary cannot be read by an older one. Pipeline and local versions are pinned and must match, because an accidental upgrade locks the pipeline out of its own state.
  • Locks need a documented recovery path. An interrupted apply leaves a lock behind. Clearing it is a normal operation with a runbook entry, not an emergency.
  • Compile and apply are not always distinct. On some tooling the command that appears to validate a change actually applies it. That is called out explicitly in the runbook, because assuming a dry run is a fast way to change production.
Decisions

Every one of these came from a failure mode with a specific cost.

Judgement

Important engineering decisions

Iteration over a keyed map rather than a positional index

Alternative rejected: count with index-addressed resources

With index addressing, deleting the second of five resources shifts everything after it down by one. The tool then sees three resources whose identity changed and destroys and recreates them. With a keyed map the address is a stable string, so removing one entry affects exactly that resource. On infrastructure that matters this is closer to a rule than a preference.

Lifecycle rules on attributes the provider misreports

Alternative rejected: accepting perpetual drift, or abandoning declarative management of those attributes

Certain blocks drifted on every plan because the provider read back values that never matched what was declared, so each apply proposed changes to every machine and restarted all of them. A configuration-only change would bounce the whole platform. Ignoring those attributes stops that, at the cost that genuine changes to them must be applied deliberately. Verified by comparing process identifiers across the fleet before and after a single-attribute change: only the intended machine restarted.

One component deliberately outside pipeline-driven apply

Alternative rejected: automating everything for consistency

The component in question runs the pipelines that manage its own infrastructure. A restarting change applied through that pipeline stops the pipeline mid-job, which then cannot write state back or release its lock, so the next run sees the same outstanding change and repeats the cycle. This is a circular dependency rather than a configuration defect, and the honest answer is to exclude it and reconcile in code afterwards.

Settings in the repository, credentials in the pipeline

Alternative rejected: pipeline variables overriding code defaults

Two pipeline variables were once silently overriding a declared endpoint and a TLS verification setting. The code said one thing and the pipeline did another, and the most security-relevant setting in the configuration was invisible unless someone opened the CI settings. Credentials still belong in the pipeline. The settings that govern how they are used do not.

Versions pinned on both sides

Alternative rejected: tracking the latest release

State written by a newer binary cannot be read by an older one, so a local run on a newer version can lock the pipeline out permanently. Pinning both sides and matching them is cheap; recovering from a forward-migrated state file is not.

Demonstrated capabilities

Measured on this build. Nothing here is a client outcome or a contracted result.

  • Reusable module set provisioning a multi-engine estate from one code path with per-environment inputs.
  • Remote state with locking, verified excluded from version control against the tracked file list.
  • Secret references resolved at apply time, with no credential in a variables file or in state output.
  • Lifecycle handling proven by process-identifier comparison across the fleet: a single-attribute change restarted only the intended machine while the others held their original processes.
  • Plan on every merge request behind a protected branch, with a deliberate gate before apply. The branch protection has blocked real direct pushes.
  • Decision record capturing the alternatives rejected, including the circular dependency that keeps one component outside automation.

What this demonstrates

This project is the reference implementation behind two Fortis service lines.

Cloud Modernization

Landing zone, automation and hardening declared in code, so an environment is reproducible and a compliance baseline can be demonstrated rather than described.

Data Platform Engineering

Multi-engine database provisioning from a single module set, with networking, backup configuration and secret injection handled consistently across engines.

Repeatable compliance posture

Provisioning standardised as code means a finding fixed once is fixed everywhere, which is the difference between a reviewed posture and a reproducible one.

Engineering judgement

The decisions that matter here are about failure modes: index shifting, provider drift restarting a fleet, circular dependencies, and invisible pipeline overrides.

All projects Start a conversation