mdp/Infrastructure/modern_data_platform_infrastructure.md

Greenfield Modern Data Platform — Azure Infrastructure Deployment Guide

Companion to: Modern Data Platform – High-Level Reference Architecture v8.0 Classification: Internal – Confidential | Date: March 2026

---

1. Azure Landing Zone Architecture

1.1 Design Philosophy

The Greenfield Modern Data Platform deploys within an Azure Landing Zone aligned with Microsoft's Cloud Adoption Framework (CAF) Enterprise-Scale architecture. The landing zone provides the secure, governed foundation required for a regulated financial institution operating under AMF, OSFI, and Law 25 requirements. Every design decision traces back to the reference architecture's principles: Security by Design, Right Tool for the Right Workload, and Unified Governance with Federated Execution.

1.2 Hub-Spoke Topology

The deployment follows a hub-spoke network topology — the standard pattern for enterprise-scale Azure deployments in regulated financial services. Shared network services (firewall, DNS, on-premises connectivity) are centralized in a hub virtual network, while each workload environment operates in its own spoke VNet peered to the hub.

Component	VNet Name	Purpose
Connectivity Hub	vnet-hub-canadacentral	Azure Firewall (Premium), ExpressRoute gateway to on-premises data centers, Azure Bastion for secure admin access, centralized Private DNS Zones, VPN gateway for backup connectivity
Data Platform – Production	vnet-data-prod-cc	Databricks workspaces (VNet-injected), ADLS Gen2 private endpoints, Databricks SQL Warehouse endpoints, Purview private endpoints, Key Vault private endpoints
Data Platform – Non-Production	vnet-data-nonprod-cc	Development and staging Databricks workspaces, test storage accounts, sandbox environments for data scientists
SAS Viya	vnet-sas-prod-cc	SAS Viya Compute Server pods on AKS (or VMs), JDBC connectivity to Databricks SQL Warehouses routed through hub firewall
Management	vnet-mgmt-cc	Azure DevOps self-hosted agents, Terraform runners, monitoring infrastructure, Manta lineage engine deployment (if IaaS)
Fabric (Overlay)	N/A (PaaS)	Microsoft Fabric is fully managed PaaS — no VNet required. Connectivity to ADLS Gen2 is through OneLake shortcuts and Microsoft-managed private endpoints from the Fabric tenant

All spoke VNets peer to the hub. Inter-spoke traffic routes through Azure Firewall for inspection. No direct spoke-to-spoke peering is permitted — this ensures all cross-environment traffic is logged and inspectable.

1.3 Hub Services Detail

Azure Firewall (Premium SKU). Centralized egress filtering with TLS inspection for outbound traffic, FQDN-based application rules, network rules for port-level control, and threat intelligence–based filtering. All spoke VNets route their default route (0.0.0.0/0) through the hub firewall via user-defined routes (UDRs). The Premium SKU is required for TLS inspection and IDPS capabilities mandated by OSFI guidelines.

ExpressRoute Gateway. Dedicated, private connectivity between Greenfield' on-premises data centers (Lévis/Montréal) and Azure. The ExpressRoute circuit terminates in the hub VNet with route propagation to all spokes. This is the primary path for source system connectivity — core banking (DB2, Oracle), insurance policy administration, mainframe extracts, and CRM systems all feed data through this circuit into ADF and Auto Loader ingestion pipelines.

Private DNS Zones. Centralized in the hub subscription and linked to every spoke VNet. Zones include privatelink.dfs.core.windows.net, privatelink.blob.core.windows.net, privatelink.vaultcore.azure.net, privatelink.azuredatabricks.net, privatelink.purview.azure.com, privatelink.servicebus.windows.net, and others. This ensures all workloads resolve private endpoint FQDNs to their private IPs regardless of which spoke they run in.

Azure Bastion. Browser-based RDP/SSH access to management VMs and jump boxes without exposing public IPs. This is the only permitted path for interactive administrative access to infrastructure resources.

1.4 Region Strategy

	Primary	Secondary
Region	Canada Central (Toronto)	Canada East (Québec City)
Role	All production and non-production workloads	Disaster recovery, geo-redundant storage replication target
Rationale	Lower latency to majority of Azure services; broader service availability	Data residency compliance (Law 25, OSFI); geographic separation for DR

Critical constraint: All data must remain within Canadian Azure regions. No replication, backup, or compute spillover to non-Canadian regions is permitted. This constraint is enforced via Azure Policy at the management group level (allowedLocations = canadacentral, canadaeast).

---

2. Resource Organization

2.1 Management Group Hierarchy

The data platform subscriptions inherit organization-wide Azure Policies from Greenfield' root management group (allowed regions, required tags, prohibited resource types, mandatory encryption). The platform-specific hierarchy:

Greenfield (Root MG)
└── Data & AI Platform (MG)
    ├── Production (MG)
    │   ├── sub-data-platform-prod
    │   ├── sub-data-sas-prod
    │   └── sub-data-fabric-prod
    ├── Non-Production (MG)
    │   └── sub-data-platform-nonprod
    └── Connectivity & Shared Services (MG)
        ├── sub-data-connectivity
        └── sub-data-management

Azure Policies applied at the "Data & AI Platform" management group level include: deny public endpoint creation on storage/Key Vault/Purview, require specific tags on all resources, enforce diagnostic settings on all supported resources, deny creation of unmanaged disks, and require TLS 1.2 minimum.

2.2 Subscription Layout

Subscription	Purpose	Key Resources
sub-data-connectivity	Hub networking, shared services	Azure Firewall, ExpressRoute gateway, Bastion, Private DNS Zones, VPN gateway
sub-data-platform-prod	Production data platform workloads	Databricks workspaces (prod), ADLS Gen2 accounts (bronze/silver/gold/staging), Key Vault, Purview account, Event Hub namespaces
sub-data-platform-nonprod	Development, staging, sandbox	Databricks workspaces (dev/stg/sandbox), ADLS Gen2 (dev/stg), Key Vault (non-prod)
sub-data-sas-prod	SAS Viya Compute Server	AKS cluster (or VM scale sets), SAS-specific staging storage, SAS license server
sub-data-fabric-prod	Microsoft Fabric capacity	Fabric F-SKU capacity resources, Fabric admin workspace for capacity management
sub-data-management	Platform operations & DevOps	Azure DevOps self-hosted agents, Terraform state storage account, Log Analytics workspace, Manta deployment (if IaaS), Azure Monitor resources, Microsoft Sentinel

Rationale for subscription isolation: Separate subscriptions provide independent RBAC boundaries (SAS team cannot accidentally modify Databricks resources), independent Azure Cost Management scopes for precise chargeback, independent quota management, and separate blast radii for misconfigurations.

2.3 Resource Group Strategy

Within sub-data-platform-prod:

Resource Group	Contents
rg-databricks-prod-cc	Databricks workspaces, managed resource groups (auto-created by Databricks), associated NSGs
rg-storage-prod-cc	ADLS Gen2 storage accounts (bronze, silver, gold, staging, archive), storage private endpoints
rg-governance-prod-cc	Microsoft Purview account, Purview managed storage, Purview private endpoints, Purview managed Event Hub
rg-keyvault-prod-cc	Azure Key Vault instances (platform secrets + encryption keys), Key Vault private endpoints
rg-monitoring-prod-cc	Log Analytics workspace (prod), diagnostic settings resources, Azure Monitor action groups, alert rules, workbooks
rg-networking-prod-cc	VNet, subnets, NSGs, route tables (UDRs), private endpoint NICs
rg-ingestion-prod-cc	Azure Data Factory instance, Event Hub namespaces (for streaming ingestion), ADF private endpoints

---

3. Networking Architecture

3.1 VNet & Subnet Design — Production Data Platform

The production spoke VNet (vnet-data-prod-cc) uses a /16 address space (e.g., 10.10.0.0/16) to accommodate Databricks' substantial IP requirements (VNet injection demands large subnets) and future growth.

Subnet	CIDR	Purpose	Delegation / Notes
snet-dbx-host-prod	10.10.0.0/22	Databricks cluster host VMs (VNet injection)	Delegation: Microsoft.Databricks/workspaces; NSG: nsg-dbx-host-prod
snet-dbx-container-prod	10.10.4.0/22	Databricks cluster container network (VNet injection)	Delegation: Microsoft.Databricks/workspaces; NSG: nsg-dbx-container-prod
snet-private-endpoints	10.10.8.0/24	Private endpoints for ADLS, Key Vault, Purview, Event Hub	No delegation; NSG restricts inbound to platform VNets only
snet-sqlwarehouse-prod	10.10.9.0/24	Databricks Serverless SQL Warehouse connectivity	Network Connectivity Config (NCC) for serverless compute
snet-adf-prod	10.10.10.0/24	ADF Integration Runtime (self-hosted, if needed for on-prem sources)	VMs running IR software
snet-services-prod	10.10.11.0/24	Supporting services, internal load balancers, utility VMs	General-purpose

Sizing note: Databricks VNet injection requires /22 or larger subnets for host and container in production to support auto-scaling across multiple concurrent clusters (data engineering, SQL warehouses, ML training). Each running node consumes 2 IPs (one host, one container). A /22 provides ~1,022 usable IPs per subnet, supporting approximately 500 concurrent nodes — sufficient for Greenfield' production workload with headroom.

3.2 Private Endpoints

Every platform service that supports private endpoints must be configured with public network access disabled. This is non-negotiable for a regulated financial institution.

Service	Private Endpoint Sub-Resource	Private DNS Zone
ADLS Gen2 (each account × 2)	dfs, blob	privatelink.dfs.core.windows.net, privatelink.blob.core.windows.net
Azure Key Vault (each instance)	vault	privatelink.vaultcore.azure.net
Microsoft Purview	account, portal	privatelink.purview.azure.com, privatelink.purviewstudio.azure.com
Databricks Workspace	databricks_ui_api	privatelink.azuredatabricks.net
Azure Event Hub	namespace	privatelink.servicebus.windows.net
Azure Data Factory	dataFactory, portal	privatelink.datafactory.azure.net, privatelink.adf.azure.com
Azure Container Registry (for SAS)	registry	privatelink.azurecr.io
Databricks Unity Catalog Metastore	(via workspace PE)	Covered by Databricks workspace PE

Total private endpoints (production): Approximately 25–30 PEs across all services. Each PE creates a NIC in snet-private-endpoints with a private IP, plus a DNS A record in the corresponding Private DNS Zone.

3.3 Network Security Groups (NSGs)

NSGs are applied at the subnet level with a default deny-all posture. Explicit allow rules are created only for required traffic flows.

Databricks host & container subnets:

• Allow all inbound/outbound between snet-dbx-host-prod and snet-dbx-container-prod (required for Spark executor ↔ driver communication)
• Allow outbound to snet-private-endpoints on ports 443 (ADLS HTTPS), 443 (Key Vault), 1433 (if Azure SQL is used)
• Allow outbound to Databricks control plane IPs (published by Databricks, region-specific) on port 443
• Allow outbound to Azure Service Tags (AzureActiveDirectory, AzureMonitor) for authentication and telemetry
• Deny all inbound from internet
• Deny all other outbound (forced through Azure Firewall via UDR)

Private endpoint subnet:

• Allow inbound from snet-dbx-host-prod, snet-dbx-container-prod, snet-adf-prod, snet-services-prod, and vnet-sas-prod-cc (via peering) on service-specific ports
• Deny inbound from all other sources
• No outbound rules needed (PEs are inbound-only destinations)

3.4 Azure Firewall Rules

The hub Azure Firewall controls all egress from spoke VNets. Key application rule collections:

Rule Collection	FQDN Targets	Protocol	Purpose
rc-databricks-control	*.azuredatabricks.net, Databricks control plane IPs	HTTPS (443)	Databricks workspace communication with Azure-managed control plane
rc-azure-services	login.microsoftonline.com, management.azure.com, *.monitor.azure.com	HTTPS (443)	Entra ID authentication, ARM management, Azure Monitor telemetry
rc-package-repos	pypi.org, files.pythonhosted.org, repo.anaconda.com, conda.anaconda.org	HTTPS (443)	Python/Conda package installation for Databricks clusters and SAS
rc-databricks-artifacts	dbartifactsprodcac.blob.core.windows.net (region-specific)	HTTPS (443)	Databricks runtime artifacts, libraries, Spark distributions
rc-sas-licensing	SAS licensing endpoints (provided by SAS)	HTTPS (443)	SAS Viya license validation
rc-deny-all	*	Any	Default deny — all egress not matching an explicit rule is blocked and logged

3.5 Connectivity to On-Premises

Source systems (core banking on DB2, Oracle, SQL Server; insurance policy administration; CRM; mainframe extracts) connect to the Azure data platform through ExpressRoute. ADF pipelines and Databricks Auto Loader access on-premises sources through the hub VNet's ExpressRoute gateway, with traffic routing controlled by BGP route propagation and UDRs. Self-hosted Integration Runtimes (in snet-adf-prod) are deployed for sources that require a local agent.

---

4. Identity & Access Management

4.1 Azure Entra ID as the Identity Foundation

All platform access is brokered through Azure Entra ID (formerly Azure Active Directory) with mandatory multi-factor authentication enforced via Conditional Access policies. Conditional Access policies require: MFA for all users, compliant device, access from Greenfield corporate network or approved VPN locations, and session lifetime controls (re-authentication every 12 hours for interactive sessions).

Component	Identity Integration	Authentication Method
Databricks Workspaces	Entra ID SSO via SCIM provisioning; Entra security groups synced to Unity Catalog for RBAC	OAuth 2.0 / SAML; MFA via Conditional Access
Microsoft Fabric	Native Entra ID integration via Microsoft 365 tenant	Entra ID SSO; MFA enforced
SAS Viya	Entra ID integration via SAML 2.0 federation or direct OIDC (depending on SAS version)	SAML 2.0 federated with Entra ID; MFA enforced
Microsoft Purview	Native Entra ID; Purview collection-level roles mapped to Entra security groups	Entra ID SSO
Azure Data Factory	Managed identity for pipeline execution; Entra groups for authoring RBAC	System-assigned Managed Identity
ADLS Gen2	Entra ID RBAC at Azure resource level (Storage Blob Data roles) + Unity Catalog ACLs at the data layer	Managed Identity / Service Principal

4.2 RBAC Model — Azure Resource Layer

Azure RBAC provides coarse-grained access at the resource level. Data-level fine-grained access (RLS, CLS, DDM) is enforced by Unity Catalog at the compute layer.

Entra Security Group	Azure RBAC Role	Scope
sg-data-platform-admins	Contributor	sub-data-platform-prod, sub-data-platform-nonprod
sg-data-engineers	Databricks Contributor, Storage Blob Data Contributor	Databricks workspaces, ADLS Gen2 accounts
sg-data-scientists	Databricks Contributor (sandbox workspace only), Storage Blob Data Reader	Sandbox workspace, Silver/Gold ADLS (read)
sg-data-analysts	Reader	Databricks analytics workspace (SQL Warehouse access via Unity Catalog grants)
sg-governance-admins	Purview Data Curator, Key Vault Administrator	Purview account, Key Vault
sg-sas-developers	Contributor	sub-data-sas-prod
sg-fabric-admins	Fabric Capacity Administrator	Fabric capacity resource
sg-finops	Cost Management Reader, Billing Reader	All data platform subscriptions

4.3 Service Principals & Managed Identities

Automated workloads use managed identities (preferred) or service principals (where managed identities are not supported). Shared secrets and embedded credentials are explicitly prohibited per the reference architecture.

Workload	Identity Type	Scope / Role Assignments
Databricks workspace (system)	System-assigned Managed Identity	Storage Blob Data Contributor on ADLS Gen2 accounts; Key Vault Secrets User on kv-data-platform-prod
ADF pipelines	System-assigned Managed Identity	Storage Blob Data Contributor on ADLS; Databricks workspace access via linked service token
SAS Viya Compute Server	Service Principal (sp-sas-compute-prod)	Storage Blob Data Reader on authorized ADLS paths only (non-sensitive datasets); JDBC access to Databricks SQL Warehouses scoped via Unity Catalog grants
Purview scanners	System-assigned Managed Identity	Storage Blob Data Reader on all ADLS accounts; Databricks metastore reader for Unity Catalog metadata harvesting
Manta lineage engine	Service Principal (sp-manta-prod)	Read-only access to Databricks repos, ADF metadata APIs, SAS code repositories; Purview Data Curator for lineage publishing
CI/CD pipelines (Terraform)	Service Principal (sp-terraform-prod)	Contributor on data platform subscriptions; User Access Administrator for RBAC assignments

SAS read-path security enforcement: Because SAS Compute Server accesses ADLS files using its service principal's Azure RBAC permissions (bypassing Unity Catalog's fine-grained RLS/CLS/DDM), the sp-sas-compute-prod service principal is granted Storage Blob Data Reader only on non-sensitive, pre-authorized ADLS paths. Those paths are registered as Unity Catalog external locations with restricted scope. For sensitive datasets, SAS must read through JDBC LIBNAME to Databricks SQL Warehouses, where Unity Catalog enforcement applies at query time.

---

5. Compute Sizing & Configuration

5.1 Databricks Workspaces

Workspace	Environment	Purpose	Configuration
dbw-data-eng-prod	Production	Data engineering: Bronze→Silver→Gold DLT pipelines, Auto Loader streaming ingestion, data quality workflows	Unity Catalog enabled; VNet injected; cluster policies enforce approved instance types (Standard_DS4_v2 8 vCPU/28 GB as default workers, Standard_DS5_v2 16 vCPU/56 GB for memory-intensive), auto-terminate 30 min, spot instances (60/40 spot/on-demand) for non-critical batch jobs; Photon enabled for DLT pipelines
dbw-analytics-prod	Production	SQL analytics, Databricks AI/BI Dashboards, Genie NL querying, ad-hoc analyst queries	Serverless SQL Warehouses (preferred); Pro SQL Warehouses for workloads requiring Classic compute; auto-suspend 10 min
dbw-mlops-prod	Production	ML/AI: MLflow experiments, model training, Feature Store serving, Model Serving endpoints, GenAI workloads	GPU instance pools (Standard_NC6s_v3 6 vCPU/112 GB/1× V100 for training; Standard_NC4as_T4_v3 for inference); CPU pools for feature engineering; Model Serving auto-scale (min replicas: 0 for dev, 1 for prod SLA endpoints)
dbw-data-eng-dev	Non-Prod	Development and testing of data engineering pipelines	Reduced cluster sizes (max 4 workers); mandatory auto-terminate 15 min; restricted to Standard_DS3_v2; Unity Catalog dev metastore (separate from prod)
dbw-sandbox	Non-Prod	Data science exploration, POCs, experimentation	Read-only access to Silver/Gold via Unity Catalog; no production write access; budget cap enforced ($X/month per user via tag-based cost alerts); auto-terminate 10 min

Cluster policy governance: All workspaces enforce cluster policies that prevent creation of non-compliant clusters. Policies define: allowed instance types (approved VM families only), maximum workers per cluster (10 for dev, 50 for prod data eng, 20 for ML), mandatory auto-termination, mandatory Unity Catalog mode (no legacy clusters), spot instance minimum ratio, custom tags (CostCenter, DataDomain required), and prohibited features (no local file system access in prod, no public IP assignment).

5.2 Databricks SQL Warehouses

SQL Warehouse	Size (DBU)	Purpose	Configuration
sqlwh-bi-serving	Medium (4–8 DBU base)	JDBC endpoint consumed by SAS Compute Server (JDBC LIBNAME) and indirectly by Power BI (via OneLake shortcut refresh queries)	Serverless; auto-suspend 10 min; scaling policy: economy (cost-optimized, queues queries rather than scaling aggressively); spot instances enabled; query result caching enabled
sqlwh-analytics	Small–Medium (2–8 DBU)	Ad-hoc analyst queries via SQL editor, JDBC/ODBC for downstream applications, notebook SQL exploration	Serverless; auto-suspend 10 min; query queuing enabled; intelligent workload management
sqlwh-etl-support	Medium–Large (4–16 DBU)	Data quality validation queries run during pipeline execution, data contract SLA checks (freshness, completeness), Gold layer aggregation queries	Pro; scheduled scaling aligned with pipeline windows (04:00–08:00 EST peak for T+1 processing); auto-suspend after pipeline completion

5.3 SAS Viya Compute Server

SAS Viya runs on Compute Server engine (not CAS — this is AD-04). Processing is sequential and batch-oriented, so compute is sized for single-threaded throughput with high memory for in-process data manipulation, not for horizontal parallelism.

Deployment option: AKS (recommended)

Component	VM Size / Pod Spec	Quantity	Notes
SAS Compute Server pods	Standard_E16s_v5 (16 vCPU, 128 GB RAM)	2–4 pods	Primary workhorses for actuarial model development, risk model validation, moderate batch scoring; memory-optimized for SAS DATA step and PROC SQL in-memory operations
SAS Programming Runtime (heavy)	Standard_E32s_v5 (32 vCPU, 256 GB RAM)	1–2 pods	Large actuarial reserving models (IFRS 17, IBNR) requiring extended memory; scheduled availability during nightly batch windows to control cost
SAS License Server	Standard_D4s_v5 (4 vCPU, 16 GB RAM)	1 (HA pair)	SAS license management service; always-on
SAS Model Manager	Standard_D8s_v5 (8 vCPU, 32 GB RAM)	1–2 pods	Model governance, model registration, champion/challenger tracking for regulatory model management
AKS System Node Pool	Standard_D4s_v5	3 nodes (across AZs)	AKS system services (CoreDNS, kube-proxy, metrics-server)

AKS configuration: Private AKS cluster (no public API endpoint); Azure CNI networking for VNet integration in vnet-sas-prod-cc; Entra ID integration for RBAC; node pool auto-scaling (min 2 / max 6 for compute pods); Azure Disk CSI driver for persistent volumes (SAS work directories); Azure Files for shared SAS configuration.

5.4 Microsoft Fabric Capacity

Fabric capacity is sized strictly for the BI serving workload (AD-03). Any request to increase capacity for data engineering or warehousing triggers an architecture review per the reference architecture's anti-pattern rules.

Capacity SKU	CU	Use Case	Governance
F64 (Production)	64 Capacity Units	Power BI Direct Lake semantic models serving 55,000 users, report rendering, paginated reports for regulatory distribution, Copilot-assisted analytics	Auto-pause during non-business hours (22:00–06:00 EST); smoothing enabled for burst absorption; capacity alerts at 70% and 90% utilization; Fabric admin monitors for non-BI workloads
F32 (Non-Production)	32 CU	Development and testing of Power BI semantic models, report prototyping, Direct Lake connectivity validation	Auto-pause aggressive (off outside business hours); BI workloads only; no Fabric notebooks or Spark
F16 (Fabric IQ POC — Horizon 2)	16 CU	Ontology POC on Customer 360 domain, Data Agent evaluation with controlled user group (per Section 4.5 prerequisites)	Isolated capacity; time-limited (6-month evaluation); requires Architecture Review Board approval before provisioning

Cost note: F64 at list price is approximately $8,000–9,000 USD/month. This is significantly less than the Databricks DBU cost that would be incurred if 55,000 Power BI users queried Databricks SQL Warehouses via DirectQuery — the entire economic justification for Fabric in this architecture (AD-03).

---

6. Storage Architecture

6.1 ADLS Gen2 Account Layout

ADLS Gen2 is the shared storage substrate across all three platforms (AD-07). All accounts have hierarchical namespace (HNS) enabled for Delta Lake compatibility and are configured with private endpoints only (public access disabled at the account level).

Storage Account	Access Tier	Container Structure	Access Pattern
stadlsbronzeprod	Hot	/bronze/{source_system}/{entity}/ (e.g., /bronze/core_banking/accounts/, /bronze/insurance/claims/)	Write: ADF managed identity, Auto Loader managed identity; Read: Databricks data engineering workspace MI, Purview scanner MI
stadlssilverprod	Hot	/silver/{domain}/{entity}/ (e.g., /silver/customer/individual/, /silver/claims/claim_header/)	Write: Databricks DLT pipelines; Read: Databricks analytics & ML workspaces, SAS via JDBC (Unity Catalog enforced)
stadlsgoldprod	Hot	/gold/{data_product}/{entity}/ (e.g., /gold/customer_360/member_profile/, /gold/risk_features/credit_scores/)	Write: Databricks DLT/SQL pipelines; Read: Power BI (OneLake shortcut → Direct Lake), Databricks SQL, SAS JDBC LIBNAME, ML Feature Store, REST API services
stadlsstagingprod	Hot	/staging/ingestion/{source}/, /staging/sas_writeback/{model_domain}/{model_name}/{run_date}/, /staging/purview_dq/	Write: ADF (ingestion landing), SAS ADLS LIBNAME (writeback staging); Read: Purview DQ scanner (pre-Bronze assessment), Databricks promotion pipelines
stadlsarchiveprod	Cool → Archive	/archive/{domain}/{entity}/{year}/	Write: Lifecycle policy tiering; Read: Rare, on-demand rehydration for regulatory retrieval

Why separate storage accounts (not just separate containers)? Separate accounts provide independent RBAC boundaries (the SAS service principal has no RBAC on stadlsbronzeprod), independent throughput limits (each account gets its own IOPS/bandwidth quota), independent private endpoints (enabling subnet-level NSG control), independent lifecycle policies, and independent diagnostic logging.

6.2 Delta Lake Configuration

All data within Bronze, Silver, and Gold containers is stored in Delta Lake format (AD-01).

Time Travel retention: 90 days minimum for Bronze (regulatory auditability per AMF requirements); 30 days for Silver and Gold (sufficient for pipeline replay and debugging), extended to 90 days for regulatory domain Gold tables (Financial Aggregates, Risk Features).

VACUUM & OPTIMIZE schedule (via Databricks Workflows):

• Bronze: VACUUM retaining 90 days, runs weekly (low urgency, append-only)
• Silver: VACUUM retaining 30 days, runs nightly; OPTIMIZE with ZORDER BY on business keys, runs nightly
• Gold: VACUUM retaining 30 days, runs nightly; OPTIMIZE with LIQUID CLUSTERING on high-cardinality filter columns (date, business_unit, product_code, member_id), runs nightly post-refresh; table statistics maintained via ANALYZE TABLE for SQL Warehouse query optimizer

Schema evolution: mergeSchema enabled on Bronze tables (Auto Loader schema inference accommodates upstream changes without pipeline failures). Silver and Gold tables enforce strict schemas via DLT expectations and data contracts — schema-breaking changes require a data product lifecycle change request.

6.3 Lifecycle Policies

Storage Account	Policy Rule	Action / Rationale
stadlsbronzeprod	Blobs not modified in 180 days → Cool tier; not modified in 365 days → Archive tier	Cost optimization for aged raw data while maintaining regulatory access; Delta Time Travel handles recent rollback needs
stadlsstagingprod	Ingestion staging blobs > 30 days → delete; SAS writeback blobs > 14 days after promotion validation → delete	Prevent staging zone bloat; staging data is ephemeral by design
stadlsarchiveprod	Archive tier by default; rehydrate on demand (Standard priority, 15 hours)	Lowest cost for long-term regulatory retention; rare access pattern
All accounts	Soft delete enabled (14-day retention); blob versioning enabled; container soft delete (7 days)	Accidental deletion recovery baseline

6.4 Encryption

All storage accounts use encryption at rest with Microsoft-managed keys (MMK) as the baseline. For data classified as Restricted (per Purview auto-classification — e.g., NAS/SIN numbers, certain financial details), customer-managed keys (CMK) stored in Azure Key Vault (kv-data-encryption-prod) are applied at the storage account level. In practice, this means stadlssilverprod and stadlsgoldprod use CMK (these contain processed, queryable sensitive data), while stadlsbronzeprod uses MMK (raw data is further protected by restricted RBAC — only data engineering roles have access).

Encryption in transit: TLS 1.2 minimum enforced on all storage endpoints. Secure transfer required = true on all accounts.

---

7. Security Architecture

7.1 Azure Key Vault

Key Vault Instance	Purpose	Access Policy
kv-data-platform-prod	Platform operational secrets: database connection strings, SAS license keys, API keys for external data feeds, ADF linked service credentials	Databricks MI, ADF MI, SAS SP → Key Vault Secrets User role; CI/CD SP → Key Vault Administrator (for secret rotation automation)
kv-data-encryption-prod	Customer-managed encryption keys: ADLS CMK, Databricks workspace CMK (managed services encryption, DBFS encryption)	Storage account MI, Databricks workspace MI → Key Vault Crypto User role; Key Vault admin group → Key Vault Administrator
kv-data-platform-nonprod	Non-production secrets (same structure, isolated values)	Non-prod workspace MIs and SPs

Hardening:

• Key Vault Firewall: enabled; access restricted to private endpoint only (from snet-private-endpoints)
• Purge protection: enabled on all instances (prevents permanent deletion, even by admins, for 90 days)
• Soft delete: enabled with 90-day retention
• Diagnostic settings: all Key Vault operations streamed to Log Analytics; alert rules configured for unexpected access patterns (access from unknown service principals, bulk secret reads, key deletion attempts)
• Secret rotation: automated rotation policies for secrets with 90-day expiry; CI/CD pipeline rotates service principal credentials quarterly

7.2 Private Endpoints — Defense in Depth

All platform services operate exclusively through private endpoints. Public endpoints are disabled at the resource level and blocked by Azure Policy. Network traffic between services never traverses the public internet.

The full private endpoint inventory (approximately 25–30 PEs in production) is detailed in Section 3.2. Each PE creates: a NIC with private IP in snet-private-endpoints, a DNS A record in the corresponding Private DNS Zone, and NSG rules governing which subnets can reach it.

7.3 Data Encryption — Layers

Layer	Encryption	Key Management
At rest — ADLS Gen2	AES-256; MMK (default) or CMK for Restricted data	CMK in kv-data-encryption-prod; auto-rotation annually
At rest — Databricks DBFS	AES-256 with CMK	CMK in kv-data-encryption-prod
At rest — Databricks managed services	CMK for notebook results, job results, ML artifacts	Workspace-level CMK configuration
In transit — all services	TLS 1.2 minimum	Azure-managed certificates; Databricks enforces TLS for all cluster ↔ storage and cluster ↔ control plane communication
In transit — ExpressRoute	MACsec (Layer 2 encryption at peering)	Optional but recommended for Express Route Direct circuits

7.4 Databricks Workspace Security Configuration

Security Control	Configuration Detail
Workspace encryption	CMK from kv-data-encryption-prod for managed services and DBFS
Cluster policies	Mandatory: enforce approved instance types, auto-termination, spot ratios, max workers, Unity Catalog mode, custom tags; prohibit legacy table ACLs, local file download, init scripts from untrusted sources
Unity Catalog	External metastore on dedicated ADLS storage; all data access routed through UC; legacy Hive metastore disabled; default catalog = none (users must specify catalog explicitly)
Token management	Personal Access Tokens (PATs) disabled in production workspaces; service principals use OAuth 2.0 M2M (client credentials flow)
IP access lists	Workspace accessible only from Greenfield corporate network IP ranges (via ExpressRoute) and Azure Bastion subnet
Audit logging	All workspace audit logs streamed to Log Analytics via diagnostic settings; captures: data access events, cluster lifecycle, admin configuration changes, SQL query history, secrets access
Secret scopes	All Databricks secret scopes backed by Azure Key Vault (kv-data-platform-prod); Databricks-native secret storage disabled
Secure cluster connectivity (No Public IP)	Enabled — cluster nodes have no public IPs; all communication via VNet and private links

7.5 Data Loss Prevention & Exfiltration Controls

• Egress restriction: All compute cluster outbound traffic routes through Azure Firewall; only whitelisted FQDNs are permitted (Section 3.4)
• DBFS download restriction: Local file download from Databricks notebooks disabled in production via workspace admin settings
• Purview DLP: Microsoft Purview DLP policies integrated with M365 prevent sensitive data from leaving via email, Teams, or SharePoint
• Clipboard/export controls: Fabric Power BI tenant settings restrict data export from reports (no CSV export of Restricted-classified datasets)

---

8. Monitoring & Observability

8.1 Centralized Monitoring Stack

All monitoring data converges on a central Log Analytics workspace (law-data-platform-prod in sub-data-management) to provide a single-pane-of-glass view.

Component	Technology	Data Collected
Infrastructure	Azure Monitor metrics + Log Analytics	VM metrics (CPU, memory, disk), AKS node/pod metrics, storage account metrics (transactions, latency, capacity), network flow logs (NSG flow logs v2), Azure Firewall logs (application + network rules)
Databricks	Databricks audit logs → Diagnostic Settings → Log Analytics; Databricks system tables (system.billing.usage, system.compute.clusters, system.query.history)	Cluster utilization & idle time, job run durations/statuses/failures, SQL Warehouse query performance (duration, bytes scanned, cache hit ratio), DBU consumption by workspace/cluster/job/user, Unity Catalog audit trail (who accessed what data)
Data pipelines	ADF monitoring → Log Analytics; Databricks Workflows run logs	Pipeline run status, duration, row counts, error details; DLT pipeline quality metrics (rows passed/failed expectations); ingestion audit log (source, timestamp, row counts, schema version)
Data quality	Purview DQ scores (Tier 1) + DLT expectation metrics (Tier 2–3) → unified DQ dashboard (Power BI)	Quality scores by domain, source system, data product; quarantine volumes; SLA compliance trends; CDE completeness tracking
Security / SIEM	Microsoft Sentinel connected to Log Analytics	Security events aggregated from all services; anomalous data access patterns; DLP alerts; Key Vault access audit; failed authentication attempts; service principal credential anomalies
Cost	Azure Cost Management + Databricks Account Console	Subscription-level cost by resource group and tag; Databricks DBU consumption by workspace, cluster type, job, and user; Fabric CU consumption; storage growth trends

8.2 Key Dashboards

Dashboard	Audience	Content
Platform Health	Platform engineering team	Cluster availability, pipeline success rates, SQL Warehouse queue times, storage latency, private endpoint health, AKS node status
Data Quality Governance	CDO office, data stewards	Unified DQ scores (all three tiers), quarantine volumes, SLA compliance by data product, CDE completeness trends, steward action items
FinOps & Cost	FinOps team, CDO office	DBU spend by team/domain, Fabric CU utilization, storage growth, reserved vs. on-demand ratio, cost anomaly detection, budget burn rate
Security & Compliance	Security team, CISO office	Sentinel incident summary, data access audit summary, DLP alert trends, service principal activity, Key Vault operations

8.3 Alerting Strategy

Alert Category	Trigger	Severity	Action
Pipeline failure	Production ingestion or DLT pipeline fails	Sev 1 (critical domains: Customer 360, Financial Aggregates, Risk Features) / Sev 2 (others)	Page on-call data engineer; auto-retry with exponential backoff (3 retries); escalate to lead if all retries fail
Data quality SLA breach	Gold data product fails freshness SLA (e.g., T+1 by 06:00 EST not met) or completeness threshold (<99.5% on CDEs)	Sev 1	Notify data product owner + steward; defer Gold refresh; CDO office escalation if unresolved within 4 hours
Security anomaly	Unexpected SP data access; bulk data download; Key Vault access from unknown identity	Sev 1	Sentinel incident auto-created; SOC investigation; auto-block identity if high-confidence threat
Cluster over-provisioning	Databricks cluster idle >30 min or CPU utilization <20% for sustained period	Sev 3	Notify FinOps team; recommend right-sizing; auto-terminate if policy allows
Fabric capacity saturation	CU utilization >90% sustained for 1 hour	Sev 2	Notify Fabric admin; evaluate workload scheduling or capacity burst
Storage anomaly	Storage account growth rate exceeds 2× historical 7-day baseline	Sev 3	Notify data engineering lead; investigate potential staging bloat or data duplication

---

9. Disaster Recovery & Business Continuity

9.1 Recovery Objectives

Tier	Workloads	RPO	RTO	Strategy
Tier 1 — Critical	Gold data products (Customer 360, Financial Aggregates, Risk Features), regulatory reporting pipelines, active ML model serving endpoints	≤ 1 hour	≤ 4 hours	GRS/RA-GRS storage replication to Canada East; Databricks workspace deployable via IaC in <2 hours; pre-provisioned standby networking in Canada East; SQL Warehouse endpoints re-creatable from IaC
Tier 2 — Important	Silver layer, ingestion pipelines (ADF + Auto Loader), SAS Viya actuarial models, Feature Store	≤ 4 hours	≤ 8 hours	GRS storage replication; IaC-based compute rebuild in secondary region; SAS model code in version control (Git); ADF pipeline definitions exported and version-controlled
Tier 3 — Standard	Bronze layer, development/sandbox environments, non-production workloads	≤ 24 hours	≤ 24 hours	GRS storage (async replication); rebuild from IaC; accept data loss up to last successful replication point

9.2 Storage Replication

All production ADLS Gen2 accounts are configured with Geo-Redundant Storage (GRS), replicating data asynchronously to Canada East. For Tier 1 storage accounts (stadlsgoldprod), Read-Access GRS (RA-GRS) is enabled, allowing read access from the secondary region during a regional outage without waiting for Microsoft-initiated failover.

Delta Lake's transaction log (_delta_log/) is included in GRS replication. After failover, replicated Delta tables are query-consistent — the transaction log ensures that only committed transactions are visible, even if replication was mid-flight during the outage.

9.3 Compute Recovery

All infrastructure is defined as code (Terraform — see Section 12). In a DR scenario:

1. Networking: Standby hub-spoke VNets in Canada East are pre-provisioned (warm standby) with peering, NSGs, and route tables ready. ExpressRoute circuit has a secondary connection to Canada East.
2. Databricks: Workspace configuration, cluster policies, Unity Catalog metastore settings, and secret scope bindings are all in Terraform. A terraform apply targeting the Canada East module deploys a functional workspace in ~60–90 minutes.
3. SAS Viya: AKS cluster definition is in Terraform; SAS container images are in Azure Container Registry (geo-replicated to Canada East). Deployment time ~2–3 hours including SAS configuration restore.
4. Fabric: Fabric capacity is region-specific. A standby F-SKU in Canada East can be provisioned on-demand (manual, ~30 minutes). Power BI semantic models would need to be re-pointed to the secondary ADLS endpoints.

9.4 Backup Strategy

Component	Backup Mechanism	Retention
Delta Lake data	Delta Time Travel (point-in-time recovery within retention window) + GRS replication	90 days Bronze, 30 days Silver/Gold
Azure Key Vault	Soft delete (90 days) + purge protection; secrets/keys recoverable even after deletion	90 days
Unity Catalog metastore	Databricks account-level configuration backup + IaC definitions	Recoverable from IaC
Purview configuration	Glossary, classification schemas, and policy definitions exported periodically to version control (JSON export)	Git history (indefinite)
Pipeline definitions	ADF ARM templates in Git; Databricks notebooks in Git (Repos); DLT pipeline code in Git	Git history (indefinite)
SAS code & models	SAS programs, macros, model code in Git; SAS Model Manager metadata exported periodically	Git history (indefinite)

DR testing cadence: Tabletop exercise quarterly; partial failover test (storage failover + compute redeploy for one domain) semi-annually; full DR simulation annually.

---

10. Cost Management & FinOps

10.1 Cost Allocation Architecture

Cost visibility is built into the infrastructure from day one through mandatory tagging (enforced by Azure Policy), subscription-level isolation, and platform-native cost tools.

Mandatory tags (enforced via Azure Policy — deny if missing):

Tag Key	Purpose	Example Values
Environment	Environment segregation	prod, nonprod, sandbox, dr
CostCenter	Chargeback to business unit	CC-CDO-1234, CC-INS-5678, CC-RISK-9012
Platform	Technology pillar identification	databricks, fabric, sas, governance, shared
Owner	Operational contact	team-data-eng@Greenfield.com
DataDomain	Domain-level cost attribution	customer, claims, risk, finance, shared
DataClassification	Align cost with sensitivity tier	public, internal, confidential, restricted

Recommended tags (enforced via Azure Policy — audit mode):

Tag Key	Purpose	Example Values
ManagedBy	IaC tool tracking	terraform, manual, bicep
Horizon	Implementation phase tracking	h1, h2, h3
ExpiryDate	Temporary resource cleanup	2026-09-30 (for POCs, sandboxes)

10.2 Budget Controls

Subscription-level budgets: Set for each subscription with alerts at 50%, 75%, 90%, and 100% of monthly budget. Breaches above 100% trigger automatic notification to CDO office and FinOps team.

Databricks cost controls:

• Cluster policies enforce max cluster sizes, auto-termination, and spot instance ratios
• Databricks Account Console provides DBU consumption dashboards by workspace, cluster type, job, and user
• Databricks budgets feature (if available) configured per workspace with alerts
• SQL Warehouses configured with economy scaling to minimize DBU for BI serving workload

Fabric capacity governance:

• Capacity sized strictly for BI serving only (AD-03)
• Auto-pause enabled (22:00–06:00 EST) — saves ~33% of capacity cost
• Smoothing enabled for burst absorption without over-provisioning
• Any request to increase capacity for non-BI workloads triggers architecture review
• Monthly Fabric utilization review by FinOps + Fabric admin

Storage cost controls:

• Lifecycle policies (Section 6.3) auto-tier aged data to lower-cost tiers
• Storage growth alerts (Section 8.3) detect anomalous growth
• Monthly review of staging zone sizes to prevent bloat

10.3 Reserved Capacity & Savings Plans

Resource	Commitment Type	Term	Estimated Savings
Databricks DBU	Databricks Committed Use Discount (DBCU)	1-year (based on 6-month usage baseline after Horizon 1 stabilization)	20–35% vs. pay-as-you-go
Azure VMs (SAS Viya AKS nodes)	Azure Savings Plan for Compute	1-year	15–25% vs. pay-as-you-go
ADLS Gen2 Storage	Azure Reserved Capacity (hot tier)	1-year (based on projected growth model)	Up to 30% on hot tier
Fabric Capacity	Fabric reservation (evaluate availability)	Evaluate after 6 months of usage data	TBD at GA pricing
ExpressRoute	ExpressRoute circuit commitment	Already committed (enterprise)	N/A — existing circuit

10.4 Chargeback Model

Cost is allocated to business units via CostCenter tags. The CDO's FinOps practice produces monthly chargeback reports:

• Direct costs (Databricks DBU, Fabric CU, SAS compute) → allocated to the business unit whose workloads consumed them
• Shared costs (storage, networking, governance, monitoring) → allocated proportionally by data volume and query consumption per domain
• Platform overhead (hub networking, Key Vault, Purview, Manta) → allocated as a shared infrastructure charge across all consuming business units

---

11. DevOps & Infrastructure as Code

11.1 Terraform — Primary IaC Tool

All Azure infrastructure is managed through Terraform. Resources are organized as reusable modules stored in a central Git repository with a clear module hierarchy.

Terraform Module	Scope	State Backend
terraform-module-networking	Hub-spoke VNets, subnets, NSGs, route tables, Private DNS Zones, Azure Firewall, ExpressRoute, VNet peering	Remote state in Azure Storage (sub-data-management); state locking via blob lease
terraform-module-databricks	Databricks workspaces, cluster policies, Unity Catalog metastore, secret scopes, workspace configuration, IP access lists	Remote state per environment (prod/nonprod); separate state file per workspace
terraform-module-storage	ADLS Gen2 accounts, containers, lifecycle policies, private endpoints, RBAC role assignments, CMK configuration	Remote state per environment
terraform-module-governance	Purview account, Purview collections, scan rules, Purview private endpoints, diagnostic settings	Remote state (governance)
terraform-module-keyvault	Key Vault instances, access policies, private endpoints, diagnostic settings, purge protection	Remote state (security)
terraform-module-sas	AKS cluster, node pools, AKS networking, SAS service principal, SAS storage	Remote state (SAS)
terraform-module-monitoring	Log Analytics workspace, diagnostic settings (applied to all resources), alert rules, action groups, Sentinel workspace	Remote state (monitoring)
terraform-module-fabric	Fabric capacity resource, Fabric admin configuration	Remote state (fabric)
terraform-module-policy	Azure Policy definitions and assignments at management group level (required tags, allowed regions, deny public endpoints)	Remote state (governance)

State management: All Terraform state files are stored in a dedicated storage account in sub-data-management with: blob lease-based state locking (prevents concurrent applies), versioning enabled (state history for rollback), encryption at rest (MMK), access restricted to CI/CD service principal only.

11.2 CI/CD Pipeline Design

Pipelines run in Azure DevOps (Greenfield standard). Self-hosted agents run in vnet-mgmt-cc to access private endpoints. The pipeline follows a 4-stage model:

Stage 1 — Validate (on every Pull Request):

• terraform fmt -check (formatting compliance)
• terraform validate (syntax and configuration validation)
• tflint (Terraform linting for best practices)
• tfsec / Checkov security scanning: validates that all resources comply with security policies (public endpoints disabled, encryption enabled, required tags present, no unmanaged secrets, private endpoints configured)
• Results posted as PR comment; PR blocked if critical findings

Stage 2 — Plan (on PR approval):

• terraform plan generates execution plan against target environment
• Plan output posted to PR as a formatted comment for human review
• No changes applied — review gate ensures human approval of all changes

Stage 3 — Apply Non-Production (on merge to develop branch):

• terraform apply deploys changes to non-production environment
• Automated smoke tests: verify VNet connectivity, private endpoint DNS resolution, Databricks workspace health, Key Vault accessibility, storage account reachability
• If smoke tests fail, automatic rollback via terraform apply with previous state

Stage 4 — Apply Production (on merge to main branch + manual approval):

• Manual approval gate (requires two approvals from platform engineering leads)
• terraform apply deploys to production
• Post-deployment validation: resource health checks, Databricks workspace connectivity, pipeline reachability, monitoring integration verification
• Deployment tagged in Git with version number and timestamp

11.3 Data Engineering CI/CD (Separate from Infrastructure)

Data engineering artifacts follow their own CI/CD lifecycle, independent of infrastructure IaC:

Artifact Type	Version Control	CI/CD Tool	Deployment Method
DLT pipeline definitions	Databricks Repos (Git-backed)	Azure DevOps pipeline	Databricks Asset Bundles (DABs) — databricks bundle deploy to prod workspace
Databricks notebooks	Databricks Repos	Azure DevOps	DABs or Repos-based deployment on merge to main
dbt models (if used)	Git repository	Azure DevOps	dbt run executed by Databricks Workflow job
ADF pipelines	ARM template export in Git	Azure DevOps	ARM template deployment to ADF instance
Great Expectations suites	Git repository	Azure DevOps	Deployed as part of pipeline package
SAS programs & macros	Git repository (external to SAS)	Azure DevOps	File deployment to SAS Compute Server shared filesystem; SAS Model Manager registration
Power BI semantic models	Power BI Desktop files in Git	Azure DevOps + Fabric REST API	Automated deployment via Fabric deployment pipelines (dev → test → prod)

Branching strategy: Gitflow — feature branches → develop (integration) → release (staging validation) → main (production). Hotfixes branch from main and merge back to both main and develop.

Testing strategy for data pipelines:

• Unit tests: DLT expectations serve as row-level quality tests; Great Expectations suites validate cross-table invariants
• Integration tests: End-to-end pipeline run in non-prod workspace against sample datasets (masked production data); validates Bronze → Silver → Gold flow, quality gates, and data contract compliance
• Regression tests: Compare Gold output schema and row counts against baseline after pipeline changes
• Performance tests: Benchmark pipeline duration and DBU consumption against historical baselines; flag regressions exceeding 20% threshold

---

Appendix A — Resource Naming Convention

General pattern: {type}-{workload}-{environment}-{region}

Resource Type	Prefix	Example
Resource Group	rg-	rg-databricks-prod-cc
Virtual Network	vnet-	vnet-data-prod-cc
Subnet	snet-	snet-dbx-host-prod
NSG	nsg-	nsg-dbx-host-prod
Storage Account (ADLS)	stadls	stadlsgoldprod (no hyphens — Azure restriction)
Key Vault	kv-	kv-data-platform-prod
Databricks Workspace	dbw-	dbw-data-eng-prod
SQL Warehouse	sqlwh-	sqlwh-bi-serving
Log Analytics	law-	law-data-platform-prod
Azure Data Factory	adf-	adf-data-platform-prod
AKS Cluster	aks-	aks-sas-viya-prod
Private Endpoint	pe-	pe-stadlsgoldprod-dfs
Managed Identity	id-	id-databricks-prod
Service Principal	sp-	sp-sas-compute-prod
Purview Account	pv-	pv-data-governance-prod
Fabric Capacity	fc-	fc-bi-serving-prod
Azure Firewall	afw-	afw-hub-canadacentral

Region abbreviations: cc = Canada Central, ce = Canada East.

---

This covers the full Azure infrastructure deployment design for the Greenfield Modern Data Platform. When you're ready, I can generate this as a formatted .docx document for download — the computer tools appear to have been experiencing intermittent issues, but I'm happy to retry whenever you'd like.