# Phase 0 — Foundation: Detailed Implementation Plan

**Parent document:** Databricks-Primary Implementation Plan (MDP v8.0)
**Duration:** Weeks 1–3 (15 working days)
**Objective:** Establish the Azure landing zone, network security, Databricks workspace, Unity Catalog metastore, and IaC foundation. At the end of Phase 0, the platform is secure, governed, and ready for data engineering workloads — but no data flows yet.

---

## 1. Week-by-Week Breakdown

### Week 1 — Azure Landing Zone & Networking

| Day | Task | Owner | Details |
|---|---|---|---|
| D1 | Subscription provisioning | Cloud Infra | Create subscription: `greenfield-mdp-prod` (Canada Central). Apply Greenfield management group policies (tagging, allowed regions, allowed SKUs). DR and DevTest subscriptions are out of scope for now. |
| D1 | Resource group structure | Cloud Infra | Create resource groups per component: `rg-mdp-network`, `rg-mdp-databricks`, `rg-mdp-storage`, `rg-mdp-governance`, `rg-mdp-keyvault`, `rg-mdp-monitoring`. |
| D1–D2 | VNet design & deployment | Cloud Infra | See §2 Network Architecture below. Deploy via Terraform. |
| D2–D3 | Private Endpoints | Cloud Infra | Deploy Private Endpoints for: ADLS Gen2, Key Vault, Purview, Databricks (back-end), Databricks (front-end via Transit VNet). Register private DNS zones in central DNS hub (if Greenfield uses hub-spoke). |
| D3 | NAT Gateway | Cloud Infra | Attach NAT Gateway to both Databricks subnets (host + container). Provides stable egress IPs for allowlisting at external source systems. |
| D3–D4 | NSG rules | Cloud Infra / Security | Apply NSGs to Databricks subnets. Allow Databricks control plane service tags. Deny all other inbound. Restrict outbound to ADLS, Key Vault, Purview, and Databricks control plane only. See §3 NSG Rule Set. |
| D4 | ADLS Gen2 storage account | Cloud Infra | Deploy storage account: `stglobal<env>mdp` with HNS enabled, private endpoint, CMK (Key Vault-backed), soft delete (7 days), immutability policy on archive container. Container layout: `landing`, `bronze`, `silver`, `gold`, `archive`, `checkpoints`. |
| D4 | Key Vault | Cloud Infra | Deploy Key Vault with private endpoint, soft delete, purge protection. Create initial secrets: Databricks PAT (placeholder), source system JDBC credentials (placeholders). Enable diagnostic logging to Log Analytics. |
| D5 | Terraform state & CI/CD | DevOps | Azure Storage account for Terraform state (separate subscription, blob versioning, container-level lease lock). Azure DevOps / GitHub Actions pipeline: `terraform plan` on PR, `terraform apply` on merge to `main`. Branch protection rules enforced. |
| D5 | Security review checkpoint | Security Architect | Review: VNet topology, NSG rules, Private Endpoint configuration, RBAC assignments. Sign-off before proceeding to Databricks deployment. |

---

### Week 2 — Databricks Workspace & Identity

| Day | Task | Owner | Details |
|---|---|---|---|
| D6 | Databricks account setup | Platform Admin | Configure Azure Databricks account via account console. Enable Unity Catalog at account level. Set account-level admins (Entra ID group: `grp-mdp-account-admins`). |
| D6 | Workspace deployment | Cloud Infra | Deploy Databricks workspace via Terraform (`azurerm_databricks_workspace`). Tier: **Premium** (required for UC, Private Link, IP access lists, audit logs). VNet injection: bind to pre-created host + container subnets. Managed resource group: `rg-mdp-databricks-managed` (auto-created by Databricks, do not modify). Enable No Public IP (NPIP) for secure cluster connectivity. |
| D6–D7 | Private Link configuration | Cloud Infra | **Back-end Private Link:** Private endpoint from Databricks VNet to control plane (workspace, DBFS, artifact). **Front-end Private Link:** Private endpoint in Transit VNet for user browser/API access. DNS: register `adb-<workspace-id>.azuredatabricks.net` in private DNS zone. |
| D7 | IP access lists | Platform Admin | Configure workspace IP access list to allow only Greenfield corporate IP ranges and the Transit VNet CIDR. Block all other access. |
| D7 | Conditional Access (Entra ID) | Security Architect | Entra ID Conditional Access policy for the Databricks enterprise application: require MFA, require compliant device, block access from non-compliant locations. |
| D7–D8 | Identity provisioning (SCIM) | Platform Admin | Enable **automatic identity management** (default for accounts created after Aug 2025) or configure SCIM provisioning connector in Entra ID. Sync the following Entra ID groups to Databricks account: |
| | | | `grp-mdp-account-admins` → Account Admin role |
| | | | `grp-mdp-platform-engineers` → Workspace Admin role |
| | | | `grp-mdp-data-engineers` → Workspace User, Can Manage clusters |
| | | | `grp-mdp-data-analysts` → Workspace User, SQL access only |
| | | | `grp-mdp-data-stewards` → Workspace User, UC metastore admin |
| | | | `grp-mdp-data-scientists` → Workspace User, ML runtime access |
| D8 | Workspace configuration | Platform Admin | Enable: audit logging (to Azure Monitor diagnostic settings), web terminal disabled (security), DBFS disabled for external access, repos enabled (Git integration), serverless compute enabled. |
| D8–D9 | Cluster policies | Platform Admin | Create cluster policies (see §5 Cluster Policies). Assign policies to groups: data engineers get `de-job-cluster` and `de-interactive`, analysts get `analyst-sql-only`, data scientists get `ds-ml-cluster`. |
| D9 | Git integration | DevOps | Configure workspace Git integration (Azure DevOps Repos or GitHub). Set default branch: `main`. Enable branch protection. Create repo structure for DABs (see §6). |
| D10 | Workspace validation | Platform Admin | Validate: user login via Private Link, cluster launch, notebook execution, ADLS access via managed identity, Key Vault secret scope. Document any issues. |

---

### Week 3 — Unity Catalog & Governance Foundation

| Day | Task | Owner | Details |
|---|---|---|---|
| D11 | Metastore creation | Platform Admin | Create Unity Catalog metastore for **Canada Central** region. Root storage: ADLS Gen2 container `unitycatalog` in the MDP storage account. Storage credential: **User-Assigned Managed Identity** (not service principal — supports storage behind network rules). Access connector: `databricks-access-connector-mdp-prod`. |
| D11 | Metastore assignment | Platform Admin | Assign the metastore to the production workspace. |
| D11–D12 | Catalog structure | Platform Admin / Data Steward | Create the initial catalog hierarchy: |

```
metastore: greenfield_canadacentral
├── catalog: raw               ← Bronze layer (ingested data)
│   ├── schema: customer
│   ├── schema: transactions
│   ├── schema: products
│   ├── schema: claims
│   └── schema: policies
├── catalog: curated            ← Silver layer (cleansed, conformed)
│   ├── schema: customer
│   ├── schema: transactions
│   ├── schema: products
│   ├── schema: claims
│   └── schema: policies
├── catalog: analytics          ← Gold layer (business-ready, Metric Views)
│   ├── schema: dimensional     ← Star schemas
│   ├── schema: metrics         ← Metric Views
│   ├── schema: reference       ← MDM, reference data
│   └── schema: customer360     ← Unified customer view
├── catalog: sandbox            ← Exploratory (data scientists, analysts)
│   └── schema: <per-user or per-team>
└── catalog: system             ← UC system tables (audit, lineage, billing)
```

| Day | Task | Owner | Details |
|---|---|---|---|
| D12 | External locations | Platform Admin | Register ADLS containers as UC external locations: `abfss://bronze@stglobal...`, `abfss://silver@...`, `abfss://gold@...`. Bind each to the managed identity storage credential. |
| D12 | Default grants | Data Steward | Apply baseline grants: |
| | | | `grp-mdp-data-engineers`: `ALL PRIVILEGES` on `raw`, `curated`; `SELECT` on `analytics` |
| | | | `grp-mdp-data-analysts`: `SELECT` on `curated`, `analytics`; `USAGE` on `sandbox` |
| | | | `grp-mdp-data-scientists`: `SELECT` on `curated`, `analytics`; `ALL PRIVILEGES` on `sandbox` |
| | | | `grp-mdp-data-stewards`: `ALL PRIVILEGES` on all catalogs (metastore admin) |
| D13 | Audit logging | Platform Admin | Enable UC system tables: `system.access.audit`, `system.access.table_lineage`, `system.billing.usage`. Create a Databricks SQL dashboard for: login events, permission changes, table access patterns. Route audit logs to Azure Monitor via diagnostic settings. |
| D13 | Purview registration | Governance Specialist | Register the Databricks workspace as a data source in Microsoft Purview. Configure the UC scan using managed identity authentication. Run initial metadata scan (catalogs, schemas — tables will come in Phase 1). Validate that catalog structure appears in Purview Unified Catalog. |
| D14–D15 | Phase 0 exit review | All | **Go/No-Go gate.** Review against exit criteria (§8). Security sign-off. Architecture sign-off. Handover to Phase 1 team. |

---

## 2. Network Architecture

### VNet Layout (Canada Central — Production)

| VNet / Subnet | CIDR | Purpose |
|---|---|---|
| `vnet-mdp-prod` | `10.100.0.0/16` | Main MDP VNet |
| `snet-dbx-host` | `10.100.1.0/24` | Databricks host subnet (254 IPs) |
| `snet-dbx-container` | `10.100.2.0/24` | Databricks container subnet (254 IPs) |
| `snet-private-endpoints` | `10.100.3.0/24` | Private endpoints (ADLS, KV, Purview, etc.) |
| `snet-sas-viya` | `10.100.4.0/24` | SAS Viya integration (Phase 2) |
| `vnet-mdp-transit` | `10.101.0.0/24` | Front-end Private Link for user access |

### Peering

- `vnet-mdp-prod` ↔ `vnet-mdp-transit` (for front-end Private Link)
- `vnet-mdp-prod` ↔ Greenfield hub VNet (for on-prem connectivity via ExpressRoute/VPN)
- `vnet-mdp-prod` ↔ `vnet-sas-viya` (when SAS is deployed in Phase 2)

### DNS

- Private DNS zones hosted in Greenfield's central DNS hub (hub-spoke model)
- Zones: `privatelink.azuredatabricks.net`, `privatelink.dfs.core.windows.net`, `privatelink.vaultcore.azure.net`, `privatelink.purview.azure.com`

---

## 3. NSG Rule Set (Databricks Subnets)

### Inbound Rules

| Priority | Name | Source | Destination | Port | Action |
|---|---|---|---|---|---|
| 100 | AllowDatabricksControlPlane | `AzureDatabricks` service tag | `snet-dbx-*` | 443 | Allow |
| 200 | AllowInternalSubnet | `snet-dbx-host` | `snet-dbx-container` | * | Allow |
| 201 | AllowInternalSubnet2 | `snet-dbx-container` | `snet-dbx-host` | * | Allow |
| 4096 | DenyAllInbound | * | * | * | Deny |

### Outbound Rules

| Priority | Name | Source | Destination | Port | Action |
|---|---|---|---|---|---|
| 100 | AllowDatabricksControlPlane | `snet-dbx-*` | `AzureDatabricks` service tag | 443 | Allow |
| 110 | AllowSQL | `snet-dbx-*` | `Sql` service tag | 3306 | Allow |
| 120 | AllowStorage | `snet-dbx-*` | `Storage` service tag | 443 | Allow |
| 130 | AllowEventHub | `snet-dbx-*` | `EventHub` service tag | 9093 | Allow |
| 200 | AllowKeyVault | `snet-dbx-*` | `snet-private-endpoints` | 443 | Allow |
| 210 | AllowADLS | `snet-dbx-*` | `snet-private-endpoints` | 443 | Allow |
| 4096 | DenyAllOutbound | `snet-dbx-*` | `Internet` | * | Deny |

**Note:** The `DenyAllOutbound` to Internet is the data exfiltration protection control. All egress goes through Private Endpoints or service tags.

---

## 4. Terraform Module Structure

```
terraform/
├── environments/
│   └── prod/
│       ├── main.tf
│       ├── variables.tf
│       ├── terraform.tfvars
│       └── backend.tf          ← State in Azure Storage
├── modules/
│   ├── networking/
│   │   ├── main.tf             ← VNet, subnets, NSGs, NAT GW, peerings
│   │   ├── variables.tf
│   │   └── outputs.tf
│   ├── databricks-workspace/
│   │   ├── main.tf             ← Workspace, VNet injection, Private Link
│   │   ├── variables.tf
│   │   └── outputs.tf
│   ├── unity-catalog/
│   │   ├── main.tf             ← Metastore, access connector, storage credential,
│   │   │                          catalogs, schemas, external locations, grants
│   │   ├── variables.tf
│   │   └── outputs.tf
│   ├── storage/
│   │   ├── main.tf             ← ADLS Gen2, containers, CMK, private endpoints
│   │   ├── variables.tf
│   │   └── outputs.tf
│   ├── keyvault/
│   │   ├── main.tf             ← Key Vault, access policies, private endpoint
│   │   ├── variables.tf
│   │   └── outputs.tf
│   ├── monitoring/
│   │   ├── main.tf             ← Log Analytics, diagnostic settings, alerts
│   │   ├── variables.tf
│   │   └── outputs.tf
│   └── identity/
│       ├── main.tf             ← Managed identities, RBAC assignments
│       ├── variables.tf
│       └── outputs.tf
└── ci/
    ├── azure-pipelines.yml     ← Plan on PR, Apply on merge
    └── scripts/
        └── validate.sh         ← terraform validate + tflint + checkov
```

### Key Terraform Resources

| Module | Key Resources |
|---|---|
| `networking` | `azurerm_virtual_network`, `azurerm_subnet`, `azurerm_network_security_group`, `azurerm_nat_gateway`, `azurerm_private_endpoint`, `azurerm_private_dns_zone` |
| `databricks-workspace` | `azurerm_databricks_workspace` (with `custom_parameters` for VNet injection), `azurerm_private_endpoint` (front-end + back-end) |
| `unity-catalog` | `databricks_metastore`, `databricks_metastore_assignment`, `databricks_metastore_data_access`, `databricks_catalog`, `databricks_schema`, `databricks_external_location`, `databricks_grants` |
| `storage` | `azurerm_storage_account` (HNS, CMK), `azurerm_storage_container`, `azurerm_private_endpoint` |
| `identity` | `azurerm_user_assigned_identity`, `azurerm_databricks_access_connector`, `azurerm_role_assignment` (Storage Blob Data Contributor on ADLS) |

---

## 5. Cluster Policies

| Policy Name | Target Group | Configuration |
|---|---|---|
| `de-job-cluster` | Data Engineers | Auto-termination: 20 min. Spot instances: up to 80% of workers. Node type: `Standard_DS3_v2` (fixed). Min/max workers: 2–8. Spark version: latest LTS. Unity Catalog enabled. |
| `de-interactive` | Data Engineers | Auto-termination: 60 min. Single-user access mode. Node type: `Standard_DS4_v2`. Max workers: 4. |
| `analyst-sql-only` | Analysts | Serverless SQL Warehouse only. No general-purpose clusters. Warehouse size: Small (2X-Small for dev). Auto-stop: 10 min. |
| `ds-ml-cluster` | Data Scientists | GPU-enabled option: `Standard_NC6s_v3`. Auto-termination: 30 min. ML runtime (latest). Max workers: 4. Single-user access mode. |
| `admin-unrestricted` | Platform Admins | Unrestricted — for platform debugging only. Requires justification tag. |

---

## 6. DABs Repository Structure

```
mdp-platform/                     ← Git repository
├── databricks.yml                 ← DAB project config
├── bundles/
│   ├── infrastructure/
│   │   ├── cluster-policies/      ← Policy JSON definitions
│   │   ├── instance-pools/        ← Pool definitions
│   │   └── secret-scopes/         ← Secret scope config
│   ├── ingestion/                 ← Phase 1 (empty in Phase 0)
│   │   ├── bronze/
│   │   └── silver/
│   ├── transformation/            ← Phase 2 (empty in Phase 0)
│   │   └── gold/
│   ├── metrics/                   ← Phase 3 (empty in Phase 0)
│   │   └── metric-views/
│   └── governance/
│       ├── grants/                ← UC grant definitions
│       └── quality/               ← DQ rule definitions
├── tests/
│   ├── unit/
│   └── integration/
└── .github/ or .azure-pipelines/
    └── ci.yml
```

---

## 7. RACI Matrix

| Task | Platform Admin | Cloud Infra | Security Architect | DevOps | Data Steward |
|---|---|---|---|---|---|
| Subscription provisioning | C | **R** | A | I | I |
| VNet & networking | C | **R** | A | I | I |
| Private Endpoints | C | **R** | A | I | I |
| NSG rules | I | **R** | **A** | I | I |
| ADLS Gen2 deployment | C | **R** | A | I | I |
| Key Vault deployment | C | **R** | A | I | I |
| Databricks workspace | **R** | C | A | I | I |
| Private Link config | C | **R** | A | I | I |
| SCIM / identity provisioning | **R** | I | A | I | I |
| IP access lists | **R** | I | **A** | I | I |
| Conditional Access | I | I | **R** | I | I |
| Cluster policies | **R** | I | C | I | I |
| UC metastore creation | **R** | I | C | I | C |
| Catalog/schema structure | C | I | I | I | **R** |
| External locations | **R** | C | C | I | I |
| Default grants | C | I | A | I | **R** |
| Audit logging | **R** | C | A | I | I |
| Purview registration | I | I | I | I | **R** |
| Terraform modules | C | **R** | C | **R** | I |
| CI/CD pipeline | I | C | I | **R** | I |
| Phase 0 exit review | **R** | **R** | **R** | C | C |

**R** = Responsible, **A** = Accountable, **C** = Consulted, **I** = Informed

---

## 8. Exit Criteria

| # | Criterion | Verification Method |
|---|---|---|
| 1 | Workspace accessible via Private Link from Greenfield corporate network | User logs in via browser → workspace UI loads. `curl` to workspace URL resolves to private IP. |
| 2 | No public internet access to workspace | Attempt access from non-corporate IP → blocked. IP access list denies connection. |
| 3 | Unity Catalog metastore operational | `SHOW CATALOGS` returns `raw`, `curated`, `analytics`, `sandbox`. |
| 4 | ADLS Gen2 accessible from workspace | Notebook: `dbutils.fs.ls("abfss://bronze@stglobal...")` succeeds via managed identity. |
| 5 | Key Vault secret scope functional | Notebook: `dbutils.secrets.get(scope="mdp-keyvault", key="test-secret")` returns value. |
| 6 | SCIM sync active | All 6 Entra ID groups visible in workspace admin console with correct members. |
| 7 | Cluster policies enforced | User in `grp-mdp-data-analysts` cannot create general-purpose cluster — only SQL Warehouse. |
| 8 | Audit logs flowing | Azure Monitor shows Databricks diagnostic logs. UC audit table `system.access.audit` contains login events. |
| 9 | Purview metadata scan completed | Purview Unified Catalog shows `raw`, `curated`, `analytics`, `sandbox` catalogs. |
| 10 | Terraform state clean | `terraform plan` returns "No changes. Your infrastructure matches the configuration." |
| 11 | Security review passed | Security architect sign-off document on file. No critical findings. |
| 12 | CI/CD pipeline operational | PR triggers `terraform plan`. Merge to `main` triggers `terraform apply`. Both succeed. |

---

## 9. Risks Specific to Phase 0

| Risk | Impact | Mitigation |
|---|---|---|
| Subscription provisioning delay (Greenfield IT) | Blocks everything | Engage cloud team 2 weeks before Phase 0 starts. Pre-approve subscription request. |
| Private DNS zone conflicts with existing hub | Workspace unreachable | Coordinate with central networking team. Use conditional forwarders if needed. |
| Databricks Premium contract not signed | Cannot deploy workspace | Procurement must close before D6. Escalate to VP if delayed. |
| SCIM sync issues (nested groups, group size limits) | Incomplete identity provisioning | Test with a small pilot group first. Flatten nested groups if needed. |
| NSG rules too restrictive | Cluster launch failures | Start permissive (log-only), harden iteratively based on traffic analysis. |
| Terraform provider version mismatch | Drift between plan and apply | Pin provider versions in `required_providers`. Validate in CI pipeline before applying. |