Azure Storage Queue Module

This OpenTofu module provisions a dedicated Azure Storage Account holding a single Azure Storage Queue, for the Notification Events that kastoria-eventnotification publishes. The events themselves are documented in Ingestion Manifest.

The account holds nothing but the queue. Its consumer is a customer-developed queue reader, so keeping it separate from the blob and table accounts means a reader's role assignment lands on an account with no PHI to mis-scope it onto, and the queue's network posture stays independent of the object store's.

Features

Standard Performance: Standard tier with LRS replication (General Purpose v2), as the queue is a transient work list rather than a store of record.
One Account, One Queue: Unlike objectstore and filestore there is no shards map. A second consumer gets a second account, which keeps every reader's grant scoped to exactly one queue.
No Credentials: shared_access_key_enabled = false, so the account has no keys and no SAS can be minted from it. Every caller authenticates with Entra, and nothing is handed over, rotated or expired.
Managed Identity Publishing: Creates its own publisher identity, umi-<base_name>-queue-publisher, and grants it Storage Queue Data Message Sender on the account: send only, never read, peek or manage. OpenTofu creates the queue, so the publisher never needs to, and runs with createQueueIfMissing off. The identity exists for exactly this queue and holds no rights on any other store, so workloads that reach object storage cannot reach the queue unless you also attach the publisher to them. Supply user_managed_identity_principal_id instead to bring your own principal and skip creating one.
Customer Reader Access: Grants Storage Queue Data Message Processor to each entry of reader_principal_ids, so a customer's reader peeks, gets and deletes messages without a secret changing hands.
Plan-Time Name Validation: The derived account name is checked against ^[a-z0-9]{3,24}$ by a terraform_data precondition — the same rule the Kastoria queue writer validates accountName with — so a bad base_name/suffix aborts the plan rather than failing at apply.
Queue Name Validation: queue_name is checked against Azure's rule (3-63 lowercase alphanumerics and hyphens, starting and ending alphanumeric, no double hyphen) at plan time.
Network Security: With enable_firewall = true (the default) the account sets default_action = "Deny", admits allowed_subnet_ids via virtual network rules and allowed_ips via IP rules, and bypasses AzureServices. With it off, the network rules block opens to Allow with no restrictions.
HTTPS Enforcement: https_traffic_only_enabled = true and min_tls_version = "TLS1_2".
Firewall Drift Tolerance: network_rules[0].ip_rules is in ignore_changes, so the temporary runner-IP whitelists that CI/CD applies to these accounts do not show up as drift.

Requirements

Name	Version
azurerm	~> 4.0

Providers

Name	Version
azurerm	~> 4.0
terraform	n/a

Modules

No modules.

Resources

Name	Type
azurerm_role_assignment.readers	resource
azurerm_role_assignment.writer	resource
azurerm_storage_account.queue	resource
azurerm_storage_queue.events	resource
azurerm_user_assigned_identity.publisher	resource
terraform_data.account_name_check	resource

Inputs

Name	Description	Type	Default	Required
allowed_ips	List of ips to whitelist. Unlike the sibling storage modules, this list also carries the customer reader's egress ranges.	`list(string)`	`[]`	no
allowed_subnet_ids	The IDs of the subnets that can access the queue.	`list(string)`	n/a	yes
base_name [d]	The base name prefix used for resources.	`string`	n/a	yes
enable_firewall	Enables storage account firewall	`bool`	`true`	no
location [d]	The location where resources will be created.	`string`	n/a	yes
queue_name [d]	The queue Notification Events are published to.	`string`	`"kastoria-events"`	no
reader_principal_ids [d]	Principal IDs granted Storage Queue Data Message Processor, for customer-developed queue readers.	`list(string)`	`[]`	no
resource_group_name [d]	The name of the resource group.	`string`	n/a	yes
suffix [d]	A suffix to append to the resource name.	`string`	n/a	yes
tags	A map of tags to apply to the resource.	`map(string)`	n/a	yes
user_managed_identity_principal_id [d]	Principal ID to grant Storage Queue Data Message Sender. Null (default) creates a dedicated publisher identity and grants that instead.	`string`	`null`	no

[d] Destructive: changing this input forces one or more resources to be destroyed and recreated (an OpenTofu/Terraform replacement) rather than updated in place.

Outputs

Name	Description
publisher_client_id	Client ID of the created publisher identity, for the configuration document's managedIdentityClientId. Null when `user_managed_identity_principal_id` is supplied
publisher_id	Resource ID of the created publisher identity, for a workload's identity_ids. Null when `user_managed_identity_principal_id` is supplied
publisher_principal_id	Principal ID of the publisher: the created identity, or the supplied user_managed_identity_principal_id
queue_name	Name of the queue Notification Events are published to
queue_url	Data plane URL of the queue, for a customer-developed reader
storage_account_id	Resource ID of the queue storage account
storage_account_name	Name of the queue storage account, which is the `accountName` the Kastoria queue writer takes

Example Usage

module "queue" {
  source = "oci://acrmerkalisdist0c66.azurecr.io/modules/azure/queue?tag=<module-version>"

  base_name           = "myapp"
  suffix              = random_id.project_prefix.hex
  resource_group_name = azurerm_resource_group.env.name
  location            = azurerm_resource_group.env.location

  # Only the compute subnet publishes; nothing else in the VNet reads this queue
  allowed_subnet_ids = [
    module.networking.created_subnets["compute"].id
  ]

  # Principal IDs of your queue reader(s), granted read/process access
  reader_principal_ids = []

  # The runner's IP must be present for plan/apply to reach the account.
  # Append your reader's egress ranges if it runs outside the VNet
  allowed_ips = var.allowed_ips
  tags        = azurerm_resource_group.env.tags
}

The module creates its own publisher identity and exports it (publisher_id, publisher_client_id, publisher_principal_id); attach it to whichever workloads should publish. To grant the Sender role to a principal you already own instead, pass user_managed_identity_principal_id — the module then creates no identity, publisher_id and publisher_client_id are null, and you wire the identity and managedIdentityClientId yourself.

Wiring the Queue Writer

Two things connect the queue to Kastoria: the publishing workload must carry the publisher identity, and the configuration document must name the queue and that identity.

Attach the publisher to the job through the compute module's user_managed_queue_app:

jobs = {
  evtnotify = {
    # ... container_config, schedule, tags ...
    user_managed_storage_app = { id = azurerm_user_assigned_identity.app_identity.id, client_id = azurerm_user_assigned_identity.app_identity.client_id }
    user_managed_queue_app   = { id = module.queue.publisher_id, client_id = module.queue.publisher_client_id }
  }
}

The event notification job reads the Changelog from object storage as well as publishing, so it carries both identities.

Kastoria learns about the queue through an optional queue_writer entry in app_config. Like system_storage and storage_tiers, it is a JSON-encoded string, and it renders as the queueWriter section of the Kastoria configuration document:

app_config = {
  organization     = "My Org"
  environment_name = "myapp-prod"
  purpose          = "production"
  system_storage   = jsonencode([])
  storage_tiers    = local.storage_tiers

  queue_writer = jsonencode({
    type = "azure-storage-queue"
    configuration = {
      accountName             = module.queue.storage_account_name
      managedIdentityClientId = module.queue.publisher_client_id
      queueName               = module.queue.queue_name
      messageEncoding         = "text" # or "base64"
    }
  })
}

No connection string. The account has no keys to build one from, and the writer refuses a connectionString alongside managedIdentityClientId.
No createQueueIfMissing. OpenTofu owns the queue, which is what lets the publisher's role stay Storage Queue Data Message Sender.
managedIdentityClientId is the publisher's client ID. It is not the AZURE_CLIENT_ID that user_managed_storage_app sets. A workload carrying both identities uses AZURE_CLIENT_ID (the storage identity) for the storage tiers, and the explicit managedIdentityClientId (the publisher) for the queue.
The publishing job must carry the publisher. user_managed_queue_app attaches it without setting any environment variable; without it the writer cannot authenticate as the publisher and every publish fails.
messageEncoding is text (the JSON as-is) or base64. Every message on the queue uses the same encoding, so choose the one your reader expects — an Azure Functions queue trigger expects base64 by default.

Leaving queue_writer unset (it defaults to null) renders a configuration document identical to one without a queue.

The Kastoria writer composes the queue endpoint itself as https://<accountName>.queue.core.windows.net/<queueName>; there is no endpoint override.

Notes

1. No Credentials Exist

shared_access_key_enabled = false removes the account keys, which removes SAS along with them — an account SAS is signed with an account key. There is therefore nothing this module can emit that grants access, and no output is marked sensitive. Access is entirely Entra role assignments:

Principal	Role	Granted via
The publishing workload	`Storage Queue Data Message Sender`	the module-created `umi-<base_name>-queue-publisher`, unless `user_managed_identity_principal_id` names another principal
A customer's queue reader	`Storage Queue Data Message Processor`	`reader_principal_ids`

Both assignments are scoped to the storage account rather than the queue. azurerm_storage_queue exports no resource_manager_id, so unlike azurerm_storage_table there is no queue-scoped ARM id to target — and with one queue in the account the two scopes grant the same thing.

A reader that is not an Entra principal (for example, a tool that only accepts a SAS or connection string) is not supported.

2. Network Access

RBAC decides who may read; the firewall decides from where. A role assignment alone is not enough: a customer reader outside the VNet also needs its egress ranges admitted, or it receives a 403 Forbidden from the firewall that no role can resolve.

In objectstore and filestore, allowed_ips is operator access only. Here it also carries the reader's egress ranges, so keep the two lists separately named in your environment and concat them when passing allowed_ips.

3. Firewall Rules Are Seeded Once, Then Managed Out of Band

network_rules[0].ip_rules is in ignore_changes, so allowed_ips seeds the initial rule set only. The guard is not optional: creating and reading the queue are requests against the account's queue endpoint (https://<account>.queue.core.windows.net/<queue>), which sits behind this firewall even though queue creation authorizes as a management action rather than a data action. Plan and apply therefore fail with 403 Forbidden if the runner's IP is not admitted. CI/CD workflows whitelist the runner IP temporarily; without the guard every plan would show a spurious diff, and an apply would strip the runner's own IP while it still had work to do.

After the first apply, a reader's egress range is added out of band:

az storage account network-rule add -g <rg> --account-name <account> --ip-address <cidr>

Adding it to allowed_ips afterwards is harmless but has no effect — the guard discards it, and the plan will not show the change.

4. Onboarding a Reader

Add the reader's Entra principal ID to reader_principal_ids and apply. This creates the Storage Queue Data Message Processor role assignment.
If the reader runs outside the allowed subnets, add its egress range out of band as shown above.
Hand the reader the storage_account_name, queue_name and queue_url outputs, along with the message encoding you configured.

Until a reader is added, the queue is closed to everyone but the publisher.

5. CI Firewall Registration, and the First Apply Failing

If your pipeline whitelists its runner IP per storage account before a plan, add this account to that list once it exists, or plans will fail with a 403 against it. It cannot be added beforehand, because the rule cannot be set on an account that does not exist.

Unless the runner's IP is already in allowed_ips, this means the apply that first creates the account is expected to fail:

The pipeline whitelists the runner IP only on accounts already in its list. The queue account is not in that list, because it does not exist yet.
azurerm_storage_account.queue is created — a control-plane call, unaffected by the account's own firewall — with default_action = "Deny" and no runner IP in ip_rules.
azurerm_storage_queue.events then cannot reach the account and fails. The account is left in state.
Add the account to the pipeline's list and run again. The runner IP is now whitelisted before the plan, and the queue is created.

No rollback is needed between the two runs: the queue references the account by ID and nothing else depends on the queue existing.

6. Deploying Principal

The principal running OpenTofu needs Microsoft.Storage/storageAccounts/queueServices/queues write (and delete, for destroy). Creating a queue is a management action, not a data action — the data actions are the message-level operations under …/queues/messages/…, which is what the workload roles in Note 1 grant, and why this one is easy to overlook.

Subscription Owner covers it. Where the deployer is not an Owner, the smallest built-in role that does is Storage Queue Data Contributor; Storage Queue Data Message Sender and Storage Queue Data Message Processor cover only message-level operations.

Grant any such role alongside your other deployer-role assignments, not in this module. Component modules grant roles only to workload identities.

7. Naming Limitations

Azure Storage Account names are strictly limited to 24 characters and must be lowercase alphanumeric. This module constructs the name using the pattern:

saq + base_name + suffix

The whole name is scrubbed to lowercase alphanumerics and truncated to 24 characters, then validated at plan time. An over-long base_name is silently truncated rather than rejected.

Available versions

v0.9.1