Telecoms · Playbook 39

Network incident triage

Thousands of alarms turned into a handful of probable incidents, each with its likely root cause, diagnostics already gathered and a draft incident record. Engineers decide the fix, and no network change happens without change control.

  • Predictive ML
  • Agents with tools
  • Event-driven integration
  • Human in the loop
  • Telecommunications (Security) Act 2021
  • Telecoms Security Code of Practice
  • Ofcom s105K reporting
  • Ofcom General Conditions
  • UK GDPR

At a glance

The problem
One fault upstream sets off thousands of alarms downstream, and the NOC spends the first hour working out what actually broke
Typical volume
Over a million raw alarms a day, a few hundred tickets, and several major incidents a month
Who's involved
NOC and network engineering, incident and problem management, change management, security, customer operations, regulatory affairs, data protection
Pilot
10–12 weeks on one network domain, after a 3–4 week discovery
Platforms
AWS, Azure, or a sovereign or on-prem stack, with the network management plane always on-prem

An illustrative scenario. The numbers are typical examples, not a client case study. The cloud services and UK rules were checked on 9 November 2026.

The business problem

When a fibre is cut or a router card fails, everything that depends on it raises an alarm: transport links, mobile sites, broadband aggregation, voice. The NOC sees a storm of thousands of alarms from dozens of systems, each with its own format and severity. Engineers suppress what they can, sort the rest by hand, and try to find the one alarm that explains all the others. On a bad night, that takes longer than the fix.

Meanwhile the clock is running on everything else. Customers and the contact centre want to know what's happening. Major incidents may need reporting to Ofcom, and broadband and landline outages can trigger automatic compensation. As landlines move to digital voice, an access outage can also cut off phone calls and telecare alarms for people who rely on them. Incident records written at 3am are often thin, and post-incident reviews have to rebuild the timeline from logs.

The tools that could help are also a risk. Anything that can reach network management systems is a privileged access path into the most sensitive part of the network. The Telecommunications (Security) Act 2021 and its Code of Practice treat the management plane and automation that can affect the network as security critical. So the design has to make diagnosis faster without giving an AI the ability to change the network.

What AI changes

  • Alarms from every domain are normalised onto one event stream and grouped in near real time, using the network topology and time, so one fault shows as one probable incident.
  • An ML model ranks the likely root cause in each group, with the evidence and a confidence, learned from past incidents and their confirmed causes.
  • An agent runs read-only diagnostics through approved tools: interface and link status, recent alarms, performance counters, recent changes and planned works on the affected kit.
  • It drafts the incident record, the impact assessment (services, customers, priority customers and emergency call routing), and customer and internal updates for a person to approve.
  • It flags when an incident looks likely to meet Ofcom's reporting thresholds, with the timeline and figures the report will need. A named incident manager decides and submits.
  • Any change to the network, even a suggested fix from the runbook, goes to an engineer and through change control. The agent has no write access to the management plane, by design rather than by instruction.

How success is measured

Agree these, and measure the baseline, before anything is built. Example targets are typical, not promises.

Time to probable cause

From the first alarm to an engineer confirming the root cause.

ExampleDown by half on the pilot domain

Alarm-to-incident ratio

How many raw alarms the NOC handles per real incident. Too aggressive grouping hides separate faults.

ExampleDown sharply, with missed separate faults tracked

Root-cause suggestions confirmed

Share of incidents where the top suggestion matched the engineer's confirmed cause.

ExampleAbove 70%, reviewed weekly

Incident record completeness

Records with timeline, impact, cause and actions filled in by the time the incident closes.

ExampleAbove 95%

Ofcom reports on time

Reportable incidents notified within Ofcom's expected timescales, with figures that didn't need correcting later.

ExampleEvery reportable incident

Changes made outside change control

Network changes made during incidents without an approved or retrospectively logged emergency change.

ExampleZero

UK regulation and controls

Cross-sector rules are on the UK regulation map.

  • DCMS / OfcomTelecommunications (Security) Act 2021 and the Electronic Communications (Security Measures) Regulations 2022

    Providers must identify and reduce the risks of security compromise, protect the functions that oversee and manage the network, control privileged access, monitor and log activity, and be able to keep running the UK network without relying on people, equipment or data outside the UK. A triage agent that can query network management systems is a new privileged access path. It gets its own read-only identity, goes through a brokered, logged route, and the platform runs in UK regions.

  • DCMSTelecommunications Security Code of Practice, version 1.1 (14 July 2026)

    Tier 1 providers (relevant turnover of £1 billion or more) and Tier 2 (£50 million to £1 billion) are measured against it. The first measures applied from March 2024 and March 2025; a further group, including management plane, signalling, supplier contracts and monitoring, is due by 31 March 2027, and new version 1.1 measures follow in 2028 and 2029. Its guidance on automation asks for validated inputs, reviewed changes, checked outputs, a rollback plan and a named owner for every automated process. The design follows that: the agent reads, a person changes.

  • OfcomSecurity compromise reporting (section 105K, Communications Act 2003)

    Providers must tell Ofcom about security compromises with a significant effect, including outages from hardware failure or human error, as soon as reasonably practicable. Ofcom's guidance sets thresholds, for example 100,000 customers affected for any duration, and expects the most serious incidents to be notified quickly, followed by a fuller report, with smaller ones in a monthly bulk report. The agent tracks the figures against the thresholds and drafts the report. The incident manager decides whether to report and submits it.

  • OfcomGeneral Condition A3: availability of services and access to emergency services

    Providers must take all necessary measures to keep services available during major failures and to keep access to emergency services uninterrupted. Any incident touching 999 and 112 call routing goes to the top of the queue and straight to a person. The model can never rank it lower.

  • OfcomGeneral Condition C5, automatic compensation and complaints

    Customers in vulnerable circumstances need extra care during outages. Under the automatic compensation scheme, a residential broadband or landline customer gets £10.34 a day once a total loss of service isn't fixed after two full working days (rates from 1 April 2026), and since 8 April 2026 complaints can go to dispute resolution after six weeks. Accurate start and end times and affected-customer lists in the incident record matter. Customer messages are drafted for a person to approve, and telecare users get a call, not just a text.

  • DCMS / OfcomPSTN switch-off (31 January 2027) and the Fixed Telecoms Modernisation Charter (24 March 2026)

    As landlines move to digital voice, a broadband or power outage can now cut off phone calls and telecare alarms. The impact assessment includes voice and telecare on affected lines, and uses the priority customer register the charter expects providers to keep.

  • ICOUK GDPR and PECR

    Alarm and performance data is mostly not personal, but affected-customer lists, customer messages and engineers' actions are. Use customer data for restoring service and keeping people informed, keep outage updates as service messages, not marketing, and do a DPIA before the pilot.

The solution

Showing AWSAzuresovereign / on-prem services. Switch with the toggle; every diagram and table on this page follows it.

Network incident triage: reference architecture
  1. 01Network and OSS

    • Network elements and element managers
    • Fault management and network inventory
    • Performance counters and probes
  2. 02Intake

    • Amazon MSKEvent Hubs (Kafka endpoint)Apache KafkaAlarm and event stream
    • Amazon NeptuneAzure Cosmos DB for Apache GremlinPostgreSQL with Apache AGETopology graph
    • Amazon OpenSearch ServiceFabric Real-Time Intelligence (Eventhouse)OpenSearchAlarm and incident history
  3. 03Intelligence

    • SageMaker AIAzure Machine LearningMLflow and Kubeflow on KubernetesAlarm grouping and root-cause model
    • Bedrock Knowledge BasesFoundry IQ (Azure AI Search)OpenSearch, or PostgreSQL with pgvectorRunbooks and past incidents
    • Agent
      Bedrock AgentCoreFoundry Agent ServiceLangGraph on Kubernetes, with open-weight models via vLLMIncident triage agent
  4. 04Orchestration

    • Step FunctionsLogic Apps / Durable FunctionsTemporalIncident workflow
    • AgentCore GatewayAPI ManagementKong, or your existing API gatewayRead-only diagnostics gateway
  5. 05Systems of record

    • Service management: incidents, problems and changes
    • Customer systems and priority customer register
    • Service status page and customer messaging
  6. 06People

    • Person
      NOC engineer: confirms cause and fix
    • Person
      Incident manager: comms and Ofcom reporting
    • Person
      Change authority: approves any network change
  • Cloud or platform service (AWSAzuresovereign / on-prem)
  • AI agent
  • A person decides
  • Your existing systems
  • Outside your organisation

Components, side by side

Network incident triage: components on AWS, Azure and a sovereign or on-prem stack
LayerAWSAzureSovereign / on-prem
Alarm intakeTake alarms from fault management, not from every element. Normalise them once, with a stable ID, so replays don't duplicate.Amazon MSK, fed by your fault management system's northbound interfaceEvent Hubs (Kafka endpoint), fed the same wayApache Kafka
Real-time groupingGroup by topology and time window. Keep the rules visible to engineers, and let them split a group the model got wrong.Amazon Managed Service for Apache FlinkFabric Real-Time Intelligence (Eventstream, Eventhouse)Apache Flink on Kubernetes
Network topologyA copy of what your inventory system says is connected to what. Inventory stays the master; stale topology is the commonest cause of wrong grouping.Amazon NeptuneAzure Cosmos DB for Apache GremlinPostgreSQL with Apache AGE
Root-cause modelTrained on past incidents with confirmed causes. Cause coding in old tickets is often poor, so budget time to clean it.Amazon SageMaker AIAzure Machine LearningMLflow and Kubeflow on Kubernetes
Runbooks and past incidents (retrieval)Approved runbooks, vendor documentation you're licensed to use and closed incident reviews, so drafts cite their source.Bedrock Knowledge BasesFoundry IQ, built on Azure AI SearchOpenSearch, or PostgreSQL with pgvector
Reasoning and draftingBoth clouds offer a choice of models, including Claude. The language model drafts and explains; the grouping and ranking come from the ML model.Amazon BedrockMicrosoft FoundryOpen-weight models served with vLLM on your own GPUs, or IONOS AI Model Hub
Agent runtimeThe original Bedrock Agents closed to new customers on 30 July 2026; new builds use AgentCore.Amazon Bedrock AgentCoreFoundry Agent ServiceAn open-source agent framework (e.g. LangGraph) on Kubernetes
Diagnostic toolsEach tool is a fixed, read-only query (show commands, counters, recent changes) through your management systems' APIs, never a free-form command line on a device.AgentCore Gateway in front of read-only APIsAPI Management in front of read-only APIsKong or your existing API gateway in front of read-only APIs
Workflow and eventsAWS Step Functions, EventBridge, SQSLogic Apps or Durable Functions, Event Grid, Service BusTemporal; Apache Kafka or RabbitMQ
Link to service managementIncidents and draft updates land in your service management tool. Change requests can only be created by a person.API Gateway, plus your existing integration platformAPI Management, plus your existing integration platformKong or your existing API gateway, plus your existing integration platform
Human reviewEngineers see the group, the suggested cause and the evidence where they already work.In the NOC's existing incident console and service management toolIn the NOC's existing incident console and service management toolIn the NOC's existing incident console and service management tool
Cloud operations agentsThese investigate cloud workloads, not radio, transport or core networks. Amazon DevOps Guru closed to new customers on 29 October 2026.AWS DevOps Agent, for the platform itselfAzure SRE Agent, for the platform itselfYour existing monitoring
Identity, keys, privileged accessThe agent has its own read-only identity, separate from engineers'. Engineers make changes only from privileged access workstations, as the Code of Practice expects.IAM Identity Center, KMS, Secrets ManagerEntra ID, Key Vault, Privileged Identity ManagementKeycloak or your existing Active Directory; OpenBao or HashiCorp Vault; an HSM
Monitoring and auditLog every tool call, every suggestion and every human decision, and feed the logs to your security monitoring.CloudWatch, CloudTrailAzure Monitor, Log AnalyticsPrometheus, Grafana, OpenTelemetry
UK data residencyNetwork data is security sensitive. Keep it in the UK and make sure the NOC can run without the platform.eu-west-2 (London)UK South / UK WestYour own UK data centre, or a UK provider

On a phone, the table shows the platform you pick. Sovereign / on-prem means open, portable components you can run in your own data centre or with a UK or European provider; IONOS is shown as one example, and others exist. What sovereign cloud means →

Hybrid and on-prem

Many organisations keep core systems and sensitive data on their own servers. This is how the same design works across both.

Stays on-prem

Network elements, element managers, fault management, inventory and every route into the management plane stay in the operator's own network, untouched. So does the change process and anything that can alter configuration.

Uses cloud AI

Alarm grouping, the root-cause model, the runbook index and the triage agent can run in a UK cloud region, or on the operator's own servers with open-weight models. They receive alarms, topology and read-only diagnostic results, never credentials that can change the network.

How they connect

Fault management publishes alarms outbound over AWS Direct Connect or Azure ExpressRoute. Diagnostic queries come back in through one brokered, read-only gateway inside the operator's network, which decides what may be asked and logs every call.

If the link drops

The NOC works exactly as it did before the AI, from fault management and its own runbooks. Alarms queue and are grouped when the link returns, and incident records are completed by hand. Nothing in the network depends on the cloud.

The agents

What each agent may do through its tools, what it may not, and where a person signs off. The limits are enforced by the platform, not the prompt.

Incident triage agent

May do

  • Read the alarm group, the model's suggested cause and the topology around it
  • Run approved read-only diagnostics: link and interface status, counters, recent alarms
  • Look up recent changes and planned works on the affected equipment
  • Search runbooks and past incidents for similar faults
  • Draft the incident record, impact assessment, customer updates and an Ofcom report

May not do

  • Change any configuration or run any command that writes to a network element
  • Raise, approve or implement a change request
  • Close or downgrade an incident, or lower the priority of anything touching emergency calls
  • Publish a status update or send a message to customers
  • Submit a report to Ofcom

Where a person signs offA NOC engineer confirms the cause and decides the fix, and any network change goes through change control, including emergency changes. The incident manager approves customer messages and decides on Ofcom reporting. The agent's permissions are enforced by the platform's identity and policy controls and by the read-only gateway, not by instructions in a prompt.

Middleware and integration

Alarms come from fault management through its northbound interface onto one event stream. Each alarm keeps its original ID and timestamps, so replays and duplicates are harmless and the grouping can be rebuilt after the event for the post-incident review.

The agent reaches the network only through a read-only diagnostics gateway inside the operator's network. Each tool is a named, fixed query with a fixed scope. The gateway checks the identity, enforces rate limits so diagnostics can't add load during an outage, and logs every call. There is no tool for changes, and no route from the gateway to the write side of any management system.

Service management stays the system of record for incidents, problems and changes. If it's slow or down, drafts wait in a queue. The AI can be switched off without touching the network: the NOC carries on from fault management, as it did before.

Process and people

  • A network engineering lead owns the grouping rules and the root-cause model for each domain, with change control like any other network system.
  • Engineers can split or merge a group with one action, and every correction is fed back as a label.
  • Every diagnostic tool is reviewed by security before it's added: what it reads, from where, and what it could reveal.
  • Emergency changes during an incident follow the existing emergency change process, logged and reviewed afterwards. The AI's suggestion is recorded, but the engineer's decision is what's approved.
  • A named incident manager on every shift owns customer communications and the decision to report to Ofcom.
  • A monthly review of the incidents where the top suggestion was wrong, and of any separate faults that were grouped together.

Delivery plan

  1. 3–4 weeks

    Discovery

    Map today's alarm flow and incident process, measure time to cause, check alarm and topology data quality, and agree the read-only boundary with security.

  2. Gate · must be true to passPilot domain chosen, baseline measured, diagnostics gateway design approved by security, DPIA started.

  3. 10–12 weeks

    Pilot

    One domain (for example transport or mobile radio), in shadow mode: groups and suggestions go to a small team of engineers and are compared with what actually happened.

  4. Gate · must be true to passRoot-cause hit rate and grouping errors within agreed limits, no write path found in testing, and the NOC manager signs off.

  5. 4–6 weeks

    Production

    Grouping and drafts go live in the NOC for the pilot domain, with monitoring, support and a rollback plan.

  6. Gate · must be true to passSupport model in place, engineers and incident managers trained, security monitoring receiving the agent's logs.

  7. Ongoing

    Scale

    Add domains one at a time, then cross-domain grouping, then problem management.

  8. Gate · must be true to passEach new domain passes the same pilot gate; each new diagnostic tool passes its own security review.

Roles and RACI

Who is responsible, accountable, consulted and informed, by activity
ActivityNOC directorNetwork engineering leadIncident managerSolutions architectI can fill thisTechnical PMI can fill thisEngineersI can fill thisSecurityI can fill thisRegulatory affairs
Business case and benefitsAccountableResponsibleConsultedConsultedConsultedInformedInformedInformed
Target architecture and read-only boundaryInformedConsultedInformedAccountable and ResponsibleConsultedConsultedConsultedInformed
Security review against the Code of PracticeInformedConsultedInformedConsultedInformedConsultedAccountable and ResponsibleConsulted
Alarm, topology and diagnostics accessInformedAccountableInformedConsultedConsultedResponsibleConsultedInformed
Build and integration with service managementInformedConsultedConsultedAccountableResponsibleResponsibleConsultedInformed
Ofcom reporting and customer comms templatesInformedInformedResponsibleConsultedInformedInformedConsultedAccountable
Model testing against past incidentsInformedAccountableConsultedConsultedConsultedResponsibleInformedInformed
Go-live decisionAccountableResponsibleConsultedConsultedResponsibleInformedConsultedConsulted
Run, monitor and improveInformedAccountableConsultedConsultedConsultedResponsibleConsultedInformed

Business case and benefits

NOC director
A · Accountable
Network engineering lead
R · Responsible
Incident manager
C · Consulted
Solutions architect
C · Consulted
Technical PM
C · Consulted
Engineers
I · Informed
Security
I · Informed
Regulatory affairs
I · Informed

Target architecture and read-only boundary

NOC director
I · Informed
Network engineering lead
C · Consulted
Incident manager
I · Informed
Solutions architect
A/R · Accountable and Responsible
Technical PM
C · Consulted
Engineers
C · Consulted
Security
C · Consulted
Regulatory affairs
I · Informed

Security review against the Code of Practice

NOC director
I · Informed
Network engineering lead
C · Consulted
Incident manager
I · Informed
Solutions architect
C · Consulted
Technical PM
I · Informed
Engineers
C · Consulted
Security
A/R · Accountable and Responsible
Regulatory affairs
C · Consulted

Alarm, topology and diagnostics access

NOC director
I · Informed
Network engineering lead
A · Accountable
Incident manager
I · Informed
Solutions architect
C · Consulted
Technical PM
C · Consulted
Engineers
R · Responsible
Security
C · Consulted
Regulatory affairs
I · Informed

Build and integration with service management

NOC director
I · Informed
Network engineering lead
C · Consulted
Incident manager
C · Consulted
Solutions architect
A · Accountable
Technical PM
R · Responsible
Engineers
R · Responsible
Security
C · Consulted
Regulatory affairs
I · Informed

Ofcom reporting and customer comms templates

NOC director
I · Informed
Network engineering lead
I · Informed
Incident manager
R · Responsible
Solutions architect
C · Consulted
Technical PM
I · Informed
Engineers
I · Informed
Security
C · Consulted
Regulatory affairs
A · Accountable

Model testing against past incidents

NOC director
I · Informed
Network engineering lead
A · Accountable
Incident manager
C · Consulted
Solutions architect
C · Consulted
Technical PM
C · Consulted
Engineers
R · Responsible
Security
I · Informed
Regulatory affairs
I · Informed

Go-live decision

NOC director
A · Accountable
Network engineering lead
R · Responsible
Incident manager
C · Consulted
Solutions architect
C · Consulted
Technical PM
R · Responsible
Engineers
I · Informed
Security
C · Consulted
Regulatory affairs
C · Consulted

Run, monitor and improve

NOC director
I · Informed
Network engineering lead
A · Accountable
Incident manager
C · Consulted
Solutions architect
C · Consulted
Technical PM
C · Consulted
Engineers
R · Responsible
Security
C · Consulted
Regulatory affairs
I · Informed

R ResponsibleA AccountableC ConsultedI Informed· not involved

Where I'd fit

Option A

Lead the architecture

If there's no architect on the programme, I'd own the design end to end: the alarm pipeline, the read-only diagnostics boundary, the integration with your service management tool and the gates, working closely with your network engineers and security team. I'd report to your programme board.

Option B

Support your architect

If your partner or your own team already has an architect, I'd take on the event pipeline, the agent and its tool gateway, and the service management integration under their lead, and give them a second senior opinion on privileged access and on whether the root-cause suggestions are good enough to trust at 3am.

Where this goes wrong

  1. Goes wrong

    The agent is given an engineer's credentials 'just for the pilot', and can change the network.

    How the design avoids it

    Its own read-only identity, a gateway with fixed queries, and a security test that tries to write before go-live.

  2. Goes wrong

    Stale topology groups unrelated faults together, and a second outage is hidden inside the first.

    How the design avoids it

    Topology refreshed from inventory, engineers able to split groups, and missed separate faults measured.

  3. Goes wrong

    Diagnostics pile load onto struggling equipment in the middle of an outage.

    How the design avoids it

    Rate limits and timeouts at the gateway, and cached results shared between engineers and the agent.

  4. Goes wrong

    An emergency-call routing fault is ranked as minor because few alarms fired.

    How the design avoids it

    A hard rule: anything touching 999 and 112 routing goes to the top and to a person, whatever the model says.

  5. Goes wrong

    Customer updates go out with the wrong scope or times, and compensation and complaints follow.

    How the design avoids it

    Drafts only, approved by the incident manager, with impact taken from service data, not the model's guess.

On the horizon

Changes already announced or under way that affect this design. More in the UK horizon scan.

  1. Statement expected
    Ofcom consulted from 12 May to 4 August 2026 on new incident reporting guidance: standard mobile thresholds, a single failed rural cell site becoming reportable, and the critical threshold falling from 3 million to 1.5 million user-hours lost.

    Build threshold checks as configuration, not code, so they can change when Ofcom publishes its final statement.

  2. 31 January 2027
    The PSTN switch-off completes. Voice and telecare on most lines then depend on broadband and local power.

    Include voice and telecare in every access outage's impact assessment, linked to the priority customer register.

  3. 31 March 2027
    Further Telecoms Security Code of Practice measures fall due, including management plane, signalling, monitoring and all existing supplier contracts. New version 1.1 measures follow in March 2028, December 2028 and December 2029.

    Map the triage platform, its gateway and its suppliers against the measures now, and add them to the security programme.

  4. Late 2026–2028
    The Cyber Security and Resilience Bill brings medium and large managed service providers into regulation, with faster incident reporting.

    If any part of the NOC or its tooling is run by a managed service provider, check its new duties and align its incident reporting with yours.

  5. 2028
    The NCSC's post-quantum cryptography milestone: a full migration plan by 2028.

    Record the cryptography used on the alarm feed, the gateway and the private links, and choose components where it can be updated.

Questions to ask your partner or vendor

Good answers are specific. Vague answers to any of these are worth following up.

Copy them into your next supplier meeting

  1. Can the agent change anything on the network? How do you prove it can't?
  2. Which diagnostics can it run, through what, and what stops them adding load during an outage?
  3. How will you measure root-cause accuracy and missed separate faults on our incidents, not a benchmark?
  4. How does the design make sure emergency-call faults are never ranked down?
  5. Who decides on Ofcom reporting, and what figures will the draft give them, from where?
  6. What does it cost to run per million alarms, at our volume?

Planning something like this?

Whether you need someone to lead the design or to support the architect you already have, let's talk it through.

Sam Naseem
Sam NaseemBirmingham, UK · UK & Europe
Available now