Written by: Boris Kauffmann, Sr. Solutions Architect at Amazon Web Services.
Co-authored by: Paul Schiller, Product Marketing Manager, and Golan Simani, Director, Cloud & Tech Operations, TAG.
In the rapidly evolving world of broadcast, the bar for quality is set higher than ever. Viewers, armed with an expanding choice of content and platforms, demand consistent and superior experiences. Within this context, the value of monitoring and quality of experience (QoE) tools in broadcast operations emerges more prominently than ever.
In this blog post, we discuss how the TAG Realtime Media Performance platform helps broadcasters maintain operational and engineering familiarity with traditional master control room (MCR) workflows by leveraging native AWS services to facilitate TAG’s alarm capabilities. We compare the solution to a traditional monitoring workflow and discuss key design considerations.
Traditional master control room workflow
Within the broadcast facility, the master control room (MCR) stands as the operational core, directing and refining the flow of content. Here, live feeds and pre-recorded content from diverse sources converge, where each signal’s integrity, from its source to its format, is monitored and processed as needed.
An MCR’s main components include routing switchers that allow operators to dynamically assign feeds to their intended outputs. In the case of technical disruptions or programming changes, the MCR operator quickly redirects, pulling from backups or alternate sources when necessary.
Automated tools are deployed for rigorous quality control, catching anomalies in video and audio. After QC clearance, signals are routed for transmission, whether via satellite uplink, terrestrial IP distribution, or internet streaming.
Lastly, multiviewers are at the heart of an MCR, where different aspects of audio, video, and metadata signals are monitored and logged for compliance purposes. The monitoring solutions enable operators to quickly respond to issues such as signal loss and black or frozen signals. This is paramount for many broadcasters, as delays in resolution can result in significant on-air gaps, impacting viewer experience and broadcaster revenue.
Challenges
With such a broad spectrum of potential issues, an operator’s ability to swiftly diagnose a problem is crucial, regardless of where a system is running (on-premises, cloud, or hybrid). But it’s not just about identifying disruptions; it’s about rapid, effective response while maintaining operational familiarity. This requirement is magnified as MCR operators face a sheer volume of channels they oversee; each channel has a unique set of content, transmission requirements, and potential pitfalls. Monitoring hundreds of channels means that operators must keep track of thousands of individual signal elements and scheduling lists.
Operators must also proactively anticipate issues, often juggling between different feeds to prioritize and which to monitor closely based on factors like viewership numbers, event significance, or historical data about a channel’s reliability. Different channels might use different equipment, protocols, or software depending on the workflow targets, such as over-the-top (OTT), Satellite, and Terrestrial distribution. Operators must often be familiar with a broad range of systems and adept at using diagnostic tools efficiently.
As broadcasters move their workflows to the cloud, providing familiar and consistent automation tools to the operators while ensuring scalability and resiliency is critical.
The Solution
TAG excels in the monitoring space, with deep probing and analysis within a user-friendly interface, ensuring easy integration with an array of broadcast tools and software, creating a holistic ecosystem. By running on AWS, TAG enables workflows to continuously adapt to the latest transport streams, codecs, and standards to monitor the health of all streams.
With 500+ deep-probing capabilities, user-defined thresholds allow notifications (including severity levels and durations) to be delivered as event-driven alarms to downstream systems. Combined with AWS’s managed services, including Amazon Managed Streaming for Apache Kafka (MSK), AWS Lambda, Amazon Simple Notification Service (SNS), and Amazon Simple Storage Service (S3), customers can automate the monitoring and switching of video content based on defined thresholds, from a simple notice to minor, major, or even critical alerts.
Figure 1 – Set thresholds and severity for over 500 types of errors with TAG Realtime Media Platform.
Signal Workflow
Figure 2 – Solution Architecture Diagram
1. Signal Acquisition
Signals from encoders, playout engines, or any other transport stream source are delivered to the cloud using AWS Elemental MediaConnect. AWS Elemental MediaConnect is an AWS-managed service that leverages Amazon’s extensive network infrastructure to provide broadcasters with secure and reliable content delivery, ensuring high-quality streams with minimal latency. The outputs of the AWS Elemental MediaConnect flows are fed into the broadcast router inputs, allowing them to be routed to other systems.
2. Signal Monitoring
The Broadcast Router sends a copy of all available sources to the TAG Realtime Media Platform, where multiviewer mosaics are created for the operator’s visual inspection and deep-probing analysis with user-defined thresholds to generate notifications about the incoming signals.
Figure 3 – Channel Events list provided by TAG
3. Event-driven Alarming System
Collection of all notification events is published by TAG to specific Topics inside the Amazon MSK cluster, carrying in the message the metadata related to the event.
Example of the JSON event published to Amazon MSK.
{
"channel": "9c8ff210-b922-4bae-ae0e-e0e2b708c8d4",
"device": "569235aa-82d7-4bc9-acc3-5c0a930550fb",
"mcm-version": "6.3.1",
"protocol": "1.0",
"label": "OnPrem-SRC",
"url": "srt://IP:PORT/",
"tags": null,
"profile": {
"label": "Default",
"uuid": "a4ade0f0-028e-4228-8b52-427d54d87e96"
},
"thresholds": {
"label": "Default",
"uuid": "b026dd10-9b84-42db-805a-16f1e97b10da"
},
"event_number": 10,
"event_type": "Configured service type lower than effective (SD/HD/RADIO)",
"access_type": "CTP",
"severity": "Critical",
"active": true,
"trigger": false,
"duration": null,
"description": "Config Service Type mismatch",
"details": null,
"created_us": 1696962277728421,
"timestamp_us": 1696962277728421
}
4. AWS Lambda trigger from Amazon MSK
The collection of all notification events is then published to Amazon MSK and used as a trigger in AWS Lambda to consume and act upon. The AWS Lambda function parses the events, analyzes which source is impacted, and takes action based on the available metadata. AWS Lambda automates the flow of all user-defined alarms created in the TAG into specific rules-based actions.
Figure 4 – Amazon MSK as AWS Lambda Trigger
5. Automatic Source Switching
Based on which source is known to be disrupted and the criticality, the AWS Lambda function sends a command to the Broadcast Router to switch the source and failover to a new valid source, a backup feed, or an alternate path. This automation workflow virtually eliminates, at minimum drastically reduces, operator resources to take any actions.
6. Logging Storage and External Notifications
For compliance purposes, all events processed are stored in Amazon S3 for search during troubleshooting or future use. Also, AWS Lambda publishes selected critical events to Amazon Simple Notification Service (SNS) to connect with other messaging systems for Application-to-Application (A2A) messaging or push SMS or e-mail notifications in an Application-to-Person (A2P) messaging model.
Summary
The integration of TAG’s Realtime Media Platform with AWS infrastructure presents a technically advanced approach to broadcast monitoring and management. By leveraging TAG’s in-depth visualization and real-time alert system, MCR operators gain an efficient toolset to monitor hundreds of channels simultaneously. On the infrastructure side, the scalability of AWS its secure cloud environment ensures that resources are allocated dynamically based on demand, aiding in system optimization. Additionally, this integration streamlines operations, reducing the need for disparate systems management. This convergence of TAG and AWS represents not just a merging of services, but a holistic solution crafted to address the intricate, technical demands of modern broadcasting monitoring.
