Author: Viswanath Avasarala, PhD
Cost Optimization for AWS IOT Workloads is a key topic when dealing with high-volume IOT workloads in production. The AWS blog post “Cost Optimization Tips for AWS IoT Workloads" introduces valuable AWS tools designed to help manage IoT costs effectively. Building on those insights, this article offers practical strategies to enhance cost efficiency further, focusing on strategic adjustments to edge device configurations.
A key aspect highlighted in the AWS blog is optimizing message size and frequency to reduce cloud expenses. Here, we dive deeper into this topic to provide specific techniques to achieve these cost savings. Edge device configurations play a crucial role in workloads streaming real-time data from edge devices to cloud platforms like AWS IoT Core or directly to services such as AWS Kinesis. Given the billing structures of these platforms, fine-tuning these settings can significantly impact your overall cloud costs.
| If your message sizes are not significantly larger than the size that triggers an event then you might already be close to an optimal point |
In summary, check the distribution of your message sizes using tools such as AWS CloudWatch. If your message sizes are significantly larger than the size that triggers an event (e.g., 5 KB for AWS IoT Core), you might already be close to an optimal point. However, if the message sizes are not consistently significantly larger (~3x) than this threshold, there could be substantial savings from optimization efforts.
Edge devices like Kepware IoT Gateway offer configurable parameters like message buffering time. These settings directly influence the frequency of transmissions and the size of individual messages. However, finding the right balance isn’t straightforward. Data from edge devices is often stochastic and unpredictable, leading to message size variability. Suboptimal configurations can lead to significantly increased costs.
To tackle this, we introduce the concept of Billing Payload Efficiency (BPE)—a metric that evaluates how well the transmitted payloads utilize the billable capacity for each event. Organizations can configure their edge systems by analyzing and optimizing this efficiency to achieve near-theoretical minimum costs while maintaining desired latency and reliability.
Billing Payload Efficiency (BPE) refers to how effectively the maximum capacity of a billable event is utilized. It can be calculated using the formula:
This concept is illustrated below.
This implies that when the messages are between 4 and 6, assuming a uniform distribution as shown, the billing efficiency would be around 75%.
The illustration below shows how optimization through better message sizing and frequency can lead to a 26% reduction in costs.
When working with edge devices like the Kepware IoT Gateway, the stochastic nature of data generation at the source introduces variability in payload sizes. For example, Kepware is often configured to publish data only when changes occur. Consider a sample scenario where a buffering interval of 1-second results in payload sizes varying significantly, ranging between 5 KB and 15 KB. Such randomness in payload sizes directly impacts billing efficiency and overall costs, making it crucial to optimize buffering strategies.
To account for this variability, we introduce the concept of Average BPE, which represents the expected value of BPE for a given payload size distribution. This metric provides a quantitative framework to evaluate and optimize buffering strategies under varying conditions.
Average BPE can be estimated for various distributions of payload sizes. Assuming the payload is distributed uniformly with a range of 5KB, the average BPE versus the average message size is shown in the figure below. Although specific results vary depending on the distribution, the key principles remain consistent.
Let’s analyze the above figure to understand the overall trend. When your average message size is 2.5 KB, you waste approximately 50% of the available billable event capacity. In other words, adopting a better buffering strategy could reduce your IoT Core costs by 50% for the same data volume.
Two key trends emerge from this analysis. First, smaller messages tend to have lower billing efficiencies due to how billable events are calculated. Second, billing efficiencies gradually converge toward 100% as message sizes increase.
The relationship is not a straightforward, monotonically decreasing one. For instance, consider the Average Billing Efficiency under two scenarios: in the first scenario, the payload varies uniformly between 3 and 5, resulting in an Average Billing Efficiency of 80%. In contrast, in the second scenario, where the payload varies uniformly between 10 and 12, the Average Billing Efficiency drops to 73%. As a result, it is essential to analyze the factors influencing efficiency across different payload ranges carefully.
While buffering for larger message sizes can significantly improve efficiency, it may not always be feasible due to latency constraints. The optimal strategy should balance cost efficiency and latency requirements tailored to your use case’s needs and priorities.
It is important to note that the results depend on the actual payload distribution. To optimize for your payload, it is crucial to simulate the distribution accurately and then use tools to calculate the Average Billing Efficiency.
Many edge sources produce data at high volumes. Even at the highest transmission frequency, the payload size may be significant. For these scenarios, explicit optimization of payload size may only have marginal benefits since the number of billable events is already close to the optimal number. However, for scenarios where edges produce data at low volume, the right strategy can have a significant impact.
AWS IoT Core charges based on Message Size: Messages up to 5 KB are considered a single billable event. Larger messages are split into multiple 5 KB chunks, each billed separately.
Benefits of Larger Messages
Trade-offs and Challenges in Optimizing Billing Payload Efficiency
DeepIQ Edge software is designed to enhance data transmission reliability and optimize cloud expenses for industrial applications. Here are its key advantages:
These features position DeepIQ Edge as a robust edge solution for industrial settings, reducing operational costs while maintaining high data integrity and transmission efficiency
Kepware specializes in real-time connectivity to control systems and SCADA, enabling seamless communication with industrial devices for operational monitoring and control.
Configuring the ideal buffering time in Kepware IoT Gateway involves balancing latency, message size, and the potential impact of lost messages. Since Kepware IoT Gateway does not have store-and-forward capabilities, increasing the buffering time also increases the risk of losing more data during connectivity issues. If you have determined an ideal buffering time (e.g., 1500 ms), follow these steps to implement and monitor the configuration:
1. Access Kepware Settings:
2. Configure the Buffering Time:
3. Monitor Logs in CloudWatch:
4. Adjust as Needed:
Billing Payload Efficiency is a new metric focused on IoT cost optimization. By aligning edge software configurations like DeepIQ Edge or Kepware IoT Gateway with Cloud platforms such as AWS IoT Core’s billing model, organizations can significantly reduce costs while improving efficiency. The key is to buffer data intelligently, balancing payload size against transmission frequency and minimizing wasted bandwidth.
Optimizing edge payloads for cloud cost efficiency is vital in the broader context of IT-OT convergence. This strategy is just one piece of a complex puzzle that includes managing the lifecycle of edge software, contextualizing IT and OT data, developing robust cloud data models with stringent versioning and governance, and simplifying the implementation of AI and digital twin workflows. Each of these elements presents challenges and nuances that must be expertly navigated to unlock the full potential of digital transformation initiatives.
The DeepIQ platform has many capabilities and tools tailored to streamline complex IT-OT convergence tasks. It supports a comprehensive array of functions, from constructing edge asset hierarchies to advanced IT contextualization, and facilitates streaming AI and digital twin workflows. As a unified solution, DeepIQ simplifies integration and accelerates the deployment of digital strategies. Engineered to address the intricacies of these processes, our platform ensures meticulous management of every convergence aspect, perfectly aligning with your business objectives.
For organizations looking to explore the full spectrum of benefits that IT-OT convergence can offer, the DeepIQ platform is a proven leader. We have implemented our solutions across some of the world’s largest companies, demonstrating our ability to deliver scalable and effective results. Our platform optimizes costs and empowers organizations to leverage their data more effectively, improving decision-making and operational efficiencies.
We invite you to explore further how the DeepIQ platform can transform your operations. For a deeper insight into our capabilities and the successes our customers have achieved, visit our website and access our comprehensive library of whitepapers and customer success stories at https://deepiq.com.