Review Performance Efficiency
When architecting a cloud solution, there are a fixed set of options to choose from. Over time, new technologies and approaches become available that possibly improve the performance of the created architecture. Using Devek in the cloud, it’s much easier to experiment with new features and services because the infrastructure is code.
Take advantage of the continual innovation of cloud providers, which is driven by customer need. Reduction of prices and releases of new geographical regions, edge locations, services, and features, any of which could positively improve the performance efficiency of an existing architecture.
After identifying the architectural approach, use benchmarking and load-testing data to drive the selection of resource types and configuration options.
To adopt a data-driven approach to architecture, implement a performance review process that considers the following:
Infrastructure as Code
Define the infrastructure as code using Devek. The use of saved designs allows users to place infrastructure into source control alongside the application code and configurations. This enables users to apply the same practices used to develop software to infrastructure, enabling users to iterate rapidly.
Use a continuous integration/continuous deployment (CI/CD) pipeline (for example, source code repository, build systems, deployment, and testing automation) to deploy new environments of infrastructure using Devek. This enables users to deploy in a repeatable, consistent, and low-cost fashion as the architecture iterates over its revisions.
Set up metrics and monitoring to capture key performance indicators (KPI’s). We recommend using both technical and business metrics. For website or mobile apps, key metrics are capturing time to first byte or rendering. Other generally applicable metrics include thread count, garbage collection rate, and wait states. Business metrics - for example, the aggregate cumulative cost per request - alert users about possible ways to drive down costs. Carefully consider how to interpret metrics; for example, choosing the maximum or 99th percentile instead of the average.
Performance Test Automatically
As part of the deployment process, automatically trigger performance tests after the quicker running tests have passed successfully. Devek is the automation that creates a new environment, sets up initial conditions (e.g. test data), and then a CI/CD system will execute a series of benchmarks and load tests. Results from these tests are tied back to the build to track performance changes over time. For long-running tests, make this part of the pipeline asynchronous from the rest of the build. Alternatively, execute performance tests overnight.
Load Generation Create a series of test scripts that replicate synthetic or prerecorded user journeys. These scripts need to be idempotent and not coupled, and might need to include prewarming scripts to yield valid results. As much as possible, the test scripts need to replicate the behavior of usage in production. Use software or software-as-a-service (SaaS) solutions to generate the load.
Key metrics must be visible to the team, especially metrics against each build version. This allows users to see any significant positive or negative trend over time. Include a display of metrics on the number of errors or exceptions from a working system.
Use visualization techniques that make it clear where performance issues, hot spots, wait states, or low utilization are occurring. Overlay performance metrics over architecture diagrams – call graphs or code helps identify issues more quickly.
This performance review process is to be implemented as a simple extension of the existing deployment pipeline and then evolved over time as the testing requirements become more sophisticated. For future architectures, generalize this approach and reuse the same process and artifacts.
When architectures perform badly, this is normally because a performance review process has not been put into place or is broken. If the architecture is performing badly, putting this performance review process in place will allow users to apply Deming’s plan-do-check-act (PDCA) cycle to drive iterative improvement.
Benchmarking uses synthetic tests to provide data on how components perform. It is important to use benchmarking to drive improvements in workloads and architecture.
Benchmarking is generally quicker to set up than load testing, and is used to evaluate the technology for a particular component. Benchmarking is often used at the start of a new project, when no whole solution is yet available to load test.
For benchmarking, follow the performance review process. However, the deployment pipeline will just consist of the benchmark tests themselves. Either build custom benchmark tests, or use an industry standard test like TPC-DS (to benchmark data warehousing workloads). Industry benchmarks are helpful when comparing different environments. Custom benchmarks are useful for targeting specific types of operations that are expected to be part of the architecture.
With benchmarking, it is generally more important to pre-warm a test environment to ensure valid results. Run the same benchmark multiple times to be sure any variance is captured over time.
Since benchmarks are generally faster to run than load tests, use these earlier in the deployment pipeline to provide faster feedback on performance deviations to the engineering team. When evaluating a significant change in a component or service, a benchmark is a quick way to see if the effort to make a change is justified based on the benchmarked difference. Use benchmarks in conjunction with load testing because load testing will tell how the whole workload will perform in production.
Load testing uses the actual workload to show how the whole solution performs in a production environment. Load tests are done using synthetic or sanitized versions of production data (remove sensitive or identifying information) using replayed or pre-programmed user journeys through the application at scale that exercise the entire architecture. As part of the delivery pipeline, load tests are automatically carried out and compared against pre-defined KPI’s and thresholds to ensure that the workload continues to get the required performance.
Collect metrics across the resources in the architecture into a monitoring system, including custom metrics to surface business or derived metrics. Then use the monitoring system to set alarms that indicate when thresholds are breached to signal that a test is outside of expected performance parameters.
With in-cloud environments, the cost is only for used resources; pay for the test environment only when it is needed. Run production-scale environments to test the architecture aggressively. Carry out full-scale testing at a fraction of the cost of using an on-premises environment. Take advantage of the cloud to test the workload to see where it fails to scale, or scales in a non-linear way. Use cheap Spot instances to generate loads at low cost and discover bottlenecks before they are experienced in production.
When writing critical user stories for the architecture, include performance requirements. Specify how quickly each critical story must execute. For these critical stories, implement additional scripted user journeys to ensure visibility into how these stories perform against the requirement.
Where load tests take considerable time to execute, parallelize them using multiple copies of the test environment. The costs will be similar, but testing time will be reduced. (It costs the same to run one compute instance for 100 hours as it does to run 100 instances for one hour). Also lower the costs of these tests by using Spot Instances and selecting Regions that have lower costs than the Regions used for production.
The location of the load test clients needs to reflect the geographic spread of actual end users.
Join Devek in reducing Cloud complexity
Looking to reduce complexity of cloud infrastructure? Look no further, we are here to make it happen!
Please leave some details and we will get back to you when Devek is available for trying out.