EXAM DP‐203 OBJECTIVES COVERED IN THIS CHAPTER:
- Monitor data storage and data processing
- Develop a batch processing solution
- Develop a stream processing solution
WHAT YOU WILL LEARN IN THIS CHAPTER:
- An overview of Azure monitoring
As you learned in Chapter 8, “Keeping Data Safe and Secure,” before you can design a data security solution, you need to perform an audit. An audit discovers the data sources and analyzes the kind of data that resides within them. You need to know what you have before you can design a solution to protect it. Monitoring the Azure platform and the applications running on it is as necessary as an audit. The information you gather from monitoring is what you use to clarify the dependability of the platform, the applications running within it, or other application dependencies. Without any data that tracks the behavior of your application, it will not be possible to confirm any reported or unreported performance or availability issues. Furthermore, without application data captured from monitoring, you would not be able to perform any root cause analysis to find the reason for application issues. Finally, application data is what is most often used to determine which actions should be taken to improve any poorly behaving applications or specific areas within the applications.
Being proactive means taking action to prevent an inevitable event from happening before it does. You do not want to wait until a customer reports a problem before realizing that you need to enable monitoring. Historically, having a monitoring strategy and implementation plan was not a very high priority. Most of the focus was on developing new features, resolving any high impact bugs, and keeping the production system running. However, running your IT solutions in the cloud, on a third‐party platform, is a reason to take monitoring more seriously. It is often very complicated to distinguish platform‐related issues from application‐related issues. And you must know that the cloud platform provider does not log any information about the applications running within it, due to privacy regulations. So, if you do not enable monitoring and your application fails, there will be no record of it. Therefore, you must enable monitoring, and you must create an end‐to‐end monitoring solution. That solution must incorporate the analysis of monitoring logs, which drives proactive actions to improve reliability and avoid problems before they happen.
The most used product on the Azure platform for implementing a monitoring strategy is Azure Monitor, which was introduced in Chapter 1, “Gaining the Azure Data Engineer Associate Certification,” and shown in Figure 1.29. Azure Monitor consists of numerous technologies and features that will help you gain insights, visualize, analyze, and respond from the telemetry produced from the platform and your applications. This chapter covers monitoring capabilities on the Azure platform and Azure Monitor and includes details about implementing, retrieving, and interpreting monitoring logs.
Monitoring Data Storage and Data Processing
This section covers how to implement, configure, store, retrieve, and interpret monitoring logs. The logs discussed focus primarily on platform performance and availability. These logs do not track your application code or your custom configurations, but they do track and measure how your solution is functioning on the platform. However, determining why your solution behaves how it does requires a deeper level of logging. The next sections discuss how you can get the most out of the monitoring products and features available on the Azure platform.
Implement Logging Used by Azure Monitor
Azure Monitor is a management workspace that helps you integrate and create a monitoring solution. Most PaaS Azure products have many of the Azure Monitor features built in. That means that you can configure, store, and analyze performance data from the monitoring features bound to the product itself in the Azure portal. For a hybrid cloud solution with on‐premises resources or IaaS products that you want to monitor with Azure Monitor, you need to have provisioned an Azure Monitor workspace. Perform Exercise 9.1, where you will create this workspace.
