5. Anomaly Detection

Now that you have sent data to MSK via. the two python scripts in Cloud9. Lets first validate that we can see the log data in OpenSearch. Then second, lets create our anomaly detector.

Step 1 - Create Index Pattern

  1. Navigate to the OpenSearch page in the AWS console
  2. Click on workshop-domain

anomoly_1

  1. Click on the OpenSearch Dashboards URL

anomoly_2

  1. You will be prompted to log in. For the user name enter OSMasterUser for the password enter AwS#OpenSearch1
  2. If an additional popup window is present after login asking about data upload click on Explore on my own
  3. If an additional popup window is present asking you to select your tenant select Global and click on Confirm
  4. In the OpenSearch Dashboard, expand the side menu and click on Stack Management under management section

stack_management

  1. On the stack management page click on Index Patterns on the left hand menu

index_pattern

  1. On the index patterns page click on Create index pattern

create_index_pattern_1

  1. Enter infa-logs-* under the index pattern name section

anomoly_3

  1. Click on Next step
  2. Click on the time field drop down and select eventtime

anomoly_4

  1. Click on Create index pattern

You have now created an index pattern! You can use the index pattern to search our log and validate that the logs have been send from MSK to OpenSearch

Step 2 - Search Logs

  1. In the OpenSearch Dashboard expand the side menu and click on Discover under the OpenSearch Dashboards section

search_1

  1. Expand the time range that OpenSearch will view to the Last 3 year

anomoly_5

You will now be able to see logs from the past 3 yrs

anomoly_6

Step 3 - Create Anomoly Detector

  1. In the OpenSearch Dashboard expand the side menu and click on Anomaly detection under the OpenSearch Plugins section

anomoly_7

  1. Click on Create detector

anomoly_8

  1. Enter cpu_detector for the name of the detector
  2. Pick infa-logs-1 for the data source of the detector

anomoly_9

  1. Pick eventtime for the timestamp of the detector
  2. Change the detector interval to 1440
  3. Click on Next

anomoly_10

  1. Enter CPU-Utilization for the feature name
  2. Select average() for the aggregation method
  3. Select cpu_util for the field

anomoly_11

  1. Click the check box to enable categorical fields
  2. Select application_id for the categorical field
  3. Click on Next

anomoly_12

  1. Click to uncheck the box to start real-time detector automatically
  2. Click the check box to run historical analysis detection
  3. Adjust the historical analysis date range to the last 3 yrs
  4. Click on Apply
  5. Click on Next

anomoly_13

  1. Click on Create detector

Step 4 - View Anomaly Detector Results

  1. Click on the Historical analysis tab

anomoly_14

You will now be able to see a heat map of the anomaly’s detected by application. Click on an of the rectangles in the heat map to see a more detailed view of the anomaly

anomoly_15

When you are ready proceed to the next step Clean Up if you want to delete the resources we used for this workshop