4. Send Logs

You need to send log data to MSK so it can subsequently be sent to OpenSearch (in real time) by the Lambda function we deployed.

You will send the log data to Kafka via. a Python Script run from Cloud9

  1. Navigate to the MSK page in the AWS console
  2. Select the msk-workshop-cluster

enviorment_set_up_msk_1

  1. Click on View client information

enviorment_set_up_msk_2

  1. Copy the plaintext bootstrap servers connection information
  2. Navigate to the Cloud9 console
  3. Click on Open IDE

enviorment_set_up_msk_4

  1. In the Cloud9 environment navigate and open Kafka_OpenSearch_Anomaly_Detection/Kafka/2_base_data.py
  2. Update the <plain_text_bootstrap_server> with the plaintext bootstrap servers connection information you copied down and save the file
  3. Run pip install pykafka
  4. Run cd ~ in the Cloud9 terminal
  5. Run python environment/Kafka_OpenSearch_Anomaly_Detection/Kafka/2_base_data.py

send_log_data_1

Allow the python script to run until the terminal looks similar to the image below

send_log_data_2

  1. Open Kafka_OpenSearch_Anomaly_Detection/Kafka/3_anomoly_data.py
  2. Update the <plain_text_bootstrap_server> with the plaintext bootstrap servers connection information you copied down and save the file
  3. Run python environment/Kafka_OpenSearch_Anomaly_Detection/Kafka/3_anomoly_data.py

send_log_data_3

Allow this python script to run until completion. This may take several minutes. If you leave the Cloud9 environment open in your web browser as a separate tab you can move on to the next steps

When the script is complete the Cloud9 terminal will look similar to the image below

send_log_data_4

We have completed the process of sending data to MSK. When you are ready begin the next step Anomaly Detection