Console Deploy (Manual)

Only complete this section if you intend to set up your AWS environment manually via. the AWS console. If you have completed CloudFormation (Automated) skip this section and proceed to Send Log Data to Kinesis Fire Hose

Step 1 - Create an OpenSearch Domain

  1. Go to the OpenSearch Console
  2. Click on Create domain

cloud_shell_button

  1. Enter the name workshop-domain for the OpenSearch Domain
  2. Under the deployment type section, select Development and testing
  3. Under the network section, select Public access
  4. Under the fine-grained access control section select Create master user
  5. For the username enter OSMasterUser
  6. For the password enter AwS#OpenSearch1
  7. Under the access policy section, select Only use fine-grain access control
  8. Leave all other settings at the default selections
  9. Click on Create

It will take approximately 5 - 10 minutes for your OpenSearch domain to be created. Upon successful creation you will see your OpenSearch domain status is active

cloud_shell_button Do not proceed to the next step until you confirm that your domain status is active

Step 2 - Create a Kinesis Firehose

  1. Go to the Kinesis Firehose Console
  2. Click on Create delivery stream

create_delivery_stream

  1. Under the choose source and destination section for the source, select Direct PUT for the destination select Amazon OpenSearch Service
  2. Under the delivery stream name section name the stream workshop-firehose

select_source_destination

  1. Under the destination settings for the OpenSearch service domain, click on Browse and select the OpenSearch domain workshop-domain this is the OpenSearch domain we created in the previous step
  2. Name the index workshop-log
  3. Select Every hour for the Index rotation. This will produce a next index every hour

index_rotation

  1. Expand the Buffer hints section
  2. Adjust the buffer interval to 60 seconds. This will write data from Kinesis Firehose to OpenSearch every 60 seconds

buffer_hints

  1. Under the backup settings under the S3 backup bucket click on Create. This will (in a new browser window) open the create S3 bucket web page

back_up_create_s3

  1. On the create bucket page provide a bucket name. You may name the bucket any valid name
  2. Click on Create bucket at the bottom of the page. Leaving all of the S3 settings the the default selections
  3. Return the browser window that we were using to configure our Kinesis Firehose and under the backup setting section for the S3 backup bucket, click on Browse
  4. Select the bucket you just created. If you do not see the bucket listed click on the small refresh button in the top right corner of the window that pops us when you click on the browse button

back_up_select_s3

  1. At the bottom of the page click on Create delivery stream leave all other settings at the default selections

Step 3 - Configure Identity Access Management (IAM) Permissions

We now have an OpenSearch domain and Kinesis Firehose created. In order to send logs to OpenSearch via Kinesis Data Firehose we need to grant the IAM role that firehose uses permissions in OpenSearch.

Map IAM Role with OpenSeach Role
  1. Go to the OpenSearch Console
  2. Click on the workshop-domain OpenSearch domain you created earlier
  3. Click on the OpenSearch Dashboard URL. This should open the URL in a web browser window

open_search_dashboard

  1. You will be prompted to log in. Using the user name OSMasterUser and password AwS#OpenSearch1 log in
  2. If an additional pop up window is present after login asking about data upload click on Explore on my own
  3. If an additional pop up windows is present asking you to select your tenant select Global and click on Confirm

You should now see a window that looks like this

select_domain

  1. Click on and expand the hamburger menu on the side bar of the OpenSearch home page
  2. Under the OpenSearch Plugins section click on Security

select_security

  1. On the security page click on Roles from the left hand menu

select_roles

  1. On the roles page search for and click on all_access

search_all_access

  1. On the all_access role page click on Mapped users
  2. Under the mapped users page click on Manage mapping

mapped_users

On the manage mapping page we need to map the IAM role the is used by Kinesis Data Firehose to the all_access OpenSearch role. This will give Kinesis Firehose the permissions it need to create, update indexes and write data.

For the purposes of this lab we will give Kinesis Firehose all_access in OpenSearch.

We need to find the ARN of the IAM role Kinesis Firehose is using. Keeping the Manage mapping page open in your browser, navigate to a new tab and

  1. Go to the Kinesis Firehose Console
  2. Click on the workshop-firehose listed. This is the Kinesis Data Fire hose we created earlier
  3. Click on the Configuration section

IAM_role

  1. On the configuration page scroll down to the permissions section
  2. Click on the IAM role

IAM_role

  1. This will open a new window in your web browser. Copy down the ARN of the IAM role

IAM_role

  1. Navigate back to the OpenSearch map user tab
  2. Enter the ARN we copied in step 18 and paste it in the backend roles section of OpenSearch console page
  3. Click on Map

IAM_role

Our AWS environment set up. When you are ready begin the next step Send Log Data to Kinesis Fire Hose