Setting up the Akamai integration

Setting up Log File Analysis with the Akamai DataStream (opens in a new tab) in Conductor Website Monitoring is very straightforward! Follow the steps outlined below to enable Log File Analysis in Conductor Website Monitoring, and set up the Akamai integration.

Setting up the Log File Analysis

The process of configuring the Log File Analysis consists of two phases:

1. Enabling the Log File Analysis feature and creating an AWS S3 bucket (opens in a new tab) in Conductor Website Monitoring

2. Configuring the Akamai DataStream in the Akamai UI and associating the DataStream to the corresponding property

1. Enabling the Log File Analysis feature and creating an AWS S3 bucket in Conductor Website Monitoring

If you navigate to Account and then Websites section in Conductor Website Monitoring, you can easily filter and see which of the websites you are monitoring in Conductor Website Monitoring are on Akamai:

From there, just follow the steps outlined below to enable Log File Analysis on the wanted website:

1. Click any website that is running on Akamai in the Websites section (opens in a new tab) of Conductor Website Monitoring
2. Click on Log File Analysis tab in settings
3. Move the Log File Analysis toggle to Enabled state

After this is done, follow the next steps to create an AWS S3 bucket in Conductor Website Monitoring:

• In the same Log File Analysis section, click on How to install link under the Log Sources

• Specify in which region should the AWS S3 bucket be created (EU or US) and click on Create bucket

• After this, Conductor Website Monitoring will automatically generate the AWS credentials and the AWS S3 bucket

• The credentials will be used to associate the AWS S3 bucket with the Akamai DataStream in the next section

2. Configuring the Akamai DataStream

The Akamai DataStream needs to be manually configured in the Akamai UI for every website. The configuration of the Akamai DataStream consists of two steps:

1. Creating a new Akamai DataStream
2. Associating the DataStream to the corresponding property (domain)

Creating a new Akamai DataStream

Follow these steps to create a new Akamai DataStream:

1. Go to "DataStream" section in your Akamai account

Log in to your Akamai Control Center (opens in a new tab) and click on DataStream in the navbar:

2. Click the Create stream button

On the "DataStream" screen, click Create stream:

3. Configure the DataStream

Specify the stream's name (e.g. "Conductor Website Monitoring LFA example.com") and pick the website for which you want to configure the Akamai DataStream in the Properties list:

4. Configure the Data Sets

Important: The log format needs to be JSON.

Choose the following required data sets to be sent to the AWS S3 bucket:

• Log information
  • Request ID
  • Request time
• Message exchange data
  • Client IP
  • HTTP status code
  • Protocol type
  • Request host
  • Request method
  • Request path
  • Request port
  • Response-Content-Type
  • User-Agent
  • SSL version
  • Query string

All the fields listed above need to be selected for the Log File Analysis feature to work properly.

You can also add other data sets, but the ones listed above are required.

Important!

We require streaming only non-PII (non-personally identifiable information) data to our AWS S3.

To do that, you can filter away non-search engine traffic based on the User Agent string when setting up the DataStream directly in Akamai UI.

This is covered in the section "Filter away non-search engine traffic" below.

5. Define a destination

• In the next step, choose "Amazon S3" as the destination where the data will be streamed.
• Fill in the "Display name" (e.g. contentking-lfa-s3-example)
• Enter the "Bucket name" generated by Conductor Website Monitoring to the "Bucket" field
• Path (default suggested by Akamai): logs/{%Y}/{%m}/{%d}/{%H}
• Enter the "Region" that you selected in Conductor Website Monitoring:
  • If you've selected US: enter us-east-2 (located in Ohio)
  • If you've selected EU: enter eu-central-1 (located in Frankfurt)
• Enter the "Access key ID" and "Secret access key" from Conductor Website Monitoring
• Once done, click on Validate & Save

6. Finish the setup

Finish the setup and click on the Save stream button:

Important: After configuring the DataStream, it might take several hours until Akamai sets everything up. Please let us know if it takes more than 24h.

Associate the created DataStream to the corresponding property

After creating the new DataStream, you need to associate it with the corresponding property in your Akamai account.

1. Go to "Properties" section in your Akamai account

Log in to your Akamai Control Center (opens in a new tab) and click on Properties in the navbar:

2. Choose the property

Click the property which you want to associate with the created DataStream:

3. Choose the property version

Click on the Property Version which is being monitored by Conductor Website Monitoring (this can be either the Staging Version or the Production Version):

4. Scroll down to the "DataStream" section in the "Property Configuration Settings"

If the "DataStream" section is missing, follow the steps below to add it. If not, skip to the step 5.

4.1 Scroll all the way down, and click "Edit new version"

4.2 Scroll down to the "Property Configuration Settings", and click "Add Behavior"

4.3 Search for "DataStream" in the modal that appears and select it

4.4 Click "Insert Behavior"

5. Choose the Stream version

In the DataStream box in the Behaviors list, choose DataStream2 as the Stream version

6. Enable the DataStream2 and specify the name

In the DataStream2 section, move the toggle to On, choose the Stream name of the new DataStream you created, and then click the Save button:

Important: Once this is done, ALL the traffic on your website will be streamed to Conductor Website Monitoring's AWS S3 bucket. However, Conductor Website Monitoring will process only search engine visits, and filter out all other traffic, including all PII data.

Follow the next steps to learn how to filter away non-search engine traffic directly in the Akamai DataStream that you've created in the previous steps.

These steps are required!

Ensure Logging of the User Agent header

1. Go to the "Property Configuration Settings"

Go to the "Property Configuration Settings" following steps 1. through 3. in the section "Associate the created DataStream to the corresponding property"

2. Scroll to the "Log Request Details"

Scroll down to the "Log Request Details" section in the "Property Configuration Settings":

In case the "Log Request Details" section is not there, follow the same steps as within step 4. in the section "Associate the created DataStream to the corresponding property" to add it.

3. Enable the "Log User-Agent header"

Make sure the "Log User-Agent header" is enabled, and then click the Save button:

Filter away non-search engine traffic

1. Go to "Property Configuration Settings"

Go to "Property Configuration Settings" following steps 1. through 3. in the section "Associate the created DataStream to the corresponding property", and click on the Add Rule button:

2. Insert the rule for specific User Agents

Choose a "Blank rule template", then give it a name (e.g. "Enable DataStream for specific user agents"), and click the Insert rule button:

3. Add "DataStream" behavior

Click on the Add Behavior button, and add "DataStream" behavior following steps 4.3 and 4.4. in "Associate the created DataStream to the corresponding property":

4. Set up the User Agent match criteria

Click on the Add Match button and set up the criteria for the User Agent to match either the Googlebot or the bingbot user agent:

• Example: User agent is one of:
  • *Googlebot*
  • *bingbot*

And finally, click the Save button:

If you want to learn how to work with the Log File Analysis data in Conductor Website Monitoring, refer to our support article: Working with the Log File Analysis data in Conductor Website Monitoring

Reinstalling the Akamai DataStream

If you want to change the AWS S3 bucket region, or reinstall the Akamai DataStream, you can do it in the following way:

1. Click on the website on which you want to reinstall the Akamai DataStream in the Websites section (opens in a new tab) of Conductor Website Monitoring
2. Click on Log File Analysis in Settings, and then on the reinstall link under Log Sources
3. If needed change the region, and click on Create bucket
4. Create a new DataStream by following the steps above

Disabling Log File Analysis

Same as with enabling Log File Analysis, you need to disable the feature in Conductor Website Monitoring and then remove the Akamai DataStream in the Akamai UI.

Disabling Log File Analysis

1. Click on the website on which you want to disable Log File Analysis in the Websites section (opens in a new tab) of Conductor Website Monitoring
2. Click on the Log File Analysis in the Settings section
3. Move the Log File Analysis toggle to Disabled state

Once this is done, Conductor Website Monitoring will automatically disable Akamai DataStream access to the AWS S3 bucket where the data was streamed.

Removing the Akamai DataStream

If you have disabled the Log File Analysis feature, you still need to remove the Akamai DataStream in your Akamai account.

This needs to be done manually, as Conductor Website Monitoring doesn't have access to your Akamai account.

Frequently asked Security related questions

For the most common security-related questions about Conductor Website Monitoring's Log File Analysis, refer to the FAQ section in the Log File Analysis support article

Need help?

In case you have any questions regarding the Akamai integration that are not covered by our documentation, don’t hesitate to contact us!