Load Balancing VMware Log Insight With KEMP’s LoadMaster Part 3 – Load Balancing Log Insight with KEMP

This article will show you how to configure a KEMP Load Master for VMware Log Insight.

Log Insight supports receipt and ingestion of Syslog messages that are sent over UDP, TCP, TCP with SSL Encryption and API.

The Load Master provides Log Insight-aware Services to optimise high availability and scalability of Log Insight deployments.

One Concern that KEMP has addressed, is when Syslog Messages are sent using methods other than UDP. Clients will often open long-lived connections that are then used for large amounts of messages. The issue is that traffic is not distributed on a close-to-even fashion across the cluster of available nodes. The Load Master provides a solution to this problem and allows messages to be parsed with a connection to allow a more even distribution across servers in a cluster, as well as simplified scalability of log Insight environments.

I have provided a logical diagram showing how the solution works.

Installing and Configuring the Load Master

In this demo we are only going to use one Load Master, however you can deploy KEMP Load Masters in a HA Pair. For configuring a KEMP HA Solution please see the following Link: KEMP High Availability Configuration Guide

The first thing we need to do is Download a Load Master from KEMP’s Site. KEMP Technologies VLM Download Page

Import the OVF or (VMDK & VMX) into your VMware Infrastructure.

You will then need to license the Load Master using your KEMP ID (Email Address) Please see the following Link: Licensing Feature Description

Adding the Log Insight Add on and Templates

Before we can start creating the Virtual Services for Log Insight, we will need to import the Add-on pack for Log Insight. you can find this at the following link: Log Insight Add-on pack

Once Downloaded, go to System Administration > Update Software

Under the Installed Addon Packages, you can browse the Addon file, once selected, click Install Addon Package.

You will now see the Addon Package for Log Insight (Log_Insight). Ensure that you obtain the same Addon version as your Firmware Version. For Example if you have a Firmware Version 7.1-20a, you will need a Addon Version of 7.1-20a.

You will need to reboot the Load Master once the install is complete for the Addon to become active.

Adding the Log Insight Templates

KEMP have simplyfyed the Deployment of Log Insight by create Templates that can be imported into the Load Master.

You can find the Template file at the following Link: Log Insight Template

The template contains the following Virtual Services: UDP, TCP, SSL, and API Services

To import the Template, Navigate to Virtual Services > Manage Templates

Select the Template File you want to Import, then click “Add New Template”

Once imported you will see the Templates for all four Log Insight Virtual Services.

Configure Log Insight Message Split Interval

The Log Insight Split Interval Value Controls how many Syslog messages should be sent to each server in the cluster before moving to the next server. If you set the Split interval to 1, then it would allow a single message to be sent Server A before sending a message to Server B , then C, then again back to Server A.

Configure the Log Insight Split Interval:

Go to the Main Menu of WUI, > System Configuration > Miscellaneous Options > L7 Configuration

I have set the Log Insight Message Split Interval to “1”

Configuring the Log Insight Virtual Services

Create the TCP Syslog Virtual Service

A TCP Syslog Virtual Service Must be create if clients will send syslog messages to Log insight TCP.

To create the Service, Navigate to Virtual Services > Add New

Enter the Shared Virtual Service and Select the Template Log Insight TCP, This will then Populate all the required fields.

Standard Options Section:

It is recommended that you use Round Robin as the Scheduling Method as Round Robin is typically best to accomplish even traffic distribution. Avoid using Least Connection as the Scheduling Method as the Log Insight Split Interval will not behave as expected.

Add the Real Servers (Master Node & Worker Node)

Firstly Configure the Real Server Check Parameters and set the Port “514”

Then add the Real Servers.

Create the UDP Syslog Virtual Service

A UDP Syslog Virtual Service Must be create if clients will send syslog messages to Log insight UDP.

To create the Service, Navigate to Virtual Services > Add New

Configuring Standard Options

Ensure that Force L7 (Layer 7) is selected and Ensure that transparency Check Box is Selected.

Transparency allows the client’s IP address to be presented to the Log Insight Servers. Depending on your Network Topology, Transparency may not be supported. You can disable this option and set the persistence to Source IP if it is not supported.

It is recommended that you set the Idle Connection Timeout value to 1 as it has been tested and shows the best performance for Log Insight’s UDP Service.

Real Servers

The UDP Real Server Check Parameter should use ICMP Ping

Add all the real Servers.

Once Complete, you will see the Virtual Service showing as up.

Create the SSL Syslog Virtual Service

A SSL Syslog Virtual Service Must be create if clients will send syslog messages to Log insight SSL.

To create the Service, Navigate to Virtual Services > Add New

For this Service, we will need to configure the Certificate from Log Insight:

Import the Log Insight Certificate:

Certificates > Import Certificate >

Assign the Certificate to the Virtual Service.

Real Servers:

enter the Checked port 514

Ensure that the Real Servers are configured with the port 514.

Log Insight API Ingest Service

“if HTTP POST requests” are used to programmatically send log information to the Log Insight cluster. A “Log Split” Content rule is required and an accompanying Virtual Service Must be created. Content rules interrogate incoming client connections and make decisions as well as header modification based on the contents of the requests.

We need to create this rule to ensure even distribution of messages across the cluster of Log Insight nodes when the API Ingest Service is Utilised.

Creating the Log Split Content Rule:

A “Log Split” content rule is required to minimise “Lumpiness” (as KEMP Call it) to accomplish a more even distribution of messages that are posted.

Navigate to Rules & Checking > Content Rules

The Log Insight template has the content rule configured, you can check the configuration by selecting modify.