A Comprehensive Guide to Efficient Logging with EFK and TD Agent

In the dynamic landscape of modern IT infrastructures, effective log management and analysis play a crucial role in maintaining systems' health, security, and performance. The EFK stack, consisting of Elasticsearch, Fluentd, and Kibana, has emerged as a powerful solution for log aggregation, storage, and visualization. When paired with the TD Agent, it becomes an even more potent tool for efficiently handling large volumes of log data.

1. Elasticsearch: A distributed, RESTful search and analytics engine that serves as the storage backend for log data.
2. Fluentd: An open-source data collector that unifies the data collection and consumption for better use and understanding. Fluentd collects, processes, and forwards log data to Elasticsearch.
3. Kibana: A powerful visualization tool that works in conjunction with Elasticsearch to help users explore, analyze, and visualize data stored in Elasticsearch indices. Together, these components provide a comprehensive solution for log management, offering scalability, flexibility, and ease of use.

1. Log Collection: td-agent is primarily used for collecting log data from various sources, such as application logs, system logs, and more. It supports a wide range of input sources.
2. Data Processing: It allows filtering, parsing, and transforming log data. This ensures that the collected data is formatted and structured according to the requirements before being forwarded.
3. Log Forwarding: The td-agent can forward log data to various output destinations, making it suitable for integrating with different storage systems or log analysis tools. Common output destinations include Elasticsearch, Amazon S3, MongoDB, and others.
4. Fluentd Integration: td-agent uses the Fluentd logging daemon as its core. Fluentd provides a flexible and extensible architecture for log data handling. It supports a wide range of plugins, making it adaptable to various environments.
5. Configuration: The configuration of td-agent is typically done through a configuration file (commonly named td-agent.conf). This file defines input sources, processing filters, and output destinations.
6. Ease of Use: td-agent is designed to be easy to install and configure. It is suitable for both small-scale deployments and large, distributed systems.
7. Community and Support: As an open-source project, td-agent benefits from an active community and ongoing development. It's well-documented, and users can find support through forums, documentation, and community channels.

• Manually logging in to the server to check application logs, container logs, nginx logs, etc.
• No centralized monitoring Dashboard.
• No search engine.
• No visualization tool.
• Security issues.
• Most team members have access to the server.

Docker and Docker Compose

Step 1: Create Directory and Change Directory

Step 2: Create docker-compose.yml

Step 3: Create Directory and Change Directory

Step 4: Create Dockerfile

Step 5: Create entrypoint.sh

Step 6:  Create Directory and Change Directory

Step 7: Create fluent.conf

Change directory to efk and run the Docker container:

Check logs for service fluentd and kibana:

Check the running container:

On the Dashboard

2. Next, click the Explore on My Own button on the welcome page below.

5. Now, input the index pattern Name as fluentd-*, set the Timestamp field to @timestamp, and click the Create index pattern button to confirm the index pattern settings.

6. Lastly, click on the top left menu (ellipsis), then click the Discover menu to show the logs monitoring.

Run a Docker container by utilizing the Fluentd log driver

Lastly, switch back to the Kibana dashboard, and click the Discover menu on the left side.

Setting up TD Agent (Treasure Data Agent) using Script allows for automation. TD Agent is commonly used for log forwarding and aggregation. In this guide, we will walk through the steps to set up TD Agent using Script.

td-agent/install_td_agent.sh

td-agent/td-agent.conf.j2

Remember to execute the script with the appropriate permissions:

Check using filter on kibana dashboard

tag- log_name: fluentd.test.follow

Prerequisites: Basic knowledge of Node.js and NPM Package.json.

Use npm to install dependencies locally:

index.js:

Run the app and go to http://public-ip:3000/ in your browser to send the logs to Fluentd:

Add the following lines to fluent.conf. After adding these lines, remove the previous Docker images. Then, run 'docker-compose up' again in the ~/efk directory. tag- log_name: hostname.system.auth.info

Login to the server again to generate auth-log:

Below are fluentd Container Logs:

On Kibana Dashboard:

Add the following lines to fluent.conf. After adding these lines, remove the previous Docker images. Then, run 'docker-compose up' again in the ~/efk directory. tag: container name:

Add the following lines to fluent.conf. After adding these lines, remove the previous Docker images. Then, run 'docker-compose up' again in the ~/efk directory. tag- log_name: hostname.system.daemon.info