How to Set Up a VPS-Based Log Aggregation Pipeline with OpenSearch and Fluentd

When you manage multiple VPS instances, application logs spread across every server — making troubleshooting, security auditing, and performance analysis a manual nightmare. A centralized log aggregation pipeline solves this by collecting logs from all your VPS nodes into a single searchable platform. This guide walks through setting up a production-grade log pipeline using OpenSearch (the open-source fork of Elasticsearch) for storage and search, Fluentd for log collection and forwarding, and OpenSearch Dashboards for visualization. All components run on your own VPS infrastructure with no external SaaS dependency. For a VPS with the RAM and storage to run this pipeline effectively, find the right VPS for your logging infrastructure.

Architecture Overview

The pipeline has three layers:

• Agent layer (Fluentd on each VPS): Tail log files, apply parsing and filtering, and forward to the aggregator.
• Aggregator layer (Fluentd on a central VPS): Receive logs from all agents, buffer them, and write to OpenSearch.
• Storage and search layer (OpenSearch on a central VPS): Index and store logs, serve search queries, and power dashboards.

Prerequisites and VPS Sizing

For a setup aggregating logs from 5–10 VPS instances at ~1 GB of logs per day each:

Component	vCPU	RAM	Storage	OS
Aggregator VPS (Fluentd)	2	4 GB	50 GB NVMe	Ubuntu 24.04
OpenSearch VPS	4	8 GB (min 50% heap)	200 GB NVMe	Ubuntu 24.04
Agent VPS (per node)	1	512 MB	10 GB	Any modern Linux

OpenSearch is memory-hungry — allocate at least 4 GB of your VPS RAM to the JVM heap (-Xms4g -Xmx4g). The remaining RAM handles OS cache for indexing. For higher volumes, scale the OpenSearch VPS vertically or add data nodes in a cluster.

Step 1: Install and Configure OpenSearch

OpenSearch 2.x is available from the official OpenSearch project repository. Install on the central VPS:

# Install OpenSearch 2.x
curl -fsSL https://artifacts.opensearch.org/releases/bundle/opensearch/2.x/opensearch-2.x.list | sudo tee /etc/apt/sources.list.d/opensearch-2.x.list
sudo apt-get update
sudo apt-get install opensearch -y

Configure heap size and security. Edit /etc/opensearch/opensearch.yml:

# /etc/opensearch/opensearch.yml
cluster.name: vps-logs
node.name: node-1
path.data: /var/lib/opensearch
path.logs: /var/log/opensearch
network.host: 0.0.0.0
http.port: 9200
discovery.type: single-node

# Security plugin settings
plugins.security.disabled: false
plugins.security.ssl.http.enabled: true
plugins.security.allow_default_init_securityindex: true

Set heap in /etc/opensearch/jvm.options:

-Xms4g
-Xmx4g

Start and enable OpenSearch:

sudo systemctl daemon-reload
sudo systemctl enable opensearch
sudo systemctl start opensearch
# Verify: curl -k -u admin:admin https://localhost:9200

Change the default admin password immediately:

curl -k -X PUT "https://localhost:9200/_plugins/_security/api/internalusers/admin" \
  -H "Content-Type: application/json" \
  -u admin:admin \
  -d '{"password": "your-new-strong-password"}'

Step 2: Install and Configure Fluentd (Aggregator)

Install Fluentd via the official td-agent package on the aggregator VPS:

# Install td-agent (Fluentd)
curl -fsSL https://toolbelt.treasuredata.com/sh/install-ubuntu-jammy-td-agent4.sh | sh
sudo systemctl enable td-agent
sudo systemctl start td-agent

Configure /etc/td-agent/td-agent.conf to receive logs from agents and forward to OpenSearch:

# /etc/td-agent/td-agent.conf
## Source: receive logs from agents via forward protocol

  @type forward
  port 24224
  bind 0.0.0.0
  
    self_hostname aggregator.vps.local
    shared_key your_shared_secret_key
  


## Parse and transform: add host tag, normalize fields

  @type record_transformer
  
    hostname ${hostname}
    @timestamp ${time.asctime}
  


## Output: send to OpenSearch

  @type opensearch
  host localhost
  port 9200
  scheme https
  user admin
  password your-new-strong-password
  ssl_verify false
  logstash_format true
  logstash_prefix vps-logs
  flush_interval 5s
  
    @type file
    path /var/log/td-agent/buffer
    flush_mode interval
    retry_max_times 5
    chunk_limit_size 8m
  

Install the OpenSearch output plugin:

sudo td-agent-gem install fluent-plugin-opensearch

Restart td-agent: sudo systemctl restart td-agent

Step 3: Configure Fluentd Agents on Each VPS Node

On each VPS that generates logs, install td-agent and configure it to forward logs to the aggregator:

# /etc/td-agent/td-agent.conf
## Source: tail Nginx access logs

  @type tail
  path /var/log/nginx/access.log
  pos_file /var/log/td-agent/nginx-access.pos
  tag nginx.access
  
    @type nginx
  


## Source: tail Nginx error logs

  @type tail
  path /var/log/nginx/error.log
  pos_file /var/log/td-agent/nginx-error.pos
  tag nginx.error
  
    @type none
  


## Source: tail system auth logs

  @type tail
  path /var/log/auth.log
  pos_file /var/log/td-agent/auth.pos
  tag system.auth
  
    @type syslog
  


## Source: tail syslog

  @type tail
  path /var/log/syslog
  pos_file /var/log/td-agent/syslog.pos
  tag system.syslog
  
    @type syslog
  


## Source: Docker container logs (if Docker is running)

  @type tail
  path /var/lib/docker/containers/*/*-json.log
  pos_file /var/log/td-agent/docker.pos
  tag docker.*
  
    @type json
  


## Filter: drop noisy debug logs

  @type grep
  
    key level
    pattern ^debug
  


## Output: forward to aggregator

  @type forward
  
    host AGGREGATOR_VPS_IP
    port 24224
  
  
    self_hostname agent-${hostname}
    shared_key your_shared_secret_key
  
  
    @type file
    path /var/log/td-agent/buffer
    flush_interval 10s
    retry_max_times 3
  

Restart td-agent on each agent: sudo systemctl restart td-agent. Logs will begin flowing to the aggregator and into OpenSearch within seconds.

Step 4: Install OpenSearch Dashboards

On the same VPS as OpenSearch (or a dedicated one for larger deployments):

sudo apt-get install opensearch-dashboards -y

Edit /etc/opensearch-dashboards/opensearch_dashboards.yml:

server.host: 0.0.0.0
server.port: 5601
opensearch.hosts: ["https://localhost:9200"]
opensearch.ssl.verificationMode: none
opensearch.username: admin
opensearch.password: your-new-strong-password

Enable and start:

sudo systemctl enable opensearch-dashboards
sudo systemctl start opensearch-dashboards

Access Dashboards at https://your-vps-ip:5601. Create an index pattern matching vps-logs-* and start exploring your aggregated logs.

Step 5: Configure Index Lifecycle Management

Without lifecycle management, OpenSearch will fill your disk until it stops accepting writes. Create an ILM policy that rolls over indices daily and deletes data older than 30 days:

curl -k -X PUT "https://localhost:9200/_plugins/_ism/policies/log_retention" \
  -u admin:your-new-strong-password \
  -H "Content-Type: application/json" \
  -d '{
    "policy": {
      "description": "Rotate logs daily, delete after 30 days",
      "default_state": "hot",
      "states": [
        {
          "name": "hot",
          "actions": [
            { "rollover": { "min_size": "50gb", "min_index_age": "1d" } }
          ],
          "transitions": [{ "state_name": "warm" }]
        },
        {
          "name": "warm",
          "actions": [],
          "transitions": [{ "state_name": "delete", "conditions": { "min_index_age": "30d" } }]
        },
        {
          "name": "delete",
          "actions": [{ "delete": {} }]
        }
      ],
      "ism_template": { "index_patterns": ["vps-logs-*"], "priority": 100 }
    }
  }'

Security Considerations

• Encrypt Fluentd forward traffic: The shared_key in the forward plugin authenticates agents, but traffic is not encrypted by default. Tunnel Fluentd traffic through WireGuard or use the fluent-plugin-forward-encrypt plugin for TLS.
• Restrict OpenSearch access: Do not expose port 9200 to the internet. If you need remote access, use an SSH tunnel or place OpenSearch behind a reverse proxy with HTTP Basic Auth.
• Log sanitization: Use Fluentd filter plugins to redact sensitive fields (passwords, API keys, PII) before forwarding. The fluent-plugin-record-modifier plugin can mask patterns.
• Disk space monitoring: Set up a Fluentd monitor or cron job that alerts when OpenSearch disk usage exceeds 80%. ILM policies prevent runaway growth but do not replace capacity planning.

Troubleshooting Common Issues

• Fluentd cannot connect to OpenSearch: Verify OpenSearch is running, check TLS settings, and confirm the password. Test with curl -k -u admin:password https://localhost:9200.
• Logs not appearing in Dashboards: Check Fluentd agent logs (/var/log/td-agent/td-agent.log) for forwarding errors. Verify the index pattern matches the logstash_prefix you configured.
• High memory usage on OpenSearch VPS: Reduce the JVM heap size proportionally (max 50% of VPS RAM), or add more RAM to the VPS. Also check if you’re indexing too many fields — disable dynamic:true mapping for verbose logs.
• Buffer overflow on Fluentd aggregator: If log volume spikes exceed OpenSearch’s indexing capacity, Fluentd’s file buffer prevents data loss. Increase chunk_limit_size and flush_interval to smooth out bursts.

With this pipeline running, you gain centralized search across all your VPS logs, real-time dashboards for monitoring, and the ability to correlate events across servers during incident response. Start with a single aggregator VPS and scale horizontally as your log volume grows. For a VPS with enough RAM and NVMe storage to run OpenSearch and Fluentd comfortably, find the right VPS for your logging infrastructure.