Managing Logstash with the Redis Client

Users of Logstash will be familiar with the stack of technologies required to implement a logstash solution:

The client that ships the logs to Redis

Redis which queues up the files for indexing

Logstash which creates the indices

Elasticsearch which stores the indices

Kibana which queries Elasticsearch

When you’re dealing with multiple components like this, things will inevitably for wrong.

For instance, say for some reason you client stops, and then you start it again 4 days later, and now the stack has to process 4 days of old log files before letting you search the latest ones.

One of the best ways to deal with this is to setup the Redis queue (“list” is the correct term) so that you can selectively remove entries from the list, so that chunks of old logs can be skipped.

Take a look at this config from the logstash shipper:


output {
  stdout { debug => false debug_format => "json"}
  redis { host => "172.32.1.172" data_type => "channel" key => "logstash-%{@type}-%{+yyyy.MM.dd.HH}" }
}

You’ll see here that I’ve modified the default key value for logstash, by adding the log file type and date stamp to the key. The default key value in the Logstash documentation is “logstash’, which means every entry goes into Redis with the same key value.

You will also notice that I have changed the data_type from the default “list” to “channel’, more of which in a moment.

To see what this means, you should now login to your Redis server with the standard redis-cli command line interface

To list all available keys, just type


KEYS *logstash*

and you will get something like


redis 127.0.0.1:6379> keys *logstash*
 1) "logstash-nodelog-2014.03.07.17"
 2) "logstash-javalog-2014.03.07.15"
 3) "logstash-applog-2014.03.07.14"
 4) "logstash-catalina-2014.03.08.23"
 5) "logstash-applog-2014.03.08.23"
 6) "logstash-catalina-2014.03.07.15"
 7) "logstash-nodelog-2014.03.07.14"
 8) "logstash-javalog-2014.03.07.14"
 9) "logstash-nodelog-2014.03.08.23"
10) "logstash-applog-2014.03.07.15"
11) "logstash-javalog-2014.03.08.23"

This shows that your log data are now stored in Redis according to log file type, and data and hour, rather than all just under the default “logstash” key. In other words, there are now multiple keys, rather than just the “logstash” key which is the default.

You also need to change the indexer configuration at this point, so that it looks for multiple keys in Redis rather than just the “logstash” key


input {
  redis {
    host => "127.0.0.1"
    type => "redis-input"
    # these settings should match the output of the agent
    data_type => "pattern_channel"
    key => "logstash*"

    # We use json_event here since the sender is a logstash agent
    format => "json_event"
  }
}

For data_type here, I am using “pattern_channel”, which means the indexer will ingest the data from any key where the key matches the pattern “logstash*”.

If you don’t change this, and you have changed your shipper, none of your data will get to Elasticsearch.

Using Redis in this way also requires a change to the default Redis configuration. When Logstash keys are stored in Redis in a List format, the List is constantly popped by the Logstash indexer, so it remains in a steady state in terms of memory usage.

When the Logstash Indexer pull data from a Redis channel, the data isn’t removed from Redis, and therefore grows.

To deal with this, you need to set up memory management in Redis, namely:

maxmemory 500mb
maxmemory-policy allkeys-lru

What this means is that when Redis reaches a limit of 500mb of used memory, it will drop keys according to a “Least Recently Used” algorithm. The default algorithm is volatile-lru, which is dependent on the TTL value of the key, but as Logstash doesn’t set the TTL on Redis keys, which need to use the allkeys-lru alternatively instead.

Now, if you want to remove a particular log file type from a particular date and time from the Logstash process, you can simply delete that data from Redis


DEL logstash-javalog-2014.03.08.23

You can also check the length of individual lists by using LLEN, to give you an idea of which logs from which dates and times will take the longest to process


redis 127.0.0.1:6379> llen logstash-javalog-2014.03.08.23
(integer) 385460

You can also check you memory consumption in Redis with:

redis 127.0.0.1:6379>info

3 thoughts on “Managing Logstash with the Redis Client

  1. Skou

    Hi,

    Interesting post , thanks for sharing your knowledge.

    I have updated my logstash config shipper and indexer as your have posted and since them I’m getting empty log entries in Kibana with the bellow fields :

    “”"
    @timestamp
    @version
    _id
    _index
    _type
    type
    “”"

    In Redis can only see a single logstash key :

    “”"
    127.0.0.1:6379> KEYS *
    1) “logstash”
    “”"

    Can you please advise ? Also can you paste the whole Redis and ElasticSearch config ?

    Kind regards,
    Skou.

  2. aengler

    Hello,

    thank you very much for this instruction. May I ask for your help regarding the first part (logstash-shipper -> redis)?

    i’ve written in my config file, as you suggested, but with host=>”localhost” instead of actual IP and i’ve changed the key to “logstash” for simplicity reason:

    output {
    redis { host => “127.0.0.1″ data_type => “channel” key => “logstash”}
    }

    it all works perfectly fine, when i use data_type => “list”. There will be a KEY and LPUSH logstash provides the values.

    following the doc on http://logstash.net/docs/1.4.2/outputs/redis#data_type it seems, that PUBLISH somehow doesn’t work. (btw. do you understand what “TODO set required true” means?)

    when connecting to the redis-server via ./redis-cli, the command KEYS * doesn’t show any key called “logstash” as would have been expected.

    my solution for now only works with data_type => “list” but it would be desired to have a “channel” instead to be able to use “channel_pattern” afterwards (reading from redis).

    best regards,
    aengler

Leave a Reply

Your email address will not be published.

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>