Collecting Logs with Logstash

Lab Solution

Copy the new Apache file to Logstash in the same way:

cp data/apache_logs-2021-05 labs/logstash/data/

Check the indices:

curl localhost:9200/_cat/indices?v

You'll see several new indices starting apache-2021.05, and one with today's date

Query the documents for today's index, e.g.

curl 'localhost:9200/apache-2021.07.18/_search?size=1&pretty'

Logstash creates a document for every log, even if it fails processing. You'll see a document like this:

{
   "@version":"1",
   "host":"d22c222ad09a",
   "path":"/data/apache_logs-2021-05",
   "tags":[
      "_grokparsefailure"
   ],
   "message":"212.84.56.58 - - [21/Jly/2021:06:05:18 +0000] \"GET /projects/xdotool/xdotool.xhtml HTTP/1.1\" 200 50112 \"http://www.google.it/url?sa=t&rct=j&q=&esrc=s&source=web&cd=2&ved=0CEQQFjAB&url=http%3A%2F%2Fwww.semicomplete.com%2Fprojects%2Fxdotool%2Fxdotool.xhtml&ei=7j0BU-3qL4nB0gWP2YHQDg&usg=AFQjCNFwZFAI0RQdN_N0kFH-oj8cLsJRNQ&bvm=bv.61535280,d.d2k&cad=rja\" \"Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:24.0) Gecko/20100101 Firefox/24.0\"",
   "@timestamp":"2021-07-18T20:42:26.490Z"
}

There's a tags field with the value _grokparsefailure, which tells you the incoming line failed to parse correctly.

It's going to be difficult to debug that - the problem is actually the date: 21/Jly/2021 uses an invalid month name. That's enough to fail the Regular Expression so none of the other fields are extracted.

Collecting Logs with Logstash

#Lab Solution

Lab Solution