logstash gork过滤器错误：提取了无效的配置

zoram 发表于 Dev

佐拉姆

我正在使用Filebeat-> Logstash-> Elasticsearch-> Kibana来了解我的glassfish日志文件。

您知道我的Logstash过滤器配置有什么问题吗？

我的过滤器配置如下所示：

filter {
  if [type] == "log" {
    grok {
      match => { "message", "(?m)\[\#\|%{TIMESTAMP_ISO8601:timestamp}\|%{LOGLEVEL:Log Level}\|%{DATA:server_version}\|%{JAVACLASS:Class}\|%{DATA:thread}\|%{DATA:message_detail}\|\#\]" }
      add_field => [ "Log level", "%{LOGLEVEL:Log Level}" ]
    }
  }
syslog_pri { }
date {
  match => {[ "timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]}
  }
}

当我启动Filebeat 5.0

sudo ./filebeat -e -c filebeat-logstash.yml -d "publish

ERR Failed to publish events caused by: EOF
2016/11/11 14:53:54.397825 single.go:91: INFO Error publishing events (retrying): EOF
2016/11/11 14:54:09.232146 logp.go:230: INFO Non-zero metrics in the last 30s:     filebeat.harvester.open_files=1 libbeat.logstash.call_count.PublishEvents=5 libbeat.logstash.publish.read_errors=5 libbeat.logstash.publish.write_bytes=5542 libbeat.publisher.published_events=2047 filebeat.harvester.running=1 libbeat.logstash.published_but_not_acked_events=10235 filebeat.harvester.started=1

Logstash日志

http://pastebin.com/Hh9ECFjd

{:timestamp=>"2016-11-11T14:40:35.247000+0000", :message=>"fetched an invalid config", :config=>"input {\n  lumberjack {\n    port => 5000\n    type => \"log\"\n    ssl => false\n    #ssl_certificate => \"/etc/pki/tls/certs/logstash-forwarder.crt\"\n    #ssl_key => \"/etc/pki/tls/private/logstash-forwarder.key\"\n  }\n}\n\ninput {\n  beats {\n   port => 5044\n   ssl => false\n   # ssl_certificate => \"/etc/pki/tls/certs/logstash-beats.crt\"\n   # ssl_key => \"/etc/pki/tls/private/logstash-beats.key\"\n  }\n}\n\nfilter {\n  if [type] == \"log\" {\n    grok {\n      match => { \"message\", \"(?m)\\[\\#\\|%{TIMESTAMP_ISO8601:timestamp}\\|%{LOGLEVEL:Log Level}\\|%{DATA:server_version}\\|%{JAVACLASS:Class}\\|%{DATA:thread}\\|%{DATA:message_detail}\\|\\#\\]\" }\n      add_field => [ \"Log level\", \"%{LOGLEVEL:Log Level}\" ]\n    }\n   }\n    syslog_pri { }\n    date {\n      match => {[ \"timestamp\", \"MMM  d HH:mm:ss\", \"MMM dd HH:mm:ss\" ]}\n   }\n}\n\n\nfilter {\n  if [type] == \"nginx-access\" {\n    grok {\n      match => { \"message\" => \"%{NGINXACCESS}\" }\n    }\n  }\n}\n\noutput {\n  elasticsearch {\n    hosts => [\"http://localhost:9200\"]\n    sniffing => true\n    manage_template => false\n    index => \"%{[@metadata][beat]}-%{+YYYY.MM.dd}\"\n    document_type => \"%{[@metadata][type]}\"\n  }\n}\n\n", :reason=>"Expected one of #, => at line 23, column 27 (byte 461) after filter {\n  if [type] == \"log\" {\n    grok {\n      match => { \"message\"", :level=>:error}

提前谢谢了。

汤玛士

仙女

您的配置有几处错误。

grok {
  match => { "message", "(?m)\[\#\|%{TIMESTAMP_ISO8601:timestamp}\|%{LOGLEVEL:log_level}\|%{DATA:server_version}\|%{JAVACLASS:Class}\|%{DATA:thread}\|%{DATA:message_detail}\|\#\]" }
}

请勿在字段名称中使用空格。要解析的字段和grokstring需要一个箭头=>。不支持它们。您也不需要添加loglevel字段，因为grok正在为您执行此操作。

date {
  match => [ "timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
}

您不需要在日期匹配中使用大括号，因为它需要一个数组。

该伐木工人输入您需要使用SSL你不能禁用它ssl => false。但是，这适用于节拍输入。

完整的配置如下所示：

input {
  lumberjack {
    port => 5000
    type => "log"
    ssl_certificate => "/etc/foo.crt"
    ssl_key => "/etc/foo.key"
  }
}
input {
  beats {
   port => 5044
  ssl => false
    }
}
filter {
if [type] == "log" {
    grok {
      match => { "message" =>  "(?m)\[\#\|%{TIMESTAMP_ISO8601:timestamp}\|%{LOGLEVEL:log_level}\|%{DATA:server_version}\|%{JAVACLASS:Class}\|%{DATA:thread}\|%{DATA:message_detail}\|\#\]" }
    }
  }
syslog_pri { }
date {
  match => [ "timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
  }
}
output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    sniffing => true
    manage_template => false
    index => "%[@metadata][beat]}-%{+YYYY.MM.dd"
    document_type => "%[@metadata][type]"
  }
}

关于调试配置的另一件事。您发布了logstash日志，logstash本身正在告诉您它无法解释的内容。

:reason=>"Expected one of #, => at line 23, column 27 (byte 461) after filter {
  if [type] == \"log\" {
    grok {
      match => { \"message\"", :level=>:error}

这样，您便可以快速找到可能错过的错误。

本文收集自互联网，转载请注明来源。

如有侵权，请联系 [email protected] 删除。