input { beats { port => 5044 no codec multiline here unless Filebeat is not doing it,…
- User
input { beats { port => 5044 # no codec multiline here unless Filebeat is not doing it, # but better to do multiline in Filebeat } }
filter { grok { match => { "message" => "^ModSecurity:\s*%{GREEDYDATA:modsec_message}\s*(?:[(?.*)])$" } tag_on_failure => ["grok_prefix_failed"] }
if "_grokparsefailure" not in [tags] and [rest] { kv { source => "rest" field_split => "] [" value_split => " " trim_key => " \t" trim_value => """ include_brackets => false value_split_pattern => " " } } }
output { elasticsearch { hosts => ["http://localhost:9200"] index => "modsecurity-%{+YYYY.MM.dd}" } stdout { codec => rubydebug } }
May 22 16:53:40 debian systemd[1]: Started logstash.service - logstash. May 22 16:53:40 debian logstash[10659]: Using bundled JDK: /usr/share/logstash/jdk May 22 16:53:58 debian logstash[10659]: Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties May 22 16:53:58 debian logstash[10659]: [2025-05-22T16:53:58,521][INFO ][logstash.runner ] Log4j configuration path used is: /etc/logstash/log4j2.properties May 22 16:53:58 debian logstash[10659]: [2025-05-22T16:53:58,529][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"9.0.0", "jruby.version"=>"jruby 9.4.9.0 (3.1.4) 2024-11-04 547c6b150e OpenJDK 64-Bit Server VM 21.0.6+7-LTS on 21.0.6+7-LTS +indy +jit [x86_64-linux]"} May 22 16:53:58 debian logstash[10659]: [2025-05-22T16:53:58,533][INFO ][logstash.runner ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED, -Dio.netty.allocator.maxOrder=11] May 22 16:53:58 debian logstash[10659]: [2025-05-22T16:53:58,588][INFO ][org.logstash.jackson.StreamReadConstraintsUtil] Jackson default value override
logstash.jackson.stream-read-constraints.max-string-lengthconfigured to200000000(logstash default) May 22 16:53:58 debian logstash[10659]: [2025-05-22T16:53:58,588][INFO ][org.logstash.jackson.StreamReadConstraintsUtil] Jackson default value overridelogstash.jackson.stream-read-constraints.max-number-lengthconfigured to10000(logstash default) May 22 16:53:58 debian logstash[10659]: [2025-05-22T16:53:58,588][INFO ][org.logstash.jackson.StreamReadConstraintsUtil] Jackson default value overridelogstash.jackson.stream-read-constraints.max-nesting-depthconfigured to1000(logstash default) May 22 16:53:59 debian logstash[10659]: [2025-05-22T16:53:59,794][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false} May 22 16:54:00 debian logstash[10659]: [2025-05-22T16:54:00,587][INFO ][org.reflections.Reflections] Reflections took 157 ms to scan 1 urls, producing 149 keys and 521 values May 22 16:54:01 debian logstash[10659]: [2025-05-22T16:54:01,385][INFO ][logstash.javapipeline ] Pipelinemainis configured withpipeline.ecs_compatibility: v8setting. All plugins in this pipeline will default toecs_compatibility => v8unless explicitly configured otherwise. May 22 16:54:01 debian logstash[10659]: [2025-05-22T16:54:01,419][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://localhost:9200"]} May 22 16:54:01 debian logstash[10659]: [2025-05-22T16:54:01,547][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}} May 22 16:54:01 debian logstash[10659]: [2025-05-22T16:54:01,672][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"http://localhost:9200/"} May 22 16:54:01 debian logstash[10659]: [2025-05-22T16:54:01,673][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch version determined (9.0.0) {:es_version=>9} May 22 16:54:01 debian logstash[10659]: [2025-05-22T16:54:01,690][INFO ][logstash.outputs.elasticsearch][main] Not eligible for data streams because config contains one or more settings that are not compatible with data streams: {"index"=>"modsecurity-%{+YYYY.MM.dd}"} May 22 16:54:01 debian logstash[10659]: [2025-05-22T16:54:01,692][INFO ][logstash.outputs.elasticsearch][main] Data streams auto configuration (data_stream => autoor unset) resolved tofalseMay 22 16:54:01 debian logstash[10659]: /usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-filter-kv-4.7.0/lib/logstash/filters/kv.rb:389: warning: character class has ']' without escape May 22 16:54:01 debian logstash[10659]: [2025-05-22T16:54:01,699][ERROR][logstash.javapipeline ][main] Pipeline error {:pipeline_id=>"main", :exception=>#<RegexpError: empty char-class: /[] []/>, :backtrace=>["org/jruby/RubyRegexp.java:914:ininitialize'", "org/jruby/RubyClass.java:922:innew'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-filter-kv-4.7.0/lib/logstash/filters/kv.rb:389:inregister'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-mixin-ecs_compatibility_support-1.3.0-java/lib/logstash/plugin_mixins/ecs_compatibility_support/target_check.rb:48:inregister'", "org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:75:inregister'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:245:inblock in register_plugins'", "org/jruby/RubyArray.java:1981:ineach'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:244:inregister_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:623:inmaybe_setup_out_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:257:instart_workers'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:198:inrun'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:150:inblock in start'"], "pipeline.sources"=>["/etc/logstash/conf.d/modsecurity.conf"], :thread=>"#<Thread:0x5c1845fe /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:138 run>"} May 22 16:54:01 debian logstash[10659]: [2025-05-22T16:54:01,700][INFO ][logstash.javapipeline ][main] Pipeline terminated {"pipeline.id"=>"main"} May 22 16:54:01 debian logstash[10659]: [2025-05-22T16:54:01,706][INFO ][logstash.outputs.elasticsearch][main] Using a default mapping template {:es_version=>9, :ecs_compatibility=>:v8} May 22 16:54:01 debian logstash[10659]: [2025-05-22T16:54:01,713][ERROR][logstash.agent ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create, action_result: false", :backtrace=>nil} May 22 16:54:01 debian logstash[10659]: [2025-05-22T16:54:01,735][INFO ][logstash.runner ] Logstash shut down. May 22 16:54:01 debian logstash[10659]: [2025-05-22T16:54:01,758][INFO ][logstash.outputs.elasticsearch][main] Failed to perform request {:message=>"Connection pool shut down", :exception=>Manticore::ClientStoppedException, :cause=>#<Java::JavaLang::IllegalStateException: Connection pool shut down>} May 22 16:54:01 debian logstash[10659]: [2025-05-22T16:54:01,763][ERROR][logstash.outputs.elasticsearch][main] Failed to install template {:message=>"Mutex relocking by same thread", :exception=>ThreadError, :backtrace=>["org/jruby/ext/thread/Mutex.java:101:inlock'", "org/jruby/ext/thread/Mutex.java:172:insynchronize'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-elasticsearch-12.0.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:486:inreturn_connection'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-elasticsearch-12.0.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:434:inwith_connection'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-elasticsearch-12.0.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:335:inperform_request'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-elasticsearch-12.0.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:343:inblock in Pool'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-elasticsearch-12.0.2-java/lib/logstash/outputs/elasticsearch/http_client.rb:410:inexists?'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-elasticsearch-12.0.2-java/lib/logstash/outputs/elasticsearch/http_client.rb:418:intemplate_exists?'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-elasticsearch-12.0.2-java/lib/logstash/outputs/elasticsearch/http_client.rb:82:intemplate_install'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-elasticsearch-12.0.2-java/lib/logstash/outputs/elasticsearch/template_manager.rb:44:ininstall'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-elasticsearch-12.0.2-java/lib/logstash/outputs/elasticsearch/template_manager.rb:32:ininstall_template'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-elasticsearch-12.0.2-java/lib/logstash/outputs/elasticsearch.rb:627:ininstall_template'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-elasticsearch-12.0.2-java/lib/logstash/outputs/elasticsearch.rb:346:infinish_register'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-elasticsearch-12.0.2-java/lib/logstash/outputs/elasticsearch.rb:303:inblock in register'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-elasticsearch-12.0.2-java/lib/logstash/plugin_mixins/elasticsearch/common.rb:172:in `block in after_successful_connection'"]} May 22 16:54:01 debian logstash[10659]: [2025-05-22T16:54:01,766][FATAL][org.logstash.Logstash ] Logstash stopped processing because of an error: (SystemExit) exit May 22 16:54:01 debian logstash[10659]: org.jruby.exceptions.SystemExit: (SystemExit) exit May 22 16:54:01 debian logstash[10659]: at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:924) ~[jruby.jar:?] May 22 16:54:01 debian logstash[10659]: at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:883) ~[jruby.jar:?] May 22 16:54:01 debian logstash[10659]: at usr.share.logstash.lib.bootstrap.environment.(/usr/share/logstash/lib/bootstrap/environment.rb:90) ~[?:?] - Assistant
The root cause of your Logstash pipeline failure is this error:
Pipeline error {:pipeline_id=>"main", :exception=>#<RegexpError: empty char-class: /[] []/>}The error indicates that the regular expression used internally in your kv filter has an invalid character class:
[].
Why this happens
In your kv filter config, you have:
kv { source => "rest" field_split => "] [" value_split => " " trim_key => " \t" trim_value => "\"" include_brackets => false value_split_pattern => " " }Specifically,
field_split => "] ["is problematic.field_splitshould be a string or a valid regex that describes how key-value pairs are separated.- The string
"] ["is converted internally to a regex character class, but because it includes]and[, it results in an invalid regex with an empty character class.
This is causing the regex compile error in the kv filter plugin when it tries to compile the regex for parsing.
Also, the log shows this warning before the failure:
warning: character class has ']' without escape
How to fix it
You want to split on literal string
] [(right square bracket, space, left square bracket).You have two options:
Option 1: Escape brackets in field_split (use regex syntax)
Set
field_splitwith a proper regex that escapes brackets:field_split => "\] \["or better, use a regex literal:
field_split => "\] \["Note: In Logstash config, this works because:
field_splitcan accept a regex string, so escaping special chars is necessary.
Option 2: Use a regular expression for field_split_pattern instead
Alternatively, you can omit
field_splitand usefield_split_patternthat accepts a regex:field_split_pattern => "\] \["and remove
field_splitline.
Recommended corrected filter section
filter { grok { match => { "message" => "^ModSecurity:\s*%{GREEDYDATA:modsec_message}\s*(?:\[(?<rest>.*)\])$" } tag_on_failure => ["grok_prefix_failed"] } if "_grokparsefailure" not in [tags] and [rest] { kv { source => "rest" field_split_pattern => "\] \[" # <- use this instead of field_split value_split => " " trim_key => " \t" trim_value => "\"" include_brackets => false } } }or
filter { grok { match => { "message" => "^ModSecurity:\s*%{GREEDYDATA:modsec_message}\s*(?:\[(?<rest>.*)\])$" } tag_on_failure => ["grok_prefix_failed"] } if "_grokparsefailure" not in [tags] and [rest] { kv { source => "rest" field_split => "\] \[" value_split => " " trim_key => " \t" trim_value => "\"" include_brackets => false } } }
Summary
- The error is caused by an invalid regex in
field_split => "] [". - Escape square brackets inside regex strings as
\]and\[. - Alternatively, use
field_split_patternwith regex. - This fix resolves the
empty char-classregex error in the kv filter and allows Logstash to start.
If after this fix you get any more issues, feel free to ask!