Write filebeat config for modsecurity log at /var/log/modsecurity/audit.log in JSON…
- User
Write filebeat config for modsecurity log at /var/log/modsecurity/audit.log in JSON format:
There's an example line from logfile: {"transaction":{"client_ip":"192.168.250.6","time_stamp":"Thu May 22 20:13:53 2025","server_id":"022d5b3351467cabff297d0ddb81cc6eed577194","client_port":47146,"host_ip":"10.100.2.2","host_port":80,"unique_id":"17479628333.483484","request":{"method":"GET","http_version":1.1,"uri":"/DVWA/vulnerabilities/sqli_blind/?Submit=Submit&id=%27case+when+cast%28pg_sleep%2815.0%29+as+varchar%29+%3E+%27%27+then+0+else+1+end+--+","headers":{"host":"192.168.250.5","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36","Cookie":"security=low; PHPSESSID=6adtn84vbe041otbj6o5r5l4qd","pragma":"no-cache","cache-control":"no-cache","referer":"http://192.168.250.5/DVWA/vulnerabilities/sqli_blind/"}},"response":{"body":"\r\n403 Forbidden\r\n\r\n403 Forbidden\r\nnginx\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n","http_code":403,"headers":{"Server":"nginx","Date":"Fri, 23 May 2025 01:13:53 GMT","Content-Length":"548","Content-Type":"text/html","X-Content-Type-Options":"nosniff","Connection":"keep-alive","X-Frame-Options":"DENY","Content-Security-Policy":"default-src 'self';"}},"producer":{"modsecurity":"ModSecurity v3.0.9 (Linux)","connector":"ModSecurity-nginx v1.0.3","secrules_engine":"Enabled","components":["OWASP_CRS/4.15.0-dev""]},"messages":[{"message":"SQL Injection Attack Detected via libinjection","details":{"match":"detected SQLi using libinjection.","reference":"v55,69","ruleId":"942100","file":"/etc/nginx/modsec/owasp-crs/rules/REQUEST-942-APPLICATION-ATTACK-SQLI.conf","lineNumber":"46","data":"Matched Data: sEkf( found within ARGS:id: 'case when cast(pg_sleep(15.0) as varchar) > '' then 0 else 1 end -- ","severity":"2","ver":"OWASP_CRS/4.15.0-dev","rev":"","tags":[],"maturity":"0","accuracy":"0"}},{"message":"appinfo leak","details":{"match":"Matched "Operator
Rx' with parameter(?i)(mysql|password|passwd|can't connect|error|exception|trace|fatal|undefined|denied|unauthorized)' against variableRESPONSE_BODY' (Value:\x0d\x0a403 Forbidden\x0d\x0a\x0d\x0a403 Forbidd (526 characters omitted)' )","reference":"o197,5o197,5v458,548","ruleId":"180003","file":"/etc/nginx/modsec/custom-rules/application-data-leaks.conf","lineNumber":"8","data":"","severity":"0","ver":"","rev":"","tags":["application-info-leak"],"maturity":"0","accuracy":"0"}}]}} - Assistant
Here is a Filebeat configuration snippet to parse ModSecurity JSON audit logs located at
/var/log/modsecurity/audit.log. This config reads the file, decodes JSON lines, and parses the nested fields properly.filebeat.inputs: - type: log enabled: true paths: - /var/log/modsecurity/audit.log json.keys_under_root: true json.add_error_key: true json.message_key: "" # empty because JSON root is an object, not just a string field json.overwrite_keys: true # You can add multiline if your JSON logs are multiline (usually not necessary for one JSON object per line) # Optional processors to rename fields or add ECS-compatible fields processors: - rename: fields: - from: "transaction.client_ip" to: "source.ip" - from: "transaction.client_port" to: "source.port" - from: "transaction.host_ip" to: "destination.ip" - from: "transaction.host_port" to: "destination.port" - from: "transaction.time_stamp" to: "@timestamp" ignore_missing: true fail_on_error: false - convert: fields: - {field: "@timestamp", type: "date", target_field: "@timestamp", formats: ["ddd MMM dd HH:mm:ss yyyy", "RFC3339", "ISO8601"]} ignore_missing: true fail_on_error: false # Output example, adjust to your Elasticsearch or Logstash setup: output.elasticsearch: hosts: ["http://localhost:9200"] index: "modsecurity-%{+yyyy.MM.dd}"Explanation:
json.keys_under_root: true: places all keys from JSON root at the top level in the event.json.add_error_key: true: adds a field if JSON decoding fails.- The example ModSecurity log is one JSON object per line, so no special multiline config is needed.
- The transaction’s
time_stampis parsed as the@timestampwith a custom date format matching the example"Thu May 22 20:13:53 2025". - Renamed IP and port fields to ECS-standard fields for better integration with Elastic stack.
If ModSecurity logs are multiline JSON objects, you'll need a multiline configuration, but based on your example, this should work.
If you want a full standalone
filebeat.ymlconfig (minimal) including the essential parts:filebeat.inputs: - type: log enabled: true paths: - /var/log/modsecurity/audit.log json.keys_under_root: true json.add_error_key: true processors: - rename: fields: - from: "transaction.client_ip" to: "source.ip" - from: "transaction.client_port" to: "source.port" - from: "transaction.host_ip" to: "destination.ip" - from: "transaction.host_port" to: "destination.port" - from: "transaction.time_stamp" to: "@timestamp" ignore_missing: true fail_on_error: false - convert: fields: - {field: "@timestamp", type: "date", target_field: "@timestamp", formats: ["E MMM dd HH:mm:ss yyyy", "RFC3339", "ISO8601"]} ignore_missing: true fail_on_error: false output.elasticsearch: hosts: ["http://localhost:9200"] index: "modsecurity-%{+yyyy.MM.dd}"Adjust output as needed for your environment.
- User
make output to logstash
- Assistant
Certainly! Here is the Filebeat config updated to send output to Logstash instead of Elasticsearch:
filebeat.inputs: - type: log enabled: true paths: - /var/log/modsecurity/audit.log json.keys_under_root: true json.add_error_key: true processors: - rename: fields: - from: "transaction.client_ip" to: "source.ip" - from: "transaction.client_port" to: "source.port" - from: "transaction.host_ip" to: "destination.ip" - from: "transaction.host_port" to: "destination.port" - from: "transaction.time_stamp" to: "@timestamp" ignore_missing: true fail_on_error: false - convert: fields: - {field: "@timestamp", type: "date", target_field: "@timestamp", formats: ["E MMM dd HH:mm:ss yyyy", "RFC3339", "ISO8601"]} ignore_missing: true fail_on_error: false output.logstash: # Replace with your Logstash server address and port hosts: ["logstash-server:5044"]Notes:
- Replace
"logstash-server:5044"with your actual Logstash host and port. - The processors remain the same to keep field names ECS-compliant and parse the timestamp.
- This config assumes Logstash is configured to receive Beats input on the specified port.
- Replace
- User
Now write logstash config to collect these logs
- Assistant
Certainly! Below is a sample Logstash configuration to receive the ModSecurity JSON logs from Filebeat, parse the
@timestampfield properly, and keep all JSON fields intact.input { beats { port => 5044 } } filter { # Parse the @timestamp field coming from Filebeat (string) into Logstash @timestamp (timestamp) if [@timestamp] { date { match => ["@timestamp", "EEE MMM dd HH:mm:ss yyyy", "ISO8601", "RFC3339"] locale => "en" timezone => "UTC" target => "@timestamp" # If parsing fails, keep original but tag the event add_tag => [ "_timestamp_parsed" ] # Uncomment this if you want to remove original in case of success # remove_field => ["@timestamp"] } } } output { elasticsearch { hosts => ["http://localhost:9200"] index => "modsecurity-%{+YYYY.MM.dd}" } stdout { codec => rubydebug } # For debugging on console; remove in production }
Explanation:
input.beatslistens on port 5044 for Filebeat (default Beats input port).- The filter uses the
dateplugin to parse the@timestampfield created by Filebeat from the ModSecurity JSON'stransaction.time_stamp. - The
matchdate formats cover your example format"Thu May 22 20:13:53 2025"as well as standard ISO and RFC formats. locale => "en"ensures month and day names are parsed correctly.- Output sends to Elasticsearch with an index pattern
modsecurity-YYYY.MM.DD. stdout { codec => rubydebug }is for debugging and can be removed in production.
If you want to further process fields (like flatten arrays, convert types, or enrich) you can add more filters — but for your JSON logs this should be a solid starting point.
- User
where i should paste this logstash configuration?
- Assistant
Great question! Here's where and how to use the Logstash configuration:
Where to place the Logstash configuration file:
-
Default config directory:
By default, Logstash loads pipeline configurations from itsconfigdirectory, usually located at:/etc/logstash/conf.d/(Linux packages)<logstash_install_dir>/config(when running from extracted archive or Windows)
-
Create a new config file:
Save your Logstash pipeline config in a file with a.confextension inside this directory. For example:/etc/logstash/conf.d/modsecurity.conf -
File content:
Paste the entire Logstash configuration you have (includinginput,filter, andoutputblocks) into that file.
After placing the config file:
-
Verify config syntax:
(Optional but recommended)
Run the following to check your Logstash configuration for errors:sudo /usr/share/logstash/bin/logstash --config.test_and_exit -f /etc/logstash/conf.d/modsecurity.confAdjust path if Logstash installed elsewhere.
-
Restart Logstash:
To apply the new configuration, restart the Logstash service:sudo systemctl restart logstashor (depending on your OS / setup):
sudo service logstash restart -
Check logs:
Monitor Logstash logs for errors or confirmation that it started successfully:sudo journalctl -u logstash -f(or check
/var/log/logstash/logstash-plain.log)
Alternative: Running Logstash manually
If you are testing or developing and want to run Logstash straight from shell without installing as a service:
/path/to/logstash/bin/logstash -f /path/to/your/modsecurity.confThis will run Logstash with your config file until you stop it (Ctrl+C).
Summary:
Step Action 1. Config location /etc/logstash/conf.d/modsecurity.conf2. Paste config Put Logstash config ( input,filter,output) inside the file3. Test syntax logstash --config.test_and_exit -f /etc/logstash/conf.d/modsecurity.conf4. Restart Logstash systemctl restart logstash5. Check logs journalctl -u logstash -f
Feel free to ask if you want help for a specific OS or a tailored Logstash startup!
-
- User
May 22 20:44:59 debian logstash[1169]: [2025-05-22T20:44:59,755][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: Illegal pattern component: R", :backtrace=>["org.logstash.config.ir.CompiledPipeline.(CompiledPipeline.java:137)", "org.logstash.execution.AbstractPipelineExt.initialize(AbstractPipelineExt.java:240)", "org.logstash.execution.AbstractPipelineExtiINVOKERinitialize.gen)", "org.jruby.internal.runtime.methods.JavaMethodINVOKERnewInstance.call(RubyClassi$newInstance.gen)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:446)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:92)", "org.jruby.ir.instructions.CallBase.interpret(CallBase.java:548)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:363)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:88)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:238)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:225)", "org.jruby.internal.runtime.m
- Assistant
The error message:
Unable to configure plugins: Illegal pattern component: Rindicates there is a problem with the
datefilter'smatchpattern in your Logstash configuration.Root cause
The Java date/time parser used by Logstash does not recognize the
Rpattern symbol used in"RFC3339"or"ISO8601"in thedatefiltermatch.In Logstash’s
datefilter plugin, valid patterns are based on Java's SimpleDateFormat, which does not supportRFC3339orISO8601as format strings.
How to fix it
1. Adjust your
datefilter'smatchclause to use only valid Java SimpleDateFormat date patterns.Your timestamp in example logs is:
Thu May 22 20:13:53 2025Which corresponds to Java SimpleDateFormat:
EEE MMM dd HH:mm:ss yyyyEEE= Day name short (Thu)MMM= Month name short (May)dd= Day of monthHH:mm:ss= 24-hour timeyyyy= 4-digit year
If your
@timestampfield might already be an ISO8601/RFC3339 timestamp (produced by Filebeat or elsewhere), you can handle that by parsing it separately or safely ignoring it.
Minimal working example for your filter block:
filter { if [@timestamp] { date { match => ["@timestamp", "EEE MMM dd HH:mm:ss yyyy"] locale => "en" timezone => "UTC" target => "@timestamp" # Uncomment if you want to tag events where parsing succeeded add_tag => [ "_timestamp_parsed" ] } } }Remove
"ISO8601"and"RFC3339"from thematcharray since those are not valid pattern strings here.
Explanation:
- The
datefilter will parse your ModSecurity timestamp format only. - If you want to parse other timestamp formats, add their corresponding valid Java date patterns, not strings like
"RFC3339".
Full working example Logstash config snippet (filter section):
filter { if [@timestamp] { date { match => ["@timestamp", "EEE MMM dd HH:mm:ss yyyy"] locale => "en" timezone => "UTC" target => "@timestamp" add_tag => ["_timestamp_parsed"] } } }
Summary of action to fix your pipeline:
- Open your Logstash
.conffile (e.g.,/etc/logstash/conf.d/modsecurity.conf). - Edit the
datefilter'smatchline to only use"EEE MMM dd HH:mm:ss yyyy". - Remove
"RFC3339"and"ISO8601"from the list since those are invalid format strings here. - Save and restart Logstash:
sudo systemctl restart logstash - Check logs again for errors:
sudo journalctl -u logstash -f
If your Filebeat sends
@timestampin ISO8601 format (which it usually does, e.g.,2025-05-22T20:13:53.000Z), you might choose to skip thedatefilter altogether, or add a conditional to only parse when the timestamp is in your custom format.
If you want me to help you build that conditional or a more robust pipeline, just ask!
- User
set elasticsearch credentials in logstash
- Assistant
To configure Elasticsearch credentials in your Logstash output, you add the
userandpasswordfields under theelasticsearchoutput plugin.Here’s a complete example snippet for your Logstash output section with authentication:
output { elasticsearch { hosts => ["http://localhost:9200"] # Adjust to your Elasticsearch host(s) index => "modsecurity-%{+YYYY.MM.dd}" user => "elastic" # Your Elasticsearch username password => "your_elasticsearch_password" # Your Elasticsearch password ssl => false # Set to true if using https and configure accordingly # If using https with self-signed certificate, you may need to add ssl_certificate_verification => false } stdout { codec => rubydebug } }Notes:
- Replace
"elastic"and"your_elasticsearch_password"with the actual username and password. - If Elasticsearch runs with HTTPS, change the
hostsURL to"https://your-es-host:9200"and setssl => true. - If your Elasticsearch cluster uses API keys or other authentication, Logstash also supports those methods but the basic user/password method is most common.
Example with HTTPS and skipping cert verification (not recommended for production):
output { elasticsearch { hosts => ["https://es-host:9200"] user => "elastic" password => "your_password" ssl => true ssl_certificate_verification => false } stdout { codec => rubydebug } }
After editing your Logstash config:
- Save the file
- Restart Logstash
- Verify it successfully connects to Elasticsearch with provided creds
Let me know if you want me to help with secured connection setup or other auth methods!
- Replace