elasticsearch - logstash fails to create an index in ES -

i trying parse log file using logstash.. filebeat-reading sample logs directory , indexed elasticsearch through logstash. ( reading input file directory through filebeat , specifying read logstash output in filebeat.yml, , parsing log file in logstash configuration file, , putting result in index in es. )

filebeat.yml

#=========================== filebeat prospectors =============================  filebeat.prospectors:    #input_type: log   #input_type: log   document_type: my_log paths:   - c:\logsa\elast.log      #----------------------------- logstash output --------------------------------     output.logstash:       # logstash hosts       hosts: ["localhost:5044"]    elast.log : (i trying parse 1 line of log in log file)       [2016-11-03 07:30:05,987] [info] [o.e.p.pluginsservice     ] [htykfft] initializing... 

logstash configuration file :

input { beats { port => "5044" } } filter { if [type] == "my_log" { grok { match => { "message" => "\[%{timestamp_iso8601:timestamp}\] \[%{data:loglevel}\] \[%{data:message}\] \[%{greedydata:message}\] %{greedydata:message1}"} } } } output { elasticsearch { hosts => ["localhost:9200"] } } 

i running filebeat.exe, logstash conf file, , elasticsearch.

i not getting errors such when running logstash configuration file...

console when running logstash conf:

c:\logstash-5.0.0\logstash-5.0.0\bin>logstash -f log-h.conf java_opts set [ -xmx1g -xx:+useparnewgc -xx:+useconcmarksweepgc -xx:+cmsp arallelremarkenabled -xx:survivorratio=8 -xx:maxtenuringthreshold=1 -xx:cmsiniti atingoccupancyfraction=75 -xx:+usecmsinitiatingoccupancyonly -xx:+heapdumponouto fmemoryerror -xx:heapdumppath="$ls_home/heapdump.hprof"]. logstash trust th ese options, , not set defaults might set sending logstash logs c:/logstash-5.0.0/logstash-5.0.0 /logs configured via log4j2.properties. [2016-11-08t17:38:02,452][info ][logstash.inputs.beats    ] beats inputs: starti ng input listener {:address=>"0.0.0.0:5044"} [2016-11-08t17:38:02,728][info ][org.logstash.beats.server] starting server on p ort: 5044 [2016-11-08t17:38:03,082][info ][logstash.outputs.elasticsearch] elasticsearch p ool urls updated {:changes=>{:removed=>[], :added=>["http://localhost:9200"]}} [2016-11-08t17:38:03,089][info ][logstash.outputs.elasticsearch] using mapping t emplate {:path=>nil} [2016-11-08t17:38:03,324][info ][logstash.outputs.elasticsearch] attempting nstall template {:manage_template=>{"template"=>"logstash-*", "version"=>50001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"_all"= >{"enabled"=>true, "norms"=>false}, "dynamic_templates"=>[{"message_field"=>{"pa th_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text" , "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"str ing", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"= >"keyword"}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date", "include_in_all" =>false}, "@version"=>{"type"=>"keyword", "include_in_all"=>false}, "geoip"=>{"d ynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_po int"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}} }}}}} [2016-11-08t17:38:03,359][info ][logstash.outputs.elasticsearch] new elasticsear ch output {:class=>"logstash::outputs::elasticsearch", :hosts=>["localhost:9200" ]} [2016-11-08t17:38:03,596][info ][logstash.pipeline        ] starting pipeline {" id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch. delay"=>5, "pipeline.max_inflight"=>500} [2016-11-08t17:38:03,612][info ][logstash.pipeline        ] pipeline main starte d [2016-11-08t17:38:03,783][info ][logstash.agent           ] started  logstash api endpoint {:port=>9600} 

it not creating index in es, not getting errors such seen in console above well.

can here? in advance.

there indentation issues filebeat configuration. should filebeat 5.x.

filebeat.prospectors: - paths:     - c:/logsa/elast.log   document_type: my_log  output.logstash:   hosts: ["localhost:5044"] 

there logstash configuration example provided in beats documenation shows how configure elasticsearch output. write data filebeat-yyyy.mm.dd index.

input {   beats {     port => "5044"   }    }     filter {   if [type] == "my_log" {     grok {       match => { "message" => "\[%{timestamp_iso8601:timestamp}\] \[%{data:loglevel}\] \[%{data:message}\] \[%{greedydata:message}\] %{greedydata:message1}"}     }      }    }     output {   elasticsearch {     hosts => "localhost:9200"     manage_template => false     index => "%{[@metadata][beat]}-%{+yyyy.mm.dd}"     document_type => "%{[@metadata][type]}"   }    }    

when using logstash must manually install filebeat index template elasticsearch.

for windows:

ps c:\program files\filebeat> invoke-webrequest -method put -infile filebeat.template.json -uri http://localhost:9200/_template/filebeat?pretty

for unix:

curl -xput 'http://localhost:9200/_template/filebeat' -d@/etc/filebeat/filebeat.template.json