I'm trying to fetch index data from Elastic search using Logstash but i'm getting few errors ""

Question

Using same conf file i'm able to copy the of index data using local elastic but using production elastic i'm getting this error

My conf file is :

`

input { elasticsearch { hosts => ["https://lion_jarvis_client:7CaVEQTFES0wiPmNZT3H@project-prod-api.elastic.services.aws.kpn.org:443"]

    input {
elasticsearch {
    hosts => ["https://lion_jarvis_client:7CaVEQTFES0wiPmNZT3H@project-prod-api.elastic.services.aws.kpn.org:443"]
                        
    index => "lion-jarvis-powerbi-transform-demo"
    ssl => true
    ca_file => "C:\Testing\KPNRootCertificate.crt"
    
    query => '{
                  "query": {
                      "bool": {
                        "must": [],
                        "filter": [
                          {
                            "match_all": {}
                          }
                        
                        ]
                      }
                    }
                  }'
    size => 500
    scroll => "5m"
    docinfo => true
   
    
   
    
   
}
}
output {
 

csv {
  fields => ["ord_no","ord_state","po_no","cust_ord_no","cust_code","service_no"]
  
    path => "\\kpnnl.local\Private\Homedirs2\Homedirs02E\kunch500\logstash1\sample.csv"
  }
}

`

I'm getting below error:

I search for it but i didn't found any related information for this issue

`Sending Logstash logs to C:/Users/kunch500/Downloads/elk stack/logstash-7.13.2/logs which is now configured via log4j2.properties [2022-03-29T20:35:49,349][INFO ][logstash.runner ] Log4j configuration path used is: C:\Users\kunch500\Downloads\elk stack\logstash-7.13.2\config\log4j2.properties [2022-03-29T20:35:49,396][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.13.2", "jruby.version"=>"jruby 9.2.16.0 (2.5.7) 2021-03-03 f82228dc32 OpenJDK 64-Bit Server VM 11.0.11+9 on 11.0.11+9 +indy +jit [mswin32-x86_64]"} [2022-03-29T20:35:49,580][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified [2022-03-29T20:35:51,839][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600} [2022-03-29T20:35:52,942][INFO ][org.reflections.Reflections] Reflections took 156 ms to scan 1 urls, producing 24 keys and 48 values [2022-03-29T20:35:55,655][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>10, "pipeline.batch.size"=>2000, "pipeline.batch.delay"=>20, "pipeline.max_inflight"=>20000, "pipeline.sources"=>["//kpnnl.local/Private/Homedirs2/Homedirs02E/kunch500/logstash1/dblogstash.conf"], :thread=>"#<Thread:0x7948c3b run>"} [2022-03-29T20:35:55,696][WARN ][logstash.javapipeline ][main] CAUTION: Recommended inflight events max exceeded! Logstash will run with up to 20000 events in memory in your current configuration. If your message sizes are large this may cause instability with the default heap size. Please consider setting a non-standard heap size, changing the batch size (currently 2000), or changing the number of pipeline workers (currently 10) {:pipeline_id=>"main", :thread=>"#<Thread:0x7948c3b run>"} [2022-03-29T20:35:57,907][INFO ][logstash.javapipeline ][main] Pipeline Java execution initialization time {"seconds"=>2.17} [2022-03-29T20:36:00,935][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"} [2022-03-29T20:36:01,150][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]} [2022-03-29T20:36:04,135][ERROR][logstash.javapipeline ][main][ec8e41a26d7fa10a9b245713304a8e1a14520d9d1642debd75d9b23af485400f] A plugin had an unrecoverable error. Will restart this plugin. Pipeline_id:main Plugin: <LogStash::Inputs::Elasticsearch ca_file=>"C:\Testing\KPNRootCertificate.crt", size=>500, hosts=>["https://lion_jarvis_client:7CaVEWUsjq0wiPmNZT3H@project-prod-api.elastic.services.aws.kpn.org:443"], query=>"{\r\n "query": {\r\n "bool": {\r\n "must": [],\r\n "filter": [\r\n {\r\n "match_all": {}\r\n }\r\n \r\n ]\r\n }\r\n }\r\n }", scroll=>"5m", index=>"lion-jarvis-powerbi-transform-demo", docinfo=>true, id=>"ec8e41a26d7fa10a9b245713304a8e1a14520d9d1642debd75d9b23af485400f", ssl=>true, enable_metric=>true, codec=><LogStash::Codecs::JSON id=>"json_d6ea0194-5374-4d8c-9720-29ab31ea5942", enable_metric=>true, charset=>"UTF-8">, docinfo_target=>"@metadata", docinfo_fields=>["_index", "_type", "_id"], connect_timeout_seconds=>10, request_timeout_seconds=>60, socket_timeout_seconds=>60>

Error: No such host is known (https) Exception: Manticore::ResolutionFailure

Stack: C:/Users/kunch500/Downloads/elk stack/logstash-7.13.2/vendor/bundle/jruby/2.5.0/gems/manticore-0.7.0-java/lib/manticore/response.rb:37:in block in initialize' C:/Users/kunch500/Downloads/elk stack/logstash-7.13.2/vendor/bundle/jruby/2.5.0/gems/manticore-0.7.0-java/lib/manticore/response.rb:79:in call' C:/Users/kunch500/Downloads/elk stack/logstash-7.13.2/vendor/bundle/jruby/2.5.0/gems/manticore-0.7.0-java/lib/manticore/response.rb:274:in call_once' C:/Users/kunch500/Downloads/elk stack/logstash-7.13.2/vendor/bundle/jruby/2.5.0/gems/manticore-0.7.0-java/lib/manticore/response.rb:158:in code' C:/Users/kunch500/Downloads/elk stack/logstash-7.13.2/vendor/bundle/jruby/2.5.0/gems/elasticsearch-transport-5.0.5/lib/elasticsearch/transport/transport/http/manticore.rb:84:in block in perform_request' C:/Users/kunch500/Downloads/elk stack/logstash-7.13.2/vendor/bundle/jruby/2.5.0/gems/elasticsearch-transport-5.0.5/lib/elasticsearch/transport/transport/base.rb:262:in perform_request' C:/Users/kunch500/Downloads/elk stack/logstash-7.13.2/vendor/bundle/jruby/2.5.0/gems/elasticsearch-transport-5.0.5/lib/elasticsearch/transport/transport/http/manticore.rb:67:in perform_request' C:/Users/kunch500/Downloads/elk stack/logstash-7.13.2/vendor/bundle/jruby/2.5.0/gems/elasticsearch-transport-5.0.5/lib/elasticsearch/transport/client.rb:131:in perform_request' C:/Users/kunch500/Downloads/elk stack/logstash-7.13.2/vendor/bundle/jruby/2.5.0/gems/elasticsearch-api-5.0.5/lib/elasticsearch/api/actions/search.rb:183:in search' C:/Users/kunch500/Downloads/elk stack/logstash-7.13.2/vendor/bundle/jruby/2.5.0/gems/logstash-input-elasticsearch-4.9.1/lib/logstash/inputs/elasticsearch.rb:349:in search_request' C:/Users/kunch500/Downloads/elk stack/logstash-7.13.2/vendor/bundle/jruby/2.5.0/gems/logstash-input-elasticsearch-4.9.1/lib/logstash/inputs/elasticsearch.rb:272:in do_run_slice' C:/Users/kunch500/Downloads/elk stack/logstash-7.13.2/vendor/bundle/jruby/2.5.0/gems/logstash-input-elasticsearch-4.9.1/lib/logstash/inputs/elasticsearch.rb:250:in do_run' C:/Users/kunch500/Downloads/elk stack/logstash-7.13.2/vendor/bundle/jruby/2.5.0/gems/logstash-input-elasticsearch-4.9.1/lib/logstash/inputs/elasticsearch.rb:238:in run' C:/Users/kunch500/Downloads/elk stack/logstash-7.13.2/logstash-core/lib/logstash/java_pipeline.rb:405:in inputworker' C:/Users/kunch500/Downloads/elk stack/logstash-7.13.2/logstash-core/lib/logstash/java_pipeline.rb:396:in block in start_input'

Thanks

Answer 1

The hosts option of an elasticsearch input accepts host:ip, not a URI. Remove the https:// and the user/password. Set the user/password using the user and password options on the input.