Sending CSV file on HTTP

Question

I want to send this CSV data to SOLR.

id,author,name,year,category
001-001,H. G. Wells,The Time Machine,1895,Fiction
001-002,H. G. Wells,The Wonderful Visit,1895,Fiction

Here is my code which doesn't work.

  private static final String postURL = "http://localhost:8983/solr/update/csv?separator=,&commit=true";

  public static void main(String[] args) throws IOException {
    BookCsvIndexer poster = new BookCsvIndexer();
    BufferedReader dbr = new BufferedReader(new FileReader("E:/solr-example/csv/books.csv"));
    String line;
    StringBuilder dsb = new StringBuilder();
    while ((line = dbr.readLine()) != null) {
      dsb.append(line);
    }
    dbr.close();
    String data = dsb.toString();
    System.out.println(data);
    poster.doPost(data);
  }

  public void doPost(String data) {
    String contentLength = Integer.toString(data.getBytes().length);
    System.out.println("Length=" + contentLength);
    StringBuilder sb = new StringBuilder();
    try {
      URL url = new URL(postURL);
      HttpURLConnection connection = (HttpURLConnection) url.openConnection();
      connection.setDoOutput(true);
      connection.setDoInput(true);
      connection.setRequestMethod("POST");
      connection.setRequestProperty("Accept", "text/plain");
      connection.setRequestProperty("Connection", "close");
      connection.setRequestProperty("Content-Type", "application/csv; charset=utf-8");
      connection.setRequestProperty("Content-Length", "" + contentLength);
      connection.connect();

      if (data.length() > 0) {
        OutputStreamWriter wr = new OutputStreamWriter(connection.getOutputStream());
        wr.write(data);
        wr.flush();
        wr.close();
      }

      BufferedReader br = new BufferedReader(new InputStreamReader((connection.getInputStream())));
      String output;
      System.out.println("Server Respose CODE=" + connection.getResponseCode());
      while ((output = br.readLine()) != null) {
        sb.append(output);
      }
    } catch (MalformedURLException e) {
      e.printStackTrace();
    } catch (IOException e) {
      e.printStackTrace();
    }
    System.out.println(sb.toString());
  }

I can use this same code for sending JSON and other text files successfully by simply changing the 'Content-Type'. I have been using this same code for hitting many REST services. I can't understand why this code is not working for sending CSV to SOLR.

Here is the SOLR log after running the above code.

INFO  - 2013-11-21 11:20:09.090; org.apache.solr.update.DirectUpdateHandler2; start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
INFO  - 2013-11-21 11:20:09.093; org.apache.solr.core.SolrDeletionPolicy; SolrDeletionPolicy.onInit: commits: num=1
    commit{dir=NRTCachingDirectory(org.apache.lucene.store.SimpleFSDirectory@E:\solr-example\solr\collection1\data\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@15b46ef; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
INFO  - 2013-11-21 11:20:09.093; org.apache.solr.core.SolrDeletionPolicy; newest commit generation = 1
INFO  - 2013-11-21 11:20:09.094; org.apache.solr.update.DirectUpdateHandler2; No uncommitted changes. Skipping IW.commit.
INFO  - 2013-11-21 11:20:09.094; org.apache.solr.search.SolrIndexSearcher; Opening Searcher@ae20f5 main
INFO  - 2013-11-21 11:20:09.095; org.apache.solr.update.DirectUpdateHandler2; end_commit_flush
INFO  - 2013-11-21 11:20:09.095; org.apache.solr.core.QuerySenderListener; QuerySenderListener sending requests to Searcher@ae20f5 main{StandardDirectoryReader(segments_1:1:nrt)}
INFO  - 2013-11-21 11:20:09.096; org.apache.solr.core.QuerySenderListener; QuerySenderListener done.
INFO  - 2013-11-21 11:20:09.096; org.apache.solr.handler.component.SpellCheckComponent$SpellCheckerListener; Building spell index for spell checker: suggest
INFO  - 2013-11-21 11:20:09.096; org.apache.solr.spelling.suggest.Suggester; build()
INFO  - 2013-11-21 11:20:09.099; org.apache.solr.core.SolrCore; [collection1] Registered new searcher Searcher@ae20f5 main{StandardDirectoryReader(segments_1:1:nrt)}
INFO  - 2013-11-21 11:20:09.100; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/solr path=/update/csv params={commit=true&separator=,} {commit=} 0 17

And the response from the last line System.out.println(sb.toString()); is

<?xml version="1.0" encoding="UTF-8"?>
<response>
    <lst name="responseHeader">
    <int name="status">0</int>
    <int name="QTime">17</int></lst>
</response>

The URL and CSV file is correct because I can POST this CSV data successfully using REST client software like POSTMAN, a Chrome extension for REST client. Now, I want to do the same using Java.

Am I missing something?

Answer 1

you can use apache commons HttpClient to post a file to server.

 File file = new File("file-path"); 
 MultipartPostMethod method = new MultipartPostMethod(post-file-url);
 method.addParameter("fileName", file.getName());
 method.addPart(new FilePart("file", "file", file));
 httpClient.executeMethod(method);
 String response = method.getResponseBodyAsString();

HttpClient dependency :

<dependency>
    <groupId>commons-httpclient</groupId>
    <artifactId>commons-httpclient</artifactId>
    <version>3.1</version>
</dependency>