How to fork and merge a REST API call that takes a list of id's as query param?

Question

I have to call a third party GET API which takes a comma seperated list of ids and returns a JSON response. Like so:

def call(sysid)
   # RestClient GET  https://test.com?ids=sysid.join(',')&....
   # parse response and returns
   # rescue - any error handling
end

RestClient

But sysid is an input and can sometimes have 300+ elements and due to the limitation of GET, my api call errors out (probably because URL length becomes too big)

How can I efficiently send API calls (in small batches) & merge results back. In doing that: -

• should I use any Parallelism
  • if one of the batch call fails, may be add retries
    • if it still fail, stop the whole process & return an error from call()

Is there a good pattern in Ruby for this type of problem?

Answer 1

I am not aware of good ruby pattern but we can solve issue by below code

def call(sysid)
  fetch_next = true
  response = {status: true, result: []}
  while fetch_next
    param_value_id = sysid[0..n-1]
    while sysid.present?
      begin
        call API with FARADAY/RestClient with its params as param_value_id.join(',')
        if success
          parse response and append in responses result
        else
          fetch_next = false
          response[:status] = false
          response[:result] = []
        end
      rescue Exception => e
        Rails.logger.warn "Exception raised due to #{e}"
        fetch_next = false
        response[:status] = false
        response[:result] = []
      end
    end
    sysid = sysid.drop(n)
    if sysid.empty?
      fetch_next = false
    end
  end
  response
end

EXPLANATION

1. setting fetch_next to true and response hash to be returned from call method.
2. till fetch_next is true, we'll fetch n ids from sysid
3. till sysid is there, we'll call third party api with intended params
4. if response is success, we append it in result key of response hash
5. if there is failure or an exception, we'll set fetch_next to false, status key of response to false and empty result key
6. In the end we drop n values from sysid array and will set fetch_next to false if sysid is empty.

you can return response object or send particular response if status is false

This is pseudo code but i think it might solve your problem.

Thank you.