An Opinionated view on delivering Large scale payloads through API call
Streaming: The real-time approach to transfer a huge payload ( in millions ) through the API calls as efficiently as possible
Recently, I stumbled upon a use case to deliver a large-scale payload ( millions of records) as part of the REST API call. Agreeing with the fact that it was not an ideal use case for blocking synchronous API calls and could have been better handled in an asynchronous fashion ( non-blocking ) using some middleware solutions like Apache Kafka or RabbitMQ, but, it led me to investigate all the viable potential solution approaches for the requirement in hand.
I was using the following technology stack
- Spring boot v2.0 as the application framework
- Apache Solr v8.6.1 as a backend data source
Through this article, I have tried to give a brief overview of all the solution approaches I investigated, scenarios where I found them to be useful, and why I had to reject them except for the one which proved to be the required solution for my use case. Not denying the fact that there can be other better solution approaches that I might have missed out on, so I request all the solution seekers to evaluate /consider them along with the ones which I have mentioned below.
Having said that let's look at each of the solution approaches which I have evaluated for my use case
- Pagination: This approach mainly aims at breaking the entire data set into smaller chunks knows as pages. The consumer is provided with the information on page count, current page data, page size, and capability to paginate backward and forward. Though this approach helps to navigate through large-scale payloads but results in making multiple server calls which provides a frustrating customer experience. Moreover, it is likely suitable for UI-centric applications with a limited set of payloads. As mine was a non UI customer expecting the entire data set, this solution approach didn’t work.
- HATEOAS: Also know as Hypermedia As The Engine Of Application State. The beauty of HATEOAS is that the requested resource response contains both data, and actions related to it. It usually augments and extends pagination by using common hypermedia formats for messaging. It allows to intelligently break down larger responses into different logical collections. Below is the sample JSON which shows how the HETEOAS response looks like
/////////////////////////illustrative depiction /////////////////{
"Employee": {
"Empid": 1,
"FirstName": "Ravi",
"SecondName": "Shankar",
"DOB": "28/01/1983",
"Designation": "Software Engineer"
},
"_links": {
"Projects": {
"href": "http://localhost:8090/projects"
},
"Hobbies": {
"href": "http://localhost:8090/hobbies"
},
"Blogs": {
"href": "http://localhost:8090/blogs"
}
}
}
This approach is suitable for use cases where you need to get the top-level data to start with and as and when needed navigate to all other methods on the same resource or related resources and sub-resources. Some use cases where the HATEAOS approach is really helpful are as follows
- Navigating a collection
- Error Resolution
- Service-Controlled Flow
- Asynchronous operations
- Saving bandwidth
More details on the above use cases can be found in this link
For my use case to send the data in the above format would have involved a lot of extra processing with increased latency and also would have left the client making multiple API calls to fetch the data for each of the embedded links. For the same reason, I had to rule out this solution approach.
3. Through Compression: This technique can be used to reduce the surface area of the API response through the compression techniques like GZIP and DEFLATE. The most common use case where it is commonly used is the client-side download functionality in different formats like PDF, XLS, CSV, and Tab-delimited. This approach can be used only when the client has the ability to un-compress the content.
This is a good technique if the right level of compression is used taking into account the performance for zipping and unzipping. This can be provided as an option in the API (request parameter) for those clients who can handle unzipping. Since it was resulting in more memory consumption and resulted in OOME issues for the payload size I had, so ruled out this solution also.
4. Streaming response: Finally I got the solution I was looking for “The non-blocking asynchronous solution approach” in the form of HTTP Streaming.
By definition, HTTP streaming is “ A push-style data transfer technique that allows a backend server to continuously send data to a client over a single HTTP connection that remains open indefinitely.”
The major advantage of HTTP streaming is that as soon as the first bit is read from the backend server it can be immediately pushed to the client. The entire resultset doesn’t have to be read /stored into the JVM memory before transferring it to the client, thereby avoiding the most notorious OOME issues.
In this approach, the backend server will not close the connection as soon as the response is sent to the client. It is configured to hold on to a specific request and as soon as the updates are available, the response is immediately pushed to the client and the server closes the connection only when it is explicitly asked to do so. To achieve this indefinite response the server will send the following response header which sets up a persistent connection with the client
Transfer Encoding: chunked
Alternatively, the data may be streamed via the Server-Sent Events (SSE) method, for which there is a standardized HTML5 API called EventSource. With SSE, data is encoded as text/event-stream in the header.
The only drawback with this approach is that it is suitable only for clients who can handle streaming.
For spring boot to provide streaming and nonblocking calls I leverage the StreamingResponseBody feature which provides asynchronous request processing and also the capability to write the data directly to the response OutputStream without blocking the servlet container thread. In this approach, the data is processed and written in chunks to the outputstream and also it is highly recommended to configure Spring MVC TaskExecutor for executing asynchronous requests.
To fetch the streaming data from the Apache Solr, I leveraged the Streaming expression of /export which provides the capability to fetch all the documents in the collection. It is one of the implicit requesthandlers provided by Solr. Below is the sample code which showcases the same.
In this approach, the stream is read continuously till the EOF is reached as shown in line number 24. As soon as the Tuple is read it is pushed to the Spring boot outputstream which provides the immediate non-blocking response to the API caller.
The above approach finally resulted in sending the required huge payload to the API caller and that too with the limited set of resources ( CPU, Memory, I/O) in hand.
Below is the complete code for reference
The above code showcase an API endpoint getDepartmentdata which fetches the complete departmental information for all departments from the Apache Solr collection ( department ). The code snippet contains all the configurations needed for CloudSolrStream. We can see that Basic authentication is enabled at Zookeeper and Solr levels as an added security layer.
Conclusion
In this article we have seen possible solution approaches to handle large payloads and how did we leverage streaming capabilities provided by Spring boot and Apache Solr for our use case. Though we could have exposed the Apache Solr /export API directly, it is not a good design to directly expose the backend data source systems. The backend must be shielded with an anti-corruption layer and that's what the spring boot controller & service layer are for. They also provide the capability to transform/polish the raw output before sending it to the API caller.
I hope you found this article to be useful and welcome all your review comments and suggestions.
References
