Introducing Infinispan OData server: Remote JSON documents querying

Dear Infinispan community,

We are proud to announce the very first available version of Infinispan OData server.

This brand new standalone HTTP server is based on odata4j framework (its core is running on OData Jersey) and using Infinispan as an underlying JSON document store.

Clients are allowed to use OData standard query language and OData service operations for communication with the server, storing and obtaining requested JSON documents.

Infinispan OData server understands an internal structure of JSON document, and therefore, clients can query the cache and ask for a collection of results in dependence on specified expression operators.

What's Open Data Protocol (OData)?

 

OData frees stored information from current data silos and exposes the data for variety of possible consumers. To achieve that, OData utilizes common web technologies: HTTP, REST, JSON and Atom Publishing Protocol.

From clients' point of view, databases offer usually diverse APIs. OData focuses its effort on a creation of an uniform access pattern for various sources of data, to unlock them and prepare for broad consummation.

The picture is used from OData introduction video on http://www.odata.org/.

Any HTTP client can access the data or there is also plenty of consumers present in OData ecosystem. And we just added another producer into that ecosystem -- Infinispan OData server which exposes JSON documents stored in caches to various number of clients using industrial standard and platform independent way.

More information about OData: http://www.odata.org/

Our challenge: 

 

Generally, NoSQL solutions lack overall portability as they are different one from each other and almost each database of this kind provide an unique interface for accessing stored data.

Infinispan is not an exception and its library mode usage is mainly focused on Java clients. However, Infinispan did a step further to solve this problem by introducing its server modules. A few implementations of an Infinispan Hot Rod client were developed to support a few different programming languages. There is also an Infinispan REST server which provides an HTTP access to key-value store capabilities. The problem is that users are not able to effectively use Infinispan for operating with documents in commonly adopted JSON format.

Sure thing that Infinispan REST server allows users to store data in JSON format. The only problem is that our REST server supports only key-value access approach as it does not understand internal structure of JSON documents. Therefore, when you have for example cars in your cache, you can access them through their unique keys only. One key -- one value, classic.

But in some cases it could be beneficial to be able to access your data using specific filters and possibly obtain even a collection of results meeting specified criteria.

This is the main advantage which Infinispan OData server offers. As you can see in the picture, OData server understands internal structure of JSON documents and (let's talk in pseudo language for a while) you can obtain "all red cars" without any problems.

And that's it!

Interface

 

You might be interested how you can interact with the server. Recently, it is enough to use any HTTP client, REST methodology and follow OData service operations exposed in server metadata document which describes server interface capabilities and is accessible at:

http://localhost:8887/ODataInfinispanEndpoint.svc/$metadata

Putting data into the cache:

HTTP POST request at:
http://localhost:8887/ODataInfinispanEndpoint.svc/odataCache_put?key='person1' 

with JSON document in the body will do the trick. For obtaining the data, you have two options:

Key-value approach:

HTTP GET request at:
http://localhost:8887/ODataInfinispanEndpoint.svc/odataCache_get?key='person1'

Query approach:

HTTP GET request at:
http://localhost:8887/ODataInfinispanEndpoint.svc/odataCache_get?$filter=
name eq 'Morpheus' and lastname eq 'Matrix'

Full interface is introduced in project README file.

Solution

 

We were able to achieve desired functionality using the @FieldBridge from Hibernate search which helps us to add fields from JSON objects into the Apache Lucene Document and get it indexed. All with help of odata4j framework (http://odata4j.org/).

For now, code is located at: https://github.com/tsykora/infinispan-odata-server/ -- please follow README where you can find information about building, dependencies, server interface and practical usage examples.

Infinispan OData server is the main deliverable of the tsykora's diploma thesis -- for further details accessible at: https://is.muni.cz/auth/th/325144/fi_m/ diploma_thesis.pdf file with the text part and tsykora_attachment.zip archive which contains essential libraries and already built server itself (version 1.0). We also benchmarked this new server against already existing Infinispan servers (Hot Rod, Memcached, and mainly REST in 4-node clustered environment) using Infinispan cakery tool which is based on PerfCake framework.

We managed to significantly improve performance of odata4j framework libraries for our needs but there is still much to investigate there. Infinispan OData server project is very young (started in June 2013) and was developed under respective time constraints.

As we have mentioned, this small Infinispan sub-project is in the early stage and we are gathering feedback and feature requests. Feel free to open new issue in GitHub project repository. Now we plan to focus on more performance improvements, code polishing and addition of more cool features, support for more OData operators and embedded JSON documents, lists, number fields etc. 

Stay tuned!