Hibari is a distributed database for cloud computing based on such key concepts as key-value storage, storage bricks and chain replication.

Hibari is a distributed, non-relational database management system (distributed non-RDBMS) based on distributed key value store technology, implemented by Gemini Mobile Technologies, it inherits important characteristics of both the distributed non-RDBMS world and of Gemini's HyperScale® Platform. Gemini is very excited to contribute Hibari to the community as an Open Source product. Please check out the details on Hibari's Open Source initiative here.

Hibari has the following characteristics to flexibly support growing volumes of data in cloud computing space.

For instance, when applied to large-scale email systems, Hibari can provide economic efficiency and thus avoid a typical situation: employees/customers occasionally must delete messages and files to free up space in limited capacities of mail sytems and file servers.

Also, when "Hibari" is applied to a large-scale archive system, a storage system can be economically developed and operated to perpetually store and manage vast amounts of information automatically generated by a wide range of systems, e.g., logs and telemetric measurements (for instance, location of cars and other moving devices, and usage of utility services).

Data Model
Hibari provides a simple data model that has five (5) attributes; keys to uniquely identify data items, values that correspond to keys, timestamps to detect race conditions for accessing, expiry to manage expiration of data, and flags to store meta data.

Hibari Key Concept: Consistency and Availability
Hibari incorporates in its architecture a concept of storage bricks to build a large storage system by using a number of commodity computers connected to a network and a concept of chain replication to provide high throughput and availability without sacrificing data consistency in a large storage system.

Chain Replication
In Hibari, a database cluster consists of multiple chains, and each chain is set up of multiple groups of brick storages (of computers connected to a network). It means that a database cluster of Hibari consists of a number of brick storages that are grouped into chains.

For chain replication, each chain is basically structured with three types of brick storages; one head brick, one tail brick, and zero or more middle bricks. Data is replicated in all brick storages within the chain.

When there is a request for data write to a chain in a database cluster, the request is sent to a head brick of the chain. The head brick receiving the data write request updates data and sends it via a middle brick to a tail brick to execute replication within the chain. A reply to the write request is returned from the tail brick. Chains of length two have no middle brick. In chain of length one, the role of head brick and tail brick are performed by the same brick (Chains of length one have no data redundancy.)

When there is a request for data read in a chain, the request is sent to a tail brick of the chain. The tail brick returns a reply.

If a failure happens in a brick storage (node) within a chain, the chain is automatically shortened to sustain normal chain operations. For instance, when a chain configuration has three brick storages, even if two of them fail simultaneously, access to data in the chain can be maintained. When a brick store is restarted, its data will be repaired automatically. When a brick's repair is finished, the brick will be automatically re-added to the chain.


Client Application (Development Language and API)
Hibari provides various development options to flexibly support different needs of development of client applications.

Hibari provides highly versatile APIs including Amazon S3, JSON-RPC-RFC4627, Universal Binary Protocol (UBF), Thrift(*), Avro(*), Google's Protocol Buffers (*): (*)=under development/roadmap and supports Java, C/C++, Python, Ruby, Erlang.


High Performance
Erlang, a programming language which excels at parallel processing, is the driver for high performance of Hibari. Erlang shares many of the design principles as HyperScale® Technology and simplifies integration with other Gemini HyperScale® software components. The following two approaches also help to build a high-performance distributed system.

The first approach is data distribution based on consistent hashing algorithm. The algorithm is used to allocate data stored in Hibari in multiple chains within a database cluster. Data allocated in multiple chains enables parallel execution of simultaneous access to data distributed to individual chains.

The second approach is storage options. Hibari can provide options of disk base storage and RAM (on-memory) base storage. For instance, when high speed is particularly required in a system, RAM based storage can be selected. In both cases, data is written and flushed to disk to protect against data center power failures. A "batch commit" technique is used to minimize disk I/O.

High Scalability
Hibari adopts an architecture that allows dynamic addition of brick storages (nodes) that construct a database cluster. The architecture enables the expansion of database clusters, such as upgrading storage capacities and access performance, without service outage.

Also, Hibari supports the following features for easy system expansion.

Hibari thus has an architecture that enables dynamic resource addition to clusters and provides automatic data management functions.

High Availability
Hibari can deliver high availability of data access based on consistent hashing algorithm and chain replication.

Consistent hashing algorithm of Hibari is the basis for partitioning of key space of database to allow distribution of data in multiple chains. Efficient distribution of data to database clusters drives parallel processing of access to individual chains, improving the availability of data access.

Also, each chain of database clusters of Hibari can be redundantly configured by chain replication. As long as at least one brick storages (nodes) within a chain is running, data within the chain can be still accessed. For instance, when a chain configuration has three bricks even if two of them fail simultaneously, access to data in the chain can be maintained.

In Hibari, all other components of a system can be also redundantly configured to totally eliminate a single point of failure (SPOF) within the system. Each brick store machine is an independent unit of failure. Failures of disk, RAM, CPU, power supply, network interface, or any other component are all considered the same by the chain replication implementation. This architecture is frequently called a "shared nothing" storage architecture.

High Reliability
Generally speaking, maintaining consistency of data items that are replicated for redundancy is an important challenge for distributed databases.

Hibari guarantees strong consistency of multiply replicated data items distributed in a database. In reply to a request for data read from a client, the system always provides the latest data for which write operation is completed.

Also, when a RAM based storage option is selected, Hibari records all data updates on disks to ensure data durability.

Economical
Hibari can realize economical system build-up, using commodity hardware, regardless of a required size of a system. Hibari's system expansion method is basically "scale-out." A size of clusters (extension or reduction) can be flexibly changed, without any service outage, in cope with a change of required processing performance or storage size of a system.

In addition, hardware of differing capacities can be mixed as nodes in a cluster. It allows the user to select the latest, cost effective hardware whenever it is necessary to extend a cluster's size.

Also, reallocation of data upon a change of cluster size or recovery after a failure can be automatically executed by automatic balancing of data allocation, simplifying operations and reducing overall operation cost.