[ACCEPTED]-Real World Use of Zookeeper-distributed
Free Software Projects Powered by ZooKeeper:
- AdroitLogic UltraESB
- Eclipse Communication Framework
- Eclipse Gyrex
- Talend ESB
Apache Projects Powered by ZooKeeper:
- Apache Accumulo
- Apache BookKeeper
- Apache CXF DOSGi
- Apache Flume
- Apache Hadoop MapReduce
- Apache HBase
- Apache Hedwig
- Apache Kafka
- Apache S4
- Apache Solr
HBase uses Zookeeper for coordinating activities 8 its "head node" was responsible 7 for prior to the current version. The move 6 to using Zookeeper means the central control 5 is no longer a single point of failure.
Zookeeper 4 is very versatile; here is an example of 3 using it to build a distributed concurrent 2 queue:
You can of course also use it to create 1 resource locks, etc, in a distributed system.
Old question, but since this page comes 3 up first on a google search for zookeeper 2 use cases, I figured it would be best to 1 give an updated listing
The Apache CXF implementation of DOSGi uses zookeeper for its service 13 registration repository. Individual containers 12 have a distributed software (dsw) bundle 11 that listens for all service events and 10 when a service status changes that has a 9 property indicating distribution. The dsw 8 talks to the discovery bundle which, in 7 the reference implementation case, uses 6 zookeeper to store service as ephemeral 5 nodes. Other instances will look for changes 4 to the node structure and register proxies 3 on their local systems. The end result 2 is you can code to plain OSGi and end up 1 with transparent distribution.
Norbert is a good example from a scalable production 18 system. I general, it integrates Netty, Protocol 17 Buffers and Zookeeper into a lightweight 16 framework for running clustered services. Protocol 15 Buffers are used to specify your service 14 API, Netty implements transport-layer abstractions 13 and Zookeeper is essentially a fault-tolerant 12 discovery service.
Every time a service instance 11 is started Norbert registers it as available 10 instance of a particular service type. From 9 implementation perspective, it creates two 8 Zookeeper trees:
- "/ServiceName/members" which lists all known instances of the service
- "/ServiceName/available" which lists currently available instances of the service
The most important property 7 for each node is the url to use to connect 6 to the corresponding service instance. It 5 enables client-side load balancing - a Norbert 4 client finds the list of urls for a given 3 service name and attempt to connect to one 2 of them is some order (e.g. round-robin 1 or random).
There is a good article ZooKeeper - The King of Coordination about ZooKeeper 7 at Elastic Cloud.
At Found, for example, we 6 use ZooKeeper extensively for discovery, resource 5 allocation, leader election and high priority 4 notifications. In this article, we'll 3 introduce you to this King of Coordination 2 and look closely at how we use ZooKeeper 1 at Found
Solr is also working to integrate ZooKeeper. Here you can see they are 2 using for dynamic config, sharding, SPOF 1 elimination (master/slave election), rebalancing, etc.
- Storm is used by a number of companies (Twitter and Groupon being two of the better known) and relies on Zookeeper.
- Kafka is used by Linkedin and relies on Zookeeper.
Storm uses Zookeeper to store all state so that 10 it can recover from an outage in any of 9 its (distributed) component services.
This 8 allows the component services to be stateless 7 and simply download or sync with the Zookeeper 6 servers when configuration data is needed. If 5 you have ever had to recover a production 4 server you will know what a headache this 3 can be!
Kafka queue consumers can use Zookeeper 2 to store information (high water mark) on 1 what has been consumed from the queue.
Zookeeper was used for many things other 3 than configuration. Here is a official list 2 of implement distributed primitives using 1 zookeeper.
In my case we are storing configuration 4 files in zookeeper ensemble for cluster 3 usage . We are using leader -> follower 2 schema . So when one zookeeper down we are 1 switched for another one (replicated mode)
Neo4j uses Zookeeper their High Availability 1 enterprise server! http://docs.neo4j.org/chunked/milestone/ha.html
datomic uses apache zookeeper to manage riak based 8 data storage.
Because Riak supports only 7 eventual consistency at this time, a Datomic 6 system running on Riak also utilizes Apache 5 ZooKeeper, a highly-available coordination 4 service. Datomic uses ZooKeeper for transactor 3 failover coordination, and for the handful 2 of keys per database that need to be updated 1 with CAS. source: http://blog.datomic.com/2012/11/riak-and-couchbase-support.html
Here's some detail on how HBase uses ZooKeeper, including information on how 4 they intend to use it in future. Generally 3 they use it for eliminating SPOF on the 2 region servers via Leader election implemented 1 using ZooKeeper.
More Related questions