Thriving Thriving Java JV
↑ 39 ?
waggle-dance ExpediaGroup/waggle-dance

Hive federation service. Enables disparate tables to be concurrently accessed across multiple Hive deployments.

HEALTH ? 89 / Healthy
// legacy hero hivehive-metastorefederationmetastoreoss-portal-listed ?

// readme

Start using

You can obtain the Waggle Dance TGZ and RPM packages from GitHub Releases. From 4.2.0 they are no longer published to Maven Central; earlier versions remain available there. The Waggle Dance libraries are published to Maven Central:

Overview

Waggle Dance is a request routing Hive metastore proxy that allows tables to be concurrently accessed across multiple Hive deployments. It was created to tackle the appearance of dataset silos that arose as our large organization gradually migrated from monolithic on-premises clusters to cloud based platforms.

In short, Waggle Dance provides a unified end point with which you can describe, query, and join tables that may exist in multiple distinct Hive deployments. Such deployments may exist in disparate regions, accounts, or clouds (security and network permitting). Dataset access is not limited to the Hive query engine, and should work with any Hive metastore enabled platform. We’ve been successfully using it with Spark for example.

We also use Waggle Dance to apply a simple security layer to cloud based platforms such as Qubole, DataBricks, and EMR. These…