spark connector for Milvus
// momentum
// why it's here
- Active: last commit today
- Health 99/100 with only 22 stars
- 22 forks for 22 stars: people build on it
// how to help
// readme
Spark-Milvus
Read and write Milvus Collection data with Apache Spark.
Spark-Milvus brings Milvus Collection data into Spark DataFrames for analysis, data cleaning and batch updates using Spark SQL, Scala or PySpark.
Why Spark-Milvus
Milvus powers online vector search. Spark-Milvus makes the same Collection data available to Spark for offline processing, connecting retrieval and data preparation workflows.
With Spark-Milvus, you can:
- Analyze Collection data: Join it with other data sources, inspect field values and measure data quality using familiar Spark operations.
- Update existing fields in batches: Clean text, standardize metadata or compute replacement values, then write the results to Milvus data files.
For example, Milvus can serve similar-product searches from a product Collection. Spark can read that Collection, join it with a product catalog to identify inconsistent descriptions, and write back corrected values. Your application controls when those updates are submitted to Milvus and become visible to online queries.