Airbyte Makes Hundreds of Data Sources Available for Artificial Intelligence Applications

New vector database connectors, including Pinecone and Chroma, enable companies to sync data from over 300 sources for Large Language Model use cases

Airbyte, creators of the fastest-growing open-source data movement platform, today made available connectors for the Pinecone and Chroma vector databases as the destination for moving data from hundreds of data sources, which then can be accessed by artificial intelligence (AI) models.

“We are the first general-purpose data movement platform to add support for vector databases – the first to build a bridge between data movement platforms and AI,” said Michel Tricot, CEO, Airbyte. “Now, Pinecone and Chroma users don’t have to struggle with creating custom code to bring in data; they can use the new Airbyte connector to select the data sources they want.”

Because vector databases have the ability to interpret data to create relationships, their usage is increasingly popular as users seek to gain more meaning from data. Vector databases are ideal for applications like recommendation systems, anomaly detection and natural language processing, and as sources for AI applications – specifically Large Language Models (LLM).

The vector database destination in Airbyte now enables users to configure the full ELT pipeline, starting from extracting records from a wide variety of sources to separating unstructured and structured data, preparing and embedding text contents of records, and finally loading them into vector databases – all through a single, user-friendly interface. These vector databases can then be accessed by LLMs. All existing advantages of the Airbyte platform are now extended to vector databases, including:

• The largest catalog of data sources that can be connected within minutes, and optimized for performance.
• Availability of the no-code connector builder that makes it possible to easily and quickly create new connectors for data integrations that addresses the “long-tail” of data sources.
• Ability to do incremental syncs to only extract changes in the data from a previous sync.
• Built-in resiliency in the event of a disrupted session moving data, so the connection will resume from the point of the disruption.
• Secure authentication for data access.
• Ability to schedule and monitor status of all syncs.

Airbyte continues to innovate and support cutting-edge technologies to empower organizations in their data integration journey. The addition of vector database support marks another significant milestone in Airbyte's commitment to providing powerful and efficient solutions for data integration and analysis.

The vector database destination is currently in alpha status and available supporting: Pinecone on both Airbyte Cloud and the Open Source Software (OSS) version; Chroma and the embedded DocArray database on Airbyte OSS; plus more options in the future.

Airbyte makes moving data easy and affordable across almost any source and destination, helping enterprises provide their users with access to the right data for analysis and decision-making. Airbyte has the largest data engineering contributor community – with more than 800 contributors – and the best tooling to build and maintain connectors.

To learn more about the new Airbyte vector database destination:

• Register for the upcoming webinar, Airbyte powers AI through vector databases, on Thursday, August 17
• Read the blog post, Why AI shouldn’t reinvent ETL
• Review the tutorial on integrating a vector database into your LLM

About Airbyte

Airbyte is the open-source data movement leader running in the safety of your cloud and syncing data from applications, APIs, and databases to data warehouses, lakes, and other destinations. Airbyte offers four products: Airbyte Open Source, Airbyte Enterprise, Airbyte Cloud, and Powered by Airbyte. Airbyte was co-founded by Michel Tricot (former director of engineering and head of integrations at Liveramp and RideOS) and John Lafleur (serial entrepreneur of dev tools and B2B). The company is headquartered in San Francisco with a distributed team around the world. To learn more, visit airbyte.com.

View source version on businesswire.com: https://www.businesswire.com/news/home/20230808045224/en/