Companies across industries are increasingly recognizing that making data-driven decisions is a necessity to compete now, in the next 5 years, in the next 20 years and beyond. Data growth, especially unstructured data growth, is at an unprecedented level, and the era of data sprawl and artificial intelligence is upon us.

Implicit in this reality is the fact that AI can meaningfully categorize and process massive amounts of data - not only for tech giants like Alphabet, Meta, and Microsoft, with their massive R&D operations and custom AI tools, but also for ordinary businesses and even small and medium-sized enterprises.

Well-designed AI-based applications can sift through extremely large data sets extremely quickly to generate new insights and ultimately drive new revenue streams that create real value for businesses. But no data growth can be truly actionable and democratized without something new and emerging - vector databases.

With the explosion of big speech modeling, vector databases have become a hot topic. With just a few lines of simple Python code, vector databases can act as a cheap but efficient "external brain" for big language models. But do we really need a specialized vector database? Are vector databases a hype or a necessity?

Recently, InfoQ had the chance to interview Lei Shi, a senior Redis architect, to hear him talk about Redis vector database technology practices.

The following is an edited version of the interview.

InfoQ: Mr. Shi Lei, can you introduce yourself first?

Shi Lei:I currently work for Redis as a Senior Product Architect, managing Redis' technical affairs in Asia Pacific. My main responsibilities are to help Redis customers optimize their Redis implementations, guide them to the best possible experience with new Redis features, and help Redis service providers in China succeed.

As a Redis original, we maintain an open source community edition and offer an enterprise edition of the software. In the Asia Pacific region, more and more customers are beginning to understand the value of Redis Enterprise. However, most people's understanding of Redis is still limited to the open source version or some modified third-party version, and they may have limited understanding of the core functionality and application landscape of Redis. Therefore, my main mission is to help customers better utilize the tools provided by Redis to solve real-world problems and meet business needs.

I joined Redis last year, after a career in research in Singapore and then in AI product development at a fintech startup. At that company, I had been using Redis for about 7 or 8 years, but mainly limited to the open source version. After joining the Redis originals, I've gotten to know the enterprise version of Redis better, and I've learned more about its powerful features. I want to use my experience to help customers realize the full potential of Redis.

InfoQ: When did you first become interested in the field of vector databases?

Shi Lei:Before joining Redis, I focused on product development in AI and Big Data. I've been involved in feature generation, access methods, and real-time AI processing, and I've worked with a number of tools, but it wasn't until I joined the original Redis team that I really realized that Redis was also supporting vector databases. However, it wasn't until I joined the Redis factory that I really realized that Redis was also supporting vector databases, a layout that Redis has actually had for quite some time.

We initially supported search through a search module that's been around since about 2018, and the search functionality in Redis 2.0 takes it to the next level by making it even easier and more convenient. Starting with Redis 2.4, in March of last year, we officially supported vector searches. Redis has been in the vector database space since before the large language model caused a stir. Since Redis is widely used in a variety of industries, many customers around the world started using vector searches as soon as they were introduced.

With the rise of large-scale language modeling, the application of vector databases has entered a new phase. At first, people may just use vector queries to process simple content such as images, videos, audio or documents to extract and search some basic vector features. But with the popularity of large-scale models, people began to explore how to better use vector databases, applying them to higher dimensions, wider scope, and faster requests. The perception of a vector database as niche and low performance is changing rapidly. In this process, Redis has undergone many trials and tribulations. As a vector database, with the rise of large-scale models, many core enterprise applications, such as ChatGPT and OpenAI, are starting to use Redis in the backend, which has led to more opportunities for Redis to work with customers to meet their needs.

At the same time, the Redis search module is also growing. By collecting first-hand information from our customers, our product team quickly translates the needs they encounter in real-world applications and new requirements in the AI and big data environments into products that better serve them.

InfoQ: I noticed you mentioned ChatGPT and OpenAI, who are already using Redis. are they using Redis as their only vector database? Is there any information you can share on this?

Shi Lei:From what I understand, ChatGPT and OpenAI are not just using Redis as their only vector database, they are also working with other vector databases. Because the technology is changing so rapidly, Redis has set up a dedicated team to research and develop vector databases, and is working with a number of different companies.

On the official Redis website, we've shown examples of collaboration with many of the big AI modeling areas, including with ChatGPT and others. However, the specifics and which real-world use cases are using Redis, and what they are, are so rapidly changing in these areas that I don't have up-to-date information on them at this time.

InfoQ: You mentioned earlier that in 2018, Redis introduced the vector search module. What was the situation at that time? Did we develop this feature based on customer demand? Or did we see this big trend ourselves?

Shi Lei:Starting in 2018, Redis introduced a search module. At that time, this search module mainly supported scalar search, not vector search. At that time, one of the pain points in using Redis was that even though it was a key-value storage system in memory, there was no good way to retrieve data based on query criteria without scanning each key individually.

Hence, the 2018 version is mainly designed to address this pain point of searching. It allows users to store a large number of keys in Redis, and these keys are very fast to retrieve. But how do you quickly find data that meets specific criteria among these keys? Through internal stacking and upgrading from version 1.0 to 2.0, we have gathered many customer requirements. These focused on how to quickly create indexes, how to quickly run queries, and how to automate these operations for the application. 2.0, which we launched in 2020, focused on these areas. Then, in version 2.4 (starting last March, before big models like ChatGPT became popular), we formally introduced vector search functionality. During this process, we received a number of requests from customers who asked if we could extend Redis' fast scalar search to search vectorized data. Our product team listened to these customer requests and initially supported a basic vector similarity search feature.

Over time, we've refined this feature by adding a variety of popular search patterns and algorithms to make it more mature. Now, Redis has received a major update in version 7.2 that brings many new features. Notably, we no longer promote search as a module, but rather as one of the main features that Redis offers. This means that Redis can be used not only as a cache and master database, but also as a vector database.

InfoQ: With the increase in functionality, has the positioning of Redis changed a bit?

Shi Lei:Yes, it is. Initially, Redis was really positioned as a memory database, focusing on providing memory storage and extending its functionality through modules. However, over time, we have repositioned Redis. Now, we offer Redis Enterprise Edition software, which integrates all of the functionality within it. With Redis Enterprise Edition, you get full functionality right out of the box, with no additional components to purchase or deploy.

For customers who are already using Redis as a cache, they can now simply store vectors in Redis and perform vector searches directly. This is very intuitive and easy to use for customers, and it adds no additional system complexity, no need to introduce other products or features.

InfoQ: I'd like to know a little bit about the development history of this module and some of the overlays it went through during the development process. Also, did you encounter any problems when integrating it with the Redis database? If there were technical problems, how did you solve them? Can you tell us about the technical implementation process?

Shi Lei:The wide range of applications of Redis in different domains prompted us to gather requirements for Redis search from various domains. Initially, some customers asked for a way to do vector searches in memory. We listened carefully to these customers and worked to implement this feature.

In the implementation, starting with the initial POC (Proof of Concept) project, we added this feature to Redis as an add-on. Over time, we evolved it into a Redis mainstay. In the process, the main Redis product has evolved in tandem with our module functionality. As an example, Redis Enterprise Edition has many features that address pain points in everyday applications, such as a powerful built-in proxy. This proxy automatically directs requests to the appropriate shards, whether they are in single-shard or clustered mode, ensuring that Redis storage and throughput can be scaled automatically without additional intervention. For customers, the built-in agent simplifies business logic, regardless of whether they are in single or clustered mode. This also solves the search problem because each Redis slice is single-threaded, and if requests are centralized on a single slice, performance will suffer. But with clustered mode, the client needs to maintain connections and understand the data on each slice, which makes the business logic complicated.

The Enterprise Edition solves these difficulties and also makes searching easier. In Redis Clustered Edition, because of the built-in agents, search requests are automatically distributed to the various shards for execution, consolidating results at minimal cost. This ensures that Redis is no longer constrained by the performance of a single slice in a search, while delivering larger volumes of faster searches. This scalability and speed of search is made possible by the agents built into Redis Enterprise Edition.

During the development of Redis Search, due to the need to maintain additional data structures such as indexes, our product team optimized to ensure fast allocation and querying of these structures, resulting in a significant performance improvement of the Redis Enterprise Edition over the Open Source Edition.

In addition, we're rolling out enterprise releases, including the latest 7.2, that already introduce features from the preview version. On the search side, we have solved the problem that searches on each slice were still limited to a single thread. With a multi-threaded approach, we have realized simultaneous searches, which in tests have resulted in performance improvements of more than 10x and even 16x. This also shows that there are many ways to further optimize the performance of search, which is an evolving and continuous process.

InfoQ: How long did it take to incorporate the above features into Redis to empower the Redis database?

Shi Lei:The evolution of this technology has taken quite a long time from initial discussions to studies, to research and development, and now to the preview version of the function. As far as I understand it, the actual testing of this function has taken at least more than a year, and the process from initial discussion to actual testing does take some time. It definitely took more than a year before these functions were planned and implemented.

The Redis product team has invested a lot of time and effort, even creating a dedicated team to identify the features that Redis needs to implement as a vector database. This team had to think about what core components could accomplish this functionality, as well as collaborate with other teams. The whole process required collaboration across multiple teams, so it was a long-term development process.

InfoQ: Given how popular big models are and the importance of databases, do you think databases are becoming especially important in this era of AI and big data? Is it necessary to develop new databases to meet the growing demand?

Shi Lei:I think vector databases are now a necessity because they solve several core problems that traditional databases cannot. Traditional databases are mainly based on precise search for keywords, i.e., patterns that exist or do not exist. Vector databases, on the other hand, provide proximity searches, so that when I provide a picture, a text, or a voice, it can find similar matches, not just 0 and 1 results. It gives approximations through a scoring mechanism, which is not possible with traditional databases.

At the same time, the indexing methods of traditional relational databases are not directly applicable to the current vector databases. The reason is that vector databases are completely different from traditional databases at the bottom level, including the computation, data storage, and application levels. Initially, vector databases may only serve as a supplement to relation databases. However, with the development of big data, big models and artificial intelligence, the storage and querying of vectors as well as the fast performance have put forward higher requirements, and only vector databases can fulfill these requirements.

InfoQ: Will the demand for vector databases continue to rise in the future?

Shi Lei:I think this is a continuous rise. With the rise of big models, the demand for vector databases continues to grow. Many traditional vector databases are being stacked and updated, and some products that didn't support vector databases before are claiming to support them and adding this feature. Therefore, this demand will continue to exist, and it is a process of constant reshuffling and elimination of inadequacies.

InfoQ: There is currently some belief that every database of the future will naturally and natively support vector embedding and vector search. What do you think about this view? If this trend does emerge, how will it affect the vector database industry, what are the possible positives, or what are the possible challenges?

Shi Lei:From a technical point of view, almost any storage system or query system can add functionality to support vector search, query or storage. From this point of view, there is no technical problem. However, in practice, we may gradually tend to one or two commonly used types, and other ways may be phased out. Although they are both data storage or database systems, they fulfill indexing and querying requirements in different ways. Traditional databases have difficulty supporting vector queries directly because of the lack of effective support for vector queries in the underlying design. While this can be accomplished by adding functionality, it can be clumsy and inconvenient.

Emerging vector databases may be more responsive to current needs, but they may introduce system complexity. For example, customers may need to use traditional databases, relation databases and vector databases at the same time, which will increase maintenance, cost and overhead. Therefore, we need to find a good balance between a system that meets the requirements of a vector database and one that meets the needs of traditional query functionality. Finding such a combination is key. As a customer, if you can use one system to efficiently accomplish different types of functionality, rather than choosing a traditional system and adding additional functionality, maintenance costs and various costs will be reduced.

In this case, I think Redis can be very good to realize this balance. Redis is not only widely used application, as the enterprise version, it also provides a complete enterprise application ecosystem, which can help customers to meet various needs. Whether it is a vector search or scalar search, in Redis is stored in the memory as key-value, query efficiency is very high. Additionally, Redis has a powerful hybrid query feature that allows querying both vectors and other types of data, such as text, numbers, or GPS information. This native hybrid querying capability gives Redis a significant advantage in vector databases while maintaining high performance.

InfoQ: As a teacher with many years of experience in the database field, what do you think are the key skills that programmers need to acquire nowadays if they wish to pursue a career in AI and vector databases?

Shi Lei:The current technology is superimposed very fast, last year's use of products and experience may have become obsolete this year, or new technologies have emerged. In this case, I think we first need to better understand the existing system. In the case of Redis, for example, most people probably know that it's great at caching, but there are other areas where Redis is used that may not be as well known. As a technology practitioner, it's important to understand the underlying architecture and functionality of mainstream products and what they can do.

We need to constantly update our knowledge, especially with the emergence of new technologies such as vector databases and big models. As a technician, it is important to embrace the new technologies and gain a deep understanding of how they work and how they can be applied. Rather than jumping from an existing technology to a new one, you should utilize your accumulated experience to apply the new technology to your existing work. While there is a cost to this transition, we need to find the most effective way to minimize it, so that the technology works for us, not as a slave to it. This requires experience, technical insight, and a spirit of discovery.

InfoQ: Your words have really given us valuable insights. Finally, we can explore a topic of great concern, that is, the future development of vector databases. Currently, vector databases have entered a popular stage, and many related technologies have become very mature, including vector indexing and traditional database technologies. However, there is a general concern about where the future is headed and what trends we should be looking at.

Shi Lei:From my personal perspective, combining my years of experience in AI and Big Data with my knowledge of traditional databases, I'll say that a few years ago I could not have predicted the extent to which the database space would evolve so quickly.

In the case of vector databases, I believe that their emergence has been driven by a strong drive to quickly eliminate unsuitable technologies, and also by the emergence of new technologies, which is a gradual selection process. In the long run, I firmly believe that vector databases will continue to mature and provide more accurate vector search results for different application scenarios.

To illustrate with a simple example, we may need real-time, fast search, or we may need large-scale feature search. In the future, these needs may gradually evolve into different dimensions. I believe there will be some domain-specific vector databases that will emerge, and there may be one or two or more types of databases that fit a particular scenario. Each type may be optimized for a particular domain, and this will be a process of integration and optimization.

InfoQ: In the future, vector databases will not be like traditional databases, there are more than 200 out in the country?

Shi Lei:I think the continued expansion of the database market is inevitable, mainly due to the very fast pace of technology upgrade, while the technology threshold is also gradually lowering. There is a huge demand, which will attract more and more databases and even vector databases to join the competition. However, from the industry's perspective, this market expansion is favorable. It can lead to more technical and business participation, and although the market will be limited to a certain extent, it will filter out the better technologies and solutions in a competition to better meet the demand.

I hope to see more competitors emerge in this space, and I look forward to seeing which technologies can withstand the test of application and prove their viability in practice. To me, this kind of market expansion should be benign. We don't want to see unhealthy competition, and we shouldn't emphasize our superiority by belittling other applications. I think this is not good for the ecology of the industry. Instead, I look forward to a kind of healthy competition that gives people more quality choices, which in turn drives progress in the field.

Guests::

Mr. Shi Lei is a Senior Solution Architect of Redis, dedicated to providing customers with services such as product architecture consulting and design, performance optimization, application and promotion of Redis technology using Redis Enterprise Edition. He has more than 10 years of experience in software architecture design and development, big data system tuning and optimization, computer vision, fintech AI, etc., and more than 8 years of experience in Redis development, operation and maintenance. Prior to joining Redis, he was Head, AI Architect at a fintech innovation company in Singapore, and Chief Architect at the Institute of Data Science (NUS), National University of Singapore.