Spotlight: Qodo Innovates Efficient Code Search with NVIDIA DGX

In retrieval-augmented generation (RAG) systems, embedding models play a crucial role by transforming text into high-dimensional vectors that capture semantic meaning. These embeddings are stored in a vector database and enable efficient similarity searches, enabling the system to retrieve the most relevant information from a knowledge base when responding to user queries.

For code-specific tasks, using an embedding model trained on both programming languages and software documentation is particularly strategic. Such a model can better understand the nuances of code syntax, function names, and technical terminology, leading to more accurate retrieval of relevant code snippets or documentation. 

This specialized embedding model can significantly enhance the performance of RAG systems in software development contexts, helping to improve code completion, bug detection, and the generation of technical documentation.

Compared to LLMs, embedding models are significantly smaller, and thus can be more efficiently distributed across multiple GPUs. This enables better utilization of hardware resources and potentially faster training times. As such, they are more amenable to data-parallel distributed training, where the entire model is replicated on each GPU worker, and batches of data are split among multiple GPUs.

Qodo trained their embedding model using a NVIDIA DGX 8x A100 80GB node. Training at bfloat16 numeric precision enabled them to use large micro-batch sizes of 256, speeding up convergence rate and training time. This is important for embedding models using a contrastive loss, especially when relying on in-batch negatives. 

A larger batch size enables the model to sample a more diverse set of negative examples, which is essential for effective learning. That diversity helps the model better distinguish between similar and dissimilar instances, leading to improved representation quality.

Qodo fine-tuned two embedding models, Qodo-Embed-1-1.5B and Qodo-Embed-1-7B, based on Qwen, an open-source LLM developed by Alibaba Cloud and designed to perform a wide range of AI tasks. They achieved SOTA accuracy, leading the Hugging Face MTEB::CoIR leaderboard with both models in their respective size category (Figure 4).

NDGC is a specific metric used to assess the quality of information retrieval.

Case study: Internal code search

A recent collaboration between NVIDIA and Qodo shows the value of Qodo’s solution through a real-world use case. The work focused on enhancing the accuracy of one of NVIDIA’s internal RAG solutions (Genie) for searching private code repositories. The end goal was to perform LLM-based queries on NVIDIA’s internal code repositories to generate accurate and precise responses. 

To achieve this goal, we substituted existing industry-standard components in the Genie project pipeline with Qodo’s specialized alternatives, improving the system’s ability to mine NVIDIA’s internal code repositories and yielding superior results.

The following Qodo components were integrated into the pipeline:

• Code indexer for GitLab and GitHub
• Code RAG Retriever
• Embedding model (Qodo-Embed-1-7B)

As discussed earlier, one of the challenges of building a code-specific RAG solution is chunking.  Large code files should be split at natural stopping points to ensure that text chunks are optimally sized for processing and storage. Otherwise, the retrieval process fails when critical relevant code sections are out of context.

The final pipeline was integrated into NVIDIA’s internal Slack system, allowing expert C++ developers to ask detailed technical questions based on repositories of interest and receive robust responses. 

Figures 6 and 7 show the example output from each pipeline, the original and the one built in collaboration with Qodo, respectively. The highlighted rectangle in Figure 6 shows that the original pipeline couldn’t respond with the specific data points.

Figure 7 shows a far more detailed result.

For testing, we used the following common graphics public SDKs:

We used Ragas to generate synthetic questions based on these datasets and compared which responses were more correct and had technical detail. Each of the three row pairs corresponds to one of the SDK repositories from the list. Each column represents a breakdown of questions, where the value in each cell includes the number of correct responses based on faithfulness and answer relevancy. The final cell in each row shows the total number of correct responses.

Conclusion7

You can experiment with Qodo’s embedding models, Qodo-Embed-1-1.5B and Qodo-Embed-1-7B, on Hugging Face. 

Startups that want to accelerate their work should explore the free benefits available through the NVIDIA Inception program.

For more information, see the following resources: