Groq, an AI chip startup valued at $2.8 billion, is taking a novel approach to challenge Nvidia's dominance in the AI chip market, particularly targeting Nvidia's CUDA software advantage. The company is offering free access to its inference service, which has attracted over 625,000 developers to use its APIs.

Groq's strategy, internally called "unleashing the beast," involves making its compute power available via a cloud instance for free, with usage caps based on requests per day or tokens per minute. This approach aims to capitalize on the growing demand for faster inference at lower costs in AI applications.

Unlike Nvidia's CUDA, which requires developers to program at the chip level, Groq's system is designed to be more accessible. The company has built over 1,800 models into its compiler, eliminating the need for kernels or CUDA libraries. This means developers can start working with built-in models without needing to program at the chip level, potentially lowering the barrier to entry for users.

Groq is focusing on inference, the type of computing that produces answers to queries asked of large language models, rather than the more energy-intensive training process. The company claims to offer some of the fastest inference speeds available, which it sees as a key competitive advantage.

The startup's approach comes with both risks and potential benefits. While it may not accumulate a stable of developers continuously improving its base software like CUDA has, it offers a potentially lower barrier to entry for users. Groq has shifted its focus, with 60% of its approximately 300 employees now being software engineers.

Groq has ambitious goals, including providing half the world's inference and establishing global joint ventures. The company aims to ship 108,000 of its language processing units (LPUs) by the first quarter of 2025 and 2 million chips by the end of that year, mostly through its cloud platform.

This strategy represents a significant challenge to Nvidia's current market dominance, particularly in the software aspect of AI chip technology. As the AI industry continues to grow rapidly, Groq's approach could potentially capture a significant market share, especially in the inference segment of AI computing.