Sidu Ponnappa
Apr 12, 2024
For typical enterprise workloads, LLMs are raw material. You will need software engineering to turn them into a scalable resource that does real work.
Moving on from a clever demo to a production-grade system that solves real business problems at scale is far from trivial. It's also where I think a lot of the current hype around LLMs falls short.
A black box only adds to the challenge. It goes against a lot of the principles and practices we have built up over decades of working with observable, debuggable systems.
As engineers, we're used to being able to introspect our systems at every level. We can trace the flow of data through our code, set breakpoints, log key events, and incrementally test our changes. But with LLMs, a lot of that goes out the window.
Suddenly, you're dealing with these massive, opaque boxes at the core of your system. You can't easily peek inside to see how they're arriving at their outputs. You can't step through the code line by line.
The best you can do is develop heuristics. This is a huge cognitive shift for engineers. We're used to being able to reason about our systems in a fairly linear, deterministic way. But LLMs introduce a level of uncertainty and non-determinism that requires a hacker's mindset.
That's what we need to cultivate as an industry. Not just prompt engineering, but across the full stack. It's an incredible opportunity for folks who can bridge that gap.
Teams that can get down and dirty, iterate faster, and shorten cycle times will be the first to gain adoption and establish a moat.