Career / Journey
History of NLP: From TF-IDF to BERT, LLMs and AI Agents, a Practitioner's View
How NLP and AI changed from statistics and random forests to BERT, RAG and agents, as one practitioner saw it, and why I now care about meaning.
On this page
- 2006 to 2011: the pieces fall into place while I study statistics
- 2012 to 2017: a breakthrough I understood only later
- Early 2018: TF-IDF, BM25 and my first neural networks
- 2019 to 2020: BERT and Transformers give words context
- 2022 to 2023: ChatGPT and open-weight LLMs, from training models to using them
- 2024 to 2026: from models to AI agents and systems
- 2026: why I now care about meaning
- What the statistician still brings
- Sixteen years of catching up
I still remember waiting more than a minute for a single BERT embedding to come back from a server running on my local CPU. That was around 2019. Today I can ask a coding agent to inspect a repository and attempt changes across many files. This post is about everything in between: how I watched the AI field, and NLP in particular, develop. Even the problems we are trying to solve have changed along the way.
Generative and agentic AI is now one of the hottest fields in technology, but for much of my education and early career it was not the AI we talk about today. I spent five years at IIT Kanpur on statistics, econometrics and mathematics, and my courses at the University of Washington covered Bayesian methods, time series and cross-country panel data. I don’t remember meeting a neural network in any of them. I suspect many practitioners of my generation have a similar story: we were trained on one family of models, and the field moved to another while we were busy elsewhere.
This is not a complete history of AI. It is how that history looked from where I stood, over about sixteen years. What I find most interesting is not how much more capable the models became. It is how the work of building AI changed, and how the bottleneck kept moving.
2006 to 2011: the pieces fall into place while I study statistics
I started at IIT Kanpur in 2010 and finished my bachelor’s and master’s in 2015. My days were p-values, t-tests, z-tests, probability distributions, regression and the assumptions behind a normal distribution.
In hindsight, the pieces of modern deep learning were being assembled elsewhere. In 2006 Geoffrey Hinton and colleagues showed that deep belief networks could be pretrained layer by layer, which renewed interest in deeper networks. NVIDIA introduced CUDA the same year, making GPUs usable for general computation. (If this part interests you, Stephen Witt’s book The Thinking Machine tells the NVIDIA side of it well.) In 2009 ImageNet gave computer vision a large dataset and a common benchmark, and neural networks began improving speech recognition. Better algorithms, more data, more computing power and shared yardsticks were arriving together.
Almost none of it reached my classrooms, at IIT Kanpur or at the University of Washington. I don’t think that was bad teaching. Neural networks had not yet earned wide trust, and the field was more confident in the methods it already knew. I was learning the foundations of statistical modelling without knowing how much the practice of building AI was about to change.
2012 to 2017: a breakthrough I understood only later
In 2012 AlexNet won the ImageNet competition with a top-five error of 15.3%, against 26.2% for the next-best entry (AlexNet paper). It showed what deep neural networks could do with large datasets and GPUs, and it is where trust in neural networks started to return. I was still studying when it happened, and I did not appreciate it.
Later, when I moved into machine learning and data engineering, my problems were classification problems, usually solved with random forests and similar models. The work was practical: prepare the data, choose an algorithm, tune the parameters, compare train and test accuracy, ship. Industry seemed to care less about the statistical theory I had spent years on and more about eval-driven tuning.
It also started to feel repetitive. Cloud platforms such as AWS began offering model training as a service, and I wondered whether classification would become fully automatic: give the system data in the right format, say what to predict, and let it train itself. At one point I felt the field had become monotonous.
Looking back, this is the part of my story I think is most typical. The deep learning revolution was real, but the methods I used still worked for the problems I had. A research breakthrough and widespread adoption are different events. A technology can show extraordinary results in one domain while practitioners elsewhere keep using what works well enough, costs less and is easier to deploy.
Early 2018: TF-IDF, BM25 and my first neural networks
In early 2018, as a senior ML engineer, I moved into NLP. We used NLTK and spaCy as base modules to analyse text and extract things like named entities, and I started exploring RNNs and LSTMs for text extraction. One of my first neural problem statements was an image captioning system: ResNet to extract features from an image, then an RNN or LSTM to write a description. It was a different way of thinking. Instead of designing every feature and rule by hand, we could let the model learn representations from the data.
At the same time we were building a search engine, and its core was traditional information retrieval: inverted indexes, TF-IDF scoring, later BM25 through Elasticsearch, and a number of query filters. For synonyms we built our own algorithm, with dictionaries maintained automatically through a mix of BM25 and fuzzy matching. It worked, but maintaining the relationships between words became a problem of its own.
Later we tried word2vec to find clusters of related words. It helped us spot terms that appear in similar contexts, but it gave each word one vector regardless of the sentence. Take the word “claim”. In insurance it is a request for payment; elsewhere it is an assertion that something is true. A single static embedding could not tell the difference. We had moved beyond matching exact words, but we had not solved meaning in context.
2019 to 2020: BERT and Transformers give words context
Around 2019 we learned about BERT (paper), built on the Transformer from the 2017 paper Attention Is All You Need. What excited me was that it could represent a word using its surrounding text and encode whole sentences, where word2vec worked word by word. After years of synonym dictionaries and relevance tuning, it felt like a real step forward.
Then came the compute problem. On my CPU machine, one embedding could take more than a minute. That was my setup, not a property of BERT, but it made the trade-off concrete: a richer representation is not useful if its latency makes the application impractical. Sentence-BERT, published in 2019, offered a more practical way to produce sentence embeddings for semantic search. Representation, latency, cost, retrieval quality and the needs of the application all count.
GPT-2 followed in 2019. I started reading about its architecture to understand it, and before I got very far, OpenAI released GPT-3. For the first time I felt that the technology I was studying might be replaced before I finished understanding it. The question was no longer just how to learn a new architecture, but how to keep building useful systems while the underlying capabilities moved this fast.
2022 to 2023: ChatGPT and open-weight LLMs, from training models to using them
ChatGPT launched at the end of 2022, built on GPT-3.5 and trained further for conversation with human feedback (OpenAI announcement). It was the most viral technology moment I have witnessed. For me the biggest change was not the interface but how I could approach a new problem. Before, a use case began with labelled data, model selection, feature engineering and a training pipeline. Now I could give a pretrained model instructions and examples, see what it could do, and decide whether the results justified more engineering. The cost of getting started had collapsed.
Open-weight models widened it further. Meta released LLaMA in February 2023, Alpaca and RedPajama followed, and quantization let us run 4-bit and 8-bit versions on modest local machines. We used them to explore the ontology and governance layers of a large data problem.
Then a new company gave me access to GPUs, and I could train and test self-hosted models instead of only calling hosted ones. Andrej Karpathy, Umar Jamil and others had published step-by-step guides to building Llama-style models from scratch, and following them gave me my first close look at how a Transformer is put together. I built a few small ones myself. It was wild to go from using models as tools to seeing how the parts fit.
Fine-tuning LLMs, and falling behind
For insurance use cases I fine-tuned Llama and Mistral models, and I trained more than 800 versions, varying the data, the settings and which layers to freeze. Every run was an attempt to find a better combination. But the field kept moving. Llama 2 arrived in July 2023 and Mistral 7B in September, and before we had found our best combination, newer and stronger models appeared. I often felt two years behind, both in understanding the architecture and in adopting the latest techniques. Being able to train a model did not mean that training one was the right decision.
In parallel we built retrieval-augmented generation systems for business problems at scale. RAG brought me back to search, with a new responsibility: retrieved text was now handed to a model to write an answer. Fine-tuning changes a model’s parameters, while retrieval supplies information at request time. If the question is about an updated insurance policy, retrieving the current policy beats training again. Sometimes the answer is a better model, and sometimes it is better data, better retrieval or a better way of supplying context. We were moving from building models for tasks to assembling systems.
2024 to 2026: from models to AI agents and systems
Over these years I watched coding move from autocomplete to vibe coding to agentic coding, and agents spread from software development into research, analysis and everyday productivity work. For me the period also meant co-founding Order.law as its CTO in January 2025 and, in March 2026, moving to head the R&D Lab at Indexnine. The steps were not a neat sequence, but the scope of what we could ask a system to do grew considerably.
Here are the milestones and shifts that mattered most to me. The dated ones come first, followed by the changes in focus.
The difference between a coding assistant and an agent is not just better code. The system around the model gives it tools and lets it carry work through several steps. That changes the engineering problem. Once a system can modify files or run commands, it is no longer enough to ask whether the answer looks right. We need to know what it changed, whether the tests passed, which permissions it used and how we recover when something goes wrong. The same shift is happening outside coding, where a system that can retrieve information, interpret it and act in an enterprise application is far more useful, and far more able to make costly mistakes.
From prompts to context engineering
As I built more capable workflows, the output depended more and more on what the model was given: instructions, relevant documents, earlier steps, tool results, constraints and state. Context engineering treats that selection and organisation as an engineering problem, not a prompt-writing trick. A coding agent on a large repository needs the relevant files, conventions, dependencies and tests, and needs to avoid being drowned in irrelevant material. More context is not automatically better context.
This is where my search years came back. We had spent years trying to retrieve the right document for a query. Now we had to decide which information an agent needs at each step, how it should use it, and whether the result stays grounded in the evidence.
The new problems arrived quickly
As AI got easier to use, the problems around it multiplied. How do we manage the flood of low-quality generated output, the AI slop? How do we stop agents from acting on wrong assumptions, or from reaching data they should not see? How do we measure whether an agent actually completed a task rather than offered a convincing explanation? Security threats, concerns about proprietary models and reverse engineering, and a more serious public debate about cognition and consciousness all joined the daily conversation. None of these can be solved by picking a more capable model. They need engineering around the model: evaluations, verification, permissions, audit trails, and limits on cost and scope.
2026: why I now care about meaning
The biggest lesson of this journey is that, in many enterprise applications, model capability is no longer the only bottleneck. A fluent agent can still be wrong if it does not understand what the business means by its own words.
Consider an insurance system. A customer, a policyholder, an insured person and a beneficiary can be four different people. They may overlap on one policy, but the concepts are not interchangeable. A model can retrieve the right document, write a grammatical answer and still confuse them, and the answer looks reasonable to anyone who does not know the domain. In a real workflow, the distinction matters. This is a different problem from finding relevant text. The information often already sits in the company’s databases and documents. The work is to make the concepts, relationships and constraints explicit enough for a system to use them correctly.
That is why my attention has moved to context engineering, ontologies and knowledge graphs. An ontology describes the concepts of a domain, the relationships between them and the rules that apply. A knowledge graph represents entities and relationships in structured form. With reliable source data, retrieval, validation and access controls, they let a system work with business meaning instead of guessing it from text. They are not magic: definitions go stale, source data is incomplete and a model can still misuse what it receives. The point is to make the important meaning explicit and give the system ways to use and check it. The big data platforms have noticed: Databricks, Microsoft and Snowflake have each added an ontology or semantic layer for agents during 2026.
There is a line back to the search systems I built years ago. We began by matching terms, then added statistical ranking, synonym dictionaries and learned representations. Contextual embeddings improved how we represented language, and RAG let us supply relevant information to a model. But retrieving relevant information and understanding what it means in a particular business are not the same thing. The problem did not disappear. The cost of getting it wrong grew.
What the statistician still brings
For a while I worried that the habits from my statistics years were becoming irrelevant. I now think the opposite. When we evaluate an AI system we need to know what an evaluation actually proves. Does the benchmark represent the cases that matter? Is the sample large and representative enough? How uncertain is the result? Are we measuring the quality of an answer, or whether the task was completed correctly? A model can score well on a benchmark and fail on an important class of real cases. An agent can finish most tasks and still make a few costly mistakes, and a better average can hide a regression exactly where correctness matters most. Statistics did not prepare me for every new architecture, but it gave me a way to think about evidence, uncertainty and the gap between a result that looks promising and one I can trust.
Sixteen years of catching up
I now work 12 to 13 hours on most days, partly because I enjoy exploring what is possible and partly because of FOMO, the fear of missing something interesting. There is always another model, framework or agent to try. I cannot explore everything, and I am slowly learning that trying every release is not the same as understanding the important developments. The better question is what actually changed, which problems got easier, and what new constraints must be solved before the technology is useful in practice.
What keeps surprising me is how often the bottleneck moves. First we had to learn to train models that worked. Then we had to make them fast and cheap enough to deploy. Pretrained models lowered the barrier to starting but not the work of making systems reliable. Now that AI takes actions instead of only producing answers, the system around the model matters as much as the model.
I started with statistics, moved through predictive models and search, and now work with systems I could not have imagined in those early experiments. I can’t imagine a more exciting time to be building. I still remember waiting a minute for one BERT embedding, and the question I ask today is the same one, in a much wider setting: what will make this useful enough to trust?