You don’t have to work in IT support to know that the very first troubleshooting step for any device or software is to try turning it off and on again. Does this apply to AI, too?

Turns out that it does. Through both personal experience of trying out GenAI tools and reading about the outcomes others have been getting from LLM chatbots, I’ve come to realize what a double-edged sword it is to give AI context. Meaning, instead of shooting a fresh new question at an LLM that doesn’t have any data surrounding that specific prompt, when we continue talking with a chatbot for extended periods of time and accumulate contextual information.

It’s counterintuitive that a machine which gets more highly personalized and relevant data about you & what you are trying to achieve may become less useful as a result of this. When building classic IT systems for business use, it was always given that more and better quality data will lead to better outcomes for users. When it comes to large language models that act as a kind of calculator for words, this isn’t always true - in the context of an individual user and their experience of working with AI.

Most of us know by now that LLMs are non-deterministic and will give different answers to the same question at different times and in different contexts. What’s more subtle and not as obvious to users of tools like ChatGPT and Copilot is that the direction where the discussion is heading has a major impact on what the outcome will be. The longer you keep working with the same AI without rebooting it, the less in control you are about what will happen next.

In his newsletter, the cognitive scientist and vocal critic of “LLMs lead to AGI” ideology, Gary Marcus, talked about the inherent limitations of current GenAI that relies solely on neural networks trained with massive amounts of data. Marcus is a heavy proponent of neuro-symbolic AI that would combine these neural networks with tools for symbolic reasoning. For example, letting LLMs run Python, like many of them can already do today. But they run it more as a “bolt-on” rather than an explicit capability of the models.

When there is training data with similar examples as those that the user is trying to get an LLM to answer, the results can be almost magical. The problem is that once you venture outside the knowledge of the model, it may not be able to produce a correct answer - but it will absolutely never give up. It is then up to the user to know when to give up. From the newsletter:

The model will assume a direction from the first prompts and then keep predicting the next tokens. It will try to hold on to the previous data in the chat - which really is what has made the experience of talking with ChatGPT so relatable to us humans. It feels like the AI chatbot is understanding more and more about who we are and what we need as the discussion continues.