Generative Artificial Intelligence relies on deep learning models (artificial neural networks consisting of three or more layers) to identify and encode patterns and contextual relationships in large data sets. These models use this information to understand user requests and generate responses. Responses can be text, images, audio, video, or software code. They often rely on transformer based large language models (LLMs) but can use other model approaches such as generative adversarial networks (GANs), variational autoencoders (VAEs), or diffusion models.

Large language models (LLMs) are a specialized form of multi-layered neural net (specifically using an architecture known as a transformer) that are trained on large collections of natural language text. This data is typically collected from the internet, user input to GenAI tools, or from other available sources. LLMs are a type of foundation model, which are models trained on extremely large data sets in order to provide foundational capabilities applicable to a wide variety of use cases, as opposed to more specialized, domain-specific models.

LLMs are designed to understand and generate text, including inferring semantic context and generating relevant responses. They do this by leveraging billions of parameters within an artificial neural network to map contextual relationships in language. These techniques can be applied to non-linguistic tasks as well, such as image or audio processing.

 A transformer is a neural network model first proposed in a famous 2017 paper by researchers from Google and the University of Toronto titled “Attention is All You Need.” Transformer models enable input strings to be processed in parallel and to selectively retain sequential information inputs. This allows the model to better account for contextual relationships, speed up training, and improve efficiency by distributing computations over multiple processing units (typically graphics processing units, or GPUs for short).

OpenAI’s GPT (Generative Pre-Trained Transformer) and Google’s BERT (Bidirectional Encoder Representations from Transformers) were the first major applications of the transformer approach. While originally developed for natural language processing (particularly machine translation), transformer models have since been adapted to a variety of applications, including natural language processing, image and video processing, audio processing, etc.

Because computers cannot understand language directly, text must be converted to numbers that machines can act upon computationally (primarily using vector and matrix mathematics). GenAI systems first break words into smaller components or subwords that can be represented as numbers. These are referred to as tokens. Tokens are then compared to other tokens to create a vector representation of semantic relationships – essentially an ordered series of numbers that depict how closely related a given token is to another. These vectors are known as embeddings. Similar tokens will have similar embeddings, thus allowing the system to determine context by applying statistical techniques to input word sequences and subsequently calculating appropriate relevant responses.

Context window refers to the number of tokens an AI model can consider at once (note that a token is smaller than a word, so the word limit will be somewhat smaller than the listed limit). It can be thought of as the memory span of the model. A larger context window allows a model to retain more information from an interaction (such as a series of prompt exchanges), providing more coherent, relevant responses. Context windows have steadily increased over time – ChatGPT’s original context window was limited to 4000 tokens, while the GPT-4o model is 128,000. Google’s latest Gemini 1.5 Pro model has a context window of 2,000,000 tokens, although this is currently limited to select developers.

While larger context windows provide significant advantages, there are nevertheless tradeoffs involved. Larger context windows require more computational resources to process requests. This can increase processing time and user costs (many GenAI services charge by token). In addition, inputting too much information (referred to as prompt stuffing) can bog down the model with extraneous details, causing the system to miss key information, resulting in less accurate results and increasing the likelihood of AI hallucination.

Retrieval Augmented Generation (RAG) is a technique that links LLMs to external knowledge bases (e.g. peer reviewed papers, company documents, Wikipedia, etc.). This improves both the efficiency and accuracy of models by allowing the LLM to search for and use relevant information as needed, rather than relying on extensive input prompts, thus reducing the need for large context windows and prompt stuffing (i.e. inputting too much information). The approach has the additional benefits of extendibility, durability (the data can be kept current), improved flexibility, and provides the ability to account for data sources and ensure factual accuracy. However, it does require greater upfront effort and complexity associated with building and managing the retrieval system.