From tokens to transformers: How AI processes information

Three business professionals presenting on stage at a conference with red wall backdrop and podium display
From left: Dr. Mohamed Aymen Meddeb, Dr.  Marius Vach and Dr. Jonathan Kottlors during an AI special course at neuroRAD 2026 in Mainz, Germany.
neurorad 2026

AI systems process information by breaking text into tokens (numerical representations), using embeddings to understand relationships between words, and applying transformer architecture to connect information across long documents. However, fluent AI responses don't guarantee accuracy—systems can invent information, struggle with unfamiliar data, and be misled by context, making local validation and monitoring essential for clinical use.

  • Tokenization breaks text into fragments and converts them into numerical IDs, allowing models to process unfamiliar medical terms using familiar components.
  • Embeddings represent tokens as lists of numbers, placing related terms close together on a multi-dimensional map learned from vast amounts of training text.
  • Transformer architecture uses self-attention to weigh relationships between tokens, helping AI understand context in long radiology reports and clinical histories.
  • LLMs generate responses one token at a time by calculating probabilities based on preceding context, but fluency doesn't guarantee accuracy—systems can invent references or confidently describe findings that don't exist.
  • Domain shift occurs when AI models validated in one hospital perform differently in another due to new scanners or imaging protocols, requiring local validation and ongoing monitoring.

Let's take the term "neuroradiology." For us, its meaning is immediately clear. For a large language model (LLM), however, the word may be split into fragments called tokens. Tokenization was one of the concepts explored during two AI special courses at neuroRAD 2026 in Mainz. 

Dr. Marius Vach, of University Hospital Düsseldorf in Düsseldorf, Germany, explained how transformer models work. Dr. Jonathan Kottlors, of University Hospital Cologne in Cologne, Germany, talked about clinical application scenarios for LLMs in neuroradiolog and Dr. Mohamed Aymen Meddeb, of the Institute of Neuroradiology at Charité – Universitätsmedizin Berlin in Berlin, Germany, discussed agentic AI and multimodal systems. 

The sessions looked beneath the surface of AI systems. How does a system read a report, interpret an image and eventually come up with an answer Understanding this might also help explain where AI should be used and where it should not.

How language models make sense

Since computers cannot read words as we do, they turn them into numbers they can calculate with. Tokenization can break text into smaller pieces: words, fragments and even punctuation marks. Each token gets a numerical ID.

This allows a model to process words it has never encountered before. An unfamiliar medical term can be represented using familiar fragments, although that does not guarantee the model will understand it correctly.

The numerical IDs alone do not tell the model how tokens relate to one another. Through embedding, each token is represented by a list of numbers. It is like placing tokens on a map with hundreds or thousands of dimensions, where related terms can sit close together.

For example, "hypophysis" might sit closer to other neuroanatomical terms than to "liver." These relationships are learned from enormous amounts of text during training.

Connecting the dots

The next key element is the transformer architecture, which helps the model represent relationships between tokens. Consider a report: "Several lesions are visible in the frontal lobe. They remain unchanged." What does "they" refer to?

Transformers use self-attention to weigh relationships between tokens. This helps a model connect "they" to "lesions," even when related words are separated by other text.

Within a model's context window, that makes the architecture useful for processing long radiology reports, previous examinations and complicated clinical histories. However, calculating all those relationships is computationally expensive.

Double the number of tokens, and conventional full attention requires roughly four times as many pairwise calculations. Some approaches reduce this burden by using smaller attention windows.

The guessing game

Given all those numbers, relationships and context, how does the model eventually produce an answer? One token at a time.

A generative LLM calculates probabilities for the next token based on the preceding context. A mathematical function called softmax turns its scores into probabilities. A decoding method selects the next token. Then the next and the next -- until an entire response appears.

The result might sound like an experienced neuroradiologist. Yet the same system can invent a reference, overlook a contradiction or confidently describe a finding that isn't there. Fluency does not guarantee accuracy, and the facts still need to be checked.

In his neuroRAD presentation, Kottlors framed clinical applications of LLMs as a path from research and prototypes to clinical validation and future use.

Why AI can be too agreeable

Part of the explanation for an AI assistant's behavior lies in its training. Language models undergo pretraining, learning patterns from vast amounts of text, typically by predicting tokens. Then comes post-training, which shapes how they respond to people.

One approach is reinforcement learning from human feedback (RLHF). Humans compare answers, and their preferences are used to train a reward model that guides further training.

Sometimes, this can encourage sycophancy: an assistant that reinforces the user's assumptions instead of challenging them. That becomes particularly problematic when someone asks whether a suspicious brain lesion is benign. A model agreeing with a suggested diagnosis is not the same as a second radiologist independently confirming it.

From words to images

Images can be broken into pieces, too. Vision transformers divide an image into patches, convert them into numerical representations and examine relationships between them.

Convolutional neural networks (CNNs) have been analyzing medical images for years. Their filters pick up local features: an edge, a texture or an anatomical boundary.

CNNs are particularly suited to extracting local detail, while transformers can help connect information across wider areas. Hybrid architectures combine these strengths.

Research architectures illustrate how these approaches can be combined. TransUNet combines convolutional processing with transformer mechanisms for medical image segmentation. Swin-Unet takes a transformer-based approach using shifted local attention windows.

One model, many jobs

Traditionally, an AI model might be trained to recognize one particular abnormality. Foundation models provide a more versatile starting point. Trained on broad datasets, often through self-supervised learning that does not require a human label for every example, they learn representations that can be adapted to different tasks.

A medical imaging foundation model might provide the starting point for segmentation, classification or other image-analysis tasks. Combine visual and language capabilities, and the possibilities expand further.

A multimodal model can process a scan alongside a patient's history, previous reports and clinical findings.

Why context matters

Abscess or tumor? During Meddeb's session on agentic AI and multimodal systems, an image was shown that could suggest either diagnosis. Then came additional clinical information. An inflammatory history pointed toward an abscess.

Radiologists know that context can help, but it can also create anchoring bias. AI is no exception. Instructions to disregard misleading cues can reduce their effect, but do not necessarily eliminate it. The question is how much of the model's answer comes from the image and how much from the story accompanying it.

When AI does its own thing

An ordinary chatbot answers a question. An AI agent can carry out a series of tasks on its own. Given a goal and access to appropriate tools, it can search for information, consult guidelines, evaluate results and select its next step.

Meddeb demonstrated a concept for a virtual neuro-oncology tumor board. One agent reviewed imaging information. Another handled pathology. Others considered radiotherapy or searched for relevant clinical trials.

Instead of one model doing everything, several specialized agents worked together. Preparing a tumor board can require substantial time to collect reports, review examinations and track down missing information. Agents could take over some of that preparation.

Such systems could support preparation, but clinical responsibility must remain with the multidisciplinary team. But every new agent, database connection and automated step introduces another place for something to go wrong. For AI agents, checking the final result is only part of the job. The steps in between matter, too.

Why AI can work here but fail elsewhere

Consider a model trained to detect intracranial hemorrhage. It performs well during validation. Then a hospital introduces a new CT scanner or changes its reconstruction protocol. The images may now look different to the algorithm, and its performance may fall. The model hasn't necessarily changed. Its environment has.

This is an example of domain shift. It helps explain why an AI system validated in one hospital may perform differently in another or deteriorate over time.

Even when the results are good, understanding exactly why a neural network made a particular decision remains difficult. Attention maps and other interpretability techniques can offer clues, but they do not prove that the model reached its conclusion for the right reason.

A CE mark, a promising study or a sophisticated architecture cannot replace local validation and monitoring.

The right AI for the right job

To summarize several previous MRI reports, an LLM might save time. To delineate a tumor, a specialized image-analysis model may be better suited. To combine imaging, pathology and clinical history, a multimodal system could help.

To organize a tumor board, AI agents may take over some of the legwork. But each comes with limitations. An LLM can invent information. An image model can struggle with unfamiliar data. A multimodal model can be misled by context. An agent can make mistakes along the way.

Understanding how these systems work makes it easier to judge what they can do, where they may fail and which one fits the task.

Page 1 of 130
Next Page