Guide
What Is a Neural Network?
A plain-English guide to neural networks: what a neural network is, how artificial neurons, weights and layers work, how a network learns through backpropagation, the main types from CNNs to transformers, how the transformer behind ChatGPT works, and how neural networks got here.
Quick answer: A neural network is a computer model made of layers of simple connected units, called artificial neurons, that learns to do a task by adjusting the strength of the connections between them until its outputs match the examples it was trained on. Neural networks are the technology underneath chatbots such as ChatGPT, Claude, Gemini, and underneath image generators, speech recognition and face unlock. The design behind those chatbots is the transformer, a neural-network architecture that Google researchers published in June 2017. The one caveat: a neural network is loosely inspired by the brain but works very differently from one, and even its builders cannot fully explain how a large network reaches a given answer.
This page is the explainer: what a neural network is, how it learns, the main types, and how the transformer fits in. For how transformers become chatbots, see what is an LLM. For one-line definitions of the rest of the vocabulary, see the AI glossary.
What is a neural network?
A neural network is a model built from many small calculating units, arranged in layers, where each unit’s output feeds into units in the next layer and the strength of every connection is learned from data rather than written by a programmer.
In one sentence that could stand on its own: a neural network takes numbers in, passes them through layers of weighted connections, and produces numbers out, and training is the process of finding the connection weights that make those outputs useful.
The numbers can stand for almost anything. A photo is a grid of pixel brightness values. A sound clip is a sequence of air-pressure readings. A sentence is a sequence of tokens, each converted into a list of numbers. Because everything is numbers, the same basic machinery can learn to recognise cats, transcribe speech or write an email.
Neural network vs machine learning vs deep learning
These terms are nested, not interchangeable.
| Term | What it means | Relationship |
|---|---|---|
| Artificial intelligence (AI) | The whole field of building systems that perform tasks associated with human intelligence | The umbrella |
| Machine learning | AI that learns from examples instead of following hand-written rules | One approach to AI |
| Neural network | A machine learning model made of layers of connected artificial neurons | One family of machine learning models, alongside decision trees, regression and others |
| Deep learning | Machine learning with neural networks that have many layers | The modern, large-scale use of neural networks |
| Transformer | A neural-network architecture built around a mechanism called attention | The design behind large language models such as the ones in ChatGPT, Claude and Gemini |
| Large language model (LLM) | A very large transformer trained on text to predict the next token | The best-known application of transformers (LLM guide) |
Every LLM is a neural network, but most neural networks are not LLMs. The network that unlocks your phone with your face and the one that recommends your next video are neural networks too.
Is a neural network like the brain?
Only loosely. The idea came from neuroscience: in 1943, Warren McCulloch and Walter Pitts published “A Logical Calculus of the Ideas Immanent in Nervous Activity”, a mathematical model of a neuron that fires or stays silent depending on its inputs. Modern artificial neurons keep that basic idea and little else.
| Biological brain | Artificial neural network | |
|---|---|---|
| Basic unit | Neuron, a living cell | A simple function: multiply, add, apply a threshold-like curve |
| Scale | About 86 billion neurons on average, measured in adult male brains (Azevedo et al., 2009) | Learned values, called parameters, range from thousands to more than a trillion |
| How signals travel | Electrical and chemical spikes, timed | Plain numbers passed layer to layer |
| How it learns | Many mechanisms | Almost always one method: backpropagation with gradient descent |
| Energy | About 20 watts (NIST) | The 2017 transformer was trained on eight GPUs for 3.5 days (paper); larger models use far more |
The useful takeaway: “neural” describes where the idea came from, not how the system works. A neural network does not think the way a person does, and comparing parameter counts to neuron counts tells you very little.
How does a neural network work?
A neural network works by passing numbers forward through layers of artificial neurons, where each neuron multiplies its inputs by learned weights, adds them up, applies a simple non-linear function and passes the result on to the next layer.
The artificial neuron
Each artificial neuron does four small things:
- Takes inputs. A list of numbers, either the raw data or the outputs of neurons in the previous layer.
- Multiplies each input by a weight. A weight is a learned number that says how much that input matters. A large positive weight amplifies an input; a negative weight suppresses it.
- Adds them up, plus a bias. The bias is another learned number that shifts the total up or down.
- Applies an activation function. A simple curve that decides how strongly the neuron “fires”. The most common is ReLU (rectified linear unit), which passes positive numbers through unchanged and turns negative numbers into zero.
On its own, a single neuron does almost nothing useful. The power comes from wiring millions or billions of them together and tuning all their weights at once.
Why the activation function matters
Without an activation function, stacking layers would be pointless: a chain of multiply-and-add steps collapses mathematically into a single multiply-and-add step, however many layers you use. The small bend that the activation function adds is what lets a deep network represent complicated patterns. Mathematicians have shown that a network with even one hidden layer and enough neurons can approximate almost any continuous function, a result known as the universal approximation theorem (George Cybenko proved an early version in 1989). In practice, deep networks with many layers learn such functions far more efficiently than wide, shallow ones.
Layers: input, hidden and output
| Layer | What it does | Example: recognising a handwritten digit |
|---|---|---|
| Input layer | Receives the raw numbers | 784 inputs, one per pixel of a 28-by-28 image |
| Hidden layers | Transform the data step by step into more useful features | Early layers respond to strokes and edges; later layers to loops and line combinations |
| Output layer | Produces the answer | 10 outputs, one score for each digit from 0 to 9 |
“Deep” in deep learning simply means many hidden layers. Each layer builds on the one before: in an image network, early layers pick up edges, middle layers combine edges into shapes such as eyes or wheels, and late layers combine shapes into objects. Nobody programs those features in. The network discovers them because they help it get the right answer.
Parameters: what a model’s size means
A network’s weights and biases together are its parameters. When a model is described as having 7 billion or 1.6 trillion parameters, that is the number of learned values inside it.
| Network | Parameters | Source |
|---|---|---|
| AlexNet (2012 image classifier) | About 60 million | Krizhevsky, Sutskever and Hinton |
| GPT-3 (2020 language model) | 175 billion | OpenAI |
| DeepSeek-V4-Pro | 1.6 trillion total, 49 billion active per token | DeepSeek model card |
More parameters give a network more capacity to store patterns, but size alone does not decide quality. Training data, training method and architecture matter as much.
How do neural networks learn?
A neural network learns by making a prediction, measuring how wrong it was, working out how much each weight contributed to the error, and nudging every weight slightly in the direction that reduces the error, then repeating that millions of times.
The training loop, step by step
- Start random. Every weight begins as a small random number, so the untrained network’s answers are noise.
- Forward pass. Feed in a training example, such as a photo labelled “cat”, and let the numbers flow through the layers to an output.
- Measure the error. A loss function turns the gap between the network’s answer and the correct answer into a single number. Lower is better.
- Backward pass (backpropagation). Work backwards from the output layer, using calculus, to calculate how much each weight in the network contributed to the error. That set of values is called the gradient.
- Update the weights (gradient descent). Move every weight a tiny step in the direction that would have made the error smaller. The size of the step is set by the learning rate.
- Repeat. Do it again with the next batch of examples, many millions of times, until the error stops falling.
Backpropagation is the key idea. David Rumelhart, Geoffrey Hinton and Ronald Williams popularised it in a 1986 paper in Nature, and it is the standard way to train neural networks, from small image classifiers to large language models.
A simple way to picture gradient descent
Imagine standing on a hillside in thick fog, trying to reach the valley floor. You cannot see the valley, but you can feel which way the ground slopes under your feet, so you take a small step downhill and check again. Gradient descent does the same thing in a landscape with millions or billions of dimensions, one per weight. The “height” is the error, and the valley floor is a set of weights where the network makes few mistakes.
Training vs inference
| Training | Inference | |
|---|---|---|
| What happens | Weights are adjusted using backpropagation | Weights are fixed; the network just runs forward |
| How often | Once per model version | Every time someone uses the model |
| Cost | Heavy: days on eight GPUs for the 2017 transformer, and far more for larger models | Light by comparison: one forward pass per output |
When you send a prompt to a chatbot, nothing is learned from it in that moment. The network only runs forward. The LLM guide covers inference, tokens and what it costs in more detail.
Overfitting
A network with enough parameters can memorise its training examples instead of learning the general pattern, and then fail on new data. This is called overfitting. Builders guard against it by testing on data the network never saw during training, by using more varied data, and with techniques such as dropout, which randomly switches off neurons during training so the network cannot rely on any single path.
Types of neural networks
Neural networks come in several architectures, each built around the shape of the data it handles. The architecture is the wiring plan: how the layers connect and what each kind of layer does.
| Architecture | Built for | How it works, in one line | Where you meet it |
|---|---|---|---|
| Feedforward network (multilayer perceptron) | Tables of numbers, simple classification | Data flows straight through fully connected layers | Credit scoring, inside larger models as a building block |
| Convolutional neural network (CNN) | Images and video | Small filters slide across the image, detecting the same pattern wherever it appears | Face unlock, photo tagging, medical scans |
| Recurrent neural network (RNN) and LSTM | Sequences such as speech and text | Reads one step at a time, carrying a memory forward | Speech recognition and translation systems of the 2010s |
| Transformer | Text, and increasingly images, audio and video | Attention lets every part of the input look at every other part at once | ChatGPT, Claude, Gemini, translation, transcription |
| Diffusion model | Generating images, video and audio | Learns to turn random noise into an image step by step | Most AI image and video generators |
| Generative adversarial network (GAN) | Generating realistic images | Two networks compete: one generates, one judges | Earlier image generators and face-swap tools |
| State-space model (such as Mamba) | Very long sequences | Processes a sequence with a compressed running state instead of full attention | Research models and hybrid language models |
Convolutional neural networks
A convolutional neural network scans an image with small learned filters, each tuned to a pattern such as a vertical edge or a patch of texture, and stacks layers of these filters to recognise larger structures. Yann LeCun and colleagues used this approach to read handwritten digits in the late 1980s and 1990s. CNNs then drove the 2012 breakthrough: an entry built on AlexNet, a CNN with about 60 million parameters, won the ImageNet competition with a top-5 error rate of 15.3%, against 26.2% for the next-best entry. ACM said the result “almost halved the error rate for object recognition and reshaped the computer vision field” (ACM).
Recurrent networks and LSTMs
A recurrent neural network reads a sequence one element at a time and passes a hidden state, a kind of running memory, from each step to the next. Plain RNNs struggle to remember things from many steps earlier. The long short-term memory (LSTM) network, published by Sepp Hochreiter and Jürgen Schmidhuber in 1997, added a memory cell with gates that control what it stores and passes on, and LSTMs powered much of speech recognition and machine translation in the 2010s. Their weakness is that they must process a sequence in order, one step after another, which makes them slow to train on very large datasets. The transformer removed that bottleneck.
Diffusion models
A diffusion model is trained by adding random noise to images in many small steps and teaching a neural network to predict and remove that noise. To generate a new image, it starts from pure noise and removes it step by step, guided by your text prompt, until a picture appears. The neural network doing the denoising is often itself a transformer or a CNN. For the tools built on diffusion, see best AI image generators and best AI video generators.
Transformers: the architecture behind ChatGPT
A transformer is a neural-network architecture that processes a whole sequence at once and uses a mechanism called attention to let every token weigh how relevant every other token is to it. It is the “T” in GPT, which stands for generative pre-trained transformer, and it is the design behind the large language models in ChatGPT, Claude and Gemini.
Where the transformer came from
Google researchers introduced the transformer in the paper “Attention Is All You Need”, first posted on 12 June 2017. Its design was “based solely on attention mechanisms, dispensing with recurrence and convolutions”. In practice that meant a model no longer had to read a sentence word by word, as an RNN does. It could process every word in parallel, which suited the GPU chips used for training and made it practical to train on vastly more text. The paper’s largest model reached 28.4 BLEU on the WMT 2014 English-to-German translation benchmark, a new state of the art at the time, after 3.5 days of training on eight GPUs.
How attention works, in plain English
Take the sentence “The trophy did not fit in the suitcase because it was too big.” To understand “it”, a model has to connect it to “trophy”, not “suitcase”. Attention is the mechanism that makes that connection.
- Every token becomes a vector. Each word fragment is turned into a list of numbers that represents its meaning, plus information about its position in the sentence.
- Each token asks a question. The network creates three versions of each vector: a query (what this token is looking for), a key (what this token offers) and a value (the information it carries).
- Tokens score each other. Each token’s query is compared with every other token’s key. A high score means “this other token is relevant to me”. For “it”, the score against “trophy” comes out high.
- Blend the information. Each token’s new representation is a weighted mix of the other tokens’ values, weighted by those scores.
- Do it many times in parallel. A transformer runs many attention “heads” side by side, each free to track a different kind of relationship (grammar, meaning, what refers to what), and stacks dozens of these layers.
Between the attention steps, each layer also has a small feedforward network that processes each token on its own. Attention moves information between tokens; the feedforward layers transform it.
Encoders, decoders and GPT
| Transformer type | What it does | Examples |
|---|---|---|
| Encoder-only | Reads a whole input and builds a rich representation of it; good for classifying and searching text | Google’s BERT, many embedding models used for search |
| Decoder-only | Generates text one token at a time, each token attending only to the tokens before it | The GPT, Claude, Gemini, Llama and DeepSeek families |
| Encoder-decoder | Reads an input with one stack and writes an output with another | The original 2017 translation model, OpenAI’s Whisper speech recogniser |
Chatbots use decoder-only transformers. They predict the next token, add it to the text, and run again, which is why answers appear word by word. The LLM guide walks through that loop, plus tokens, context windows and training, step by step.
Why transformers took over
- Parallel training. Transformers process a sequence all at once, so training spreads across thousands of chips. RNNs could not do that.
- Long-range connections. Attention links any two tokens directly, however far apart, instead of passing information along a chain step by step.
- They keep improving with scale. Making transformers bigger and training them on more data has produced more capable models: GPT-3 in 2020 had 175 billion parameters, against about 60 million for AlexNet in 2012.
- They work beyond text. The Vision Transformer (2020) applied the same design to images by splitting a picture into small patches and treating each patch like a word. Transformers also handle images, audio, video and code, which is why one model can be multimodal.
Transformer limits, and what might come next
Attention compares every token with every other token, so its cost grows roughly with the square of the input length: doubling the text quadruples the attention work. That is the main reason long context windows are expensive to serve. Researchers have proposed alternatives. Mamba, a state-space model published by Albert Gu and Tri Dao in December 2023, scales linearly with sequence length, and its authors report “5× higher throughput than Transformers” at inference. AI21 Labs’ Jamba, released in March 2024, is a hybrid that mixes transformer and Mamba-style layers.
Neural network examples in everyday life
| Where you meet it | What the network does | Architecture |
|---|---|---|
| Chatbots and AI assistants | Answers questions, writes and writes code | Transformer (decoder-only) |
| Phone face unlock and photo search | Recognises faces and objects | CNN or vision transformer |
| Dictation and voice assistants | Turns speech into text | Transformer, earlier RNN/LSTM |
| Translation apps | Translates text and speech | Transformer |
| Image and video generators | Creates pictures and clips from a prompt | Diffusion model, often with a transformer inside |
| Streaming and shopping apps | Recommends what to watch or buy next | Feedforward and transformer-based recommenders |
| Banking | Flags payments that look fraudulent | Feedforward networks, alongside other machine learning |
| Search engines | Matches the meaning of a query to pages | Transformer encoders and embedding models |
For the products built on these networks, see best AI models for the models themselves and best AI chatbots for the apps. For neural networks you can download and run on your own hardware, see best open-source LLMs and best local LLMs, and what is open-source AI for what “open” means.
Limitations of neural networks
- The black-box problem. A large network’s knowledge is spread across billions of weights, so nobody can read off why it gave a particular answer. Interpretability research has explained parts of it: in May 2024 Anthropic reported extracting millions of “features”, patterns of neuron activity tied to concepts such as the Golden Gate Bridge, from inside its Claude 3 Sonnet model, and in March 2025 it published evidence that Claude plans rhyming words ahead when writing poetry.
- Confident mistakes. A network always produces an output, even for inputs unlike anything it was trained on. Language models are the most visible case, stating false facts fluently. See AI hallucinations.
- Data-hungry. Deep networks need large amounts of training data, and they absorb whatever biases that data contains.
- Expensive to train. Training a large network takes many specialised chips running for days or longer; using it afterwards costs far less per request.
- Brittle in odd ways. Small changes to an input, such as a few altered pixels or a reworded prompt, can change a network’s answer. In a 2023 study, formatting changes alone to a few-shot prompt swung LLaMA-2-13B’s accuracy by up to 76 points (see what is prompt engineering).
A short history of neural networks
| Year | Milestone |
|---|---|
| 1943 | Warren McCulloch and Walter Pitts publish a mathematical model of the neuron |
| 1958 | Frank Rosenblatt describes the perceptron, a single-layer network that learns from examples |
| 1969 | Marvin Minsky and Seymour Papert’s book Perceptrons shows the limits of single-layer networks, and interest in the field cools |
| 1982 | John Hopfield publishes a paper on the network now named after him, the Hopfield network, which stores and recalls patterns |
| 1986 | Rumelhart, Hinton and Williams publish their backpropagation paper in Nature, showing how to train multi-layer networks |
| 1997 | Hochreiter and Schmidhuber publish the long short-term memory (LSTM) network |
| 1998 | Yann LeCun and colleagues publish LeNet-5, a convolutional network for reading handwritten characters |
| 2012 | AlexNet wins ImageNet by a wide margin |
| 2017 | Google researchers publish the transformer |
| 2019 | Yoshua Bengio, Geoffrey Hinton and Yann LeCun receive the 2018 ACM Turing Award for their work on deep neural networks |
| 2020 | The Vision Transformer applies transformers to images |
| 2022 | OpenAI launches ChatGPT on 30 November, built on a transformer language model |
| 2023 | Gu and Dao publish Mamba, an alternative to attention for long sequences |
| 2024 | John Hopfield and Geoffrey Hinton win the Nobel Prize in Physics “for foundational discoveries and inventions that enable machine learning with artificial neural networks” |
How to learn more about neural networks
| If you want to | Start here |
|---|---|
| Understand how neural networks become chatbots | What is an LLM |
| Look up a term quickly | AI glossary |
| See how networks are used as agents that take actions | What is agentic AI |
| Take a structured course | Best AI courses |
| Run a neural network on your own computer | Best local LLMs |
| Work in the field | Machine learning jobs |
Frequently asked questions
What is a neural network in simple terms?
A neural network is a computer model made of layers of simple connected units that learns a task from examples. Each connection has a strength, called a weight, and training adjusts those weights until the network’s answers match the examples. Neural networks power chatbots, image generators, speech recognition and face unlock.
How does a neural network work?
A neural network passes numbers through layers of artificial neurons. Each neuron multiplies its inputs by learned weights, adds them up, applies a simple activation function and passes the result to the next layer. The final layer produces the answer, such as a label for an image or the next word in a sentence.
How does a neural network learn?
A neural network learns through backpropagation and gradient descent. It makes a prediction, measures how wrong it was, calculates how much each weight contributed to the error, and adjusts every weight slightly to reduce it. Repeating this over millions of examples gradually turns random weights into a useful model.
What is the difference between a neural network and deep learning?
A neural network is a type of machine learning model made of layers of connected artificial neurons. Deep learning is the use of neural networks with many layers, often dozens or more. All deep learning uses neural networks, but a small network with one or two layers is not usually called deep learning.
Are neural networks the same as AI?
Neural networks are one approach within AI, not the whole of it. AI also includes rule-based systems, search algorithms and other kinds of machine learning such as decision trees. Neural networks are the approach behind chatbots, image generators and speech recognition.
What is a transformer model?
A transformer is a neural-network architecture that processes a whole sequence at once and uses attention to let each part of the input weigh every other part. Google researchers introduced it in 2017. It is the architecture behind the large language models in ChatGPT, Claude and Gemini, and it is the “T” in GPT.
Is ChatGPT a neural network?
ChatGPT is an app built on neural networks. The models inside it are large transformer neural networks, called large language models, that generate text one token at a time. ChatGPT adds the chat interface, memory, web search and other tools around them.
What are the main types of neural networks?
The main types are feedforward networks for simple data, convolutional neural networks (CNNs) for images, recurrent networks and LSTMs for sequences, transformers for language and increasingly everything else, diffusion models for generating images and video, and generative adversarial networks (GANs).
How is a neural network different from the human brain?
A neural network borrows the idea of connected neurons from the brain but works very differently. An adult human brain has about 86 billion living neurons that communicate with timed electrical and chemical signals. An artificial neural network is a set of mathematical functions passing plain numbers between layers, and it learns almost entirely through one method, backpropagation.
Who invented neural networks?
No single person invented neural networks. Warren McCulloch and Walter Pitts published the first mathematical model of a neuron in 1943, Frank Rosenblatt described the perceptron in 1958, and David Rumelhart, Geoffrey Hinton and Ronald Williams popularised backpropagation in 1986. Hinton and John Hopfield won the 2024 Nobel Prize in Physics for foundational work on neural networks.
What are parameters in a neural network?
Parameters are the learned numbers inside a neural network, mainly the weights on its connections and the biases on its neurons. A model’s size is usually quoted in parameters: the 2012 image network AlexNet had about 60 million, OpenAI’s GPT-3 had 175 billion, and DeepSeek-V4-Pro, released in April 2026, has 1.6 trillion.