The whole pipeline on one page: from web text to chat assistant

The exercise

The intent of this project is to demonstrate, simplistically and at a high-level, the end-to-end topology of a general LLM pipeline. Such a pipeline may be considered in two phases. Phase 1 - Data curation and pre-training processing and Phase 2 - Training and inference.

Training an LLM is an expensive endeavor with large, realistic datasets. Reducing the incoming training data helps mitigate this cost. We want to train our model on high-quality data. We want the data to be “safe”, accurate, and free of duplication. We perform several stages of “filtering” to remove undesirable data, avoid prohibitively large datasets, remove personal information, constrict our data to desired language/s, and remove non-human-language content i.e. html/css/etc. code.

Once we have filtered and sanitized our dataset we manipulate the data into an encoding ingestible by a neural network. This process of “tokenization” is concerned with identifying human words and common relationships between such words. What is a human word to a machine? In a human language, a word is a sequence of single characters. To a machine a word is a sequence of characters each represented by a byte-length (8-bit) numerical value. We will consider the UTF-8 character encoding for our purposes. UTF-8 defines a symbol space with 2^8 symbols giving us 255 total possible symbols. Thus a human word, to a machine, is a sequence of these symbols. Let’s take the common word “hi” as an example. Considering the UTF-8 values of this word, we could observe how often ‘h’ appears before ‘i’ in the English language. It is very common. We could consider adding a 257th symbol that is the concatenation of ‘h’ and ‘i’. This expands our vocabulary and truncates the overall sequence length.

Consider the human word “love”. The UTF-8 sequence is 108 111 118 101. Through repeated merges, first ‘l’ and ‘o’, then ‘lo’ and ‘v’, then ‘lov’ and ‘e’, this sequence of symbols could become a single new minted token. Each merge mints one token: after the first we have 2^8+1 or 257 tokens in our possible vocabulary, and we have expanded our vocabulary space. The process of repeatedly assigning a new symbol ID to the most frequent adjacent pair is called “byte-pair encoding” (BPE). In case it’s not obvious, to expand our symbol table above the 2^8 = 255 symbols in UTF-8, we will need to choose a larger integer type to express the full range of our new vocabulary. To recap, we began with a dataset of UTF-8-character-based human words, we now have a dataset of machine tokens, a compact vocabulary, and a truncated input dataset ready for ingestion by a NN. It is worth briefly noting that some words into sub-word tokens, for instance the word helpful may become two tokens help and ful, since ful could likely be a sub-word token in another word, for instance

Before moving on, let’s pause to consider the concept of inference on a human level… “Inference is the cognitive process of drawing a logical conclusion or deduction based on evidence, reasoning, or prior knowledge”. How do we instill this ability in a machine?

If we consider the end product of an LLM (inference), we are asking for a black box that given a sequence of human words, accurately predicts and returns the next reasonably correct word in the sequence. This does not have to be human words BTW, it could be pixels, or audio slices. It’s important to illustrate that the machine doesn’t know the difference. To the machine this is a field of probability functions, tuned on numerical data - a lens through which to filter incoming data and produce a prediction for the continuation of that data, based on learned patterns in training data. Humans speak in words, LLMs speak in “tokens”. The question to the LLM is “Given an incomplete sequence of tokens, what token should appear next?” If I were to propose a well known phrase to a human and ask what comes next, for instance “To be or not to ”, the human would provide the response “be” (assuming they were trained to know Shakespeare). An LLM is a tool humans have created for asking the same question, but posed to a machine. We now understand that machines speak in tokens. Such machines, when properly trained, are statistical analysis machines that evaluate mathematical expressions to emulate the human response to such a query to a degree that appears human. So how might we accomplish this?

This brings us to “training.” An LLM is backed by a neural network: one very large function with millions, or even billions, of adjustable parameters called weights. Given a sequence of tokens, this function outputs a probability for every token in the vocabulary being the next one. Weights are randomized prior to training. If you never read Shakespeare, you couldn’t finish “To be or not to…”; our machine has to “read” Shakespeare too. The weights are its memory.

The first layer of the NN creates a vector of values giving them a point in high-dimensional space. The values are initially random and are updated at each training pass. Words that are related in language, such as love and adore will have vectors that are near each other in the vector space. Similarly, we can notice how often the word love is followed by you and encode that relationship as well. This is referred to as Attention and we will talk a lot more on this later.

Training passes update the weights by giving the NN a real piece of text, asking it to predict the next token, then comparing it to what we know to be the expected token. When the wrong token is returned we use backpropagation to calculate each weight’s contribution to the error and we use gradient descent to nudge the weight values in the direction that would make a correct answer more likely. BP and GD are beyond the scope of this writing and are complex enough to deserve their own posts which are forthcoming.

Following training we fine-tune our model by feeding it example conversations written by humans to create a more human-like experience, and to train-away potentially harmful or misleading behavior. We will save this discussion for a future post as it commands a deeper treatment of its own. This will also be a central focus of the deeper work of confronting the challenges in demanding truth from inference and combatting hallucinations.

At this point we should have a functional chat assistant, though given the toy size of our model we should keep our expectations low for producing intelligent responses. Our goal here, was to demonstrate the full pipeline in a trivial example that anyone can run on their machine.

Please check out the companion code and run each module in sequence to observe the intermediate outputs of the pre-training stages to understand each stage and notice the human-machine-human translation/conversation that occurs.

Project

Every stage of an LLM pipeline, on a demonstrative level:

1. Seed URLs. A hand-picked list of about forty pages, each labelled with whether it should survive filtering (below).
2. Fetch. Download each page politely: a user agent, a delay between requests, and robots.txt respected. Save the raw HTML.
3. Extract. Pull the main text out of each page with trafilatura, the extractor FineWeb uses.
4. Filter. Run the kept text through a datatrove pipeline on the local executor: language ID, Gopher repetition and quality filters, C4 heuristics, then MinHash deduplication, which catches exact copies and near-copies alike.
5. Score the filters. Compare what survived with the labels in the seed list. Every filter makes mistakes, and this is the first small eval of the series.
6. Tokenize. Train a small byte-level BPE tokenizer (4,096 tokens) on the surviving text.
7. Pretrain. Train a tiny GPT on it: about a million parameters, in the style of Karpathy’s nanoGPT. Forty pages is a few hundred thousand tokens at most, so it will mostly memorize. That’s expected.
8. Fine-tune for chat. Supervised fine-tuning on 500 conversations from smol-smoltalk (Apache-2.0), in a chat template, with the loss on the assistant’s replies only.
9. Chat. A small loop that takes a question and prints the model’s reply.

The seed list

Forty-one URLs in four groups, chosen so that each filter has something to catch. The full list is seed_urls.tsv in the repo.

# url <tab> group <tab> expected
https://simple.wikipedia.org/wiki/Quantum_mechanics	clean	keep
https://www.gutenberg.org/cache/epub/84/pg84.txt	book	keep
https://www.gutenberg.org/cache/epub/84/pg84-images.html	duplicate	drop
https://de.wikipedia.org/wiki/Physik	junk	drop

Run it

git clone https://github.com/dannyrrushton/llm-foundations
cd llm-foundations
./setup_venv.sh f1-mini-pipeline       # its own virtual environment, with requirements.txt installed
source f1-mini-pipeline/.venv/bin/activate
cd f1-mini-pipeline

python fetch.py seed_urls.tsv          # raw pages into data/raw/
python extract_and_filter.py           # trafilatura + datatrove, into data/clean/
python score_filters.py                # survived vs. expected, per filter
python train_tokenizer.py              # BPE, a few thousand tokens
python pretrain.py                     # a ~1M-parameter GPT on data/clean/
python sft.py                          # 500 smol-smoltalk conversations
python chat.py