Lecture 22 - Large Language Models¶
22.1 Introduction to LLMs¶
Large Language Models (LLMs) are a class of Deep Neural Networks designed to understand and generate natural human language. LLMs achieved state-of-the-art performance across various NLP tasks.
LLMs are a result of many years of research and advancement in NLP and Machine Learning. Important phases in NLP development include:
Statistical language models (1980s-2000s): developed to predict the probability of a word in a text sequence based on the preceding words. Examples of statistical language models include Bag-Of-Words models based on N-grams. These models were used in tasks like speech recognition and machine translation, but struggled with capturing long-range dependencies and context-related information in text.
Neural network models (2000-2017): Fully-connected NNs and Recurrent NNs emerged as an alternative to statistical language models. Long Short-Term Memory (LSTM) RNN models were used for sequence-to-sequence tasks (such as machine translation) and they formed the basis for several early LLMs. Similar to statistical language models, RNNs struggled with capturing context-related information. Other limitations of RNNs include the inability to parallelize the data processing, and the gradients can become unstable during training.
Transformer network models (2017-present): Transformer networks introduced the self-attention mechanism as a replacement for the recurrent layers in RNNs. This architecture enabled the development of more powerful and efficient LLMs, laying the foundation for BERT, GPT, and modern LLMs.
22.1.1 Architecture of Large Language Models¶
The architecture of modern LLMs is based on Transformer Networks, which we covered in Lecture 20. The main components of the Transformer Networks architecture include:
Input embeddings, are fixed-size continuous vector embeddings that represent tokens in input text.
Positional encodings, are fixed-size continuous vectors that are added to the input embeddings to provide information about the relative positions of the tokens in the input text sequence.
Encoder, is composed of a stack of multi-head attention modules and fully-connected (feed-forward) modules. The encoder block also includes dropout layers, residual connections, and applies layer normalization.
Decoder, is composed of a stack of multi-head self-attention modules and fully-connected (feed-forward) modules similarly to the encoder block. The decoder block has an additional masked multi-head attention module, that applies masking to the next words in the text sequence to ensure that the module does not have access to those words for predicting the next token.
Output fully-connected layer, the output of the decoder is passed through a fully-connected (dense, linear) layer to produce the next token in the text sequence.
Figure: Pretraining LLMs. Source: [2].
The architecture of Transformer Networks includes multiple successive encoder and decoder blocks to create deep networks with many layers that allow learning complex patterns in input text. For example, the original Transformer Network has 6 encoder and 6 decoder blocks, as shown in the above figure.
The self-attention mechanism is a key component of the Transformer Network architecture that enables the model to weigh the importance of each token with respect to the other tokens in a sequence. It allows to capture long-range dependencies and relationships between the tokens (words) and helps the model to understand the context and structure of the input text sequence.
22.1.2 Variants of Transformer Network Architectures¶
Various LLMs have been built on top of the Transformer Network architecture. The popular variants include:
Decoder-only models: are autoregressive models that utilize only the decoder part of the Transformer Network architecture. These models are particularly suitable for generating text and content. An example of decoder-only LLMs is the family of GPT models.
Encoder-only models: use only the encoder part of the Transformer Network architecture, and perform well on tasks related to language understanding, such as classification and sentiment analysis. An example is the BERT model.
Encoder-decoder models: employ the original Transformer Network architecture and combine encoder and decoder sub-networks, enabling to both understand language and generate content. These models can be used for various NLP tasks with minimal task-specific modifications. An example of this class of models is T5 (Text-to-Text Transfer Transformer).
List of LLMs¶
A large number of LLMs have been developed in the past several years. Some of the most well-known LLMs include:
GPT (Generative Pretrained Transformers): Developed by OpenAI, the GPT family are the best-known LLMs. They include GPT 1, 2, 3, 3.5 (initial ChatGPT), 4, 4o (current ChatGPT), and o1 (where o stands for omni, meaning that the model can process multi-modal inputs, including text, images, video, audio, etc.). According to some sources, GPT-4 has 1.76 trillion parameters, and it is trained on 13T tokens.
LlaMA (Large Language Model Meta AI): Developed by Meta AI, LlaMA is an open-source LLM, which can be used for both research and commercial uses. It consists of several models including LlaMA base model, LlaMA-Chat, and Code-LlaMA. Released versions include LlaMA 2, LlaMA 3, LLaMA 3.1, and LlaMA 3.2. The latest LlaMA 3.2 includes smaller test models with 1B and 3B parameters, and multi-modal 11B and 90B parameters, trained on 9T tokens.
Claude: Developed by Anthropic, the latest version Claude 3 has three models named Haiku, Sonnet, and Opus. These models rank very high on the benchmarking leaderboards for many tasks, and they are currently the main competitor to OpenAI’s GPT models.
Gemini: Developed by Google, offers four models named Nano, Flash, Pro, and Ultra. The number of parameters is not known. The smaller models are designed for smartphones, whereas the larger models are multimodal and can process images, video, code, and other inputs, beside text.
Mixtral: Developed by Mistral, these LLM use mixture-of-experts (MOE) architecture, which allows them to be competitive with larger models, despite having fewer parameters. Current models have 8 mixture-of-experts with 7B and 22B parameters.
Grok: Developed by xAI, Grok is trained on data from X (formerly Twitter) and has 314B parameters. It also uses a mixture-of-experts (MOE) architecture.
BERT (Bidirectional Encoder Representations from Transformers): Developed by Google in 2018, BERT is an early LLM with 340M parameters that can understand natural language and answer questions.
Cohere LLM: Developed by Cohere, it is a family of LLMs with 6B, 13B, and 52B parameters, designed for enterprise use cases.
Vicuna: Developed by LMSYS, Vicuna is a 13B parameters chat assistant finetuned from LLaMA on user-shared conversations.
Alpaca: Developed by Stanford, it is a 7B LLM finetuned from instruction-following samples by LLaMA.
Falcon: Developed by UAE’s Technology Innovation Institute (TII), it is an open-source family of models with 1.3B, 7.5B, 40B, and 180B parameters, trained on 3.5T tokens.
DBRX and Dolly: Developed by Databricks, DBRX has 132B parameters, whereas Dolly is a smaller LLM language model with 12B parameters.
22.2 Creating LLMs¶
Creating modern LLMs typically involves three main phases:
Pretraining, the model extracts knowledge from large unlabeled text datasets.
Supervised finetuning, the model is refined to improve the quality of generated responses.
Alignment, the model is further refined to generate safe and helpful responses that are aligned with human preferences.
22.2.1 Pretraining¶
The first step in creating LLMs is pretraining the model on massive amounts of text data. The datasets usually consist of a large collection of web pages or e-books comprising billions or trillions of tokens, and ranging from gigabytes to terabytes of text. During pretraining, the model learns the structure of the language, grammar rules, facts about the world, and reasoning rules. And, it also learns biases and harmful content present in the training data.
Pretraining is performed using unsupervised learning techniques. Two common approaches for pretraining LLMs are:
Causal Language Modeling, also known as autoregressive language modeling, involves training the model to predict the next token in the text sequence given the previous tokens. This approach is more common with modern LLMs.
Masked Language Modeling, where a certain percentage of the input tokens are randomly masked, and the model is trained to predict the masked tokens based on the surrounding context. BERT and earlier LLMs were pretrained with masked language modeling.
The following figure depicts the pretraining phase with Causal Language Modeling, where the model learns to predict the next word in a sentence given the previous words.
Figure: Pretraining LLMs. Source: [3].
Pretraining allows to extract knowledge from very large unlabeled datasets in unsupervised learning manner, without the need for manual labeling. Or, to be more precise, the “label” in LLMs pretraining is the next word in the text, to which we already have access since it is part of the training text. Such pretraining approach is also called self-supervised training, since the model uses each next word in the text to self-supervise the training.
Note that pretraining LLMs from scratch is computationally expensive and time-consuming. As we stated before, the pretraining phase can cost millions of dollars (e.g., the estimated cost for training GPT-4 is $100 million). Also, pretraining LLMs requires access to large datasets and technical expertise with strong understanding of deep learning workflows, working with distributed software and hardware, and managing model training with thousands of GPUs simultaneously.
22.2.2 Supervised Finetuning¶
After the pretraining phase, the model is finetuned on a much smaller dataset, which is carefully generated with human supervision. This dataset consists of samples where AI trainers provide both queries (instructions) and model responses (outputs), as depicted in the following figure. That is, instruction is the input text given to the model, and output is the desired response by the model. The model takes the instruction text as input (e.g., “Write a limerick about a pelican”) and uses next-token prediction to generate the output text (e.g., “There once was a pelican so fine …”).
The finetuning process involves updating the model’s weights using supervised learning techniques. The objective of supervised finetuning is to improve the quality of the generated responses by the pretrained LLM.
To compile datasets for supervised finetuning, AI trainers need to write the desired instructions and responses, which is a laborious process. Typical datasets include between 1K and 100K instruction-output pairs. Based on the provided instruction-output pairs, the model is finetuned to generate responses that are similar to those provided by AI trainers.
Figure: Finetuning a pretrained LLM. Source: [3].
22.2.3 Alignment¶
To further improve the performance and align the model responses with human preferences, LLMs are typically refined in one additional phase. This ensures that the responses generated by LLMs are aligned with human preferences, making the models more useful and safer for interaction with users. The alignment phase is essential for reducing harmful, biased, or otherwise undesirable outputs.
Two main strategies for LLM alignment include Reinforcement Learning from Human Feedback (RLHF) with Proximal Policy Optimization (PPO) and Reinforcement Learning with Direct Policy Optimization (DPO).
Reinforcement Learning from Human Feedback (RLHF) with Proximal Policy Optimization (PPO)
LLM alignment with Reinforcement Learning from Human Feedback (RLHF) by employing Proximal Policy Optimization (PPO) is depicted in the figure below and involves the following steps:
Collect human feedback. For this step a new dataset is created by collecting sample prompts from a database or by creating a set of new prompts. For each prompt, multiple responses are generated by the supervised finetuned model. Next, AI trainers are asked to rank by quality all responses generated by the model for the same prompt, from best to worst. Such feedback is used to define the human preferences and expectations about the responses by the model. Although this ranking process is time-consuming, it is usually less labor-intensive than creating the dataset for supervised finetuning, since ranking the responses is faster than writing the responses.
Create a reward model. The collected data with human feedback containing the prompts and the ranking scores of the different responses are used to train a Reward Model (denoted with RM in the figure). The task for the Reward Model is to predict the quality of the different responses to a given prompt and output a ranking score. The ranking scores provided by AI trainers are used to establish the ground-truth for training the Reward Model. Note that the Reward Model is a different model than the LLM that is being finetuned, and it only needs to rank the generated responses by the LLM.
Finetune the LLM with RL. The LLM is finetuned using the Reinforcement Learning (RL) algorithm Proximal Policy Optimization (PPO). For a new prompt, the original LLM generates a response, which the Reward Model evaluates and calculates a reward score \(r_k\). Next, the PPO algorithm uses the reward score \(r_k\) to finetune the LLM so that the total rewards for the generated responses by the LLM are maximized. I.e., the goal is to generate responses by the LLM that maximize the predicted reward scores, and by that, the responses become more aligned with human preferences and are more useful to human users.
Iterative improvement. The RLHF process is performed iteratively, with multiple rounds of collecting additional feedback from human labelers, re-training the Reward Model, and applying Reinforcement Learning. This leads to continuous refinement and improvement of the LLM’s performance.
Figure: Reinforcement Learning from Human Feedback. Source: [4].
In summary, the RLHF approach creates a reward system that is augmented by human feedback and is used to teach LLMs which responses are more aligned with human preferences. Through these iterations, LLMs can be better aligned with our human values and can lead to higher-quality responses, as well as improved performance on specific tasks.
Note also that there are several variants of the RLFH approach for finetuning LLMs. For example, LlaMA models employ two reward models: one based on the ranks of helpfulness of the responses, and another based on the ranks of safety of the responses. The final reward score is obtained as a combination of the helpfulness and safety scores.
Reinforcement Learning with Direct Policy Optimization (DPO)
RL with Direct Policy Optimization (DPO) is another approach for LLM alignment that has been popular recently, as it is simpler than RLHF with PPO. DPO uses a different optimization approach in comparison to RL with PPO, where DPO optimizes the LLM directly based on user preferences, without the need for training a separate Reward Model. I.e., DPO aims to directly maximize the reward function to produce model outputs that align with human preferences. Detailed explanation of RL with DPO is beyond the scope of this lecture.
22.3 Finetuning LLMs¶
Finetuning LLMs involves updating the weights of an LLM model on new data to improve its performance on a specific task and make the model more suitable for a specific use case. It involves additional re-training of the model on a new dataset that is specific to that task. That is, finetuning is a transfer learning technique, where the gained knowledge by a trained model is transferred to improve the performance on a target task.
To adapt LLMs to a custom task, different finetuning techniques have been applied. Full model finetuning is a method that finetunes all the parameters of all the layers of a pretrained model. Full model finetuning typically can achieve the best performance, but it is also the most resource-intensive and time-consuming. Performance-efficient finetuning involves updating only a small number of the parameters to reduce the required computational resources and costs.
In this section, we will demonstrate how to finetune LlaMA 2, an open-source LLM developed by Meta AI. Released in July 2023, LlaMA 2 was the first LLM that is open for both research and commercial use. LlaMA 2 is a successor model to the original LlaMA developed by Meta AI as well. LlaMA 2 has three variants with 7B, 13B, and 70B parameters. It has been trained on 2 trillion tokens, and it has a context window of 4,096 tokens enabling to process large documents. For instance, for the task of summarizing a pdf document the context can include the entire text of the pdf document, or for dialog with a chatbot the context can include the previous conversation history with the chatbot. Furthermore, specialized versions of LlaMA 2 include LlaMA-2-Chat optimized for dialog generation, and Code LlaMA optimized for code generation tasks.
22.3.1 Parameter-Efficient Finetuning (PEFT)¶
Finetuning LLMs is challenging since the large number of parameters of modern LLMs requires substantial computational resources for storing the models and for re-training the weights. Thus, it can be prohibitively expensive for most users. For instance, to load the largest version of the LlaMA 2 model with 70 billion parameters into the GPU memory requires approximately 280 GB of RAM. Full model finetuning of LlaMA 2 model with 70 billion parameters requires 780 GB of GPU memory. This is equivalent to 10 A100s GPUs that have 80 GB RAM each, or 48 T4 GPUs that have 16 GB RAM each. The free version of Google Colab offers one T4 GPU with 16 GB RAM.
Fortunately, several Parameter-Efficient FineTuning (PEFT) techniques have been introduced recently, which allow updating only a small number of the model weights. Consequently, these techniques enable finetuning LLMs using lower computational resources by reducing memory usage and speeding up the training process. PEFT techniques include prompt tuning, prefix tuning, adding additional adapter layers in the transformer block, and low-rank adaptation (LoRA).
Hugging Face has developed a PEFT library that contains implementations of common finetuning techniques. We will use the PEFT library to finetune LlaMA 2 on a custom dataset using a quantized version of the LoRA method.
22.3.2 Low-Rank Adaptation (LoRA)¶
Low-Rank Adaptation (LoRA) involves freezing the pretrained model and finetuning a small number of additional weights. After the additional weights are updated, these weights are merged with the weights of the original model.
This is depicted in the following figure, where regular finetuning is shown in the left figure, and it involves updating all weights \(W\) in a pretrained model. As we know, the weight update matrix \(\nabla{W}\) is calculated based on the negative gradient of the loss function. Finetuning with LoRA is shown in the right figure, where the weight update matrix \(\nabla{W}\) is decomposed into two smaller matrices, \(\nabla{W}=W_A*W_B\), with size \(W_A \in \mathbb{R}^{A \times r}\) and \(W_B \in \mathbb{R}^{r \times B}\). The matrices \(W_A\) and \(W_B\) are called low-rank adapters, since they have lower rank \(r\) in comparison to the original weight matrix, i.e., they have fewer number of columns or rows, respectively. During training, gradients are backpropagated only through the matrices \(W_A\) and \(W_B\), while the pretrained weights \(W\) remain frozen.
For instance, if the full weight matrix \(W\) is of size \(100 \times 100\), this is equal to \(10,000\) elements (model weights). If we decompose the weight update matrix \(\nabla{W}\) by using rank \(r=5\), the total number of elements of \(W_A \in \mathbb{R}^{100 \times 5}\) and \(W_B \in \mathbb{R}^{5 \times 100}\) will be \(500 + 500 = 1,000\). Hence, with LoRA the number of elements was reduced from \(10,000\) to \(1,000\).
Figure: Regular finetuning versus LoRA finetuning . Source: [5].
22.3.3 Quanitized LoRA (QLoRA)¶
Quanitized LoRA (QLoRA) is a modified version of LoRA that uses 4-bit quantized weights. Quantization reduces the precision for the values of the network weights. In TensorFlow and PyTorch, the network weights by default are stored with 32-bit floating-point precision. With quantization techniques, the network weights are stored with lower precision, such as 16-bit, 8-bit, or 4-bit precision.
This approach introduces a new 4-bit quantization format called “nf4” (normalized float 4) where the range of values is normalized to the range [-1, 1] by dividing the values evenly into 16 bins (4-bit allows \(2^4=16\) values). While 4-bit floating point precision (fp4) applies non-linear floating point representation of the original values and results in unequal spacing of the values, normalized float 4 precision (nf4) applies linear quantization of the original values into equally spaced bins and follows a normal distribution.
QLoRA combines 4-bit quantization of the model weights in the pretrained model and LoRA that adds low-rank adaptor layers. The benefits of QLoRA with 4-bit quantization of the model weights include reduced size of the model and increased inference speed, while having a modest decrease in the overall model performance.
For example, with QLoRA a 70B parameter model can be finetuned with 48 GB VRAM, in comparison to 780 GB VRAM required for finetuning all weights of the original model (using 32-bit floating-point precision). Similarly, QLoRA enables to train the smaller version of LlaMA 2 with 7B parameters on a T4 GPU (provided by Google Colab) that has 16 GB VRAM. In cases when only a single GPU is available, using quantization is necessary for finetuning LLMs.
22.3.4 Finetuning Example: Finetuning LlaMA-2 7B¶
Import Libraries¶
We will begin by installing the required libraries and importing modules from these packages. These include accelerate (for optimized training on GPUs), peft (for Parameter-Efficient Fine-Tuning), bitsandbytes (to quantize the LlaMA model to 4-bit precision), transformers (for working with Transformer Networks), and trl (for supervised finetuning, where trl stands for Transformer Reinforcement Learning).
[ ]:
!pip install -q accelerate==0.21.0 peft==0.4.0 bitsandbytes==0.40.2 transformers==4.31.0 trl==0.4.7
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 116.9/116.9 kB 6.5 MB/s eta 0:00:00
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 244.2/244.2 kB 16.5 MB/s eta 0:00:00
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 72.9/72.9 kB 5.3 MB/s eta 0:00:00
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 92.5/92.5 MB 22.2 MB/s eta 0:00:00
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.4/7.4 MB 57.9 MB/s eta 0:00:00
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 77.4/77.4 kB 6.0 MB/s eta 0:00:00
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.8/7.8 MB 78.0 MB/s eta 0:00:00
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 480.6/480.6 kB 30.1 MB/s eta 0:00:00
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 116.3/116.3 kB 9.2 MB/s eta 0:00:00
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 179.3/179.3 kB 15.3 MB/s eta 0:00:00
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 134.8/134.8 kB 10.4 MB/s eta 0:00:00
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 194.1/194.1 kB 13.4 MB/s eta 0:00:00
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
gcsfs 2024.10.0 requires fsspec==2024.10.0, but you have fsspec 2024.9.0 which is incompatible.
sentence-transformers 3.2.1 requires transformers<5.0.0,>=4.41.0, but you have transformers 4.31.0 which is incompatible.
[ ]:
import os
import torch
from datasets import load_dataset
from transformers import (AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig,
HfArgumentParser, TrainingArguments, pipeline, logging)
from peft import LoraConfig, PeftModel, get_peft_model
from trl import SFTTrainer
Load the Model¶
We will download the smallest version of LlaMA-2-Chat model with 7B parameters from Hugging Face. Understandably, the larger LlaMA 2 models with 13B and 70B parameters require larger memory and computational resources for finetuning.
Also, we will use the BitsAndBytes library to apply quantization with 4-bit precision format for loading the model weights. Loading a quantized model reduces the GPU memory requirement and makes it possible to train the model with a single GPU, as a tradeoff for some loss in precision. In the next cell we define the configuration for BitsAndBytes, and afterward we will use the configuration in the from_pretrained function to load the LlaMA 2 model. The parameters in BitsAndBytes
configuration are described in the commented code below.
The compute type in the cell below refers to the data format for performing computations, and it can be either “float16”, “bfloat16”, or “float32” because computations are performed in either 16 or 32-bit precision. In this case, we specified to use "torch.float16" compute data type (i.e., 16-bit floating-point numbers) for memory-saving purposes. Note that although the model weights are loaded with 4-bit precision, the weights are dequantized to 16-bit precision for performing the
calculations for the forward and backward passes through the network, since 4-bit precision is too low for performing the calculations.
[ ]:
# The model is Llama 2 from the Hugging Face hub
model_name = "NousResearch/Llama-2-7b-chat-hf"
[ ]:
# BitsAndBytes configuraton
bnb_config = BitsAndBytesConfig(
# Load the model using 4-bit precision
load_in_4bit=True,
# Quantization type (fp4 or nf4)
# nf4 is "normalized float 4" format, uses a symmetric quantization scheme with 4-bit precision
bnb_4bit_quant_type="nf4",
# Compute dtype for 4-bit models
bnb_4bit_compute_dtype= torch.float16,
# Use double quantization for 4-bit models
# Double quantization applies further quantization to the quantization constants
bnb_4bit_use_double_quant=False,
)
We will use AutoModelForCausalLM to load the model with the from_pretrained function, and we will use the above BitesAndBytes configuration to load the model parameters with 4-bit precision.
In the following cell we will load the corresponding tokenizer for LlaMA 2 by using AutoTokenizer and from_pretrained.
[ ]:
# Load Llama 2 model from Hugging Face
model = AutoModelForCausalLM.from_pretrained(
model_name,
# Apply quantization by using the bnb configuration from the previous cell
quantization_config=bnb_config,
# Don't cache the model weights, load the model weights from Hugging Face
use_cache=False,
# Trade-off parameter in Llama-2, less important, it should be 1 in most cases
pretraining_tp=1,
# Load the entire model on the GPU if available
device_map="auto"
)
/usr/local/lib/python3.10/dist-packages/huggingface_hub/file_download.py:797: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
warnings.warn(
[ ]:
# Load tokenizer from Hugging Face
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
# Needed for LLaMA tokenizer
tokenizer.pad_token = tokenizer.eos_token
# Fix an overflow issue with fp16 training
tokenizer.padding_side = "right"
Define LoRA Configuration¶
Next, the model will be packed into the LoRA format, which will introduce additional weights and keep the original weights frozen. The parameters in the LoRA configuration include:
r, determines the rank of update matrices, where lower rank results in smaller update matrices with fewer trainable parameters, and greater rank results in more trainable parameters but more robust model.lora_alpha, controls the LoRA scaling factor.lora_dropout, is the dropout rate for LoRA layers.bias, specifies if the bias parameters should be trained.task_type, is Causal LLM for the considered task.
[ ]:
# LoRA configuration
peft_config = LoraConfig(
# LoRA rank dimension
r=64,
# Alpha parameter for LoRA scaling
lora_alpha=16,
# Dropout rate for LoRA layers
lora_dropout=0.1,
bias="none",
task_type="CAUSAL_LM",
)
In order to understand how LoRA impacts the finetuning of LlaMA 2 model, let’s compare the total number of trainable parameters in LLaMA 2 and the trainable parameters for the LoRA model. As we can note in the cell below, the LoRA model has about 67M trainable parameters, which is about 1% of the 7B total trainable parameters in LlaMA 2. This makes it possible to finetune the model on a single GPU.
[ ]:
def print_number_of_trainable_model_parameters(model, use_4bit=True):
trainable_model_params = 0
all_model_params = 0
for _, param in model.named_parameters():
all_model_params += param.numel()
if param.requires_grad:
trainable_model_params += param.numel()
if use_4bit:
all_model_params *= 2
trainable_model_params *= 2
print(f"Total model parameters: {all_model_params:,d}. Trainable model parameters: {trainable_model_params:,d}. Percent of trainable parameters: {100 * trainable_model_params/ all_model_params:4.2f} %")
[ ]:
# compare the number of trainable parameters to QLoRA model
qlora_model = get_peft_model(model, peft_config)
# print trainable parameters
print_number_of_trainable_model_parameters(qlora_model)
Total model parameters: 7,067,934,720. Trainable model parameters: 67,108,864. Percent of trainable parameters: 0.95 %
Load the Dataset¶
We will use the Lamini docs dataset, which contains questions and answers about the framework Lamini for training and developing Language Models. The dataset contains 1,260 question/answer pairs. Here are a few samples from the dataset.
Question |
Answer |
|---|---|
Does Lamini support generating code |
Yes, Lamini supports generating code through its API. |
How do I report a bug or issue with the Lamini documentation? |
You can report a bug or issue with the Lamini documentation by submitting an issue on the Lamini GitHub page. |
Can Lamini be used in an online learning setting, where the model is updated continuously as new data becomes available? |
It is possible to use Lamini in an online learning setting where the model is updated continuously as new data becomes available. However, this would require some additional implementation and configuration to ensure that the model is updated appropriately and efficiently. |
A preprocessed version of the dataset in a format that matches the instruction-output pairs for LlaMA 2 is available on Hugging Face, and we will directly load the preprocessed version of the dataset.
[ ]:
# Lamini dataset
dataset = load_dataset("mwitiderrick/llamini_llama", split="train")
[ ]:
print(f'Number of prompts: {len(dataset)}')
Number of prompts: 1260
Model Training¶
The next cell defines the training arguments, and the commented notes describe the arguments. Note that we will finetune the model for only 1 epoch (if we finetune for more than 1 epoch it will take longer but it will probably result in improved performance).
[ ]:
# Set training parameters
training_arguments = TrainingArguments(
# Output directory where the model predictions and checkpoints will be stored
output_dir="./results",
# Number of training epochs
num_train_epochs=1,
# Batch size per GPU for training
per_device_train_batch_size=8,
# Number of update steps to accumulate the gradients for
gradient_accumulation_steps=2,
# Optimizer to use
optim="paged_adamw_32bit",
# Save checkpoint every number of steps
save_steps=0,
# Log updates every number of steps
logging_steps=10,
# Initial learning rate (AdamW optimizer)
learning_rate=2e-4,
# Weight decay to apply
weight_decay=0.001,
# Enable fp16/bf16 training (set bf16 to True with an A100)
fp16=False,
bf16=False,
# Maximum gradient normal (gradient clipping)
max_grad_norm=0.3,
# Group sequences with same length into batches (to minimize padding)
# Saves memory and speeds up training considerably
group_by_length=True,
# Learning rate schedule
lr_scheduler_type="constant",
# Disable reporting to external tools (e.g., WandB, TensorBoard)
report_to="none"
)
Next, we will use the SFTTrainer class in Hugging Face to create an instance of the model by passing the loaded LlaMA 2 model, training dataset, PeFT configuration, tokenizer, and the training arguments. SFTTrainer stands for Supervised Fine-Tuning Trainer.
[ ]:
# Set supervised finetuning parameters
trainer = SFTTrainer(
model=model,
train_dataset=dataset,
peft_config=peft_config,
tokenizer=tokenizer,
args=training_arguments,
# Column in the dataset that contains the data
dataset_text_field="text",
# Maximum sequence length to use
max_seq_length=None,
# Pack multiple short examples in the same input sequence to increase efficiency
packing=False,
)
/usr/local/lib/python3.10/dist-packages/peft/utils/other.py:102: FutureWarning: prepare_model_for_int8_training is deprecated and will be removed in a future version. Use prepare_model_for_kbit_training instead.
warnings.warn(
/usr/local/lib/python3.10/dist-packages/trl/trainer/sft_trainer.py:159: UserWarning: You didn't pass a `max_seq_length` argument to the SFTTrainer, this will default to 1024
warnings.warn(
Finally, we can train the model with the train() function in Hugging Face. In the output of the cell we can see the loss for every 10 training steps, because we set logging_steps=10 in the training arguments.
The training took about 15 minutes on a T4 GPU with High-RAM memory on Google Clab Pro.
[ ]:
# Train the model
trainer.train()
You're using a LlamaTokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.
/usr/local/lib/python3.10/dist-packages/torch/_dynamo/eval_frame.py:632: UserWarning: torch.utils.checkpoint: the use_reentrant parameter should be passed explicitly. In version 2.5 we will raise an exception if use_reentrant is not passed. use_reentrant=False is recommended, but if you need to preserve the current default behavior, you can pass use_reentrant=True. Refer to docs for more details on the differences between the two variants.
return fn(*args, **kwargs)
| Step | Training Loss |
|---|---|
| 10 | 2.648700 |
| 20 | 1.850300 |
| 30 | 0.876900 |
| 40 | 0.658500 |
| 50 | 0.609400 |
| 60 | 0.532200 |
| 70 | 0.581600 |
TrainOutput(global_step=79, training_loss=1.0302405297001707, metrics={'train_runtime': 885.2212, 'train_samples_per_second': 1.423, 'train_steps_per_second': 0.089, 'total_flos': 5700938654515200.0, 'train_loss': 1.0302405297001707, 'epoch': 1.0})
Generate Text¶
To generate text with the trained model we will use the Hugging Face pipeline with the task set to "text-generation". We can set the length of the generated text tokens with the max_length argument.
The output displays the start <s>[INST] and end [/INST] of the instruction prompt, followed by the generated output by the model.
[ ]:
# Run text generation pipeline with the finetuned model
prompt = "What are Lamini models?"
pipe = pipeline(task="text-generation", model=model, tokenizer=tokenizer, max_length=200)
output = pipe(f"<s>[INST] {prompt} [/INST]")
print(output[0]['generated_text'])
Xformers is not installed correctly. If you want to use memory_efficient_attention to accelerate training use the following command to install Xformers
pip install xformers.
/usr/local/lib/python3.10/dist-packages/transformers/generation/utils.py:1270: UserWarning: You have modified the pretrained model configuration to control generation. This is a deprecated strategy to control generation and will be removed soon, in a future version. Please use a generation configuration file (see https://huggingface.co/docs/transformers/main_classes/text_generation )
warnings.warn(
/usr/local/lib/python3.10/dist-packages/torch/utils/checkpoint.py:87: UserWarning: None of the inputs have requires_grad=True. Gradients will be None
warnings.warn(
<s>[INST] What are Lamini models? [/INST] Lamini is an open-source library for building and training LLMs (Large Language Models) in Python. everybody can use Lamini to build and train LLMs. Lamini models are trained on large datasets of text and can be used for a variety of natural language processing tasks, such as language translation, text summarization, and chatbots.
Lamini models are based on the transformer architecture, which is a type of neural network that is particularly well-suited for natural language processing tasks. The transformer architecture uses self-attention mechanisms to allow the model to consider the entire input sequence when generating each output element, rather than just relying on a fixed-length context window or recurrence. This allows the model to capture long-range dependencies in the input sequence and generate more coherent and contextually appropriate output.
Lamini models are
[ ]:
# Run text generation pipeline with the finetuned model
prompt = "How to evaluate the quality of the generated text with Lamini models"
pipe = pipeline(task="text-generation", model=model, tokenizer=tokenizer, max_length=500)
output = pipe(f"<s>[INST] {prompt} [/INST]")
print(output[0]['generated_text'])
<s>[INST] How to evaluate the quality of the generated text with Lamini models [/INST] Lamini is a Python library for generating text with LLMs. everybody can use Lamini to generate text with LLMs. However, the quality of the generated text can vary depending on the model and the input prompt. Here are some ways to evaluate the quality of the generated text with Lamini models:
1. Perplexity: Perplexity measures how well the generated text fits the input prompt. A lower perplexity score indicates a better fit. You can calculate perplexity using the `perplexity` function in Lamini.
2. BLEU score: BLEU (Bilingual Evaluation Understudy) is a metric that measures the quality of machine-generated text. It compares the generated text to a reference translation. You can use the `bleu` function in Lamini to calculate the BLEU score.
3. ROUGE score: ROUGE (Recall-Oriented Understudy for Gisting Evaluation) is another metric that measures the quality of machine-generated text. It compares the generated text to a reference translation. You can use the `rouge` function in Lamini to calculate the ROUGE score.
4. Fluency: Fluency measures how natural and coherent the generated text is. You can evaluate the fluency of the generated text by reading it aloud or by asking a human evaluator to assess its fluency.
5. Relevance: Relevance measures how well the generated text answers the input prompt. You can evaluate the relevance of the generated text by checking if it contains the requested information or if it is relevant to the topic.
6. Length: Length measures how long the generated text is. You can evaluate the length of the generated text by checking its length against the desired length.
7. Sentiment analysis: Sentiment analysis measures the emotional tone of the generated text. You can evaluate the sentiment of the generated text by using a sentiment analysis tool or by asking a human evaluator to assess its sentiment.
8. Named entity recognition: Named entity recognition measures the accuracy of the generated text in identifying and mentioning named entities such as people, places, and organizations. You can evaluate the named entity recognition of the generated text by
[ ]:
# Run text generation pipeline with the finetuned model
prompt = "Write a poem about Data Science"
pipe = pipeline(task="text-generation", model=model, tokenizer=tokenizer, max_length=500)
output = pipe(f"<s>[INST] {prompt} [/INST]")
print(output[0]['generated_text'])
<s>[INST] Write a poem about Data Science [/INST] In the realm of code and math,
everybody's talking about data science.
A field of wonder, a world of facts,
Where insights are found, and secrets are cracked.
With algorithms and models, we dive deep,
Into the ocean of data, where truths are kept.
We analyze and visualize, with skill and grace,
The patterns and trends that shape our world's embrace.
From predicting the future to understanding the past,
Data science is the key to unlocking the vast.
With every line of code, we make progress,
And bring the world closer to perfection.
So let us embrace this field of wonder,
And unlock the secrets of the data thunder.
For in the realm of data science, we find,
The answers to questions that have long been blind.
With every discovery, we make a new start,
And bring the world closer to a brighter heart.
For data science is the key to unlocking the truth,
And bringing the world closer to a brighter youth.
So let us celebrate this field of art,
And the beauty that it brings to every part.
For in the realm of data science, we find,
A world of wonder, a world of mind.
22.3.5 Retrieval Augmented Generation (RAG)¶
Retrieval Augmented Generation (RAG) refers to using external sources of information for improving the quality of generated responses by LLMs. RAG enables LLMs to retrieve facts from external sources (such as Wikipedia, news articles) and provide responses that are more accurate and/or are up-to-date.
In general, the internal knowledge of LLMs is static, as it is fixed by the date of the used training dataset. Therefore, LLMs cannot answer questions about current events, and they are stuck in the time moment of their training data. Updating LLMs with knowledge about current events requires to continuously retrain the models on new data. Such a process is very expensive, as it requires collecting updated datasets and finetuning the model to update the weights.
RAG enables to avoid expensive LLMs retraining, by retrieving information from updated external databases to generate responses. The RAG approach involves two phases: retrieval and content generation. The retrieval phase includes performing relevancy search of external databases regarding a user query, and retrieving supporting documents and snippets of important information. Afteward, in the content generation phases, these supporting documents are used as a context that is appended to the user query, and are fed to the LLMs for generating the final response.
Instead of relying only on the information contained in the training dataset used for training an LLM, RAG provides an interface to external knowledge to ensure that the model has access to the most current and reliable facts. E.g., in enterprise setting, external sources of information for RAG can comprise of various company-specific files, documents, and databases. Employing RAG can result in more relevant responses and it can reduce the problem of hallucination by LLMs. It also allows the users to review the sources that were used by LLMs and verify the accuracy of generated responses.
22.3.6 Prompt Engineering¶
Prompt engineering is a technique for improving the performance of LLMs by providing detailed context and information about a specific task. It involves creating text prompts that provide additional information or guidance to the model, such as the topic of the generated response. With prompt engineering, the model can better understand the kind of expected output and produce more accurate and relevant results.
The following tips for creating effective prompts as part of prompt engineering can improve the performance of LLMs:
Use clear and concise prompts: The prompt should be easy to understand and provide enough information for the model to generate relevant output. Avoid using jargon or technical terms.
Use specific examples: Providing specific examples can help the model better understand the expected output. For example, if you want the model to generate a story about a particular topic, include a few sentences about the setting, characters, and plot.
Vary the prompts: Use prompts with different styles, tones, and formats to obtain more diverse outputs from the model.
Test and refine: Test the prompts on the model and refine them by adding more detail or adjusting the tone and style.
Use feedback: Use feedback from users or other sources to identify areas where the model needs more guidance and make adjustments accordingly.
Chain-of-thought technique involves providing the LLM with a series of instructions to help guide the model and generate a more coherent and relevant response. This technique is useful for obtaining well-reasoned responses from LLMs.
An example of a chain-of-thought prompt is as follows: “You are a virtual tour guide from 1901. You have tourists visiting Eiffel Tower. Describe Eiffel Tower to your audience. Begin with (1) why it was built, (2) how long it took to build, (3) where were the materials sourced to build, (4) number of people it took to build it, and (5) number of people visiting the Eiffel tour annually in the 1900’s, the amount of time it completes a full tour, and why so many people visit it each year. Make your tour funny by including one or two funny jokes at the end of the tour.”
22.4 Limitations and Ethical Considerations of LLMs¶
Although LLMs have demonstrated impressive performance across a wide range of tasks, there are several limitations and ethical considerations that raise concerns.
Limitations:
Computational resources: Training LLMs requires significant computational resources, making it difficult for researchers with limited access to GPUs or specialized hardware to develop and use these models.
Data bias: LLMs are trained on vast amounts of data from the internet, which often contain biases present in the data. As a result, the models may unintentionally learn and reproduce biases in their generated responses.
Producing hallucinations: LLMs can produce hallucinations, which are responses that are false, inaccurate, unexpected, or contextually inappropriate. One example of hallucination by ChatGPT is when asked to list academic papers by an author, and it provides papers that don’t exist.
Inability to explain: LLMs are inherently black-box models, making it challenging to explain their reasoning or decision-making processes, which is essential in certain applications like healthcare, finance, and legal domains.
Ethical considerations:
Privacy concerns: LLMs memorize information from their training data, and can potentially reveal sensitive information or violate user privacy.
Misinformation and manipulation: Text generated by LLMs can be exploited to create disinformation, fake news, or deepfake content that manipulates public opinion and undermines trust.
Accessibility and fairness: The computational resources and expertise required to train LLMs may lead to an unequal distribution of benefits, where only a few organizations have the resources to develop and control these powerful models.
Environmental impact: The large-scale training of LLMs consumes a significant amount of energy contributing to carbon emissions, which raises concerns about the environmental sustainability of these models.
Conclusively, it is important to encourage transparency, collaboration, and responsible AI practices to ensure that LLMs benefit all members of society without causing harm.
22.5 Foundation Models¶
Foundation Models are extremely large NN models trained on tremendous amounts of data with substantial computational resources, resulting in high capabilities for transfer learning to a wide range of downstream tasks. In other words, these models are scaled along each of the three factors: number of model parameters, size of the training dataset, and amount of computation. And, they are typically trained using self-supervised learning on unlabeled data. The scale of Foundation Models leads to new emergent capabilities, such as the ability to perform well on tasks that the models were not explicitly trained to do. This allows few-shot learning, which refers to finetuning Foundation Models to new downstream tasks by using only a few training data instances for the new task. Similarly, zero-shot learning extends this concept even further, and refers to a model’s ability to generalize to new tasks for which the model hasn’t seen any examples during the training.
LLMs represent early examples of Foundation Models, because LLMs are trained at scale and can be adapted for various NLP tasks, even for tasks they were not trained to perform.
The term Foundation Models is more general than LLMs, and they generally refer to large models that are trained on multimodal data, where the inputs can include text, images, audio, video, and other data sources.
The importance of Foundation Models is in their potential to replace task-specific ML models that are specialized in solving one task (i.e., optimized to perform well on one dataset) with general models that have the capabilities to solve multiple tasks. I.e., these models can serve as a foundation that is adaptable to a broad range of applications.
Figure: Foundation model. Source: link.
22.6 AI Agents¶
AI Agents are a collection of AI systems that are based on advanced LLMs and Foundation Models and can independently perform complex tasks. Differently from traditional ML models that have capabilities to make predictions given a dataset, or LLMs and Foundation Models that have capabilities to answer questions or generate text, images, and other content, Agentic AI systems demonstrate advanced capabilities such as making autonomous decisions and taking concrete actions like scheduling meetings, replying to emails, querying databases, and controlling physical devices. Through interaction with other AI systems, AI agents can complete multiple steps to perform tasks either without human guidance or with minimal human intervention.
For instance, an AI agent can plan a vacation or a business trip by visiting websites to book flights, reserve hotels, schedule events, and perform similar actions. In customer service an AI agent can communicate with a customer and perform actions such as confirming sales and orders, canceling subscriptions, and completing customer support tickets from beginning to end. In finance, an AI agent can reconcile invoices, initiate transactions, handle insurance claims, etc. In healthcare, an AI agent can assist patients by answering questions and booking appointments, support physicians by summarizing information and generating treatment suggestions, and can support administrative processes and manage records.
AI Agents represent one of the most impactful areas of AI development today, and it is expected that in 2025 various AI Agents will be developed for specific applications by training on domain-specific knowledge databases. Future AI Agents have potential to create personalized experiences by remembering user preferences and provide contextually aware assistance across complex tasks.
Appendix: Unsloth Library for LLM Training and Inference¶
Unsloth is another library for training and inference of LLMs, offering tools to facilitate optimization of LLMs (link) The library applies various optimization techniques to reduce the training and inference time in comparison to the Hugging Face library and other related libraries. As you will notice in the following code, the Unsloth tools use pre-built components from Hugging Face (such as transformers, trl) and adapt them to optimize various workflows
for model training and inference.
The following code [10] provides an example of finetuning LlaMA-3.1 8B model using a single T4 GPU. For this example, the training time was similar to training LlaMA 2 7B with the Hugging Face library above, as in both cases training for 1 epoch took about 15 minutes. On the other hand, while the largest batch size (in multiples of 2) with Hugging Face was 8 samples, Unsloth allowed to use a batch size of 16, meaning that Unsloth optimized the memory usage. Training LLMs with larger batch sizes is related to reduced training variance and more stable gradient updates, which typically result in improved performance. In addition, the inference with Unsloth was faster.
[ ]:
# Note: to install unsloth in this notebook, I had to interupt the currently running kernel, and start a new kernel
%%capture
!pip install -q unsloth
# Also get the latest nightly Unsloth!
!pip uninstall unsloth -y && pip install --upgrade --no-cache-dir "unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git"
[ ]:
from unsloth import FastLanguageModel
import torch
model, tokenizer = FastLanguageModel.from_pretrained(
model_name = "unsloth/Meta-Llama-3.1-8B-Instruct-bnb-4bit",
max_seq_length = 2048,
dtype = None,
load_in_4bit = True,
)
🦥 Unsloth: Will patch your computer to enable 2x faster free finetuning.
==((====))== Unsloth 2024.11.7: Fast Llama patching. Transformers = 4.46.2.
\\ /| GPU: Tesla T4. Max memory: 14.748 GB. Platform = Linux.
O^O/ \_/ \ Pytorch: 2.5.1+cu121. CUDA = 7.5. CUDA Toolkit = 12.1.
\ / Bfloat16 = FALSE. FA [Xformers = 0.0.28.post3. FA2 = False]
"-____-" Free Apache license: http://github.com/unslothai/unsloth
Unsloth: Fast downloading is enabled - ignore downloading bars which are red colored!
[ ]:
model = FastLanguageModel.get_peft_model(
model,
r = 16, # Choose any number > 0 suggested 8, 16, 32, 64, 128
target_modules = ["q_proj", "k_proj", "v_proj", "o_proj",
"gate_proj", "up_proj", "down_proj",],
lora_alpha = 16,
lora_dropout = 0, # Supports any, but = 0 is optimized
bias = "none", # Supports any, but = "none" is optimized
# [NEW] "unsloth" uses 30% less VRAM, fits 2x larger batch sizes!
use_gradient_checkpointing = "unsloth", # True or "unsloth" for very long context
random_state = 3407,
use_rslora = False, # Supports rank stabilized LoRA
loftq_config = None, # And LoftQ
)
Unsloth 2024.11.7 patched 32 layers with 32 QKV layers, 32 O layers and 32 MLP layers.
[ ]:
from datasets import load_dataset
# Load the Lamini dataset
dataset = load_dataset("mwitiderrick/llamini_llama", split="train")
[ ]:
from trl import SFTTrainer
from transformers import TrainingArguments
from unsloth import is_bfloat16_supported
trainer = SFTTrainer(
model = model,
tokenizer = tokenizer,
train_dataset = dataset,
dataset_text_field = "text",
max_seq_length = 2048,
dataset_num_proc = 2,
packing = False, # Can make training 5x faster for short sequences.
args = TrainingArguments(
per_device_train_batch_size = 16,
gradient_accumulation_steps = 2,
warmup_steps = 5,
num_train_epochs = 1,
learning_rate = 2e-4,
fp16 = not is_bfloat16_supported(),
bf16 = is_bfloat16_supported(),
logging_steps = 5,
optim = "adamw_8bit",
weight_decay = 0.01,
lr_scheduler_type = "linear",
seed = 3407,
output_dir = "outputs",
report_to="none"
),
)
[ ]:
trainer.train()
==((====))== Unsloth - 2x faster free finetuning | Num GPUs = 1
\\ /| Num examples = 1,260 | Num Epochs = 1
O^O/ \_/ \ Batch size per device = 16 | Gradient Accumulation steps = 2
\ / Total batch size = 32 | Total steps = 39
"-____-" Number of trainable parameters = 41,943,040
| Step | Training Loss |
|---|---|
| 5 | 2.841300 |
| 10 | 1.681000 |
| 15 | 0.779200 |
| 20 | 0.694200 |
| 25 | 0.670900 |
| 30 | 0.671200 |
| 35 | 0.616100 |
TrainOutput(global_step=39, training_loss=1.0871606301038692, metrics={'train_runtime': 915.5008, 'train_samples_per_second': 1.376, 'train_steps_per_second': 0.043, 'total_flos': 1.57759722651648e+16, 'train_loss': 1.0871606301038692, 'epoch': 0.9873417721518988})
[ ]:
# Perform inference
FastLanguageModel.for_inference(model)
prompt = "What are Lamini models?"
inputs = tokenizer([prompt.format(
"", # instruction
"", # input
"", # output
)], return_tensors = "pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=200, use_cache=True)
decoded_output = tokenizer.batch_decode(outputs)
print("\n".join(decoded_output))
<|begin_of_text|>What are Lamini models? Lamini models are pre-trained language models that can be fine-tuned or customized for specific tasks, such as generating text, answering questions, or completing tasks. These models are trained on a large dataset of text and can learn patterns and relationships in language that enable them to generate coherent and contextually relevant text.
Lamini models are based on the transformer architecture, which is a type of neural network that is particularly well-suited for natural language processing tasks. They are trained using a variety of techniques, including masked language modeling, next sentence prediction, and language modeling.
Some of the key features of Lamini models include:
1. Pre-trained language models: Lamini models are pre-trained on a large dataset of text, which allows them to learn patterns and relationships in language.
2. Fine-tuning: Lamini models can be fine-tuned or customized for specific tasks, such as generating text, answering questions, or completing tasks.
3. Contextual understanding: Lamini models can understand the
[ ]:
# Perform inference
prompt = "Write a poem about Data Science"
inputs = tokenizer([prompt.format(
"", # instruction
"", # input
"", # output
)], return_tensors = "pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=500, use_cache=True)
decoded_output = tokenizer.batch_decode(outputs)
print("\n".join(decoded_output))
<|begin_of_text|>Write a poem about Data Science
In realms of code and numbers bright,
A new breed of heroes takes flight,
With data as their guiding light,
They weave a tapestry of insight and might.
Their tools of trade, a Python script,
Or R, or SQL, a language to connect,
They dance with algorithms, a wondrous sight,
As they unravel secrets, hidden from the light.
With machine learning, they learn and grow,
As models trained on data start to show,
The paths to answers, hidden deep inside,
Are revealed with clarity, as the data does provide.
Their quest for truth, a noble goal,
A pursuit of knowledge, that makes the soul,
Their work, a symphony of code and art,
A harmony of logic, that touches the heart.
In data science, they find their home,
Where numbers and code, are not unknown,
But a language, that speaks to the mind,
And reveals the secrets, left behind.
With data as their guide, they navigate,
The vast expanse of information, that creates,
A world of wonder, where insights abound,
And the mysteries of data, are finally found. #datascience #poetry
In the realm of code and numbers bright,
A new breed of heroes takes flight,
With data as their guiding light,
They weave a tapestry of insight and might.
Their tools of trade, a Python script,
Or R, or SQL, a language to connect,
They dance with algorithms, a wondrous sight,
As they unravel secrets, hidden from the light.
With machine learning, they learn and grow,
As models trained on data start to show,
The paths to answers, hidden deep inside,
Are revealed with clarity, as the data does provide.
Their quest for truth, a noble goal,
A pursuit of knowledge, that makes the soul,
Their work, a symphony of code and art,
A harmony of logic, that touches the heart.
In data science, they find their home,
Where numbers and code, are not unknown,
But a language, that speaks to the mind,
And reveals the secrets, left behind.
With data as their guide, they navigate,
The vast expanse of information, that creates,
A world of wonder, where insights abound,
And the mysteries of data, are finally found. #datascience #poetry
This poem is a tribute to the world of Data Science, where data is the guiding light that helps us navigate the vast expanse of information and uncover hidden secrets
References¶
Introduction to Large Language Models, by Bernhard Mayrhofer, available at https://github.com/datainsightat/introduction_llm.
Understanding Encoder and Decoder LLMs, by Sebastian Raschka, available at https://magazine.sebastianraschka.com/p/understanding-encoder-and-decoder.
LLM Training: RLHF and Its Alternatives, by Sebastian Raschka, available at https://magazine.sebastianraschka.com/p/llm-training-rlhf-and-its-alternatives.
Training Language Models to Follow Instructions with Human Feedback, by Long Ouyang et al., available at https://arxiv.org/abs/2203.02155.
Parameter-Efficient LLM Finetuning With Low-Rank Adaptation (LoRA), by Sebastian Raschka, available at https://sebastianraschka.com/blog/2023/llm-finetuning-lora.html.
How to Fine-tune Llama 2 With LoRA, by Derrick Mwiti, available at https://www.mldive.com/p/how-to-fine-tune-llama-2-with-lora.
Fine-Tuning Llama 2.0 with Single GPU Magic, by Chee Kean, available at https://ai.plainenglish.io/fine-tuning-llama2-0-with-qloras-single-gpu-magic-1b6a6679d436.
Fine-Tuning LLaMA 2 Models using a single GPU, QLoRA and AI Notebooks, by Mathieu Busquet, available at https://blog.ovhcloud.com/fine-tuning-llama-2-models-using-a-single-gpu-qlora-and-ai-notebooks/.
Getting started with Llama, by Meta AI, available at https://ai.meta.com/llama/get-started/.
Llama-3.1 8b + Unsloth 2x faster finetuning, by Unsloth AI, available at https://colab.research.google.com/drive/1Ys44kVvmeZtnICzWz0xgpRnrIOjZAuxp?usp=sharing.
BACK TO TOP