![](\"images/transformer.jpg\"){kind=spacer}
\n","\n","*Figure: Pretraining LLMs.* Source: [2]."],"metadata":{"id":"aP-g3MgRUYUI"}},{"cell_type":"markdown","source":["The architecture of Transformer Networks includes multiple successive encoder and decoder blocks to create deep networks with many layers that allow learning complex patterns in input text. For example, the original Transformer Network has 6 encoder and 6 decoder blocks, as shown in the above figure.\n","\n","The **self-attention mechanism** is a key component of the Transformer Network architecture that enables the model to weigh the importance of each token with respect to the other tokens in a sequence. It allows to capture long-range dependencies and relationships between the tokens (words) and helps the model to understand the context and structure of the input text sequence."],"metadata":{"id":"5eNRiCQhWXlN"}},{"cell_type":"markdown","source":["### 22.1.2 Variants of Transformer Network Architectures ![](\"images/pretraining.jpg\")
\n","\n","*Figure: Pretraining LLMs.* Source: [3].\n","\n","Pretraining allows to extract knowledge from very large unlabeled datasets in unsupervised learning manner, without the need for manual labeling. Or, to be more precise, the \"label\" in LLMs pretraining is the next word in the text, to which we already have access since it is part of the training text. Such pretraining approach is also called self-supervised training, since the model uses each next word in the text to self-supervise the training.\n","\n","Note that pretraining LLMs from scratch is computationally expensive and time-consuming. As we stated before, the pretraining phase can cost millions of dollars (e.g., the estimated cost for training GPT-4 is $100 million). Also, pretraining LLMs requires access to large datasets and technical expertise with strong understanding of deep learning workflows, working with distributed software and hardware, and managing model training with thousands of GPUs simultaneously."],"metadata":{"id":"wJ5rp47DXf2w"}},{"cell_type":"markdown","source":["### 22.2.2 Supervised Finetuning ![](\"images/finetuning.jpg\")
\n","\n","*Figure: Finetuning a pretrained LLM.* Source: [3]."],"metadata":{"id":"LnAjaqBPXlJC"}},{"cell_type":"markdown","source":["### 22.2.3 Alignment ![](\"images/RLHF.jpg\")
\n","\n","*Figure: Reinforcement Learning from Human Feedback.* Source: [4].\n","\n","In summary, the RLHF approach creates a reward system that is augmented by human feedback and is used to teach LLMs which responses are more aligned with human preferences. Through these iterations, LLMs can be better aligned with our human values and can lead to higher-quality responses, as well as improved performance on specific tasks.\n","\n","Note also that there are several variants of the RLFH approach for finetuning LLMs. For example, LlaMA models employ two reward models: one based on the ranks of helpfulness of the responses, and another based on the ranks of safety of the responses. The final reward score is obtained as a combination of the helpfulness and safety scores.\n","\n","**Reinforcement Learning with Direct Policy Optimization (DPO)**\n","\n","RL with Direct Policy Optimization (DPO) is another approach for LLM alignment that has been popular recently, as it is simpler than RLHF with PPO. DPO uses a different optimization approach in comparison to RL with PPO, where DPO optimizes the LLM directly based on user preferences, without the need for training a separate Reward Model. I.e., DPO aims to directly maximize the reward function to produce model outputs that align with human preferences. Detailed explanation of RL with DPO is beyond the scope of this lecture.\n"],"metadata":{"id":"0s2BsqGjXlLg"}},{"cell_type":"markdown","source":["## 22.3 Finetuning LLMs ![](\"images/LoRA.png\")
\n","\n","*Figure: Regular finetuning versus LoRA finetuning .* Source: [5]."],"metadata":{"id":"XS2wNFlwlgD1"}},{"cell_type":"markdown","source":["### 22.3.3 Quanitized LoRA (QLoRA) ![](\"images/foundation_model.jpg\")
\n","\n","*Figure: Foundation model.* Source: [link](https://blogs.nvidia.com/blog/2023/03/13/what-are-foundation-models/)."],"metadata":{"id":"BnFBFVAEe1my"}},{"cell_type":"markdown","source":["## 22.6 AI Agents "],"metadata":{"id":"_R5CX-pQi_48"}},{"cell_type":"markdown","source":["\n","**AI Agents** are a collection of AI systems that are based on advanced LLMs and Foundation Models and can independently perform complex tasks. Differently from traditional ML models that have capabilities to make predictions given a dataset, or LLMs and Foundation Models that have capabilities to answer questions or generate text, images, and other content, Agentic AI systems demonstrate advanced capabilities such as making autonomous decisions and taking concrete actions like scheduling meetings, replying to emails, querying databases, and controlling physical devices. Through interaction with other AI systems, AI agents can complete multiple steps to perform tasks either without human guidance or with minimal human intervention.\n","\n","For instance, an AI agent can plan a vacation or a business trip by visiting websites to book flights, reserve hotels, schedule events, and perform similar actions. In customer service an AI agent can communicate with a customer and perform actions such as confirming sales and orders, canceling subscriptions, and completing customer support tickets from beginning to end. In finance, an AI agent can reconcile invoices, initiate transactions, handle insurance claims, etc. In healthcare, an AI agent can assist patients by answering questions and booking appointments, support physicians by summarizing information and generating treatment suggestions, and can support administrative processes and manage records.\n","\n","AI Agents represent one of the most impactful areas of AI development today, and it is expected that in 2025 various AI Agents will be developed for specific applications by training on domain-specific knowledge databases. Future AI Agents have potential to create personalized experiences by remembering user preferences and provide contextually aware assistance across complex tasks."],"metadata":{"id":"t-MKsiV8jG3i"}},{"cell_type":"markdown","source":["## Appendix: Unsloth Library for LLM Training and Inference "],"metadata":{"id":"wqMGfKQFqepe"}},{"cell_type":"markdown","source":["**Unsloth** is another library for training and inference of LLMs, offering tools to facilitate optimization of LLMs ([link](https://unsloth.ai/)) The library applies various optimization techniques to reduce the training and inference time in comparison to the Hugging Face library and other related libraries. As you will notice in the following code, the Unsloth tools use pre-built components from Hugging Face (such as `transformers`, `trl`) and adapt them to optimize various workflows for model training and inference.\n","\n","The following code [10] provides an example of finetuning LlaMA-3.1 8B model using a single T4 GPU. For this example, the training time was similar to training LlaMA 2 7B with the Hugging Face library above, as in both cases training for 1 epoch took about 15 minutes. On the other hand, while the largest batch size (in multiples of 2) with Hugging Face was 8 samples, Unsloth allowed to use a batch size of 16, meaning that Unsloth optimized the memory usage. Training LLMs with larger batch sizes is related to reduced training variance and more stable gradient updates, which typically result in improved performance. In addition, the inference with Unsloth was faster."],"metadata":{"id":"tV3OTXmItWMa"}},{"cell_type":"code","source":["# Note: to install unsloth in this notebook, I had to interupt the currently running kernel, and start a new kernel\n","%%capture\n","!pip install -q unsloth\n","# Also get the latest nightly Unsloth!\n","!pip uninstall unsloth -y && pip install --upgrade --no-cache-dir \"unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git\""],"metadata":{"id":"PLD3oFA_Z6Hj"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["from unsloth import FastLanguageModel\n","import torch\n","\n","model, tokenizer = FastLanguageModel.from_pretrained(\n"," model_name = \"unsloth/Meta-Llama-3.1-8B-Instruct-bnb-4bit\",\n"," max_seq_length = 2048,\n"," dtype = None,\n"," load_in_4bit = True,\n",")"],"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":298,"referenced_widgets":["e5e7e416af46439daec974646d0ea4f7","d275be345fc64126b2a7ea9b978a0bbb","6d2754fa7c584e15b41445b2a9a14fe5","cae2866e69eb476fb33c55b481855d9c","d916318be1554302af0c086d3d0ab058","9c4c1ea43ffe419b84867c45378667b2","09ae798102cc488fbbe63406dbdfb484","0abf2ed04d994594832d0ee1aa6c2cc4","69e4e4cf23a744a79724230b0c4abe5e","45705056203241cc82a9093cb2095d28","6edd9a04774b463fa899b31723e7a1c3","183faafaa5de458885ceeb37367f0584","e02e952e6859409485ae729b8ac913d2","84a2d590ff984fcebbcf03c0c2f34de6","82b04d51c46c4b39a21684a7efd50882","29b86e0f788345d0a1bdefd72b45e756","1e8487619f3445e797bc14401bf0c590","8519d5ec774740fa91a2120f99790df5","6fda3e71d8db490ea5e9aae65543746e","55bad581711e4badb2e8830fd136e2e6","711e5a54ca2943948ffe5e5196bd43ed","b2b4e3081438434b99a71db14807f793","acb70a0efbe74c37b7e57f726fd90036","f6edb1a98831423b8a007b9e7b4c69cd","01622b0416cb491f904d0aae80d4519c","aacd82e0bc694e60928e92f3fbf5fb17","699f8c2894a74e09b1e30ec058ee3c54","a10da8a602d544059ead3d89ea799fb7","6dfde5ff6bfd473ebbaf6388619bebfe","352ac1fe560e44a3a6b3811a691fbd6b","31543da5d7b84dc68bc55ac04b64793f","57469e905c0d4ca086ea65cedd6d4044","927ffedf579849cdb585be9a67bec8aa","f436913f7187460990933ec6f41445d6","16be3956178e4b4883933767c1c3d90c","c4e756e10e4a4a28b2ba250cbe4f6462","ed223bdbf2ab482385513275442d58dc","17173c7cafe7428faeeb79afc1273073","75e46bf54bca4c6a8d9273b10ff8a97e","ef65aa221b214d03801f23ce1f904f9f","cbf8760463814d78ba6b735bcea62daa","fc67ebc00205455cb526b272a28a10b6","323ca860db28457f8090819ca7449924","f9e12920746b4a88a9dd3ce47f61f82a","addc9a3b16ad4737858d1aeae75cafb6","b4108af85a0e41e8a3e64e2299c45a45","a1a2c6c92f144c7ab683f0c92f440898","f2c6fde1f3b142f8b57ff1b9fa0cea11","08ec3947201047708fd0064f6339b401","fbd0115eba55464abdba87f49e1a0bff","1dfafae804964572af67d314c44aa570","c3cf0f3465224a0696c23531c4c7734e","8de9414fe05d4fc784d0ee218837bab9","6f80de026504479aa92551a5eee17236","50eec38214184192b70add3f6deeb18b"]},"id":"1jnwKADYr8Gn","executionInfo":{"status":"ok","timestamp":1731777055551,"user_tz":420,"elapsed":54678,"user":{"displayName":"Aleksandar Vakanski","userId":"07675307153279708378"}},"outputId":"badc4541-dac6-47eb-d8a1-c860bf844bb6"},"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["🦥 Unsloth: Will patch your computer to enable 2x faster free finetuning.\n","==((====))== Unsloth 2024.11.7: Fast Llama patching. Transformers = 4.46.2.\n"," \\\\ /| GPU: Tesla T4. Max memory: 14.748 GB. Platform = Linux.\n","O^O/ \\_/ \\ Pytorch: 2.5.1+cu121. CUDA = 7.5. CUDA Toolkit = 12.1.\n","\\ / Bfloat16 = FALSE. FA [Xformers = 0.0.28.post3. FA2 = False]\n"," \"-____-\" Free Apache license: http://github.com/unslothai/unsloth\n","Unsloth: Fast downloading is enabled - ignore downloading bars which are red colored!\n"]},{"output_type":"display_data","data":{"text/plain":["model.safetensors: 0%| | 0.00/5.70G [00:00 0 suggested 8, 16, 32, 64, 128\n"," target_modules = [\"q_proj\", \"k_proj\", \"v_proj\", \"o_proj\",\n"," \"gate_proj\", \"up_proj\", \"down_proj\",],\n"," lora_alpha = 16,\n"," lora_dropout = 0, # Supports any, but = 0 is optimized\n"," bias = \"none\", # Supports any, but = \"none\" is optimized\n"," # [NEW] \"unsloth\" uses 30% less VRAM, fits 2x larger batch sizes!\n"," use_gradient_checkpointing = \"unsloth\", # True or \"unsloth\" for very long context\n"," random_state = 3407,\n"," use_rslora = False, # Supports rank stabilized LoRA\n"," loftq_config = None, # And LoftQ\n",")"],"metadata":{"id":"dTxPMx9jZ6J2","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1731777062453,"user_tz":420,"elapsed":6904,"user":{"displayName":"Aleksandar Vakanski","userId":"07675307153279708378"}},"outputId":"45f59e03-c05c-4d3e-c964-bdfa2b989a6a"},"execution_count":null,"outputs":[{"output_type":"stream","name":"stderr","text":["Unsloth 2024.11.7 patched 32 layers with 32 QKV layers, 32 O layers and 32 MLP layers.\n"]}]},{"cell_type":"code","source":["from datasets import load_dataset\n","\n","# Load the Lamini dataset\n","dataset = load_dataset(\"mwitiderrick/llamini_llama\", split=\"train\")"],"metadata":{"id":"0VNtZ0BPsf2z","colab":{"base_uri":"https://localhost:8080/","height":81,"referenced_widgets":["bf99e5d1321b41d787b14680af6bc853","4179f0ffa444421db141260cdf5e0c86","bbb3a0c090b84d76ad7a58f62e61e921","7616e56620e64e1d9c9f43b8fa373ee7","679787c064e943e294105d7485f27db3","7ff78ec2155d41b8a170fc9552281598","e60c351dbd6e429198272b8100ed2519","6b68bd2b35084fbfa6e2f5ca3183921a","39d62487318d44158663b30e0d2513d6","bfe15461b1b248eba8f72ddfe76faec2","d34cd5e679954580b3b3d6b78529ed54","9816f6eff79448449ff4d5d748c08516","9824f4ed8bd8467d8c155d40ded17a85","29836bb0da3446dabf0f38c5ed18ef6a","3784907f730e49a2bdd86093984b5cf2","11ae8cabe37740d39c3027b29e6d9d4a","5c28514607ab4177b136d494e34074c8","6ee0c978f7814e3ba50eb1f8b8a78a66","2751550f53a4454ebaff4304943cd481","312ccaa96d2c43769abc3ee67b8aebaf","339397339da1416a89691a85ec626e0f","d2e21bdc396f47ab9bb59b489e7173ea"]},"executionInfo":{"status":"ok","timestamp":1731777067388,"user_tz":420,"elapsed":3683,"user":{"displayName":"Aleksandar Vakanski","userId":"07675307153279708378"}},"outputId":"3a7096d8-2772-46f6-db0d-32cfb99efafe"},"execution_count":null,"outputs":[{"output_type":"display_data","data":{"text/plain":["lamini2.csv: 0%| | 0.00/1.14M [00:00"],"text/html":["\n"," \n"," \n"," \n"," [39/39 14:49, Epoch 0/1]\n","
\n"," | Step | \n","Training Loss | \n","
|---|---|
| 5 | \n","2.841300 | \n","
| 10 | \n","1.681000 | \n","
| 15 | \n","0.779200 | \n","
| 20 | \n","0.694200 | \n","
| 25 | \n","0.670900 | \n","
| 30 | \n","0.671200 | \n","
| 35 | \n","0.616100 | \n","