{ "cells": [ { "cell_type": "markdown", "id": "182c44d2", "metadata": {}, "source": [ "# Tutorial 10 - Bash Scripting" ] }, { "cell_type": "markdown", "id": "5aad412e-ceab-4acc-9aa4-2a9aad4a08d3", "metadata": {}, "source": [ "[![View notebook on Github](https://img.shields.io/static/v1.svg?logo=github&label=Repo&message=View%20On%20Github&color=lightgrey)](https://github.com/avakanski/Fall-2024-Applied-Data-Science-with-Python/blob/main/docs/Lectures/Tutorials/Tutorial_10-Bash/Tutorial_10-Bash.ipynb)\n", "[![Open In Collab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/avakanski/Fall-2024-Applied-Data-Science-with-Python/blob/main/docs/Lectures/Tutorials/Tutorial_10-Bash/Tutorial_10-Bash.ipynb)" ] }, { "cell_type": "markdown", "id": "f6aa84e4-7ae5-4ac8-a66b-94c020db11be", "metadata": {}, "source": [ "" ] }, { "cell_type": "markdown", "id": "6dd37bf8", "metadata": {}, "source": [ "This tutorial is adapted from a blog post on the website [Bash Scripting Tutorial](https://www.freecodecamp.org/news/bash-scripting-tutorial-linux-shell-script-and-command-line-for-beginners/), by Freecodecamp." ] }, { "cell_type": "markdown", "id": "6dc0b898-a27d-4f90-9b2b-9a278cfcf3d2", "metadata": {}, "source": [ "**Bash** (Bourne Again SHell) is a shell and command language interpreter for the GNU operating system and is widely used as the default shell on Linux systems. The terms `shell` and `bash` are often used interchangeably, though there is a subtle distinction between them. A \"shell\" is a program that provides a command-line interface for interacting with an operating system. Bash (Bourne-Again SHell) is one of the most widely used shells in Unix/Linux systems and serves as the default shell in many Linux distributions. While Bash is one type of shell, other options are also available, including the Korn shell (ksh), C shell (csh), and Z shell (zsh). Each shell has its own syntax and unique features, yet they all serve the shared purpose of providing a command-line interface for interacting with the operating system.\n", "\n" ] }, { "cell_type": "markdown", "id": "e57368a1", "metadata": {}, "source": [ "## Advantages of Bash Scripts\n", "\n", "Bash scripts' simplicity allows users to quickly write and understand scripts, even with minimal programming knowledge, which is particularly useful for beginners and system administrators alike. By automating repetitive tasks, Bash scripts save time and reduce human error, which is crucial for maintaining consistency in frequently performed operations like backups, file management, and system monitoring. Additionally, Bash allows users to run sequences of commands as a single command, making complex workflows easier to manage and execute. The ease of integrating other tools and programs within Bash scripts further enhances their utility, allowing for complex and efficient automation. Moreover, Bash’s wide availability across UNIX-based systems ensures that scripts are highly portable, making it an ideal choice for tasks that need to be executed across multiple systems. Overall, Bash scripting improves productivity by streamlining workflows, reducing manual effort, and enabling efficient system administration." ] }, { "cell_type": "markdown", "id": "e7c0e944", "metadata": {}, "source": [ "## Basic Shell Interface " ] }, { "cell_type": "markdown", "id": "5d64358a", "metadata": {}, "source": [ "When a shell is used interactively, it displays a `$` prompt to indicate that it is waiting for a command from the user.\n", "\n", "Below is an example of a command-line interface for a shell." ] }, { "cell_type": "markdown", "id": "ac3f0329", "metadata": {}, "source": [ "" ] }, { "cell_type": "markdown", "id": "ad71b19f", "metadata": {}, "source": [ "You can identify the shell type by using the `ps` command:\n", "\n", "" ] }, { "cell_type": "markdown", "id": "1dfec3a4", "metadata": {}, "source": [ "### Basic Bash Commands " ] }, { "cell_type": "markdown", "id": "846931c8", "metadata": {}, "source": [ "Generally, commands follow this syntax: `command [OPTIONAL] arguments`. The text `[OPTIONAL]` is for flags and inputs that are not required to run the command or that have default values. The text `arguments` is for flags or inputs that are required to run the command. These arguments must be supplied by the user or the command will not execute or produce an error. Flags start with the character `-`, such as the the flag `-v` to enable verbose mode. \n", "\n", "Commonly used Bash commands are:\n", "\n", "* `cd`: Change the directory to a different location.\n", "* `ls`: List the contents of the current directory.\n", "* `mkdir`: Create a new directory.\n", "* `touch`: Create a new file.\n", "* `rm`: Remove a file or directory.\n", "* `cp`: Copy a file or directory.\n", "* `mv`: Move or rename a file or directory.\n", "* `echo`: Print text to the terminal.\n", "* `cat`: Concatenate and print the contents of a file.\n", "* `grep`: Search for a pattern in a file.\n", "* `chmod`: Change the permissions of a file or directory.\n", "* `sudo`: Run a command with administrative privileges.\n", "* `df`: Display the amount of disk space available.\n", "* `history`: Show a list of previously executed commands.\n", "* `ps`: Display information about running processes." ] }, { "cell_type": "markdown", "id": "e17b3084", "metadata": {}, "source": [ "Several examples are shown below.\n", "\n", "`ls`: Display the contents of the current directory.\n", "\n", "" ] }, { "cell_type": "markdown", "id": "a0b708b8", "metadata": {}, "source": [ "`echo`: Prints a string or value of a variable in the terminal.\n", "\n", "" ] }, { "cell_type": "markdown", "id": "2353b95f", "metadata": {}, "source": [ "`cat`: Displays the contents of a file or multiple files line by line.\n", "\n", "" ] }, { "cell_type": "markdown", "id": "bf44f3c2", "metadata": {}, "source": [ "`sudo apt install package-name`: Install software package with administrative privilege. \n", "\n", "\n" ] }, { "cell_type": "markdown", "id": "570b530a", "metadata": {}, "source": [ "`date`: Displays the current date.\n", "\n", "" ] }, { "cell_type": "markdown", "id": "cff3d1e2", "metadata": {}, "source": [ "`pwd`: Displays the present working directory.\n", "\n", "" ] }, { "cell_type": "markdown", "id": "e774d5c0", "metadata": {}, "source": [ "`man command`: Look up the details of the command in the command manual.\n", "\n", "Example: `man ls` provides the manual for the command `ls`.\n", "\n", "" ] }, { "cell_type": "markdown", "id": "183df56c-d2e4-4339-9bca-5b531dc78bf6", "metadata": {}, "source": [ "`help`: Provides an overview of the Bash commands.\n", "\n", "" ] }, { "cell_type": "markdown", "id": "fa07c9ca", "metadata": {}, "source": [ "### Creating Bash Script\n", "\n", "Let's create a Bash script that reads and displays the user input. Specifically, the user will be asked to enter a path, and the content of that directory will be listed in the terminal. \n", "\n", "1) Write `vi run_all.sh` to create a file named `run_all.sh`, where `vi` opens the vi editor, which is a text editor for Unix-like systems.\n", "2) In the opened text editor, insert the following commands and save the script:\n", "\n", " #!/bin/bash\n", " echo \"Today is \" `date`\n", " echo -e \"\\nenter the path to directory\"\n", " read the_path\n", " echo -e \"\\n your path has the following files and folders: \"\n", " ls $the_path\n" ] }, { "cell_type": "markdown", "id": "f50e9b10", "metadata": {}, "source": [ "In the above code, the Bash script begins with a combination of the `#` (hash) and `!` (bang) symbols called `shebang`, followed by the path to the Bash shell `/bin/bash/`. This is typically the first line of a Bash script, and it instructs the shell to run the script using the Bash interpreter. Essentially, the shebang provides the absolute path to the Bash interpreter." ] }, { "cell_type": "markdown", "id": "621bbc65", "metadata": {}, "source": [ "Next, the `echo` command is used to display the current date and time in the terminal. Note that the date is in backticks. In the next line, the user is asked to enter a path to a directory. The `read` command reads the input and stores it in the variable `the_path`. The `ls` command takes the variable with the stored path and displays the current files and folders in that directory." ] }, { "cell_type": "markdown", "id": "d7abfe17", "metadata": {}, "source": [ "3) To make the script executable, grant execution permissions to the user with this command:\n", "\n", " chmod u+x run_all.sh" ] }, { "cell_type": "markdown", "id": "cd77639e", "metadata": {}, "source": [ "In the above code, `chmod` changes the ownership of a file for the current user `u` and `+x` adds the execution rights, therefore the current user can run the file. `run_all.sh` is the file to run.\n", "\n", "The script can be run using any of the following methods:\n", "\n", "* `bash run_all.sh`\n", "* `sh run_all.sh`\n", "* `./run_all.sh`" ] }, { "cell_type": "markdown", "id": "fe32bda1", "metadata": {}, "source": [ "" ] }, { "cell_type": "markdown", "id": "d4bfe2f4", "metadata": {}, "source": [ "### Comments In Bash Scripting\n", "\n", "In Bash scripting, comments begin with `#`, as in Python scripts. Any line starting with `#` is treated as a comment and will be ignored by the interpreter.\n", "\n", " # This is an example comment" ] }, { "cell_type": "markdown", "id": "e27df3ec", "metadata": {}, "source": [ "### Variables and Data Handling in Bash\n", "\n", "Variables allow to read, access, and manipulate data throughout your script. Bash does not have defined data types. A variable can hold numeric values, single characters, or text strings.\n", "\n", "You can define and use variable values in Bash in the following ways:\n", "\n", "1) Directly assign the value: e.g., `country=USA`\n", "2) Assign a value based on the output of a program or command using command substitution, where `$` is used to access the value of an existing variable: e.g, `new_country=$country`\n", "\n", "\n", "\n", "To access the value of a variable, prefix the variable name with `$`. For instance, the following script reads the user input and displays the user name in the terminal. \n", "\n", " #!/bin/bash \n", " echo \"What's your name?\" \n", " read entered_name \n", " echo -e \"Welcome to Bash tutorial\" $entered_name" ] }, { "cell_type": "markdown", "id": "0319d7bb", "metadata": {}, "source": [ "### Variable Naming Conventions\n", "\n", "In Bash scripting, follow these conventions for naming variables:\n", "\n", "* Begin variable names with a letter or an underscore (_).\n", "* Variable names can include letters, numbers, and underscores (_).\n", "* Variable names are case-sensitive.\n", "* Avoid spaces and special characters in variable names.\n", "* Choose descriptive names that convey the variable's purpose.\n", "* Do not use reserved keywords (e.g., `if`, `then`, `else`, etc.) as variable names." ] }, { "cell_type": "markdown", "id": "98381baa", "metadata": {}, "source": [ "### Arithmetic Operators\n", "`+`, `-`, `*`, `/`, `% ` (Addition, Subtraction, Multiplication, Division, Remainder).\n", "\n", "### Comparison Operators for Evaluating Integers\n", "\n", "* `-lt`: Less than (<)
\n", "* `-gt`: Greater than (>)
\n", "* `-le`: Less than or equal to (<=)
\n", "* `-e`: Greater than or equal to (>=)
\n", "* `-eq`: Equal to (==)
\n", "* `-ne`: Not equal to (!=)
\n", "\n", "### Logical Operators \n", "* `-a`: AND
\n", "* `-o`: OR
" ] }, { "cell_type": "markdown", "id": "9d1a423e", "metadata": {}, "source": [ "### Conditional Statements (if/else)\n", "\n", "There are several methods for evaluating conditions in Bash, including `if`, `if-else`, `if-elif-else`, and nested conditional statements. The syntax is as follows:\n", "\n", " if [ condition ]; then\n", " statement\n", " elif [ condition ]; then\n", " statement \n", " else\n", " do this by default\n", " fi" ] }, { "cell_type": "markdown", "id": "bf9e81eb", "metadata": {}, "source": [ "For example, the following Bash script named `test_odd.sh` uses `if-elif-else` statements to determine if a number inputted by the user is positive, negative, or zero. The end of an `if` statement is marked with `fi`.\n", "\n", " #!/bin/bash\n", " echo \"Please enter a number: \"\n", " read num\n", "\n", " if [ $num -gt 0 ]; then\n", " echo \"$num is positive\"\n", " elif [ $num -lt 0 ]; then\n", " echo \"$num is negative\"\n", " else\n", " echo \"$num is zero\"\n", " fi" ] }, { "cell_type": "markdown", "id": "35ce764b", "metadata": {}, "source": [ "" ] }, { "cell_type": "markdown", "id": "fafb5bca", "metadata": {}, "source": [ "### Looping and Branching in Bash\n", "\n", "#### While Loop\n", "\n", "A `while` loop checks a condition and continues to execute as long as the condition is true. " ] }, { "cell_type": "markdown", "id": "0e305da5", "metadata": {}, "source": [ "In the example below, create a file called `while_loop.sh`, where `(( i += 1 ))` serves as the counter statement, incrementing the value of `i`. This loop will execute 10 times.\n", "\n", " #!/bin/bash\n", " i=1\n", " while [[ $i -le 10 ]] ; do\n", " echo \"$i\"\n", " (( i += 1 ))\n", " done" ] }, { "cell_type": "markdown", "id": "5abd0ae3", "metadata": {}, "source": [ "" ] }, { "cell_type": "markdown", "id": "88c62fbe", "metadata": {}, "source": [ "#### For Loop\n", "\n", "Similar to the `while` loop, the `for` loop executes statements a specific number of times. \n", "\n", "In the example below, the file called `for_loop.sh` runs the loop for 5 iterations.\n", "\n", " #!/bin/bash\n", " for i in {1..5}\n", " do\n", " echo $i\n", " done\n" ] }, { "cell_type": "markdown", "id": "ca06ca13", "metadata": {}, "source": [ "" ] }, { "cell_type": "markdown", "id": "df41ff09", "metadata": {}, "source": [ "### Case Statements\n", "\n", "In Bash, case statements allow comparing a value against a list of patterns and executes a specific block of code for the first matching pattern. The syntax for a case statement in Bash is as follows:" ] }, { "cell_type": "markdown", "id": "4884e7b6", "metadata": {}, "source": [ " case expression in\n", " pattern1)\n", " # code to execute if expression matches pattern1\n", " ;;\n", " pattern2)\n", " # code to execute if expression matches pattern2\n", " ;;\n", " pattern3)\n", " # code to execute if expression matches pattern3\n", " ;;\n", " *)\n", " # code to execute if none of the above patterns match expression\n", " ;;\n", " esac" ] }, { "cell_type": "markdown", "id": "043ed892", "metadata": {}, "source": [ "In the example in the next cell, since the value of fruit is `\"apple\"`, the first pattern matches, and the code block that outputs `\"This is a red fruit.\"` runs. If fruit was `\"banana\"` instead, the second pattern would match, and the code block that outputs `\"This is a yellow fruit.\"` would execute, and so on. If `\"fruit\"` does not match any of the specified patterns, the default case runs, outputting `\"Unknown fruit.\"` The end of a `case` statement is marked with the reverse word `esac`." ] }, { "cell_type": "markdown", "id": "2e4309f0", "metadata": {}, "source": [ " fruit=\"apple\"\n", "\n", " case $fruit in\n", " \"apple\")\n", " echo \"This is a red fruit.\"\n", " ;;\n", " \"banana\")\n", " echo \"This is a yellow fruit.\"\n", " ;;\n", " \"orange\")\n", " echo \"This is an orange fruit.\"\n", " ;;\n", " *)\n", " echo \"Unknown fruit.\"\n", " ;;\n", " esac" ] }, { "cell_type": "markdown", "id": "28c8566f", "metadata": {}, "source": [ "### File Scheduling Scripts Using Cron in Bash\n", "\n", "Cron is a powerful job scheduling utility available on Unix-like operating systems. With Cron, you can configure automated jobs to run on a daily, weekly, monthly, or specified schedule. \n", "\n", "The syntax for scheduling cron jobs is:\n", "\n", " # Cron job example\n", " * * * * * sh /path/to/script.sh\n", "\n", "The `*` symbols represent in order: minutes, hours, days, months, and weekdays." ] }, { "cell_type": "markdown", "id": "b630a04a", "metadata": {}, "source": [ "Here are some examples of scheduling Cron jobs.\n", "\n", "" ] }, { "cell_type": "markdown", "id": "4f77f8be", "metadata": {}, "source": [ "**Using Crontab**\n", "\n", "The `crontab` utility allows add and edit cron jobs. For instance, `crontab -l` lists the scheduled jobs for a specific user. You can add or modify cron jobs with `crontab -e`." ] }, { "cell_type": "markdown", "id": "e7689600", "metadata": {}, "source": [ "**Batch Jobs**\n", "\n", "To start a single batch job, use `runbatch.sh [Input URL] [Output URL] []`. This command initiates a batch job with specified input and output URLs indicating where input data for the batch job is located, and where the output of the batch job will be saved.\n", " \n", "To start a chain of batch jobs, execute `runbatch.sh`. This command initiate a chain of batch jobs defined in the `Batch.properties` file.\n", " \n", "To start a batch job with job files, use `runbatch.sh -job [-name ] ... `. It initiates a batch job or chain using specific job files ` ... `, instead of URLs or a properties file. It also allows to assign a name to the job for identification, which is optional. \n", " \n", "To stop a job, sue `runbatch.sh -stop `. This command stops the batch job or chain gracefully." ] }, { "cell_type": "markdown", "id": "4987e88e", "metadata": {}, "source": [ "### Debugging and Troubleshooting Bash Scripts\n", "\n", "A valuable technique for debugging Bash scripts is to enable the `set -x` option at the start of the script. This activates debugging mode, making Bash print each command it runs to the terminal, prefixed by a `+` sign. This can be extremely useful for pinpointing where errors occur in the script.\n", "\n", "\n", " #!/bin/bash\n", " set -x\n", " # Your script goes here" ] }, { "cell_type": "markdown", "id": "1e258f26", "metadata": {}, "source": [ "When Bash encounters an error, it assigns an exit code to indicate the type of error. You can check the exit code of the most recent command using the `$?` variable. An exit code of `0` signifies success, while any other value indicates an error.\n", "\n", " #!/bin/bash\n", " # Your script goes here\n", " if [ $? -ne 0 ]; then\n", " echo \"Error occurred.\"\n", " fi" ] }, { "cell_type": "markdown", "id": "a9b39816", "metadata": {}, "source": [ "### References:\n", "\n", "1. Franklin, Arnab. \"Bash Scripting Tutorial – Linux Shell Script and Command Line for Beginners.\" FreeCodeCamp, 13 Jan. 2023, https://www.freecodecamp.org/news/bash-scripting-tutorial-linux-shell-script-and-command-line-for-beginners/.\n", "2. Mandelberg, Mike G., and Mitch Frazier. \"Bash Programming - Introduction HOWTO.\" The Linux Documentation Project, 2000, https://tldp.org/HOWTO/Bash-Prog-Intro-HOWTO.html#toc9." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.5" } }, "nbformat": 4, "nbformat_minor": 5 }