| Both sides previous revision Previous revision Next revision | Previous revision |
| dcc:itsol:whisper:setup [2025/10/17 08:56] – changed title giulio | dcc:itsol:whisper:setup [2026/09/25 10:19] (current) – alba |
|---|
| {{indexmenu_n>1}} | {{indexmenu_n>1}} |
| ===== Initial Setup ===== | |
| |
| ==== Setting up on HPC using the portal ==== | ====== Before you start ====== |
| |
| The guide below will take you through some basic steps to start running automatic transcription jobs on the High Performance Computing (HPC) cluster from a Windows computer. Before you can get started with setting Whisper up, you need to first [[https://portal.hb.hpc.rug.nl/public/start.html|Request a Hábrók (HPC) account or log into the HPC portal]] | Whisper runs on Hábrók and reads your recordings from one directory there, writing the transcripts to another. You need: |
| |
| {{ :dcc:itsol:whisper:hb_portal_1.png?direct&900 | }} | * a Hábrók account; |
| | * space in your Hábrók ''%%scratch%%'' directory for the recordings and the results; and |
| | * the language spoken in the recording, if you know it. |
| |
| Once you have an HPC account, you can navigate to the ''Files'' tab in the top menu bar and select ''Home Directory''. | ===== Get access ===== |
| |
| {{ :dcc:itsol:whisper:hb_portal_2.png?direct&900 | }} | **Request an account.** Staff members and PhD students request a P-account. Students request an S-account, which needs approval from their course, thesis, or project supervisor. Both request forms are on the [[https://portal.hb.hpc.rug.nl/public/start.html|Hábrók web portal start page]], and [[:habrok:introduction:policies|Accounts, access and policies]] explains who is eligible. You can skip this step if you already use Hábrók. |
| |
| In this window, you will be able to manage your files and folders in HPC. After you finish the Whisper installation, you can upload your audio files from here. You can also download the transcripts from this page, as well as remove any old audio files that have already been transcribed. To begin with, make sure that you are in the Home directory, then open the **terminal** as shown below. | **Log in to the web portal.** Once your account is active, open the [[https://portal.hb.hpc.rug.nl/public/start.html|start page]] and use the Login link in the top right corner. Everything in this guide happens in the portal. |
| |
| {{ :dcc:itsol:whisper:insta_portal.png?direct&900 | }} | {{ :dcc:itsol:whisper:login.webm?800x450 |}} |
| |
| The next steps explain how to create the Whisper environment you will be working in: | ===== Check your scratch space ===== |
| |
| * Create a folder “whisper_audio” in the root folder of your HPC session by inputting the following command: | The app reads recordings from and writes results to your ''%%scratch%%'' directory. The default quota is 250 GB. ''%%scratch%%'' is not backed up, so keep the results you need somewhere else. See [[:habrok:data_management:storage_areas|Storage areas]] for details. |
| * <code> mkdir $HOME/whisper_audio </code> {{ :dcc:itsol:whisper:insta_1a.png?direct&600 | }} | |
| * In the same way, you should also create an output folder in the same place by inputting: | |
| * <code> mkdir $HOME/whisper_output </code> | |
| * If this step of the installation succeeded, then you should be able to display the new folders in MobaXterm as shown in the figures below. {{ :dcc:itsol:whisper:insta_2a.png?direct&600 | }} | |
| * You can also use the HPC portal and its folder navigation window to check if the new folders are there. | |
| |
| ==== Building the virtual environment and installing Whisper ==== | ===== Create input and output directories ===== |
| |
| **Note: This step is only needed the first time you set up Whisper.** After you have installed the program for the first time, you can skip directly to the next part of the guide to run the program. | You only need to do this once. |
| |
| When logged into your session in the terminal, you will have a prompt where you can enter commands. In order to run Whisper, you will need to create the proper environment in your HPC session. To do so, copy the grey-highlighted lines below one by one into your terminal and run them separately by pressing enter. | - In the web portal, open the ''%%Files%%'' menu and choose your ''%%scratch%%'' directory, which is ''%%/scratch/%%'' followed by your username, for example ''%%/scratch/p123456%%''. The file browser opens. |
| | - Click ''%%New Directory%%'', type ''%%whisper%%'', and click ''%%OK%%''. |
| | - Click the new ''%%whisper%%'' directory to open it. |
| | - Create an ''%%input%%'' and an ''%%output%%'' directory inside it in the same way. |
| |
| **Note**: To copy text into the terminal, ''ctrl+V'' will not work. Use either the right mouse click, then select paste from the drop-down menu, or, if you have a mouse wheel, click on the terminal with the mouse wheel to paste the text directly after you copied it. | Your ''%%scratch%%'' directory now contains: |
| |
| **Steps to follow to install whisper**: | <code> |
| | whisper/ |
| | ├── input/ |
| | └── output/ |
| | </code> |
| |
| * First, you need to load a module that Whisper will need to run. To do so, copy-paste the line highlighted in grey below into the terminal, as shown in the figure. | Use these lowercase names. The form then proposes the right directories for you, inside the ''%%scratch%%'' directory you opened in step 1. If your recordings live elsewhere, for example in a group ''%%scratch%%'' directory, check the ''%%Folder for input files%%'' and ''%%Folder for output files%%'' fields when you fill in the form; see [[dcc:itsol:whisper:running#input_and_output_directories|Input and Output Directories]]. |
| * <code> module load PyTorch/2.1.2-foss-2023a-CUDA-12.1.1</code> {{ :dcc:itsol:whisper:insta_4a.png?direct&600 | }} | |
| * Then you need to create the virtual environment where you will install whisper. Copy-paste the line below into the terminal. | |
| * <code> python3 -m venv $HOME/.envs/whisper </code> {{ :dcc:itsol:whisper:insta_5a.png?direct&600 | }} | |
| * Now, activate the newly created environment by copy-pasting the line below. | |
| * <code> source $HOME/.envs/whisper/bin/activate </code> {{ :dcc:itsol:whisper:insta_6a.png?direct&600 | }} | |
| * Before you install Whisper, you need to make sure to have the latest version of some programs. Copy the two lines below separately into the terminal, as shown in the figures: | |
| * <code> pip install --upgrade pip </code> {{ :dcc:itsol:whisper:insta_7a.png?direct&600 | }} | |
| * <code> pip install --upgrade wheel </code> {{ :dcc:itsol:whisper:insta_8a.png?direct&600 | }} | |
| * Finally, you can install Whisper by running the command below: | |
| * <code> pip install git+https://github.com/openai/whisper.git </code> {{ :dcc:itsol:whisper:insta_9a.png?direct&600 | }} | |
| * If everything went well, you should see the following screen: {{ :dcc:itsol:whisper:insta_10a.png?direct&600 | }} | |
| * As a final step, type <code> deactivate </code> into the terminal, then press "enter". After this initial installation, you won't need to manually activate the whisper environment anymore. {{ :dcc:itsol:whisper:insta_11a.png?direct&600 | }} | |
| * If you wish to fully close the environment and also close the HPC session directly, type <code> exit </code> instead of <code> deactivate </code>. | |
| |
| | Put only the recordings for the current run in ''%%whisper/input%%''. **The app hands every entry in the input directory to Whisper, so any other file makes the job fail**. The job script is the usual cause for this: leave ''%%Script Location%%'' pointing at ''%%whisper/%%'', never at ''%%whisper/input%%'', because the script and the job log would then be treated as recordings. |
| |
| **Note**: The version numbers displayed in this guide for the programs you have installed and upgraded reflect the most recent versions at the time this guide was written. The numbers you will see displayed might have changed if newer versions have been released. | {{ :dcc:itsol:whisper:create_dirs.webm?800x450 |}} |
| |
| | ===== Upload recordings ===== |
| | |
| | - In the file browser, open ''%%whisper%%'' and then ''%%input%%''. |
| | - Click ''%%Upload%%''. |
| | - Drag the audio or video files into the window, or click ''%%browse files%%'' and select them. |
| | - Click the green ''%%Upload%%'' button, which shows the number of files, for example ''%%Upload 3 files%%''. |
| | - Wait until every upload has finished before you start the job. |
| | |
| | {{ :dcc:itsol:whisper:upload.webm?800x450 |}} |
| | |
| | <WRAP half column>[[dcc:itsol:whisper|← Previous: Whisper Guide]]</WRAP> |
| | <WRAP half column rightalign>[[dcc:itsol:whisper:running|Next: Run a Whisper Job →]]</WRAP> |
| | <WRAP clear></WRAP> |
| |
| [[dcc:itsol:whisper:scripts| → Move to the next step]] | |