| Both sides previous revision Previous revision Next revision | Previous revision |
| dcc:itsol:whisper:datamanage [2025/09/10 13:44] – Minor text changes alba | dcc:itsol:whisper:datamanage [2026/09/25 10:26] (current) – alba |
|---|
| {{indexmenu_n>4}} | {{indexmenu_n>5}} |
| ===== Data management safety measures ===== | |
| |
| Because you are handling data containing the **voices of (multiple) people**, your data is considered one of the most sensitive kinds of data. For this reason, data uploaded to HPC to perform the transcription should not be left idling on HPC. As soon as you are aware that the transcription has been performed, you should take steps to **download the results from HPC to your UG work environment and to remove all traces of the data from HPC**. | ====== Data management safety measures ====== |
| |
| There are **three main areas** that you need to clear to secure your data: | Because you are handling data containing the **voices of (multiple) people**, your data is considered one of the most sensitive kinds of data. For this reason, data uploaded to Hábrók to perform the transcription should not be left idling there. As soon as you know that the transcription has been performed, **download the results to your UG work environment and remove all traces of the data from Hábrók**. |
| * The **audio** that you provided on **input** | |
| * The **transcripts** that Whisper created | |
| * The **SLURM file** created by HPC to record the job details | |
| |
| Once all three of these locations/files are downloaded and cleared from HPC, you are **ready for a new round of transcriptions**. | Hábrók is a computing cluster, intended for the short-term storage of data you are actively working on. Scratch is not backed up, so anything you want to keep must be copied elsewhere first; deleting the only copy is irreversible, so check that your saved copy opens correctly before you remove anything. |
| |
| Last but not least, and independently of the sensitivity of your data, **HPC is a computing cluster** and therefore only intended for the **short-term storage of mutable data**. In order to ensure proper performance, data should not be stored long-term in the cluster. | Three things have to be cleared before you are done: |
| |
| ==== Input Audio ==== | ^What ^Where ^ |
| | |The recordings you uploaded |''%%whisper/input/%%'' | |
| | |The transcripts Whisper created|''%%whisper/output/%%'' | |
| | |The job log |''%%whisper/slurm-<JobID>.out%%''| |
| |
| The files contained in the folder ''whisper_audio'' need to be **removed before you can run a new transcription**. This is because the script is designed in such a way that it will transcribe anything that is located in the ''whisper_audio'' folder. **If you do not remove old audio data, Whisper will transcribe that audio again**, potentially running out of time to finish the transcription job. | Once all three are downloaded and cleared, you are ready for a new round of transcriptions. |
| |
| Before removing the audio files, we advise you to first check if the transcripts are acceptable. Should you have to run the transcription again with a modified script (i.e., to force a language that Whisper did not automatically identify), then having the audio still on HPC will save you time. | ===== The recordings ===== |
| |
| If the transcripts are what you expect them to be, however, then the audio should be removed promptly. Please consider doing a brief check of the transcripts, rather than going through them line by line. You can check the details of the transcription in your work environment directly later on. | Check the transcripts before you delete the audio. If a job has to be run again — because the wrong language was detected, for example — having the recordings still on Hábrók saves you from uploading them a second time. A quick check that the transcription succeeded is enough; the detailed reading can happen in your own work environment. |
| |
| ==== Output Text ==== | Once the transcripts are what you expect, remove the recordings promptly. Leaving transcribed audio on the cluster serves no purpose and increases the risk of a data leak. |
| |
| The files contained in the folder ''whisper_output'' can be removed at any time, even after a new transcription job has been launched. We do advise you, however, to **clean your HPC environment completely, before running a new transcription**. | ===== The transcripts ===== |
| |
| The transcripts created by Whisper come in five different formats: | Download everything you need from ''%%whisper/output/%%'' and then empty it. The output files can be removed at any time, but clear them before starting a new job so that one run's results are never mistaken for another's. |
| * ''txt'' → This is most likely the format you will want to work in. It is a **simple text format**, with only the transcription recorded and minimal line formatting done. | |
| * ''srt'' → This file is useful if you want to display **subtitles** in a video containing the audio you transcribed. | |
| * ''vtt'' → This is the //Video Text to Track// file. It contains the text but also the **time stamps** of the transcription. | |
| * ''tsv'' → This is a simple //Tab Separated Value// table. It is mostly equivalent to the ''txt'' file. | |
| * ''json'' → This file is **machine readable** and can come in handy if you need to pass the transcription on to a different program. | |
| |
| ==== Job Information File ==== | ===== The job log ===== |
| |
| Finally, there is one last file that needs to be removed before you are done cleaning your HPC environment. In your HOME folder (the one you are directed to when you connect to HPC), there is a file called: | The job log sits next to the job script, in the ''%%whisper/%%'' directory rather than in the output directory, and is named after the job: ''%%slurm-<JobID>.out%%''. |
| * ''slurm-<jobID>.out''. | |
| |
| This file is created by HPC when you launch a job, and it is tagged with the ''jobID'' displayed when executing the script. It is used to record what happened while the job was running. Apart from the information on the status of the job and how it was completed, HPC also records the actual transcription here. This means that **the transcription of your audio can also be read by displaying this file**. In order to ensure that all data related to your audio is removed from HPC, these files need to be deleted as well. | It records what Whisper did — the model it loaded, the files it processed, and any errors. It does not contain the transcript itself. It does contain the file names of your recordings, which can be identifying on their own, so delete it along with the rest. |
| |
| **Note**: If you were curious, SLURM stands for //Simple Linux Utility for Resource Management//. | <WRAP half column>[[dcc:itsol:whisper:troubleshooting|← Previous: Troubleshooting]]</WRAP> |
| | <WRAP half column rightalign>[[dcc:itsol:start|Return to DCC IT Solutions Guides →]]</WRAP> |
| | <WRAP clear></WRAP> |
| |
| [[dcc:itsol:start | → Return to DCC IT Solutions Guides]] | |