Differences

This shows you the differences between two versions of the page.

--- habrok:connecting_to_the_system:web_portal [2023/07/11 13:32] – [Hábrók Desktop] fokke
+++ habrok:connecting_to_the_system:web_portal [2023/11/08 08:50] (current) – [Jupyter Notebook] fokke
@@ Line 2: / Line 2: @@
 **
-''The Web Portal for Hábrók is now available, but some functionality is still lacking. For instance, the interactive apps for remote desktop, MATLAB, COMSOL, TensorFlow, and PyTorch are not available yet.''
+''The Web Portal for Hábrók is now available, but some functionality is still lacking. For instance, the interactive apps for TensorFlow and PyTorch are not available yet.''
 **
 ====== Web Portal ======
-We have implemented a new web portal for Hábrók, based on software developed at the [[https://www.osc.edu/|Ohio Supercomputer Center]], called [[http://openondemand.org/|OpenOnDemand]] (affectionately known as [[https://en.wikipedia.org/wiki/Ood|OOD]]). Using your username and password just as you would do on Hábrók, you can login to this portal at the following address:
+We have implemented a new web portal for Hábrók, based on software developed at the [[https://www.osc.edu/|Ohio Supercomputer Center]], called [[http://openondemand.org/|OpenOnDemand]] (affectionately known as [[https://en.wikipedia.org/wiki/Ood|OOD]]). Using your username and password, just as you would do on Hábrók, you can log in to this portal at the following address:
 https://portal.hb.hpc.rug.nl
@@ Line 14: / Line 14: @@
 ===== Logging in =====
-  - If you already have a Hábrók account, you can just go to [[https://portal.hb.hpc.rug.nl]] and click on the <html><span class="glyphicon glyphicon-log-in hidden-sm"></span></html> Login button at the top-right corner. You will be prompted to enter your username, password and MFA token, using the RUG single sign on page, after which you will be logged in to the Web Portal.
+  - If you already have a Hábrók account, you can just go to [[https://portal.hb.hpc.rug.nl]] and click on the <html><span class="glyphicon glyphicon-log-in hidden-sm"></span></html> Login button at the top-right corner. You will be prompted to enter your username, password, and MFA token, using the RUG single sign-on page, after which you will be logged in to the Web Portal.
   - If you don't have a Hábrók account, then go the [[https://iris.service.rug.nl|IRIS]] support system to request an account (more information at the [[..:introduction/policies|"Accounts and policies"]] page). Once your account has been added to Hábrók, follow from step 1 here to login.
 ===== The Dashboard =====
@@ Line 22: / Line 22: @@
 {{dashboard.png|}}
-Here, in addition the “Message of the Day” that you see when you ''ssh'' into Hábrók, we will also display messages relevant only to the web portal (such as any planned maintenance of the portal).
+Here, in addition to the “Message of the Day” that you see when you ''ssh'' into Hábrók, we will also display messages relevant only to the web portal (such as any planned maintenance of the portal).
 The menu has several items, the most useful ones being:
@@ Line 38: / Line 38: @@
 ===== The Files menu =====
-In the ''%%Files%%'' menu you have access to your ''%%/homeN%%'', ''%%/projects%%'' and ''%%/scratch%%'' folders, as well as any group folders in ''%%/scratch%%'' for groups you belong to, and which are names as ''%%hb-<group-name>%%''. When accessing any of the submenus in ''%%Files%%'', the corresponding folder will be opened in a file explorer in a new browser tab. From here you can navigate, perform file operations (copy, delete, etc.), view and even edit files (the file editor has syntax highlighting for a large number of text file formats).
+In the ''%%Files%%'' menu you have access to your home, ''%%/projects%%'' and ''%%/scratch%%'' folders, as well as any group folders in ''%%/scratch%%'' for groups you belong to, and which are named as ''%%hb-<group-name>%%''. When accessing any of the submenus in ''%%Files%%'', the corresponding folder will be opened in a file explorer in a new browser tab. From here you can navigate, perform file operations (copy, delete, etc.), view and even edit files (the file editor has syntax highlighting for a large number of text file formats).
 ===== The Jobs menu =====
-In the ''%%Jobs%%'' menu, there are two subitems: ''%%Active Jobs%%'' and ''%%Job Composer%%''. The ''%%Job Composer%%'' allows you to define a submit a job directly from the web portal, but at the moment it is a bit cumbersome and counter-intuitive in our opinion, so we do not recommend using it just yet. The ''%%Active Jobs%%'' provides and overview of jobs on the cluster, either for yourself or all users. It is similar to the ''%%squeue%%'' from the command line.
+In the ''%%Jobs%%'' menu, there are two subitems: ''%%Active Jobs%%'' and ''%%Job Composer%%''. The ''%%Job Composer%%'' allows you to define a submit a job directly from the web portal, but at the moment it is a bit cumbersome and counter-intuitive in our opinion, so we do not recommend using it just yet. The ''%%Active Jobs%%'' provides an overview of jobs on the cluster, either for yourself or all users. It is similar to the ''%%squeue%%'' from the command line.
 ===== The Clusters menu =====
@@ Line 52: / Line 52: @@
 This is where things become more interesting. At the moment we have a single interactive app (with more to come), which you can launch as jobs from this menu:
+==== Jupyter Notebook ====
-==== Servers ====
+This app will launch a Jupyter notebook on a compute node and will allow you to connect to it and run notebooks. It currently only supports Python 3.10.4.
-=== Jupyter Notebook ===
+You can now also choose which filesystem Jupyter starts in, through a dropdown box. Currently the only options are ''/scratch/$USER'' and any other group directory you have in ''/scratch''.
-This app will launch a Jupyter notebook on a compute node and will allow you to connect to it and run notebooks. It supports Python 3.6.4 (CUDA and non-CUDA), Python 3.7.2, 3.7.4, and R 3.6.1.
+=== Python  ===
+Before launching the Jupyter notebook, this interactive app loads the module: IPython (versions vary). Since this is just a basic Python, you might want to build you own virtual environments and have them available in the Jupyter notebook.
-== Python  ==
-Before launching the Jupyter notebook, this interactive app loads two modules: IPython and TensorFlow (versions vary). IPython already contains a lot of Python packages, including pandas, scipy, etc. Nevertheless, you might want to build you own virtual environments and have them available in the Jupyter notebook.
 **Building the Python virtual environment**
-We suggest you put your Python virtual environment on the ''/data'' filesystem, rather than ''/home'', since the number of files can be quite high, particularly if you have several virtual environments. Before building the virtual environment, you need to load the appropriate module:
+We suggest you put your Python virtual environment in your home directory, rather than ''/scratch'', since the number of files can be quite high, particularly if you have several virtual environments. Before building the virtual environment, you need to load the appropriate module:
 <code>
@@ Line 72: / Line 71: @@
 where ''<ipython_version>'' should be replaced by the appropriate IPython version. We support the following versions:
-  * ''<ipython_version> = IPython/6.4.0-foss-2018a-Python-3.6.4'' for ''Python 3.6.4'' and ''Python 3.6.4 CUDA''
+  * ''<ipython_version> = IPython/8.5.0-GCCcore-11.3.0'' for ''Python 3.10.4''
-  * ''<ipython_version> = IPython/7.7.0-foss-2019a-Python-3.7.2'' for ''Python 3.7.2''
+  * ''<ipython_version> = IPython/7.26.0-GCCcore-11.2.0'' for ''Python 3.8.16''
-  * ''<ipython_version> = IPython/7.9.0-foss-2019b-Python-3.7.4'' for ''Python 3.7.4''
-  * ''<ipython_version> = IPython/7.9.0-fosscuda-2019b-Python-3.7.4'' for ''Python 3.7.4 GPU''
-Now build a virtual environment in ''/data/$USER/.envs'':
+Now build a virtual environment in ''$HOME/venvs'':
 <code>
-python3 -m venv /data/$USER/.envs/my_env
+python3 -m venv $HOME/venvs/my_env
 </code>
 Activate the virtual environment you just built:
 <code>
-source /data/$USER/.envs/my_env/bin/activate
+source $HOME/venvs/my_env/bin/activate
+</code>
+It is advisable to upgrade the version of pip for you new virtual environment, to suppress the warning that you will get anytime you install a new package. Also installing or updating the ''wheels'' and ''setuptools'' modules is required.
+<code>
+pip install --upgrade pip wheel setuptools
 </code>
-Optionally, you can upgrade the version of pip for you new virtual environment, to suppress the warning that you will get anytime you install a new package:
+Before you are able to build a Jupyter Notebook kernel for your new virtual environment, there are a few additional packages that you need to install:
 <code>
-pip install --upgrade pip
+pip install tomlkit jupyter-contrib-core
 </code>
-Before your are able to build a Jupyter Notebook kernel for your new virtual environment, there are a few additional packages that you need to install. The precise list of packages will depend on which IPython module you loaded, so here are the different options:
-  * IPython/6.4.0-foss-2018a-Python-3.6.4: ''pip install decorator''
-  * IPython/7.7.0-foss-2019a-Python-3.7.2: ''pip install six decorator pygments python-dateutil''
-  * IPython/7.9.0-foss-2019b-Python-3.7.4: ''pip install six decorator pygments python-dateutil''
-  * IPython/7.9.0-fosscuda-2019b-Python-3.7.4: ''pip install six decorator pygments python-dateutil''
-There may be a few warnings about missing packages, so you can optionally install also the following: ''pip install jinja2 jupyter-contrib-core''. These are not strictly required, but we do suggest you install them nevertheless.
 Finally, you can generate a Jupyter Notebook kernel from this virtual environment:
@@ Line 106: / Line 98: @@
 </code>
-You can check that the kernel has been properly generated by having a look at its file, which is usually in ''$HOME/.local/share/jupyter/kernels/my_env/kernel.json''. If you see a reference to your virtual environment in this file, something like ''/data/$USER/.envs/my_env/bin/python'', this means that everything is properly setup.
+You can check that the kernel has been properly generated by having a look at its file, which is usually in ''$HOME/.local/share/jupyter/kernels/my_env/kernel.json''. If you see a reference to your virtual environment in this file, something like ''$HOME/venvs/my_env/bin/python'', this means that everything is properly set up.
-That's it! You have now access to the virtual environment you just built from the Jupyter Notebook, through a kernel named ''my_env'' (You can, of course, use any name you please).
+That's it! You now have access to the virtual environment you've just built from the Jupyter Notebook (note that you may need to reload the Jupyter web page), through a kernel named ''my_env'' (You can, of course, use any name you please).
 Any packages you install in this virtual environment will be available from the Jupyter Notebook immediately. We suggest you use a ''requirements.txt'' file to keep track of the packages you've installed.
-You can now also choose which filesystem Jupyter starts in, through a dropdown box. The options are ''/home/$USER/'', ''/data/$USER'', ''/scratch/$USER'' and any group directory you have in ''/data''.
+==== Hábrók Remote Desktop ====
-=== TensorFlow GPU ===
-This app is a special case of the Jupyter Notebook app, designed to make it easy for you to run TensorFlow in a Jupyter notebook on a GPU node.
-By default, there is one Jupyter kernel available, which has TensorFlow built in. In case you wish to use your own virtual environment, see the instructions below.
-== Building the Python virtual environment ==
-We suggest you put your Python virtual environment on the ''/data'' filesystem, rather than ''/home'', since the number of files can be quite high, particularly if you have several virtual environments. Before building the virtual environment, you need to load the appropriate modules:
-<code>
-module load IPython/7.9.0-fosscuda-2019b-Python-3.7.4
-</code>
-Now build a virtual environment in ''/data/$USER/.envs'':
-<code>
-python3 -m venv /data/$USER/.envs/my_env
-</code>
-Activate the virtual environment you just built:
-<code>
-source /data/$USER/.envs/my_env/bin/activate
-</code>
-Have a look at the section above on the [[habrok:connecting_to_the_system:web_portal#jupyter_notebook|Jupyter Notebook]] to install some extra packages.
-Generate a jupyter notebook kernel from this virtual environment:
-<code>
-python -m ipykernel install --user --name=my_env
-</code>
-Deactivate the virtual environment:
-<code>
-deactivate
-</code>
-You will now have access to the virtual environment you just built from the Jupyter notebook, through a kernel named ''my_env'' (You can, of course, use any name you please).
-Any packages you install in this virtual environment will be available from the Jupyter notebook immediately. We suggest you use a ''requirements.txt'' file to keep track of the packages you've installed.
-== Using the App ==
-Using the TensorFlow GPU app is very similar to using the generic Jupyter Notebook app. You can choose your TensorFlow version (only 2.3.1 so far),  which type of GPU you want to use (K40, V100), number of hours, cores, amount of memory and notebook root folder.
-=== PyTorch GPU ===
-This app is a special case of the Jupyter Notebook app, designed to make it easy for you to run PyTorch in a Jupyter notebook on a GPU node.
-By default, there is one Jupyter kernel available, which has PyTorch built in. In case you wish to use your own virtual environment, see the instructions below.
-== Building the Python virtual environment ==
-We suggest you put your Python virtual environment on the ''/data'' filesystem, rather than ''/home'', since the number of files can be quite high, particularly if you have several virtual environments. Before building the virtual environment, you need to load the appropriate modules:
-<code>
-module load IPython/7.9.0-fosscuda-2019b-Python-3.7.4
-</code>
-Now build a virtual environment in ''/data/$USER/.envs'':
-<code>
-python3 -m venv /data/$USER/.envs/my_env
-</code>
-Activate the virtual environment you just built:
-<code>
-source /data/$USER/.envs/my_env/bin/activate
-</code>
-Have a look at the section above on the [[habrok:connecting_to_the_system:web_portal#jupyter_notebook|Jupyter Notebook]] to install some extra packages.
-Generate a jupyter notebook kernel from this virtual environment:
-<code>
-python -m ipykernel install --user --name=my_env
-</code>
-Deactivate the virtual environment:
-<code>
-deactivate
-</code>
-You will now have access to the virtual environment you just built from the Jupyter notebook, through a kernel named ''my_env'' (You can, of course, use any name you please).
-Any packages you install in this virtual environment will be available from the Jupyter notebook immediately. We suggest you use a ''requirements.txt'' file to keep track of the packages you've installed.
-== Using the App ==
-Using the PyTorch GPU app is very similar to using the generic Jupyter Notebook app. You can choose your PyTorch version (only 1.6.0 so far),  which type of GPU you want to use (K40, V100), number of hours, cores, amount of memory and notebook root folder.
+The Remote Desktop app will launch a graphical desktop environment (based on ([[https://mate-desktop.org/|MATE]]) on a compute node, allowing you to interactively run tools with a graphical user interface with some more resources than what's available on the login/interactive nodes. By submitting to the ''gpu'' partition, you can even make use of a GPU and do OpenGL hardware rendering. The desktop environment itself has some tools like a web browser, text editors, and terminal available, but in a terminal you can also make use of all the modules. This would allow you to run tools like MATLAB, Mathematica, ParaView, or VMD.
+{{portal_remote_desktop.png?1600|}}
 ===== Testing =====