{"type":"doc","content":[{"type":"panel","attrs":{"panelType":"warning"},"content":[{"type":"paragraph","content":[{"text":"Introduction","type":"text","marks":[{"type":"strong"}]},{"text":": The workshop session will provide a quick tour covering high-level concepts, commands and processes for using Linux and HPC on our MedicineBow cluster. It will cover enough to allow an attendee to access the cluster and to perform analysis associated with this workshop. ","type":"text"}]}]},{"type":"panel","attrs":{"panelType":"note"},"content":[{"type":"paragraph","content":[{"text":"Goals","type":"text","marks":[{"type":"strong"}]},{"text":":","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Introduce ARCC and what types of services we provide including “what is HPC?”","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Define “what is a cluster”, and how is it made of partitions and compute nodes.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"How to access and start using ARCC’s MedicineBow cluster - using our OnDemand service.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"How to start an interactive desktop and open a terminal to use Linux commands within.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Introduce the basics of Linux, the command-line, and how its File System looks on MedicineBow.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Introduce Linux commands to allow navigation and file/folder manipulation.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Introduce Linux commands to allow text files to be searched and manipulated.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Introduce using a command-line text-editor and an alternative GUI based application.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"How to setup a Linux environment to use R(/Python) and start RStudio, by loading modules.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"How to start interactive sessions to run on a compute node, to allow computation, requesting appropriate resources.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"How to put elements together to construct a workflow that can be submitted as a job to the cluster, which can then be monitored.","type":"text"}]}]}]}]},{"type":"rule"},{"type":"heading","attrs":{"level":2},"content":[{"text":"We will not be covering:","type":"text"}]},{"type":"panel","attrs":{"panelType":"warning"},"content":[{"type":"paragraph","content":[{"text":"We will not covering, but workshops are available, on:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Using a terminal to SSH onto the Cluster - see ","type":"text"},{"text":"Intro to Accessing the Cluster","type":"text","marks":[{"type":"link","attrs":{"href":"https://arccwiki.atlassian.net/wiki/spaces/DOCUMENTAT/pages/2184937473"}}]},{"text":".","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Data Management","type":"text","marks":[{"type":"link","attrs":{"href":"https://arccwiki.atlassian.net/wiki/spaces/DOCUMENTAT/pages/2103312388"}}]},{"text":" nor ","type":"text"},{"text":"Data Transfer","type":"text","marks":[{"type":"link","attrs":{"href":"https://arccwiki.atlassian.net/wiki/spaces/DOCUMENTAT/pages/2173599767"}}]},{"text":" (such as using Globus).","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Using / Creating ","type":"text"},{"text":"Conda Environments","type":"text","marks":[{"type":"link","attrs":{"href":"https://arccwiki.atlassian.net/wiki/spaces/DOCUMENTAT/pages/2135883799"}}]},{"text":" - one method for installing your own software.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Using the ","type":"text"},{"text":"Jupyter Service","type":"text","marks":[{"type":"link","attrs":{"href":"https://arccwiki.atlassian.net/wiki/spaces/DOCUMENTAT/pages/2205646897"}}]},{"text":" via OnDemand.","type":"text"}]}]}]}]},{"type":"rule"},{"type":"heading","attrs":{"level":1},"content":[{"text":"Sections","type":"text"}]},{"type":"extension","attrs":{"layout":"default","extensionType":"com.atlassian.confluence.macro.core","extensionKey":"toc","parameters":{"macroParams":{"minLevel":{"value":"1"},"maxLevel":{"value":"1"},"outline":{"value":"false"},"style":{"value":"none"},"type":{"value":"list"},"printable":{"value":"true"}},"macroMetadata":{"macroId":{"value":"9afd11da-8be2-4637-b8ff-a1737db71f32"},"schemaVersion":{"value":"1"},"title":"Table of Contents"}},"localId":"be7c9559-a869-43e5-a0a2-2a577722c3c8"}},{"type":"rule"},{"type":"rule"},{"type":"heading","attrs":{"level":1},"marks":[{"type":"alignment","attrs":{"align":"center"}}],"content":[{"text":"*** Class 01 ***","type":"text"}]},{"type":"rule"},{"type":"rule"},{"type":"heading","attrs":{"level":1},"content":[{"text":"00 Introduction and Setting the Scope: ","type":"text"}]},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"paragraph","content":[{"text":"HPC Skills to Learn","type":"text","marks":[{"type":"link","attrs":{"href":"https://arccwiki.atlassian.net/wiki/spaces/DOCUMENTAT/pages/2156527636/Workshops+and+Tutorials#HPC-Skills-to-Learn"}}]},{"text":": The roadmap to becoming a proficient HPC user can be long, complicated, and varies depending on the user. ","type":"text"}]}]},{"type":"panel","attrs":{"panelType":"warning"},"content":[{"type":"paragraph","content":[{"text":"Typically what we’re going to cover over the next two classes we’d use two full days. ","type":"text"}]},{"type":"paragraph","content":[{"text":"So bear in mind that we’ll be introducing key high-level concepts, with not as much time for questions/exercises that we’d normally provide.","type":"text"}]},{"type":"paragraph","content":[{"text":"The classes will be more hands-on demonstrations for you to listen to, follow along where you can - but you’ll need to and be expected to work through these in your own time.","type":"text"}]},{"type":"paragraph","content":[{"text":"More extensive and in-depth information and walkthroughs are available on our wiki and under workshops/tutorials. You are welcome to dive into those in your own time. Content within them will provide you with a lot more detail and examples of the foundational concepts you would need to be familiar with to become a proficient HPC user.","type":"text"}]}]},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"paragraph","content":[{"text":"Links","type":"text","marks":[{"type":"strong"}]},{"text":":","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"General Wiki","type":"text","marks":[{"type":"link","attrs":{"href":"https://arccwiki.atlassian.net/wiki/spaces/DOCUMENTAT"}}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Workshops and Tutorials","type":"text","marks":[{"type":"link","attrs":{"href":"https://arccwiki.atlassian.net/wiki/spaces/DOCUMENTAT/pages/2156527636"}}]}]}]}]}]},{"type":"rule"},{"type":"heading","attrs":{"level":1},"content":[{"text":"01 About UW ARCC and HPC","type":"text"}]},{"type":"panel","attrs":{"panelType":"note"},"content":[{"type":"paragraph","content":[{"text":"Goals: ","type":"text","marks":[{"type":"strong"}]}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Describe ARCC’s role at UW.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Provide resources for ARCC Researchers to seek help.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Introduce staff members, including those available throughout the workshop.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Introduce the concept of an HPC cluster, it’s architecture and when to use one.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Introduce the MedicineBow HPC architecture, hardware, and partitions.","type":"text"}]}]}]}]},{"type":"rule"},{"type":"heading","attrs":{"level":2},"content":[{"text":"About ARCC and how to reach us","type":"text"}]},{"type":"panel","attrs":{"panelType":"error"},"content":[{"type":"paragraph","content":[{"text":"Wiki Front Page: About ARCC","type":"text","marks":[{"type":"strong"},{"type":"link","attrs":{"href":"https://arccwiki.atlassian.net/wiki/spaces/DOCUMENTAT"}}]}]}]},{"type":"extension","attrs":{"layout":"default","extensionType":"com.atlassian.confluence.macro.core","extensionKey":"excerpt-include","parameters":{"macroParams":{"":{"value":"ARCC Wiki"},"name":{"value":"AboutARCC"}},"macroMetadata":{"macroId":{"value":"391c877e-43fc-4540-8b43-274713815948"},"schemaVersion":{"value":"1"},"title":"Insert excerpt"}},"localId":"e8df238a-9a20-4b16-9baf-888b0890991b"}},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"In short, we maintain internally housed scientific resources including more than one HPC Cluster, data storage, and several research computing servers and resources.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"We are here to assist UW researchers like yourself with your research computing needs. ","type":"text"}]}]}]}]},{"type":"panel","attrs":{"panelIconId":"atlassian-info","panelIcon":":info:","panelColor":"#E3FCEF","panelType":"custom"},"content":[{"type":"paragraph","content":[{"text":"Exercise","type":"text","marks":[{"type":"strong"}]},{"text":": Navigate to our ","type":"text"},{"text":"Service Portal","type":"text","marks":[{"type":"link","attrs":{"href":"https://arccwiki.atlassian.net/servicedesk/customer/portals"}}]},{"text":" and submit a ","type":"text"},{"text":"General Research Computing Support","type":"text","marks":[{"type":"strong"}]},{"text":" question.","type":"text"}]},{"type":"paragraph","content":[{"text":"Under the ","type":"text"},{"text":"Please further describe your issue","type":"text","marks":[{"type":"strong"}]},{"text":" section make sure to enter the work “Test”.","type":"text"}]}]},{"type":"rule"},{"type":"heading","attrs":{"level":2},"content":[{"text":"What is HPC","type":"text"}]},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"paragraph","content":[{"text":"HPC stands for ","type":"text"},{"text":"High Performance Computing","type":"text","marks":[{"type":"textColor","attrs":{"color":"#bf2600"}},{"type":"strong"}]},{"text":" and is one of UW ARCC’s core services. HPC is the practice of aggregating computing power in a way that delivers a much higher performance than one could get out of a typical desktop or workstation. HPC is commonly used to solve large problems, and has some common use cases: ","type":"text"}]},{"type":"orderedList","attrs":{"order":1},"content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Performing computation-intensive analyses on large datasets: MB/GB/TB in a single or many files, computations requiring RAM in excess of what is available on a single workstation, or analysis performed across multiple CPUs (cores) or GPUs.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Performing long, large-scale simulations: Hours, days, weeks, spread across multiple nodes each using multiple cores.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Running repetitive tasks in parallel: 10s/100s/1000s of small short tasks.","type":"text"}]}]}]}]},{"type":"table","attrs":{"layout":"default","width":1297.0,"localId":"2bfb2cc9-cc99-4479-84c5-89fbfd7a08ca"},"content":[{"type":"tableRow","content":[{"type":"tableHeader","attrs":{"colspan":1,"background":"#ffffff","rowspan":1,"colwidth":[407.0]},"content":[{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Users log in from their clients (desktops, laptops, workstations) into a login node. ","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"In an HPC Cluster, each compute node can be thought of as it’s own desktop, but the hardware resources of the cluster are available collectively as a single system.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":" Users may request specific allocations of resources available on the cluster - beyond that of a single node.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Allocated resources may include CPUs (Cores), Nodes, RAM/Memory, GPUs, etc.","type":"text"}]}]}]}]},{"type":"tableHeader","attrs":{"colspan":1,"background":"#ffffff","rowspan":1,"colwidth":[890.0]},"content":[{"type":"mediaSingle","attrs":{"layout":"align-start","width":875,"widthType":"pixel"},"content":[{"type":"media","attrs":{"width":1411,"id":"17a3eb13-834f-45a9-a7cb-668675f6c262","collection":"contentId-2319876097","type":"file","height":689}}]}]}]}]},{"type":"rule"},{"type":"heading","attrs":{"level":2},"content":[{"text":"What is a Compute Node?","type":"text"}]},{"type":"mediaSingle","attrs":{"layout":"center","width":400,"widthType":"pixel"},"content":[{"type":"media","attrs":{"width":829,"id":"11c181e7-720d-40c9-b45b-ab6db085e238","collection":"contentId-2319876097","type":"file","height":916}}]},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"We typically have multiple users ","type":"text"},{"text":"independently","type":"text","marks":[{"type":"textColor","attrs":{"color":"#bf2600"}},{"type":"strong"}]},{"text":" running jobs concurrently across compute nodes - ","type":"text"},{"text":"multi-tentancy","type":"text","marks":[{"type":"textColor","attrs":{"color":"#bf2600"}},{"type":"strong"}]},{"text":".","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Resources are shared, but ","type":"text"},{"text":"do not interfere with any one else’s resources","type":"text","marks":[{"type":"textColor","attrs":{"color":"#bf2600"}},{"type":"strong"}]},{"text":".","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"i.e. you have your own cores, your own block of memory. ","type":"text"}]}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"If someone else’s job fails it does NOT affect yours.","type":"text"}]}]}]}]},{"type":"rule"},{"type":"heading","attrs":{"level":2},"content":[{"text":"Homogeneous vs Heterogeneous HPCs","type":"text"}]},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"paragraph","content":[{"text":"There are 2 types of HPC systems:","type":"text"}]},{"type":"orderedList","attrs":{"order":1},"content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Homogeneous","type":"text","marks":[{"type":"textColor","attrs":{"color":"#bf2600"}},{"type":"strong"}]},{"text":": All compute nodes in the system share the same architecture. CPU, memory, and storage are the same across the system. (Ex: NWSC’s Derecho)","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Heterogeneous","type":"text","marks":[{"type":"textColor","attrs":{"color":"#bf2600"}},{"type":"strong"}]},{"text":": The compute nodes in the system can vary architecturally with respect to CPU, memory, even storage, and whether they have GPUs or not. Usually, the nodes are grouped in partitions. ","type":"text"},{"text":"MedicineBow is a heterogeneous cluster","type":"text","marks":[{"type":"textColor","attrs":{"color":"#bf2600"}},{"type":"strong"}]},{"text":".","type":"text"}]}]}]}]},{"type":"rule"},{"type":"heading","attrs":{"level":2},"content":[{"text":"Cluster: Heterogeneous: Partitions","type":"text"}]},{"type":"mediaSingle","attrs":{"layout":"center","width":750,"widthType":"pixel"},"content":[{"type":"media","attrs":{"width":1924,"id":"d8dad389-972c-49f6-8eee-dbe04061f3f4","collection":"contentId-2319876097","type":"file","height":809}}]},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"paragraph","content":[{"text":"MedicineBow Hardware Summary Table","type":"text","marks":[{"type":"link","attrs":{"href":"https://arccwiki.atlassian.net/wiki/spaces/DOCUMENTAT/pages/2117402626"}}]},{"text":": Understand what resources are available.","type":"text"}]}]},{"type":"rule"},{"type":"heading","attrs":{"level":1},"content":[{"text":"02 Using OnDemand to access the MedicineBow HPC Cluster","type":"text"}]},{"type":"panel","attrs":{"panelType":"note"},"content":[{"type":"paragraph","content":[{"text":"Goals:","type":"text","marks":[{"type":"strong"}]}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Demonstrate how users log into OnDemand. ","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Demonstrate requesting and using a XFCE Desktop Session","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Introduce the Linux File System and how it compares to common workstation environments ","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Introduce HPC specific directories and how they’re used","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Introduce MedicineBow specific directories and how they’re used","type":"text"}]}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Demonstrate how to access files using the MedicineBow File Browsing Application","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Demonstrate the use of emacs, available as a GUI based text-editor","type":"text"}]}]}]}]},{"type":"rule"},{"type":"heading","attrs":{"level":2},"content":[{"text":"Log in and Access the Cluster","type":"text"}]},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"paragraph","content":[{"text":"Open OnDemand Dashboard","type":"text","marks":[{"type":"link","attrs":{"href":"https://arccwiki.atlassian.net/wiki/spaces/DOCUMENTAT/pages/2209579064"}}]},{"text":": This service allows users to access MedicineBow cluster over a web-based portal, via a browser.","type":"text"}]}]},{"type":"panel","attrs":{"panelIconId":"atlassian-info","panelIcon":":info:","panelColor":"#E3FCEF","panelType":"custom"},"content":[{"type":"paragraph","content":[{"text":"Exercise","type":"text","marks":[{"type":"strong"}]},{"text":": Open an ","type":"text"},{"text":"Interactive Desktop","type":"text","marks":[{"type":"link","attrs":{"href":"https://arccwiki.atlassian.net/wiki/spaces/DOCUMENTAT/pages/2209448009/Interactive+Applications#Interactive-Desktop"}}]},{"text":": ","type":"text"}]}]},{"type":"rule"},{"type":"heading","attrs":{"level":2},"content":[{"text":"Structure of the Linux File System and HPC Directories","type":"text"}]},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"paragraph","content":[{"text":"From within the Interactive Desktop:","type":"text"}]}]},{"type":"panel","attrs":{"panelIconId":"atlassian-info","panelIcon":":info:","panelColor":"#E3FCEF","panelType":"custom"},"content":[{"type":"paragraph","content":[{"text":"Linux File Structure","type":"text","marks":[{"type":"strong"}]},{"text":": Double click on the ","type":"text"},{"text":"Home","type":"text","marks":[{"type":"strong"}]},{"text":" icon, and then ","type":"text"},{"text":"File System","type":"text","marks":[{"type":"strong"}]},{"text":".","type":"text"}]},{"type":"paragraph","content":[{"text":"This is specific to the MedicineBow HPC but most Linux environments will look very similar:","type":"text"}]}]},{"type":"mediaSingle","attrs":{"layout":"center","width":920,"widthType":"pixel"},"content":[{"type":"media","attrs":{"width":920,"alt":"image-20240916-002145.png","id":"f8b69dc2-3343-4dbb-aabc-615adbafb196","collection":"contentId-2319876097","type":"file","height":543}}]},{"type":"rule"},{"type":"heading","attrs":{"level":3},"content":[{"text":"Linux Operating Systems (Generally)","type":"text"}]},{"type":"mediaSingle","attrs":{"layout":"align-start","width":384,"widthType":"pixel"},"content":[{"type":"media","attrs":{"width":384,"id":"2dc05d57-e3c5-40e3-917d-bcfd75676f09","collection":"contentId-2319876097","type":"file","height":461}}]},{"type":"rule"},{"type":"heading","attrs":{"level":3},"content":[{"text":"Compare and Contrast: Linux, HPC Specific, MedicineBow Specific ","type":"text"}]},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"paragraph","content":[{"text":"Based on: ","type":"text"},{"type":"inlineCard","attrs":{"url":"https://arccwiki.atlassian.net/wiki/spaces/DOCUMENTAT/pages/2117337115"}},{"text":" ","type":"text"}]}]},{"type":"panel","attrs":{"panelType":"warning"},"content":[{"type":"paragraph","content":[{"text":"The project name for this class is: ","type":"text"},{"text":"genomicdatasci","type":"text","marks":[{"type":"strong"}]}]}]},{"type":"panel","attrs":{"panelIconId":"atlassian-info","panelIcon":":info:","panelColor":"#E3FCEF","panelType":"custom"},"content":[{"type":"paragraph","content":[{"text":"Exercise","type":"text","marks":[{"type":"strong"}]},{"text":": File Browsing in OnDemand GUI: ","type":"text"},{"text":"The Files Category and App","type":"text","marks":[{"type":"link","attrs":{"href":"https://arccwiki.atlassian.net/wiki/spaces/DOCUMENTAT/pages/2209546295/Applications+in+Open+OnDemand#The-Files-Category-and-App"}}]}]}]},{"type":"rule"},{"type":"heading","attrs":{"level":1},"content":[{"text":"03 Using Linux and the Command Line","type":"text"}]},{"type":"panel","attrs":{"panelType":"note"},"content":[{"type":"paragraph","content":[{"text":"Goals","type":"text","marks":[{"type":"strong"}]},{"text":":","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Introduce the shell terminal and command line interface","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Demonstrate starting a MedicineBow SSH shell using OnDemand","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Demonstrate information provided in a command prompt","type":"text"}]}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Introduce Policy for HPC Login Nodes","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Demonstrate how to navigate the file system to create and remove files and folders using command line interface (CLI)","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"mkdir","type":"text","marks":[{"type":"code"}]},{"text":", ","type":"text"},{"text":"cd","type":"text","marks":[{"type":"code"}]},{"text":", ","type":"text"},{"text":"ls","type":"text","marks":[{"type":"code"}]},{"text":", ","type":"text"},{"text":"mv","type":"text","marks":[{"type":"code"}]},{"text":", ","type":"text"},{"text":"cp","type":"text","marks":[{"type":"code"}]}]}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Demonstrate the use of ","type":"text"},{"text":"man","type":"text","marks":[{"type":"code"}]},{"text":", ","type":"text"},{"text":"--help","type":"text","marks":[{"type":"code"}]},{"text":" and identify when these should be used","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Demonstrate using a command-line text editor, ","type":"text"},{"text":"vi","type":"text","marks":[{"type":"code"}]}]}]}]}]},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"paragraph","content":[{"text":"Based on Workshop","type":"text","marks":[{"type":"strong"}]},{"text":": ","type":"text"},{"text":"Intro to Linux Command-Line: The File System","type":"text","marks":[{"type":"link","attrs":{"href":"https://arccwiki.atlassian.net/wiki/spaces/DOCUMENTAT/pages/2185166869"}}]}]}]},{"type":"rule"},{"type":"heading","attrs":{"level":2},"content":[{"text":"Exercise: Shell Terminal Introducing Command Line","type":"text"}]},{"type":"panel","attrs":{"panelIconId":"atlassian-info","panelIcon":":info:","panelColor":"#E3FCEF","panelType":"custom"},"content":[{"type":"paragraph","content":[{"text":"Getting Started: ","type":"text","marks":[{"type":"strong"}]},{"text":"Using the OnDemand service","type":"text"},{"text":": ","type":"text","marks":[{"type":"strong"}]},{"text":"Using the Terminal","type":"text","marks":[{"type":"link","attrs":{"href":"https://arccwiki.atlassian.net/wiki/spaces/DOCUMENTAT/pages/2184413241"}}]},{"text":":","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Start MedicineBow Shell Access","type":"text","marks":[{"type":"link","attrs":{"href":"https://arccwiki.atlassian.net/wiki/spaces/DOCUMENTAT/pages/2184413241/Using+the+Terminal#Start-MedicineBow-Shell-Access"}}]},{"text":": ","type":"text"}]}]}]}]},{"type":"rule"},{"type":"heading","attrs":{"level":2},"content":[{"text":"What am I Using?","type":"text"}]},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"paragraph","content":[{"text":"Remember","type":"text","marks":[{"type":"strong"}]},{"text":":","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"The MedicineBow Shell Access opens up a new browser tab that is ","type":"text"},{"text":"running on a login node","type":"text","marks":[{"type":"textColor","attrs":{"color":"#bf2600"}},{"type":"strong"}]},{"text":". Do not run any computation on these.","type":"text"},{"type":"hardBreak"},{"text":"[@mblog1/2 ~]$","type":"text","marks":[{"type":"code"}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"The OnDemand Interactive Desktop (terminal) is already ","type":"text"},{"text":"running on a compute node","type":"text","marks":[{"type":"textColor","attrs":{"color":"#bf2600"}},{"type":"strong"}]},{"text":".","type":"text"},{"type":"hardBreak"},{"text":"[@mbcpu-001 ~]$","type":"text","marks":[{"type":"code"}]}]}]}]}]},{"type":"rule"},{"type":"heading","attrs":{"level":2},"content":[{"text":"Login Node Policy","type":"text"}]},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"paragraph","content":[{"text":"As a courtesy to your colleagues, please do not run the following on any login nodes: ","type":"text"}]},{"type":"orderedList","attrs":{"order":1},"content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Anything compute-intensive (tasks using significant computational/hardware resources - Ex: using 100% cluster CPU)","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Any collection of a large # of tasks resulting in a similar hardware footprint to actions mentioned previously. ","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Either start an Interactive Desktop, an interactive session (","type":"text"},{"text":"salloc","type":"text","marks":[{"type":"code"}]},{"text":") or submit a job (","type":"text"},{"text":"sbatch","type":"text","marks":[{"type":"code"}]},{"text":") ","type":"text"},{"text":"These will be covered later","type":"text","marks":[{"type":"textColor","attrs":{"color":"#bf2600"}},{"type":"strong"},{"type":"em"}]},{"text":".","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"See more on our ","type":"text"},{"text":"ARCC HPC Policies","type":"text","marks":[{"type":"link","attrs":{"href":"https://arccwiki.atlassian.net/wiki/spaces/DOCUMENTAT/pages/983114"}}]},{"text":".","type":"text"}]}]}]}]},{"type":"rule"},{"type":"heading","attrs":{"level":2},"content":[{"text":"Demonstrating how to get help in CLI","type":"text"}]},{"type":"table","attrs":{"layout":"default","width":1302.0,"localId":"cec1ea4b-28b3-48bf-83fc-3e1653aa192b"},"content":[{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[376.0]},"content":[{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"man","type":"text","marks":[{"type":"code"}]},{"text":" - Short for the manual page. This is an interface to view the reference manual for the application or command. ","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"man pages are only available on the login nodes.","type":"text"}]}]}]},{"type":"paragraph"}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[923.0]},"content":[{"type":"codeBlock","content":[{"text":"[@mblog1 ~]$ man pwd\nNAME\n pwd - print name of current/working directory\nSYNOPSIS\n pwd [OPTION]...\nDESCRIPTION\n Print the full filename of the current working directory.\n -L, --logical\n use PWD from environment, even if it contains symlinks\n -P, --physical\n avoid all symlinks\n --help display this help and exit\n --version\n output version information and exit\n If no option is specified, -P is assumed.\n NOTE: your shell may have its own version of pwd, which usually supersedes the version described here. Please refer to your shell's documentation\n for details about the options it supports.","type":"text"}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[376.0]},"content":[{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"--help","type":"text","marks":[{"type":"code"}]},{"text":" - a built-in command in shell. It accepts a text string as the command line argument and searches the supplied string in the shell's documents. ","type":"text"}]}]}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[923.0]},"content":[{"type":"codeBlock","content":[{"text":"[@mblog1 ~]$ cp --help\nUsage: cp [OPTION]... [-T] SOURCE DEST\n or: cp [OPTION]... SOURCE... DIRECTORY\n or: cp [OPTION]... -t DIRECTORY SOURCE...\nCopy SOURCE to DEST, or multiple SOURCE(s) to DIRECTORY.","type":"text"}]}]}]}]},{"type":"rule"},{"type":"heading","attrs":{"level":2},"content":[{"text":"Demonstrating file navigation in CLI","type":"text"}]},{"type":"table","attrs":{"layout":"default","width":1302.0,"localId":"fa08fbbe-4b16-43a0-96fc-ed3a812937ff"},"content":[{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[449.0]},"content":[{"type":"paragraph","content":[{"text":"File Navigation demonstrating the use of: ","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"pwd","type":"text","marks":[{"type":"code"}]},{"text":" (Print Working Directory)","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"ls","type":"text","marks":[{"type":"code"}]},{"text":" (“List” lists information about directories and any type of files in the working directory)","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"ls","type":"text","marks":[{"type":"code"}]},{"text":" flags","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"-l","type":"text","marks":[{"type":"code"}]},{"text":" (tells the mode, # of links, owner, group, size (in bytes), and time of last modification for each file)","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"-a","type":"text","marks":[{"type":"code"}]},{"text":" (Lists all entries in the directory, including the entries that begin with a ","type":"text"},{"text":".","type":"text","marks":[{"type":"strong"}]},{"text":" which are hidden)","type":"text"}]}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"cd","type":"text","marks":[{"type":"code"}]},{"text":" (Change Directory)","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"cd ..","type":"text","marks":[{"type":"code"}]},{"text":" (Change Directory - up one level)","type":"text"}]}]}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[851.0]},"content":[{"type":"codeBlock","content":[{"text":"[@mblog1 ~]$ pwd\n/home/\n[@mblog1 ~]$ ls\nDesktop Documents Downloads ondemand R\n[@mblog1 ~]$ cd /project/genomicdatasci\n[@mblog1 genomicdatasci]$ pwd\n/project/genomicdatasci\n[@mblog1 genomicdatasci]$ cd \n[@mblog1 ]$ ls -la\ntotal 2.0K\ndrwxr-sr-x 2 genomicdatasci 4.0K May 23 11:05 .\ndrwxrws--- 80 root genomicdatasci 4.0K Jun 4 14:39 ..\n[@mblog1 ]$ pwd\n/project/genomicdatasci/\n[@mblog1 ]$ cd ..\n[@mblog1 genomicdatasci]$ pwd\n/project/genomicdatasci","type":"text"}]}]}]}]},{"type":"rule"},{"type":"heading","attrs":{"level":2},"content":[{"text":"Demonstrating how to create and remove files and folders using CLI","type":"text"}]},{"type":"table","attrs":{"layout":"default","width":1310.0,"localId":"cceb4e02-8cb6-4e0c-81a6-10c25862236f"},"content":[{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[468.0]},"content":[{"type":"paragraph","content":[{"text":"Creating, moving and copying files and folders:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"touch","type":"text","marks":[{"type":"code"}]},{"text":" (Used to create a file without content. The file created using the touch command is empty)","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"mkdir","type":"text","marks":[{"type":"code"}]},{"text":" (Make Directory - to create an empty directory)","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"mv","type":"text","marks":[{"type":"code"}]},{"text":" (Move - moves a file or directory from one location to another)","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"cd..","type":"text","marks":[{"type":"code"}]},{"text":" (Change Directory - up one level)","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"cp","type":"text","marks":[{"type":"code"}]},{"text":" (Copy - copies a file or directory from one location to another)","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"-r","type":"text","marks":[{"type":"code"}]},{"text":" flag (Recursive)","type":"text"}]}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"~","type":"text","marks":[{"type":"code"}]},{"text":" (Alias for ","type":"text"},{"text":"/home/user","type":"text","marks":[{"type":"code"}]},{"text":")","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"rm","type":"text","marks":[{"type":"code"}]},{"text":" (Remove - removes a file or if used with ","type":"text"},{"text":"-r","type":"text","marks":[{"type":"code"}]},{"text":", removes directory and recursively removes files in directory)","type":"text"}]}]}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[841.0]},"content":[{"type":"codeBlock","content":[{"text":"[@mblog1 genomicdatasci]$ cd \n[@mblog1 ]$ touch testfile\n[@mblog1 ]$ mkdir testdirectory\n[@mblog1 ]$ ls\ntestdirectory testfile\n[@mblog1 ]$ mv testfile testdirectory\n[@mblog1 ]$ ls\ntestdirectory\n[@mblog1 ]$ cd testdirectory\n[@mblog1 testdirectory]$ ls\ntestfile\n[@mblog1 testdirectory]$ cd .. \n[@mblog1 ]$ cp -r testdirectory ~\n[@mblog1 ]$ cd ~\n[@mblog1 ~]$ pwd\n/home/\n[@mblog1 ~]$ ls\nDesktop Documents Downloads ondemand R testdirectory \n[@mblog1 ~]$ cd testdirectory\n[@mblog1 ~]$ ls\ntestfile\n[@mblog1 ~]$ rm testfile\n[@mblog1 ~]$ ls\n[@mblog1 ~]$","type":"text"}]}]}]}]},{"type":"rule"},{"type":"heading","attrs":{"level":1},"content":[{"text":"04 Text Editors","type":"text"}]},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"paragraph","content":[{"text":"See Workshop: ","type":"text"},{"text":"Intro to Text Editors in Linux","type":"text","marks":[{"type":"link","attrs":{"href":"https://arccwiki.atlassian.net/wiki/spaces/DOCUMENTAT/pages/2185167098"}}]}]}]},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"paragraph","content":[{"text":"You can use Text Editors from:","type":"text"}]},{"type":"orderedList","attrs":{"order":1},"content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"from the command-line.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"via GUI, from an Interactive Desktop, using emacs: Applications > Accessories > Emacs","type":"text"}]}]}]}]},{"type":"mediaSingle","attrs":{"layout":"center","width":845,"widthType":"pixel"},"content":[{"type":"media","attrs":{"width":845,"alt":"image-20240916-144953.png","id":"30eac68c-4df8-4a5b-9a3e-a6cd77c31a4b","collection":"contentId-2319876097","type":"file","height":392}}]},{"type":"rule"},{"type":"rule"},{"type":"heading","attrs":{"level":1},"marks":[{"type":"alignment","attrs":{"align":"center"}}],"content":[{"text":"*** Class 02 ***","type":"text"}]},{"type":"rule"},{"type":"rule"},{"type":"heading","attrs":{"level":1},"content":[{"text":"05 Using Linux to Search/Parse Text Files","type":"text"}]},{"type":"panel","attrs":{"panelType":"note"},"content":[{"type":"paragraph","content":[{"text":"Goals","type":"text","marks":[{"type":"strong"}]},{"text":":","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Using the command-line, demonstrate how to search and parse text files.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Show how ","type":"text"},{"text":"export","type":"text","marks":[{"type":"code"}]},{"text":" can be used to setup environment variables and ","type":"text"},{"text":"echo","type":"text","marks":[{"type":"code"}]},{"text":" to see what values they store.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Linux Commands:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"find","type":"text","marks":[{"type":"code"}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"cat","type":"text","marks":[{"type":"code"}]},{"text":" / ","type":"text"},{"text":"head","type":"text","marks":[{"type":"code"}]},{"text":" / ","type":"text"},{"text":"tail","type":"text","marks":[{"type":"code"}]},{"text":" / ","type":"text"},{"text":"grep","type":"text","marks":[{"type":"code"}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"sort","type":"text","marks":[{"type":"code"}]},{"text":" / ","type":"text"},{"text":"uniq","type":"text","marks":[{"type":"code"}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Pipe ","type":"text"},{"text":"|","type":"text","marks":[{"type":"code"}]},{"text":" output from one command to the input of another, and redirect to a file using ","type":"text"},{"text":">","type":"text","marks":[{"type":"code"}]},{"text":", ","type":"text"},{"text":">>","type":"text","marks":[{"type":"code"}]},{"text":".","type":"text"}]}]}]}]}]}]},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"paragraph","content":[{"text":"Based on Workshop","type":"text","marks":[{"type":"strong"}]},{"text":": ","type":"text"},{"text":"Intro to Linux Command-Line: View Find and Search Files","type":"text","marks":[{"type":"link","attrs":{"href":"https://arccwiki.atlassian.net/wiki/spaces/DOCUMENTAT/pages/2185167027"}}]}]}]},{"type":"rule"},{"type":"heading","attrs":{"level":2},"content":[{"text":"Your Environment: Echo and Export","type":"text"}]},{"type":"codeBlock","content":[{"text":"# View the settings configured within your environment.\n[~]$ env\n\n# View a particular environment variable\n# PATH: Where you environment will look for execuatables/commands.\n[~]$ echo $PATH\n\n# Create an environment variable that points to the workshop data folder.\n[~] export TEST_DATA=/project/genomicdatasci/software/test_data\n\n# Check it has been correctly set.\n[~]$ echo $TEST_DATA\n/project/genomicdatasci/software/test_data","type":"text"}]},{"type":"rule"},{"type":"heading","attrs":{"level":2},"content":[{"text":"Use Our Environment Variable","type":"text"}]},{"type":"codeBlock","content":[{"text":"# Lets use it.\n# Navigate to your home.\n[~]$ cd\n\n# Navigate to the workshop data folder.\n[~]$ cd $TEST_DATA\n[test_data]$ pwd\n/project/genomicdatasci/software/test_data","type":"text"}]},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"These are only available within this particular terminal/session.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Once you close this terminal, they are gone.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"They are not available across other terminals.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Advanced","type":"text","marks":[{"type":"strong"}]},{"text":": To make 'permanent' you can update your ","type":"text"},{"text":"~/.bashrc","type":"text","marks":[{"type":"code"}]}]}]}]}]},{"type":"rule"},{"type":"heading","attrs":{"level":2},"content":[{"text":"Search for a File","type":"text"}]},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"paragraph","content":[{"text":"Based on","type":"text","marks":[{"type":"strong"}]},{"text":": ","type":"text"},{"text":"Search for a File","type":"text","marks":[{"type":"link","attrs":{"href":"https://arccwiki.atlassian.net/wiki/spaces/SAP/pages/2134900799/Intro+to+Linux+Command-Line+View+Find+and+Search+Files#02-Search-for-a-File"}}]}]}]},{"type":"panel","attrs":{"panelType":"warning"},"content":[{"type":"paragraph","content":[{"text":"Linux is ","type":"text"},{"text":"case-sensitive","type":"text","marks":[{"type":"strong"}]},{"text":".","type":"text"}]}]},{"type":"codeBlock","content":[{"text":"[test_data]$ cd /project/genomicdatasci/software/test_data\n# Find a file using its full name.\n[test_data]$ find . -name \"epithelial_overrep_gene_list.tsv\"\n./scRNASeq_Results/epithelial_overrep_gene_list.tsv\n\n# Remember, Linux is case sensitive\n# Returned to command prompt with no output.\n[test_data]$ find . -name \"Epithelial_overrep_gene_list.tsv\"\n[test_data]$\n\n# Use case-insensitive option:\n[test_data]$ find . -iname \"Epithelial_overrep_gene_list.tsv\"\n./scRNASeq_Results/epithelial_overrep_gene_list.tsv","type":"text"}]},{"type":"rule"},{"type":"heading","attrs":{"level":2},"content":[{"text":"Use Wildcards *","type":"text"}]},{"type":"codeBlock","content":[{"text":"# Use Wildcards:\n[test_data]$ find . -name \"epithelial*\"\n./scRNASeq_Results/epithelial_overrep_gene_list.tsv\n./scRNASeq_Results/epithelial_de_gsea.tsv\n\n[test_data]$ find . -name \"*.tsv\"\n./Grch38/Hisat2/exons.tsv\n./Grch38/Hisat2/splicesites.tsv\n./DE_Results/DE_sig_genes_DESeq2.tsv\n./DE_Results/DE_all_genes_DESeq2.tsv\n./scRNASeq_Results/epithelial_overrep_gene_list.tsv\n./scRNASeq_Results/epithelial_de_gsea.tsv\n./Pathway_Results/fc.go.cc.p.down.tsv\n./Pathway_Results/fc.go.cc.p.up.tsv\n./BatchCorrection_Results/DE_genes_uhr_vs_hbr_corrected.tsv","type":"text"}]},{"type":"rule"},{"type":"heading","attrs":{"level":2},"content":[{"text":"View the Contents of a File","type":"text"}]},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"paragraph","content":[{"text":"Based on","type":"text","marks":[{"type":"strong"}]},{"text":": ","type":"text"},{"text":"View/Search a File","type":"text","marks":[{"type":"link","attrs":{"href":"https://arccwiki.atlassian.net/wiki/spaces/SAP/pages/2134900799/Intro+to+Linux+Command-Line+View+Find+and+Search+Files#01-View/Search-a-File"}}]}]}]},{"type":"codeBlock","content":[{"text":"[]$ cd /project/genomicdatasci/software/test_data/scRNASeq_Results\n\n# View the contents of a TEXT based file:\n# Prints everything.\n[scRNASeq_Results]$ cat epithelial_overrep_gene_list.tsv\n\n# View 'page-by-page'\n# Press 'q' to exit and return to the command-line prompt.\n[scRNASeq_Results]$ more epithelial_overrep_gene_list.tsv","type":"text"}]},{"type":"rule"},{"type":"heading","attrs":{"level":2},"content":[{"text":"View the Start and End of a File","type":"text"}]},{"type":"codeBlock","content":[{"text":"# View the first 10 items.\n[]$ head epithelial_overrep_gene_list.tsv\n\n# View the first 15 items.\n[]$ head -n 15 epithelial_overrep_gene_list.tsv\n\n# View the last 10 items.\n[]$ tail epithelial_overrep_gene_list.tsv\n\n# View the last 5 items.\n[]$ tail -n 5 epithelial_overrep_gene_list.tsv\n\n# On a login node, remember you can use 'man head' \n# or tail --help to look up all the options for a command.","type":"text"}]},{"type":"rule"},{"type":"heading","attrs":{"level":2},"content":[{"text":"Search the Contents of a Text File","type":"text"}]},{"type":"codeBlock","content":[{"text":"[]$ cd /project/genomicdatasci/software/test_data/scRNASeq_Results\n# Find rows containing \"Zfp1\"\n# Remember: Linux is case-sensitive\n# Searching for all lower case: zfp1\n[]$ grep zfp1 epithelial_overrep_gene_list.tsv\n[]$ \n\n# Searching with correct upper/lower case combination: Zfp1\n# Returns all the lines that contain this piece of text.\n[]$ grep Zfp1 epithelial_overrep_gene_list.tsv\nZfp106\nZfp146\nZfp185\nZfp1","type":"text"}]},{"type":"rule"},{"type":"heading","attrs":{"level":2},"content":[{"text":"Grep-ing with Case-Insensitive and Line Numbers","type":"text"}]},{"type":"codeBlock","content":[{"text":"# Grep ignoring case.\n[]$ grep -i zfp1 epithelial_overrep_gene_list.tsv\nZfp106\nZfp146\nZfp185\nZfp1\n\n# What line numbers are the elements on?\n[]$ grep -n -i zfp1 epithelial_overrep_gene_list.tsv\n696:Zfp106\n1998:Zfp146\n2041:Zfp185\n2113:Zfp1","type":"text"}]},{"type":"rule"},{"type":"heading","attrs":{"level":2},"content":[{"text":"Pipe: Count, Sort","type":"text"}]},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"paragraph","content":[{"text":"Based on","type":"text","marks":[{"type":"strong"}]},{"text":": ","type":"text"},{"text":"Output Redirection and Pipes","type":"text","marks":[{"type":"link","attrs":{"href":"https://arccwiki.atlassian.net/wiki/spaces/DOCUMENTAT/pages/2184872009"}}]}]}]},{"type":"codeBlock","content":[{"text":"[]$ cd /project/genomicdatasci/software/test_data/scRNASeq_Results\n\n# Pipe: direct the output of one command to the input of another.\n# Count how many lines/rows are in a file.\n[]$ cat epithelial_overrep_gene_list.tsv | wc -l\n2254\n\n# Alphabetically soft a file:\n[] sort epithelial_overrep_gene_list.tsv\n...\nZswim4\nZyx\nZzz3\nZzz3\n\n# Count lines after sorting.\n[]$ sort epithelial_overrep_gene_list.tsv | wc -l\n2254","type":"text"}]},{"type":"rule"},{"type":"heading","attrs":{"level":2},"content":[{"text":"Uniq","type":"text"}]},{"type":"codeBlock","content":[{"text":"# Find and list the unique elements within a file.\n# You need to sort your elements first.\n[] sort epithelial_overrep_gene_list.tsv | uniq\n...\nZswim4\nZyx\nZzz3\n\n# You can pipe multiple commands together.\n# Find, list and count the unique elements within a file:\n[] sort epithelial_overrep_gene_list.tsv | uniq | wc -l\n2253","type":"text"}]},{"type":"rule"},{"type":"heading","attrs":{"level":2},"content":[{"text":"Redirect Output into a File","type":"text"}]},{"type":"codeBlock","content":[{"text":"# Redirect an output into a file.\n# > : Over writes a file \n# >> : Appends to a file.\n[] sort epithelial_overrep_gene_list.tsv > sorted.tsv\n# This will fail for anyone else.\n-bash: sorted.tsv: Permission denied\n\n# You do not have write permission within this folder.\n[]$ cd ..\n[]$ ls -al\ndrwxr-sr-x 2 genomicdatasci 4096 May 31 13:50 scRNASeq_Results\n\n# Redirect to a location where you do have write permission - you home folder.\n[]$ cd scRNASeq_Results/\n[]$ sort epithelial_overrep_gene_list.tsv > ~/sorted.tsv\n[]$ ls ~\n... sorted.tsv ...\n[]$ head ~/sorted.tsv\n","type":"text"}]},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"paragraph","content":[{"text":"For further details on permissions, read through ","type":"text"},{"text":"File Ownership and Permissions","type":"text","marks":[{"type":"link","attrs":{"href":"https://arccwiki.atlassian.net/wiki/spaces/DOCUMENTAT/pages/2184642658/The+Linux+File+System#File-Ownership-and-Permissions"}}]},{"text":".","type":"text"}]}]},{"type":"rule"},{"type":"heading","attrs":{"level":1},"content":[{"text":"06 Lets start using R(/Python) and RStudio","type":"text"}]},{"type":"panel","attrs":{"panelType":"note"},"content":[{"type":"paragraph","content":[{"text":"Goals","type":"text","marks":[{"type":"strong"}]},{"text":":","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Using a terminal (via an Interactive Desktop), demonstrate how to load modules to setup an environment that uses R/RStudio and how to start the GUI.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Mention how the module system will be used, in later workshops, to load other software applications.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"(Indicate how this relates to setting up environment variables behind the scenes.)","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Further explain the differences between using a login node that requires an ","type":"text"},{"text":"salloc","type":"text","marks":[{"type":"code"}]},{"text":" to access a compute node, and that you're already running on a compute node (with limited resources) via an interactive desktop.","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Confirm arguments for ","type":"text"},{"text":"partition","type":"text","marks":[{"type":"code"}]},{"text":", ","type":"text"},{"text":"gres/gpu","type":"text","marks":[{"type":"code"}]},{"text":".","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Note that can confirm a GPU device is available by running ","type":"text"},{"text":"nvidia-smi -L","type":"text","marks":[{"type":"code"}]},{"text":" from the command-line.","type":"text"}]}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Show how the resources from the Interactive Desktop configuration start mapping to those used by ","type":"text"},{"text":"salloc","type":"text","marks":[{"type":"code"}]},{"text":".","type":"text"}]}]}]}]},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"paragraph","content":[{"text":"Based on Workshops","type":"text","marks":[{"type":"strong"}]},{"text":":","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Intro to Accessing the Cluster","type":"text","marks":[{"type":"link","attrs":{"href":"https://arccwiki.atlassian.net/wiki/spaces/DOCUMENTAT/pages/2184937473"}}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Intro to using the Module System","type":"text","marks":[{"type":"link","attrs":{"href":"https://arccwiki.atlassian.net/wiki/spaces/DOCUMENTAT/pages/2185167302"}}]}]}]}]}]},{"type":"rule"},{"type":"heading","attrs":{"level":2},"content":[{"text":"Open a Terminal","type":"text"}]},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"paragraph","content":[{"text":"You can access a Linux terminal from OnDemand by:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Opening up an Interactive Desktop and opening a terminal.","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Running on a compute node: Command prompt: ","type":"text"},{"text":"[@t402 ~]$","type":"text","marks":[{"type":"code"}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Only select what you require","type":"text","marks":[{"type":"strong"}]},{"text":":","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"How many hours? Your session will NOT run any longer that the amount of hours you requested.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Some Desktop Configurations will NOT work with some GPU Types.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Do you actually need a GPU?","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Unless you software/library/package has been developed to utilize a GPU, simply selected one will NOT make any difference - this won’t make you code magically run faster.","type":"text"}]}]}]}]}]}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Selecting a MedicineBow Shell Access which opens up a new browser tab.","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Running on the login node: ","type":"text"},{"text":"[@mblog1/2 ~]$","type":"text","marks":[{"type":"code"}]}]}]}]}]}]}]},{"type":"panel","attrs":{"panelType":"warning"},"content":[{"type":"paragraph","content":[{"text":"To run any GUI application, you ","type":"text"},{"text":"must","type":"text","marks":[{"type":"strong"},{"type":"em"}]},{"text":" use OnDemand and an Interactive Desktop.","type":"text"}]}]},{"type":"rule"},{"type":"heading","attrs":{"level":2},"content":[{"text":"Setting Up a Session Environment","type":"text"}]},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"paragraph","content":[{"text":"Across the class, you’ll be using a number of different environments.","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Running specific software applications.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Programming with R and using various R libraries.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Programming with Python and using various Python packages.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Environments build with ","type":"text"},{"text":"Miniconda","type":"text","marks":[{"type":"link","attrs":{"href":"https://arccwiki.atlassian.net/wiki/spaces/DOCUMENTAT/pages/7504145"}}]},{"text":" - a package/environment manager.","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"Since the cluster has to cater for ","type":"text"},{"text":"everyone","type":"text","marks":[{"type":"textColor","attrs":{"color":"#bf2600"}},{"type":"strong"},{"type":"em"}]},{"text":" we can not provide a simple desktop environment that provides ","type":"text"},{"text":"everything","type":"text","marks":[{"type":"textColor","attrs":{"color":"#bf2600"}},{"type":"strong"},{"type":"em"}]},{"text":".","type":"text"}]},{"type":"paragraph","content":[{"text":"Instead we provide ","type":"text"},{"text":"modules","type":"text","marks":[{"type":"textColor","attrs":{"color":"#ff5630"}},{"type":"strong"}]},{"text":" that a user will ","type":"text"},{"text":"load","type":"text","marks":[{"type":"textColor","attrs":{"color":"#ff5630"}},{"type":"strong"}]},{"text":" that configures their environment for their particular needs within a session.","type":"text"}]},{"type":"paragraph","content":[{"text":"Loading a module configures various environment variables within that Session.","type":"text"}]}]},{"type":"rule"},{"type":"heading","attrs":{"level":2},"content":[{"text":"What is Available?","type":"text"}]},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"paragraph","content":[{"text":"We have environments available based on compilers, ","type":"text"},{"text":"Singularity","type":"text","marks":[{"type":"link","attrs":{"href":"https://arccwiki.atlassian.net/wiki/spaces/DOCUMENTAT/pages/59277532"}}]},{"text":" containers, Conda, Linux Binaries","type":"text"}]}]},{"type":"codeBlock","content":[{"text":"[]$ module avail\n[]$ gcc --version\ngcc (GCC) 11.4.1 20230605 (Red Hat 11.4.1-2)\n\n[]$ which gcc\n/usr/bin/gcc\n\n[]$ echo $PATH\n/home//.local/bin:/home//bin:/apps/s/arcc/1.0/bin:/apps/s/slurm/latest/bin:\n/usr/share/Modules/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin","type":"text"}]},{"type":"rule"},{"type":"heading","attrs":{"level":2},"content":[{"text":"Is Python and/or R available?","type":"text"}]},{"type":"codeBlock","content":[{"text":"# An old version of Python is available on the System.\n# Systems are updated! Do NOT rely on them for you environment regards versions/reproducability.\n[]$ which python\n/usr/bin/python\n\n[]$ python --version\nPython 3.9.18\n\n# R is NOT available.\n[]$ which R\n/usr/bin/which: no R in (/home//.local/bin:/home//bin:\n/apps/s/arcc/1.0/bin:/apps/s/slurm/latest/bin:/usr/share/Modules/bin:\n/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin)\n\n# Nothing returned.\n[]$ echo $R_HOME\n[]$ ","type":"text"}]},{"type":"rule"},{"type":"heading","attrs":{"level":2},"content":[{"text":"Load a Compiler","type":"text"}]},{"type":"codeBlock","content":[{"text":"# What's avail for a compiler?\n[]$ module load gcc/14.2.0\n[]$ module avail\n# Notice there are a lot more applications available under this loaded compiler.\n\n[]$ gcc --version\ngcc (Spack GCC) 14.2.0\n\n[]$ which gcc\n/apps/u/spack/gcc/11.4.1/gcc/14.2.0-vzbrz6i/bin/gcc\n\n# Notice that the environment variables have been extended.\n[]$ echo $PATH\n/apps/u/spack/gcc/11.4.1/gcc/14.2.0-vzbrz6i/bin:/apps/u/spack/gcc/14.2.0/zstd/1.5.5-4jnrrl7/bin:\n/home//.local/bin:/home//bin:/apps/s/arcc/1.0/bin:\n/apps/s/slurm/latest/bin:/usr/share/Modules/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin\n\n# Notice R is now available and newer versions of Python are available under gcc/14.2.0","type":"text"}]},{"type":"panel","attrs":{"panelType":"error"},"content":[{"type":"paragraph","content":[{"text":"Note","type":"text","marks":[{"type":"strike"},{"type":"strong"}]},{"text":": For this class, for R, until you hear otherwise, we will actually be using R using the","type":"text","marks":[{"type":"strike"}]},{"text":" ","type":"text"},{"text":"gcc/13.2.0","type":"text","marks":[{"type":"code"}]},{"text":" ","type":"text"},{"text":"compiler.","type":"text","marks":[{"type":"strike"}]}]}]},{"type":"panel","attrs":{"panelType":"note"},"content":[{"type":"paragraph","content":[{"text":"20240926: Update","type":"text","marks":[{"type":"strong"}]},{"text":": ARCC has been updating compilers and libraries over the last few weeks, and we are nlow recommending to use the ","type":"text"},{"text":"gcc/14.2.0","type":"text","marks":[{"type":"code"}]},{"text":" version of ","type":"text"},{"text":"r/4.4.0","type":"text","marks":[{"type":"code"}]},{"text":".","type":"text"}]},{"type":"paragraph","content":[{"text":"All this means is that ","type":"text"},{"text":"r/4.4.0","type":"text","marks":[{"type":"code"}]},{"text":" has been built with a newer compiler. The core functionality and language remains exactly the same, and you will not see any difference running R scripts.","type":"text"}]}]},{"type":"rule"},{"type":"heading","attrs":{"level":2},"content":[{"text":"Load a Newer Version of Python","type":"text"}]},{"type":"codeBlock","content":[{"text":"[]$ module load python/3.10.6\n\n[]$ which python\n/apps/u/spack/gcc/14.2.0/python/3.10.6-6lvrsdd/bin/python\n\n[]$ python --version\nPython 3.10.6","type":"text"}]},{"type":"rule"},{"type":"heading","attrs":{"level":2},"content":[{"text":"Typically Loading R","type":"text"}]},{"type":"codeBlock","content":[{"text":"[]$ module load r/4.4.0\n\n# Notice the environment variable has now been set.\n[]$ echo $R_HOME\n/apps/u/spack/gcc/14.2.0/r/4.4.0-w7xoohc/rlib/R\n\n[]$ which R\n/apps/u/spack/gcc/14.2.0/r/4.4.0-w7xoohc/bin/R\n\n# Notice ALL the dependencies:\n[] module list\nCurrently Loaded Modules:\n 1) slurm/latest (S) 42) libxau/1.0.8\n 2) arcc/1.0 (S) 43) libxdmcp/1.1.4\n...\n 40) libpthread-stubs/0.4 81) r/4.4.0\n 41) xproto/7.0.31\n\n[]$ R --version\nR version 4.4.0 (2024-04-24) -- \"Puppy Cup\"","type":"text"}]},{"type":"panel","attrs":{"panelType":"warning"},"content":[{"type":"paragraph","content":[{"text":"You then perform: ","type":"text"},{"text":"install.packages","type":"text","marks":[{"type":"code"}]},{"text":" and manage these yourself.","type":"text"}]}]},{"type":"panel","attrs":{"panelType":"warning"},"content":[{"type":"paragraph","content":[{"text":"Same with Python: You perform the ","type":"text"},{"text":"pip install","type":"text","marks":[{"type":"code"}]},{"text":" to install which ever Python packages you require.","type":"text"}]}]},{"type":"rule"},{"type":"heading","attrs":{"level":2},"content":[{"text":"Using module purge to reset you session/environment","type":"text"}]},{"type":"codeBlock","content":[{"text":"[]$ module purge\nThe following modules were not unloaded:\n (Use \"module --force purge\" to unload all):\n\n 1) slurm/latest 2) arcc/1.0\n\n# ml is a shortcut for module list\n[salexan5@mblog2 testdirectory]$ ml\nCurrently Loaded Modules:\n 1) slurm/latest (S) 2) arcc/1.0 (S)\n\n Where:\n S: Module is Sticky, requires --force to unload or purge","type":"text"}]},{"type":"rule"},{"type":"heading","attrs":{"level":2},"content":[{"text":"Modules Specific for this Class","type":"text"}]},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"paragraph","content":[{"text":"We have created two modules specifically for this class:","type":"text"}]}]},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"paragraph","content":[{"text":"R/4.4.0 + Library of 477 R Packages (this is the original ","type":"text","marks":[{"type":"strong"}]},{"text":"gcc/13.2.0","type":"text","marks":[{"type":"code"}]},{"text":" built library)","type":"text","marks":[{"type":"strong"}]}]},{"type":"codeBlock","content":[{"text":"[]$ ls /project/genomicdatasci/software/r/libraries/\nabind DBI ggnewscale libcoin RcppAnnoy sourcetools\nalabaster.base dbplyr ggplot2 lifecycle RcppArmadillo sp\nalabaster.matrix DelayedArray ggplotify limma RcppEigen spam\n...","type":"text"}]}]},{"type":"panel","attrs":{"panelType":"note"},"content":[{"type":"paragraph","content":[{"text":"The new ","type":"text"},{"text":"gcc/14.2.0","type":"text","marks":[{"type":"code"}]},{"text":" version can be found under: ","type":"text"},{"text":"/project/genomicdatasci/software/r/libraries_gcc14/","type":"text","marks":[{"type":"code"}]}]}]},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"paragraph","content":[{"text":"R/4.3.3 and R Package Pigengene","type":"text","marks":[{"type":"strong"}]}]}]},{"type":"panel","attrs":{"panelType":"warning"},"content":[{"type":"paragraph","content":[{"text":"Due to ","type":"text"},{"text":"dependency hell","type":"text","marks":[{"type":"textColor","attrs":{"color":"#bf2600"}},{"type":"strong"},{"type":"em"}]},{"text":" issues, we could not install Pigengene within the R library collection.","type":"text"}]},{"type":"paragraph","content":[{"text":"There are ","type":"text"},{"text":"two separate environments","type":"text","marks":[{"type":"strong"},{"type":"em"}]},{"text":".","type":"text"}]},{"type":"paragraph","content":[{"text":"With ","type":"text"},{"text":"different versions of R","type":"text","marks":[{"type":"strong"},{"type":"em"}]},{"text":".","type":"text"}]}]},{"type":"rule"},{"type":"heading","attrs":{"level":2},"content":[{"text":"Using R/4.4.0 + Library","type":"text"}]},{"type":"codeBlock","content":[{"text":"[]$ module purge\n[]$ module use /project/genomicdatasci/software/modules/\n[]$ module avail\n...\n------------------- /project/genomicdatasci/software/modules -------------------\n pigengene/3.18 r/4.4.0-genomic r/4.4.0-genomic-gcc14\n...","type":"text"}]},{"type":"panel","attrs":{"panelType":"warning"},"content":[{"type":"paragraph","content":[{"text":"If you do not call the ","type":"text"},{"text":".libPaths()","type":"text","marks":[{"type":"code"}]},{"text":" command from within R (or an R script) you will not get access to the packages.","type":"text"}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Version: ","type":"text"},{"text":"r/4.4.0-genomic","type":"text","marks":[{"type":"code"}]},{"text":" (deprecated)","type":"text"}]},{"type":"panel","attrs":{"panelType":"error"},"content":[{"type":"paragraph","content":[{"text":"This original library was built use ","type":"text"},{"text":"gcc/13.2.0","type":"text","marks":[{"type":"code"}]},{"text":" and covered at the start of the class- I would recommend NOT using this one.","type":"text"}]}]},{"type":"codeBlock","content":[{"text":"[]$ module purge\n[]$ module load r/4.4.0-genomic\n-------------------------------------------------------------------------------\nThe following dependent module(s) are not currently loaded: zlib-ng/2.1.4_zen4 (required by: gcc/13.2.0), zstd/1.5.5_zen4__programs_True (required by: gcc/13.2.0)\n-------------------------------------------------------------------------------\n\n[]$ R\nR version 4.4.0 (2024-04-24) -- \"Puppy Cup\"\n...\n> .libPaths(c('/project/genomicdatasci/software/r/libraries', '/apps/u/spack/gcc/13.2.0/r/4.4.0-pvzi4gp/rlib/R/library'))","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Version: ","type":"text"},{"text":"r/4.4.0-genomic-gcc14","type":"text","marks":[{"type":"code"}]}]},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"paragraph","content":[{"text":"This later version has been built using ","type":"text"},{"text":"gcc/14.2.0","type":"text","marks":[{"type":"code"}]},{"text":" - I would recommend using this version.","type":"text"}]}]},{"type":"codeBlock","content":[{"text":"[]$ module purge\n[]$ module load r/4.4.0-genomic-gcc14\n[]$ R\nR version 4.4.0 (2024-04-24) -- \"Puppy Cup\"\n...\n> .libPaths(c('/project/genomicdatasci/software/r/libraries_gcc14', '/apps/u/spack/gcc/14.2.0/r/4.4.0-w7xoohc/rlib/R/library'))","type":"text"}]},{"type":"rule"},{"type":"heading","attrs":{"level":2},"content":[{"text":"R/4.3.3 and R Package Pigengene","type":"text"}]},{"type":"codeBlock","content":[{"text":"[salexan5@mblog2 testdirectory]$ module purge\n[salexan5@mblog2 testdirectory]$ module use /project/genomicdatasci/software/modules/\n[salexan5@mblog2 testdirectory]$ module load pigengene/3.18\n[salexan5@mblog2 testdirectory]$ R --version\nR version 4.3.3 (2024-02-29) -- \"Angel Food Cake\"\n...\n# Start R\n[salexan5@mblog2 testdirectory]$ R\nR version 4.3.3 (2024-02-29) -- \"Angel Food Cake\"\n...\n> library(Pigengene)\nLoading required package: graph\nLoading required package: BiocGenerics\n...\n","type":"text"}]},{"type":"panel","attrs":{"panelType":"warning"},"content":[{"type":"paragraph","content":[{"text":"Due to ","type":"text"},{"text":"dependency hell","type":"text","marks":[{"type":"textColor","attrs":{"color":"#bf2600"}},{"type":"strong"},{"type":"em"}]},{"text":" issues, we could not install Pigengene within the R library collection.","type":"text"}]},{"type":"paragraph","content":[{"text":"There are two separate environments.","type":"text"}]},{"type":"paragraph","content":[{"text":"With different versions of R.","type":"text"}]}]},{"type":"rule"},{"type":"heading","attrs":{"level":2},"content":[{"text":"Using RStudio with R/Library of Packages for ","type":"text"},{"text":"this","type":"text","marks":[{"type":"em"}]},{"text":" Class","type":"text"}]},{"type":"panel","attrs":{"panelType":"warning"},"content":[{"type":"paragraph","content":[{"text":"Since we are using ","type":"text"},{"text":"RStudio","type":"text","marks":[{"type":"textColor","attrs":{"color":"#bf2600"}},{"type":"strong"}]},{"text":", which is an IDE for R, i.e. a ","type":"text"},{"text":"GUI","type":"text","marks":[{"type":"strong"}]},{"text":", you need to perform this from an ","type":"text"},{"text":"Interactive Desktop","type":"text","marks":[{"type":"strong"}]},{"text":", via ","type":"text"},{"text":"OnDemand","type":"text","marks":[{"type":"strong"}]},{"text":".","type":"text"}]},{"type":"paragraph","content":[{"text":"From the Interactive Desktop, open a terminal:","type":"text"}]}]},{"type":"mediaSingle","attrs":{"layout":"center","width":568,"widthType":"pixel"},"content":[{"type":"media","attrs":{"width":568,"alt":"image-20240916-161047.png","id":"bcb5314c-c8a9-437b-89b6-d75a620c898b","collection":"contentId-2319876097","type":"file","height":292}}]},{"type":"codeBlock","content":[{"text":"[]$ module use /project/genomicdatasci/software/modules/\n[]$ module avail\n...\n------------------- /project/genomicdatasci/software/modules -------------------\n pigengene/3.18 r/4.4.0-genomic r/4.4.0-genomic-gcc14\n...\n[]$ module load r/4.4.0-genomic-gcc14\n[]$ module spider rstudio\n----------------------------------------------------------------------------\n rstudio:\n----------------------------------------------------------------------------\n Versions:\n rstudio/2024.04.1\n rstudio/2024.04.2\n----------------------------------------------------------------------------\n For detailed information about a specific \"rstudio\" package (including how to load the modules) use the module's full name.\n Note that names that have a trailing (E) are extensions provided by other modules.\n For example:\n $ module spider rstudio/2024.04.2\n----------------------------------------------------------------------------\n \n[]$ module load rstudio/2024.04.1\n[]$ rstudio\n\n# From within R Studio:\n> .libPaths(c('/project/genomicdatasci/software/r/libraries_gcc14', '/apps/u/spack/gcc/14.2.0/r/4.4.0-w7xoohc/rlib/R/library'))\n# Notice how the list of Packages updates.","type":"text"}]},{"type":"panel","attrs":{"panelType":"warning"},"content":[{"type":"paragraph","content":[{"text":"Remember","type":"text","marks":[{"type":"strong"}]},{"text":": If you do not call the ","type":"text"},{"text":".libPaths()","type":"text","marks":[{"type":"code"}]},{"text":" command from within R (or an R script) you will not get access to the packages.","type":"text"}]}]},{"type":"rule"},{"type":"heading","attrs":{"level":2},"content":[{"text":"Using RStudio and R/Pigengene for ","type":"text"},{"text":"this","type":"text","marks":[{"type":"em"}]},{"text":" Class","type":"text"}]},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"paragraph","content":[{"text":"Remember","type":"text","marks":[{"type":"strong"}]},{"text":": Since we are using ","type":"text"},{"text":"RStudio","type":"text","marks":[{"type":"textColor","attrs":{"color":"#bf2600"}},{"type":"strong"}]},{"text":", which is an IDE for R, i.e. a ","type":"text"},{"text":"GUI","type":"text","marks":[{"type":"strong"}]},{"text":", you need to perform this from an ","type":"text"},{"text":"Interactive Desktop","type":"text","marks":[{"type":"strong"}]},{"text":", via ","type":"text"},{"text":"OnDemand","type":"text","marks":[{"type":"strong"}]},{"text":".","type":"text"}]},{"type":"paragraph","content":[{"text":"From the Interactive Desktop, open a terminal:","type":"text"}]}]},{"type":"codeBlock","content":[{"text":"[]$ module use /project/genomicdatasci/software/modules/\n[]$ module load pigengene/3.18\n[]$ export PATH=$PATH:/project/genomicdatasci/software/pigengene/3.18/bin/\n[]$ export RSTUDIO_WHICH_R=/project/genomicdatasci/software/pigengene/3.18/lib/R/bin/R\n[]$ module load rstudio/2024.04.1\n[]$ rstudio","type":"text"}]},{"type":"panel","attrs":{"panelType":"warning"},"content":[{"type":"paragraph","content":[{"text":"If you do not ","type":"text"},{"text":"export the detailed environment variables","type":"text","marks":[{"type":"strong"}]},{"text":", RStudio will not pick up the version of R.","type":"text"}]}]},{"type":"rule"},{"type":"heading","attrs":{"level":2},"content":[{"text":"Other Class Modules","type":"text"}]},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"paragraph","content":[{"text":"Using the ","type":"text"},{"text":"module use","type":"text","marks":[{"type":"code"}]},{"text":" command will also group and show other applications relating to this class:","type":"text"}]}]},{"type":"codeBlock","content":[{"text":"[]$ module use /project/genomicdatasci/software/modules/\n[]$ module avail\n...\n------------------- /project/genomicdatasci/software/modules -------------------\n bam-readcount/0.8.0 kentutils/1.04.0 regtools/1.0.0\n bedops/2.4.41 multiqc/1.24.1 rseqc/5.0.3\n fastp/0.23.4 picard/3.2.0 samtools/1.20 (D)\n fastqc/0.12.1 pigengene/3.18 sratoolkit/3.1.1\n hisat-genotype/1.3.3 r/4.4.0-genomic-gcc14 subread/2.0.6\n hisat2/2.2.1 r/4.4.0-genomic tophat/2.1.1\n...","type":"text"}]},{"type":"rule"},{"type":"heading","attrs":{"level":2},"content":[{"text":"Request Interactive Session (Compute Node) from a Login Node","type":"text"}]},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"paragraph","content":[{"text":"Based on Workshop","type":"text","marks":[{"type":"strong"}]},{"text":": ","type":"text"},{"text":"Intro to job Scheduling","type":"text","marks":[{"type":"link","attrs":{"href":"https://arccwiki.atlassian.net/wiki/spaces/DOCUMENTAT/pages/2185167366"}}]}]}]},{"type":"table","attrs":{"layout":"default","width":1002.0,"localId":"bc9b9275-092d-4b39-8c5e-0cd7c97b3bf0"},"content":[{"type":"tableRow","content":[{"type":"tableHeader","attrs":{"colspan":1,"background":"#ffffff","rowspan":1,"colwidth":[391.0]},"content":[{"type":"mediaSingle","attrs":{"layout":"center","width":374,"widthType":"pixel"},"content":[{"type":"media","attrs":{"width":563,"alt":"image-20240915-232354.png","id":"ced5d372-6d5b-475d-a72b-53446185be89","collection":"contentId-2319876097","type":"file","height":487}}]},{"type":"paragraph"}]},{"type":"tableHeader","attrs":{"colspan":1,"background":"#ffffff","rowspan":1,"colwidth":[611.0]},"content":[{"type":"codeBlock","content":[{"text":"# Short form:\n# Notice we can request more memory.\n[@mblog1 ~]$ salloc -A genomicdatasci -t 4:00:00 --mem=4G -c 1 \n# Long form\n# MUST define account/A and time/t\n[@mblog1 ~]$ salloc --account=genomicdatasci --time=4:00:00 --mem=4G --cpus-per-task=1 \nsalloc: Granted job allocation 3735901\nsalloc: Waiting for resource configuration\nsalloc: Nodes mbcpu-024 are ready for job\n[@mbcpu-024 ~]$\n\n[@mbcpu-024 ~]$ exit\nexit\nsalloc: Relinquishing job allocation 3735901\n\n[@mblog1 ~]$ salloc --help","type":"text"}]}]}]}]},{"type":"rule"},{"type":"heading","attrs":{"level":2},"content":[{"text":"Request Interactive Session (Compute Node) with a GPU","type":"text"}]},{"type":"table","attrs":{"layout":"default","width":1011.0,"localId":"54c82888-d6ab-4071-9459-784412e03b8f"},"content":[{"type":"tableRow","content":[{"type":"tableHeader","attrs":{"colspan":1,"background":"#ffffff","rowspan":1,"colwidth":[399.0]},"content":[{"type":"mediaSingle","attrs":{"layout":"center","width":382,"widthType":"pixel"},"content":[{"type":"media","attrs":{"width":560,"alt":"image-20240915-232444.png","id":"97c11640-12df-4da8-aa82-24c35301637f","collection":"contentId-2319876097","type":"file","height":592}}]},{"type":"paragraph"}]},{"type":"tableHeader","attrs":{"colspan":1,"background":"#ffffff","rowspan":1,"colwidth":[612.0]},"content":[{"type":"codeBlock","content":[{"text":"[@mblog2]$ salloc -A genomicdatasci -t 8:00:00 --mem=8G -c 2 -p mb-a30 --gres=gpu:1\nsalloc: Granted job allocation 3735902\nsalloc: Nodes mba30-002 are ready for job\n[@mba30-002]$ nvidia-smi -L\nGPU 0: NVIDIA A30 (UUID: GPU-769d8459-cfc2-0a41-0b61-ba5ab798662b)\n[@mba30-002]$ exit\nexit\nsalloc: Relinquishing job allocation 3735902\n[@mblog2]%\n\n# If you don't ask, you don't get:\n[@mblog2]$ salloc -A genomicdatasci -t 8:00:00 --mem=8G -c 2 -p mb-a30\nsalloc: Granted job allocation 3735903\nsalloc: Nodes mba30-002 are ready for job\n[@mba30-002]$ nvidia-smi -L\nNo devices found.\n[@mba30-002]$ exit\n[@mblog2]$","type":"text"}]}]}]}]},{"type":"rule"},{"type":"heading","attrs":{"level":2},"content":[{"text":"Request what you Need!","type":"text"}]},{"type":"codeBlock","content":[{"text":"# You're telling this command to use 4 threads - 4 cores\n[]$ hisat2-build -p 4 ...\n\n# Create an interactive session with only 1 core.\n[]$ salloc --account=genomicdatasci --time=30:00 \n# Setup the Environment\n[@mbcpu-001]$ hisat2-build -p 4 --ss $INDEX/splicesites.tsv --exon $INDEX/exons.tsv $REFERENCE/chr22_with_ERCC92.fa $INDEX/chr22\n...\nTotal time for call to driver() for forward index: 00:01:09\n\n# Create an interactive session with 4 cores.\n[@mblog1]$ salloc --account=genomicdatasci --time=30:00 -c 4\n# Setup the Environment\n[@mbcpu-001]$ hisat2-build -p 4 --ss $INDEX/splicesites.tsv --exon $INDEX/exons.tsv $REFERENCE/chr22_with_ERCC92.fa $INDEX/chr22\n...\nTotal time for call to driver() for forward index: 00:00:43","type":"text"}]},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"paragraph","content":[{"text":"The first instance only request one core and ran slower than when we correctly requested 4 cores.","type":"text"}]}]},{"type":"rule"},{"type":"heading","attrs":{"level":1},"content":[{"text":"07 Create a basic workflow and submitting jobs.","type":"text"}]},{"type":"panel","attrs":{"panelType":"note"},"content":[{"type":"paragraph","content":[{"text":"Goals","type":"text","marks":[{"type":"strong"}]},{"text":":","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Since RStudio is a GUI, demonstrate moving from running a script within RStudio to running using Rscript from the command-line.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Put the various elements of loading modules, moving into a folder, running an R file, that make up a basic workflow, into a script that can be submitted using ","type":"text"},{"text":"sbatch","type":"text","marks":[{"type":"code"}]},{"text":" to Slurm.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Map the ","type":"text"},{"text":"salloc","type":"text","marks":[{"type":"code"}]},{"text":" arguments to ","type":"text"},{"text":"#SBATCH","type":"text","marks":[{"type":"code"}]},{"text":".","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Show how to monitor a jobs using ","type":"text"},{"text":"squeue","type":"text","marks":[{"type":"code"}]},{"text":" as well as using the email related Slurm options.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Show how to request GPU compute nodes and defining ","type":"text"},{"text":"gres","type":"text","marks":[{"type":"code"}]},{"text":" to specifically request a GPU.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Provide a basic template.","type":"text"}]}]}]}]},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"paragraph","content":[{"text":"Based on Workshop","type":"text","marks":[{"type":"strong"}]},{"text":": ","type":"text"},{"text":"Intro to Job Scheduling","type":"text","marks":[{"type":"link","attrs":{"href":"https://arccwiki.atlassian.net/wiki/spaces/DOCUMENTAT/pages/2185167366"}}]}]},{"type":"paragraph","content":[{"text":"General Wiki","type":"text","marks":[{"type":"strong"}]},{"text":": ","type":"text"},{"text":"Slurm Workload Manager","type":"text","marks":[{"type":"link","attrs":{"href":"https://arccwiki.atlassian.net/wiki/spaces/DOCUMENTAT/pages/3113024"}}]}]}]},{"type":"rule"},{"type":"heading","attrs":{"level":2},"content":[{"text":"Why Submit a Job","type":"text"}]},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"paragraph","content":[{"text":"A single computation can take, minutes, hours, days, weeks, months. An interactive session quickly becomes impractical.","type":"text"}]},{"type":"paragraph","content":[{"text":"Submit a job to the Slurm queue - Slurm manages everything for you.","type":"text"}]},{"type":"paragraph","content":[{"text":"Everything you do on the command-line, working out your workflow, is put into a script.","type":"text"}]},{"type":"paragraph","content":[{"text":"Workflow:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"What resources you require? (Interactive desktop configuration, ","type":"text"},{"text":"salloc","type":"text","marks":[{"type":"code"}]},{"text":" options)","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"What modules are loaded.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Which folder you’re running you computation within. Where the data is stored. Where you want the results.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Command-line calls being called.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Software applications being run.","type":"text"}]}]}]}]},{"type":"rule"},{"type":"heading","attrs":{"level":2},"content":[{"text":"Submit a Job to the Cluster","type":"text"}]},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"paragraph","content":[{"text":"Convert ","type":"text"},{"text":"salloc","type":"text","marks":[{"type":"code"}]},{"text":" command-line options to an ","type":"text"},{"text":"sbatch","type":"text","marks":[{"type":"code"}]},{"text":" related script.","type":"text"}]},{"type":"paragraph","content":[{"text":"Options have ","type":"text"},{"text":"defaults","type":"text","marks":[{"type":"em"}]},{"text":" if not defined.","type":"text"}]}]},{"type":"codeBlock","content":[{"text":"# salloc\n[@mblog1 ~]$ salloc -A genomicdatasci -t 8:00:00 --mem=8G -c 2 -p mb-a30 --gres=gpu:1 \n# sbatch\n# Options within your bash script.\n#SBATCH --account=genomicdatasci # Account. MUST be defined.\n#SBATCH --time=8:00:00 # Time. MUST be defined. \n#SBATCH --mem=8G # Memory.\n#SBATCH --mem-per-cpu=1G # Alternaitve: Commented out. Default is 1G if no memory values defined.\n#SBATCH --cpus-per-task=2 # CPUs per Task - default is 1 if not defined.\n#SBATCH --partition=mb-a30 # Partition - If not defined, Slurm will select.\n#SBATCH --gres=gpu:1 # Generic Resources","type":"text"}]},{"type":"rule"},{"type":"heading","attrs":{"level":2},"content":[{"text":"Additional ","type":"text"},{"text":"sbatch","type":"text","marks":[{"type":"code"}]},{"text":" Options","type":"text"}]},{"type":"codeBlock","content":[{"text":"#SBATCH --job-name=\n#SBATCH --nodes=<#nodes> # Default is 1 if not defined. \n#SBATCH --ntasks-per-node=<#tasks/node> # Default is 1 if not defined.\n#SBATCH --mail-type=ALL\n#SBATCH --mail-user=\n#SBATCH --output=_%A.out # Postfix the job id to \n # If not defined: slurm-.out","type":"text"}]},{"type":"rule"},{"type":"heading","attrs":{"level":2},"content":[{"text":"Example Script: What Goes into It?","type":"text"}]},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"paragraph","content":[{"text":"The bash script can contain:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Linux/bash commands and script.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Module loads.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Application command-line calls.","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":" Lets consider our R workflow. I have:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"R scripts copied into my ","type":"text"},{"text":"/gscratch","type":"text","marks":[{"type":"code"}]},{"text":" folder.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"R related modules to load.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"R scripts to run.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"to track the time the job starts and ends.","type":"text"}]}]}]}]},{"type":"rule"},{"type":"heading","attrs":{"level":2},"content":[{"text":"Example Script: Running R Script","type":"text"}]},{"type":"codeBlock","content":[{"text":"#!/bin/bash\n# Comment: The first line 'shebang' is followed by the interpreter or the command that should be used to execute the script.\n#SBATCH --job-name=r_job\n#SBATCH --account=genomicdatasci\n#SBATCH --time=10:00\n#SBATCH --mail-type=ALL\n#SBATCH --mail-user=\n#SBATCH --output=r_%A.out\nexport R_FILES=/gscratch/$USER\necho \"R Workflow Example\"\nSTART=$(date +'%D %T')\necho \"Start:\" $START\necho \"SLURM_JOB_ID:\" $SLURM_JOB_ID\necho \"SLURM_JOB_NAME:\" $SLURM_JOB_NAME\necho \"SLURM_JOB_NODELIST:\" $SLURM_JOB_NODELIST\nmodule use /project/genomicdatasci/software/modules\nmodule purge\nmodule load r/4.4.0-genomic-gcc14\ncd $R_FILES\nRscript test_r_libraries_gcc14.R\nEND=$(date +'%D %T')\necho \"End:\" $END","type":"text"}]},{"type":"rule"},{"type":"heading","attrs":{"level":2},"content":[{"text":"Submit your Job","type":"text"}]},{"type":"codeBlock","content":[{"text":"# From your Working Directory - the folder you are currently in.\n[@mblog1]$ ls\nrun_r.sh test_data\n\n# You can submit the job from the login node.\n# Make a note of the job id.\n[@mblog1]$ sbatch run_r.sh\nSubmitted batch job 16054193\n\n# ST Column: Status of P means Pending / R means Running.\n[@mblog1]$ squeue -u \n JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)\n 16054193 teton r_job R 0:06 1 t402\n\n# Once the job is running, the defined output file will be generated.\n[@mblog1]$ ls\nr_16054193.out run_r.sh test_data","type":"text"}]},{"type":"rule"},{"type":"heading","attrs":{"level":2},"content":[{"text":"Monitor your Job","type":"text"}]},{"type":"codeBlock","content":[{"text":"# You can view the contents of your output file:\n[@mblog1]$ cat r_16054193.out\nR Workflow Example\nStart: 06/05/24 14:02:01\nSLURM_JOB_ID: 16054193\nSLURM_JOB_NAME: r_job\nSLURM_JOB_NODELIST: m221\nSleeping...\n[@mblog1]$ squeue -u \n JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)\n 16054193 teton r_job R 0:18 1 t402\n\n# If the job id is nolonger in the queue then it means the job is no longer running.\n# It might have completed, or failed and exited.\n[@mblog1]$ squeue -u \n JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)","type":"text"}]},{"type":"rule"},{"type":"heading","attrs":{"level":2},"content":[{"text":"Why is my Job Not Running?","type":"text"}]},{"type":"panel","attrs":{"panelType":"warning"},"content":[{"type":"paragraph","content":[{"text":"Typically because the resources you are requesting are not ","type":"text"},{"text":"currently","type":"text","marks":[{"type":"em"}]},{"text":" available.","type":"text"}]},{"type":"paragraph","content":[{"text":"Slurm will add your job to the queue, but it will be ","type":"text"},{"text":"PENDING (P)","type":"text","marks":[{"type":"textColor","attrs":{"color":"#bf2600"}},{"type":"strong"}]},{"text":" while it waits for the necessary resources to become available.","type":"text"}]},{"type":"paragraph","content":[{"text":"As soon as they are, your job will start, and it’s status will update to ","type":"text"},{"text":"RUNNING (R)","type":"text","marks":[{"type":"textColor","attrs":{"color":"#bf2600"}},{"type":"strong"}]},{"text":".","type":"text"}]},{"type":"paragraph","content":[{"text":"Slurm ","type":"text"},{"text":"manages","type":"text","marks":[{"type":"textColor","attrs":{"color":"#bf2600"}},{"type":"strong"},{"type":"em"}]},{"text":" this for you.","type":"text"}]}]},{"type":"rule"},{"type":"heading","attrs":{"level":2},"content":[{"text":"Monitor your Job: Continued…","type":"text"}]},{"type":"codeBlock","content":[{"text":"# You can monitor the queue and/or log file to check if running.\n[@mblog1 ]$ cat r_16054193.out\nR Workflow Example\nStart: 06/05/24 14:02:01\nSLURM_JOB_ID: 16054193\nSLURM_JOB_NAME: r_job\nSLURM_JOB_NODELIST: t402\nSleeping...\nLoading required package: SeuratObject\nLoading required package: sp\nAttaching package: ‘SeuratObject’\nThe following objects are masked from ‘package:base’:\n intersect, t\nEnd: 06/05/24 14:02:29\n# OR...","type":"text"}]},{"type":"rule"},{"type":"heading","attrs":{"level":2},"content":[{"text":"Alternative Monitoring of Job via Email: Job Efficiency","type":"text"}]},{"type":"codeBlock","content":[{"text":"# Monitor your email:\nEmail 1:\nSubject: medicinebow Slurm Job_id=16054193 Name=r_job Began, Queued time 00:00:01\n\nEmail 2: Job Efficieny:\nSubject: medicinebow Slurm Job_id=16054193 Name=r_job Ended, Run time 00:00:28, COMPLETED, ExitCode 0\nJob ID: 16054193\nCluster: medicinebow\nUser/Group: /\nState: COMPLETED (exit code 0)\nCores: 1\nCPU Utilized: 00:00:07\nCPU Efficiency: 25.00% of 00:00:28 core-walltime\nJob Wall-clock time: 00:00:28\nMemory Utilized: 0.00 MB (estimated maximum)\nMemory Efficiency: 0.00% of 1000.00 MB (1000.00 MB/core)","type":"text"}]},{"type":"rule"},{"type":"heading","attrs":{"level":2},"content":[{"text":"Example Script 2","type":"text"}]},{"type":"paragraph","content":[{"text":"This might look like something your cover in later sessions:","type":"text"}]},{"type":"codeBlock","content":[{"text":"#!/bin/bash\n#SBATCH --job-name=hisat2\n#SBATCH --account=genomicdatasci \n#SBATCH --time=8:00:00 \n#SBATCH --cpus-per-task=4 \n#SBATCH --mem=8G\n#SBATCH --mail-type=ALL\n#SBATCH --mail-user=\n#SBATCH --output=hisat2_%A.out\nSTART=$(date +'%D %T')\necho \"Start:\" $START\necho \"SLURM_JOB_ID:\" $SLURM_JOB_ID\necho \"SLURM_JOB_NAME:\" $SLURM_JOB_NAME\necho \"SLURM_JOB_NODELIST:\" $SLURM_JOB_NODELIST\nmodule load gcc/12.2.0 hisat2/2.2.1\nexport REFERENCE=/project/genomicdatasci/software/test_data/Grch38/fasta\nexport INDEX=/project/genomicdatasci/software/test_data/Grch38/Hisat2\n# Comment: Location of the splicesites.tsv file.\ncd /gscratch/$USER\nhisat2-build -p 4 --ss $INDEX/splicesites.tsv --exon $INDEX/exons.tsv $REFERENCE/chr22_with_ERCC92.fa $INDEX/chr22\nEND=$(date +'%D %T')\necho \"End:\" $END","type":"text"}]},{"type":"panel","attrs":{"panelType":"warning"},"content":[{"type":"paragraph","content":[{"text":"See: ","type":"text"},{"text":"Slurm: Common Questions and Issues and How to Resolve","type":"text","marks":[{"type":"link","attrs":{"href":"https://arccwiki.atlassian.net/wiki/spaces/DOCUMENTAT/pages/2237693956"}}]}]}]},{"type":"rule"},{"type":"heading","attrs":{"level":2},"content":[{"text":"Being a Good Cluster Citizen","type":"text"}]},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"paragraph","content":[{"text":"Based on: ","type":"text"},{"text":"Slurm: Workflows and Best Practices","type":"text","marks":[{"type":"link","attrs":{"href":"https://arccwiki.atlassian.net/wiki/spaces/DOCUMENTAT/pages/2231795764"}}]},{"text":":","type":"text"}]},{"type":"paragraph","content":[{"text":"Specifically the topic: ","type":"text"},{"text":"How can I be a good cluster citizen","type":"text","marks":[{"type":"link","attrs":{"href":"https://arccwiki.atlassian.net/wiki/spaces/DOCUMENTAT/pages/2231795764/Slurm+Workflows+and+Best+Practices#How-can-I-be-a-good-cluster-citizen?"}}]},{"text":":","type":"text"}]}]},{"type":"rule"},{"type":"heading","attrs":{"level":1},"content":[{"text":"08 Summary and Next Steps","type":"text"}]},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"paragraph","content":[{"text":"Examples and cheat sheets can be found under: ","type":"text"},{"text":"/project/genomicdatasci/arcc_notes","type":"text","marks":[{"type":"code"}]}]}]},{"type":"panel","attrs":{"panelType":"note"},"content":[{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"We’ve covered the following high-level concepts, commands and processes:","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"What is HPC and what is a cluster - focusing on ARC’s MedicineBow cluster.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"An introduction to Linux and its File System, and how to navigate around using an Interactive Desktop and/or using the command-line.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Linux command-line commands to view, search, parse, sort text files.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"How to pipe the output of one command to the input of another, and how to redirect output to a file.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Using vim as a command-line text editor and/or emacs as a GUI within an Interactive Desktop.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Setting up your environment (using modules) to provide R/Python environments, and other software applications.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Accessing compute nodes via a OnDemand Interactive Desktop, and requesting different resources (cores, memory, GPUs).","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Requesting interactive sessions (from a login node) using ","type":"text"},{"text":"salloc","type":"text","marks":[{"type":"code"}]},{"text":".","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Setting up a workflow, within a script, that can then be submitted to the Slurm queue using ","type":"text"},{"text":"sbatch","type":"text","marks":[{"type":"code"}]},{"text":", and how to monitor jobs.","type":"text"}]}]}]}]},{"type":"paragraph"},{"type":"paragraph"},{"type":"paragraph"},{"type":"paragraph"}],"version":1}

Browser not supported