class: center, middle, inverse, title-slide .title[ # VanML 2024: Using supercomputers effectively ] .author[ ### Dr. Lloyd T. Elliott   2024/02/22 ] --- # High performance computing (HPC) Pros: - Run multiple analyses in parallel - Collaborate with shared data directories Cons: - Remote development can be difficult --- # Recipe 1: Launch Amazon instance .center[ <img src="Figure01.png" width="105%" /> ] Create a new Amazon AWS account --- .center[ <img src="Figure02.png" width="105%" /> ] Enter an email and account name --- .center[ <img src="Figure03.png" width="105%" /> ] Enter verification code from email --- .center[ <img src="Figure04.png" width="105%" /> ] Choose a password --- .center[ <img src="Figure05.png" width="105%" /> ] Enter your contact information --- .center[ <img src="Figure06.png" width="105%" /> ] Enter credit card information. There may be a hold of 1USD, but you will not be further charged provided you remain in the free tier --- .center[ <img src="Figure07.png" width="105%" /> ] Enter a cell number to receive another verification code --- .center[ <img src="Figure08.png" width="105%" /> ] Enter verification code from phone --- .center[ <img src="Figure09.png" width="105%" /> ] Click 'complete sign up' --- .center[ <img src="Figure10.png" width="105%" /> ] Click 'go to the aws management console' --- .center[ <img src="Figure11.png" width="105%" /> ] Enter your email again --- .center[ <img src="Figure12.png" width="105%" /> ] Enter your password --- .center[ <img src="Figure13.png" width="105%" /> ] In the 'search' bar, type ec2 --- .center[ <img src="Figure14.png" width="105%" /> ] Click on ec2 under 'services' --- .center[ <img src="Figure16.png" width="105%" /> ] Optional: Click on the drop down for 'ohio' in the top right, and then click 'canada (central)'. This will make it so that the computer you connect to is in Canada --- .center[ <img src="Figure15.png" width="105%" /> ] Click on 'instances' --- .center[ <img src="Figure17.png" width="105%" /> ] Click on 'launch instance' --- .center[ <img src="Figure18.png" width="105%" /> ] Enter a name for your instance (this is what it will be called in other places in the console) --- .center[ <img src="Figure19.png" width="105%" /> ] Scroll down to see the default settings for the instance. These settings will create a linux computer on the free tier --- .center[ <img src="Figure20.png" width="105%" /> ] Scroll down further to 'key pair (login)' and click 'create new key pair' --- .center[ <img src="Figure21.png" width="105%" /> ] Enter a name for the keypair (again, this is what it will be called in other places in the console). Click 'create key pair' --- .center[ <img src="Figure22.png" width="105%" /> ] You may be prompted to download the key pair. Click 'allow' --- .center[ <img src="Figure23.png" width="105%" /> ] Scroll down further and click 'launch instance' --- .center[ <img src="Figure24.png" width="105%" /> ] Click 'connect to instance' --- .center[ <img src="Figure25.png" width="105%" /> ] Click 'ssh client' --- .center[ <img src="Figure26.png" width="105%" /> ] Copy the 'ssh' command under 'example:'. Then, in a terminal change to the directory the downloaded key pair is in, and then enter the command in step 3, and then paste/enter the copied 'ssh' command --- .center[ <img src="Figure27.png" width="105%" /> ] You are now connected to your Amazon AWS machine --- # Recipe 1 complete! - Higher level tutorial here: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EC2_GetStarted.html --- # Amazon AWS free tier .center[ <img src="Figure28.png" width="105%" /> ] - Details: https://aws.amazon.com/free/ - 750 hours/month and 100GB data-out/month --- # Amazon AWS pricing - Reach: airbnb, netflix, NASA, twitch - Pricing: <$0.09/GB data-out. Details: https://aws.amazon.com/ec2/pricing/ --- # Amazon AWS pricing .center[ <img src="Figure99.png" width="105%" /> ] --- # Recipe 2: Set Amazon AWS budget .center[ <img src="Figure29.png" width="105%" /> ] - Log in to https://aws.amazon.com --- .center[ <img src="Figure30.png" width="105%" /> ] - Type 'billing' into the 'search' bar, and click on 'billing and cost management' --- .center[ <img src="Figure31.png" width="105%" /> ] - Scroll down on the left pane and click on 'budgets' (under 'budgets and planning') --- .center[ <img src="Figure32.png" width="105%" /> ] - Click 'create a budget' --- .center[ <img src="Figure33.png" width="105%" /> ] - Scroll down through the default settings for a 'zero cost budget' --- .center[ <img src="Figure34.png" width="105%" /> ] - Scroll down through the default settings for a 'zero cost budget', and enter your email address under 'email recipients'. Click 'create budget' --- # Recipe 2 complete! - Now you will be emailed once you exceed your free tier usage - More details here: https://aws.amazon.com/aws-cost-management/aws-budgets/ --- # Recipe 3: Terminate the instance .center[ <img src="Figure35.png" width="105%" /> ] - Log in to https://aws.amazon.com --- .center[ <img src="Figure35.png" width="105%" /> ] - Click on 'ec2' (or, type 'ec2' in the search bar and click on 'ec2') --- .center[ <img src="Figure36.png" width="105%" /> ] - Click on 'instances'. Note: if you don't see your instance running, you may have to switch regions by clicking the down arrow in the upper right --- .center[ <img src="Figure37.png" width="105%" /> ] - Click the check box to the left of your instance --- .center[ <img src="Figure38.png" width="105%" /> ] - Click 'instance state' and then 'terminate' --- .center[ <img src="Figure39.png" width="105%" /> ] - Click 'terminate' on the confirmation dialog --- # Recipe 3 complete! - Your instance is now terminated, and will not contribute further to your free tier --- # Remote development: Level 0 One way to develop code that you want to run on a supercomputer is to develop it locally, and then copy it over using 'scp' or 'sftp' - This may be paired with some remote editing through the terminal using 'vim' or 'emacs' - If you take this approach: PLEASE! use 'tmux' or 'screen' https://www.linuxtrainingacademy.com/tmux-tutorial/ --- # Remote development: Level 1 To make it easier to transfer files between a supercomputer and your local computer, you can mount a shared filesystem. An easy way to do that is with dropbox Pros: - This can help collaboration Cons: - Synchronization can take a moment, so it's possible to momentarily lose consistency --- # Recipe 4: Dropbox for Amazon AWS - Go to https://dropbox.com and create a new account ('Start your free trial/Free for 30 days') - Install the dropbox desktop app: https://www.dropbox.com/desktop you may be prompted to log in. Or, after installing an icon may appear on your status bar or desktop, and you may be able to interact with that icon to log in --- .center[ <img src="Figure40.png" width="105%" /> ] - Now there should be a directory called "Dropbox" in your home directory. Here, we have changed to that directory and created a sub-directory called 'VanML' --- .center[ <img src="Figure41.png" width="105%" /> ] - The sub-directory we just created now shows up on our dropbox landing page --- - Next, let's get another "terminal-accessible" version of this dropbox account mounted on Amazon AWS. Let's pick up from the end of Recipe 1: .center[ <img src="Figure42.png" width="105%" /> ] --- .center[ <img src="Figure43.png" width="105%" /> ] - Enter the command 'sudo yum install fuse' to install a driver that allows filesystems such as dropbox to be mounted --- .center[ <img src="Figure44.png" width="105%" /> ] - Enter the command 'sudo yum install python-pip' to install the 'python install packages' system --- .center[ <img src="Figure45.png" width="105%" /> ] - Enter the command 'pip3 install dbxfs' to install the dropbox mounter --- .center[ <img src="Figure46.png" width="105%" /> ] - Enter the commands 'mkdir mydropbox', 'dbxfs mydropbox' to create/sync 'mydropbox' to your account. Follow the steps --- .center[ <img src="Figure47.png" width="105%" /> ] - Now '~/mydropbox/' is linked with your local environment. We add a poem to the sub-directory 'VanML' --- .center[ <img src="Figure48.png" width="105%" /> ] - The poem appears in our local environment --- # Recipe 4 complete! - Details: https://ostechnix.com/dbxfs-mount-dropbox-folder-locally-as-virtual-file-system-in-linux/ - SFU has a total head count Microsoft 365 license, including OneDrive. You can mount OneDrive in a similar way to the above dropbox slides: https://sfu.teamdynamix.com/TDClient/255/ITServices/KB/ArticleDet?ID=3820#sign_in "SET UP ONEDRIVE ON DESKTOP (NON-MOBILE) DEVICES" --- # Remote development: Level 2 While working in terminal can be fast, IDEs are a GUI alternative Pros: - More features, can be easier to use features - Good for project organization Cons: - Can have a higher learning curve - Can be harder to set up on a remote machine --- # Remote development: Level 2 .center[ <img src="Figure49.png" width="105%" /> ] XCode is an example of an IDE --- # Other IDEs - VSCode - Jupyter notebooks - posit cloud/RStudio cloud https://posit.cloud/ --- # Summary - Recipes - Launch Amazon AWS instance - Set Amazon AWS budget - Terminate the instance - Dropbox for Amazon AWS --- # Summary - Remote development - Level 0: Terminal - Level 1: Shared filesystems - Level 2: Remote IDEs