Memento Mori

Archiving a digital body of work

The archiving of websites is a significant issue at the Museum of Old and New Art. When I first joined, there were 40+ websites running on a single production server. Years of festival sites, running (solely for reference purposes) along side production websites. This needlessly cost the museum thousands of dollars a month, while ensuring performance and scalability issues.

Below is a process I've begun for archiving modern, API driven websites.

---

Sufficient Memory

(Archiving Goals)

• Safe deprovisioning of servers, databases, CDN's and third party API's.
• Historical accuracy of web content, mainly used for internal reference.
• Cost effectiveness by running offline, or on temporary cloud servers.
• Low maintenance, as close to zero as possible.

---

---

Archived + Offline API's

I wanted to find a way to have our API's work offline. I was inspired by how Insomnia saves a history of all requests made inside of their client.

While there are many tried and true ways of caching API responses, they are mainly focussed on speed, short (impermanent) lifespans, and would ultimately add an additional layer to our overall infrastructure.

This led me to create custom middleware for Koa. Many of the sites built at the Museum of Old and New Art are powered by Koa REST API's. It's worth noting that this concept could be easily applied to any basic web server.

import vault from 'koa-vault'

app.use(
  vault({
    directory: './vault',
    offline: false
  })
)

Koa-Vault works by saving JSON responses from your API to a local directory. When running with the offline option disabled, your requests will run as normal, and a JSON file is is output to a vault directory. The most recent version will always be saved.

→ program → mulatu-astatke.json
→ program → nakhane.json
→ program → neneh-cherry.json
→ program → oneohtrix-point-never.json

File names are sanitised for storage on your filesystem, but kept human readable -- which means these responses can be used for reference and/or debugging purposes.

When a website is archived, the offline option can be enabled. This will allow your API to respond with only the saved JSON files. It's a bit like a time machine; please delete your production infrastructure as you see fit!

---

---

🐝

We've started using this method to archive our recent festival websites. Multiple archives can be kept in sync with pre, present, and post festival phases. The saved responses can be committed to our git repositories, and versioned branches can be made for each versioned archive. A beautiful, low-impact, high return solution!

Koa-Vault is available to use on GitHub and NPM.