Fetching on the edge

The edge is a buzz – and for good reason!

Serverless has many benefits. At it's core – scalable cloud functions.

Gone are the days of paying for servers to run 24/7, even if traffic is 0. And no more worrying if your servers are powerful enough for your next spike in traffic.

With serverless, you still get a server, but only when you need it! With each cloud function invokation, your serverless provider will intiailize a new instance of your operating environment, install your packages, and execute your code. After the program exits, your small partition of that server is erased, freeing space for another users instance.

Small ephemeral operating systems spun up on demand.

The problem? Initializing a disk image and installing packages takes some time.

Typically, the exection environment is nodejs, and even with only a few packages, this can take 3 seconds. 3 seconds is a long time for users to wait for a network response.

There are other things you can do to improve perceived network response speed, such as a network-cache layer like tanstack-query or swr.

The V8 runtime

The V8 runtime is a high performance javasript execution environment, developed by the chromium project in 2008. Node.js itself runs in the v8 runtime.

So if installing node.js and it's packages are the biggest problem to serverless latency, can we remove node.js and execute our code in the V8 runtime directly?

The answer is yes! And edge servers already do. Network of edge servers were originally built as "content delivery networks" to cache static assets (e.g. html files) physically close to users (to avoid long distance trips to a mega data center). Typically, edge servers simply read request headers and redirect traffic. But now, companies like cloudflare and vercel are allowing us to execute our own code directly in these edge servers.

There are other limitations of course - most notably that most npm packages cannot be installed. But the V8 runtime does support some important standard Web APIs, such as Fetch!

This means, that if your cloud function only needs to fetch data from a database, then you can reduce your network responses to milliseconds.

Beware of cache

Another major benefit of the edge is caching. The edge has been used as a cache layer for content delivery for many years. Now, it can even cache the result of your code execution! Similar to memoization, if a user makes a request to your edge api route with the same parameters as another user, then the edge server will return the cached result immediately. We're talking less than 10 milliseconds.

However, will all the new layers of cache that are appearing, it can be hard to track which data is fresh and from where. And in some cases, cache cannot be invalidated easily. So it's usually best to keep your cache invalidation low, or use a "stale while revalidate" pattern. This pattern will return the cached data immediately, but then re-execute the code and cache a fresh response, just in case.

Conclusion and the edge's future

I have begun extracting as many data fetching functions into edge api routes as possible. This way, most data is returned to the user in milliseconds, to keep them in the flow of the app.

By the time a user needs a function that requires node.js, they are usually deeper in the app, where waiting is not a problem. They expect it to take some time, because you're app is performing an important function for them.

I'm excited to see how the edge evolves. The next step seems to be full databases on the edge (such as redis). I'll be wathching closely.

Add improved version from coda