```yaml
---
slug: time-based-queues-serverless-docmq
published: 2022-08-31T12:39:00-07:00
description: Nerding out on time-ordered priority queues
changelog: ~
---
```

Building a [serverless-friendly queue system](https://taskless.io) continues to be an adventure. Recently, we had to say goodbye to our Redis queues and replace them with something more durable. The journey there involved a lot about time storage, visibility windows, and how the most popular queue solutions solve these problem. So, if you were curious what makes Taskless reliable, the short answer is a database optimized for time-based queries. For a longer answer, we need to go on a journey.

## First, it's Time for Events

The first rule of the JAMStack is that everything is an event. Because servers aren't standing by and waiting for your every request, events are the only way for a serverless provider to know your code needs to run. In practical terms, when you make a request to a Next.js app or AWS lambda, that small chunk of code is invoked in response to the URL request, executed, sends the reply, and then shuts down. This is well and good for operations like "forward this blog post to a friend" or "leave a comment about how you definitely wouldn't forward this blog post to a friend," given both examples begin with an obvious interaction. But what about an operation like "send this to my inbox tomorrow morning"?

It turns out, time ruins everything.

Time sucks, and if you don't believe me, go [read Zack Holman's excellent piece on the matter](https://zachholman.com/talk/utc-is-enough-for-everyone-right) and return once you've come to a similar conclusion. Now that we agree the concept of time was a mistake, let's talk about sending something to your inbox "tomorrow". To avoid headaches, we're going to make some poor assumptions about what tomorrow means. In Luxon, we can express this arbitrary definition of "tomorrow" as `DateTime.now().zone("America/Los_Angeles").plus({days: 1}).startOf("day").set({hour: 8})`. Bam. 8am "tomorrow" at of this exact moment. All we need to do is "event" at that time, and we're set!

## Store the Time in the Queue Itself - aka The SQS Experiment

The purest of serverless ambitions would be to never persist anything under our ownership. To that end, [Amazon SQS](https://aws.amazon.com/sqs/) and lambdas are an excellent fit. The platform has limitations, but those could be worked around with a bit of duct tape.

First, the maximum delay on a lambda is 15 minutes, so SQS would need to send the job to a lambda at the end of the maximum delay, which would only report success if it could requeue the job for another 15 minutes. We'd repeat this ad-nauseum until we reached our magical "8am tomorrow" requirement and then run our job.

This worked really well (and was insanely cheap). Each job would require a maximum of 96 SQS calls per day in a pending state, and it's only $0.50 per million requests. That's 10,000 jobs per month, delaying every job up to a maximum of one day, with just AWS tools. In practice, the system has a lot of opportunity to fail. Specifically 672 opportunities for every day the job sits in SQS and must be constantly requeued,

Honestly, if SQS didn't have an arbitrary 15 minute delay, we'd be done. But it doesn't so we're not.

## Tables, Indexed by Time

It turns out that databases are really good at time stuff, and not just because [ISO-8601](https://en.wikipedia.org/wiki/ISO_8601) is both readable and easy to sort by. But also because really smart people made time (and its storage) one of the primitives in their db systems.

Databases are also really good and indexing non-time things. So imagine you have a table

| column | type |
| :--- | :--- |
| id | any |
| failed | boolean = false |
| visible | time, nullable |
| deleted | time, nullable |
| ack | any, nullable, unique (on non-null) |
| payload | text, nullable |

Now it's easy to describe the operations in order to take, process, and report on jobs.

* Your next jobs are at `visible <= now() AND deleted IS NULL`
* Your failed jobs are at `deleted IS NOT NULL AND failed = TRUE`
* Your completed jobs are at `deleted IS NOT NULL AND failed = FALSE`
* You can retry a job by changing its `visible` time
* You can complete/fail a job by setting `deleted` and `failed` in the same transaction
* A unique `ack` value lets you find a single job in the table so that you are complete/fail/retrying only one specific job

As an added bonus, using just-a-database means that we have separated our queue (what do we need to run) from our worker (the actual running). The pattern was simple enough that I bundled it up as [DocMQ](https://github.com/jakobo/docmq). It has a persistent worker if that's something you want, or you can just run DocMQ for adding jobs to your queue database and process them on your own time.

## Time to Return to Events

The beautiful part about using a database is that once it's in the database, it's _there_. We don't need persistence unless we want it, and polling is good enough for most use cases. All we need to do is poll in response to an event; that event is _time itself changed_. 🤯

Stick with me here. Imagine if we have a cron job via [Netlify Scheduled Functions](https://docs.netlify.com/netlify-labs/experimental-features/scheduled-functions/) or [lambda cron](https://docs.aws.amazon.com/lambda/latest/dg/services-cloudwatchevents-expressions.html) that runs once per minute (or per-second if you'd like to YOLO). A single query tells you what capacity you need:

```sql
SELECT count(*) FROM your_table WHERE visible <= now() AND deleted IS NULL
```

You get 20 rows back, spin up 20 serverless functions, each to claim 1 row:

```sql
UPDATE your_table SET ack = gen_random_uuid(), visible = visible + interval '30 seconds'
WHERE visible <= now() AND deleted IS NULL
LIMIT 1
```

💥Instant scalability. If you need more granularity, tighten the cron. If you need sub-second control then you probably do want a server. Real talk though, if you're sending emails the latency isn't your job system.

## How This Relates to Taskless

At [Taskless](https://taskless.io) we ran on the cron job / polling design until we were invoking an unreasonable number of serverless functions on the minute and customers wanted sub-minute responses. Since our scheduled function infrastructure didn't let us go to per-second, we switched to persistence. In short, we ran the servers so you could go 100% serverless. 🎉

Because [DocMQ](https://github.com/jakobo/docmq) is on GitHub, you can replicate it yourself. There's even a [postgres prototype](https://github.com/jakobo/docmq/tree/labs-postgres) that will get merged down soon in case you're a fan of Neon, Supabase, or other serverless-friendly postgres instances.

### Some Quick Answers

Added because these came up in casual conversation

* Does the DB matter? - No, as long as your DB understands time
* Biggest drawback? - Performance. A relational db will be slower than an in-memory solution in almost every use case. Unless you're building a real-time application such as chat, the latency of a DB touching disk is not going to be why things are slow
* What DB does Taskless use? - We use MongoDB Atlas currently. We like their time series support



Designing A Time-Based Queue for Serverless

```yaml
---
slug: nodejs-memory-leak-heroku
published: 2022-07-19
description: You can connect to your production Heroku instance, debug it live, and even trace down memory leaks.
keywords: heroku, r14, r15, memory leak
changelog: ~
---
```

There's nothing more frustrating that your node.js app leaking memory. If you're coming here because of Heroku R14 / R15 errors, take some deep breaths, it'll be okay. Heroku's smart enough to swap to disk when this happens, and will kill and restart your app if things get absolutely wild.

## Background

![Heroku Dashboard with a lovely orange band of R14 errors](https://user-images.githubusercontent.com/1795/163650457-2371246f-bc80-4482-b42e-8bfdbf664a1a.png)

It started with my email box getting pelted with warnings that the memory quota on a project I advise for was getting hit with R14 errors. Going to the Heroku metrics page confirmed that yes, something was clearly wrong. In the above picture, you can see again and again the memory usage surge and then Heroku responding with an R15 and force rebooting the server. Moments later, the memory usage was growing again.

It's a Memory Leak, Ruin of Productivity and afternoons. 😭 

However, since the last time I dealt with a memory leak (2011) the tools have come a long way. So here's how you triage a production memory issue, run analysis against your currently leaking production boxes, and stamp out the issue.

## Modern Tools

The tools for dealing with a memory leak have gotten much better, especially if you're on Heroku. You only need two:

* A current version of Chrome (which comes with the amazing v8 debugger)
* The Heroku CLI (if on Heroku)

The first does 90% of the heavy lifting, while the second now has all the commands we need to turn on a debugger in production and forward debug data to our local machine. Make sure you've logged in to the Heroku CLI if you're triaging Heroku.

## Opening Chrome's Inspector

The Chrome Developer Tools are always available on `chrome://inspect`, which can be put in your browser's address bar. The DevTools are useful for debugging everything from node.js processes to Service workers. For now, we want to stay on the `Devices` tab, make sure `Discover network targets` is checked, and leave that window open while we set other things up. You may even see a few device targets show up that Chrome's discovered. In my case, apparently my TV can be seen. Odd, but neat.

![The Chrome developer tools](https://user-images.githubusercontent.com/1795/163651415-d75abf95-e781-45fb-ab2c-3e9adcb6ba1d.png)

## Enabling the Debugger With a Signal

The most common way to start debugging node is to add the `--inspect` CLI arg, either directly (via `node --inspect <your file>`), or in the appropriate env values ( `NODE_OPTIONS="--inspect" node <your file>`). In production environments, it may not be desirable to enable the debugger on every start; in the case of Heroku which randomly assigns ports, you can't even guarantee the debug port of `9229` is available. But, you can enable the debugger a different way.

From the [node.js debugging page](https://nodejs.org/en/docs/guides/debugging-getting-started/)

> Node.js will also start listening for debugging messages if it receives a SIGUSR1 signal. (SIGUSR1 is not available on Windows.) In Node.js 7 and earlier, this activates the legacy Debugger API. In Node.js 8 and later, it will activate the Inspector API.

Sending a signal is as easy as using the unix `kill` command, where `<PID>` is your node.js process ID:

```sh
$ kill -usr1 <PID>
```

You can try this locally by running `ps aux | grep node` to find your node.js process, and then using the `PID` for the above `kill` command. Personally, I find that the `usr1` signal is not only more reliable (works everywhere) but also means you don't have to completely restart the server. When you send `usr1`, node just sees the signal and starts up its debugger. It'll pop up moments later in the Chrome DevTools.

![local development showing the inspector starting](https://user-images.githubusercontent.com/1795/163651616-5d423ec2-bdd9-4bcd-828c-29e4bbf5335e.png)

![the development tools showing the new remote target](https://user-images.githubusercontent.com/1795/163651579-b7181e72-86d9-47e9-9d46-eb4c3cf76bc4.png)

Hit the `inspect` link and you'll be taken to a very familiar DevTools interface, but optimized for node.js debugging. In addition to the Console and Sources, you'll also have "Profiler" for CPU profiling, and "Memory" for memory usage profiling,

![connected development tools, ready to go](https://user-images.githubusercontent.com/1795/163651715-22c7a39a-81cb-43da-bc97-4e17523ffb56.png)

## Connecting to Heroku in Production

Once that's all working, it's time to connect to our production instance in Heroku and take some live samples. Close the inspector and local development server if needed, and make sure your `heroku` command works. If you're using a different environment such as Digital Ocean, look up SSH instructions to get to your host. They'll be different on every provider.

For heroku, there's the `ps:exec` command which gives you a shell on your default process for dyno 1. Be sure to specify your `<APP_NAME>` so that you're ssh-ing into the correct box, and if you need to connect to a dyno process that's not `web`, add the `-d <dyno>` flag.

```sh
$ heroku ps:exec -a <APP_NAME>
```

![connecting to a remote box over SSH](https://user-images.githubusercontent.com/1795/163652037-8309d4c9-452a-4bf5-848a-98f466346471.png)

From here, we can search our node process with `ps aux | grep node` and start the debugger with `kill -usr1 <PID>`. Our debugger will start, but obviously we can't see it yet. Heroku (and other providers) don't expose port `9229` to the internet for security reasons. Open another terminal window and use heroku's forwarding command to create a local tunnel for your debugger (and add the optional `-d` flag if needed).

```sh
$ heroku ps:forward -a <APP_NAME>
```

And just like that, your production instance shows up in the Chrome DevTools.

## Sampling

Taking samples is straightforward. Wait for the VM instance size to grow, click the trash can icon (force GC) to ensure it's actually grown by a reasonable amount, and then hit the **Take Snapshot** button to capture a memory heap. Chrome DevTools will be busy for a bit, but it'll create a Heap, which it adds to the left-hand side of the Memory profiler. Every heap is a detailed analysis of what was in JavaScript at that exact moment. Every string, object, key, event, closure, callback; you name it, it's here.

Collect several samples (5-6 is ideal) since you'll be looking for identifiable trends. I also recommend waiting about 2-3 MB between each sample. We want our growth in memory to be noticeable, and whatever's leaking to stick out.

Here, I've got 6 production samples of a leaking app. In the time I sampled, we added about 15MB to the footprint. 😱 

![chrome devtools with several heaps loaded](https://user-images.githubusercontent.com/1795/163652520-5df538be-aa9a-4ae7-bf41-17617abe030f.png)

What matters is we can see the memory growing, we have several samples of the Heaps in question (which contain our growing memory), and those Heaps are readable by the Chrome DevTools. We can save the Heaps, disconnect from our production machine (which also closes the debugger), and get to our analysis.

## Using Heap Compare

Select the newest (largest) Heap, and above the list of objects in the heap, change from `Summary` to `Comparison`. This changes the Heap view from telling you what's in the Heap to what changed **between** Heaps.

![how to change from summary to comparison view](https://user-images.githubusercontent.com/1795/163652680-cd5f2530-6b12-4c00-8014-326df53c8a80.png)

Sort the table by `# Delta`, and the problem suddenly becomes very obvious.

![a sorted by delta comparison](https://user-images.githubusercontent.com/1795/163652782-5832abe0-c4c3-4e79-b804-100f1916fa47.png)

We can compare against a few other Heaps (that's why we captured them) to isolate the source of the memory leak. Here, I'm comparing Heap 5 to Heap 4, where the number of `(concatenated string)` is much smaller and easier to manage. There's a lot of weird date strings like "-137 days, 21" being created and retained. Clicking on one gives us the full Retainer log. It's part of a larger string, that's part of a larger string, that's in `discountTimeLeft` in the update cycle of a React component called `DiscountBanner`, inside of a... `Timeout`.

![The memory leak source](https://user-images.githubusercontent.com/1795/163653410-35a401c4-0b85-4eb6-9c14-4eec51878a0d.png)

Opening up the `DiscountBanner` culprit, there's a `setInterval` in the `constructor`. Fixing this and redeploying closed the memory leak. It turned out that when next.js was running in Heroku, the `setInterval` would just keep on running and building its string about how many days were left on the discount. Since that string was locked in the `setInterval` callback, it could never be garbage collected by v8. Add in a few thousand requests per second, and those discount strings would ultimately take up all of the available memory and cause Heroku to restart the server with an R15 error.

## More than Heaps

I only needed the Heap compare, but the CPU profiler and the Memory timeline (Allocation instrumentation on timeline) can help narrow the issue even further in a larger application. The hardest part is getting a reproducible issue. Looking at production not only makes it easier, but ensures you're solving the right problem. And when you're done, you'll never put a `setInterval` in a React component constructor again. 😆 

Solving a Node.js Memory Leak on Heroku

```yaml
---
slug: use-github-as-a-cms
published: 2022-01-11T11:40:00-08:00
description: GitHub's rich GraphQL API gives you a built-in headless CMS
changelog: ~
---
```

First thing's first. I really love [Ghost](https://ghost.org/). It's got an excellent content and admin API and I'd honestly consider it better than many other Headless CMS options. Second, this isn't a post about Ghost. There were two major shortcomings in Ghost that ultimately I couldn't find good workarounds for.

1. Custom structured data. As it stands I was jamming everything into `codeinjection_foot` as YAML and hoping I didn't ever need that field. I also needed to comment my code explaining all my additional metadata resided in the field not called `metadata`. Yikes.
2. Something that isn't Mobiledoc. I understand why it's used. I don't like that my parsing process on Codedrift requires that I start from the final HTML and then throw it all in [rehype](https://github.com/jakobo/codedrift/blob/9d0bd286355f5b80d1ba71561d78603a9a388ed0/src/components/markup/rehype.js#L25) to [fix URL links, remove widowed text, and more](https://github.com/jakobo/codedrift/blob/0f5991456f576dd031f2e95888af5878a63f16c3/src/graphql/util.js).

HTML to AST back to HTML is not a good feeling and the pipeline was brittle. When I realized I didn’t want to re-add tags on my posts to Codedrift because of the complexity, I looked for alternatives.

I saw [Casual JavaScript](https://github.com/casualjavascript/blog) which was running the entire site off of [GitHub Issues](https://github.com/casualjavascript/blog/blob/gh-pages/index.html). It wasn't perfect, but GitHub Issues are in Markdown, and Markdown _does_ support frontmatter, and frontmatter _is literally designed for metadata_.

The missing piece was something that wasn't issue based. [Discussions](https://github.blog/changelog/2021-02-23-github-discussions-graphql-api-public-beta/). Sold.

## Core Tools
The original Codedrift had a GraphQL micro server that ran everything. It was overengineered (like all good personal projects should be), but in the rewrite I wanted to remove as much of my code as possible. The entire GraphQL infrastructure became a simple [http proxy](https://github.com/jakobo/codedrift/blob/9d5b0349dc204cd1076b54fd5e475dffe449c84d/src/pages/api/proxy/api.github.com/graphql.ts) using [http-proxy-middleware](https://www.npmjs.com/package/http-proxy-middleware). This way I wouldn't need to expose a personal access token in the browser.

The next step was to use [graphql-code-generator](https://www.graphql-code-generator.com/) for Typescript typed Urql hooks. I'm still a fan of Urql, and their [docs for next.js integration](https://formidable.com/open-source/urql/docs/advanced/server-side-rendering/#nextjs) are easy to follow and set up.

Because scope creep is a thing, the site's now also in Typescript. 🥳 

So now I've got typed GraphQL queries to GitHub's API without exposing the access token. All that's left is to get our Discussions, turn them into blog posts, and solve for weird cases like [tag-based searches](https://github.com/jakobo/codedrift/blob/main/src/pages/thunked/tag/%5Bname%5D.tsx#L19) which need to go through the `search` endpoint instead of the `repository` endpoint. Consistency (and sanity) between queries came from the Post fragment.

## The Post Fragment
Fragments are [better explained by graphql.org](https://graphql.org/learn/queries/#fragments), but in context they are the reusable unit of documents. In this case, a fragment on `Discussions` will let me get the same fields every time for our discussion, regardless if I arrived at the Discussion via the `search` endpoint or a `repository` endpoint. Even though it might overfetch a little for its use case, a fragment pushed through graphql-code-generator will give you a typed object representing the returned data structure.

In other words, having this GraphQL fragment
```gql
fragment PostDetails on Discussion {
  id
  title
  lastEditedAt
  url
  labels(first: 100) {
    nodes {
      id
      name
      description
    }
  }
  author {
    avatarUrl(size: 10)
    ... on User {
      id
      name
    }
  }
  body
}
```

Will give us this type in our generated file:
```ts
export type PostDetailsFragment = {
  __typename?: "Discussion";
  id: string;
  title: string;
  lastEditedAt?: any | null | undefined;
  url: any;
  body: string;
  labels?:
    | {
        __typename?: "LabelConnection";
        nodes?:
          | Array<
              | {
                  __typename?: "Label";
                  id: string;
                  name: string;
                  description?: string | null | undefined;
                }
              | null
              | undefined
            >
          | null
          | undefined;
      }
    | null
    | undefined;
  author?:
    | { __typename?: "Bot"; avatarUrl: any }
    | { __typename?: "EnterpriseUserAccount"; avatarUrl: any }
    | { __typename?: "Mannequin"; avatarUrl: any }
    | { __typename?: "Organization"; avatarUrl: any }
    | {
        __typename?: "User";
        id: string;
        name?: string | null | undefined;
        avatarUrl: any;
      }
    | null
    | undefined;
};
```

Even a `Partial<PostDetailsFragment>` adds excellent safeguards to a function when I expect to receive data shaped like the GraphQL response for a Discussion.

## Slugs and Search
Fetching the most recent posts is [a straightforward process](https://github.com/jakobo/codedrift/blob/cbcc0f4beac26e611c14cb789a7adc5b3ba7c55a/src/graphql/posts.graphql#L27) since the API can go from a repository to its discussions and limit those to a single category. For accessing a single post by anything other than its ID, there's only the `search` endpoint.

A generic search on GitHub takes a search query as a string. It's the same string used in [GitHub's Advanced Search](https://github.com/search/advanced?r=jakobo%2Fcodedrift) which at least means it's easy to test and iterate on, even if there is no official query builder. The result is a very generic GraphQL query for searching, which can be used for both slugs and tags. It could probably also be used for the recently created posts, but I'd prefer less magic strings where possible.

```gql
query SelectPostsWithSearch($search: String!, $first: Int = 1, $after: String) {
  search(first: $first, after: $after, type: DISCUSSION, query: $search) {
    nodes {
      ... on Discussion {
        ...PostDetails
      }
    }
    discussionCount
    pageInfo {
      endCursor
      hasNextPage
      hasPreviousPage
      startCursor
    }
  }
}
```

The first of two major downside to this approach is I need to constantly cast the node collection to `Discussion`, but I feel that's more my inexperience with Typescript than the language itself. The other is that because when running a search, there's always a risk of returning two records for a search. I briefly thought about changing the URLs to be `slug/post_id` but that felt like it was binding me too closely to GitHub's ID system.

## The New Post Workflow
Posts on Codedrift start in [my public notes](https://codedrift.com/notes), but eventually they’re well-formed enough to get ideas and opinions. At that point, there’s discussion on the draft, which used to exist exclusively over email and Twitter DMs. These processes also worked, but it always felt like I was missing an opportunity for more open discourse about a topic.

So now, when a post grows out of Codedrift, it’ll become an [issue](https://github.com/jakobo/codedrift/issues?q=is%3Aopen+is%3Aissue+label%3A%22%E2%9C%92+Draft%22), When it’s ready to be published, it can be moved to a Discussion, and GitHub saves all of the relevant comments and edits in the transfer. This has the added benefit of matching my mental model. Unposted content is an issue which certainly needs resolution. Posted content deserves a discussion.

The code for all of this is in the [Codedrift repository](https://github.com/jakobo/codedrift), though now that I'm just calling a GitHub API with no interference, the code feels very vanilla. And that feels pretty good.

Hey 👋🏼, I’m Jakob

Designing A Time-Based Queue for Serverless

Solving a Node.js Memory Leak on Heroku

GitHub as a CMS