Traveling Coderman Blog

Avoid Runtime Errors with Optionals

2019-08-19T00:00:00Z

Null is bad for representing the absence of a value, use Optionals instead. Use Optional.flatMap to chain multiple operations that might not return a value.

The problem with null 🔗

Let's recap what null is and what problem it entails.

The language Java allows every non-primitive variable to take the value null. If a variable holds the value null, then all method calls to it will fail with a NullPointerException at runtime. The compiler can only give limited hints if a method call on a variable is safe. Therefore, the programmer needs to make assumptions if a method call is safe. If they are wrong with their assumptions, NullPointerExceptions might occur during runtime, interrupting the control flow, even possibly taking down the application. If the programmer is over-cautious, then the code is riddled with (necessary and unnecessary) null-checks.

Why use null anyway? 🔗

In many programs, a null value is used to indicate the absence of a value. This is a less noisy option than to signalize each absence of a value with an exception. In some cases, a null value is used to indicate that a variable has not been initialized yet. Some use cases might even assign the null value of a variable a dedicated meaning.

To motivate the usage of Optionals instead of nulls, let's first take a look at an example using null to represent the absence of a value. After that, we solve the example in a better way with Optionals.

Modeling with null 🔗

Let's consider an interface Person. It contains a method getMiddleName to retrieve the middle name of the person. Not every person has a middle name, therefore this method returns null when the person does not have a middle name.

interface Person {
  public String getMiddleName();
}

Additionally, we define a function to retrieve a person by id. This function fetches a person from a data storage. In case no person with the given ID exists, the function returns null.

public static Person findPerson(int id) { /* … */ }

Now assume we want to write a function returning the middle name of a person by the ID of the person.

public static String getPersonMiddleName(int id) {
  return findPerson(id).getMiddleName();
}

We fetch a person by its ID and then return its middle name. Unfortunately, this naive implementation is erroneous although it compiles without errors. It will return with a NullPointerException in case no person with the given ID exists. To fix this, we can return null from getPersonMiddleName if findPerson returned null. This entails that the caller of getPersonMiddleName can not determine the reason for the result being null. For our use case, we consider this to be fine.

Note The Either type goes a step further than the Optional type and can fix this.

Trying to solve the NullPointerException without Optionals, we are required to inspect the return value of findPerson and differentiate between a null return value and a non-null return value.

public static String getPersonMiddleName(int id) {
  Person person = findPerson(id);
  if (person == null) {
    return null;
  }
  return person.getMiddleName();
}

Now, the function can not result in a NullPointerException anymore. But it requires three additional lines of null handling and the compiler did not support us in preventing the NullPointerException in the first place.

Refining with Optional 🔗

Let's take a look at the alternative approach with Optionals.

First, we change the method getMiddleName to explicitly state that it might return an absent value.

interface Person {
  public Optional<String> getMiddleName();
}

Now, the method returns either the middle name in an Optional or an empty Optional. It will by contract never return a null value. Second, we make the same change to findPerson.

public static Optional<Person> findPerson(int id) { /* … */ }

A call to findPersonalways returns a non-null Optional. It either contains the found person or it is empty. Let's build the function getPersonMiddleName. As we did with the null solution, let's first implement it in the naive way.

public static Optional<String> getPersonMiddleName(int id) {
  return findPerson(id).getMiddleName();
}

This time, the compiler will fail because the types do not match. The expression findPerson(id) resolves to an instance of Optional<Person> and therefore getMiddleName() can not be called on the return value. Instead of running into an issue at runtime, we have caught the mistake already at compile time. We can use Optional.flatMap to define what should happen in case findPerson(id) leads to a result. The method flatMap in the context of Optionals automatically handles the case where the Optional is empty and will return an empty Optional in that case.

public static Optional<String> getPersonMiddleName(int id) {
  return findPerson(id)
    .flatMap(person -> person.getMiddleName());
}

If findPerson(id) resolves to an empty Optional, then the lambda person -> person.getMiddleName() won't be called and getPersonMiddleName will return an empty Optional. If findPerson(id) resolves to a person inside of an Optional, then the lambda person -> person.getMiddleName() will be called with the person. In that case, two further distinctions happen. If person.getMiddleName() resolves to an empty Optional, then the overall result of getPersonMiddleName is an empty Optional as well. If person.getMiddleName() resolves to a string inside of an Optional, then both calls findPerson(id) and person.getMiddleName() have yielded a present value and the overall result of getPersonMiddleName is the middle name of the person with the given ID inside of an Optional.

Summary 🔗

With Optionals, we detected a runtime bug already at compile time. The method Optional.flatMap allowed us to not have an explicit if-else-case distinction.

The One-Hour Interview Process

2021-12-14T00:00:00Z

Evaluating software engineers within the timespan of an interview is tough. You want to figure out if the applicant can help your team and at the same time make them want to work with you. And this ideally in an hour without requiring the applicant to beforehand complete a lengthy at-home assignment. In this post you find a guide to interview a software engineer applicant.

What's wrong with the status quo in interviewing? 🔗

In the last couple of years, I have sat on both sides of software engineering interviews and the interview process never felt quite right. Many companies have a multi-step process involving multiple interviews and at-home assignments. There are several issues I've seen with this process.

At-home assignments They require a lot of the applicants time. This is limiting the pool of applicants to those that are able and willing to invest that time. It puts persons in a full-time job, other duties outside of the job and those evaluating several companies at a disadvantage.
Focus on evaluating the applicant In an interview, both the company and the applicant are evaluating each other. If the interview isn't conveying realistic problems but instead uses an artificial coding problem, then you are missing an opportunity to make the applicant excited for your company.
Focus on evaluating skills instead of potential If you are evaluating the knowledge with a specific framework or library, then this requires a lot of preparation time for the applicant or you limit yourself to those engineers that have worked with that framework or library but might not be able to adapt once your tech stack changes. Instead, evaluate for timeless skills, for the ability to quickly grasp new new concepts and for soft skills.
Focus on deep knowledge instead of broad knowledge Especially pure coding assignments focus a lot on the same skill and evaluate it deeply, instead of touching several topics.
No different opinions after several interviews For a vast majority of applicants, the opinions of the different interviewers whether to hire or not to hire are identical. Every time this is the case, then one interview could have been sufficient.

The Goals of an Interview 🔗

Let's remind ourselves what we want to achieve with the interview process.

Assess the applicants knowledge: Are they able to work within a service that offers an HTTP API and/or a frontend? Can they work with an SQL database and can they work with your programming style?
Assess the applicants soft skills: Do they ask the right questions? Are they curious and eager for feedback? Do they collaborate and how do they react to mistakes or criticism?
Assess the seniority of the applicant: Should you hire them as junior, mid-level or senior engineers?
Make the applicant want to work at your company: What type of applications do you write? What kind of problems will they face in your daily work? What decisions will the applicant own? What skills will they learn and what can you teach them?
Give equal opportunity to every applicant: Interviews are stressful for applicants, you want them to feel as comfortable as possible such that it doesn't matter whether you are facing an introvert or extrovert, whether a person from a minority or majority, whether a person applying first-time or thirty years into the profession.
Minimize time: Applicants might apply at multiple positions, still work at a current job or have other duties to care for. You don't want to waste their time with a home-assignment or multiple lengthy interview sessions. You don't want to miss out on some engineers because they didn't have the time for the interview process.

The Framework 🔗

This guide assumes a role that involves the development of a product, which involves a backend server application with an SQL database, an HTTP API and a dedicated web frontend. For such a role, it is crucial to figure out how skilled the applicant is regarding HTTP APIs, general coding, frontend development and databases. Additionally, as a functional programmer, I'd like to assess for familiarity with functional programming paradigms. Each of these topics makes up a block of similar time. A block consists of questions or a problem statement that you as the interviewer ask or formulate within a conversation with the applicant. With each block, it is the idea that junior applicants can figure out an answer and that more senior applicants can go into depth outlining different aspects or multiple solution or initiate discussions on them. There are no clear correct or wrong answers.

Note Nervousness arises if the applicant does not know what you expect from them. Be upfront with the topics you want to cover and what skills you are looking for such that the applicant feels comfortable and isn't limited by their nervousness.

Each topic aims to figure out where the knowledge of the applicant ends. Some applicants might struggle with the first question of the block. In that case, try to lead them to an answer and then skip the rest of the topic moving on to the next topic. Afterwards, ask yourself if you can teach that missing knowledge to the applicant. Should they have known this considering their past experience? Were they curious about the answer they didn't know?

Note Don't reveal how many questions you want to ask in each topic! You want to be able to move on without leaving the applicant feel like they missed half of the things they could have known. If you feel like the applicant would appreciate the feedback, you can tell them afterwards in which topics you would usually have asked more and how they can advance their knowledge in these topics.

Without any further ado, let's look into the details of the five blocks.

APIs 🔗

Has the applicant knowledge about HTTP APIs, how to use them and how to model them?
Does the applicant ask clarifying questions and makes reasonable assumptions?
Convey to the applicant how you model APIs

Question Which common HTTP methods besides of GET can you think of and how would you use them?

With this question you can check if the applicant has either modeled or used an HTTP API. With explicitly mentioning GET in the question, you avoid confusion because of unfamiliarity with the term HTTP method. If the applicant answers this question without a blink of an eye, then you can talk with them about the properties of these methods, with for example two identical PUT calls leading to the same state as if only one of them would have been executed.

GET: Retrieve an entity or a list of entities
POST: Create an entity or send command
PUT: Create or exchange an entity
DELETE: Delete an entity
PATCH: Partially change an existing entity (Bonus)

Question Let's model an HTTP API of organizations and users where each organization consists of multiple users. There already exists an endpoint GET /orgs that returns a list of all organizations when called. Can you please list several endpoints you would introduce to allow creating, changing and retrieving of both organizations and users?

This question aims to figure out how much experience the applicant has in modeling HTTP APIs. There are multiple solutions which are justifiable, the most senior applicants will list name them with their advantages and disadvantages.

Note Only ask this question, if the applicant has answered the previous question confidently.

The main catch with this question is how organizations and users relate. The problem statement only states that organizations consists of users, it doesn't make any statement whether users can exist on their own, whether they can belong to multiple organizations or if they can change organizations. The most senior applicant will ask clarifying questions or outline to you the different ways how to model the API based on these assumptions.

If the applicant assumes that users are bound to organizations, then they'll likely come up with an API design similar to this one.

GET /orgs
GET /orgs/{id}
POST /orgs
PUT /orgs/{id}
GET /orgs/{id}/users
GET /orgs/{id}/users/{id}
POST /orgs/{id}/users
PUT /orgs/{id}/users/{id}

If they do and you feel like the applicant is able to discuss their solution confidently, you might want to challenge their design with a follow-up question.

Question Now a use case comes up that a user now changes its organization and we would like to cover this with the API. How would you approach this?

If the applicant came up with the solution modeling the user as a sub resource of the organization /orgs/{id}/users, then they might now change their mind and model the user as a top-level resource /users with the user entity referencing the ID of the organization { orgId: '123' }. Some applicants might also come up with a non-RESTful solution like POST /orgs/{id}/users/switch. That can be a starting point for a conversation about REST with more senior applicants.

GET /orgs
GET /orgs/{id}
POST /orgs
PUT /orgs/{id}
GET /users
GET /users/{id}
POST /users
PUT /users/{id}

Functional programming 🔗

This block aims to assess the ability of the applicant to utilize functional programming paradigms. You might want to skip this section if it does not apply to your software development philosophy.

Has the applicant applied functional programming before?
Establish that the coding challenge in upcoming blocks can be solved with a functional paradigm if this applicant sees it as suitable.

Question Across various programming languages, which ways do you know to model the absence of a value?

Most Java and JavaScript programmers are able to name null and/or undefined. Java programmers with some functional background should be able to name Optional from the standard library, JavaScript programmers might mention one of the similar options from third-party libraries. Java programmers might mention the null object design pattern.

null / undefined
Optional / Option / Maybe
Null Object Pattern (Bonus)

If the applicant mentions Optional, Option or Maybe, you can assume some functional background and check how deep the knowledge is. If the applicant does not mention it, you can skip to the next block.

Question Assume you have two existing functions Optional<User> lookupUserByName(String name) and Optional<Org> lookupOrgByUser(User user). Could you please implement a function Optional<Org> lookupOrgByUserName(String userName) that returns the organization of a user given its user name?

This question specifically aims for the usage of flatMap.

public Optional<Org> lookupOrgByUserName(String userName) {
    return lookupUserByName(userName)
        .flatMap(user -> lookupOrgByUser(user));
}

Applicants, who have heard of Optional but are coming from a more procedural programming style, might fall back to a solution with isPresent and get. This might not be a problem, you need to ask yourself the questions: Can you teach them? Will they be able to work with your code if it is using more advanced functional programming patterns?

public Optional<Org> lookupOrgByUserName(String userName) {
    var user = lookupUserByName(userName);
    if (!user.isPresent()) {
        return Optional.empty();
    }
    return lookupOrgByUser(user.get());
}

Coding 🔗

The coding block aims to determine basic programming ability.

Is the applicant able to code? Do they struggle with syntax? How do they structure the code, do they find a starting point?
Do they clarify unclear criteria, think about testing and communicate during coding?
Can the applicant stick to existing coding conventions and adapt their style?
Convey a semi-realistic problem that could come up during work

This block expects the applicant to be convenient with a programming language but not with a specific framework or library. It formulates a problem that will likely come up during work in some way or the other: You have some data in the database, some other information available via an API and you want to apply some logic based on which you are sending out notifications.

Question We now would like to implement a daily job that sends out notifications to all users of organizations on their founding date. There is an external API that allows us to retrieve the founding date of an organization and there is an existing function to send out notifications.

It gives the applicant some existing models (User, Org and Notification) and some existing functionality (services NotificationService, FoundingDateService and OrgDao in Java, functions sendNotification, retrieveFoundingDate and selectAllOrgs in TypeScript) and expects the applicant to assemble these together into a solution.

Note Junior applicants might struggle to find a starting point. While a senior applicant should be able to find that starting point thyself, you might want to give a junior applicant a skeleton of a service with an empty method (Java) or a function with parameters (TypeScript).

These are the classes and methods you can provide to the applicant. The left out body is not relevant for the code the applicant needs to write, they can focus on the signatures.

class User {
    public String getName() { /* ... */ }
}
class Org {
    public List<User> getUsers() { /* ... */ }
    public String getName() { /* ... */ }
}
class Notification {
    public Notification(String orgName, String userName) { /* ... */ }
}
class NotificationService {
    public void sendNotification(Notification notification) { /* ... */ }
    public void sendNotifications(List<Notification> notifications) { /* ... */ }
}
class FoundingDateService {
    public Date retrieveFoundingDate(String orgName) { /* ... */ }
    public Map<String, Date> retrieveFoundingDates(List<String> orgNames) { /* ... */ }
}
class OrgDao {
    public List<Org> selectAllOrgs() { /* ... */ }
}
class DateUtil {
    public static boolean haveSameDayOfYear(Date date1, Date date2) { /* ... */ }
}
interface Job {
    void execute();
}

In TypeScript, the existing code can look like this.

interface User {
  name: string;
}
interface Org {
  users: User[];
  name: string;
}
interface Notification {
  orgName: string;
  userName: string;
}
async function sendNotification(notification: Notification): Promise<void> {
  /* ... */
}
async function sendNotifications(notifications: Notification[]): Promise<void> {
  /* ... */
}
async function retrieveFoundingDate(orgName: string): Promise<Date> {
  /* ... */
}
async function retrieveFoundingDates(
  orgNames: string[]
): Promise<Record<string, Date>> {
  /* ... */
}
async function selectAllOrgs(): Promise<Org[]> {
  /* ... */
}
function haveSameDayOfYear(date1: Date, date2: Date): boolean {
  /* ... */
}

Beside the basic coding skills, this challenge also allows you to assess some other things.

Does the applicant consider performance and error-handling (sendNotification vs. sendNotifications and retrieveFoundingDate vs. retrieveFoundingDates). The single versions might be slower because they might result in single calls and they might fail individually.
Does the applicant utilize an imperative or functional paradigm?
Does the applicant tend to over-engineering?

A functional solution in Java could look like this.

public class FoundingDateNotificationJob implements Job {

    private final OrgDao orgDao;
    private final FoundingDateService foundingDateService;
    private final NotificationService notificationService;

    public FoundingDateNotificationJob(OrgDao orgDao, FoundingDateService foundingDateService, NotificationService notificationService) {
        this.orgDao = orgDao;
        this.foundingDateService = foundingDateService;
        this.notificationService = notificationService;
    }

    public void execute() {
        var orgs = this.orgDao.selectAllOrgs();
        var foundingDates = this.foundingDateService.retrieveFoundingDates(extractOrgNames(orgs));
        var today = new Date();
        var notifications = createNotifications(orgs, foundingDates, today);
        this.notificationService.sendNotifications(notifications);
    }

    private static List<String> extractOrgNames(List<Org> orgs) {
        return orgs.stream()
            .map(Org::getName)
            .collect(toList());
    }

    private static List<Notification> createNotifications(List<Org> orgs, Map<String, Date> foundingDates, Date today) {
        return orgs.stream()
            .filter(org -> DateUtil.haveSameDayOfYear(foundingDates.get(org.getName()), today))
            .flatMap(org -> org.getUsers().stream()
                .map(user -> new Notification(org.getName(), user.getName()))
            )
            .collect(toList());
    }
}

An imperative version (no judgment!) could look like this.

public class FoundingDateNotificationJob implements Job {

    private final OrgDao orgDao;
    private final FoundingDateService foundingDateService;
    private final NotificationService notificationService;

    public FoundingDateNotificationJob(OrgDao orgDao, FoundingDateService foundingDateService, NotificationService notificationService) {
        this.orgDao = orgDao;
        this.foundingDateService = foundingDateService;
        this.notificationService = notificationService;
    }

    public void execute() {
        var orgs = this.orgDao.selectAllOrgs();
        var today = new Date();
        for (var org : orgs) {
            var foundingDate = this.foundingDateService.retrieveFoundingDate(org.getName());
            if (DateUtil.haveSameDayOfYear(foundingDate, today)) {
                for (var user : users) {
                    var notification = new Notification(org.getName(), user.getName());
                    this.notificationService.sendNotification(notification);
                }
            }
        }
    }
}

A functional solution in TypeScript looks similar to the functional solution in Java.

async function executeFoundingDateJob(): Promise<void> {
  const orgs = await selectAllOrgs();
  const foundingDates = await retrieveFoundingDates(
    orgs.map((org) => org.name)
  );
  const today = new Date();
  const notifications = createNotifications(orgs, foundingDates, today);
  await sendNotifications(notifications);
}

function createNotifications(
  orgs: Org[],
  foundingDates: Record<string, Date>,
  today: Date
): Notification[] {
  return orgs
    .filter((org) => haveSameDayOfYear(foundingDates[org.name], today))
    .flatMap((org) =>
      org.users.map((user) => ({
        orgName: org.name,
        userName: user.name,
      }))
    );
}

The imperative solution in TypeScript could look like this.

async function executeFoundingDateJob() {
  const orgs = await selectAllOrgs();
  const today = new Date();
  for (const org of orgs) {
    const foundingDate = await retrieveFoundingDate(org.name);
    if (haveSameDayOfYear(foundingDate, today)) {
      for (const user of users) {
        const notification = new Notification(org.name, user.name);
        await sendNotification(notification);
      }
    }
  }
}

Frontend 🔗

If you are developing for a frontend or full-stack role, you might want to check for the applicants ability to work in the frontend. While the previous coding block represents a typical backend problem, it does not require any library or framework knowledge. Therefore, it is equally useful to assess the coding skills of a frontend developer. This block aims to assess the frontend skills without requiring a time-consuming coding problem.

Is the applicant able to conceptualize a frontend feature?
Does the applicant consider usability and accessibility?
How deep is the frontend experience of the applicant?

Question We now would like to introduce a search in the frontend to allow to find a user by their name. After a search term has been entered, a list of matching users should appear and it should be possible to select one of them to navigate to a detail view of that user. The endpoint GET /users already got extended to support a query parameter q. Can you elaborate how you would implement the frontend part of such a search such that it is highly usable and leads users fast towards the search result they were looking for?

This question can be answered in one sentence but also in a lengthy conversation. If the applicant considers the question answered after a sentence, try to challenge their solution or remind them of different aspects or edge cases.

A standard solution should have these or similar properties.

A text input field to allow enter a search term
A backend request is executed either when
- the user has stopped typing for a duration
- the user has pressed enter or clicked a search button
When a backend request is executed, there is some loading indicator
When a backend request returns with results, the loading indicator disappears and a list of results is shown
A shown result can be clicked, opening a detail view

Additionally, other things might be mentioned or can be utilized by you to check the depth of the applicants experience.

If there are no results, then a text is shown indicating that there are no results, asking the user to refine the term or making an alternative suggestion
Once a loading indicator is shown, previous results are not shown anymore
If a search term is empty or less than a number of characters, then no search is started
When a user continues typing during a backend request, then the request is stopped and the loading indicator disappears
The shown results indicate why they have been returned (Highlighting of search term, etc.)
The user can use the up and down keys to go through the list and press enter to open the detail view. The user can still type letters to refine the search term. (The focus never leaves the input field)
On hover over a result it is highlighted

Databases 🔗

This last block aims to assess the familiarity with modeling data in a relational database. You might want to exchange this block if you are primarily using document-based databases.

Is the applicant familiar with models in a relational database?
Is the applicant able to interact with an SQL database?
Is the applicant able to work in an existing database schema?

Question Let's take the example of orgs and users and assume that both entities have only an id and a name. Furthermore users are always within exactly one org. Which tables and with which columns would you create in an SQL database to model these two entities?

This question covers basic relational modeling with one table referencing the other table in a one-to-many relationship. Applicants might struggle with how to represent these tables. Remind them that already table names, column names and their types are sufficient. If they continue to struggle, you might want to give them the orgs table and ask them to give you the users table based on that.

Table orgs: id primary key, name text/varchar
Table users: id primary key, name text/varchar, org_id foreign key

Question We now want to query the database for a combined list of all user names of the organizations Google and Microsoft. How would such a query look like? You can use plain SQL or your preferred way of how you would make such a query.

This question aims at the applicant doing some form of Join. Since there are multiple ways to do a Join, there can be a lot of variations.

Note I would recommend asking this question even if the applicant was not able to model the two tables in the previous question. The applicant might not have modeled a database thyself, but they might be able to query against an existing database.

select u.name from users u, orgs o where users.org_id = orgs.id and o.name in ("Google", "Microsoft")

This question also aims at the usage of in. In case the applicant uses two equality comparisons, you might want to challenge them with a new requirement of a dynamic list orgs of organizations passed to the query.

Question We now realize that users are not necessarily in a single organization but can be in multiple. The database is running in production for thousands of users around the globe and downtime and disruptions should be avoided. How would you approach such a change on a high level step-by-step?

Junior applicants will very likely not be able to answer this question, you might want to leave it out even if they answered the previous question successfully. This question can help you though to assess the seniority of the applicant.

Add an additional table userorgs: user_id, org_id and primary_key(user_id, org_id)
Persist new org changes of users in both the new table and the old column
Migrate all existing org_ids from users in that table
Change select queries to use the new table instead of the org_id column
Remove the old org_id column

Conclusion 🔗

We have discussed the problems with existing interview processes and based on the goals of an interview outlined a flexible framework to assess an applicant while minimizing the time interviewees and interviewers need to spend on it. The framework consists of several blocks that can be dynamically adjusted before and during the interview to figure out the knowledge and the level of seniority of the applicant. I hope this guide helps you become a better interviewer and to evaluate more applicants in less time for everybody.

Scrum As Code

2021-12-16T00:00:00Z

Jira is bad. It's tough to use, exists in some space outside of your familiar development setup and only somehow integrates with your continuous integration and deployment process. Let's imagine a world where your Scrum process happens where you feel comfortable. In code.

In the past decade, more and more tooling has shifted to be defined as code. Tools like Terraform allow you to define infrastructure as code, other tools allow you to define documentation as code. This shift brings the advantage that there is a precise textual definition which can be version controlled, reviewed, tested, documented, automated and reproducible build. This increases the quality and decreases the likeliness for bugs since every change is reviewed and tested. It encourages a larger group of people to make changes since they can be sure that another person will look at their change. Let's leverage these advantages within product management.

Product Management Tool Requirements 🔗

There are multiple requirements development teams have for a product management tool.

It should show the things that are currently being done and the things that could be done in the near future.
It should offer a shared understanding of the current work and a starting point for discussion on future work.
The product manager and the engineers should be able to refine the work, adding descriptions, priority, status and acceptance criteria.
The actual progress in the code should be in sync with the status in the product management tool.
From the work in the product management tool it should be possible to find the related code changes. From the code changes it should be possible to find the related work in the product management tool.

From the mentioned requirements, the two last requirements are tough to solve outside of code. If the code and the product management work is positioned in two different places, they are inevitably out-of-sync. This can be addressed with relating code changes via identifiers (ticket numbers) to the product management tool. But it requires manual work of correctly relating them and can therefore be error-prone. It requires for each code change a counterpart to be manually created (You just want to fix this one-liner bug? Well, first create a ticket).

Also, is the status of the ticket in sync with the status of the pull request? You might be able to setup an automation to change the status on merge and deploy, but is the ticket actually closed with that pull request? Or will there be another pull request following after that? What if more details worth mentioning arose during implementation, shouldn't these be added first before closing the ticket?

Another complication arises with the product management tool and code being in two separate version control systems. Did the description of the ticket change after or before the change of the code? Where does this new requirement come from, was it already there during the planning meeting or with whom was it discussed afterwards?

All these things can be addressed with product management happening in the same place as coding.

Note Using code does not mean there can't be a graphical representation. It only means that the source-of-truth is a textual representation (code). There can still be graphical views derived from that.

The Idea 🔗

Representing Scrum as code requires us to find a textual file system representation of a product backlog.

The main entity that needs to be represented is the ticket (or story, backlog item). In Scrum, multiple tickets are arranged into a sprint. A ticket can only be part of one sprint. These properties allow us to represent tickets as files and sprints as directories. Creating a ticket is then the same as creating a file, moving a ticket into a sprint is moving a file from one directory to another and starting a sprint is creating a pull request moving and creating tickets into a directory.

- current-sprint
  - create-new-feature.md
  - fix-bug.md
  - fix-another-bug.md
- sprint-1
  - create-another-feature.md

Tickets are typically both loosely structured content (description, acceptance criteria, risks, notes) and structured content (priority, status, assignee). A suitable file representation of such tickets can therefore be Markdown with front-matter: Markdown allows to put arbitrary loosely structured content, the front-matter allows to define additional properties.

---
priority: 1
status: open
assignee: Alice
---

# Create a new feature

## Description

As a user, I want to have a new feature such I can use it.

- The new feature can be clicked.
- The new feature works on desktop and mobile.

## Out of Scope

- The new feature does not need to work on tablet yet.

The Graphical Representation 🔗

Structuring sprints and tickets in this way has another advantage: Static site generators can be used to render the sprints and tickets as websites. Not only can sprints be displayed with their tickets, also other views can be created that show selections of tickets by predefined criteria. Since it's purely a website, you can do any level of customization you need (yes, you could even rebuild Jira). Since it's defined in code, you can deploy that site to a CDN and have an immutable tailored graphical representation for you and your stakeholders with the right amount of detail you need. Editing still solely happens within code, the code remains the source of truth.

Other Advantages 🔗

With the Scrum process being defined as code, the typical as-code advantages arise.

Reviews: Changes to tickets and sprints are reviewed, ensuring multiple people have a look at a change and encouraging people to make changes that would otherwise be hesitant.
Tests: Each story in the current sprint needs to have acceptance criteria and story points? Validation for these properties can be setup and integrated into the CI checks.
Automation: Do you want to send out a reminder to review and/or close tickets that have not been started or touched for a few weeks? You can add a script and have it run as a scheduled job in a CI environment.

Conclusion 🔗

The Scrum process can be defined as code, leveraging several advantages of reviews, testing, automation and version control that an external third-party product management tool is lacking. Sprints can be represented as directories and tickets can be represented as files.

I'd appreciate your feedback on this idea. Would you define your Scrum tickets as code? What potential problems do you see?

How future languages will prevent Log4Shell

2021-12-22T00:00:00Z

The Log4Shell exploit makes conceptual problems in conventional programming languages apparent. Third-party code is able to execute arbitrary side-effects. Modern programming languages introduce concepts to avoid such vulnerabilities by design.

Log4Shell in a nutshell 🔗

The Log4Shell vulnerability boils down to a log statement doing more than just logging.

public class MyClass {
    private static final Logger LOG = LogManager.getLogger("MyClass");

    public static void main(String[] args) {
        LOG.info(args[0]);
    }
}

On execution of such a program with java MyProgram abc, you'll expect the program to print abc. And for the majority of inputs, it will do exactly that. However, the vulnerability in Log4J evaluated certain inputs in a way that untrusted Java code was downloaded and executed. There are a lot more details to Log4Shell which other posts excellently explain.

The conceptual issue 🔗

The underlying issue that led to this vulnerability already starts with the log statement LOG.info() being able to do something else than logging. In Java, every method void f() {} is able to execute any side-effect. It can read and write on the file system, it can log text on stdout, it can download resources from the internet and execute them. The language doesn't restrict a method in its side-effects and it does not give any indication what side-effect a method is executing.

The conceptual solution 🔗

There are some newer programming languages that address this conceptual issue. One of these languages is Koka. The fundamental idea is encoding the side-effects of each function in its type signature.

fun multiplyAndPrint(x: int, y: int): console int {
    result := x * y
    println(result)
    return result
}

The function multiplyAndPrint specifies that it returns an integer but also that it accesses the console. If the function does not specify that it access the console, then the compiler rejects the program with errors.

fun multiply(x: int, y: int): int {
    result := x * y
    println(result) // <-- Error
    return x*y
}

The predefined function println defines that it uses console but multiply does specify that it does not use console, hence the compiler rejects the program.

fun println(s: string): console () {
    ...
}

If a function specifies only the usage of console, then it also means it can not access the network, it can not download code and can not execute it.

Conclusion 🔗

The presented language concept can prevent third-party code from executing unwanted side-effects. While a logging library could still define a log function to download or execute code, this becomes transparent if expressed in the type signature and enables consumers to make a decision on if giving these permissions is a worthy trade-off.

Implementing a light/dark mode toggle with pure JS/CSS

2022-01-04T00:00:00Z

Statically generated websites are great, but for some things you still need some minimal JS running in the browser. Let's take a look at how to implement a minimal light/dark mode toggle in Eleventy.

What's the goal? 🔗

We want to have a button that toggles between light and dark mode on click. Initial visitors should see the site in dark mode. After a toggle, upcoming visits will remember the mode and show the site in that respective mode. The toggle button should represent the current mode and, on hover, indicate the toggle to the other mode.

In a nutshell, how do we get there? 🔗

We remember the mode with a local storage variable light-mode. If set with any value (we use "set"), then light mode is active. If the variable does not exist, then dark mode is active. On toggle button click, the variable in the local storage is created or removed. On each page load, the variable is read and if set, leads to a CSS class .light being set on the :root element. Some CSS rules lead to several CSS variables being populated with different values if :root.light is set. Also, the representation of the toggle button itself is changed via CSS rules if :root.light is set.

So what's the code? 🔗

The toggle and logic goes into a Nunjucks file theme-switch.njk. This file is included where it should appear on the page. In this case, it's the header of the site.

<style>
  {% include "_includes/theme-switch.css" %}
</style>

<button type="button" class="theme-icon">
  <i
    aria-hidden="true"
    class="fas fa-moon"
    title="Toggle between dark and light mode"
  ></i>
  <i
    aria-hidden="true"
    class="fas fa-sun"
    title="Toggle between dark and light mode"
  ></i>
  <span class="sr-only">Toggle between dark and light mode</span>
</button>

<script type="text/javascript">
  {% include "_includes/theme-switch.js" %}
</script>

The HTML structure contains the toggle button with a sun and a moon icon indicating light mode and dark mode and an alternative text for screen readers. We use Font Awesome icons, but you can also use own or different icons to eliminate this dependency.

Some CSS defines the two themes with CSS variables. The :root element refers to the html element on web pages.

:root:not(.light) {
  --background-color: #044965;
  --primary-color: #05668d;
  --font-color: white;
  --secondary-font-color: rgba(226, 232, 240, 1);
  --box-shadow: 0 4px 6px -1px rgba(0, 0, 0, 0.1), 0 2px 4px -1px rgba(0, 0, 0, 0.06);
}

:root.light {
  --background-color: #f8f9fa;
  --primary-color: #dee2e6;
  --font-color: #212529;
  --secondary-font-color: #495057;
}

The CSS styles the toggle button itself to have some fixed width and height, a bit of color, border and box-shadow as well as a Flexbox to center the visible icon vertically and horizontally. Also, it changes the visible icon depending on the selected theme. If the dark mode is active, then the moon should be shown. If the light mode is active, then the sun should be shown. We use the display property and the CSS selectors :root.light and :root:not(.light) to achieve that.

.theme-icon {
  width: 2.5rem;
  height: 2.5rem;
  box-shadow: var(--box-shadow);
  border-width: 1px;
  border-style: solid;
  border-radius: 0.5rem;
  border-color: rgba(226, 232, 240, 0);
  background-color: var(--primary-color);
  display: flex;
  justify-content: center;
  align-items: center;
  cursor: pointer;
  font-size: 1em;
}

:root:not(.light) .theme-icon .fa-sun {
  display: none;
}

:root:not(.light) .theme-icon .fa-moon {
  display: inline-block;
  color: white;
}

:root.light .theme-icon .fa-sun {
  display: inline-block;
}

:root.light .theme-icon .fa-moon {
  display: none;
}

.sr-only {
  height: 1px;
  width: 1px;
  overflow: hidden;
}

The JavaScript needs to achieve two things:

If the toggle button is clicked, then the theme should be toggled and the setting should be persisted into the local storage.
If the page is loaded, then the setting should be read from the local storage and the theme should be set.

We have two functions toggleRootClass and toggleLocalStorageItem to change the currently shown theme and to persist the setting into the local storage. On button click, both of these functions are called. On page load, only the root class is toggled since the setting doesn't change in this situation.

function isLight() {
  return localStorage.getItem("light-mode");
}

function toggleRootClass() {
  document.querySelector(":root").classList.toggle("light");
}

function toggleLocalStorageItem() {
  if (isLight()) {
    localStorage.removeItem("light-mode");
  } else {
    localStorage.setItem("light-mode", "set");
  }
}

if (isLight()) {
  toggleRootClass();
}

document.querySelector(".theme-icon").addEventListener("click", () => {
  toggleLocalStorageItem();
  toggleRootClass();
});

Conclusion 🔗

With some minimal CSS and JavaScript, it is possible to implement a lightweight dark and light mode toggle without introducing a big third-party dependency.

Identifying bad code with function usage patterns

2022-01-13T00:00:00Z

As software engineers, we aim to write code that is easy to understand and change. But it is not always obvious what kind of code achieves that. Let me explain how I look at function usage patterns to identify and refactor bad code.

Let me first introduce some vocabulary so that we are talking about the same things.

If I'm saying "a function is used in a place", then I'm talking about it being referred to in a specific static spot.

function f() {
  g();
}

The function g is used in the function f. If it appears in a for-loop, then it is still only "used in one place".

function f() {
  for (const i = 0; i < 2; i++) {
    g();
  }
}

It appears in two places, if its name appears multiple times in the code. It doesn't matter if its usage is in the same or another function. In the following example g is used two times.

function f() {
  g();
  g();
}

If a function is passed to another function as an argument or used in a lambda, then only its appearance where its passed is a usage. In the following example g is used one time.

function f() {
  [1, 2, 3].map((x) => g(x));
}

So I'm talking purely about static analysis, not the dynamic call patterns at runtime.

Hence, as an example, in the following code the function g is used three times.

function f() {
  g(); // 1
  if (false) {
    g(); // 2
  }
}

function h() {
  return [1].map(g); // 3
}

It's always about the program as a whole, independently from if the functions are distributed over multiple files.

Note For all the object-oriented folks out there, you can apply these function usages similarly to methods within objects instead of functions.

So much for the boring part, let's talk about how I use usage patterns to evaluate the quality of code.

1 to 1 usage 🔗

That's the default you want to have in your program. A function is used in exactly one place.

function createBooks(count) {
  for (const i = 0; i < count; i++) {
    createBook();
  }
}

There are multiple advantages of 1-to-1 usages.

If createBook is changed, then only createBooks might break or change in behavior.
A signature change to createBook only affects this one place in createBooks.
The input parameters and performance of createBook can be optimized for exactly this one usage createBooks.
If used exclusively, then your function usages form a tree which is easier to reason about than a graph.
Because the usage of createBook is so limited, you don't necessarily need to test it in isolation but can test createBooks instead.
You only need to look at the last item of a stack trace and know in which context the function is used.

However, in some cases, you don't want to limit yourself to 1 to 1 usages.

2-5 to 1 usage 🔗

This usage pattern refers to a function being used in a limited number of places. I choose the numbers 2-5 because above that number it can be overwhelming looking at all the usages of this function.

// On `POST /books`
function postBook(book) {
  synchronizeBook(book);
}

// Scheduled cron job
function refreshAllBooks(books) {
  for (const book of books) {
    synchronizeBook(book);
  }
}

function synchronizeBook(book) {
  // ...
}

A usage pattern like this can be very powerful, but also introduces risk since there are now multiple places that would break if synchronizeBook is faulty. Hence, if this pattern is used, there absolutely need to be tests for synchronizeBook.

Furthermore, this function needs to have a meaningful and short name. If its name is something like createOrUpdateBookIfAuthorized, then its naming can either be improved or the 2-5 to 1 usage should rather be higher or lower in the call chain. On the other hand, the name needs to be so specific such that people would not want to use it in additional places. In the example, there are only a finite number of places like putBook where someone would want to use synchronizeBook as well.

n to 1 usage 🔗

This usage pattern is super dangerous and software engineers love it. It is the pattern for functions exposed by a library. And who does not love to write a library?

// On `POST /books`
function postBook(book) {
  synchronizeEntity(book);
}

// On `POST /animal`
function postAnimal(animal) {
  synchronizeEntity(animal);
}

// Scheduled cron job
function refreshAllBooks(books) {
  for (const book of books) {
    synchronizeEntity(book);
  }
}

// Scheduled cron job
function refreshAllAnimals(animals) {
  for (const animal of animals) {
    synchronizeEntity(animal);
  }
}

function synchronizeEntity(entity) {
  // ...
}

In this example, the function synchronizeEntity is used n to 1.

My take on this: Use this usage pattern very, very rarely within a software product developed in a single team.

There are multiple disadvantages of n to 1 usages.

If synchronizeEntity is changed, then god beware what might break.
A signature change to synchronizeEntity affects your whole program.
For individual usages, the performance of synchronizeEntity is close to impossible to optimize without risking bugs in other usages.
Because synchronizeEntity is so generic, you will have trouble to test it properly. You might achieve 100% test coverage for that function and it's still not properly tested for actual usages.

This pattern should mostly be reserved for libraries with clear and well-designed APIs. The library needs to be at least maintained by a different team, better by a different company or as an open-source project.

However, in some cases this pattern can be fine to use. Consider an API with endpoints which all check for authorization.

function postBook(user, book) {
  if (!isAdmin(user)) {
    throw new UnauthorizedException();
  }
  // ...
}

function postAnimal(user, animal) {
  if (!isAdmin(user)) {
    throw new UnauthorizedException();
  }
  // ...
}

In this example, isAdmin is used n to 1. However, it is always used in the same way. Some endpoint uses it and uses the return value to leave the function early with an exception. The implementation of isAdmin can be rather simple compared to a function synchronizeEntity and it can be expected to stay stable.

Some middleware frameworks (like Express) also allow you to centralize these usages.

app.post("/books", onlyAdmins, postBook);
app.post("/animals", onlyAdmins, postAnimal);

This way, you don't have the function usages of onlyAdmins spread across your whole program if you gather the endpoint definitions in a single app file.

1 to n usage 🔗

This usage pattern is in the vast majority of cases bad but luckily also the easiest to spot. It is a single function that uses many more other functions and has a tendency to grow over time. It sometimes comes in combination with an n to 1 usage as a 1 to n to 1 usage as the generality of the function is often the problem that leads to it being overly huge.

function synchronizeEntity(
  user,
  config,
  db,
  http,
  /* ..., */ entityId,
  entityType
) {
  if (config.requireAdmin && !isAdmin(user)) {
    throw new UnauthorizedException();
  }
  const entity = db.read(entityType, entityId);
  let mappedEntity;
  if (entityType === "book") {
    mappedEntity = mapBook(entity);
  } else if (entityType === "animal") {
    mappedEntity = mapAnimal(entity);
  } else {
    throw new InvalidArgumentException();
  }
  let httpPath;
  if (entityType === "book") {
    httpPath = "/books";
  } else if (entityType === "animal") {
    httpPath = "/animals";
  } else {
    throw new InvalidArgumentException();
  }
  http.post(httpPath, mappedEntity);
}

Such a function will continue to grow. With more entity types being added, more function usages like mapBook and mapAnimal will come up. In this case, this problem can be resolved with introducing separate functions synchronizeBook and synchronizeAnimal. Such functions would not continue to grow unexpectedly and are rather 2-5 to 1 to 2-5 usages.

function synchronizeBook(user, db, http, bookId) {
  if (!isAdmin(user)) {
    throw new UnauthorizedException();
  }
  const book = db.read("book", entityId);
  const mappedBook = mapBook(book);
  http.post("/books", mappedBook);
}

Conclusion 🔗

If you make yourself conscious about the usage patterns of the functions within your program, you can use this knowledge to identify code smells and a direction on how to get rid of them. While 1 to 1 usages should be your default, 2-5 to 1 usages can be powerful if used consciously with care, n to 1 usages should be rare or reserved for libraries maintained outside of the team and 1 to n usages are to be avoided.

Building a 1000 line Java program in 6 lines of Haskell

2022-01-15T00:00:00Z

Some languages are very expressive and some languages are Java. Let's disassemble an overly expressive six-line algorithmical Haskell program. I'll walk you through all the tricks, weirdness and beauty.

Note I'm not assuming any Haskell knowledge, but some familiarity with another programming language is of advantage.

The six-line program that we are disassembling solves a coding interview problem called the knights tournament. Let me give you a quick overview of it.

The Knights Tournament 🔗

There is an arbitrary but fixed number of knights n. These knights are healthy and strong, the healthier they are the more health points they have, the stronger they are the more damage points they deal. But these knights have an unlucky fate: They are battling each other until only one knight is left. The rules are strict. They stand in a circle and can only hit the knight to their right. The order of hits is well defined: After a knight was hit, they are allowed to make the next hit. Well, if they are still standing. If that is not the case, then the knight to the right that would have been hit can take their turn to hit. This continues as long as there is only one knight left. There can only be one winner.

The Approach 🔗

Before we dig into the code, let's conceptualize a solution. Haskell works with immutable data structures, so typically the solution concepts already look way different from their mutable counterparts. Trying to make a mutable solution concept work in a functional programming language is madness and can quickly lead to frustration.

The main entity we are dealing with is a knight. A knight has a name, health points and damage points. We express the circular arena of knights as an immutable list of these knights.

The tournament is divided into turns when a knight hits another knight. We define a turn as a function that takes the arena of knights as an input and returns a new arena of knights. That new arena describes the state of the arena after the hit of the turn was done. That is, it reduces the health of the hit knight and might even remove the knight if its health below zero.

How do we express which turn it is? With a turn being done, we move the first knight to the end of the arena list. So it's always the first knight's turn. We kind of use the list as a queue, always taking the first knight and moving them to the back of the queue. This continues as long as the size of the queue becomes 1.

The Knight data type 🔗

In Java, we would go ahead and define a class Knight with fields, a constructor and possibly getters and setters and a toString and would already be way beyond six lines. In Haskell it is one line.

data Knight = Knight String Int Int deriving (Show, Read)

We define the data type Knight with three fields of type String (the name), Int (the damage points) and Int (the health points). The fields are identified by their position rather than a name. There are also data types that allow to identify fields by a name, but identifying by their position has advantages later on.

The keyword data expresses that we want to define such a data type. The first Knight defines the name of the type. The second Knight defines the constructor to create an instance of the type. This constructor accepts the three fields and can be used with Knight "James" 3 12. This constructs a knight James with twelve health points and three damage points.

Note In Haskell, you don't have to use brackets () for function calls. You can imagine it to be like bash where you also write git status instead of git(status) or git("status").

The keyword deriving instructs Haskell to automatically provide us with some functions for this data type. The type class Show provides us with a show function that transforms a knight into a string representation. Haskell gives us the default implementation inferred from the constructor name and field names which is Knight "James" 3 12. The type class Read provides us with the counterpart to show. It defines a function read that takes a string representation of a knight and constructs a knight from it. These two methods allow us to interact with the input of the user from the command-line.

Taking a hit 🔗

We're reaching line 2 and 3. This is a small helper function takeHit to express what it means for a knight to take a hit. It takes two arguments: A knight and the damage points dealt to that knight. It returns a single element list of that knight with that damage dealt to it. Or an empty list, well, if the knight decided to drop out of the tournament.

takeHit (Knight name damage health) points =
  map (Knight name damage) $ filter (>0) [health-points]

The first line declares the signature of the function takeHit accepting the knight and the points. We use a technique called destructuring for the knight. We could write takeHit knight points and would have the knight available as a variable knight in the body of the function. But we are actually only interested in its fields. So we destructure the knight into its fields such that we can refer to name, damage and health directly.

Note Haskell is a statically typed language with powerful type inference. The function might look to you dynamically typed, but Haskell is actually able to figure out that Knight name damage health is of type Knight and points is of type Int and that takeHit returns a list of knights [Knight]. Nevertheless, it can be beneficial for better error messages and readability to add the optional type signature takeHit :: Knight -> Int -> [Knight] manually.

The second line is the function body. There are multiple things going on there. First, writing f $ g x is syntactic sugar for f (g x). It avoids having to count closing brackets if there is a longer call chain f (g (h (k (x)))) where you can instead write f $ g $ h $ k x.

We define a list [health-points]. It is a single element list that would be [7] for a knight with 12 health points and 5 damage points dealt to it. The resulting health can be negative, in that case, we want to have an empty list instead. To achieve that, we filter the list with a function >0. This expression >0 uses a Haskell language feature called partial application. It is equivalent to a lambda \x -> x>0. By leaving out the left part of the comparison, Haskell knows that >0 is not true or false, but rather a function where the left part of the comparison still needs to be passed before it resolves into a true or false.

So with filter (>0) [health-points], we either have a single element list of positive health points or an empty list. We now want to construct a new knight from it (or rather the same knight with different health points). To achieve that, we map the list. Knight name damage is again a usage of partial application: The health points still need to be provided. Hence, Knight name damage is a function accepting health points and returning a knight with that name, damage and health points.

The result of the full body is therefore either a single element list of a knight with the same name and damage points and reduced health points or and empty list.

The turn 🔗

We now want to define the function that takes the arena of knights, performs a single knight's turn on it and returns the updated arena.

turn (knight@(Knight _ damage _) : knight' : knights) =
  takeHit knight' damage ++ knights ++ [knight]

We use a slightly more ambitious form of destructuring in the signature of turn. The whole expression (knight@(Knight _ damage _) : knight' : knights) is a list of knights [Knight]. With knight : knight' : knights, we are telling Haskell to allow us to refer to the first knight of the list as knight, to the second knight of the list as knight' and to the other knights as knights. With knight@(Knight _ damage _), we are saying that we are not interested in the name and health since we use the placeholder _. But we want to be able to refer to the knight as a whole with knight. That is achieved with the syntax knight@(...).

The resulting arena consists of three parts that are added together with the list concatenation function ++. The first part takeHit knight' damage is the second knight that is taking the hit of the first knight and should afterwards be the new first knight since its their turn afterwards. Remember that the return value of takeHit is either a single element list of the knight or an empty list. In case it is an empty list, it will be the turn of the first knight of the other knights afterwards (the third knight). The second part knights are all the other knights. The third part [knight] is the previously first knight that now is at the end.

The main loop 🔗

We want to read a list of knights [Knight "Bob" 3 12, Knight "Alice" 4 24, Knight "Jean" 3 16] from the command-line and print the winning knight Knight "Alice" 4 9 at the end after all turns have been done and only one knight is still standing.

main = getLine >>= putStrLn . show . head . until ((1==).length) turn . read

The function main has a dedicated role in Haskell like in Java. It is the entry point of the program. The expression getLine tells Haskell to read a single line from the command-line as a String. This line, we pass as argument to function putStrLn . show . head . until ((1==).length) turn . read.

Two things:

The dot . is a composition function. An expression f (g x) is equivalent to (f . g) x. So if we write f . g, it is equivalent to the longer version \x -> f (g x).
The >>= function is like the dot ., but works for the dedicated type IO String that getLine returns because it works with the outside world (the command line).

Let's disassemble putStrLn . show . head . until ((1==).length) turn . read. Such an expression, you read from the right to the left. We have our line of type String and pass it to the function read. The function read tries to parse a list of knights [Knight] from that String.

Note How does the generic function read knows that it should parse a list of knights? Because the following expression until ((1==).length) turn requires its argument to be a list of knights and hence requires the return type of read to be a list of knights. If following expressions would not make the type clear, then Haskell would compile with an error about unresolved ambiguity.

The expression until ((1==).length) turn performs turns on the list of knights until it is a single element list, that is there is a winner. The expression 1== is again a usage of partial application and (1==).length is a function that first gets the length of the list of knights and then checks if it is equal to one or not.

Once we have a list of a single knight, we take that one knight out of the list with head. Then we transform them into a string representation with show and write them to the console with putStrLn.

If we execute it, it looks like this.

$ ghci knights.hs
*Main> main
[Knight "Bob" 3 12, Knight "Alice" 4 24, Knight "Jean" 3 16]
Knight "Alice" 4 9

All together 🔗

The full six line program looks like this.

data Knight = Knight String Int Int deriving (Show, Read)

takeHit (Knight name damage health) points =
  map (Knight name damage) $ filter (>0) [health-points]

turn (knight@(Knight _ damage _) : knight' : knights) =
  takeHit knight' damage ++ knights ++ [knight]

main = getLine >>= putStrLn . show . head . until ((1==).length) turn . read

Conclusion 🔗

We looked at a Haskell six-line program implementing the tournament of knights. We saw how to define a data type and how to get Haskell to provide show and read functions. We learned about destructuring, partial application and function composition. While this program took conciseness to an extreme, I hope I could show you how various functional concepts can be powerful tools to build an expressive application.

Extended version 🔗

A slightly longer version that I would prefer for cases where readability and maintainability are a more important concern than conciseness would look like this. It includes function signatures, input validation and uses the Maybe type to represent a knight that might be alive or gone.

module KnightTournament ( Knight, game ) where

import Data.Foldable
import Data.Either
import Data.Maybe

data Knight = Knight
  String -- Name
  Int -- Health
  Int -- Damage
  deriving Show

type ValidationError = String

validateKnight :: Knight -> [ValidationError]
validateKnight (Knight name health damage) =
  map snd $ filter fst validations
  where validations = [
          (health <= 0, "Knight " ++ name ++ " already dead!"),
          (damage < 0, "Knight " ++ name ++ " can not deal negative damage!")]

validateArena :: Arena -> [ValidationError]
validateArena = concat.(map validateKnight)

-- The arena is a queue of knights
type Arena = [Knight]

-- The knight takes a hit and maybe leaves the tournament
takeHit :: Int -> Knight -> Maybe Knight
takeHit points (Knight name health damage)
  | points < health = Just $ Knight name (health - points) damage
  | otherwise = Nothing

-- The first knight hits the second one and will then be attached to the end.
-- If the health of the second knight is below zero, he is dead and will be removed.
-- If not, he is the next knight who gets the chance to hit.
turn :: Arena -> Arena
turn (knight@(Knight _ _ damage) : knight' : knights) =
  toList (takeHit damage knight') ++ knights ++ [knight]

game :: Arena -> Either [ValidationError] Knight
game arena = case validateArena arena of
  [] -> Right $ head $ until ((1==).length) turn arena
  errors -> Left errors

Working as a Senior Software Engineer

2022-01-26T00:00:00Z

Early in the career of a software engineer, a lot is about developing technical skills and contributing code towards features. With more seniority, the expectations shift. A post on working as a Senior Software Engineer.

As a junior or mid-level software engineer, your work is mostly about individual contribution. You participate in the sprint meetings, pick a ticket or get one assigned, come up with a solution, discuss it with your peers and implement it. You might involve yourself in architecture discussions, review your peers pull requests or fix a reported bug in production. The most of your work is focused though on contributing code towards features. During that you refine your skills as a software engineer. You gain experience on which code is good and which is bad. You learn to navigate within your codebase. You learn about various technologies, libraries, frameworks and languages.

As a Senior Software Engineer, this work as an individual contributor shifts towards work as an accelerator of a whole technical team. It's not anymore about what you are able to contribute as an individual towards the success of the team. Instead, you set a basis to make your team members successful. You have proven yourself to be able to write high-quality code autonomously. Now it's time to elevate your team members to achieve that too.

There are multiple roles that aim for that. One is the role of the engineering manager. These are often targeting people and processes. They guide their team members in their career and manage expectations. They figure out structural problems within and surrounding the team that prevent it from succeeding on its mission and they communicate internally and externally to resolve these structural problems.

The role of a Senior Software Engineer is instead targeting the technical and product basis. As a Senior Software Engineer, you need to ensure that the technical quality of your products does not block the further development of it. You need to find ways to implement new product requirements and discuss and adjust them to make sure they provide value and don't reduce the technical quality.

As a Junior Software Engineer, most parts of your work are within the team. As a Senior Software Engineer, you are taking on additional internal responsibilities as well as new external responsibilities.

Team-internal responsibilities 🔗

Leading and defining epics: Your work shifts away from completing predefined work items and towards taking vague problem statements and defining the required work items to solve that problem. While you iterate on the solution, you ensure that the set of work items still works towards solving the problem and you define additional work items and refine existing work items based on the insights you gain during implementation.
Design architectures: As part of leading epics, you need to come up with suitable architectures to support the long-term success of the solution you are building. This also includes iterating on an existing architecture and anticipating required architectural changes due to changes in the nature of the problem or how the product is being used.
Ensure value delivery: You need to ensure that the implementations of your team actually provide value to your users. Sometimes there might be small obstacles that prevent an implementation to be usable end-to-end.
Guide other engineers: During pair-programming and PR reviews, a Senior Engineer can guide other engineers towards a common coding style and agile practices. That is not acting as a police, but as a guide for the other engineers aligning them towards a common technical direction.
Take on the hard, high-impact work: There is always work, that engineers are hesitant to do. As a Senior Engineer, there is no more hiding. If you don't take it on, there is no Super Senior Engineer that will take it over for you.
Anticipate and resolve blockers: Similarly to hard, high-impact work, there are blockers for other engineers or the whole team that can come up. As a Senior Engineer, you resolve or delegate these blockers making sure they are addressed. Ideally, you anticipate them before they actually block other engineers.

Team-external responsibilities 🔗

Present learnings: Your team is doing great work. Other teams are interested in hearing about the technical solutions you applied to solve problems. They might encounter similar problems and will want to reach out to you to adopt a similar solution.
Adopt learnings: Other teams are doing great work as well. You want to inspect their work, reach out to them, listen to their presentations to reevaluate the approaches you took and are about to take.
Push back on hard, low-impact work: Sometimes stakeholders outside of the team don't realize how complicated a small feature request is. It's now your job to push back on this kind of work such that your team can focus on high-impact work and easy low-impact work.
Communicate problems: If outside expectations can't be met, then it is now your job to communicate that. That needs to happen as early as possible and always with an alternative plan in mind. Pushing the timeline, reducing scope, reconsidering technical approaches. It's important to offer a solution when bringing up problems.

Conclusion 🔗

As a Senior Software Engineer, your work shifts away from individual contribution towards working as an accelerator of your team. You take over different team-internal work and new team-external work. Much of your focus moves from work items to epics, from code to architecture and from coding to communication.

2031 - A Software Engineering Utopia

2022-01-31T00:00:00Z

It's 2031. Software engineering is on a new level. Open APIs and SaaS apps, funded open-source and an open cloud platform accelerate teams. We join Naomi, Javier and Kalina developing a new product in this environment.

It's July, 16th 2031, 8:00am.

Naomi is getting ready for the day. She will start today engineering a new project in the area of accessibility. Ten years ago it would have been tough to get funding for such a project. There is not much money in that market and projects used to focus on where the money is. But ever since the "Common Open Source Funding Foundation" (COSFF) was founded, this has changed. Almost every big company is contributing funds towards this foundation now. In exchange they get partial access to its unique talent pool, besides getting naturally free access to the developed open source.

Naomi is employed by the COSFF. But she is free to choose from the projects of the sponsoring companies. Also, from all the social, ecological and open-source initiatives that are participating in the COSFF as well. Since the sponsoring companies are providing the funding, she is obligated to spend 40% of her time on the projects of the those companies. 20% of the time, she is free to use. Some people use it for blogging, others to educate themselves, others use it as additional time to recover or to spend additional time with their families. The remaining 40% of her time Naomi can use for the social, ecological and open-source initiatives that she wants to advance. After working a year at an initiative to decentralize power supply and storage, she has now decided to move to this new project for accessibility.

Naomi logs in online to talk to her team. They are a newly formed team of three people that will bootstrap an initial prototype. Naomi, as a software engineer, is focusing more towards the engineering aspects. Kalina is the manager and representation to users and Javier is the designer. Also Kalina and Javier are employed by the COSFF and the three of them have worked on projects together before.

Kalina: So, I got the budget setup in the budget planner. They use the open cloud platform here internally for development, which makes sense: Our project is acknowledged as a social initiative, so we can use it for free. Outside the company, the application is of course expected to run on all cloud platforms, so let's adhere to the open standard for cloud platforms. Then we have all platforms covered.
Naomi: Hehe, all platforms...
Kalina rolls her eyes: Also Amazon will finally get to it. They have realized that they won't keep their competitive advantage if they reject the standard. They are working towards it. If we develop against the open standard, then Microsoft Azure, Google Cloud, Apple Cloud and the open cloud platform will all already work. Well, and AWS eventually.

One of the bigger open source projects that came to live through the COSFF is the "Open Cloud Platform" (OCP). Ten years ago there were two main players on the cloud platform market: Microsoft with Azure and Amazon with AWS. Other companies like Google were trying to push into the market, but weren't really able to establish themselves. That changed in 2025, when Google came together with smaller cloud providers to create an open standard for cloud platforms. They founded a consortium and defined what a cloud platform is and what functionality it needs to offer in which way. They standardized how infrastructure is defined as code, how personal identifiable information (PII) needs to be stored and how users can access it, how the APIs to access application metrics look and how meta data on the used libraries and third-party software services is accessed. This gave Google and Oracle a competitive advantage. Because of the open APIs, third-party companies were able to build helpful products on top: Some to gain insights from application metrics, some to analyze GDPR aspects, some to find performance bottlenecks. These companies were now able to build products for cloud platforms that adhere to the open standard. Before, they needed to integrate with each cloud platform on its own.

Kalina: Do you see the architecture as similar to one of your previous projects?
Naomi: You mean: Can I just copy some files from an existing project and make some magic happen?
Javier: Well, you can, am I guessing right?
Naomi: It's actually pretty much a standard single service setup what we need right now. I couldn't do a setup yet since the budget wasn't approved yet. But I can use my standard template repository and bootstrap it from there.
Kalina: What's included with that?
Naomi: It's a CDN-deployed frontend based on an SSG, a dynamically scaled microservice with a document-based database. User management including GDPR-compliant PII management, application and user metrics collection with some customized dashboards and an application for DB monitoring.

Javier looks at her confused.

Naomi: It's a web page. We'll use an existing user management system and some systems to get insights into our app. I would suggest to add the 'SaaS hosting dashboard' and the 'SaaS cost dashboard' directly. The 'SaaS payment processor' we can add later on, doesn't really add a lot of value until we have first users where their payments in various currencies need to be processed.
Kalina: The 'SaaS hosting dashboard' and 'SaaS cost dashboard' are already globally setup for the company.
Naomi: Ok, great, then I exclude the files defining the overall setup and only include the two files that setup the two team-specific dashboards. Then we'll have an overview of our own used software and the cost associated with it.

Of course, not all functionality provided by cloud platforms is part of the open standard. The open standard only defines the minimal requirements on specific aspects that a cloud platform needs to adhere to. Different cloud providers continue to develop unique selling points that give them a competitive advantage. For example, monetarization was first developed by AWS with the 'AWS Payment Processor'. Soon, other cloud providers saw the value in it and developed alternatives. After two years, Google, as a key driver of the open standard, pushed to make monetarization one aspect of the open standard. Since then, new products like the "SaaS cost dashboard", the "SaaS budget planner" and the "SaaS payment processor" have been created on top of that standard. All these products work by default on every cloud platform. There is even a product to automatically create a tax report and send the according telefax message to the responsible government agency.

Javier: Can we also add this new cool component design tool? It's supposed to be really great since I can describe the components vocally and get both an initial visual design and a textual description that I can version with together with your code.
Naomi: Sure, just added it. Should be provisioned on the cloud platform in a minute.
Kalina: Wait, what does it cost?
Naomi: It's payment-by-usage and you should see it on your cost dashboard now. It's, like, 15 bucks a month for the three of us, can this be correct?
Javier: It's open source. The majority of the funding comes from the COSFF and since we're acknowledged as a social initiative, it's even less. We only need to pay the one third open source maintenance part, the two thirds for hosting on the open cloud platform isn't required to be payed by us.
Naomi: Ah, yes, alright, it even says here in the dashboard. Would be 60 bucks if we wouldn't be acknowledged as a social initiative.
Kalina: I got a warning here that it doesn't manage personal identifiable information compliant with GDPR. 'Do you use this application as a customer-facing application?', it says.
Naomi: Sorry, forgot to specify this in the setup code. You can click 'no' and it will commit the change back and the warning should be gone. Few other things... I'll add 'sgit' support on GitHub. This will track all our structured data semantically, which makes reviewing stuff way easier, since we'll see semantic diffs instead of changes line-by-line. Even outside the application code, it should help you with the work item definitions in the product management tool.
Kalina: Apropos, product management tool...
Naomi interrupts: Oh no...
Kalina ignores her and goes on: I know you like to work with your fancy text files but I want to work in a nice web UI.
Naomi: That's supported with 'sgit' now. Since the work items are stored and versioned semantically, it can be displayed in whatever way the viewer likes to see it. If I look at the work items in my code editor, it is really just a view on top of the semantic data. If you look at it in the browser, it is a different view. Hell yeah, I think that there is even a retro version of Jira that works as a view of it, if you really want to shoot yourself in the foot.
Kalina: Oh... thanks, I guess? I'd rather go with the lightweight PM-View instead. And the integration with the 'Open Objectives' still works?
Naomi: Let me check... Yes, 'Open Objectives' doesn't care where the work items are persisted as long as they adhere to the format of the 'Open Work Item Standard'. Do you want me to include the setup of 'Open Objectives' such that it gets provisioned?
Kalina: Yes, please. It will be important to keep users informed, especially since we receive public funding. It will also allow us to get easier into contact with people with similar objectives.
Javier: So, what are our objectives?
Kalina: Let's take a break first and clarify this later. We made good progress setting up all the tooling and can go on defining our vision, objectives and how we measure that we are accomplishing them.

Sounds good. See you then!

About the Node.js series

2022-02-07T00:00:00Z

In a series of blog posts, I'm going to describe an opinionated Node.js microservice architecture. Let me tell you about the why, what and how of this series.

For the development of a microservice, there are plenty of frameworks and languages to choose from. Many of these options like Java with Spring Boot and Node.js with Nest.js come with a pre-defined architecture and a set of libraries. This allows for a quick start bootstrapping a service. There is one clear and defined way on how to structure the service. There is documentation of a solution for many of the common problems.

However, after a quick start, such frameworks can become a burden. Configuration- and annotation-heavy as they are, they take away choices you could express in code in exchange for hiding complexity. Anything away from the common path like adding your own custom authentication mechanism or introducing a different logging mechanism like JSON-logging requires more work and more complexity with the framework and might even not be possible at all.

For that reason, I avoid frameworks.

Instead, I'm defining an own loose architecture and choose and connect own library choices. It allows me to express solutions to problems close to their inherent complexity. It gives me the ability to exchange each library and each functionality if there is a need to.

However, this approach gives less guidance on how to structure an application. With multiple developers working on it, consistency can get lost if a desired structure is not defined. Also, there is no advantage of a quick start since you need to start for each service from scratch. That's where this opinionated architecture guide comes in.

In a series of blog posts, I'm describing an opinionated Node.js microservice architecture that my team successfully used for three microservices. Over the series of posts, we investigate a dedicated example microservice that I assemble out of the experiences with the three microservices. We talk about various aspects like organizing controllers, configuration management, database connection and querying, API testing, authentication, CI and CD workflows, Kubernetes runtime and more.

From the presented architecture example, you will be able to bootstrap your own production-ready microservice. You will be able to exchange individual libraries to accommodate for your own requirements. You will be able to choose a different architectural structure where you need it. You will be able to harness the full power of a Turing-complete programming language, not being limited by pre-determined choices of a framework.

The high-level architecture 🔗

On a high-level, this architecture consists of a Node.js application that is hosted as a Kubernetes service with replication.

It uses a PostgreSQL database for persistence and both normalized tables for structured data as well as JSON-columns for unstructured data. Locally, the database runs in docker. On production, the application connects with a managed database.

The service offers a REST-like HTTP API. The endpoints are secured with JWT authentication. They are described and validated with an OpenAPI specification.

The repository is version controlled in git and hosted on GitHub. Continuous integration and deployment are set up with GitHub Actions.

For testing, we focus heavily on unit tests for database queries, middleware, API endpoints and complicated logic.

The choice of libraries 🔗

In a framework-less approach, there are plenty of library choices to make. For the microservice that we are building, we use a fixed set of libraries.

Wait, isn't this guide then a framework?

Yes, if one is to copy the bare skeleton without understanding the copied code and uses it to build a service, then what you are reading is a framework. However, together with the explanations in these blog posts, you are able to adjust any parts to your liking. Modifying, exchanging or removing any functionality or library. Adding or removing any structure or layers. This guide is a composition of functionality. In this regard, it is different to a framework. There is no default behavior, therefore no overriding of default behavior and no hidden complexity.

In this fixed set of libraries, there are two libraries that are hard to exchange.

Node.js: Node.js as the runtime environment is not exchangeable. If you would exchange this library for Deno (for example), this would impact the choice of each other library. Hence, in this opinionated architecture, Node.js can not be exchanged.
Express.js: While it's not impossible to exchange Express.js in this architecture, much of the middleware is build for Express.js. It's not as heavy as Node.js, but exchanging Express.js requires significantly more effort than exchanging the other libraries.

Other libraries can be exchanged in a less expensive way but are used across various parts of the architecture.

axios: This library allows us making HTTP requests.
knex: With an SQL database as the persistence layer, we use Knex to have fluent schema migrations and build SQL queries. Knex is not an ORM. That makes it more lightweight than its ORM alternatives.
pino: Logging with JSON output.
jest: Environment for test execution.
lodash/fp: Some utility functions, especially for collections.

Wait, what are we using for dependency injection?

Nothing. We are ruthlessly calling functions from other modules directly and are mocking these modules in tests.

Apart from these libraries, there are libraries that solve specific problems without being referenced all over the application. Therefore, they can be easily exchanged.

prettier: Enforce consistent code styling.
eslint: Find bugs earlier.
express-openapi-validator: This library allows us to validate incoming requests against an OpenAPI specification.
jsonwebtoken: Decoding and verifying a JWT token.
testcontainers: Running a PostgreSQL database in test cases.
supertest: Testing of HTTP endpoints.
memoizee: Memoize (cache) return values of functions.

Beside these libraries there are more smaller libraries that provide glue code to connect different libraries or that are plugged in for aspects like compression or CORS.

What we are about to cover 🔗

The topics are not yet set in stone but we will roughly cover the following overarching topics and subtopics in that order. Some topics might end up in the same blog posts, some blog posts might cover several topic and some topics might be covered much earlier or later.

Building an API 🔗

We build a REST-like API, organize and test the endpoints and implement various middleware.

Working with a database 🔗

Without a persistence technology, the API can't store data over the lifetime of the application. We setup a database, build and test queries for it and develop the database schema over time.

Sending HTTP requests 🔗

For many applications, it is crucial to be able to talk to other services. We setup Axios and make authenticated calls to services.

Scheduled jobs 🔗

Some tasks either can not be executed synchronously with a request (if some effects are happen at specific times like delay notifications). Other tasks should not be executed synchronously because they are time-intensive. We architecturally setup a batch job that can run on a schedule.

Application subcommands
Asynchronous batch jobs
Idempotent cron jobs with at-least-once delivery
Knex database pagination

Working in a team 🔗

Great products are developed within a team. The team members might have different machines and environments, they require tooling to get ready quickly and to reset a broken state. Continuous integration and deployment get changes into production immediately with a pull request.

Developer configurations
Continuous integration and deployment
Evolving architecture over time

Advanced topics 🔗

Here we will cover various advanced topics that might not be necessary for each microservice.

Running it in Kubernetes
Feature Flags
Paging
Rate-limiting
Scheduled and batch jobs
Incoming asynchronous communication

What we are building 🔗

Across these blog posts, we are looking at different aspects of the same application. Since these series is more about how to technically build such an application rather than tackling a compelling problem, we are building an application for a well-known problem.

Todos.

Because applications for todos are widely known, we can focus on the technical aspects. Let me describe the requirements of such a todo application.

A todo is an entity with a short name, an assignee, a due date and an ID. The application should have a REST endpoint to manage todos.

POST /todos: Creates a todo
PUT /todos/:id: Changes the todo with the ID
DELETE /todos/:id: Deletes the todo with the ID
GET /todos: Returns a list of all todos
GET /todos/:id: Returns the todo with the ID

There is an additional endpoint that can only be called from users with admin tokens to prevent accidental deletions.

DELETE /todos: Deletes all todos

A reasonable time after a todo is delayed, the assignee should be notified. There is a pre-existing notification service, that offers an endpoint POST /notification to do that.

That's it. Nothing fancy, but it covers the different technical aspects.

Way forward 🔗

At the time of this introduction being written, the implementation of the application itself and the related posts are still mostly on the horizon. This means that it's possible to incorporate feedback. If you are having any input on additional topics that you think should be covered or would like to see some topics covered earlier, let me know.

Organizing Express.js endpoints

2022-02-09T00:00:00Z

We start the Node.js Microservice Architecture with the organization of endpoints. Learn how to organize the files and folders, why a REST resource should have its own DAO and why two layers are sufficient.

When developing a service, you want to start with the user-facing code. This way, you are fast to get something running. For a backend service, the user-facing code is the API. In our example, we provide a REST-like HTTP API. We want to provide five endpoints.

POST /todos: Creates a todo
PUT /todos/:id: Changes the todo with the ID
DELETE /todos/:id: Deletes the todo with the ID
GET /todos: Returns a list of all todos
GET /todos/:id: Returns the todo with the ID

Note how these endpoints are all related.

They define a resource /todos.
They work with the same entity type of a Todo.
They act against the same storage.

If any of these aspects resource name, entity type or storage changes, then it affects all endpoints. Things that change together should be close to each other. Following that guideline, changes are rather local to folders and not distributed across the system. This makes it easier to examine what needs to be changed. And it makes it easier to review the code.

Remember: Things that change together should be close to each other.

File System Organization 🔗

With this guideline, the organization of the endpoints within this architecture looks like this in the src folder.

src
|-- main.ts
|-- app.ts
`-- controllers
    `-- todos
        |-- todo.router.ts
        |-- todo.type.ts
        |-- todo.dao.ts
        |-- delete-todo.controller.ts
        |-- get-todo.controller.ts
        |-- get-todos.controller.ts
        |-- post-todo.controller.ts
        `-- put-todo.controller.ts

The folder controllers/todos is a self-contained module. It does not reference other parts of the application and its single entrypoint is the router todo.router.ts. In this regard, the self-contained module controllers/todos follows the advice of deep modules by Ousterhout [^1]: A module should have a simple interface (in this case an object of type Router) and should encapsulate complexity.

Todo Router 🔗

From the outside, app.ts refers to todo.router.ts to mount the /todos resource into the application.

import express from "express";
import { todosRoute } from "./controllers/todos/todo.router";

const app = express();

// Enable JSON body parsing
app.use(express.json({ limit: "1mb" }));

// Mount the `/todos` resource
app.use("/todos", todosRoute);

export default app;

The todo router itself refers to the different controllers.

import express from "express";
import { deleteTodoController } from "./delete-todo.controller";
import { getTodoController } from "./get-todo.controller";
import { getTodosController } from "./get-todos.controller";
import { postTodoController } from "./post-todo.controller";
import { putTodoController } from "./put-todo.controller";

export const todosRoute = express.Router();

todosRoute.get("/:id", getTodoController);
todosRoute.get("", getTodosController);
todosRoute.put("/:id", putTodoController);
todosRoute.delete("/:id", deleteTodoController);
todosRoute.post("", postTodoController);

Todo Type 🔗

The type todo.type.ts defines the todo type including the ID.

export type TodoId = string;

export interface Todo {
  id: TodoId;
  name: string;
  assignee: string;
  dueDate: string;
}

If a todo is sent as a body to the endpoint POST /todos, then the ID field is expected to be missing. In these cases, it's possible to refer to the todo with the type Omit<"id", Todo>.

The same type for the representation in the API and in the database? Doesn't that expose implementation details? Nope. There is no need to have a TypeScript type to map to a different structure externally in the API or internally in the database. In later chapters we will see how we can define a format in the OpenAPI specification, how we can map the todo type in the data access layer and how we can use TypeScripts type system to express differences in representation in different layers without duplicating identical fields of todos. We will also discuss how architecture can evolve over time with its requirements.

Data Access Layer 🔗

The data access object bundles all the database communication of all the todo controllers. We cover the implementation of the data access layer in a later post.

For more information on this, checkout 'Why TypeScript is awesome for SQL queries'

import { v4 as uuid } from "uuid";
import { Todo, TodoId } from "./todo.type";

export async function getTodo(id: TodoId): Promise<Todo | undefined> {
  /* ... */
}

export async function getTodos(): Promise<Todo[]> {
  /* ... */
}

export async function createTodo(todo: Todo): Promise<Todo> {
  /* ... */
}

export async function updateTodo(id: TodoId, todo: Todo): Promise<Todo> {
  /* ... */
}

export async function deleteTodo(id: TodoId): Promise<void> {
  /* ... */
}

Why are there separate files for controllers but not for each DAO function? It's all about testing. In TypeScript, each file is a module. With jest, a module can be mocked. Each controller *.controller.ts has different requirements on the mock. Therefore, bundling all controllers into one file todo.controller.ts and having one test file todo.controller.spec.ts would make the mocking of the DAO more complicated. On the other hand, the module todo.dao.ts does not require jest mocks since it does not depend on internal modules. We will cover these aspects in more detail in following chapters.

Dedicated data access object 🔗

A common architecture is the layered architecture that separates between endpoints, services and data access layers. In this layered architecture, I often see that endpoints, services and data access are in different folders. This is necessary, if a function in the lower layers is called from multiple places in an upper layer.

As an example, let's take the endpoint GET /todos. Consider additionally a job that sends notifications for todos that reached their due date. Both the endpoint and the job need to access the database to receive todos. In a layered architecture, there would be one data access function selectTodos(): Todo[] independent of the call side. However, such a function is build with the assumption that both the endpoint and the job access the todos in the same way.

There are several scenarios where this is not the case:

The job might want to filter directly with the due date.
The endpoint GET /todos might want to expose additional filters.
The job might want to select each todo with the information if it already sent a notification.

Implementing these requirements all in the same selectTodos function leads to an unnecessarily complex generic function. Such a function is hard to reason about and complex to optimize. Therefore, I recommend to implement two separate functions selectTodos(assignee: string): Todo[] and selectTodos(): Todo[] in two different dedicated data access objects controllers/todos/todo.dao.ts and jobs/notifications/todo.dao.ts.

Remember: Ideally, each data access function is only called from one place.

These functions are optimized for their one specific usage. This means that they are simpler, that their bugs can only affect one endpoint, and that the blast radius of future changes is reduced.

For more information on this, checkout 'Identifying bad code with function usage patterns'

However, endpoints like GET /todos and POST /todos are tightly related. After a call to POST /todos, the user expects the same todo to be returned including a generated id. It would be unexpected if the endpoint GET /todos returns new fields or hides fields. Hence, in this architecture, a resource like /todos uses one type todo.type.ts and one local data access object todo.dao.ts.

Controllers 🔗

Each controller is a plain function that uses the data access layer. The controller is concerned about HTTP requests, responses and status codes.

As an example, the endpoint GET /todos/:id can result in a 404. The data access function getTodo returns undefined if the todo does not exist. It is the role of the controller to map this to a 404 response.

import { NextFunction, Request, Response } from "express";
import { getTodo } from "./todo.dao";

export async function getTodoController(
  request: Request,
  response: Response,
  next: NextFunction
): Promise<void> {
  try {
    const todo = await getTodo(request.params.id);
    if (todo) {
      response.send(todo);
    } else {
      response.sendStatus(404);
    }
  } catch (error) {
    next(error);
  }
}

Elimination of service layer 🔗

The classical layered architecture separates between endpoints, services and data access layers. In my experience, the service layer is an unnecessary indirection between the endpoints and the data access layers. I acknowledge though that there is complicated logic that should neither be in the endpoints nor in the data access layers. In this architecture, this complicated logic is accessed beside the data.

Endpoints
- Data Access
- Complicated Logic

In code, this looks like this.

import { NextFunction, Request, Response } from "express";
import { sendNotification } from "../../jobs/notifications/send-notification";
import { updateTodo } from "./todo.dao";

export async function putTodoController(
  request: Request,
  response: Response,
  next: NextFunction
): Promise<void> {
  try {
    const updatedTodo = await updateTodo(request.params.id, request.body);
    await sendNotification(updatedTodo);
    response.send(updatedTodo);
  } catch (error) {
    next(error);
  }
}

The controller accesses the database via the data access function updateTodo and then calls a function sendNotification to execute complicated logic in a separate module. We will talk about this aspect of the architecture in a later post about modules for complicated logic.

Conclusion 🔗

We explored how controllers are organized in the file system with data access functions, router and types. There is no service layer and there are no separate types within the layers. The next blog post is about testing these controllers.

Testing Express.js endpoints with supertest

2022-02-11T00:00:00Z

Did you ever publish code without writing tests for it? Well, I certainly did in the last post! Let's fix that with writing some unit tests based on jest and supertest for the controllers.

Well-written unit tests enable you to have fast feedback loops. A failure of one of them gives you a clear indication of a bug in your code. None of them failing gives you a clear indication of your code working.

As a first aspect, we want to have isolated tests. A test for a module should test exactly that module. It should not fail due to bugs in another module.

As a second aspect, we want to have completeness. If there is a bug in the code, then one of the tests should fail.

As a third aspect, we want the tests to be minimal. For one bug in the code, there should be exactly one test that fails. That's the hardest aspect to achieve (but we can thrive for it).

As a fourth aspect, we want the tests to be correct. A test should verify the property you want it to verify.

Remember: Great unit tests are isolated, complete, minimal and correct.

For completeness, there are coverage tools. For correctness, it's best to strive for simplicity and obviousness in both tests and application code. For minimalism, you can reason about the different inputs and outputs a function can have.

For isolation, we need to consider internal and external dependencies. While external dependencies outside of our software are rather stable, internal dependencies within our software are a frequent point of failure. Hence, it's important to mock internal dependencies while external dependencies are not that crucial to be mocked.

For the API endpoints implemented by todo controllers, external dependencies are HTTP and Express. Internal dependencies are the todo type todo.type.ts and the todo DAO todo.dao.ts. We want to mock these two internal dependencies.

Todo stub 🔗

In types, you want to utilize the benefits of mandatory fields. If you define a field assignee of a todo to be mandatory instead of optional, then you don't need to deal with it being undefined in the logic.

// This is better...
export interface Todo {
  id: TodoId;
  name: string;
  assignee: string;
  dueDate: string;
}

// ...than this.
export interface Todo {
  id?: TodoId;
  name?: string;
  assignee?: string;
  dueDate?: string;
}

However, this means that every time you create a todo, you need to provide all the fields.

const todo: Todo = {
  id: "123",
  name: "Some Name",
  assignee: "Yet another field?",
  dueDate: "2022-02-07",
};

This is cumbersome in tests for several reasons.

It's a lot of code.
It distracts from what's actually under test.
It's tough to figure out if the seamingly random assigned values are relevant.
With each addition of a mandatory field in the Todo type, all occurrences need to be adjusted.

To avoid these problems, it is helpful to create a function that creates a stub of a todo for test cases.

import { v4 as uuid } from "uuid";
import { Todo } from "./todo.type";

export const createStubTodo = (): Todo => ({
  id: uuid(),
  name: "Name",
  assignee: "Assignee",
  dueDate: new Date().toISOString(),
});

This way, tests can refer to createStubTodo() and a change to the Todo type only requires a change to the stub.

it("...", async () => {
  // ...
  const todo = createStubTodo();
  // ...
});

In some cases, you want to have a slightly different stub. You can achieve this with the omit function to remove fields.

it("...", async () => {
  // ...
  const todo = omit("id", createStubTodo());
  // ...
});

You can use the spread syntax to add or overwrite fields.

it("...", async () => {
  // ...
  const todo = createStubTodo();
  const createdTodo = {
    ...todo,
    id: uuid(),
  };
  // ...
});

I recommend to not have a stub function configurable with parameters createStubTodo(options?: StubOptions). It might be tempting in some cases, but it's worth it to keep these stub functions as simple as possible.

Let's assume the type Todo would also have a list of tasks.

interface Task {
  description: string;
}

interface Todo {
  // ...
  tasks: Task[];
}

In that case, I'd recommend to return the most minimal todo from the stub createStubTodo(): A todo without tasks.

export const createStubTodo = (): Todo => ({
  // ...
  tasks: [],
});

If a test requires a todo with tasks, they can use an additional stub createStubTask() and the spread syntax and compose the required todo.

const todo: Todo = {
  ...createStubTodo(),
  tasks: [createStubTask(), createStubTask()],
};

Remember: Keep stubs simple stupid. If the implementation of stubs is not trivial, then they themselves are a possible source of failure, risking correctness of the tests.

Mocking data access 🔗

Beside the internal dependencies of the todo type todo.type.ts, we also want to mock the data access layer todo.dao.ts. Since the TypeScript file todo.dao.ts forms a module, it can be mocked at runtime with jest jest.mock('./todo.dao').

import { v4 as uuid } from "uuid";
import { createStubTodo } from "./todo.stub";

jest.mock("./todo.dao");

describe("getTodoController", () => {
  // ...

  it("returns the todo with the id", async () => {
    const getTodo = require("./todo.dao").getTodo;
    const todo = createStubTodo();
    getTodo.mockResolvedValue(todo);
    // ...
    expect(getTodo).toHaveBeenCalledWith(todo.id);
  });

  it("returns a 404 if no todo with that id exists", async () => {
    const getTodo = require("./todo.dao").getTodo;
    const todoId = uuid();
    getTodo.mockResolvedValue(undefined);
    // ...
    expect(getTodo).toHaveBeenCalledWith(todoId);
  });
});

Note You need to use the import at runtime require("./todo.dao") such that you indeed get the mocked module. Remember that the controller module *.controller.ts is statically referring to the DAO module todo.dao.ts with import * from 'todo.dao'. Jest is overriding this at runtime.

Spinning up the HTTP server 🔗

We want to embrace and utilize the external dependency of HTTP and Express. There are be multiple approaches to test an API.

Call the controller functions directly.
Spin up the full API and send and receive HTTP requests and responses.
Spin up a minimal HTTP server and send and receive HTTP requests and responses.

Calling the controller functions directly looks first like a straightforward and isolated test approach without depending on HTTP and the runtime of Express. But the controller functions accept a request of the Express type Request and a response of the Express type Response as inputs. It is tough to properly mock and verify these input parameters to the controllers. Also, the generic type signatures of these types rather complicate the tests than that they are helping.

Note While we discard the option of testing controller functions directly, this is very much an option for testing of middleware functions in later posts.

Spinning up the full API is the approach closest to the production reality. However, in mature stages of the software with authentication and request logging, it leads to depending on these aspects which affects isolation of the controller tests. Also, tests become more complicated due to additional effort to pass access tokens.

The third way avoids that by spinning up a dedicated test HTTP server that only configures the required middleware.

import express, { Express } from "express";

export function server(configure: (express: Express) => void): Express {
  const app = express();
  app.use(express.json({ limit: "1mb" }));
  configure(app);
  return app;
}

This function can be used by each controller test to spin up the relevant endpoint.

import { server } from "../../test.functions";
import { putTodoController } from "./put-todo.controller";

describe("putTodoController", () => {
  const route = "/todos/:id";

  const app = server((app) => {
    app.put(route, putTodoController);
  });

  // ...
});

In later posts, we will see how we can utilize this server function to add error handling middleware and to mock authentication middleware.

Sending HTTP requests 🔗

For sending the HTTP requests, we are using a library called supertest. Given an Express app app: Express, it allows to send requests and verify the responses.

const response = await request(app)
  .put(`/todos/${todo.id}`)
  .send(todo)
  .expect(200);
expect(response).toHaveProperty("body", todo);

Controller test structure 🔗

With the todo stub, data access mocking, the server function and supertest, we have everything in place to write controller tests. We can define a general structure to all of these tests.

import request from "supertest";
import { server } from "../../test.functions";
import { createStubTodo } from "./todo.stub";
import { putTodoController } from "./put-todo.controller";

// Mock DAO module
jest.mock("./todo.dao");

describe("putTodoController", () => {
  // Define route under test
  const route = "/todos/:id";

  // Spin up http server
  const app = server((app) => {
    // Setup endpoint
    app.put(route, putTodoController);
  });

  it("updates a todo", async () => {
    // Get data access function mock
    const updateTodo = require("./todo.dao").updateTodo;
    // Create stub (if necessary)
    const todo = createStubTodo();
    // Setup mocks
    updateTodo.mockResolvedValue(todo);
    // Send request and expect response
    const response = await request(app)
      .put(route.replace(":id", todo.id))
      .send(todo)
      .expect(200);
    // Expect response body
    expect(response).toHaveProperty("body", todo);
    // Expect data access function to be called
    expect(updateTodo).toHaveBeenCalledWith(todo.id, todo);
  });
});

We mock the data access module and spin up a minimal server with only the required middleware and endpoint. A series of tests check for the expected properties of the endpoints. Each test gets the mock of the relevant data access function and sets it up. A request is send and a status code and response body expected. Additionally, it is verified that the controller called the data access function for the execution of a side effect.

Conclusion 🔗

We saw how we can build isolated controller unit tests with type stub, data access function mock and a minimal HTTP server. With these, we defined a general structure for controller tests. But at this point, all controllers accept all inputs without validation. In the next post, we are adding validation based on an OpenAPI specification.

OpenAPI input validation in Express.js

2022-02-16T00:00:00Z

With the creation of an OpenAPI specification, we can both achieve input validation and make the API visually explorable.

Up to this point, we have neglected to validate the inputs of the endpoints.

The ID parameter of the endpoints PUT /todos/:id, DELETE /todos/:id and GET /todos/:id needs to be verified to be a valid UUID.
The request bodies of POST /todos and PUT /todos/:id need to be verified to be valid todos.

Additionally, we don't provide any information outside of the source code on the available endpoints, what to send to them and what to expect to receive as response.

We want to fix both of these issues with an OpenAPI specification. With an OpenAPI specification openapi.json defined, we can use an Express middleware to validate inputs. Also, we can use a tools like Swagger UI or ReDoc to graphically explore the API.

Defining the OpenAPI spec directly as a JSON file openapi.json instead of using some generation-based approach, gives us the most flexibility. There exists a huge amount of documentation for writing OpenAPI specs. Code editors can autocomplete and lint since OpenAPI specs are described with a JSON schema. Also, there is no library in between that shadows and imitates the possibilities of OpenAPI. Therefore, if the OpenAPI spec advances, we don't rely on some library in between to be updated before we can use the new OpenAPI spec features.

Defining the OpenAPI spec 🔗

On a high-level, the OpenAPI specification defines some meta information, the various endpoints and the structure of requests and responses.

{
  "openapi": "3.0.1",
  "info": {
    "title": "Todo Service",
    "version": "0.0.1"
  },
  "servers": [
    {
      "url": "/"
    }
  ],
  "tags": [],
  "paths": {
    "/todos/{id}": {
      "get": {
        // ...
      },
      "put": {
        // ...
      },
      "delete": {
        // ...
      }
    },
    "/todos": {
      "post": {
        // ...
      },
      "get": {
        // ...
      }
    }
  },
  "components": {
    "schemas": {
      "Todo": {
        // ...
      }
    }
  }
}

Note You need to additionally define a second server URL { "url": "<url>" } if your application is not running on root in production. Example: If your API is accessible at /api/v1/, then this path needs to be present besides /. If one of these paths is missing, the input validation fails on either production or locally.

At first, we define a todo with its fields as a OpenAPI component.

The ID can be specified to be a UUID.
The name and assignee can be limited to be a string of a certain length.
The due date can be defined to be a date.

{
  "components": {
    "schemas": {
      "Todo": {
        "type": "object",
        "properties": {
          "id": {
            "type": "string",
            "format": "uuid"
          },
          "name": {
            "type": "string",
            "maxLength": 40
          },
          "assignee": {
            "type": "string",
            "maxLength": 160
          },
          "dueDate": {
            "type": "string",
            "format": "date"
          }
        },
        "required": ["name", "assignee", "dueDate"]
      }
    }
  }
}

Note We don't include the ID in the required fields. This way, we can use the same type for the POST and PUT request body. However, this lack of precision means that the response bodies don't guarantee the ID to be present. This is still better though than the complication through the alternative of using allOf.

Additionally, we define a schema for the response body on a 400 status code.

{
  "components": {
    "schemas": {
      "BadRequestError": {
        "type": "object",
        "properties": {
          "message": {
            "type": "string"
          },
          "errors": {
            "type": "array",
            "items": {
              "type": "object",
              "properties": {
                "errorCode": {
                  "type": "string"
                },
                "message": {
                  "type": "string"
                },
                "path": {
                  "type": "string"
                }
              }
            }
          }
        }
      }
    }
  }
}

With these schemas, we can define the endpoints. As an example, let's define the endpoint PUT /todos/:id. The complete endpoints can be found on GitHub.

{
  "paths": {
    "/todos/{id}": {
      "put": {
        "summary": "Updates a todo",
        "operationId": "updateTodo",
        "tags": ["Todos"],
        "parameters": [
          {
            "in": "path",
            "name": "id",
            "description": "The id of the todo",
            "required": true,
            "schema": {
              "type": "string",
              "format": "uuid"
            }
          }
        ],
        "requestBody": {
          "required": true,
          "content": {
            "application/json": {
              "schema": {
                "$ref": "#/components/schemas/Todo"
              }
            }
          }
        },
        "responses": {
          "200": {
            "description": "The updated todo",
            "content": {
              "application/json": {
                "schema": {
                  "$ref": "#/components/schemas/Todo"
                }
              }
            }
          },
          "400": {
            "description": "Bad request",
            "content": {
              "application/json": {
                "schema": {
                  "$ref": "#/components/schemas/BadRequestError"
                }
              }
            }
          }
        }
      }
    }
  }
}

We define that the endpoint PUT /todos requires the path parameter id to be a valid UUID. Additionally, it requires a JSON body to be send and this body needs to match the format of a todo. The endpoint can result in a success 200 or validation error 400. On a success, we assert that the response body matches the format of a todo. On a validation error, we assert the response body to have the structure of an error.

Input validation 🔗

There is an npm package express-openapi-validator that acts as an Express middleware before the controllers are reached. We create a file openapi.ts in a new folder pre-request-handlers. A file error-handler.ts in a folder error-handling contains an Express middleware to respond with an error response in case the OpenAPI validation finds errors. The openapi.json is positioned in an assets folder outside the source folder.

assets
`-- openapi.json
src
|-- pre-request-handlers
|   `-- openapi.ts
|-- error-handling
|   `-- error-handler.ts
|-- controllers
|   `-- todos
|       `-- todo.router.ts
`-- test.functions.ts

In the openapi.ts, we create a middleware validateInputs.

import path from "path";
import * as OpenApiValidator from "express-openapi-validator";

const spec = path.join("assets", "openapi.json");

export const validateInputs = OpenApiValidator.middleware({
  apiSpec: spec,
  validateRequests: true,
  validateResponses: true,
});

The application loads the static asset assets/openapi.json and passes it to the express-openapi-validator to validate the inputs.

Note In production systems, I recommend to set validateResponses to false. Response validation is helpful during development. On production, you don't want to return the user an error that the server response does not match the expected format. In later posts, we will have a configuration to distinguish production from local system.

The middleware needs to be registered in the todo router todo.router.ts.

Why not in the app.ts? This would lead to the validation of all incoming requests. However, there are endpoints you might not want to include in the OpenAPI specification. These can be the OpenAPI specification itself GET /openapi.json or health checks GET /healthcheck.

import express from "express";
import { validateInputs } from "../../pre-request-handlers/openapi";
import { deleteTodoController } from "./delete-todo.controller";
import { getTodoController } from "./get-todo.controller";
import { getTodosController } from "./get-todos.controller";
import { postTodoController } from "./post-todo.controller";
import { putTodoController } from "./put-todo.controller";

export const todoRoute = express.Router();

// Validating inputs before all todo controllers
todoRoute.use(validateInputs);

todoRoute.get("/:id", getTodoController);
todoRoute.get("", getTodosController);
todoRoute.put("/:id", putTodoController);
todoRoute.delete("/:id", deleteTodoController);
todoRoute.post("", postTodoController);

Globally in the app.ts, we need to make sure that on errors an error response is send.

import express from "express";
import { todoRoute } from "./controllers/todos/todo.router";
import { sendErrorResponse } from "./error-handling/error-handler";

const app = express();

// ...

app.use("/todos", todoRoute);

// If an error occurred, then send an HTTP error response
app.use(sendErrorResponse);

export default app;

The function sendErrorResponse is defined to handle errors past to the Express next function. For OpenAPI validation errors, it sends the status code 400 that the library express-openapi-validator specifies.

import { NextFunction, Request, Response } from "express";

export function sendErrorResponse(
  error: Error & { status?: number; errors?: unknown[] },
  _request: Request,
  response: Response,
  _next: NextFunction
): void {
  const status = error.status ?? 500;
  response.status(status).json({
    message: error.message,
    errors: error.errors,
  });
}

Inspecting the input validation 🔗

With input validation and error responses in place, we can take a look how the application behaves.

Note I'm using a tool httpie but curl or some graphical tool also does the job.

Starting the application with npm start, it runs on http://localhost:3000. A delete request with an invalid ID yields us correctly a bad request response 400.

> http DELETE localhost:3000/todos/123
HTTP/1.1 400 Bad Request
Connection: keep-alive
Content-Length: 174
Content-Type: application/json; charset=utf-8
Date: Thu, 10 Feb 2022 21:19:40 GMT
ETag: W/"ae-f4bL7i+yZUTT/K6oR9ojSGs7bsA"
Keep-Alive: timeout=5
X-Powered-By: Express

{
    "errors": [
        {
            "errorCode": "format.openapi.validation",
            "message": "should match format \"uuid\"",
            "path": ".params.id"
        }
    ],
    "message": "request.params.id should match format \"uuid\""
}

If we send a todo with multiple missing fields to the endpoint POST /todos, then we receive a response with all the occurred errors.

> http POST localhost:3000/todos dueDate=2022-02-04
HTTP/1.1 400 Bad Request
Connection: keep-alive
Content-Length: 363
Content-Type: application/json; charset=utf-8
Date: Thu, 10 Feb 2022 21:23:47 GMT
ETag: W/"16b-nWRX3gNUYyoDnSJNwgQS2UbbbQo"
Keep-Alive: timeout=5
X-Powered-By: Express

{
    "errors": [
        {
            "errorCode": "required.openapi.validation",
            "message": "should have required property 'name'",
            "path": ".body.name"
        },
        {
            "errorCode": "required.openapi.validation",
            "message": "should have required property 'assignee'",
            "path": ".body.assignee"
        }
    ],
    "message": "request.body should have required property 'name', request.body should have required property 'assignee'"
}

This is very valuable. A naive input validation would only return the first error it found, leaving the user in a cascade of discovering and fixing mistakes.

Testing 🔗

In the tests, you want to include the validation as well. This way, it is possible to test the input validation in the tests of the controllers.

import express, { Express } from "express";
import { sendErrorResponse } from "./error-handling/error-handler";
import { validateInputs } from "./pre-request-handlers/openapi";

export function server(configure: (express: Express) => void): Express {
  const app = express();
  app.use(express.json({ limit: "1mb" }));
  // Validating inputs before all controllers
  app.use(validateInputs);
  configure(app);
  // Send an error response if an error occurred
  app.use(sendErrorResponse);
  return app;
}

With the input validation in place, the controller tests can be extended with tests of bad requests. As an example, let's take a look at passing an invalid ID to the endpoint DELETE /todos/:id.

import request from "supertest";
import { v4 as uuid } from "uuid";
import { server } from "../../test.functions";
import { deleteTodoController } from "./delete-todo.controller";

jest.mock("./todo.dao");

describe("deleteTodoController", () => {
  const route = "/todos/:id";

  const app = server((app) => {
    app.delete(route, deleteTodoController);
  });

  // ...

  it("rejects an invalid ID", async () => {
    const deleteTodo = require("./todo.dao").deleteTodo;
    const todoId = "123";
    deleteTodo.mockResolvedValue();
    const response = await request(app)
      .delete(route.replace(":id", todoId))
      .expect(400);
    expect(response).toHaveProperty(
      "body.message",
      'request.params.id should match format "uuid"'
    );
    expect(deleteTodo).not.toHaveBeenCalled();
  });
});

Serving the OpenAPI spec 🔗

So far, we only use the OpenAPI specification for input validation. Additionally, we can expose it via an endpoint GET /openapi.json. This requires a small adjustment in the pre-request-handlers/openapi.ts to introduce a function serveOpenapiSpec. This function serves the static file assets/openapi.json.

import path from "path";
import * as express from "express";
// ...

const spec = path.join("assets", "openapi.json");

export const serveOpenapiSpec = express.static(spec);

export const validateInputs = /* ... */;

In the application app.ts, the OpenAPI specification can now be served.

import express from "express";
// ...
import { serveOpenapiSpec } from "./pre-request-handlers/openapi";

const app = express();

// ...
app.use("/openapi.json", serveOpenapiSpec);
// ...

export default app;

This makes the OpenAPI specification available at GET /openapi.json. With tools like Swagger UI or ReDoc, this OpenAPI specification can be visualized.

Conclusion 🔗

We have set up input validation based on a declarative OpenAPI specification. The OpenAPI specification is tested within the controllers. It is exposed via an endpoint such that the API can be visually explored. In the next post, we take a deeper look at error handling and logging.

Advancing your company as a Software Engineer

2022-03-07T00:00:00Z

Life as a software engineer can be frustrating. You know that this new lib will make everybodies life easier. Nevertheless, you can't just introduce it: You need to get an approval. Reflecting on my past experiences of 12 successful and 3 unsuccessful advancements, I show you key learnings on how to inflict technical and organizational advancements.

You would like to get rid of Java and utilize Kotlin, TypeScript or Python? You would like to replace the cumbersome Hibernate ORM with a more lightweight, less frustrating approach? You would like to base your teams priorities on data and metrics?

As a software engineer, these decisions are not fully in your hands. However, these decisions are neither fully out of your hands. Such decisions are a result of the input of a large set of people. As a software engineer, you have the ability to influence decisions at a company into a favorable direction.

For this post, I reflected on a dozen of past experiences where I participated in or drove technical or organizational advancements. During these advancements, I have been in the position of a Senior Software Engineer. As such, I had some but not full control over languages, tech stack and processes. With the learnings from this article, you will be able to approach such advancements from such a or from a similar position.

Reflecting the twelve experiences, I investigated what the common factors have been that made the advancements possible.

Support 🔗

Introducing a technology or advancing a process is not a one-man-show. It doesn't matter if an advancement is objectively beneficial. If you can not convince your team members and stakeholders and address their concerns, then it is not beneficial to implement that change. In the following sections, I show different strategies to gain the required support of team members, stakeholders and the company.

Convincing team members 🔗

There can be many reasons why team members could not agree with your advancement. For one technology change, a team member told me that they are afraid that their previous knowledge would become useless after that advancement. For a change to a more functional programming style, some team members were used to an imperative programming style and preferred it. For introducing an event-based mechanism, there have been team members that made bad experiences with an event-based mechanism before.

To address such concerns, I found it helpful to go smaller steps. This makes the changes more approachable and leaves the option to revert.

Instead of moving from Java to Kotlin, we utilized the library Vavr as a functional advancement to Java collections. Later, we used more and more functional data types like Try and Either from the same library. Eventually, a team decided to write a new service in Kotlin.

Instead of immediately basing a new service on Kubernetes, we first got it working on the old runtime. We avoided to introduce blockers for replication and found solutions that would work within Kubernetes. Later, we moved the service to run on Kubernetes.

Another strategy is to reduce scope.

Instead of introducing RxJava in various places across a microservice, we limited its usage to a single class. Later, we gathered experience with it and team members started to see the value and adopted it in other places.

Instead of replacing all internal SDKs with OkHttp at once, we replaced a single SDK and monitored it in production. Later, we replaced the remaining SDKs.

Convincing stakeholders 🔗

Also, depending on the organizational setup, external stakeholders outside the team need to be convinced.

I found that some strategies are effective to convince stakeholders.

Gain broad support of the team
Separating controversial from non-controversial changes
If all fails: Going rogue

If you are able to get more team members to vouch for a technical advancement, then an external stakeholder is more likely to allow it. As an example, when introducing Node.js in the team, all team members including Java developers were convinced that it was the right tool for the job. When introducing testcontainers to allow isolated component tests, the shared e2e-test suite was widely considered as a problem.

If you are able to separate the controversial changes from the non-controversial changes, then the controversial change that needs a stakeholder approval is smaller, entails lower risk and is more likely to get approved. The switch to Kubernetes entailed the risk that the replication would not work with the way we build our service logic. During the implementation on the old runtime, we kept an eye on its readiness for Kubernetes. Avoiding local in-memory state, long-running transactions, application-internal locks, etc. Ensuring this is non-controversial: No external stakeholder will tell you to please store variables in global application variables or to keep this transaction open over three class boundaries (Well, if they do, run!). However, a buzzword like Kubernetes is more likely to catch the attention of an external stakeholder and is hence more controversial. For the introduction of Kubernetes, we managed to avoid introducing blockers via non-controversial work and then had a lower-risk less controversial switch to Kubernetes.

The last strategy of going rogue needs to be used rarely since if used wrongly, it can massively undermine trust. When I am referring to going rogue, I am talking about consciously acting against expectations of what you are doing for a small amount of time to prove a thesis that you were not able convince people otherwise. In the very few times I was going rogue, it was always for a few days only and never without the knowledge of at least one trusted person.

Note: With more freedom in time organization in higher positions, going rogue becomes less and less necessary. With fewer expectations on what specific tasks you spend your time on, investigating is just part of your job and nobody would consider it to be going rogue.

If you go rogue, it is mandatory that you are correct. Ensuring this, requires that you align your views with at least some person beforehand. If you can't find support, it is likely that you are wrong. Then don't go rogue, rather try to broaden your perspective. There is something that you are not seeing that would change your view if you knew it.

Convincing a company 🔗

If you are striving for huge changes like moving from imperative to functional programming, from top-down to bottom-up management or from service-oriented to domain-driven architecture, then this requires a continuous change of a companies culture.

Such changes are not happening in a sprint or in a quarter. Even a person in charge can not mandate that everybody now writes functional instead of imperative code. There has to happen a continuous process of cultural change.

There are strategies to achieve such a change over a period of years.

Knowledge sharing: Distribute resources for people to become more knowledgeable and familiar in the topic
Templates and best practices: Make it work somewhere and share recipes to adopt the approach elsewhere
Facilitate change: Act as a knowledge hub and offer support for adoption

As an example, we aimed for an increase in the frontend development efficiency with the company-wide adoption of micro frontends. To achieve this, we first got a micro frontend working within the team. After that, there was a template to adopt and a small optional library to use. The approach and the knowledge was shared in a presentation and documentation for the adoption was written. When opportunities in the different teams opened up, some of them reached out for and got support. After a year, we had eleven micro frontends across six teams in four tribes. The culture successfully changed from monolithic frontend development to micro frontends.

Gaining the support of the team is mandatory, gaining the support of external stakeholders is beneficial, changing a culture is necessary for huge changes. Another important factor I encountered was timing.

Timing 🔗

For eleven out of twelve advancements, there has been some opportunity that opened up. For three of these advancements, it was a new service that was being bootstrapped. That is the case for the introduction of Jdbi as a Hibernate alternative, the introduction of input validation with an OpenAPI spec and the introduction of Node.js within the team.

Note: That is one of the benefits of a microservice architecture. It allows to continuously open up opportunities for technical advancements.

For the introduction of Vavr, the opportunity was the director mentioning this technology from a conversation with another team. For the introduction of RxJava, long wait times for users required a solution, with no parallelization in place up to this point. For the introduction of product metrics within the team, it was the director pushing for more OKR involvement of the teams. For the introduction of a zero-bug policy, it was a bug scrubbing week eliminating all major bugs of the team. For the introduction of micro frontends, it was the engineering manager and director pushing for a general speedup in frontend development.

Timing and opportunity are crucial for technical and organizational advancements.

In three major unsuccessful advancements that I have pushed for, either the timing was not good or the opportunity was not big enough. For the introduction of a search service, the intended functionality of a type-ahead search didn't justify the effort of the setup of a new service in an environment where at that time setting up new services was rare and complex. For the introduction of a Kafka event bus to solve performance issues in the communication of two services, this would have risked the timely delivery of the functionality. For the introduction of a document database, the loose structure of the data types were to small of a problem to justify this effort of setting up additional backup and compliance strategies across the company.

Overall, it is essential to listen for signs when it is a good time and when it is a bad time to push for advancements. You don't want to repeatedly bring up a topic in times where customer complaints are gathering, where a deadline looms or where the value of a service itself needs yet to be proven. On the other hand, you want to listen for opportunities that higher-ups are creating and for problems in the customer value delivery that need a solution.

Context 🔗

I am a huge fan of pure functional programming languages. Nevertheless, I have never pushed for my team to write a service in Haskell. The reason for this is context.

There is no objective truth to what the best technologies are. In different teams, different technologies can be the best choice to make.

In a team of mostly Java developers like my first three teams, a step towards Kotlin is logical as an advancement. In a team of mostly Angular frontend developers like my fourth team, a step in the backend towards Node.js is a more reasonable advancement.

Additionally, not only your team is part of the context you need to consider, but also existing code. When we replaced Hibernate, there were options to go for Jooq or Jdbi. I was first vouching for Jooq because of its modularity. Nevertheless, with the context of a Dropwizard service, we went for the similarly powerful option Jdbi because of its integration with Dropwizard.

As a software engineer trying to advance the tech stack, it is crucial to stay flexible towards the technologies. Don't fixate on solutions, but on the problems that require solving.

Problem 🔗

Most importantly, but still often overlooked, is that an advancement solves a problem. The twelve advancements I investigated, had an underlying concrete problem that was becoming apparent.

Many technologies solve an abstract problem.

Node.js solves the problem of verbosity of Java
Testcontainers solves the problem of running tests against a database in unit test suites
RxJava solves the problem of complicated multi-threading of explicit threads

However, if you want to advance a technology stack, there needs to be a concrete problem in your context at your current point in time. The benefits of solving a concrete problem are more apparent then the abstract benefits of solving an abstract problem.

Node.js solved the problem of writing tons of Java data classes in the context of a JSON-heavy API at a time where we wanted to bootstrap and iterate on a new service quickly
Testcontainers solved the problem of the bottleneck of a common integration test suite in the context of a Java microservice at a time where the number of engineers was doubling per year
RxJava solved the problem of complicated multi-threading in the context of a single-threaded Java microservice at a time where the user load started to stress the service

Analyzing and understanding the underlying problem is crucial. Knowing the solutions you can supply to specific sets of problems allows you to more easily identify the problem.

Conclusion 🔗

When introducing technical and organizational advancements, I have four types of advice.

Gather the support of the team, external stakeholders or the entire organization depending on the scale.
Seize opportunities that open up.
Consider the context of your team and existing code.
Solve a concrete apparent problem.

Appendix: List of advancements 🔗

Introducing Vavr 🔗

Problem: In an increasingly functional architecture, the Java Streams API is verbose and makes testing difficult.
Timing/Opportunity: Director mentions the library.
Context: Java microservice with Java engineers.
Support: Create a draft pull request showcasing to team members a part of the production code with Vavr instead of Java Streams API.

Introducing Jdbi 🔗

Problem: The mutation-based Hibernate ORM prevents a functional architecture.
Timing/Opportunity: New service is being bootstrapped.
Context: Java microservice with Java engineers.
Support: Compare available alternatives (Jooq and Jdbi) and develop and present first usage.

Introducing RxJava 🔗

Problem: Parallelization and asynchronicity with threads is complicated, error-prone and difficult to test.
Timing/Opportunity: Maturing service causes long waiting times for users.
Context Single-threaded Java microservice with Java engineers.
Support: Introduce RxJava simplistically in a single class and continuously utilize RxJava features.

Introducing OkHttp 🔗

Problem: A dependency on a company-internal configured HTTP client causes problems with updating and version conflicts in the dependency tree.
Timing/Opportunity: Wide acknowledgment that the company-internal SDK is an anti-pattern.
Context: Java microservice with Java engineers with internal SDK usage.
Support: Discuss and continuously replace different SDKs with OkHttp over a few of days.

Introducing validation with OpenAPI 🔗

Problem: The published OpenAPI specification is lacking precision and gets out of sync with the input validation.
Timing/Opportunity: New service is being bootstrapped.
Context: Mandatory OpenAPI specs and new Node.js service with primarily frontend engineers.
Support: Discuss and setup mandatory OpenAPI specification and then add the validation.

Introducing testcontainers 🔗

Problem: A long-running integration test suite becomes a bottleneck for independent releases.
Timing/Opportunity: None (or I forgot)
Context: Java microservice with no integration tests with Java engineers.
Support: Try and compare different libraries and get it running on a service.

Using Kubernetes for the first service in the company 🔗

Problem: A company-wide release train becomes the bottleneck for team velocity and prevents fast feedback loops in the development of a new product.
Timing/Opportunity: A platform team did the ground work for Kubernetes and the director is pushing for a first service in Kubernetes.
Context: Central Kubernetes knowledge in platform team and no prior experience in team of Java engineers
Support: Prevent blockers up-front and in a sprint of interteam-collaboration setup the Kubernetes deployment.

Introducing Node.js in the team 🔗

Problem: A new JSON-heavy product becomes unfeasible to maintain in a Java application reducing team velocity.
Timing/Opportunity: New service is being bootstrapped.
Context: Team of mostly frontend engineers and few Java engineers, bad team experience with existing Java service, JSON-heavy service.
Support: Evaluate two existing Express.js and Nest.js services, gather feedback and team support, get approval from external stakeholders

Introducing a micro frontend in the team 🔗

Problem: The development of a small frontend within a frontend monolith for a backend-heavy product risks to reduce team velocity.
Timing/Opportunity: A basic user-facing UI is required for a backend-heavy application.
Context: Backend-heavy team with little to no frontend experience
Support: Evaluate existing options with other team, frontend lead and infrastructure engineer, talk with product owner and team, get approval from external stakeholders

Introducing micro frontends in the company 🔗

Problem: Growing engineering head count and complexity of the frontend increases release times and decreases engineering velocity.
Timing/Opportunity: Engineering manager pushing for general solution and a new part in the frontend is being developed
Context: Different micro frontend technologies working in two other teams, weaknesses known
Support: Spike webcomponent-based solution with manager and director, implement product frontend, address challenges, present to engineering organization, write setup guide, offer setup support

Introducing product metrics as the first team in the company 🔗

Problem: The team is able to deliver faster than qualitative user feedback can be gathered.
Timing/Opportunity: Director is pushing for more team involvement in OKRs and access to tools is being allowed to request.
Context: Existing operational metrics in a team with no product metrics experience
Support: Implement product metrics and start investigating and refining them in sprint reviews additionally to the existing review method.

Pushing an engineering-wide RFC process 🔗

Problem: Growing engineering head count makes it tougher to coordinate and develop a common engineering culture.
Timing/Opportunity: Staff engineer recommends RFC process, frontend people complain they need to be in a meeting to stay informed and frontend lead pushes for the usage of first RFCs.
Context: Meeting-based exchange between teams and the engineering organization.
Support: Start writing first RFCs and share them with the organization, gathering and incorporating feedback.

Introducing a zero-bug policy in the team 🔗

Problem: A huge backlog of old bugs reduces team velocity.
Timing/Opportunity: In a company-wide bug fixing week all important bugs are eliminated.
Context: Scrum sprints with a backlog of hundreds of untouched bugs older than six months
Support: Mention zero-bug policy of Microsoft multiple times, seize opportunity.

Introducing a search service 🔗

Unsuccessful
Problem: A fully overhauled memory-heavy type ahead search risks the stability of the main application.
Timing/Opportunity: Missing organizational experience in setting up new service, advancement to big in comparison to the problem
Context: Monolithic application with few services around it

Introducing an event-based approach 🔗

Unsuccessful
Problem: A new high-traffic service heavily affects the availability of the main application.
Timing/Opportunity: Advancement to big in comparison to the problem.
Context: Local Kafka's running semi-successfully

Introducing a document-based database 🔗

Unsuccessful
Problem: A new domain-specific type with many subtypes risks to reduce team velocity in a schema-based database.
Timing/Opportunity: Advancement to big in comparison to the problem.
Context: Node.js microservice with mainly TypeScript engineers

JSON Logging with Pino in Express.js

2022-03-10T00:00:00Z

Logging remains one of the most effective techniques to gain insight into what is happening in a program. Furthermore, structured logging improves tracing and filtering. In this post, we add a JSON logger into the opinionated Node.js microservice architecture.

Pino is one of the better known JSON loggers in the Node.js ecosystem. We install it as a dependency.

npm install --save pino

We create a global logger object in a dedicated file src/logger.ts.

import pino from "pino";

export const logger = pino({
  level: "info",
});

This logger can be used throughout the application.

import { logger } from "./logger";

logger.info({ some: "data" }, "This is a test");

It outputs a single line with a JSON containing meta information on log level, time, process and host. Additionally, it embeds the custom JSON data and the log message.

{
  "level": 30,
  "time": 1646582909278,
  "pid": 12345,
  "hostname": "fboeller",
  "some": "data",
  "msg": "This is a test"
}

Logging incoming requests 🔗

There are log lines that we don't want to repeat throughout the application. For an application serving an HTTP API, it is useful to always have a log line for each incoming request. This way, we avoid that for some endpoints this log line is forgotten or that the logs are structurally different.

To achieve that, we define an Express.js middleware as a pre-request handler.

npm install --save express-pino-logger
npm install --save-dev @types/express-pino-logger

We create a file src/pre-request-handlers/log-request.ts that defines a middleware logRequest.

import expressPino from "express-pino-logger";

export const logRequest = expressPino({
  level: "info",
  enabled,
});

In the application src/app.ts, we register this middleware.

// ...

const app = express();

app.use(logRequest);

// ...

export default app;

Now, with a server being started with npm start and a request being sent with http http://localhost:3000/todos, we see a log line with information about the request and the response.

{
  "level": 30,
  "time": 1646583745857,
  "pid": 14865,
  "hostname": "fboeller",
  "req": {
    "id": 1,
    "method": "GET",
    "url": "/todos",
    "query": {},
    "params": {},
    "headers": {
      "host": "localhost:3000",
      "user-agent": "HTTPie/0.9.8",
      "accept-encoding": "gzip, deflate",
      "accept": "*/*",
      "connection": "keep-alive"
    },
    "remoteAddress": "::ffff:127.0.0.1",
    "remotePort": 48864
  },
  "res": {
    "statusCode": 200,
    "headers": {
      "x-powered-by": "Express",
      "content-type": "application/json; charset=utf-8",
      "content-length": "2",
      "etag": "W/\"2-l9Fw4VUd7kr8CvBat4zaMCqXZ2w\""
    }
  },
  "responseTime": 25,
  "msg": "request completed"
}

Note: By default, all header and query parameters are logged. Therefore, be careful, when access tokens are sent. In a later post on authentication, we will change the default serialization to exclude access tokens from logging.

Logging in the context of requests 🔗

On some occasions, you might want to log additional information in the context of an incoming request. In a controller, you might be tempted to call logger.info(...) to achieve that. However, once you want to trace a specific request in production, such a log line can not be associated with the request.

The express-pino-logger attaches a dedicated request logger request.log to the request object.

import { NextFunction, Request, Response } from "express";

export async function myController(
  request: Request,
  response: Response,
  next: NextFunction
): Promise<void> {
  request.log.info({ some: "data" }, "My custom log line");
}

With the usage of this logger, the log line includes an identical request id. This way, you can search for all log lines from a specific incoming request.

{
  "level": 30,
  "time": 1646588777952,
  "pid": 19777,
  "hostname": "fboeller",
  "req": {
    "id": 1,
    "method": "GET",
    "url": "/todos"
  },
  "some": "data",
  "msg": "My custom log line"
}

Request logging in tests 🔗

Once you use the logger request.log, you run into errors in the test cases.

expected 200 "OK", got 500 "Internal Server Error"

Since the logRequest middleware is not registered in the test cases, there is no object log defined on request and hence a call to request.log.info(...) fails. However, registering the logRequest middleware causes log lines in the test cases. This reduces the readability of the test results and is therefore valuable to avoid.

The middleware expressPino exposes an option enabled to disable the logger. We adjust the logRequest function in the file src/pre-request-handlers/log-request.ts to use that option.

import expressPino from "express-pino-logger";

export const logRequest = (enabled: boolean) =>
  expressPino({
    level: "info",
    enabled,
  });

In the application src/app.ts, the value true is passed such that the logger is enabled.

// ...

const app = express();

app.use(logRequest(true));

// ...

export default app;

In the server function of the file src/test.functions.ts, the logger is turned off.

import express, { Express } from "express";
// ...

export function server(configure: (express: Express) => void): Express {
  const app = express();
  app.use(logRequest(false));
  // ...
  return app;
}

This way, request.log is now defined in test cases. The tests don't fail anymore and still no log lines are logged in tests.

Request-independent logging in tests 🔗

Also, calls to logger.info and alike should not lead to log lines in tests. Mocking the module logger.ts allows both to suppress the log lines and also spy on the calls to the logger.

jest.mock("./logger", () => ({
  logger: {
    info: jest.fn(),
    error: jest.fn(),
  },
}));

Now, expectations towards the logger can be formulated.

expect(request("./logger").error).toHaveBeenCalledTimes(1);

Prettify log lines 🔗

Looking at a single-lined colorless JSON log in your local shell might not be the desired user experience you wish to have in your daily work.

Server listening on port 3000!
{"level":30,"time":1646585672385,"pid":16618,"hostname":"fboeller","req":{"id":1,"method":"GET","url":"/todos","query":{},"params":{},"headers":{"host":"localhost:3000","user-agent":"HTTPie/0.9.8","accept-encoding":"gzip, deflate","accept":"*/*","connection":"keep-alive"},"remoteAddress":"::ffff:127.0.0.1","remotePort":49494},"res":{"statusCode":200,"headers":{"x-powered-by":"Express","content-type":"application/json; charset=utf-8","content-length":"2","etag":"W/\"2-l9Fw4VUO7kr8CvBlt4zaMCqXZ0w\""}},"responseTime":20,"msg":"request completed"}

Fortunately, there is a tool pino-pretty for that.

npm install --save-dev pino-pretty

This shell tool takes input from stdin, makes it prettier and outputs it to stdout. In the scripts of the package.json, we pipe the output of the server watch to pino-pretty with some reasonable default parameters.

{
  "scripts": {
    "_nodemon": "nodemon --legacy-watch --watch ./dist --inspect=0.0.0.0:9229 dist/main.js -- server | npx pino-pretty -i time,hostname,module,__in,name,pid"
  }
}

In the development server npm start, the output of the logs is now more readable.

Server listening on port 3000!

INFO: request completed
    req: {
      "id": 1,
      "method": "GET",
      "url": "/todos",
      "query": {},
      "params": {},
      "headers": {
        "host": "localhost:3000",
        "user-agent": "HTTPie/0.9.8",
        "accept-encoding": "gzip, deflate",
        "accept": "*/*",
        "connection": "keep-alive"
      },
      "remoteAddress": "::ffff:127.0.0.1",
      "remotePort": 49602
    }
    res: {
      "statusCode": 200,
      "headers": {
        "x-powered-by": "Express",
        "content-type": "application/json; charset=utf-8",
        "content-length": "2",
        "etag": "W/\"2-l9Fw4VUO7kr8CvBlt4zaMCqXZ0w\""
      }
    }
    responseTime: 21

Note: Add a reasonable default call to pino-pretty on every command that outputs JSON log in your package.json. Whether you start the application locally or you retrieve production logs, you'll always want to have it formatted.

Conclusion 🔗

We added a structured logger pino to the Node.js microservice architecture. It logs every incoming request with the express-pino-logger and allows to define custom logs inside and outside of the context of requests. In test cases, the logger does not pollute the test output. Locally, the structured logs are prettified with pino-pretty.

Managing configs with TypeScript code in Node.js

2022-03-21T00:00:00Z

In this blog post, we implement different configurations within TypeScript code for local development and production environment in the Node.js microservice architecture.

It's beneficial to keep local and production software similar. This reduces the chance to have bugs in a production environment that are hard to reproduce locally. Still, minor differences like disabling access token expiration locally can make your developers life easier.

We create a new folder configuration within the src directory.

configuration
|-- config.ts
|-- config.type.ts
`-- configs
    |-- get-config.ts
    |-- get-config.spec.ts
    |-- get-local-config.ts
    |-- get-local-config.spec.ts
    |-- get-production-config.ts
    `-- get-production-config.spec.ts

At this point, we distinguish between local and production config. In a later post, we provide configurations for individual engineers. This allows for additional adjustments like Macs dockerhost and Linux localhost as the domain of the local Docker environment.

Note: The configuration is a TypeScript file in this setup. With full deployments within minutes, there is no need to view configuration as something external and exchangeable from the application. Treating the configuration as code brings type-safety and testability.

The config object 🔗

The main idea is that the file config.ts is the single entrypoint of the configuration folder. It defines a global constant config that can be read from every point in the application.

import { ProcessVariables } from "./config.type";
import { getConfig } from "./configs/get-config";

export const config = getConfig(process.env as unknown as ProcessVariables);

This constant config is initialized from a pure testable function getConfig. The function getConfig receives the process variables of type ProcessVariables and returns the configuration of type Config. The type ProcessVariables defines the options that can be passed via environment variables to the application. The type Config defines the common format of production and local configuration.

import { Level } from "pino";

export type Environment =
  // The service running in a production cluster available for customers
  | "production"
  // The service running locally on a development machine
  | "local";

export interface Config {
  environment: Environment;
  logLevel: Level;
}

export interface ProcessVariables {
  ENV?: Environment;
  LOG_LEVEL?: Level;
}

As a first option, we allow the logLevel to be configurable.

Configuration selection 🔗

The creation of the configuration getConfig dispatches to the proper function for production and local configuration depending on the environment.

import { Config, Environment, ProcessVariables } from "../config.type";
import { getProductionConfig } from "./get-production.config";
import { getLocalConfig } from "./get-local.config";

export function getConfig(processVariables: ProcessVariables): Config {
  const environment: Environment = processVariables.ENV || "local";
  switch (environment) {
    case "production":
      return getProductionConfig(processVariables);
    case "local":
      return getLocalConfig(processVariables);
  }
}

Note: We default to the local environment in case the environment variable ENV is not set. This is a general pattern: Local first. The goal is to reduce the required setup for engineers by not requiring additional input like ENV=local to commands. For the production version, especially in an infrastructure-as-code approach, the value production can be passed for the environment variable ENV.

The function getConfig can be tested.

import { getConfig } from "./get-config";

jest.mock("./get-local.config", () => ({
  getLocalConfig: () => ({ environment: "local" }),
}));
jest.mock("./get-production.config", () => ({
  getProductionConfig: () => ({ environment: "production" }),
}));

describe("the configuration", () => {
  it("defaults to the local environment", () => {
    expect(getConfig({})).toHaveProperty("environment", "local");
  });

  it("returns the local config for a local environment", () => {
    expect(getConfig({ ENV: "local" })).toHaveProperty("environment", "local");
  });

  it("returns the production config for a production environment", () => {
    expect(getConfig({ ENV: "production" })).toHaveProperty(
      "environment",
      "production"
    );
  });
});

Environment-specific configuration 🔗

The files get-local.config.ts and get-production.config.ts define how the respected environment configurations look like. The local configuration get-local.config.ts defaults the log level to debug.

import { Config, ProcessVariables } from "../config.type";

export function getLocalConfig(processVariables: ProcessVariables): Config {
  return {
    environment: "local",
    logLevel: processVariables.LOG_LEVEL ?? "debug",
  };
}

It has dedicated tests. At the current simplicity of the configuration, having dedicated tests is over-engineered. However, in following posts with incoming authentication, outgoing authentication and multi-region support such a configuration becomes increasingly complex.

import { getLocalConfig } from "./get-local.config";

describe("the local configuration", () => {
  it("prefers the log level from the environment", () => {
    expect(getLocalConfig({ LOG_LEVEL: "fatal" })).toHaveProperty(
      "logLevel",
      "fatal"
    );
  });

  it("defaults the log level to debug", () => {
    expect(getLocalConfig({})).toHaveProperty("logLevel", "debug");
  });
});

The production configuration get-production.config.ts is similar and can be viewed in the full code on GitHub.

Using the config object 🔗

Within the logger src/logger.ts, it is now possible to use the global object config.

Note: This approach leads to the configuration being initialized on the first execution of a module that imports the configuration.

import pino from "pino";
import { config } from "./configuration/config";

export const logger = pino({
  level: config.logLevel,
});

Within the request logger src/pre-request-handlers/log-request.ts, the log level can also be considered.

import expressPino from "express-pino-logger";
import { config } from "../configuration/config";

export const logRequest = (enabled: boolean) =>
  expressPino({
    level: config.logLevel,
    enabled,
  });

Passing environment variables 🔗

For local exploration testing with different environment variables, the changed environment variables can be passed directly to npm start.

LOG_LEVEL=fatal npm start
ENV=production npm start
ENV=production LOG_LEVEL=error npm start

Mocking the config object 🔗

Within tests, it is possible and advisable to mock the configuration. It is sufficient to only provide the configuration options that are relevant in the scope of the test.

jest.mock("./configuration/config", () => ({
  config: {
    logLevel: "info",
  },
}));

Okta Breach - Don't outsource your business model

2022-03-25T00:00:00Z

With the handling of the LAPSUS$ breach, Okta has shown how to not do security and communication. Three core learnings from #OktaBreach.

In March 2022, the hacker group LAPSUS$ claimed that they have gotten access to customers of Okta. Okta responded in multiple updated statements to the claims. Their response gives us insights into how they handle incidents.

From an outside naive view, I want to present what I think went wrong at Okta and how to fix it.

Don't outsource your core business model 🔗

Okta is selling a single sign on product. As such, security is a core part of the product they are selling.

Okta outsourced their customer support to a third party company. For customer support to reproduce issues and resolve them, they need access to the customer's data. Therefore, Okta not only outsourced customer support but also part of their security.

Hence, when an attempt occurred to add an additional multi-factor at the third party in January, they could not simply investigate within Okta. They needed to request a report and wait for it.

The issue here is not that the report arrived to late. Even if the report came back in a week, it is too slow of a reaction to a security incident. The full handling of a security incident must be happening within the company with fast reaction times. Especially if security is the companies business model.

Hence, don't outsource your core business model.

Don't publish wrong information 🔗

Okta's first response denied the claims to be true. While in an update they corrected that statement, such information shouldn't have been published in the first place if the investigation is still ongoing.

It's important to respond fast. But if there is no certainty, then it's better to communicate: "We don't have evidence of a breach. Investigations are still ongoing."

Especially in the B2B-Business, published information is cascaded. If a wrong statement is published, it is forwarded internally in the customer companies and its correction not only damages Okta's reputation but also the reputation of the security departments within the companies.

On the bright side, at least Okta published a correction.

But still, don't publish possibly wrong information if investigations are still ongoing.

Don't redirect the blame 🔗

In Okta's communication, the focus lied on talking the incident small, reassuring the customers and redirecting the blame to the third party. That doesn't help any customer that is possibly affected. It is focused on protecting Okta's assets, not the customers.

When blaming the third party and its long time to complete a report, Okta ignores that they are in charge of setting up the processes that lead to it. They fail to see that customers not only expect this one incident to be handled, but also upcoming incidents to be prevented by adjusting the processes.

While it's possible that Okta internally learns and externally attempts to protect their image, open and transparent communication has repeatedly shown to be better to protect an image.

Hence, don't redirect the blame, apologize and show how you prevent such incidents from occurring in the future.

Conclusion 🔗

Don't outsource your core business model. The processes around your core business model need to stay fast.

Don't publish wrong information. It's okay to say that you are still investigating.

Don't redirect the blame. You are in charge of setting up the processes.

JWT authentication with JWKS in Express.js

2022-04-02T00:00:00Z

We discuss how to implement JWT authentication with JWKS, how to disable auth locally and how to suppress logging tokens in production logs.

There are multiple ways to approach authentication in an application. A currently popular way is the usage of JSON web token (JWT).

JWT are created and signed by a central authority like a dedicated authentication service. A user receives a JWT from that service, the application receives the signing key from the service. The user attaches the JWT on every request to our application. Our application validates the signature against the signing key.

For authorization, the JWT can contain some structured information on user and permission. We will cover authorization in a later post.

Authenticating endpoints 🔗

In our application, the access to some endpoints should be authenticated, to others not. Unauthenticated endpoints include for example the endpoint GET /openapi.json. If these endpoint would require authentication, tools like Swagger UI or ReDoc could not access it. Additionally, endpoints like healthchecks, readiness probes and metrics should rather be unauthenticated.

We write a middleware authenticate that is registered on the relevant routes. It is registered before the input validation such that a 401 (unauthenticated) or 403 (forbidden) are prioritized.

import express from "express";
import { authenticate } from "../../pre-request-handlers/authenticate";
import { validateInputs } from "../../pre-request-handlers/openapi";
import { getTodoController } from "./get-todo.controller";

export const todoRoute = express.Router({ mergeParams: true });

todoRoute.use(authenticate);
todoRoute.use(validateInputs);

todoRoute.get("/:id", getTodoController);
// ...

Configuration 🔗

Before we implement the authenticate middleware, we extend the configuration to contain authentication information.

export interface Config {
  environment: Environment;
  logLevel: Level;
  authentication: {
    enabled: boolean;
    jwksUrl: string;
  };
}

Locally, authentication should be turned off and the JWKS URL is irrelevant.

export function getLocalConfig(processVariables: ProcessVariables): Config {
  return {
    environment: "local",
    logLevel: processVariables.LOG_LEVEL ?? "debug",
    authentication: {
      enabled: false,
      jwksUrl: "",
    },
  };
}

On production, authentication is enabled and the JWKS URL is passed as an environment variable.

export function getProductionConfig(
  processVariables: ProcessVariables
): Config {
  return {
    environment: "production",
    logLevel: processVariables.LOG_LEVEL ?? "info",
    authentication: {
      enabled: true,
      jwksUrl:
        processVariables.JWKS_URL ??
        "<JWKS_URL> needs to be set in production environment",
    },
  };
}

Authentication middleware 🔗

The middleware authenticate resides in the file src/pre-request-handlers/authenticate.ts. It defines a jwksClient that retrieves and caches the signing keys from the external authentication service.

const jwksClient = JwksRsa({
  jwksUri: config.authentication.jwksUrl,
});

For JWT and JSON web key set (JWKS), we use the libraries jsonwebtoken and jwks-rsa.

npm install --save jsonwebtoken jwks-rsa
npm install --save-dev @types/jsonwebtoken

The middleware function authenticate consists of four steps executed in sequence.

export async function authenticate(
  request: Request,
  response: Response,
  next: NextFunction
): Promise<unknown> {
  try {
    // 1. Extract JWT from header and return 401 on failure
    // 2. Decode JWT and return 403 on failure
    // 3. If auth is enabled, then verify JWT and return 403 on failure
    // 4. Make decoded JWT payload accessible to controllers
    next();
  } catch (error) {
    next(error);
  }
}

The first step extracts the JWT from the header Authorization and removes the Bearer prefix. If such a header does not exist, then the user is unauthorized.

const encodedToken =
  request.headers.authorization?.replace("Bearer ", "") || "";
if (!encodedToken) {
  return response.status(401).end();
}

The second step decodes the JWT. It does not verify the signature, hence it only fails if the JWT is wrongly structured. In that case, the user is forbidden to access the resource.

const decodedToken = jwt.decode(encodedToken, { complete: true });
if (!decodedToken) {
  return response.status(403).end();
}

The third step verifies the JWT against the public signing key of the authentication service. This step should only be performed if authentication is enabled. If the JWT can't be verified, then the user is forbidden to access the resource.

if (config.authentication.enabled) {
  try {
    const signingKey = await jwksClient.getSigningKey(decodedToken.header.kid);
    jwt.verify(encodedToken, signingKey.getPublicKey(), {
      algorithms: ["RS256"],
    });
  } catch {
    return response.status(403).end();
  }
}

Why only disabling the verification? The payload of a JWT is also used for authorization. If no JWT is sent locally, then the controllers can't inspect the payload to decide if an admin or regular user accesses the endpoint. That's why we expect a JWT locally but don't check for expiration and validity against the signing key. A set of expired JWTs with invalid signature can be stored in version control and used by engineers.

The fourth step assigns the decoded JWT payload to the object response.locals. This way, the controllers can access it to make authorization decisions.

response.locals.token = decodedToken.payload;

Testing the authentication middleware 🔗

For the tests of controllers, we introduced a helper function server that spins up an HTTP server. For the testing of middleware, we create a helper function mockControllerInputs that allows us to test the middleware in isolation.

import express, { Express, NextFunction, Request, Response } from "express";
import { sendErrorResponse } from "./error-handling/error-handler";
import { logRequest } from "./pre-request-handlers/log-request";
import { validateInputs } from "./pre-request-handlers/openapi";

export function mockControllerInputs(request: Partial<Request> = {}): {
  request: Request;
  response: Response;
  next: NextFunction;
} {
  return {
    request: {
      log: {
        error: jest.fn().mockImplementation(),
      },
      params: {},
      ...request,
    } as unknown as Request,
    response: {
      locals: {},
      status: jest.fn().mockReturnThis(),
      sendStatus: jest.fn(),
      send: jest.fn(),
      end: jest.fn().mockReturnThis(),
    } as unknown as Response,
    next: jest.fn() as NextFunction,
  };
}

The tests of the authentication middleware mock the configuration and the jwks-rsa library.

jest.mock("../configuration/config", () => ({
  config: {
    authentication: {
      enabled: true,
      jwksUrl: "https://some-url.com",
    },
  },
}));

jest.mock("jwks-rsa", () => () => ({
  getSigningKey: () =>
    Promise.resolve({
      getPublicKey: () => `-----BEGIN RSA PUBLIC KEY-----
      abc
      -----END RSA PUBLIC KEY-----`,
    }),
}));

We create a stub JWT.

const jwtToken: jwt.Jwt = {
  payload: {
    id: "42",
    username: "adm@company.org",
  },
  header: {
    kid: "kid",
    alg: "RS256",
  },
  signature: "",
};

In the individual tests, we can adjust the return values of decode and verify and define the expectations.

it("returns a 401 if no authorization header is present", async () => {
  const { request, response, next } = mockControllerInputs({
    headers: {},
  });
  jest.spyOn(jwt, "decode").mockReturnValue(jwtToken);
  jest.spyOn(jwt, "verify").mockReturnValue();
  await authenticate(request, response, next);
  expect(response.status).toHaveBeenCalledWith(401);
  expect(next).not.toHaveBeenCalled();
});

The full code examples on GitHub contains three more tests of the authentication middleware.

Local exploration testing 🔗

Locally, authentication should not create a barrier for engineers. Since authentication verification is turned off locally, also expired and invalid JWT are accepted. Therefore, such a JWT can be stored in a shell script local.sh exporting it to an environment variable TOKEN.

export TOKEN=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c

Note: You need to call this script either with . ./local.sh or source ./local.sh for the variable $TOKEN to be accessible in the following commands.

Then, a request can be made against the running server npm start.

http http://localhost:3000/todos Authorization:"Bearer $TOKEN"
HTTP/1.1 200 OK
Connection: keep-alive
Content-Length: 2
Content-Type: application/json; charset=utf-8
Date: Mon, 07 Mar 2022 20:33:18 GMT
ETag: W/"2-l9Fw4VUO7kr8CvBlt4zaMCqXZ0w"
Keep-Alive: timeout=5
X-Powered-By: Express

[]

Suppressing token logging 🔗

When expecting the request logging of the server, you might have noticed that the JWT is logged. On the production server, a JWT must not be logged. This is to protect personal identifiable information (PII) as well as to prevent the unauthorized usage of that JWT.

{
  "authorization": "Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c"
}

To prevent this, we can add a custom serializer for requests.

import expressPino from "express-pino-logger";
import { config } from "../configuration/config";

export const logRequest = (enabled: boolean) =>
  expressPino({
    level: config.logLevel,
    enabled,
    serializers: {
      req: (request) => ({
        method: request.method,
        url: request.url,
      }),
    },
  });

This reduces the log output and does not log the headers.

INFO: request completed
    req: {
      "method": "GET",
      "url": "/todos"
    }
    res: {
      "statusCode": 200,
      "headers": {
        "x-powered-by": "Express",
        "content-type": "application/json; charset=utf-8",
        "content-length": "2",
        "etag": "W/\"2-l9Fw4VUO7kr8CvBlt4zaMCqXZ0w\""
      }
    }
    responseTime: 8
    v: 1

Conclusion 🔗

We configured JWT authentication with a JWKS leading to 401 and 403 responses. Locally, expired and invalid tokens are accepted, simplifying the life for engineers.

Why I dislike test-driven development

2022-04-04T00:00:00Z

I dislike test-driven development. There are better alternatives. A plea for writing tests later.

I think that with every tool and process, there are use cases where they are more beneficial and use cases where they are less beneficial. While test-driven development as a process is beneficial in some use cases, I found it to be counterproductive in the majority of my use cases.

Why test-driven development 🔗

The main goal of test-driven development are shorter feedback loops. I absolutely agree that this is a goal worth pursuing. Coding is a process of continuous learning: About the problem you are trying to solve and the solution you are developing. The shorter the feedback loops, the faster the learning.

Test-driven development consists of three phases: Writing a test, writing the code to satisfy the test expectation and refactoring the code.

My main issue with test-driven development: The additional code of the tests slows down the feedback loops in the refactoring phase instead of accelerating them.

Reasons for slow feedback loops 🔗

In the following sections, let me explain to you where the feedback loops of test-driven development are too slow and what's faster instead: Utilizing the type system and writing tests later.

Signature refactoring 🔗

One of the phases with test-driven development, is the refactoring phase. After a test and the code satisfying the test have been created, the code is improved to be more maintainable while still satisfying the test expectation.

There are two scenarios of a refactoring. If the refactoring does not include a function signature change of the code, then the code can independently be changed and the test can verify that the code is still solving the problem correctly. However, if the refactoring does include a function signature change of the code, then also the test case needs to be changed. This scenario leaves you with at least two places that require changes, which decreases the speed of the feedback loop.

function findBookNamesOfAuthorsWithMultipleBooks(
    books: Book[],
    Record<BookId, Author>
  ): BookName[] {
  // ...
}

it('returns no book name if there are no books', () => {
  expect(
    findBookNamesOfAuthorsWithMultipleBooks(
      [],
      { '1': 'Goethe' }
    )
  ).toEqual([]);
})

Now you might argue that while it is slower, in the case of no test case there would not be a feedback loop at all. However, there still exists a feedback loop: The type system.

Since the code change includes a function signature change, it also includes changes in the types. As an example, consider the function findBookNamesOfAuthorsWithMultipleBooks being passed two arguments Book[] and Record<BookId, Author>.

If I learn during coding that in multiple places we need a book together with their author information, then I might decide to pass a single argument AuthoredBook[] and aggregate the two data structures before passing them to the function.

function findBookNamesOfAuthorsWithMultipleBooks(
  authoredBooks: AuthoredBook[]
): BookName[] {
  // ...
}

This signature change causes the compiler to complain at the call side of this function and within the function body itself. Which leads me to react to that fast feedback and make the type system work again.

Note: You might argue that such signature changes are rare. However, the intention for having fast feedback loops is faster learning and hence faster adaption. If such changes are indeed rare, then adaption due to learning and hence fast feedback loops are not necessary at all.

With utilizing the type system, we avoid the additional refactoring in the test cases. Also, a type-system feedback loop is faster than a test feedback loop.

Implementation refactoring 🔗

Another form of refactoring is breaking down one function into two or more smaller functions. In this case, the signature of the function remains the same and hence the test case does not need to be changed.

As an example, consider a function synchronizeBooks that load books from an API and saves them to the database.

async function synchronizeBooks() {
  const books = await http
    .get<Book>("https://my.url/books")
    .then((response) => response.payload ?? []);
  for (const book of books) {
    await saveBookToDatabase(book);
  }
}

The level of detail of the loading does not match the level of detail of the save. Therefore, we decide to extract the loading into a function loadBooksFromAPI.

function loadBooksFromAPI() {
  return http
    .get<Book>("https://my.url/books")
    .then((response) => response.payload ?? []);
}

async function synchronizeBooks() {
  const books = await loadBooksFromAPI();
  for (const book of books) {
    await saveBookToDatabase(book);
  }
}

If there is a test for synchronizeBooks, we would not need to change it since the signature has not changed. However, the reason for breaking down a function is that you want to split the complexity. By not touching the tests, the complexity of the single test remains. You continue to test the parted functions as a single unit.

Continuing on that path every test that started as a test of a small unit eventually becomes an integration (or big unit) test. Therefore, not splitting the tests as well as the functions leads to slower feedback loops down the line with many integration tests and fewer small unit tests. For that reason, both not changing the tests and changing the tests lead to slower feedback loops. Hence, I argue, just write the tests once the code is more stable later on.

Unnecessary tests due to improper typing 🔗

Another source of slow feedback loops within test-driven development are unnecessary tests. These tests verify a property that could also have been verified with improved types or an improved type system.

As an example, imagine not using strict mode in TypeScript or, well, using Java.

A function count(book: Book[]): number can be called with count(null). You could write a test for the null behavior. However, if this would be handled by the type system, then the same benefits applied as before.

Let's take a look at a more sophisticated example where the type system can replace a test case without a change in the compiler or a switch to a different language.

The function calculate represents a calculator. It is passed a mathematical operation, either addition or negation. An addition sums two values. A negation negates a single value.

interface MathOperation {
  type: "add" | "negate";
  value1: number;
  value2?: number;
}

function calculate(op: MathOperation): number {
  switch (op.type) {
    case "add":
      return op.value1 + op.value2;
    case "negate":
      return -op.value1;
  }
}

The mathematical operation is imprecisely represented. It covers all valid instances like { type: "add", value1: 5, value2: 7 } and { type: "negate", value1: 11 }. But it also covers invalid instances like { type: "add", value1: 5 } and { type: "negate", value1: 11, value2: 13 }. Because of that, both the code and the tests need to account for that.

it("returns the negation of the first value", () => {
  expect(calculate({ type: "negate", value1: 11 })).toEqual(-11);
});

it("ignores the second value", () => {
  expect(calculate({ type: "negate", value1: 11, value2: 13 })).toEqual(-11);
});

If we represent the types properly, then the second test case is not required anymore. It would even cause compiler errors.

interface Addition {
  type: "add";
  leftValue: number;
  rightValue: number;
}

interface Negation {
  type: "negate";
  value: number;
}

type MathOperation = Addition | Negation;

function calculate(op: MathOperation): number {
  switch (op.type) {
    case "add":
      return op.leftValue + op.rightValue;
    case "negate":
      return -op.value;
  }
}

We represent a math operation correctly as either an addition or a negation. An addition always requires exactly two values. A negation always requires exactly one value. The second test is not required anymore.

But what about the first test?

Since the types are now properly represented and more restrictive, the function body is also able to do less and therefore less wrong. Given the operation is a negation, the case block only has access to the parameter op of the restricted type Negation. The value op.type is a string and can therefore not be returned. The value op.value is a number and can be returned. The value op.value is the only realistic return value that the function could return apart from the correct return value -op.value. Everything else would be a visible bug in the code itself without the need of a test case.

Still, even for this one scenario of returning op.value a test case would not be beneficial. The reason for this is that the test case would be of the same complexity as the code itself.

Note: Great function signatures hide complex implementations. Great tests are as simple as the function signature and less complex than the implementation.

it("returns the negation of the value", () => {
  expect(calculate({ type: "negate", value: 11 })).toEqual(11);
});

This test case is wrongly implemented. The operator - is missing in the test case. The bug in the test is as likely to occur as the bug in the code. If I still write the test, then it would only be a double check instead of a test. But by that reasoning one could also require that a function is always required to be implemented twice instead of requiring test cases. In great tests bugs are easier to spot than in the code.

Unnecessary tests due to unnecessarily complex code 🔗

Let's take a look at an example where the types are precise but the implementation is unnecessarily complex.

interface Book {
  cost: number;
}

function costs(books: Book[]): number[] {
  let costs = [];
  for (let i = 0; i < books.length; ++i) {
    costs.push(books[i].cost);
  }
  return costs;
}

The expression i < books.length vs. i <= books.length is a common off-by-one bug. Also, zero-based arrays i = 0 vs. i = 1 can be a cause for bugs. We could write a test for it.

it("sums up the total costs of all books", () => {
  expect(totalCosts([{ cost: 3 }, { cost: 5 }])).toEqual([3, 5]);
});

Instead, it is a better option to refactor the implementation towards a more declarative style. Such a style would prevent off-by-one errors to occur in the first place.

function costs(books: Book[]): number[] {
  return books.map((book) => book.cost);
}

Without index handling in the implementation, there is no need to test the index handling in the tests. With the more declarative implementation, we again reached the point that the test would be as complex as the code.

Where I like test-driven development 🔗

For every tool, there are cases where they are more beneficial and cases where they are less beneficial. For test-driven development, there are also cases where I like to use it.

Reproducing bugs 🔗

I found test-driven development to be particularly beneficial for reproducing bugs.

Assuming a bug occurs in production and I only have a bug report and ideally some log traces. Then first writing a test gives me three advantages:

I confirm that my suspicion is indeed the source of the bug.
I know when the bug is fixed.
The bug is less likely to reappear in the future.

Coding against specification 🔗

Sometimes there are testable properties that you know in advance.

You want a GET endpoint to return the same result after two immediate calls.
You want a merge of two instances of an unordered data structure to be commutative.
You want the sorting of a returned list to be deterministic.

Tests of such properties verify a usually more stable signature of a bigger unit. It can make sense to write these tests in advance.

Conclusion 🔗

Instead of test-driven development I like to refer to type-driven development. Types are a more expressive mechanism with faster feedback loops than tests. Types make the refactoring phase faster than tests. Still, I find test-driven development to be helpful for coding against specification and reproducing bugs.

Resolve a list of promises synchronously

2022-04-15T00:00:00Z

The JavaScript function Promise.all executes promises in parallel. How to write a function to execute and resolve the results synchronously instead.

TLDR; 🔗

In case you just want the code (no judgment!), here it is.

async function allSynchronously<T>(
  resolvables: (() => Promise<T>)[]
): Promise<T[]> {
  const results = [];
  for (const resolvable of resolvables) {
    results.push(await resolvable());
  }
  return results;
}

Long version 🔗

Let's assume we want to write a function getBooks(ids: string[]): Promise<Book[]>. For the list of IDs ids: string[] we want to resolve the associated books Book[]. There is an endpoint GET /books/:id. The function getBook(id: string): Promise<Book> makes that endpoint accessible.

interface Book {
  id: string;
}

function getBook(id: string): Promise<Book> {
  // GET /books/:id
}

An implementation of getBooks could use Promise.all to return a promise of books that resolves once the last endpoint call to GET /books/:id returns.

function getBooks(ids: string[]): Promise<Book[]> {
  return Promise.all(ids.map(getBook));
}

The function Promise.all runs in parallel. This makes it fast. However, there are situations where you want it to run synchronously, one after the other. The API could be rate-limited or you might want to distribute the load.

For these cases, we can write a synchronous version of Promise.all. Such a function can not have the signature allSynchronously<T>(promises: Promise<T>[]): Promise<T[]>. If the function accepted a list of promises as an argument, then at that point in time these promises would already run in parallel. Therefore, the signature accepts a list of zero-argument functions allSynchronously<T>(promises: (() => Promise<T>)[]): Promise<T[]>

async function allSynchronously<T>(
  resolvables: (() => Promise<T>)[]
): Promise<T[]> {
  const results = [];
  for (const resolvable of resolvables) {
    results.push(await resolvable());
  }
  return results;
}

Now, an implementation of getBooks can use this function allSynchronously to run the requests synchronously instead of in parallel.

function getBooks(ids: string[]): Promise<Book[]> {
  return allSynchronously(ids.map((id) => () => getBook(id)));
}

Connecting a Node.js app with a Postgres database

2022-04-16T00:00:00Z

We are setting up a local Postgres database with docker-compose for our Node.js service such that we can persist data.

A database allows us to persist data beyond the lifetime of the application. Postgres is an SQL database which we can access with the Node.js library Knex and the relevant Postgres driver. You can read in a different post why TypeScript is awesome for SQL queries.

Local docker database 🔗

First, we create a file docker-compose.yml to manage the local Postgres database.

version: "3.1"
services:
  database:
    image: "postgres:14"
    environment:
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: secret
      POSTGRES_DB: postgres
    ports:
      - "5432:5432"
    volumes:
      - database-data:/var/lib/postgresql/data/
volumes:
  database-data:

The file docker-compose.yml specifies that a Postgres database should be running. It has a fixed user, database name and password. The database port 5432 is exposed on localhost:5432 such that the Node application npm start can access it. The data is persisted on a named docker volume database-data.

With the docker-compose configuration specified, the docker-compose tool can be run.

Note The tool docker-compose needs to be natively installed on your machine.

docker-compose up -d

The running container can be inspected.

docker-compose ps

The container can be shut down.

docker-compose down

Database configuration 🔗

We need to install both the Node.js packages knex and pg. The knex package contains the TypeScript types for Knex. The pg package requires the package @types/pg package for the types.

npm install --save knex pg
npm install --save-dev @types/pg

The application needs to know where it can find the database and how to access it. We capture this in the application configuration.

import { Knex } from "knex";

export interface Config {
  environment: Environment;
  // ...
  database: Knex.PgConnectionConfig;
}

export interface ProcessVariables {
  ENV?: Environment;
  // ...
  DATABASE_URL?: string;
}

The process variables define a variable for the connection string. This string has the format postgresql://username:password@host:port/dbname. Alternatively, the host, port, user, password and database name could be passed independently via environment variables.

The configuration contains an object database that defines the connection configuration expected by Postgres.

Production configuration 🔗

In the production configuration, we assign the database object from the environment variable DATABASE_URL.

import { Config, ProcessVariables } from "../config.type";

export function getProductionConfig(
  processVariables: ProcessVariables
): Config {
  return {
    environment: "production",
    // ...
    database: {
      connectionString: processVariables.DATABASE_URL,
      ssl: true,
    },
  };
}

We can add a test to verify the correct assignment.

import { Knex } from "knex";
import { getProductionConfig } from "./get-production.config";

describe("the production configuration", () => {
  // ...

  it("reads the database configuration from the environment", () => {
    expect(
      getProductionConfig({
        DATABASE_URL: "postgresql://username:password@host:port/dbname",
      })
    ).toHaveProperty("database", {
      connectionString: "postgresql://username:password@host:port/dbname",
      ssl: true,
    } as Knex.PgConnectionConfig);
  });
});

Local configuration 🔗

The local configuration consists of constant values which are the counterpart to the values defined in the docker-compose configuration docker-compose.yml.

import { Config, ProcessVariables } from "../config.type";

export function getLocalConfig(processVariables: ProcessVariables): Config {
  return {
    environment: "local",
    // ...
    database: {
      user: "postgres",
      host: "localhost",
      database: "postgres",
      port: 5432,
      password: "secret",
      ssl: false,
    },
  };
}

Connection pool 🔗

With the configuration set up, we create a constant db that holds the global database connection pool of the application. By default, it holds a handful of connections. You can provide more configuration to change that.

import { Knex } from "knex";
import { config } from "../configuration/config";

export const db = require("knex")({
  client: "pg",
  connection: config.database,
}) as Knex;

Accessing the database 🔗

An endpoint GET /health allows us from the outside to verify if the application can access the database. We send a select query to one of Postgres already existing tables to see if it succeeds. If the database can not be accessed, the query would throw an error and lead to a 500 response.

import { NextFunction, Request, Response } from "express";
import { db } from "../database/db";

export async function healthController(
  _request: Request,
  response: Response,
  next: NextFunction
): Promise<void> {
  try {
    await db.from("information_schema.tables").select();
    response.sendStatus(204);
  } catch (error) {
    next(error);
  }
}

We register the health check in the application app.ts.

import express from "express";
import { healthController } from "./controllers/health.controller";

const app = express();

// ...
app.use("/health", healthController);
// ...

export default app;

Now, calling the endpoint yields a success 204 if the database is accessible.

http http://localhost:3000/health

HTTP/1.1 204 No Content
Connection: keep-alive
Date: Sat, 16 Apr 2022 14:40:26 GMT
ETag: W/"a-bAsFyilMr4Ra1hIU5PyoyFRunpI"
Keep-Alive: timeout=5
X-Powered-By: Express

If the database is inaccessible (forgot to run docker-compose up -d?), then it returns an error 500.

http http://localhost:3000/health

HTTP/1.1 500 Internal Server Error
Connection: keep-alive
Content-Length: 49
Content-Type: application/json; charset=utf-8
Date: Sat, 16 Apr 2022 14:41:46 GMT
ETag: W/"31-N3CMxmHVH2QRVY7BhlpI3H4+usU"
Keep-Alive: timeout=5
X-Powered-By: Express

{
    "message": "connect ECONNREFUSED 127.0.0.1:5432"
}

Conclusion 🔗

We have setup a local Postgres database. It is accessible for the local application and can be configured via environment variables for production.

Knex.js schema migrations with Postgres in Node.js

2022-04-20T00:00:00Z

We create the first database table with Knex schema migrations and make sure that we can reset the database state locally.

We want to be able to continuously evolve the database schema with fluent migrations. The library Knex bases migrations on a directory of TypeScript files. Each file describes one migration including a rollback step. Knex manages the execution state within a dedicated database table migrations.

For Knex to find the database locally and on production, we provide it with the connection settings in a file ./src/database/knexfile.ts.

Note I position the Knexfile at this position ./src/database/knexfile.ts instead of top level ./. This way, TypeScript considers it part of the code base and it can import the configuration ./src/configuration/config and type definitions.

import { Knex } from "knex";
import { config } from "../configuration/config";

module.exports = {
  client: "pg",
  connection: config.database,
  migrations: {
    directory: "./migrations",
  },
} as Knex.Config;

This leads to the following directory structure.

src
`-- database
    |-- db.ts
    |-- knexfile.ts
    `-- migrations
        |-- 1-<name>.ts
        |-- 2-<name>.ts
        `-- <n>-<name>.ts

Migrations 🔗

Knex knows now where the database is. We can write the first migration and want the todo entities to be persisted. Hence, we create a table todos.

import { Knex } from "knex";

export async function up(knex: Knex): Promise<void> {
  await knex.schema.createTable("todos", (table) => {
    table.uuid("id").primary();
    table.string("name").notNullable();
    table.string("assignee").notNullable();
    table.date("dueDate").notNullable();
  });
}

export async function down(knex: Knex): Promise<void> {
  await knex.schema.dropTable("todos");
}

We can run the migrations with a shell command.

npx knex --knexfile ./src/database/knexfile.ts migrate:latest

Knex executes all migrations that have not been successfully executed yet. If we add a second migration script later, then Knex inspects the migrations database table and only executes the second script.

Note On production, this migration command should be run on each release. If you run multiple replicas of the application, the migration should only run once instead of per replica. The specific configuration depends on your runtime technology.

Seeds 🔗

Locally, it is helpful to populate the database with example data. This makes it easier to experiment with the application. To achieve this, Knex offers seed scripts.

In the Knexfile ./src/database/knexfile.ts, we tell Knex where to find the seed scripts.

import { Knex } from "knex";
import { config } from "../configuration/config";

module.exports = {
  client: "pg",
  connection: config.database,
  migrations: {
    directory: "./migrations",
  },
  seeds: {
    directory: "./seeds",
  },
} as Knex.Config;

In a file ./src/database/seeds/todos.ts, we insert three todos into the empty table.

import { Knex } from "knex";
import { Todo } from "../../controllers/todos/todo.type";

const todos: Todo[] = [
  {
    id: "142e8446-f5ba-4813-bfb5-f3192a37f1bf",
    name: "Create Todo",
    assignee: "Naomi",
    dueDate: "2025-01-05",
  },
  {
    id: "bf6091cc-69f8-4a9e-93ed-0294c3a8ac2f",
    name: "Delete Todo",
    assignee: "Peter",
    dueDate: "2029-06-11",
  },
  {
    id: "6e5bad4c-7f1c-4ab4-9361-c6bd697b4257",
    name: "Change Todo",
    assignee: "Sheila",
    dueDate: "2032-11-27",
  },
];

export async function seed(knex: Knex): Promise<void> {
  await knex("todos").truncate();
  await knex("todos").insert(todos);
}

Similar to the migrations, we can execute the seed scripts.

npx knex --knexfile ./src/database/knexfile.ts seed:run

Utility scripts 🔗

The commands with the custom Knexfile ./src/database/knexfile.ts are rather long. Therefore, we create scripts in the package.json to migrate and seed. Additionally, we provide scripts to reset the local database to a default state.

There are three steps in a reset of the local database:

Wipe: Deleting the existing database docker volume
Migrate: Create the tables
Seed: Populate example data

{
  // ...
  "scripts": {
    // ...
    "db:wipe": "docker-compose down; docker volume rm nodemicroservicearchitecture_database-data; docker-compose up -d",
    "db:migrate": "npx knex --knexfile ./src/database/knexfile.ts migrate:latest",
    "db:seed": "npx knex --knexfile ./src/database/knexfile.ts seed:run",
    "db:reset": "npm run db:wipe && npm run db:migrate && npm run db:seed"
  }
  // ...
}

The script db:wipe removes the docker volume and can only do so if the Postgres database is shut down with docker-compose down. Afterwards it spins it up again with docker-compose up -d.

Note The volume name nodemicroservicearchitecture_database-data is assembled by docker-compose and might change between versions. You might need to adjust it on a docker upgrade.

Accessing the new table 🔗

The new table todos is created with the first migration. We can change the health endpoint to access this table and verify that it was created.

import { NextFunction, Request, Response } from "express";
import { db } from "../database/db";

export async function healthController(
  _request: Request,
  response: Response,
  next: NextFunction
): Promise<void> {
  try {
    await db.from("todos").select("id").limit(1);
    response.sendStatus(204);
  } catch (error) {
    next(error);
  }
}

Conclusion 🔗

We have setup the first Postgres database table with Knex schema migrations. New migrations can be added and can be executed with scripts from the package.json. The local setup includes example data and allows for a one-step database reset.

Creating a Node.js DAO with Knex.js

2022-04-24T00:00:00Z

We use Knex to build a data access module (DAO) to execute SQL queries to create, update, delete and retrieve todos from the database.

So far, we stored the todos in-memory in a variable todos in a file todo.storage.

import { TodoMap } from "./todo.type";

export const todos: TodoMap = {};

The data access module todo.dao acted against this variable todos.

import { v4 as uuid } from "uuid";
import { todos } from "./todo.storage";
import { Todo, TodoId } from "./todo.type";

export async function getTodo(id: TodoId): Promise<Todo | undefined> {
  return todos[id];
}

export async function getTodos(): Promise<Todo[]> {
  return Object.values(todos);
}

export async function createTodo(todo: Todo): Promise<Todo> {
  const id = uuid();
  const createdTodo = {
    ...todo,
    id,
  };
  todos[id] = createdTodo;
  return createdTodo;
}

export async function updateTodo(
  id: TodoId,
  todo: Todo
): Promise<Todo | "NotFound"> {
  if (!(id in todo)) {
    return "NotFound";
  }
  const updatedTodo = {
    ...todo,
    id,
  };
  todos[id] = updatedTodo;
  return updatedTodo;
}

export async function deleteTodo(id: TodoId): Promise<void> {
  delete todos[id];
}

Now, we exchange the implementation step-by-step to execute SQL queries against the database instead.

GET /todos 🔗

The endpoint GET /todos returns a list of all todos in the database. It utilizes the DAO function getTodos.

import { db } from "../../database/db";
import { Todo } from "./todo.type";

export function getTodos(): Promise<Todo[]> {
  return db.table<Todo>("todos").select("*");
}

The query uses the db connection pool that we created in the previous post. It chooses to act on the table with the name todos and selects every row and every column.

With httpie or curl, we can check the retrieval against the running application npm start and the running database docker-compose up -d.

Note Initialize the environment variable $TOKEN by running the script . ./local.sh.

http http://localhost:3000/todos Authorization:"Bearer $TOKEN"

HTTP/1.1 200 OK
Connection: keep-alive
Content-Length: 329
Content-Type: application/json; charset=utf-8
Date: Sun, 17 Apr 2022 17:01:27 GMT
ETag: W/"149-ffBiGJ6cbX5lQsi4xlpAB7ZE9T8"
Keep-Alive: timeout=5
X-Powered-By: Express

[
    {
        "assignee": "Naomi",
        "dueDate": "2025-01-05",
        "id": "142e8446-f5ba-4813-bfb5-f3192a37f1bf",
        "name": "Create Todo"
    },
    {
        "assignee": "Peter",
        "dueDate": "2029-06-11",
        "id": "bf6091cc-69f8-4a9e-93ed-0294c3a8ac2f",
        "name": "Delete Todo"
    },
    {
        "assignee": "Sheila",
        "dueDate": "2032-11-27",
        "id": "6e5bad4c-7f1c-4ab4-9361-c6bd697b4257",
        "name": "Change Todo"
    }
]

The query returns the three todos that are initialized with the local seeding ./database/seeds/todos.ts created in the previous post.

GET /todos/:id 🔗

The endpoint GET /todos/:id returns a single specific todo using the DAO function getTodo.

import { db } from "../../database/db";
import { Todo, TodoId } from "./todo.type";

export function getTodo(id: TodoId): Promise<Todo | undefined> {
  return db.table<Todo>("todos").where("id", id).first<Todo>();
}

From the table todos we select the rows with that specific ID. Since the ID is defined as a primary key in the migrations ./database/migrations, we know that there is only one row with that ID. Therefore, we define to retrieve the first value or undefined and explicitly type the result to be of the type Todo.

If we query for one of the existing todo IDs, we get the corresponding todo with all fields.

http http://localhost:3000/todos/6e5bad4c-7f1c-4ab4-9361-c6bd697b4257 Authorization:"Bearer $TOKEN"

HTTP/1.1 200 OK
Connection: keep-alive
Content-Length: 109
Content-Type: application/json; charset=utf-8
Date: Sun, 17 Apr 2022 17:02:27 GMT
ETag: W/"6d-Vx+CUC5e2HeHaKEJfis7cAgay/0"
Keep-Alive: timeout=5
X-Powered-By: Express

{
    "assignee": "Sheila",
    "dueDate": "2032-11-27",
    "id": "6e5bad4c-7f1c-4ab4-9361-c6bd697b4257",
    "name": "Change Todo"
}

Similarly, we can query for a non-existing ID. The controller ./controllers/todos/get-todo.controller translates the undefined from the DAO into a not found error 404.

http http://localhost:3000/todos/747c58f7-842e-4782-8897-9d81ad640f27 Authorization:"Bearer $TOKEN"

HTTP/1.1 404 Not Found
Connection: keep-alive
Content-Length: 9
Content-Type: text/plain; charset=utf-8
Date: Sun, 17 Apr 2022 17:36:33 GMT
ETag: W/"9-0gXL1ngzMqISxa6S1zx3F4wtLyg"
Keep-Alive: timeout=5
X-Powered-By: Express

Not Found

POST /todos 🔗

The endpoint POST /todos creates a new todo using the DAO function createTodo. When a user creates a todo, we need to ignore an incoming ID and generate a unique ID. The todo with this generated ID can then be both inserted into the database and returned to the user.

import { v4 as uuid } from "uuid";
import { db } from "../../database/db";
import { Todo } from "./todo.type";

export async function createTodo(todo: Todo): Promise<Todo> {
  const todoWithId = {
    ...todo,
    id: uuid(),
  };
  await db.table<Todo>("todos").insert(todoWithId);
  return todoWithId;
}

On a request to the endpoint POST /todos, we can observe the todo with the ID being returned.

http POST http://localhost:3000/todos Authorization:"Bearer $TOKEN" name=New assignee=Francis dueDate=2045-12-03

HTTP/1.1 200 OK
Connection: keep-alive
Content-Length: 102
Content-Type: application/json; charset=utf-8
Date: Sun, 17 Apr 2022 17:04:23 GMT
ETag: W/"66-i4v1yOpk5ZXd9In2M+wglJR27Tg"
Keep-Alive: timeout=5
X-Powered-By: Express

{
    "assignee": "Francis",
    "dueDate": "2045-12-03",
    "id": "7a071521-62f7-4fce-b99f-9bf7c5c21716",
    "name": "New"
}

PUT /todos/:id 🔗

On a request to the endpoint PUT /todos/:id, an existing todo is supposed to be updated using the DAO function updateTodo. The relevant todo is identified with the ID from the route. An additional ID that might be passed in the body is ignored.

import { db } from "../../database/db";
import { Todo, TodoId } from "./todo.type";

export async function updateTodo(
  id: TodoId,
  todo: Todo
): Promise<Todo | "NotFound"> {
  const todoWithId = {
    ...todo,
    id,
  };
  const changedRowCount = await db
    .table<Todo>("todos")
    .where("id", id)
    .update(todoWithId);
  return changedRowCount === 0 ? "NotFound" : todoWithId;
}

If a user tries to update a non-existing todo, then we try the update and inspect if a row was changed.

We can check the endpoint with an existing ID and update the due date. The endpoint returns the todo with the updated field.

http PUT http://localhost:3000/todos/7a071521-62f7-4fce-b99f-9bf7c5c21716 Authorization:"Bearer $TOKEN" name=New assignee=Francis dueDate=2046-12-03

HTTP/1.1 200 OK
Connection: keep-alive
Content-Length: 102
Content-Type: application/json; charset=utf-8
Date: Sun, 17 Apr 2022 17:12:47 GMT
ETag: W/"66-Yt1SoQRe6m7HzcEOLvlGf/vVo/s"
Keep-Alive: timeout=5
X-Powered-By: Express

{
    "assignee": "Francis",
    "dueDate": "2046-12-03",
    "id": "7a071521-62f7-4fce-b99f-9bf7c5c21716",
    "name": "New"
}

For a non-existing ID, the API returns a not found error 404.

http PUT http://localhost:3000/todos/747c58f7-842e-4782-8897-9d81ad640f27 Authorization:"Bearer $TOKEN" name=New assignee=Francis dueDate=2046-12-03

HTTP/1.1 404 Not Found
Connection: keep-alive
Content-Length: 9
Content-Type: text/plain; charset=utf-8
Date: Sun, 17 Apr 2022 18:06:34 GMT
ETag: W/"9-0gXL1ngzMqISxa6S1zx3F4wtLyg"
Keep-Alive: timeout=5
X-Powered-By: Express

Not Found

DELETE /todos/:id 🔗

Last but not least, the endpoint DELETE /todos/:id deletes an existing todo using the DAO function deleteTodo.

import { db } from "../../database/db";
import { Todo, TodoId } from "./todo.type";

export async function deleteTodo(id: TodoId): Promise<"Ok" | "NotFound"> {
  const deletedRowCount = await db
    .table<Todo>("todos")
    .where("id", id)
    .delete();
  return deletedRowCount === 0 ? "NotFound" : "Ok";
}

If a todo with this ID does not exist, then the DAO reports it to be not found. The API still returns a success 204, since the user intent of the absence of the todo with this ID is achieved.

We can verify the deletion with httpie or curl.

http DELETE http://localhost:3000/todos/7a071521-62f7-4fce-b99f-9bf7c5c21716 Authorization:"Bearer $TOKEN"

HTTP/1.1 204 No Content
Connection: keep-alive
Date: Sun, 17 Apr 2022 17:13:32 GMT
ETag: W/"a-bAsFyilMr4Ra1hIU5PyoyFRunpI"
Keep-Alive: timeout=5
X-Powered-By: Express

Conclusion 🔗

The data access module todo.dao provides five functions to retrieve, create, update and delete todos in the database.

Testing Knex SQL queries with Jest and testcontainers

2022-05-20T00:00:00Z

In the previous post, we created a module with queries to the Postgres database.

import { v4 as uuid } from "uuid";
import { db } from "../../database/db";
import { Todo, TodoId } from "./todo.type";

export function getTodo(id: TodoId): Promise<Todo | undefined> {
  return db.table<Todo>("todos").where("id", id).first<Todo>();
}

export function getTodos(): Promise<Todo[]> {
  return db.table<Todo>("todos").select("*");
}

export async function createTodo(todo: Omit<Todo, "id">): Promise<Todo> {
  const todoWithId = {
    ...todo,
    id: uuid(),
  };
  await db.table<Todo>("todos").insert(todoWithId);
  return todoWithId;
}

export async function updateTodo(
  id: TodoId,
  todo: Omit<Todo, "id">
): Promise<Todo | "NotFound"> {
  const todoWithId = {
    ...todo,
    id,
  };
  const changedRowCount = await db
    .table<Todo>("todos")
    .where("id", id)
    .update(todoWithId);
  return changedRowCount === 0 ? "NotFound" : todoWithId;
}

export async function deleteTodo(id: TodoId): Promise<"Ok" | "NotFound"> {
  const deletedRowCount = await db
    .table<Todo>("todos")
    .where("id", id)
    .delete();
  return deletedRowCount === 0 ? "NotFound" : "Ok";
}

In this post, we create test cases that check if the SQL queries executed by these methods work as expected and that they work with the database schema. We configure the test cases to automatically spawn a single Postgres database, make the test cases and the underlying application connect with it and tear it down once the test cases ran.

Available Options 🔗

Before we delve into the implementation, let's take a look at the options and their pros and cons.

In our setup, we have tests that require a database and we have tests that don't require a database. Ideally, the tests that don't require a database should be in the majority.

We have several options:

One database per test file: Each test file requiring a database sets up and tears down a dedicated database. Great isolation, but performance does not scale and lots of docker containers. On the bright side, if you run npm run test -- --watch and no database test is affected, then no database is spawned.
Single database, started on-demand: The first test file requiring a database sets it up. Better performance, worse isolation. Unclear how to share connection details, where and when to tear it down and database setup errors will appear in unexpected places.
Single database, always started: Before all test files, a single database is set up globally and teared down afterwards. Better performance, worse isolation. There is always a database spawned regardless if a database test is executed.

Note: These are the options if we don't separate database tests from other unit tests. I believe that it is best to not separate and instead to have a single command that runs all checks. This command runs identically locally and on CI. Any separation would lead to some checks running less frequently which undermines their purpose of catching defects early.

In this post, I present how to achieve the setup with a single database that is always started. I believe it is the best compromise between isolation and performance.

Installing Testcontainers 🔗

We use the Node.js package of testcontainers to spawn docker containers. There is an npm package that allows to use TypeScript code to spawn containers.

npm install --save-dev testcontainers

First test case 🔗

Before we configure the setup of the database, we create a first test case in a file src/controllers/todos/todo.dao.spec.ts. If we ask for the todos in an empty database, then an empty list of todos should be returned.

import { getTodos } from "./todo.dao";

describe("the todo dao", () => {
  it("retrieves no todos if the database is empty", async () => {
    await expect(getTodos()).resolves.toEqual([]);
  });
});

Right now it fails, since there is no database to connect to.

Note: If you directly want to see the complete working code, check out the GitHub repo. In this post, there are a lot of incremental changes to the files and the the snippets you see are still extended in later sections.

Global setup 🔗

Jest provides an option in the jest.config.js to run a global setup script before the execution of all tests. If you run npm run test, then it is run exactly once before all tests. If you run npm run test -- --watch, then it runs before all tests after each file system change trigger.

process.env = Object.assign(process.env, {
  DISABLE_LOGGER: "true",
});

module.exports = {
  // ...
  globalSetup: "<rootDir>/src/database/db-test.setup.ts",
};

In this script src/database/db-test.setup.ts, we spawn the database container.

function spawnDatabase(): Promise<StartedTestContainer> {
  return new GenericContainer("postgres:14")
    .withEnv("POSTGRES_USER", "postgres")
    .withEnv("POSTGRES_DB", "postgres")
    .withEnv("POSTGRES_PASSWORD", "secret")
    .withExposedPorts(5432)
    .withTmpFs({ "/temp_pgdata": "rw,noexec,nosuid,size=65536k" })
    .start();
}

module.exports = spawnDatabase;

We specify the container name and version postgres:14. This is the name and version how it is defined on DockerHub. We pass environment variables to the Postgres docker container. The standard port 5432 of a Postgres database is exposed with a random port on localhost.

Note: It's always a random port on localhost such that the tests are not depending on the port to be free. This is however also the source of complication to share this random port to the tests and application.

We also configure the docker container to use a temporary file system /temp_pgdata for the storage of the database data.

With this global setup, we have a database container running during the tests. However, since it is exposed at a random port on localhost, the tests and the application can not find it. The application still expects the database to be on port 5432.

npm run test

  ● the todo dao › retrieves no todos if the database is empty

    connect ECONNREFUSED 127.0.0.1:5432

Sharing this random port is tricky.

The Jest global setup does not have a return value that is accessible in the tests. It even runs in a separate node environment. That means that global variables written in the Jest global setup are not accessible in the tests.

To overcome this challenge, we utilize a workaround that also Puppeteer used: We write the database port to a temporary file and read it from there in the tests.

This workaround requires us to take multiple steps:

We need to write the port to a temporary file.
We need to introduce a different configuration that reads the database port from this file.
We need to configure Jest to start with that configuration.

Writing the database port to a temporary file 🔗

In the Jest global setup db-test.setup.ts, we create a second function shareDatabaseConfig that is executed after the spawning of the database.

import { GenericContainer, StartedTestContainer } from "testcontainers";
import { mkdir, writeFile } from "fs/promises";
import os from "os";
import path from "path";

function spawnDatabase(): Promise<StartedTestContainer> {
  // ...
}

async function shareDatabaseConfig(container: StartedTestContainer) {
  const variablesDir = path.join(
    os.tmpdir(),
    "jest_testcontainers_global_setup"
  );
  await mkdir(variablesDir, { recursive: true });
  await writeFile(
    path.join(variablesDir, "databasePort"),
    container.getMappedPort(5432).toString()
  );
}

async function setupDatabase(): Promise<void> {
  const container = await spawnDatabase();
  await shareDatabaseConfig(container);
}

module.exports = setupDatabase;

In this function shareDatabaseConfig, we first create a directory jest_testcontainers_global_setup in temporary directory of the operating system. Afterwards, we create a file jest_testcontainers_global_setup/databasePort and fill it with the randomly assigned port of the database.

Note: In case you are unfamiliar with why there are two ports 5432 and a random port: The Postgres container expects traffic on port 5432. The docker environment allows to map this port to a different host port, such that you can run potentially multiple databases.

Introducing a test configuration 🔗

We only want to read that port in test cases, not on local execution and not on production. Therefore, we introduce a new environment test in the config type config.type.ts beside the already existing environments local and production.

// ...

export type Environment =
  // The service running in a production cluster available for customers
  | "production"
  // The service running locally on a development machine
  | "local"
  // The service running with Jest executing tests
  | "test";

export interface Config {
  environment: Environment;
  // ...
}

// ...

Then, we can return a different config for this environment in the file get-config.ts.

import { Config, Environment, ProcessVariables } from "../config.type";
import { getTestConfig } from "./get-test.config";
import { getLocalConfig } from "./get-local.config";
import { getProductionConfig } from "./get-production.config";

export function getConfig(processVariables: ProcessVariables): Config {
  const environment: Environment = processVariables.ENV || "local";
  switch (environment) {
    case "production":
      return getProductionConfig(processVariables);
    case "local":
      return getLocalConfig(processVariables);
    case "test":
      return getTestConfig(processVariables);
  }
}

In the file get-test.config.ts, we read the database port from the temporary file.

import path from "path";
import os from "os";
import { Config, ProcessVariables } from "../config.type";
import { readFileSync } from "fs";

function readDatabasePort(): number | undefined {
  const variablesDir = path.join(
    os.tmpdir(),
    "jest_testcontainers_global_setup"
  );
  const port = readFileSync(path.join(variablesDir, "databasePort"), "utf8");
  return port ? parseInt(port) : undefined;
}

export function getTestConfig(processVariables: ProcessVariables): Config {
  return {
    environment: "test",
    logLevel: processVariables.LOG_LEVEL ?? "info",
    authentication: {
      enabled: false,
      jwksUrl: "<jwksUrl> is not set in a test environment",
    },
    database: {
      user: "postgres",
      host: "localhost",
      database: "postgres",
      port: readDatabasePort(),
      password: "secret",
      ssl: false,
    },
  };
}

Now, we can configure Jest in the jest.config.js to use the test environment instead of the local environment.

process.env = Object.assign(process.env, {
  DISABLE_LOGGER: "true",
  ENV: "test",
});

module.exports = {
  // ...
};

Migrating the database schema 🔗

If we run the tests npm run test, then the database can now be accessed. However, it complains about the non-existing table todos.

npm run test

  ● the todo dao › retrieves no todos if the database is empty

    expect(received).resolves.toEqual()

    Received promise rejected instead of resolved
    Rejected to value: [error: select * from "todos" - relation "todos" does not exist]

We need to run the database schema migrations before the tests are executed. In a file src/database/db-test.functions.ts we create a function migrateDatabase.

import { db } from "./db";

export async function migrateDatabase() {
  await db.migrate.latest({ directory: "./src/database/migrations" });
}

This function allows us to put an expressive statement beforeAll(migrateDatabase) into each test file requiring the database. Only the first call actually migrates the database. In the other cases, Knex discovers that the migrations already took place and skips the schema migration.

import { migrateDatabase } from "../../database/db-test.functions";
import { getTodos } from "./todo.dao";

describe("the todo dao", () => {
  beforeAll(migrateDatabase);

  it("retrieves no todos if the database is empty", async () => {
    await expect(getTodos()).resolves.toEqual([]);
  });
});

Note Ideally, we would extend the Jest global setup to migrate the database schema after the database is spawned. However, I am not able to get the migrations to run in the global setup due to some module import errors. It seems like the Jest environment does not like something about the module format of Knex or how Knex dynamically loads the TypeScript migration scripts from the source code. All fixes I found on the web did not work in this setup with Knex running in Jest. If you know how to fix it, let me know on Mastodon @tcoderman@techhub.social and I will improve this section referring to you.

Adding more tests 🔗

Now that the tables are created, we can add more tests.

import { omit } from "lodash/fp";
import { migrateDatabase } from "../../database/db-test.functions";
import {
  createTodo,
  deleteTodo,
  getTodo,
  getTodos,
  updateTodo,
} from "./todo.dao";
import { createStubTodo } from "./todo.stub";
import { v4 as uuid } from "uuid";

describe("the todo dao", () => {
  beforeAll(migrateDatabase);

  it("retrieves no todos if the database is empty", async () => {
    await expect(getTodos()).resolves.toEqual([]);
  });

  it("inserts a todo and retrieves it", async () => {
    const todo = omit("id", createStubTodo());
    const createdTodo = await createTodo(todo);
    expect(createdTodo).toHaveProperty("id");
    await expect(getTodos()).resolves.toEqual([createdTodo]);
    await expect(getTodo(createdTodo.id)).resolves.toEqual(createdTodo);
  });

  it("updates an existing todo", async () => {
    const toBeCreatedTodo = omit("id", createStubTodo());
    const createdTodo = await createTodo(toBeCreatedTodo);
    const toBeUpdatedTodo = omit("id", createStubTodo());
    const updatedTodo = await updateTodo(createdTodo.id, toBeUpdatedTodo);
    await expect(getTodos()).resolves.toEqual([updatedTodo]);
    await expect(getTodo(createdTodo.id)).resolves.toEqual(updatedTodo);
  });

  it("deletes an existing todo", async () => {
    const toBeCreatedTodo = omit("id", createStubTodo());
    const createdTodo = await createTodo(toBeCreatedTodo);
    await expect(deleteTodo(createdTodo.id)).resolves.toEqual("Ok");
    await expect(getTodos()).resolves.toEqual([]);
    await expect(getTodo(createdTodo.id)).resolves.toEqual(undefined);
  });

  it("does not find a non-existing todo on update", async () => {
    const todo = omit("id", createStubTodo());
    await expect(updateTodo(uuid(), todo)).resolves.toEqual("NotFound");
  });

  it("does not find a non-existing todo on delete", async () => {
    await expect(deleteTodo(uuid())).resolves.toEqual("NotFound");
  });
});

On execution, we quickly notice that the tests affect each other. The database is not cleaned up after each test.

Cleaning the tables 🔗

In the file db-test.functions.ts, we create a function truncateTables that cleans all tables. A truncate is a fast delete of all entries in a table.

import { db } from "./db";

export async function migrateDatabase() {
  // ...
}

export async function truncateTables() {
  await db.table("todos").truncate();
}

In each test file, we can call this function after each test.

beforeAll(migrateDatabase);
afterEach(truncateTables);

Note: The table name todos is hardcoded in the function truncateTables. In a more complex application, you might want to go for a more sophisticated approach to avoid forgetting to clean some tables.

Shutting down the database 🔗

If you run npm run test, then a special docker container ryuk from testcontainers will take care of shutting down the database container once the node environment has shut down (that is once the command npm run test terminated).

However, if you run npm run test -- --watch, then the node environment never shuts down. You can inspect this if you run watch docker ps while running npm run test -- --watch.

Still though, Jest runs the global setup on each trigger of file system changes and therefore spawns more and more databases. To avoid consuming more and more resources with a long running npm run test -- --watch locally, we shut down the database in the global Jest teardown script.

// ...

module.exports = {
  // ...
  globalSetup: "<rootDir>/src/database/db-test.setup.ts",
  globalTeardown: "<rootDir>/src/database/db-test.teardown.ts",
};

In the Jest global setup db-test.setup.ts, we share the spawned container to a global variable.

import { GenericContainer, StartedTestContainer } from "testcontainers";
import { mkdir, writeFile } from "fs/promises";
import os from "os";
import path from "path";

function spawnDatabase(): Promise<StartedTestContainer> {
  // ...
}

async function shareDatabaseConfig(container: StartedTestContainer) {
  // ...
}

function shareContainerForTeardown(container: StartedTestContainer) {
  (globalThis as any).__DATABASE_CONTAINER__ = container;
}

async function setupDatabase(): Promise<void> {
  const container = await spawnDatabase();
  await shareDatabaseConfig(container);
  shareContainerForTeardown(container);
}

module.exports = setupDatabase;

Wait, what? Why didn't we share the database port the same way? In contrast to the individual test cases, the global teardown runs in the same node environment as the global setup. Hence, we can access global variables from the global setup in the global teardown.

In the global teardown db-test.teardown.ts, we call stop() on the started container __DATABASE_CONTAINER__.

export async function teardownDatabase(): Promise<void> {
  (globalThis as any).__DATABASE_CONTAINER__.stop();
}

module.exports = teardownDatabase;

Appendix: Date workaround 🔗

There is something weird going with Knex and date formats passed to Postgres. It's fixable via specifying a date parser in the database file db.ts.

import { Knex } from "knex";
import { types } from "pg";
import { config } from "../configuration/config";

types.setTypeParser(1082, (date: string) => date);

export const db = require("knex")({
  client: "pg",
  connection: config.database,
}) as Knex;

Conclusion 🔗

We added tests for the DAO todo.dao.ts. These tests verify the SQL queries of the DAO via executing them against a real Postgres database with the latest database schema. The tests spawn one shared database, clean it after each test and shut it down after all tests.

Single schema multi-tenancy in Node.js with Postgres

2022-05-21T00:00:00Z

We want to allow multiple tenants to use the todos of this service independently in dedicated workspaces. The JWT provided from the token issuer contains a workspace ID to identify the workspace the authenticating user is operating in.

{
  "sub": "1234567890",
  "name": "John Doe",
  "workspaceId": "83fdfa88-737d-4550-87e4-c78bf954fcf2",
  "iat": 1516239022
}

In this post, we pass this workspace id from the endpoint controllers through the data access module to the database to achieve multi-tenancy.

Multi-tenancy options 🔗

When it comes to separating workspaces in the database, there are three different options.

Separate databases: Very expensive, very slow to create tenants, but great isolation of tenants.
Separate schemas: Expensive, slow to create tenants, still good isolation of tenants, but isolation needs to be handled in application.
Same schema, additional workspace ID column: Cheap, fast to create tenants, isolation of tenants fully in application, no isolated backups possible, admin queries over multiple workspaces possible.

All of these are valid options and depending on your requirements your chosen option will vary. The more sensible the information, the higher the isolation needs to be. You need to handle credit card information differently than a list of todos.

Since our service deals with non-sensible information, we favor speed over isolation and go for an implementation in the same schema.

Adding a workspace ID column 🔗

We add a second schema migration migrations/2-add-workspace-id.ts that adds a workspace ID column to thetodos table.

import { Knex } from "knex";

export async function up(knex: Knex): Promise<void> {
  await knex.schema.alterTable("todos", (table) => {
    table.uuid("workspaceId").notNullable();
    table.dropPrimary();
    table.primary(["id", "workspaceId"]);
  });
}

export async function down(knex: Knex): Promise<void> {
  await knex.schema.alterTable("todos", (table) => {
    table.dropPrimary();
    table.primary(["id"]);
  });
}

First, we create a UUID column for the workspace ID. Then, we drop the previous primary constraint of the id column. At last, we add a new primary constraint requiring the combination of id and workspaceId to be unique.

Note: If we were to create a table tasks referring to todos, then it would need to reference the combined foreign key of id and workspaceId. This makes JOIN queries more cumbersome and is a drawback of this approach. However, it reduces the risk of accidentally joining with data of different workspaces.

Workspace ID type 🔗

The workspace ID is referenced through the application in controllers and data access modules. To reduce the risk of confusing other IDs with workspace IDs, we create a type alias WorkspaceId in a file src/workspace-id.type.ts.

export type WorkspaceId = string;

export type WithWorkspaceId = {
  workspaceId: WorkspaceId;
};

Additionally, we create a type WithWorkspaceId. Every entity in the database has a workspace ID. But API users don't expect the workspace ID to be returned in the entity since for them all entities are part of the same workspace. The type WithWorkspaceId allows us to define a type type DbTodo = Todo & WithWorkspaceId for the usage within data access modules. Outside of the data access modules, we can refer to the type Todo without the workspace ID.

Database seed 🔗

Now that there is a workspace ID column in the todos table, the seeding of the database for the local environment needs to change. Instead of inserting three todos, we now insert five todos: Three todos in a first workspace and two todos in a second workspace.

import { Knex } from "knex";
import { DbTodo } from "../../controllers/todos/todo.type";
import { WorkspaceId } from "../../workspace-id.type";

const workspaceId1: WorkspaceId = "a9393008-eab2-48e8-b820-0e03447f881c";
const workspaceId2: WorkspaceId = "83fdfa88-737d-4550-87e4-c78bf954fcf2";

const todos: DbTodo[] = [
  {
    workspaceId: workspaceId1,
    id: "142e8446-f5ba-4813-bfb5-f3192a37f1bf",
    name: "Workspace 1 - Todo 1",
    assignee: "Naomi",
    dueDate: "2025-01-05",
  },
  {
    workspaceId: workspaceId1,
    id: "bf6091cc-69f8-4a9e-93ed-0294c3a8ac2f",
    name: "Workspace 1 - Todo 2",
    assignee: "Peter",
    dueDate: "2029-06-11",
  },
  {
    workspaceId: workspaceId1,
    id: "6e5bad4c-7f1c-4ab4-9361-c6bd697b4257",
    name: "Workspace 1 - Todo 3",
    assignee: "Sheila",
    dueDate: "2032-11-27",
  },
  {
    workspaceId: workspaceId2,
    id: "1d2be9f2-4fe9-4509-b8ec-d0d9425c3685",
    name: "Workspace 2 - Todo 1",
    assignee: "Ousmane",
    dueDate: "2025-07-14",
  },
  {
    workspaceId: workspaceId2,
    id: "e868f10d-d21a-4139-9c5d-b8c73c62735a",
    name: "Workspace 2 - Todo 2",
    assignee: "Carla",
    dueDate: "2034-09-07",
  },
];

export async function seed(knex: Knex): Promise<void> {
  await knex("todos").truncate();
  await knex("todos").insert(todos);
}

Local JWT tokens 🔗

Using jwt.io, we refer to these two workspace IDs in two tokens. We adjust the script local.sh to initialize these two tokens.

export TOKEN1=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwid29ya3NwYWNlSWQiOiJhOTM5MzAwOC1lYWIyLTQ4ZTgtYjgyMC0wZTAzNDQ3Zjg4MWMiLCJpYXQiOjE1MTYyMzkwMjJ9.TBClI8i5Urt4JDEnAo0g6LB59-S3hQwdt64MaAVulI4
export TOKEN2=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwid29ya3NwYWNlSWQiOiI4M2ZkZmE4OC03MzdkLTQ1NTAtODdlNC1jNzhiZjk1NGZjZjIiLCJpYXQiOjE1MTYyMzkwMjJ9.Pzcmnyvyvqm4PKbTm5Cu6yflYnC3oFAyFaib4Oo0cj4
export INVALID_TOKEN=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c

Additionally, we add an invalid token without a workspace ID. This way, we can manually check the behaviour of the application if the token issuer issues a faulty token.

Expecting a workspace ID in DAO functions 🔗

The data access module todo.dao.ts needs to accept the workspace ID as a parameter and utilize it in the Knex SQL queries. We adjust every function getTodo, getTodos, createTodo, updateTodo and deleteTodo.

getTodo 🔗

When retrieving an individual todo, we need to define an additional WHERE clause in the SQL query.

export function getTodo(
  workspaceId: WorkspaceId,
  id: TodoId
): Promise<Todo | undefined> {
  return db
    .table<DbTodo>("todos")
    .where("id", id)
    .andWhere("workspaceId", workspaceId)
    .first<Todo>(todoFields);
}

Additionally, we want to keep the fact that the workspaces are implemented via workspace ID column an implementation detail of the data access module and the database. Therefore, we only select the fields todoFields of a Todo. These are defined in the module todo.type.ts as export const todoFields: (keyof Todo)[] = ["id", "name", "assignee", "dueDate"];

getTodos 🔗

Similarly, when retrieving all todos, we pass the workspace ID in a WHERE clause and only select the fields todoFields.

export function getTodos(workspaceId: WorkspaceId): Promise<Todo[]> {
  return db
    .table<DbTodo>("todos")
    .where("workspaceId", workspaceId)
    .select(todoFields);
}

createTodo 🔗

When inserting a todo, we populate the workspace ID column with the passed parameter workspaceId.

export async function createTodo(
  workspaceId: WorkspaceId,
  todo: Omit<Todo, "id">
): Promise<Todo> {
  const todoWithId: Todo = {
    ...todo,
    id: uuid(),
  };
  await db.table<DbTodo>("todos").insert({
    ...todoWithId,
    workspaceId,
  });
  return todoWithId;
}

updateTodo 🔗

On update of a todo, we define an additional WHERE clause to make sure we don't update a todo of a different workspace with the same todo ID.

export async function updateTodo(
  workspaceId: WorkspaceId,
  id: TodoId,
  todo: Omit<Todo, "id">
): Promise<Todo | "NotFound"> {
  const todoWithId = {
    ...todo,
    id,
  };
  const changedRowCount = await db
    .table<DbTodo>("todos")
    .where("id", id)
    .andWhere("workspaceId", workspaceId)
    .update({
      ...todoWithId,
      workspaceId,
    });
  return changedRowCount === 0 ? "NotFound" : todoWithId;
}

deleteTodo 🔗

Also, on the deletion of a todo, we make sure to only delete the todo with the todo ID if the workspace ID matches.

export async function deleteTodo(
  workspaceId: WorkspaceId,
  id: TodoId
): Promise<"Ok" | "NotFound"> {
  const deletedRowCount = await db
    .table<DbTodo>("todos")
    .where("id", id)
    .andWhere("workspaceId", workspaceId)
    .delete();
  return deletedRowCount === 0 ? "NotFound" : "Ok";
}

Testing the data access module 🔗

With the signature of the functions of the data access module changed, we need to adjust the test cases. For each of the existing test cases, we generate a random workspace ID and pass it to the data access functions. Apart from that, nothing changes.

import { omit } from "lodash/fp";
import {
  migrateDatabase,
  truncateTables,
} from "../../database/db-test.functions";
import {
  createTodo,
  deleteTodo,
  getTodo,
  getTodos,
  updateTodo,
} from "./todo.dao";
import { createStubTodo } from "./todo.stub";
import { v4 as uuid } from "uuid";

describe("the todo dao", () => {
  beforeAll(migrateDatabase);
  afterEach(truncateTables);

  it("retrieves no todos if there are none in this workspace", async () => {
    const workspaceId = uuid();
    await expect(getTodos(workspaceId)).resolves.toEqual([]);
  });

  it("inserts a todo and retrieves it", async () => {
    const workspaceId = uuid();
    const todo = omit("id", createStubTodo());
    const createdTodo = await createTodo(workspaceId, todo);
    expect(createdTodo).toHaveProperty("id");
    await expect(getTodos(workspaceId)).resolves.toEqual([createdTodo]);
    await expect(getTodo(workspaceId, createdTodo.id)).resolves.toEqual(
      createdTodo
    );
  });

  it("updates an existing todo", async () => {
    const workspaceId = uuid();
    const toBeCreatedTodo = omit("id", createStubTodo());
    const createdTodo = await createTodo(workspaceId, toBeCreatedTodo);
    const toBeUpdatedTodo = omit("id", createStubTodo());
    const updatedTodo = await updateTodo(
      workspaceId,
      createdTodo.id,
      toBeUpdatedTodo
    );
    await expect(getTodos(workspaceId)).resolves.toEqual([updatedTodo]);
    await expect(getTodo(workspaceId, createdTodo.id)).resolves.toEqual(
      updatedTodo
    );
  });

  it("deletes an existing todo", async () => {
    const workspaceId = uuid();
    const toBeCreatedTodo = omit("id", createStubTodo());
    const createdTodo = await createTodo(workspaceId, toBeCreatedTodo);
    await expect(deleteTodo(workspaceId, createdTodo.id)).resolves.toEqual(
      "Ok"
    );
    await expect(getTodos(workspaceId)).resolves.toEqual([]);
    await expect(getTodo(workspaceId, createdTodo.id)).resolves.toEqual(
      undefined
    );
  });

  it("does not find a non-existing todo on update", async () => {
    const workspaceId = uuid();
    const todo = omit("id", createStubTodo());
    await expect(updateTodo(workspaceId, uuid(), todo)).resolves.toEqual(
      "NotFound"
    );
  });

  it("does not find a non-existing todo on delete", async () => {
    const workspaceId = uuid();
    await expect(deleteTodo(workspaceId, uuid())).resolves.toEqual("NotFound");
  });
});

Additionally, we add new test cases that explicitly check that the workspaces are properly separated. These tests are crucial, as they are the only safeguard to prevent a forgotten WHERE clause and exposing data of a different workspace.

We check for a specific scenario: There exists a todo with the todo ID in a different workspace. In this case, the following statements should be true.

The todo from the different workspace should not be retrieved.
The todo from the different workspace should not be updated.
The todo from the different workspace should not be deleted.

import { omit } from "lodash/fp";
import {
  migrateDatabase,
  truncateTables,
} from "../../database/db-test.functions";
import {
  createTodo,
  deleteTodo,
  getTodo,
  getTodos,
  updateTodo,
} from "./todo.dao";
import { createStubTodo } from "./todo.stub";
import { v4 as uuid } from "uuid";

describe("the todo dao", () => {
  beforeAll(migrateDatabase);
  afterEach(truncateTables);

  // ...

  it("does not retrieve a todo of a different workspace", async () => {
    const workspaceId1 = uuid();
    const workspaceId2 = uuid();
    const todo = omit("id", createStubTodo());
    const createdTodo = await createTodo(workspaceId1, todo);
    await expect(getTodos(workspaceId2)).resolves.toEqual([]);
    await expect(getTodo(workspaceId2, createdTodo.id)).resolves.toEqual(
      undefined
    );
  });

  // ...

  it("does not update a todo of a different workspace", async () => {
    const workspaceId1 = uuid();
    const workspaceId2 = uuid();
    const toBeCreatedTodo = omit("id", createStubTodo());
    const createdTodo = await createTodo(workspaceId1, toBeCreatedTodo);
    const toBeUpdatedTodo = omit("id", createStubTodo());
    const updatedTodo = await updateTodo(
      workspaceId2,
      createdTodo.id,
      toBeUpdatedTodo
    );
    expect(updatedTodo).toEqual("NotFound");
    await expect(getTodos(workspaceId1)).resolves.toEqual([
      { ...toBeCreatedTodo, id: createdTodo.id },
    ]);
    await expect(getTodos(workspaceId2)).resolves.toEqual([]);
    await expect(getTodo(workspaceId1, createdTodo.id)).resolves.toEqual({
      ...toBeCreatedTodo,
      id: createdTodo.id,
    });
    await expect(getTodo(workspaceId2, createdTodo.id)).resolves.toEqual(
      undefined
    );
  });

  // ...

  it("does not delete a todo of a different workspace", async () => {
    const workspaceId1 = uuid();
    const workspaceId2 = uuid();
    const toBeCreatedTodo = omit("id", createStubTodo());
    const createdTodo = await createTodo(workspaceId1, toBeCreatedTodo);
    await expect(deleteTodo(workspaceId2, createdTodo.id)).resolves.toEqual(
      "NotFound"
    );
    await expect(getTodos(workspaceId1)).resolves.toHaveLength(1);
    await expect(getTodos(workspaceId2)).resolves.toHaveLength(0);
    await expect(getTodo(workspaceId1, createdTodo.id)).resolves.not.toEqual(
      undefined
    );
    await expect(getTodo(workspaceId2, createdTodo.id)).resolves.toEqual(
      undefined
    );
  });

  // ...
});

Get workspace ID from JWT 🔗

Before we are able to pass the workspace ID from the endpoint controllers to the data access functions, we need to retrieve the workspace ID from the JWT.

We create a file src/jwt-token.ts that offers a function getToken. This function returns the token from the variable response.locals.token. This variable is initialized in the pre request handler pre-request-handlers/authenticate.ts introduced in the authentication post.

For more information on this, checkout 'JWT authentication with JWKS in Express.js'

import { Response } from "express";
import { JwtPayload } from "jsonwebtoken";
import { WithWorkspaceId } from "./workspace-id.type";

export type JwtTokenPayload = JwtPayload & WithWorkspaceId;

export function getToken(response: Response): JwtTokenPayload {
  return response.locals.token;
}

Defining the function getToken in a separate module has two advantages:

The response is properly typed as JwtTokenPayload.
We can mock the function getToken in tests of the controllers.

In this function getToken, we are relying on the issuer of the JWT that the workspace ID is present. To protect us from bugs in that service, we validate the token in the pre request handler authenticate.ts.

import { NextFunction, Request, Response } from "express";
import jwt from "jsonwebtoken";
// ...
import { JwtTokenPayload } from "../jwt-token";

// ...

export async function authenticate(
  request: Request,
  response: Response,
  next: NextFunction
): Promise<unknown> {
  try {
    // ...

    if (!isValidPayload(decodedToken.payload)) {
      throw new Error("Unexpected structure of JWT payload");
    }

    response.locals.token = decodedToken.payload;

    next();
  } catch (error) {
    next(error);
  }
}

function isValidPayload(
  payload: string | jwt.JwtPayload
): payload is JwtTokenPayload {
  return typeof payload !== "string" && "workspaceId" in payload;
}

If the token payload is not an object or does not contain the workspace ID, then we throw an error. This thrown error results in an internal server error response 500 due to a post request handler.

Passing the workspace ID from the controllers 🔗

Now that there is a function getToken to retrieve the workspace ID of the JWT, we can adjust the endpoint controllers.

The pattern is identical in all endpoint controllers. We call getToken and pass the workspace ID to the data access function.

import { NextFunction, Request, Response } from "express";
import { getToken } from "../../jwt-token";
import { deleteTodo } from "./todo.dao";

export async function deleteTodoController(
  request: Request,
  response: Response,
  next: NextFunction
): Promise<void> {
  try {
    const workspaceId = getToken(response).workspaceId;
    await deleteTodo(workspaceId, request.params.id);
    response.sendStatus(204);
  } catch (error) {
    next(error);
  }
}

In the tests, we mock the function getToken.

import request from "supertest";
import { v4 as uuid } from "uuid";
import { server } from "../../test.functions";
import { deleteTodoController } from "./delete-todo.controller";

const workspaceId = uuid();

jest.mock("./todo.dao");
jest.mock("../../jwt-token", () => ({
  getToken: () => ({
    workspaceId,
  }),
}));

describe("deleteTodoController", () => {
  const route = "/todos/:id";

  const app = server((app) => {
    app.delete(route, deleteTodoController);
  });

  it("deletes the todo and returns a 204", async () => {
    const deleteTodo = require("./todo.dao").deleteTodo;
    const todoId = uuid();
    deleteTodo.mockResolvedValue();
    await request(app).delete(route.replace(":id", todoId)).expect(204);
    expect(deleteTodo).toHaveBeenCalledWith(workspaceId, todoId);
  });

  // ...
});

Conclusion 🔗

We implemented multi-tenancy in the todo service. Each database table has a column workspaceId. The workspace ID is taken from JWT and passed from the controller through the data access module to the database.

Organizing and testing HTTP requests

2022-05-24T00:00:00Z

We want to send HTTP requests to other services. On production, we should contact the services. Locally and in tests, we want to contact mocks instead.

We assume the existence of a notification service. If a todo is deleted, then a notification should be created in that notification service. The notification service offers an API endpoint POST /notifications to do that.

We install the library axios to perform HTTP requests.

npm install --save axios
npm install --save-dev @types/axios

Sending an HTTP request 🔗

We create a module controllers/todos/notification.service.ts. Similarly to the data access module controllers/todos/todo.dao, it contains the precise queries we want to perform in the context of the todo controllers. It's not the goal to build a full TypeScript representation of the complete API of the notification service. Instead, the notification.service contains only the required function sendNotification.

import axios from "axios";
import { config } from "../../configuration/config";

export function sendNotification(message: string): Promise<void> {
  return axios.post(`${config.http.servicesUrl}/notify/notifications`, {
    message,
  });
}

The API endpoint POST /notifications provides more options. But we only implement the options that are required in the todo service. This avoids complexity and makes testing easier.

Configuring the services URL 🔗

In the configuration configuration/config.ts, we put the URL servicesUrl.

import { Knex } from "knex";
import { Level } from "pino";

export type Environment =
  // The service running in a production cluster available for customers
  | "production"
  // The service running locally on a development machine
  | "local"
  // The service running with Jest executing tests
  | "test";

export interface Config {
  // ...
  http: {
    servicesUrl: string;
  };
}

export interface ProcessVariables {
  // ...
  SERVICES_URL?: string;
}

On the production environment, we read the URL from an environment variable.

import { Config, ProcessVariables } from "../config.type";

export function getProductionConfig(
  processVariables: ProcessVariables
): Config {
  return {
    // ...
    http: {
      servicesUrl:
        processVariables.SERVICES_URL ??
        "<SERVICES_URL> needs to bet set in production environment",
    },
  };
}

Locally, we set the URL to port 3001 on localhost.

import { Config, ProcessVariables } from "../config.type";

export function getLocalConfig(processVariables: ProcessVariables): Config {
  return {
    // ...
    http: {
      servicesUrl: "http://localhost:3001",
    },
  };
}

At this local URL http://localhost:3001, we spin up a server that responds with mock responses.

Note: In my experience, it also works well to connect for local development with a production environment with test workspaces. This has the advantage of getting responses of actual production services without the overhead of running these other services locally. However, it also makes it easy to create a lot of coupling between services.

Mocking responses in local development 🔗

There are several options for mocking HTTP responses locally. We choose the npm package mockserver. It allows to organize the responses with the file system structure.

We create a file mocks/notify/notifications/POST.mock. The package mockserver returns this response on a call to the route POST /notify/notifications.

HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8

{
   "message": "Notification created"
}

In the package.json, we add a command start:mocks to start the mock server on http://localhost:3001 with the folder mocks. The npm package concurrently allows us to run the app, the TypeScript compiler and the mock server in parallel.

{
  // ...
  "scripts": {
    // ...
    "start": "npm run build && npx concurrently \"npm:start:*\"",
    "start:app": "nodemon --legacy-watch --watch ./dist --inspect=0.0.0.0:9229 dist/main.js -- server | npx pino-pretty -i time,hostname,module,__in,name,pid",
    "start:ts": "tsc --watch --incremental",
    "start:mocks": "npx mockserver -p 3001 -m mocks"
    // ...
  }
  // ...
}

The command npm run start:mocks starts the mock server and keeps it running.

npm run start:mocks

> node-microservice-architecture@0.0.1 start:mocks
> npx mockserver -p 3001 -m mocks

Mockserver serving mocks {verbose:true} under "mocks" at http://localhost:3001

Sending a notification 🔗

We adjust the controller of the endpoint DELETE /todos/:id to send a notification immediately after a successful delete.

Note: If the creation of the notification fails, then also DELETE /todos/:id returns with an error. But the todo will still be created in the database. I don't recommend to send notifications synchronously with the incoming request. In a later post, we improve on this with asynchronous notification creation.

import { NextFunction, Request, Response } from "express";
import { getToken } from "../../jwt-token";
import { sendNotification } from "./notification.service";
import { deleteTodo } from "./todo.dao";

export async function deleteTodoController(
  request: Request,
  response: Response,
  next: NextFunction
): Promise<void> {
  try {
    const workspaceId = getToken(response).workspaceId;
    await deleteTodo(workspaceId, request.params.id);
    await sendNotification(`Deleted todo ${request.params.id}`);
    response.sendStatus(204);
  } catch (error) {
    next(error);
  }
}

The unit test of the controller should not try to send a notification. Instead, we mock the function sendNotification. Since the function is in its own module, mocking this function is identical to mocking any other internal module.

import request from "supertest";
import { v4 as uuid } from "uuid";
import { server } from "../../test.functions";
import { deleteTodoController } from "./delete-todo.controller";

// ...
jest.mock("./notification.service");
// ...

describe("deleteTodoController", () => {
  const route = "/todos/:id";

  const app = server((app) => {
    app.delete(route, deleteTodoController);
  });

  it("deletes the todo, sends a notification and returns a 204", async () => {
    const deleteTodo = require("./todo.dao").deleteTodo;
    const sendNotification = require("./notification.service").sendNotification;
    const todoId = uuid();
    deleteTodo.mockResolvedValue();
    await request(app).delete(route.replace(":id", todoId)).expect(204);
    expect(deleteTodo).toHaveBeenCalledWith(workspaceId, todoId);
    expect(sendNotification).toHaveBeenCalledWith(`Deleted todo ${todoId}`);
  });

  // ...
});

Conclusion 🔗

We did setup axios to send HTTP requests and organized the calls into a service module similarly to the organization of data access modules. Locally, the HTTP requests go to a mock server. In tests, the encapsulating module notification.service is mocked.

In the following posts, we setup connection pooling, logging and authentication of outgoing requests.

Connection pooling with Axios and agentkeepalive

2022-05-25T00:00:00Z

If you are using plain Axios, then with each request a new connection is opened. However, the server your application is running on can only handle a limited number of connections at the same time. If no new connections can be created, then this can lead to all sort of weird errors in production. Let's avoid this and use a connection pool.

Luckily, there is an npm package agentkeepalive that implements the concept of a connection pool.

npm install agentkeepalive --save

We only need to configure axios to use this package agentkeepalive.

HTTP Agents 🔗

In a module src/http/create-http-agents.ts, we specify an HTTP agent and an HTTPS agent. The configuration depends on your production environment and requirements on the service. You might want to play around with different settings.

import Agent from "agentkeepalive";

export function createHttpAgent() {
  return new Agent({
    maxSockets: 100,
    maxFreeSockets: 10,
    timeout: 60000,
    freeSocketTimeout: 30000,
  });
}

export function createHttpsAgent() {
  return new Agent.HttpsAgent({
    maxSockets: 100,
    maxFreeSockets: 10,
    timeout: 60000,
    freeSocketTimeout: 30000,
  });
}

Connecting Axios and agentkeepalive 🔗

In the same directory src/http, we create a module configure-http.ts. It contains a function that registers the HTTP agents on the global axios object.

import axios from "axios";
import { createHttpAgent, createHttpsAgent } from "./create-http-agents";

export function configureHttp(): void {
  axios.defaults.httpAgent = createHttpAgent();
  axios.defaults.httpsAgent = createHttpsAgent();
}

If the function configureHttp has been called, then all requests made with the global object axios use the connection pool.

import axios from "axios";
import { config } from "../../configuration/config";

export function sendNotification(message: string): Promise<void> {
  return axios.post(`${config.http.servicesUrl}/notify/notifications`, {
    message,
  });
}

Configuring Axios on application start 🔗

Now, we only need to make sure the function is called when the application starts. We do this as one of the first steps in the function main in the module src/main.ts.

import app from "./app";
import * as http from "http";
import { logger } from "./logger";
import { configureHttp } from "./http/configure-http";

async function main(_args: string[]) {
  const port = process.env.PORT ?? 3000;
  configureHttp();
  const server = http.createServer(app);
  server.listen(port, () => {
    logger.info(`Server listening on port ${port}!`);
  });
}

main(process.argv).catch((error) => {
  // ...
});

// ...

Conclusion 🔗

We configured a connection pool with Axios. Every outgoing HTTP request done with the global axios object now utilizes a connection from the connection pool.

Pino JSON logging of Axios HTTP requests

2022-05-25T00:00:00Z

Other services can respond in unexpected ways or not be reachable at all. Therefore, we track the requests and responses to other services. This way, we can see if the source of an issue is within or outside of our service.

We configure the Pino JSON logger to log a reasonable amount of detail for each outgoing HTTP request and response.

Response Interceptor 🔗

Axios offers the concept of interceptors. A response interceptor is called after each incoming response.

We create an interceptor logResponse in a new module src/http/log-response.interceptor.ts. With the log level INFO, we log the status code, the HTTP method and the URL. With the log level DEBUG, we additionally log the response body.

import { AxiosResponse } from "axios";
import { logger } from "../logger";

export function logResponse(response: AxiosResponse): AxiosResponse {
  logger.info(
    {
      status: response.status,
      method: response.config?.method?.toUpperCase(),
      url: response.config?.url,
    },
    "Response"
  );
  logger.debug(
    {
      status: response.status,
      method: response.config?.method?.toUpperCase(),
      url: response.config?.url,
      body: response.data,
    },
    "Response Details"
  );
  return response;
}

We add a unit test to verify the format of the log lines.

import { AxiosResponse } from "axios";
import { logResponse } from "./log-response.interceptor";

jest.mock("../logger");

describe("logging responses", () => {
  it("should log a concise info and more debug info", () => {
    const logger = require("../logger").logger;
    logResponse({
      status: 200,
      data: {
        some: "response",
      },
      config: {
        data: {
          some: "request",
        },
        method: "get",
        url: "https://some.org/api",
      },
    } as AxiosResponse);
    expect(logger.error).not.toHaveBeenCalled();
    expect(logger.info).toHaveBeenCalledWith(
      {
        status: 200,
        method: "GET",
        url: "https://some.org/api",
      },
      "Response"
    );
    expect(logger.debug).toHaveBeenCalledWith(
      {
        status: 200,
        method: "GET",
        url: "https://some.org/api",
        body: {
          some: "response",
        },
      },
      "Response Details"
    );
  });
});

The function configureHttp is called in the main function of the application. We add the interceptor such that it is registered with Axios.

import axios from "axios";
// ...
import { logResponse } from "./log-response.interceptor";

export function configureHttp(): void {
  // ...
  axios.interceptors.response.use(logResponse);
}

Request interceptor 🔗

Locally, we are also interested in the outgoing request body. We add a request interceptor that logs the outgoing request in debug mode. This way, the observed log in npm start indicates an outgoing HTTP request.

import { AxiosRequestConfig } from "axios";
import { logger } from "../logger";

export function logRequest(request: AxiosRequestConfig): AxiosRequestConfig {
  logger.debug(
    {
      method: request.method?.toUpperCase(),
      url: request.url,
      body: request.data,
    },
    "Request"
  );
  return request;
}

We add a test to verify the logged JSON.

import { AxiosRequestConfig } from "axios";
import { logRequest } from "./log-request.interceptor";

jest.mock("../logger");

describe("logging requests", () => {
  it("should log a concise message and the full body in debug mode", () => {
    const logger = require("../logger").logger;
    logRequest({
      data: {
        some: "body",
      },
      method: "get",
      url: "https://some.org/api",
    } as AxiosRequestConfig);
    expect(logger.info).not.toHaveBeenCalled();
    expect(logger.error).not.toHaveBeenCalled();
    expect(logger.debug).toHaveBeenCalledWith(
      {
        method: "GET",
        url: "https://some.org/api",
        body: { some: "body" },
      },
      "Request"
    );
  });
});

In the axios configuration function configureHttp, we register the request interceptor.

import axios from "axios";
// ...
import { logRequest } from "./log-request.interceptor";
import { logResponse } from "./log-response.interceptor";

export function configureHttp(): void {
  // ...
  axios.interceptors.request.use(logRequest);
  axios.interceptors.response.use(logResponse);
}

Resulting log lines 🔗

We start the application locally with npm start. A request . ./local.sh && http DELETE http://localhost:3000/todos/6e5bad4c-7f1c-4ab4-9361-c6bd697b4257 Authorization:"Bearer $TOKEN1" results in a log line for the request and two log lines for the response. On production, only the response is logged.

npm start

DEBUG: Request
    method: "POST"
    url: "http://localhost:3001/notify/notifications"
    body: {
      "message": "Deleted todo 6e5bad4c-7f1c-4ab4-9361-c6bd697b4257"
    }
INFO: Response
    status: 200
    method: "POST"
    url: "http://localhost:3001/notify/notifications"
DEBUG: Response Details
    status: 200
    method: "POST"
    url: "http://localhost:3001/notify/notifications"
    body: {
      "message": "Notification created"
    }

Conclusion 🔗

We added a request and a response interceptor to Axios. Locally in debug mode, requests and responses are logged with their body. On production, there is only one log entry for the response of the request.

Axios HTTP error handling in Express.js

2022-05-26T00:00:00Z

HTTP communication with other services is error-prone.

A status code 2xx means success and indicates that we can proceed with the regular happy path.

A status code 4xx indicates an error on we made with the request. Some of these errors indicate a bug in our service. An authentication error 401 or 403 should not occur in our outgoing authentication since we always send a valid JWT. Other errors might be regular cases we want to handle in the business logic. For example, a not found error 404 indicates that the resource we are looking for does not exist.

A status code 5xx indicates an error at the other service or in the network between our and the other service. Some of these can be minimized with concepts like circuit breakers between the services. The remaining errors 5xx should result in an internal server error 500 of our service and an error log line. The internal server error returned to the user should contain minimal detail for a support engineer to track down the cause. The error log line should be more detailed and notify the team.

Beside of error status codes, there can also be timeouts. If Axios does not get a response in time, then also these errors should result in an internal server error and an error log line.

In this post, we implement this behaviour.

The Axios error 🔗

When we do an HTTP request with Axios, then by default every non-2xx response results in an error AxiosError. We do not catch it on purpose: There is nothing reasonable we could do about an error here.

import axios from "axios";
import { config } from "../../configuration/config";

export function sendNotification(message: string): Promise<void> {
  return axios.post(`${config.http.servicesUrl}/notify/notifications`, {
    message,
  });
}

If we knew that there are error responses that we can react to, then we can use the option validateStatus. All accepted status codes don't lead to an error but to a regular response instead. However, note that this requires you to inspect the status code to know how to parse the response body.

import axios from "axios";
import { config } from "../../configuration/config";

export function sendNotification(message: string): Promise<void> {
  return axios.post(
    `${config.http.servicesUrl}/notify/notifications`,
    {
      message,
    },
    {
      validateStatus: (status) =>
        (status >= 200 && status < 300) || status == 404,
    }
  );
}

Handling all Axios errors 🔗

For all unhandled errors, we create an Express.js error handler handleHttpError.

import express from "express";
import { todoRoute } from "./controllers/todos/todo.router";
import { handleHttpError } from "./http/http-error-handler";
// ...

const app = express();

// ...

app.use("/todos", todoRoute);

app.use(handleHttpError);
// ...

export default app;

Express.js calls this handler for all errors that have been passed with next(error) in a controller. All our controllers catch exceptions and pass them to next.

import { NextFunction, Request, Response } from "express";
// ...

export async function deleteTodoController(
  request: Request,
  response: Response,
  next: NextFunction
): Promise<void> {
  try {
    // ...
  } catch (error) {
    next(error);
  }
}

The error handler handleHttpError uses a function isAxiosError to verify that the error is of type AxiosError.

import { AxiosError } from "axios";

function isAxiosError<T>(error: Error | AxiosError<T>): error is AxiosError<T> {
  return "isAxiosError" in error && error.isAxiosError;
}

If the error is not an AxiosError, it is passed to the next error handler.

Otherwise, we write a detailed error log line with the Pino logger. Since we have the object request, we use the logger request.log. This logger automatically attaches the request id to the log line and allows us to filter the log lines for the associated request.

import { AxiosError } from "axios";
import { NextFunction, Request, Response } from "express";

export function handleHttpError(
  error: Error,
  request: Request,
  response: Response,
  next: NextFunction
): void {
  if (isAxiosError(error)) {
    request.log.error(
      {
        method: error.config.method?.toUpperCase(),
        url: error.config.url,
        headers: error.config.headers,
        params: error.config.params,
        message: error.message,
        response: {
          status: error.response?.status,
          data: error.response?.data,
        },
        stack: error.stack,
      },
      "Error Response"
    );
    response.status(500).json({
      message: `Failed request`,
      method: error.config.method?.toUpperCase(),
      url: error.config.url,
    });
  } else {
    next(error);
  }
}

Also, we send an internal server error 500 to the user. We include the method and url. This makes it easier to find the log line and possibly already allows the reproduction of the issue.

Testing the handler 🔗

We add a test to verify the log output and response. Another test verifies that indeed the next error handler is called if the request did not result in an AxiosError.

import { AxiosError } from "axios";
import { mockControllerInputs } from "../test.functions";
import { handleHttpError } from "./http-error-handler";

describe("the HTTP error handler", () => {
  it("should log the HTTP error and send a 500", () => {
    const { request, response, next } = mockControllerInputs({
      headers: {},
    });
    const error = new AxiosError(
      "message",
      "code",
      {
        method: "get",
        url: "https://some.url/",
        headers: {
          a: "b",
        },
        params: {
          c: "d",
        },
      },
      {},
      {
        status: 200,
        statusText: "statusText",
        data: {},
        headers: {},
        config: {},
      }
    );
    handleHttpError(error, request, response, next);
    expect(request.log.error).toHaveBeenCalledWith(
      {
        method: "GET",
        url: "https://some.url/",
        headers: {
          a: "b",
        },
        params: {
          c: "d",
        },
        message: "message",
        response: {
          status: 200,
          data: {},
        },
        stack: error.stack,
      },
      "Error Response"
    );
    expect(next).not.toHaveBeenCalled();
  });

  it("should skip different errors", () => {
    const { request, response, next } = mockControllerInputs({
      headers: {},
    });
    const error = new Error("No HTTP error");
    handleHttpError(error, request, response, next);
    expect(request.log.error).not.toHaveBeenCalled();
    expect(next).toHaveBeenCalledWith(error);
  });
});

Conclusion 🔗

We added an Express.js error handler. It logs errors in the communication with other services. Also, it responds with an internal server error 500 to users.

Axios HTTP request authentication with JWT

2022-05-26T00:00:00Z

We want to implement authentication of outgoing HTTP requests with JWT in a multi-tenancy setup.

Our example todo service operates in a microservice environment with multi-tenancy. A user is using one JWT to authenticate with the whole microservice environment but only for one workspace. An authority issues JWTs for the microservice environment. The JWT contains the workspace ID that all services need to consider to limit the operating scope of the user.

Let's assume a user makes an HTTP request to our service authenticating with a JWT. Then our service could use that JWT to authenticate against other services in the microservice environment.

However.

There are several issues with that approach:

It limits the interactions of our service with other services to what the user is permitted to do.
It binds us to the expiration time of the JWT. We can't make requests after that.
Other services can't trivially distinguish between requests from our service and requests from the user. These services might apply rate-limiting to us because they think our service is the user.

To bypass these issues, we request separate JWTs that authenticate our service as itself. Depending on the nature of our request, we request a different scope for the JWT.

For the majority of requests we want to have a JWT restricted to the workspace of the user.
For some requests we want to have a JWT that can act across workspaces.
For other requests we might want to have a JWT that not only acts within a workspace but also with the permissions of the user.

Requesting authentication 🔗

On deletion of a todo, we send an HTTP request to the notification service to create a notification. The todo is deleted within a workspace and the notification should be created in the same workspace.

The endpoint POST /notify/notifications requires a header Authorization with a JWT limited to a workspace.

We already have a module with a function sendNotification that encapsulates the logic of sending this request without authentication. We could retrieve a JWT here and pass it to Axios in the Authorization header.

// ...

export function sendNotification(
  workspaceId: WorkspaceId,
  message: string
): Promise<void> {
  const Authorization = await getJWT(workspaceId);
  return axios.post(
    `${config.http.servicesUrl}/notify/notifications`,
    { message },
    {
      headers: {
        Authorization,
      },
    }
  );
}

However, this exposes details about authentication to every module sending HTTP requests. Also, in every of these tests we would always need to mock the function getJWT.

Instead, we only want to pass the desired scope of the JWT. We leave it up to an Axios interceptor to retrieve the JWT and attach the authorization header.

import axios, { AxiosRequestConfig } from "axios";
import { config } from "../../configuration/config";
import { workspaceAccess } from "../../http/service-access.type";
import { WorkspaceId } from "../../workspace-id.type";

export function sendNotification(
  workspaceId: WorkspaceId,
  message: string
): Promise<void> {
  return axios.post(
    `${config.http.servicesUrl}/notify/notifications`,
    { message },
    {
      access: {
        type: "workspace",
        workspaceId,
      },
    } as AxiosRequestConfig
  );
}

The service access type 🔗

In a module src/http/service-access.type.ts, we define the different options that can be passed as parameter access to an Axios request.

import { WorkspaceId } from "../workspace-id.type";

export type ServiceAccess = AdminAccess | WorkspaceAccess;

export interface AdminAccess {
  type: "admin";
}

export type WorkspaceAccess = {
  type: "workspace";
  workspaceId: WorkspaceId;
};

We allow an object { type: "admin" } to be passed to express full access across workspaces. An object { type: "workspace", workspaceId: "<uuid>" } expresses access to one workspace. Additionally, we could define user access with { type: "user", userId: "<uuid>", workspaceId: "<uuid>" }. We leave this option out for simplicity.

Additionally, in the same module, we create two constructors.

export function workspaceAccess(workspaceId: WorkspaceId): WorkspaceAccess {
  return {
    type: "workspace",
    workspaceId,
  };
}

export const adminAccess: AdminAccess = {
  type: "admin",
};

The usage of these constructors ensures that a valid service access object ServiceAccess is passed to the Axios interceptor.

axios.post(url, body, {
  access: workspaceAccess(workspaceId),
} as AxiosRequestConfig);

axios.post(url, body, {
  access: adminAccess,
} as AxiosRequestConfig);

Retrieving a JWT 🔗

We add a module src/http/get-jwt.ts to retrieve a JWT. This logic might vary a lot depending on your way of issuing tokens.

In this setup, there exists a service auth with an endpoint POST /token. From the configuration src/configuration/config.ts, our service receives a client ID clientId and a client secret clientSecret.

// ...

export type Environment =
  // The service running in a production cluster available for customers
  | "production"
  // The service running locally on a development machine
  | "local"
  // The service running with Jest executing tests
  | "test";

export interface Config {
  // ...
  http: {
    // ...
    clientId: string;
    clientSecret: string;
  };
}

export interface ProcessVariables {
  // ...
  CLIENT_ID?: string;
  CLIENT_SECRET?: string;
}

For more information on this, checkout 'Managing configs with TypeScript code in Node.js'

We are using client ID and secret to authenticate with basic authentication against the auth service. This way, the auth service recognizes our service and allows it to issue the JWT.

import axios from "axios";
// ...
import { config } from "../configuration/config";

function toBase64(data: string): string {
  return Buffer.from(data).toString("base64");
}

async function retrieveJWT(scope?: string): Promise<string> {
  const tokenResponse = await axios.post(
    `${config.http.servicesUrl}/auth/token`,
    "",
    {
      params: {
        grant_type: "client_credentials",
        scope,
      },
      headers: {
        Authorization:
          "Basic " +
          toBase64(`${config.http.clientId}:${config.http.clientSecret}`),
        "content-type": "application/x-www-form-urlencoded;charset=utf-8",
      },
    }
  );
  return tokenResponse.data.access_token;
}

With the parameter scope, we decide what permissions the JWT should have. If set to undefined, then the JWT is not limited to a single workspace. If set to workspaceId=<uuid>, then the JWT is limited to the workspace.

Caching the JWT 🔗

We want to avoid asking for a new JWT with every incoming HTTP request. For speed, to avoid load on the token issuer and to avoid a source of HTTP errors.

There is an npm package memoizee that caches the return value of a function given that the input parameters are identical.

npm install --save memoizee
npm install --save-dev @types/memoizee

In the same module src/http/get-jwt.ts, we create a function getJWT. It caches the JWT of the function retrieveJWT for 50 minutes. This time assumes a JWT to become invalid after 60 minutes. You might want to adjust this time depending on the expiration time of JWT in your setup.

export const getJWT = memoize(retrieveJWT, {
  promise: true,
  maxAge: 50 * 60 * 1000, // 50 minutes
  max: 100,
});

We test this function to correctly assemble a JWT request.

import { getJWT } from "./get-jwt";

jest.mock("../configuration/config", () => ({
  config: {
    http: {
      servicesUrl: "https://eu.company.org",
      clientId: "todos",
      clientSecret: "secret",
    },
  },
}));
jest.mock("axios");

describe("the retrieval of JWT", () => {
  it("should retrieve a JWT with admin access", async () => {
    const post = require("axios").post;
    post.mockResolvedValue({
      data: {
        access_token: "my_admin_token",
      },
    });
    await expect(getJWT()).resolves.toEqual("my_admin_token");
    expect(post).toHaveBeenCalledWith("https://eu.company.org/auth/token", "", {
      headers: {
        Authorization: "Basic dG9kb3M6c2VjcmV0",
        "content-type": "application/x-www-form-urlencoded;charset=utf-8",
      },
      params: {
        grant_type: "client_credentials",
      },
    });
  });

  it("should retrieve a JWT with workspace access", async () => {
    const post = require("axios").post;
    post.mockResolvedValue({
      data: {
        access_token: "my_token",
      },
    });
    const scope = "workspaceId=1234-5678";
    await expect(getJWT(scope)).resolves.toEqual("my_token");
    expect(post).toHaveBeenCalledWith("https://eu.company.org/auth/token", "", {
      headers: {
        Authorization: "Basic dG9kb3M6c2VjcmV0",
        "content-type": "application/x-www-form-urlencoded;charset=utf-8",
      },
      params: {
        grant_type: "client_credentials",
        scope,
      },
    });
  });
});

Attaching the Authorization header 🔗

We create an Axios interceptor attachAuthorizationHeader that is called before each outgoing request. We register this interceptor in the function configureHttp that is called in the main function of the program.

import axios from "axios";
import { attachAuthorizationHeader } from "./authorization-header.interceptor";
// ...

export function configureHttp(): void {
  // ...
  axios.interceptors.request.use(attachAuthorizationHeader);
  // ...
}

The interceptor attachAuthorizationHeader resides in the module src/http/authorization-header.interceptor.ts.

If it detects that the passed object access is invalid, then it does nothing. If it detects that the header Authorization is already set, then it also does nothing.

Note: It is crucial that an existing Authorization header is not overridden. Otherwise, the basic authentication for the JWT retrieval would be overridden.

If the object access is valid and the header Authorization is not yet set, then the interceptor retrieves the JWT with the requested scope. Once the retrieval succeeded, the interceptor attaches the header Authorization to the request.

import { AxiosRequestConfig } from "axios";
import { ServiceAccess, WorkspaceAccess } from "./service-access.type";
import { getJWT } from "./get-jwt";

function isValidServiceAccess(access: unknown): access is ServiceAccess {
  return typeof access === "object" && !!access && "type" in access;
}

function isWorkspaceAccess(access: ServiceAccess): access is WorkspaceAccess {
  return "type" in access && access.type === "workspace";
}

export async function attachAuthorizationHeader(
  axiosRequestConfig: AxiosRequestConfig
): Promise<AxiosRequestConfig> {
  const access = (axiosRequestConfig as { access: unknown }).access;
  if (
    !axiosRequestConfig.headers?.Authorization &&
    isValidServiceAccess(access)
  ) {
    const jwt = await getJWT(
      isWorkspaceAccess(access)
        ? `workspaceId=${access.workspaceId}`
        : undefined
    );
    if (!axiosRequestConfig.headers) {
      axiosRequestConfig.headers = {};
    }
    axiosRequestConfig.headers["Authorization"] = `Bearer ${jwt}`;
  }
  return axiosRequestConfig;
}

We can verify the different scenarios with Jest.

import { AxiosRequestConfig } from "axios";
import { attachAuthorizationHeader } from "./authorization-header.interceptor";
import { adminAccess, workspaceAccess } from "./service-access.type";

jest.mock("./get-jwt");

describe("the authorization header interceptor", () => {
  it("should attach an admin token if access type is admin", async () => {
    const getJWT = require("./get-jwt").getJWT;
    getJWT.mockResolvedValue("my_admin_token");
    const request = await attachAuthorizationHeader({
      access: adminAccess,
    } as AxiosRequestConfig);
    expect(getJWT).toHaveBeenCalledWith(undefined);
    expect(request.headers).toHaveProperty(
      "Authorization",
      "Bearer my_admin_token"
    );
  });

  it("should attach a workspace token if target access is requested", async () => {
    const getJWT = require("./get-jwt").getJWT;
    getJWT.mockResolvedValue("my_token");
    const request = await attachAuthorizationHeader({
      access: workspaceAccess("1234"),
    } as AxiosRequestConfig);
    expect(getJWT).toHaveBeenCalledWith("workspaceId=1234");
    expect(request.headers).toHaveProperty("Authorization", "Bearer my_token");
  });

  it("should not attach a token if the access type is invalid", async () => {
    const getJWT = require("./get-jwt").getJWT;
    getJWT.mockResolvedValue("my_token");
    const request = await attachAuthorizationHeader({
      access: {},
    } as AxiosRequestConfig);
    expect(getJWT).not.toHaveBeenCalled();
    expect(request.headers).toEqual(undefined);
  });

  it("should fail if a JWT can not be retrieved", async () => {
    const getJWT = require("./get-jwt").getJWT;
    getJWT.mockRejectedValue("Failure");
    await expect(
      attachAuthorizationHeader({
        access: workspaceAccess("1234"),
      } as AxiosRequestConfig)
    ).rejects.toEqual("Failure");
  });
});

Mocking on local environment 🔗

In an earlier post, we setup mockserver to mock the requests to other services locally.

For more information on this, checkout 'Organizing and testing HTTP requests'

In the folder mocks, we create a file auth/token/POST.mock. This file is interpreted as the response to requests to the endpoint POST http://localhost:3001/auth/token.

HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8

{
   "access_token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwiY2xpZW50SWQiOiJ0b2RvcyIsImlhdCI6MTUxNjIzOTAyMn0.YD_HJROnQHJIROzVBagK27D862QoNvokSDFxDbA8_Ug"
}

Locally, we always send the returned JWT to the mock server. Hence, it suffices if it decodes to a dummy payload.

{
  "sub": "1234567890",
  "clientId": "todos",
  "iat": 1516239022
}

Conclusion 🔗

We added a request interceptor to Axios to add an Authorization header. This header is populated with a JWT with the specified scope. For each scope, the JWT is cached until 10 minutes before expiration. Locally, we mock the JWT retrieval to return a dummy JWT.

Node.js subcommands without CLI library

2022-06-02T00:00:00Z

When building a Node.js server application, most of the time we want to start the application as a server. It should continously run exposing an HTTP API and respond to incoming requests.

Beside that, we want to be able to execute the application to run a task and terminate. These are tasks that regularly run as a job in production, like database cleanup jobs. Also, these are tasks that we want to run locally, to quickly access specific functionality like retrieving a fresh JWT.

There are several powerful libraries for sophisticated CLIs like commander and docopt.

For our purposes, we want to implement a simpler approach. After all, we only want to be able to distinguish between subcommands. We don't need to provide further arguments. We are in control of the source code. If we need to configure something, we can simply adjust the code.

%%{init: {'theme': 'base', 'themeVariables': {"darkMode":true,"background":"#003890","primaryColor":"#0063b1","fontFamily":"Montserrat, sans-serif","fontSize":"1rem","lineColor":"white"} }}%%
graph TD
    A(main)
    A-->B(server)
    A-->C(send-delay-notifications)

We want to accept the commands server and send-delay-notifications. The command server should be the default.

Selecting the command in main 🔗

In the function main of the entrypoint file main.ts, we assign the third element of the input array to the variable command. If this element does not exist, we specify the default value to be server.

import { logger } from "./logger";
import { cmdServer } from "./commands/server.command";
import { cmdSendDelayNotifications } from "./commands/send-delay-notifications.command";

async function main([_node, _script, command = "server"]: string[]) {
  switch (command) {
    case "server":
      await cmdServer();
      break;
    case "send-delay-notifications":
      await cmdSendDelayNotifications();
      break;
    default:
      logger.error({ command }, "Command does not exist");
      process.exit(1);
  }
}

We switch over the variable command and execute the appropiate function. If no command matches, then we log an error and exit with a non-zero status code.

The file main.ts continues to contain default error handling for uncaught errors and promise rejections.

main(process.argv).catch((error) => {
  logger.error(error, `Uncaught error: ${error.message}`);
  process.exit(1);
});

process
  .on("unhandledRejection", (reason) => {
    logger.error(
      reason ?? {},
      `Unhandled rejection: ${(reason as Error)?.message}`
    );
  })
  .on("uncaughtException", (error) => {
    logger.error(error, `Uncaught Exception: ${error?.message}`);
    process.exit(1);
  });

Organizing commands 🔗

We create a top-level directory src/commands. Each file in this directory represents one command.

src
`-- commands
    |-- server.command.ts
    `-- send-delay-notifications.command.ts

The command server configures outgoing HTTP and spins up an HTTP server.

import * as http from "http";
import app from "../app";
import { logger } from "../logger";
import { configureHttp } from "../http/configure-http";

export async function cmdServer(): Promise<void> {
  const port = process.env.PORT ?? 3000;
  configureHttp();
  const server = http.createServer(app);
  server.listen(port, () => {
    logger.info(`Server listening on port ${port}!`);
  });
}

The command send-delay-notifications also configures outgoing HTTP. Then it determines the current date assuming the server is running in UTC. It follows a call to the function sendDelayNotifications containing the logic. At last, it explicitly exits with a zero status code to terminate the application.

import { configureHttp } from "../http/configure-http";
import { sendDelayNotifications } from "../delay-notifications/send-delay-notifications";

export async function cmdSendDelayNotifications(): Promise<void> {
  configureHttp();
  const now = new Date().toISOString().split("T")[0];
  await sendDelayNotifications(now);
  process.exit(0);
}

At this point, the logic sendDelayNotifications only logs two lines. We fill in the logic in later posts.

import { logger } from "../logger";

export async function sendDelayNotifications(
  startDate: string
): Promise<boolean> {
  logger.info({ startDate }, "Starting to send delay notifications.");
  // ...
  logger.info({ startDate }, "Finished to send delay notifications.");
  return true;
}

Convenient `npm run` commands 🔗

We want to make these commands easily accessible to all engineers. Hence, we add scripts to the package.json.

The script command allows us to execute npm run command <cmd> with a value cmd like server or send-delay-notifications. We make sure to build the application before execution.

The script pretty-logs converts the JSON log lines of Pino.js into a colourful readable format for the CLI.

We keep using nodemon for the script start:app since we want the application to rebuild after changes.

{
  // ...
  "scripts": {
    "postinstall": "tsc",
    "build": "tsc",
    "start": "npm run build && npx concurrently \"npm:start:*\"",
    "start:app": "nodemon --legacy-watch --watch ./dist --inspect=0.0.0.0:9229 dist/main.js -- server | npm run pretty-logs",
    // ...
    "command": "f() { npm run build && node dist/main.js \"$1\" | npm run pretty-logs; }; f",
    "pretty-logs": "npx pino-pretty -i time,hostname,module,__in,name,pid"
    // ...
  }
  // ...
}

Conclusion 🔗

We added the capability to run sub commands of the application. These sub commands execute a setup and then delegate to functions executing logic. The commands are easily accessible with properly formatted logs with the script npm run command <cmd>.

Custom-themed Mermaid diagrams in Eleventy

2022-06-03T00:00:00Z

Would you like to have custom-themed diagrams like this on your Eleventy blog?

%%{init: {'theme': 'base', 'themeVariables': {"darkMode":true,"background":"#003890","primaryColor":"#0063b1","fontFamily":"Montserrat, sans-serif","fontSize":"1rem","lineColor":"white"} }}%%
graph TD
    A(Add mermaid.js)-->B(Have diagrams)

Directly defining the snipped in the markdown file?

```mermaid-graph
graph TD
    A(Add mermaid.js)-->B(Have diagrams)
```

I guide you through the steps to achieve that.

%%{init: {'theme': 'base', 'themeVariables': {"darkMode":true,"background":"#003890","primaryColor":"#0063b1","fontFamily":"Montserrat, sans-serif","fontSize":"1rem","lineColor":"white"} }}%%
flowchart TD
    A(1. Load mermaid.js)
    B(2. Configure markdown highlighter)
    C(3. Add custom theme)
    D(4. Add diagrams)
    A-->B
    B-->C
    C-->D

Load mermaid.js 🔗

Somewhere in a template file in the directory _includes, you define the head element of the page. In my case, this is the Nunjucks file _includes/page.njk.

<!DOCTYPE html>
<html lang="en">
  <head>
    <!-- ... -->
    {% if mermaid %}
    <script src="/node_modules/mermaid/dist/mermaid.min.js"></script>
    <script>
      mermaid.initialize({ startOnLoad: true, theme: "base" });
    </script>
    {% endif %}
  </head>
  <!-- ... -->
</html>

Install the npm package mermaid with npm install --save mermaid. Then you can load the file mermaid.min.js in the script tag.

With the call to mermaid.initialize, you tell mermaid to render all diagrams it can find with the class mermaid. We specify the theme base, as it is the best to customize.

By enclosing the script tags into an if block, we ensure that mermaid is only loaded when the template requests it via defining mermaid as true in the front-matter.

---
layout: page.njk
mermaid: true
---

# My page

Configure markdown highlighter 🔗

There is a post that introduces the idea of adding a markdown highlighter. We add such a highlighter to the eleventy config .eleventy.js.

module.exports = function (eleventyConfig) {
  const highlighter = eleventyConfig.markdownHighlighter;
  eleventyConfig.addMarkdownHighlighter((str, language) => {
    if (language === "mermaid") {
      return `<pre class="mermaid">${str}</pre>`;
    }
    return highlighter(str, language);
  });
};

The rendering with mermaid can only happen in the browser due to its dynamic layouting. Hence, the graph code is wrapped in an element with the class mermaid for mermaid to pick it up at runtime.

Add custom theme 🔗

Custom theme variables can be overridden in individual diagrams. However, we don't want to replicate this custom theming in each diagram. Therefore, we prepend it in the markdown highlighter.

module.exports = function (eleventyConfig) {
  const highlighter = eleventyConfig.markdownHighlighter;
  eleventyConfig.addMarkdownHighlighter((str, language) => {
    if (language === "mermaid") {
      const themeVariables = JSON.stringify({
        darkMode: true,
        background: "#044965",
        primaryColor: "#05668d",
        fontFamily: "Montserrat, sans-serif",
        fontSize: "1rem",
        lineColor: "white",
      });
      const init = `%%{init: {'theme': 'base', 'themeVariables': ${themeVariables} }}%%`;
      return `<pre class="mermaid">${init + "\n" + str}</pre>`;
    }
    return highlighter(str, language);
  });
};

You can find information about the customization options in the mermaid documentation

Add diagrams 🔗

With these steps setup, you can add diagrams to your posts.

---
layout: page.njk
mermaid: true
---

# My page

```mermaid-graph
graph TD
    A(Add mermaid.js)-->B(Have diagrams)
```

Building an asynchronous batch job

2022-06-03T00:00:00Z

In the previous post, we setup the architecture for application subcommands. Now we can implement a command send-delay-notifications. This command can run locally with npm run command send-delay-notifications. It can also be triggered on a schedule in production.

In this post, we implement this command. We retrieve delayed todos from the database and send out a notification for each one. We put a special focus on the separation of this code from the endpoint logic. Also, we take a look at error handling and logging in such asynchronous jobs.

Structure 🔗

The functionality of the command send-delay-notifications is implemented in a directory src/send-delay-notifications. The command commands/send-delay-notifications.command.ts is the entry point and delegates to the function sendDelayNotifications.

%%{init: {'theme': 'base', 'themeVariables': {"darkMode":true,"background":"#003890","primaryColor":"#0063b1","fontFamily":"Montserrat, sans-serif","fontSize":"1rem","lineColor":"white"} }}%%
graph TD
    A(cmdSendDelayNotifications)
    B(sendDelayNotifications)
    C(getDelayedTodos)
    D(sendDelayNotification)
    E(sendNotification)
    A-->B
    B-->C
    B-->D
    D-->E

Retrieving delayed todos 🔗

In the top-level directory src/delay-notifications, we create a file delayed-todo.dao.ts. This data access module contains a Knex SQL query to retrieve all the delayed todos.

Note There is already a file src/controllers/todos/todo.dao used in the controllers. We don't add functions to that data access module nor do we make it more generic to use it. Instead we create this new data access module delayed-todo.dao. The controllers and this job only share the schema in the database, not a data access module. This ensures that future changes in code can only affect and break one functionality at a time.

import { db } from "../database/db";
import { DbTodo } from "./todo.type";

export function getDelayedTodos(
  now: string
): Promise<Pick<DbTodo, "id" | "workspaceId">[]> {
  return db
    .table<DbTodo>("todos")
    .where("dueDate", "<=", now)
    .select("id", "workspaceId");
}

The data access module retrieves delayed todos across all workspaces. We can retrieve them across workspaces since we didn't separate the workspaces into different database schemas. Instead, workspaces are represented with a column workspaceId.

The returned todos only contain the fields id and workspaceId. We reflect this in a type DbTodo in a file src/delay-notifications/todo.type.

import { WithWorkspaceId } from "../workspace-id.type";

export type TodoId = string;

export interface Todo {
  id: TodoId;
  dueDate: string;
}

export type DbTodo = Todo & WithWorkspaceId;

Note The type Todo is also defined by the controllers in a file src/controllers/todos/todo.type. However, the controllers need different fields and in future posts, the todos of the functionality send-delay-notifications need a field that the controllers don't need. Like for the data access module, only the database schema is shared. We don't share common types between different functionalities.

Notification service 🔗

There exists an API endpoint POST /notify/notifications of a different service notify. We implement a service src/delay-notifications/notification.service that implements a TypeScript interface for the relevant options.

import axios, { AxiosRequestConfig } from "axios";
import { config } from "../configuration/config";
import { workspaceAccess } from "../http/service-access.type";
import { WorkspaceId } from "../workspace-id.type";

export function sendNotification(
  workspaceId: WorkspaceId,
  message: string
): Promise<void> {
  return axios.post(
    `${config.http.servicesUrl}/notify/notifications`,
    { message },
    {
      access: workspaceAccess(workspaceId),
    } as AxiosRequestConfig
  );
}

Sending a single delay notification 🔗

We add a module send-delay-notification.ts to send a single delay notification. In addition to what the notification service provides, it encapsulates error handling.

In comparison to endpoints, error handling and logging need to be used more carefully in batch jobs. Since endpoints are executed synchronously, a user is waiting for a response. If something goes wrong, an error can be reported to that user.

This is different in asynchronous jobs. There is no user waiting for a response and hence there is no user to report an error to. A user that is waiting synchronously for a reponse has context about their request. Somebody investigating a failure of an asynchronous job is missing this context. Hence, it is important to provide this context via logging and specific error handlers.

The function sendDelayNotification handles the common error of a failing API request. In that case, it logs the error, describes how it will continue and provides an identifier of the skipped entity.

import { logger } from "../logger";
import { sendNotification } from "./notification.service";
import { DbTodo } from "./todo.type";

export async function sendDelayNotification(
  todo: Pick<DbTodo, "id" | "workspaceId">
): Promise<boolean> {
  try {
    await sendNotification(todo.workspaceId, `Todo ${todo.id} is delayed`);
    return true;
  } catch (error) {
    logger.error(
      {
        workspaceId: todo.workspaceId,
        todoId: todo.id,
        error,
      },
      "Could not send delayed todo. Skipping it..."
    );
    return false;
  }
}

Assembling everything together 🔗

In the function sendDelayNotifications, we retrieve the delayed todos and send a notification for each of them. If the retrieval of the delayed todos fails, we log an error and abort the job.

import { logger } from "../logger";
import { getDelayedTodos } from "./delayed-todo.dao";
import { sendDelayNotification } from "./send-delay-notification";

export async function sendDelayNotifications(
  startDate: string
): Promise<boolean> {
  logger.info({ startDate }, "Starting to send delay notifications.");
  const todos = await getDelayedTodos(startDate).catch((error) => {
    logger.error(
      { startDate, error },
      "Could not retrieve delayed todos from the database. Aborting job..."
    );
    return null;
  });
  if (!todos) {
    return false;
  }
  for (const todo of todos) {
    await sendDelayNotification(todo);
  }
  logger.info({ startDate }, "Finished to send delay notifications.");
  return true;
}

Note GitHub sponsors have access to tests for the functions sendDelayNotifications, sendDelayNotification and getDelayedTodos in the GitHub repo.

Adding delayed todos in the local database 🔗

In an earlier post, we added a Knex.js seed to populate the local database with example data.

For more information on this, checkout 'Knex.js schema migrations with Postgres in Node.js'

We adjust this seed to contain two delayed todos. The todo of Peter now has a due date of 2020-06-11 and the todo of Ousmane has a due date of 2020-07-14.

import { Knex } from "knex";
import { DbTodo } from "../../controllers/todos/todo.type";
import { WorkspaceId } from "../../workspace-id.type";

const workspaceId1: WorkspaceId = "a9393008-eab2-48e8-b820-0e03447f881c";
const workspaceId2: WorkspaceId = "83fdfa88-737d-4550-87e4-c78bf954fcf2";

const todos: DbTodo[] = [
  {
    workspaceId: workspaceId1,
    id: "142e8446-f5ba-4813-bfb5-f3192a37f1bf",
    name: "Workspace 1 - Todo 1",
    assignee: "Naomi",
    dueDate: "2025-01-05",
  },
  {
    workspaceId: workspaceId1,
    id: "bf6091cc-69f8-4a9e-93ed-0294c3a8ac2f",
    name: "Workspace 1 - Todo 2",
    assignee: "Peter",
    dueDate: "2020-06-11",
  },
  {
    workspaceId: workspaceId1,
    id: "6e5bad4c-7f1c-4ab4-9361-c6bd697b4257",
    name: "Workspace 1 - Todo 3",
    assignee: "Sheila",
    dueDate: "2032-11-27",
  },
  {
    workspaceId: workspaceId2,
    id: "1d2be9f2-4fe9-4509-b8ec-d0d9425c3685",
    name: "Workspace 2 - Todo 1",
    assignee: "Ousmane",
    dueDate: "2020-07-14",
  },
  {
    workspaceId: workspaceId2,
    id: "e868f10d-d21a-4139-9c5d-b8c73c62735a",
    name: "Workspace 2 - Todo 2",
    assignee: "Carla",
    dueDate: "2034-09-07",
  },
];

export async function seed(knex: Knex): Promise<void> {
  await knex("todos").truncate();
  await knex("todos").insert(todos);
}

Running it 🔗

With npm start, we spin up the database and the mockserver for the notify service. We can use npm run command send-delay-notifications to execute our implemented logic locally.

$ npm run command send-delay-notifications

INFO: Starting to send delay notifications.
    startDate: "2022-06-03"
DEBUG: Request
    method: "POST"
    url: "http://localhost:3001/notify/notifications"
    body: {
      "message": "Todo bf6091cc-69f8-4a9e-93ed-0294c3a8ac2f is delayed"
    }
DEBUG: Request
    method: "POST"
    url: "http://localhost:3001/auth/token"
    body: ""
INFO: Response
    status: 200
    method: "POST"
    url: "http://localhost:3001/auth/token"
DEBUG: Response Details
    status: 200
    method: "POST"
    url: "http://localhost:3001/auth/token"
    body: {
      "access_token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwiY2xpZW50SWQiOiJ0b2RvcyIsImlhdCI6MTUxNjIzOTAyMn0.YD_HJROnQHJIROzVBagK27D862QoNvokSDFxDbA8_Ug"
    }
INFO: Response
    status: 200
    method: "POST"
    url: "http://localhost:3001/notify/notifications"
DEBUG: Response Details
    status: 200
    method: "POST"
    url: "http://localhost:3001/notify/notifications"
    body: {
      "message": "Notification created"
    }
DEBUG: Request
    method: "POST"
    url: "http://localhost:3001/notify/notifications"
    body: {
      "message": "Todo 1d2be9f2-4fe9-4509-b8ec-d0d9425c3685 is delayed"
    }
DEBUG: Request
    method: "POST"
    url: "http://localhost:3001/auth/token"
    body: ""
INFO: Response
    status: 200
    method: "POST"
    url: "http://localhost:3001/auth/token"
DEBUG: Response Details
    status: 200
    method: "POST"
    url: "http://localhost:3001/auth/token"
    body: {
      "access_token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwiY2xpZW50SWQiOiJ0b2RvcyIsImlhdCI6MTUxNjIzOTAyMn0.YD_HJROnQHJIROzVBagK27D862QoNvokSDFxDbA8_Ug"
    }
INFO: Response
    status: 200
    method: "POST"
    url: "http://localhost:3001/notify/notifications"
DEBUG: Response Details
    status: 200
    method: "POST"
    url: "http://localhost:3001/notify/notifications"
    body: {
      "message": "Notification created"
    }
INFO: Finished to send delay notifications.
    startDate: "2022-06-03"

We see that the endpoint POST /notify/notifications is called two times. Before these calls, the configured Axios instance retrieves and attaches JWTs. Since both todos are in different workspaces, two different JWT are retrieved and the cache is not utilized.

Conclusion 🔗

We started to implement the asynchronous batch job send-delay-notifications. The job retrieves all delayed todos and sends a notifications for each of them. In the following posts, we make the job idempotent to avoid duplicate notifications on repeated runs. Also, we implement paging for the delayed todos to avoid exhaustion of the memory.

Showing Plausible view counts in an Eleventy blog

2022-06-08T00:00:00Z

So you have your statically generated Eleventy blog running, Plausible Analytics setup to track its traffic and now want to show and utilize the number of views of your blog posts? In this post, I show you how to get the page views from Plausible into your blog while keeping everything nicely statically generated at build time.

Steps 🔗

Retrieve and provide Plausible stats as Eleventy global data
Plausible API authentication
Setup a daily build
Show page views on posts
Sort posts with page views

Tradeoffs 🔗

There are multiple architectural approaches to integrate the page views.

Retrieve page views from Plausible in the browser
- Pros User sees latest data
- Cons Requires client-side JavaScript, runtime dependency on Plausible API
Retrieve and cache page views (as JSON in git) in a scheduled job
- Pros Build and deploy process independent of Plausible API
- Cons Less frequently updated data, build command acts on cache
Retrieve and cache page views before deployment
- Pros Latest page views on deployment
- Cons Deployment depends on Plausible API availability
Retrieve page views before each build
- Pros Latest page views on deployment, latest page views locally
- Cons Deployment and local development depends on Plausible API availability

In this post I am presenting the retrieval of page views before each build. Additionally, we setup a daily build. This way, page views update also if there is no manual deployment.

Retrieve and provide Plausible stats as Eleventy global data 🔗

Eleventy allows to compute data for the templates dynamically. We create a global data file _data/pageStats.js. The exported value from this module is made available in the templates via <post>.data.pageStats. We want this global variable pageStats to be a map from page URLs to the page stats.

{
  "/posts/post-1": {
    "pageviews": 1784
  },
  "/posts/node/post-3": {
    "pageviews": 8463
  }
}

Hence, in the file pageStats.js, we first retrieve the page stats and then format them.

const axios = require("axios");

module.exports = async () => {
  const response = await requestPagestats();
  return formatResponse(response);
};

function requestPagestats() {
  // ...
}

function formatResponse(response) {
  // ...
}

In the function requestPagestats, we use axios to request the page stats. The Plausible API endpoint /stats/breakdown returns statistics per page.

We pass it several parameters:

The site_id is the domain of your site.
The period should ideally be all, but is limited to 12mo maximum currently in the API.
The metrics is a comma-separated list. We are interested in the pageviews.
The property allows us to specify based on what to breakdown the stats. In our case, this is the page url event:page.
The filters allow us to only retrieve posts by matching the URL.
The Authorization header contains a Bearer authentication token which we setup in the next section.

function requestPagestats() {
  return axios.get(`https://plausible.io/api/v1/stats/breakdown`, {
    params: {
      site_id: "traveling-coderman.net",
      period: "12mo",
      metrics: "pageviews",
      property: "event:page",
      filters: "event:page==/posts/**/*",
    },
    headers: {
      Authorization: `Bearer ${process.env.PLAUSIBLE_TOKEN}`,
    },
  });
}

The response of the Plausible API is a list of the post URLs with their page views.

{
  "results": [
    {
      "page": "/posts/post-1",
      "pageviews": 1784
    },
    {
      "page": "/posts/node/post-3",
      "pageviews": 8463
    }
  ]
}

We transform the Plausible API response into our desired format.

function formatResponse(response) {
  return response.data.results.reduce(
    (acc, page) => ({
      ...acc,
      [page.page]: {
        pageviews: page.pageviews,
      },
    }),
    {}
  );
}

Plausible API authentication 🔗

In the Plausible settings, you can setup API tokens.

I recommend to setup two distinct tokens for local development and your production deployment pipeline. This way, you can revoke them independently in case of a security incident. If you work in a team, then setup tokens for each engineer.

Note I am hosting on Netlify. The retrieval of the page stats works independently of the hosting provider. You just need to make sure to make the environment variable PLAUSIBLE_TOKEN accessible to the Eleventy build command.

Passing the token locally 🔗

We use dotenv to provide the token locally to the Eleventy build.

npm install --save-dev dotenv

We create a file .env with the environment variable PLAUSIBLE_TOKEN. Replace <...> with your created development token.

PLAUSIBLE_TOKEN=<...>

We exclude the file .env from being committed in git via the .gitignore.

// ...
.env

In the Eleventy config .eleventy.js, we load dotenv after the imports but before any code execution.

// ... imports ...

require("dotenv").config();

module.exports = function (eleventyConfig) {
  // ...
};

Passing the token in the deployment 🔗

This step is specific to the hosting provider. In Netlify, you can configure an environment variable to be passed to each deployment of your site. Use the created Plausible production API token.

Setup a daily build 🔗

We want the latest daily page stats to be visible on the website.

This step is also specific to the hosting provider. In Netlify, there are currently no scheduled builds. However, Netlify offers build hooks.

Once we create a build hook, we can pass it to external tools to call it on a schedule. There are multiple options available to achieve this. My repository is hosted on GitHub. Hence, I am using GitHub Actions to avoid the introduction of a new third-party dependency.

In the folder .github/workflows, we create a file daily-netlify-build.yml.

name: Daily Netlify Build

on:
  schedule:
    # Run at 15:30 daily
    - cron: "30 15 * * *"
  workflow_dispatch:

jobs:
  build:
    name: Daily Netlify Build
    runs-on: ubuntu-latest
    steps:
      - name: Call Netlify build hook
        run: curl -X POST -d {} $

This workflow is executing each day at 15:30 and calls the Netlify build hook. Additionally, we add the option workflow_dispatch. This allows for testing of this workflow without waiting for 15:30.

Since the build hook URL is sufficient to authenticate, we store it as a secret in GitHub.

Show page views on posts 🔗

With the previous steps, the pageStats variable is available in templates. From this point, you can use it in pretty much any way you like. Probably the most common scenario is to show the page views on the individual posts.

In the individual post template, we access and render the page views.

{% if pageStats[page.url].pageviews %}
<span class="pageviews">
  <i class="fas fa-eye" aria-hidden="true"></i>
  <a
    href="https://plausible.io/traveling-coderman.net?page={{ page.url }}&period=all"
    target="_blank"
  >
    {{ pageStats[page.url].pageviews if pageStats[page.url].pageviews
    >= 100 else '<100' }} views
    <i class="fas fa-external-link-alt"></i>
  </a>
</span>
{% endif %}

In the variable pageStats, we can lookup the page stats with the URL of the page page.url. Since the page might be new and not be present in the Plausible stats, we check if the value is set.

Font Awesome provides an eye icon . We add aria-hidden="true" since the word views describes the span.

We link to the specific Plausible stats of the page https://plausible.io/<your_domain>?page={{ page.url }}&period=all.

Note While the period all exists in this link, it is not supported in the Plausible API at the time of publishing this post.

We show the page views with an interpolation {{ pageStats[page.url].pageviews if pageStats[page.url].pageviews >= 100 else '<100' }}. This leads to the page views being shown as <100 if there are less than 100 views.

The Font Awesome icon fa-external-link-alt indicates the external link. With a bit of CSS, we limit it to be shown on hover.

.pageviews .fa-external-link-alt {
  display: none;
}

.pageviews a:hover .fa-external-link-alt {
  display: inline-block;
}

Sort posts with page views 🔗

Besides showing the number of page views, we can use them to define the order of displayed posts.

In the Eleventy config .eleventy.js, we can create new collections with custom sorting. I am using a tag post on each template file that represents a post. Additionally, I use other tags to further distinguish posts into "categories".

module.exports = function (eleventyConfig) {
  const tags = ["post", "coding", "process" /* ... */];
  for (const tag of tags) {
    eleventyConfig.addCollection(`${tag}-popular`, (collectionApi) =>
      collectionApi.getFilteredByTag(tag).sort(byPageviews)
    );
  }
};

The function byPageviews compares two posts according to their number of page views.

function byPageviews(post1, post2) {
  return (
    post2.data.pageStats[post2.url]?.pageviews -
    post1.data.pageStats[post1.url]?.pageviews
  );
}

Note While the page stats pageStats are defined globally, they are also available to each post in the object data. The way we defined it, post1.data.pageStats is equal to post2.data.pageStats.

We can use these collections in templates.

{% for post in collections["post-popular"] %}
<!-- ... -->
{% endif %}

Conclusion 🔗

We setup the retrieval of Plausible API page stats into our Eleventy build. These stats are retrieved locally and on deployment builds. We setup a daily build such that the production site shows recent data. We rendered the page views on each post and sorted posts according to their page views.

Building an idempotent cron job with at-least-once delivery

2022-06-09T00:00:00Z

In the previous post, we implemented a batch job to send a notification for each delayed todo of the Postgres database. So far, on each execution, this batch job sends another notification for each delayed todo. Instead, we want to achieve that the notification for a delayed todo is not send again after a successful delivery. In this post, we implement such an idempotent cron job with at-least-once delivery.

At-least-once delivery 🔗

Note the difference between at-least-once, exactly-once and at-most-once delivery.

There can always be errors in network communication. Even worse, in a network outage, it can't even be identified if the other side has received and processed the message or not. Therefore, it is technically challenging to achieve exactly-once delivery.

Instead, it is more straightforward to either go for at-least-once delivery or at-most-once delivery. For the use case of our notifications, both options are viable. We go for at-least-once delivery. This way, we can guarantee a user to receive a notification, but we can not guarantee that there are no duplicated notifications.

Persisting if a notification has been sent 🔗

In the existing table todos, we create a column delayNotificationSent with a Knex.js schema migration. It persists a boolean variable. It is set to false on creation since the notification has not yet been sent.

Note We are mixing user-provided data (assignee, due date, etc.) with internal data (delay notification sent) in the database. It is the purpose of the data access modules to act on the relevant subset of columns. One could separate into two tables todos and notifications. However, although these would have a one-to-one relation, we would need joins to query over both. It is good to avoid that complexity.

import { Knex } from "knex";

export async function up(knex: Knex): Promise<void> {
  await knex.schema.alterTable("todos", (table) => {
    table.boolean("delayNotificationSent").defaultTo(false).notNullable();
  });
}

export async function down(knex: Knex): Promise<void> {
  await knex.schema.alterTable("todos", (table) => {
    table.dropColumn("delayNotificationSent");
  });
}

Filtering for todos without sent notification 🔗

The function getDelayedTodos in the data access module delay-notifications/delayed-todos.dao.ts returns the list of delayed todos. With the new column being created, we can add a WHERE clause filtering for todos with the value false in the column delayNotificationSent.

import { db } from "../database/db";
import { WorkspaceId } from "../workspace-id.type";
import { DbTodo, TodoId } from "./todo.type";

export function getDelayedTodos(
  now: string
): Promise<Pick<DbTodo, "id" | "workspaceId">[]> {
  return db
    .table<DbTodo>("todos")
    .where("dueDate", "<=", now)
    .andWhere("delayNotificationSent", false)
    .select("id", "workspaceId");
}

Marking a todo as processed 🔗

In the same data access module, we create a function setDelayNotificationSent. This function updates the column delayNotificationSent to true.

import { db } from "../database/db";
import { WorkspaceId } from "../workspace-id.type";
import { DbTodo, TodoId } from "./todo.type";

export async function setDelayNotificationSent(
  workspaceId: WorkspaceId,
  todoId: TodoId
): Promise<void> {
  await db
    .table<DbTodo>("todos")
    .where("id", todoId)
    .andWhere("workspaceId", workspaceId)
    .update("delayNotificationSent", true);
}

Typing the new column 🔗

We create dedicated types Todo and DbTodo for the send-delay-notifications functionality. We add the column delayNotificationSent to be part of these todos. The columns dueDate and assignee are not relevant in the context of the job send-delay-notifications. They are therefore left out.

import { WithWorkspaceId } from "../workspace-id.type";

export type TodoId = string;

export interface Todo {
  id: TodoId;
  dueDate: string;
  delayNotificationSent: boolean;
}

export type DbTodo = Todo & WithWorkspaceId;

Sending a notification and marking the todo 🔗

We assemble everything in the function sendDelayNotification. It is rather long due to error handling. Nevertheless, it only does two things.

Send a notification for the todo with sendNotification.
Mark the todo as processed with setDelayNotificationSent.

import { logger } from "../logger";
import { setDelayNotificationSent } from "./delayed-todo.dao";
import { sendNotification } from "./notification.service";
import { DbTodo } from "./todo.type";

export async function sendDelayNotification(
  todo: Pick<DbTodo, "id" | "workspaceId">
): Promise<boolean> {
  try {
    await sendNotification(todo.workspaceId, `Todo ${todo.id} is delayed`);
    try {
      await setDelayNotificationSent(todo.workspaceId, todo.id);
      return true;
    } catch (error) {
      logger.error(
        {
          workspaceId: todo.workspaceId,
          todoId: todo.id,
          error,
        },
        "Could not mark in the database that the notification for this todo has been sent. The notification will be send another time in the next run."
      );
      return false;
    }
  } catch (error) {
    logger.error(
      {
        workspaceId: todo.workspaceId,
        todoId: todo.id,
        error,
      },
      "Could not send delayed todo. Skipping it..."
    );
    return false;
  }
}

In this function, we can observe the implementation of at-least-once delivery. It is defined by the order of the calls to sendNotification and setDelayNotificationSent. If we first marked the todo and then sent the notification, we would have at-most-once delivery.

With this implementation, a user could receive two notifications in the following scenario.

We call the endpoint POST /notifications of the service notify.
The service notify creates the notification and tries to respond with a success 200.
The network fails.
After a time of x seconds, Axios errors with a timeout.
We don't mark the todo as processed.
In a subsequent run of the job, the todo is picked up again and a new notification is sent.

Note Exactly-once delivery could be achieved if the creation of notifications itself would be idempotent. This would require us to pass a unique identifier for a created notification like delayed-<todoId>. Then, the service notify would not create another notification, but update the existing notification. The endpoint POST /notify/notifications would then rather be PUT /notify/notifications/:id.

Conclusion 🔗

We implemented at-least-once delivery of notifications in an idempotent cron job. The job can be run as often as desired and only sends notifications for todos if they have not been successfully sent before.

Why Microfrontends?

2022-06-10T00:00:00Z

Microfrontends are a technique to eliminate or reduce dependencies between multiple frontend teams. The assumption is that work within teams is faster than work between teams. Hence, if we eliminate dependencies between teams, then there is more work within teams and less work between teams. As a result, the teams are faster.

Delivery pipeline with frontend monolith:

%%{init: {'theme': 'base', 'themeVariables': {"darkMode":true,"background":"#003890","primaryColor":"#0063b1","fontFamily":"Montserrat, sans-serif","fontSize":"1rem","lineColor":"white"} }}%%
graph LR
  B1(Team 1: Build)
  B2(Team 2: Build)
  D(Team 1 + Team 2: Deploy)
  UI(Team 1 + Team 2: UI)

  B1-->D
  B2-->D
  D-->UI

Delivery pipeline with microfrontends:

%%{init: {'theme': 'base', 'themeVariables': {"darkMode":true,"background":"#003890","primaryColor":"#0063b1","fontFamily":"Montserrat, sans-serif","fontSize":"1rem","lineColor":"white"} }}%%
graph LR
  B1(Team 1: Build)
  D1(Team 1: Deploy)
  B2(Team 2: Build)
  D2(Team 2: Deploy)
  UI(Team 1 + Team 2: UI)

  B1-->D1-->UI
  B2-->D2-->UI

Therefore, microfrontends can only be valuable in a setting with multiple teams. In a small startup or a company with few frontend work, microfrontends don't provide value. At the same time, they introduce complexity. If you think about introducing microfrontends, I would therefore recommend to take a look at the amount of complexity introduced vs. the amount of decoupling achieved. With the YouTube video and these accompanying blog posts, you will get a sense of both the complexity and the value provided to be able to make a call.

What are advantages of microfrontends? 🔗

Let's take a look what this "decoupling of teams" actually means.

Independent deployment pipelines If you use a frontend monolith, then this monolith builds and deploys as a whole. If a deployment fails and the pipeline needs to be repaired, then this is blocking all frontend teams. If more frontend teams are created, then the monolithic deployment pipeline slows down for all teams. Microfrontends utilize independent deployment pipelines. If a deployment fails, then other teams are not blocked. If a new frontend team is created, then the speed of the individual deployment pipelines is not affected.
Freedom from existing codebase If you use a frontend monolith, then some technologies are set in stone. Frameworks like Angular, but also libraries like RxJS or xstate, and style libraries like TailwindCSS or Bootstrap. Microfrontends create a safe space to experiment. A Svelte microfrontend can be integrated into an Angular platform, the Bootstrap platform styles remain unaffected by the TailwindCSS styles of a microfrontend.
Vertical services If you think about using microfrontends, you likely already started using microservices. Both techniques combined allow teams to be responsible for a domain. Since users work and think in these domains rather than frontend and backend, the work of the teams matches the domains of the users more closely. As a result, teams can relate better to users and anticipate their interactions.

%%{init: {'theme': 'base', 'themeVariables': {"darkMode":true,"background":"#003890","primaryColor":"#0063b1","fontFamily":"Montserrat, sans-serif","fontSize":"1rem","lineColor":"white"} }}%%
graph TD
  A1(Team 1: Frontend)
  B1(Team 1: Backend)
  C1(Team 1: Database)
  A2(Team 2: Frontend)
  B2(Team 2: Backend)
  C2(Team 2: Database)
  D(Team 3: Infrastructure)

  A1-->B1-->C1-->D
  A2-->B2-->C2-->D

What are disadvantages of microfrontends? 🔗

Every technique has its tradeoffs. Microfrontends introduce complexity to achieve decoupling.

Runtime integration complexity A monolith is naturally integrated with itself. Microfrontends need to have their independent deployment location, routing to be on the same domain, loading at runtime, etc. These things require thought and time. We hope that the video on the series help you to reduce that thought and time. However, it is still more than with a monolith.
UI integration complexity Users should not notice that they are working with microfrontends. Overlays, localization, theming, bookmarks and other aspects should just work. With web-component based microfrontends this can be achieved. But while theming and overlays can be achieved more easily, localization and routing for bookmarks are still achieved less easily.
Bundle sizes A web-component based Angular microfrontend weighs 1MB. If used heavily, you might need to invest more time and introduce more complexity to reduce this footprint.

Why based on web components and not iframes? 🔗

Another way of integrating independent frontends are iframes. While iframes allow a strong decoupling, they decouple so much that overlays and theming become an additional effort to couple again. Web components couple a micro frontend more tightly to the platform, reducing the effort for some aspects.

When should you use microfrontends? 🔗

If you are unsure if you need microfrontends, these are indicators that you might want to think about them.

Slow deployments Individual deployments take a lot of time. Engineers are delaying their merges or deployments because there are ongoing deployments. They bundle multiple changes together into a deployment to reduce the time until the changes are in production.
Blocked deployment pipelines Engineers are waiting a substantial amount of time for engineers of other teams to fix a blocked deployment pipeline.
Unclear responsibilites There is code and there are tests in the frontend codebase that nobody feels responsible for. If a deployment fails, then it is unclear which team is able to resolve it. There exist tests that can fail due to changes of code of different teams.

Conclusion 🔗

Introducing microfrontends involves a tradeoff between decoupling and complexity. They require a less complex integration than iframes to achieve the same user experience as a monolith. But they entail more complexity than a monolith at the advantage of more decoupled and therefore faster teams.

I hope we could show you why and when it makes sense to think about adopting microfrontends based on web components.

Rendering a file structure with Prism in Eleventy

2022-06-15T00:00:00Z

In this post, I show you how I use prism-treeview to render file system structures in an Eleventy blog.

src
|-- index.njk
`-- assets
    `-- portrait.png

While keeping the mark up simple.

src/
|-- index.njk
`-- assets/
    `-- portrait.png

Prism Treeview 🔗

The Prism plugin prism-treeview extends Prism with a language treeview to display file system structures. The JavaScript file registers the language and defines the logic how to convert the markup into HTML. The Stylesheet file includes the formatting.

_includes
`-- treeview.css
11ty
`-- prism-treeview.js
.eleventy.js

In the Eleventy config .eleventy.js, I load the JavaScript file require("./11ty/prism-treeview.js");. The Eleventy Docs explain how to setup general syntax highlighting.

Reducing the footprint of the CSS 🔗

The stylesheet file treeview.css has a footprint of 17.6 KB. This is due to the icons it directly embeds.

Since I already load Font Awesome icons on every page, I changed the CSS file to use these icons instead. Font Awesome describes how to use icons in pseudo elements.

// ...

.treeview-part .entry-name:before {
  display: inline-block;
  font-style: normal;
  font-variant: normal;
  text-rendering: auto;
  -webkit-font-smoothing: antialiased;
  font-family: "Font Awesome 5 Free";
  color: white;
  font-weight: 900;
}

.treeview-part .entry-name[class*="ext-"]:before {
  content: "\f15b";
}

.treeview-part .entry-name.dir:before {
  content: "\f07b";
}

.treeview-part .entry-name:is(.ext-ts, .ext-js, .ext-njk, .ext-css):before {
  content: "\f1c9";
}

.treeview-part .entry-name.ext-js:before {
  font-family: "Font Awesome 5 Brands";
  content: "\f3b9";
}

.treeview-part .entry-name.ext-css:before {
  font-family: "Font Awesome 5 Brands";
  content: "\f38b";
}

.treeview-part .entry-name.ext-png:before {
  content: "\f03e";
}

If nothing else is specified .entry-name[class*="ext-"], then the file icon \f15b is used. Every class like .dir and .ext-ts overrides the default to use a different Font Awesome 5 icon. To use brand icons for CSS and JavaScript, the font family needs to be adjusted to Font Awesome 5 Brands.

With this technique, the CSS file has a size of 1.3KB.

Appendix: Full CSS file 🔗

For reference, this is the full CSS file. The JS file I used unchanged.

.treeview-part .entry-line {
  position: relative;
  text-indent: -99em;
  display: inline-block;
  vertical-align: top;
  width: 1.2em;
}

.treeview-part .entry-line:before,
.treeview-part .line-h:after {
  content: "";
  position: absolute;
  top: 0;
  left: 50%;
  width: 50%;
  height: 100%;
}

.treeview-part .line-h:before,
.treeview-part .line-v:before {
  border-left: 1px solid #ccc;
}

.treeview-part .line-v-last:before {
  height: 50%;
  border-left: 1px solid #ccc;
  border-bottom: 1px solid #ccc;
}

.treeview-part .line-h:after {
  height: 50%;
  border-bottom: 1px solid #ccc;
}

.treeview-part .entry-name {
  position: relative;
  display: inline-block;
  vertical-align: top;
  padding: 0 0 0 1.8em;
}

.treeview-part .entry-name:before {
  content: "";
  position: absolute;
  top: 0;
  left: 0.25em;
  height: 100%;
  width: 20px;
  background: no-repeat 50% 50% / contain;
  display: inline-block;
  font-style: normal;
  font-variant: normal;
  text-rendering: auto;
  -webkit-font-smoothing: antialiased;
  font-family: "Font Awesome 5 Free";
  color: white;
  font-weight: 900;
}

.treeview-part .entry-name[class*="ext-"]:before {
  content: "\f15b";
}

.treeview-part .entry-name.dir:before {
  content: "\f07b";
}

.treeview-part .entry-name:is(.ext-ts, .ext-js, .ext-njk, .ext-css):before {
  content: "\f1c9";
}

.treeview-part .entry-name.ext-js:before {
  font-family: "Font Awesome 5 Brands";
  content: "\f3b9";
}

.treeview-part .entry-name.ext-css:before {
  font-family: "Font Awesome 5 Brands";
  content: "\f38b";
}

.treeview-part .entry-name:is(.ext-png):before {
  content: "\f03e";
}

Injecting an HTML snippet into every 11ty post

2022-07-21T00:00:00Z

If you render your post with {{ content | safe }} in your layout, then it shows up as a continuous post without interruption. You can render additional markup before or after the post with Before post {{ content | safe }} After post. However, in some cases, you might want to show additional markup within the post itself. This way, it catches the attention after the visitor decided to read the post but before they fully read the post.

In this post, I show how you can inject such additional markup at a specified position into each post.

Injecting a snippet 🔗

Instead of rendering the content directly with {{ content | safe }}, we assign the additional markup to a variable and pass it and the content to a filter inject.

{% set snippet %}
  {% include "_includes/my-snippet.njk" %}
{% endset %}

{{ content | inject(snippet) | safe }}

This filter inject takes the content and the snippet and injects the snippet at a specific position into the content.

Since the content is a string with HTML, we need to parse it. We use the library cheerio for that. It allows us to parse, manipulate and serialize HTML.

npm install --save-dev cheerio

The inject filter 🔗

In the eleventy config .eleventy.js, we import cheerio and define the filter inject. It consists of three steps: Parse, manipulate and serialize.

const cheerio = require("cheerio");

module.exports = function (eleventyConfig) {
  // ...
  eleventyConfig.addFilter("inject", (content, injection) => {
    const $ = cheerio.load(content);
    $("body > h2:nth-of-type(3)").before(injection);
    return $.html($("body"));
  });
  // ...
};

By default, cheerio adds a body around the HTML. To ensure the same output structure as input structure, we instruct cheerio to serialize the content of the body tag with $.html($("body")). In the manipulation, we use the existence of the body tag to only select the direct descendents of it. For my use case, I selected the third headline in each post and instructed cheerio to add the additional markup in front of it.

With these steps, you will get an injected HTML snippet into every post.

Pitfall: Asynchronous short code 🔗

With the usage of include within a variable assignment block set, you need to use synchronous Nunjucks filters within the snippet my-snippet.njk.

{% set snippet %}
  {% include "_includes/my-snippet.njk" %}
{% endset %}

In my use case, this was relevant to render an optimized image.

{% illustration "./assets/my-illustration.svg" %}
  <span>...</span>
  <ul>...</ul>

The Eleventy documentation includes an example of how to define a synchronous short code for images.

Best of Lodash

2023-06-02T00:00:00Z

Lodash has a lot of functions.

Some are now present natively and some are not that useful on a regular basis.

Here, I have put together a couple of helpful functions for working with collections and objects that I use on a regular basis.

Example data definitions 🔗

const what = {
  label: "What?",
  section: "General",
  answer: "Yes",
};
const where = {
  label: "Where?",
  section: "General",
  answer: "No",
};
const when = {
  label: "When?",
  section: "Time",
  answer: "Maybe",
};
const which = {
  label: "Which?",
  section: undefined,
  answer: "Yes",
};

Helpful functions with example 🔗

toPairs({ "What?": what, "Where?": where });
// => [['What?', what], ['Where?', where]]

fromPairs([
  ["What?", what],
  ["Where?", where],
]);
// => { 'What?': what, 'Where?': where }

mapKeys((label) => label.toUpperCase(), { "What?": what, "Where?": where });
// => { 'WHAT?': what, 'WHERE?': where }

mapValues(
  (question) => ({ ...question, section: question.section ?? "Other" }),
  { "What?": what, "Which?": which }
);
// => { 'What?': what, 'Which?': { label: 'Which?', section: 'Other', answer: 'Yes' } }

omit(["section"], what);
// => { label: 'What?', answer: 'Yes' }

pick(["label", "answer"], what);
// => { label: 'What?', answer: 'Yes' }

differenceBy((question) => question.label, [what, where, which], [which]);
// => [what, where]

intersectionBy(
  (question) => question.label,
  [what, where, which],
  [what, when]
);
// => [what]

unionBy((question) => question.label, [what, where], [what, when]);
// => [what, where, when]

uniqBy((question) => question.label, [what, where, what]);
// => [what, where]

sortBy((question) => question.label, [what, where, when, which]);
// => [what, when, where, which]

groupBy((question) => question.section ?? "Other", [what, where, when, which]);
// => { General: [what, where], Time: [when], Other: [which] }

partition((question) => question.answer === "Yes", [what, where, when, which]);
// => [[what, which], [where, when]]

dropRight(2, [what, where, when, which]);
// => [what, where]

last([what, where, when, which]);
// => which

Declaring over updating RxJS observables

2023-06-02T00:00:00Z

With RxJS Observables, there are two ways to compute new observables from existing ones.

Updating 🔗

In the first way, you update the computed observable each time one of the sources change.

Let’s take the example of addition. Two numbers exist that can each change independently. A third number should be computed as the sum of both numbers.

In an updating approach, we emit a new sum value each time one of the sources change. With two numbers, that is still realistic. If an observable is computed though from a variety of different sources, then it can be easy to forget emitting a new value for the computed observable.

number1$ = new BehaviorSubject<number>(0);
number2$ = new BehaviorSubject<number>(0);
sum$ = new BehaviorSubject<number>(0);

incrementFirstNumber() {
  number1$.next(number1$.value + 1);
  sum$.next(number1$.value + number2$.value);
}

incrementSecondNumber() {
  number2$.next(number2$.value + 1);
  sum$.next(number1$.value + number2$.value);
}

Declaring 🔗

A different, preferred, approach requires you to declare what the sum is. Then RxJS takes care of updating it each time one of the sources change.

While the declaration looks a bit more complex, it doesn’t require updating the sum after one of the numbers has changed.

number1$ = new BehaviorSubject<number>(0);
number2$ = new BehaviorSubject<number>(0);
sum$: Observable<number>;

ngOnInit() {
	sum$ = combineLatest([number1$, number2$])
		.pipe(
			map(([number1, number2]) => number1 + number2),
			shareReplay(1)
		);
}

incrementFirstNumber() {
  number1$.next(number1$.value + 1);
}

incrementSecondNumber() {
  number2$.next(number2$.value + 1);
}

Note: The expression shareReplay(1) is helpful to ensure that the computed observable has a value.

Angular-specific outlook: Signals 🔗

With the upcoming Angular signals, declaring computed values becomes simpler.

Note the simpler syntax. Also, how the sum is guaranteed to always have a value, which was not the case with RxJS observables.

number1 = signal(0);
number2 = signal(0);
sum = computed(() => number1() + number2());

incrementFirstNumber() {
  number1.update((value) => value + 1);
}

incrementSecondNumber() {
  number2.update((value) => value + 1);
}

Traveling Coderman Blog

Avoid Runtime Errors with Optionals

The problem with null 🔗

Why use null anyway? 🔗

Modeling with null 🔗

Refining with Optional 🔗

Summary 🔗

The One-Hour Interview Process

What's wrong with the status quo in interviewing? 🔗

The Goals of an Interview 🔗

The Framework 🔗

APIs 🔗

Functional programming 🔗

Coding 🔗

Frontend 🔗

Databases 🔗

Conclusion 🔗

Scrum As Code

Product Management Tool Requirements 🔗

The Idea 🔗

The Graphical Representation 🔗

Other Advantages 🔗

Conclusion 🔗

How future languages will prevent Log4Shell

Log4Shell in a nutshell 🔗

The conceptual issue 🔗

The conceptual solution 🔗

Conclusion 🔗

Further reading 🔗

Implementing a light/dark mode toggle with pure JS/CSS

What's the goal? 🔗

In a nutshell, how do we get there? 🔗

So what's the code? 🔗

Conclusion 🔗

Identifying bad code with function usage patterns

1 to 1 usage 🔗

2-5 to 1 usage 🔗

n to 1 usage 🔗

1 to n usage 🔗

Conclusion 🔗

Building a 1000 line Java program in 6 lines of Haskell

The Knights Tournament 🔗

The Approach 🔗

The Knight data type 🔗

Taking a hit 🔗

The turn 🔗

The main loop 🔗

All together 🔗

Conclusion 🔗

Extended version 🔗

Working as a Senior Software Engineer

Team-internal responsibilities 🔗

Team-external responsibilities 🔗

Conclusion 🔗

2031 - A Software Engineering Utopia

About the Node.js series

The high-level architecture 🔗

The choice of libraries 🔗

What we are about to cover 🔗

Building an API 🔗

Working with a database 🔗

Sending HTTP requests 🔗

Scheduled jobs 🔗

Working in a team 🔗

Advanced topics 🔗

What we are building 🔗

Way forward 🔗

Organizing Express.js endpoints

File System Organization 🔗

Todo Router 🔗

Todo Type 🔗

Data Access Layer 🔗

Dedicated data access object 🔗

Controllers 🔗

Elimination of service layer 🔗

Conclusion 🔗

Testing Express.js endpoints with supertest

Todo stub 🔗

Mocking data access 🔗

Spinning up the HTTP server 🔗