Intro

Despite of all the jokes saying it is not even a real programming language, more that 20 years since its creation, CSS is still one of the most misunderstood web technologies.

While it allows us to describe complex layouts in a high-level fashion, sometimes we bump into issues with corner cases. Also the CSS spec is not getting any smaller with time, and while I certainly do not miss juggling around with float and clear, it is hard to deny that flexbox and grid brought a lot of complexity to CSS.

TL;DR

I stumbled upon an issue with flexbox and white-space: nowrap, causing a flex child width to be larger than its parents. Visit this codesandbox to view the solution.

Flexbox all the things

I must admit that I might be overusing flexbox, but the reality is that other methods for centering content both vertically and horizontally are way too hacky.

I had this screen from the app I am currently working on whose outer-most wrapper was a flex container. The screen is supposed to be responsive, so there was no limits for its width. Deeply nested into the HTML tree there is another flex container whose children contained text that could potentially spread across multiple lines.

I was able to create a minimum reproducible example, so please forgive me if it looks a bit contrived.

Here is the markup:

<div class="outer-flex-container">
  <div class="outer-flex-child">
    <div class="inner-flex-container">
      <div class="inner-flex-child">
        Lorem ipsum dolor sit amet, consectetur adipiscing elit. Donec nec
        blandit massa, sed fringilla odio. Quisque nec ipsum molestie,
        maximus.
      </div>
    </div>
  </div>
</div>

And here are the styles:

.outer-flex-container {
  border: 2px dashed red;
  display: flex;
  justify-content: center;
  align-items: center;
  height: 100vh;
  padding: 16px;
}

.outer-flex-child {
  border: 2px solid orangered;
  padding: 16px;
}

.inner-flex-container {
  border: 2px dashed teal;
  display: flex;
  justify-content: center;
}

.inner-flex-child {
  background: paleturquoise;
  padding: 16px;
}

The result:

The issue

Because of layout constraints, I need the text in .inner-flex-child to be displayed in a single line, so I happily went along with:

 .inner-flex-child {
   background: paleturquoise;
   padding: 16px;
+  overflow: hidden;
+  text-overflow: ellipsis;
+  white-space: nowrap;
 }

The result however surprised me:

Oddly the .inner-flex-child now stopped respecting its parent's width. Furthermore, it seems like it is forcing both .inner-flex-container and .outer-flex.child to overflow .outer-flex-container width.

Okay, let's just:

 .inner-flex-child {
   background: paleturquoise;
   padding: 16px;
   overflow: hidden;
   text-overflow: ellipsis;
   white-space: nowrap;
+  width: 100%;
 }

And that will do it!

Nop! That did not work at all. Results were still the same. I tried then every possible combination of width, max-width, min-width, flex-basis I could think of without success.

For example, if I set a fixed width for .inner-flex-child:

 .inner-flex-child {
   background: paleturquoise;
   padding: 16px;
   overflow: hidden;
   text-overflow: ellipsis;
   white-space: nowrap;
+  width: 500px;
 }

I would get:

Works as expected if the viewport is larger than 500px, but expectedly if I had a smaller screen, the problem would be back:

Switching width to max-width however did not do the trick.

Another attempt had me setting flex-basis, flex-grow and flex-shrink:

 .inner-flex-child {
   background: paleturquoise;
   padding: 16px;
   overflow: hidden;
   text-overflow: ellipsis;
   white-space: nowrap;
+  flex: 200px 0 1;
 }

The idea here was to set the flex-basis to a fixed width, do not allow .inner-flex-child to grow, but do allow it to shrink. However, the result was even weirder:

Even though the content size was respecting the 200px I set, .inner-flex-container was still overflowing its parents. My head was now a complete mess.

Halt! DuckDuckGo time

Defeated, I resorted to the knowledge of the world wide web. I started digging with DuckDuckGo (just like Google, but it respects your privacy) and found this example. Apparently setting min-width: 0; in .inner-flex-child was supposed to do the trick:

 .inner-flex-child {
   background: paleturquoise;
   padding: 16px;
   overflow: hidden;
   text-overflow: ellipsis;
   white-space: nowrap;
+  min-width: 0;
 }

But it did not! I was bummed out at first, but I kept on digging. Of course there was something regarding that in StackOverflow. However the solution pointed to the same min-width: 0 trick. I was almost dismissing it when I saw it had a link to the spec.

The linked spec session starts with:

Note: The auto keyword, representing an automatic minimum size, is the new initial value of the min-width and min-height properties. The keyword was previously defined in this specification, but is now defined in the CSS Sizing module.

From that I understood that flex children have the value of min-width and min-height default to auto if not specified.

Then it goes on:

To provide a more reasonable default minimum size for flex items, the used value of a main axis automatic minimum size on a flex item that is not a scroll container is a content-based minimum size; for scroll containers the automatic minimum size is zero, as usual.

That's a mouthful. I was in rush/lazy mode by the time I was reading this, so I gave up. This is a behavioral pattern I fallback into when I am tired and with enough training I learnt how to recognize this and went out — of the room, since I am writing this in the middle of the Covid-19 pandemic, I could not leave the apartment — for a while.

After a cup of coffee and trying to put myself in the right mental state, I went back to the same SO question. It actually does a good job "translating" the spec into something more digestible:

With regard to the auto value...

On a flex item whose overflow is visible in the main axis, when specified on the flex item’s main-axis min-size property, specifies an automatic minimum size. It otherwise computes to 0.

In other words:

• The min-width: auto and min-height: auto defaults apply only when overflow is visible.
• If the overflow value is not visible, the value of the min-size property is 0.
• Hence, overflow: hidden can be a substitute for min-width: 0 and min-height: 0.

But here is the problem: .inner-flex-child is already a flex container, because it has overflow: hidden set. Damn you, CSS!

Reading on, I reached this section:

You've applied min-width: 0 and the item still doesn't shrink?

Nested Flex Containers

If you're dealing with flex items on multiple levels of the HTML structure, it may be necessary to override the default min-width: auto / min-height: auto on items at higher levels.

Basically, a higher level flex item with min-width: auto can prevent shrinking on items nested below with min-width: 0.

That was exactly the problem I was facing. I was only meddling with .inner-flex-child and .inner-flex-container the whole time. However, I had more flex content above in the DOM hierarchy. Dumbass!

So I tried the following:

 .outer-flex-child {
   border: 2px solid orangered;
   padding: 16px;
+  min-width: 0;
 }

Aaaaaand bingo!

I decided to play around with the working example and found out that the following approaches have the same effect:

 .outer-flex-child {
   border: 2px solid orangered;
   padding: 16px;
+ overflow: hidden;
 }

According to the spec, this fixes the issue because .outer-flex-child is now a scroll container, so min-width default value falls-back to 0. The downside of this is that if it contained any children that might be out of bounds — such as a tooltip or a modal with position: absolute — they would be clipped out.

 .outer-flex-child {
   border: 2px solid orangered;
   padding: 16px;
+  max-width: 100%;
 }

While not directly mentioned in the linked spec section, I noticed that .outer-flex-container width itself was not affected by the overflowing content. When using % as unit, it will be always relative to the parent's dimensions, so setting max-width: 100% does the trick as well, at least in this specific case.

Here is a codesandbox with the example I used throughout this article and the possible solutions.

That's all folks!

From the dark ol’ days of writing an entire application and only then starting to test it (often, manually) till nowadays, I have scoured a painful path of unending bug-fixing in production through the nights, many times not even knowing what was causing those bugs.

Yeah, something crazy is going on!

Since I first heard of Test Driven Development, it changed the way I think about software development.

I will not be digressing about TDD philosophy and its implications here, because a lot of more qualified people have done it before me. So let’s get to the code!

First, the problem and its solution

A long time ago in a galaxy far far away, I ended up in a problem: I had to monitor a “stream” (more like a polling) of events that were being created at a certain application in my Node.JS backend. This “stream” was not uniform and, most of the time, no event occurred.

I could not use websockets, so I would have to buffer these events in my backend. I thought using a database (even an in-memory one like Redis) just for that was too much. Then I decided that I would keep the events in memory and as my application did not care for all events that ever happened, I would keep only the last N of them.

Since Node.JS arrays are dynamic, they did not fit my needs. I did not want a fixed-size array implementation, what I needed was a fixed-sized first-in/first-out (FIFO) data structure, AKA a queue, which instead of overflowing when full, should pop its first element and then add the new one at the end.

Expected behavior

You better do that!

The data structure described above is rather simple. Its expected behavior could be summarized as follows:

Adding elements:

• When it is not full, it should add the new element to the end; its size should be increased by 1.
• When it is full, it should remove the first element and then add the new element to the end; its size must not change.
  • The removed element should be returned.

Removing elements:

• When it is not empty, it should remove the first element and return it; its size should be decreased by 1.
• When it is empty, it should throw an error.

A Mocha to go, please!

Looks delicious! ;9

From the docs:

Mocha is a feature-rich JavaScript test framework running on Node.js and in the browser, making asynchronous testing simple and fun. Mocha tests run serially, allowing for flexible and accurate reporting, while mapping uncaught exceptions to the correct test cases. Hosted on GitHub.

Installation

yarn add --dev mocha
# or with NPM:
# npm install --save-dev mocha

Writing tests

To create a test suite, you use a globally defined function called describe. To add test cases to a suite, you should use another global function it:

Suites can be nested indefinitely when you want to group your test cases. Mocha will collect all your suites recursively and execute all test cases it find within them in the order they are declared.

And that’s probably about all you need to tedknow about Mocha to get star (at least for basic usage). It excels so much for simplicity and extensibility, that it allows you to use whatever assertion library and other plugins you want.

Running tests

yarn mocha '<path-to-test-file>'
# or with NPM's npx:
# npx mocha '<path-to-test-file>'

Enter Chai

I used to think that Node.JS developers only like coffee… Guess I was not totally right (:

By default, Mocha can be used along with Node.js native assert module. It works just fine, however I don't find its developer experience to be exactly great. For that reason, we will use a 3rd-party assertion library called Chai.

From the docs:

Chai is a BDD / TDD assertion library for node and the browser that can be delightfully paired with any JavaScript testing framework.

Installation

yarn add --dev chai
# or with NPM:
# npm install --save-dev chai

Usage

Chai offers 3 different styles for writing assertions:

Chai allows you to pick your own poison.

All of them have the same capabilities, so choosing one or another is more a matter of preference than of objective facts. I like to use the expect interface.

Oh, tests! Oh, dreaded tests!

These bug’s bites hurt a lot; better kill’em before they kill you!

Going back to our original problem, let’s translate the expected behavior into mocha test suites. But first, let’s do some setup:

const chai = require("chai");
const expect = chai.expect;

const RoundQueue = require("./round-linked-queue");

describe("Round-Queue", () => {
});

Source

Testing queue creation

The main reason why we are creating this data structure is that it has to be a limited size, so let's make sure it has such property:

const chai = require("chai");
const expect = chai.expect;

const RoundQueue = require("./round-linked-queue");

describe("Round-Queue", () => {
  describe("When creating an instance", () => {
    it("Should properly set the maxLength property", () => {
      const queueLength = 3;

      const queue = new RoundQueue(queueLength);

      expect(queue.maxLength).to.equal(queueLength);
    });
  });
});

Source and diff

Next we implement just enough code to make the test above pass:

class RoundLinkedQueue {
  constructor(maxLength) {
    this._maxLength = maxLength;
  }

  get maxLength() {
    return this._maxLength;
  }
}

module.exports = RoundLinkedQueue;

Source

To run the suite, we do:

yarn mocha round-linked-queue.test.js

Keep moving and we must ensure that a queue is created empty:

it("Should initially set the length to zero", () => {
  const queueLength = 3;

  const queue = new RoundQueue(queueLength);

  expect(queue.length).to.equal(0);
});

Source

In order to make the new test pass, we can do as follows:

class RoundLinkedQueue {
  constructor(maxLength) {
    this._maxLength = maxLength;
    this._length = 0;
  }

  get maxLength() {
    return this._maxLength;
  }

  get length() {
    return this._length;
  }
}

Source and diff.

Testing adding elements

Next we create another test suite inside the top-level suite to test the behavior of adding elements to a queue.

Our base use case happens when the queue is empty and we want to add an element to it:

describe("When adding elements", () => {
  it("Should add an element to an empty queue", () => {
    const queue = new RoundQueue(3);
    const originalLength = queue.length;
    const elementToAdd = 1;

    queue.add(elementToAdd);

    // Element should've been added to the end of the queue
    expect(queue.last).to.equal(elementToAdd);
    // But since it is now the only element, it should also be the at beginning as well
    expect(queue.first).to.equal(elementToAdd);
    // Length should've been increased by 1
    expect(queue.length).to.equal(originalLength + 1);
  });
});

Source

If you run the test suite right now, you will get the following error:

D'oh!

The test failed because we didn't implement the add method yet. Now we add just enough code to make this first test case pass.

Important: the code bellow is not entirely correct, we will have to modify it further in order to make the add method work as expected. However, it does make our first test case "adding element to an empty queue" pass.

class RoundLinkedQueue {
  // ...

  add(element) {
    this._root = element;
    this._first = element;
    this._last = element;

    this._length += 1;
  }
}

Source and diff

Now let's try adding a test for when the queue is not empty anymore and yet we still want to add an element to it:

it("Should add an element to the end of a non-empty queue", () => {
  const queue = new RoundQueue(3);
  const previousElement = 1;
  const elementToAdd = 2;
  // Make the queue non-empty
  queue.add(previousElement);

  queue.add(elementToAdd);

  // Element should've been added to the end of the queue
  expect(queue.last).to.equal(elementToAdd, "last not properly set");
  // But the first pointer must remain the first element added
  expect(queue.first).to.equal(previousElement, "first not properly set");
  // Length should've been increased by 2
  expect(queue.length).to.equal(2, "length not properly set");
});

If we once again run the test suite without changing the implementation, we will get a failure:

The more attentive readers should probably be expecting this error because the way we implemented the add method before would simply overwrite the elements in the queue. To fix this, we will need some more code:

class RoundLinkedQueue {
  // ...

  add(element) {
    const node = {
      data: element,
      next: null,
    };

    if (!this._root) {
      this._root = node;
      this._first = node;
      this._last = node;
    } else {
      const previousLast = this._last;
      previousLast.next = node;

      this._last = node;
    }

    this._length += 1;
  }
}

Source and diff

We had to convert our _root, _first and _last into a node object containing data — the actual value of the item — and next — a pointer to the next node in the linked list.

Moving on, now it's time to something a little bit more challenging. Whenever our queue is at capacity, adding a new element should should cause the removal of the element that was first added:

it("Should remove the first element and add the new element to the end of a full queue", () => {
  const queue = new RoundQueue(3);
  queue.add(1);
  queue.add(2);
  queue.add(3);

  queue.add(4);

  // Element should've been added to the end of the queue
  expect(queue.last).to.equal(4, "last not properly set");
  // The second element should've been shifted to the first position
  expect(queue.first).to.equal(2, "first not properly set");
  // Length should still be the same
  expect(queue.length).to.equal(3, "length not properly set");
});

Source

Running tests once more we get:

Looks like we will need some conditionals to make the new test case pass along with the previous ones:

class RoundLinkedQueue {
  // ...

  add(element) {
    const node = {
      data: element,
      next: null,
    };

    if (this.length < this.maxLength) {
      if (!this._root) {
        this._root = node;
        this._first = node;
        this._last = node;
      } else {
        const previousLast = this._last;
        previousLast.next = node;

        this._last = node;
      }

      this._length += 1;
    } else {
      this._root = this._root.next;
      this._last.next = node;
      this._first = this._root;
      this._last = node;
    }
  }
}

Source and diff

Halt! Refactor time

So far we were writing code in a rather linear fashion: make a failing test, implement code to make it pass; make another failing test, write just enough code to make it pass, and so on.

In TDD jargon, creating a failing test is called the red phase, while implementing the code that will make it pass is the green phase.

In reality, things are not so pretty-neaty. You will not always get how to write the best code possible the first time. The truth is we've been cheating a little: we were skipping the refactor phase of the TDD cycle:

The gist of TDD

Right now I see some possible improvements in our data structure:

1. Having both _root and _first properties seem redundant.
2. There is some duplication of code in the add method (remember DRY?)

Because we already know the expected behavior, which is coded in our test suite, we are comfortable to refactor mercilessly.

class RoundLinkedQueue {
  // ...

  add(element) {
    const node = {
      data: element,
      next: null,
    };

    if (this.length < this.maxLength) {
      if (!this._first) {
        this._first = node;
        this._last = node;
      }

      this._length += 1;
    } else {
      this._first = this._first.next;
    }

    this._last.next = node;
    this._last = node;
  }
}

Source and diff

Hopefully, our tests are still green:

Taking some shortcuts

Now we are going to cheat a little bit.

The last requirement is that the add method should return the removed element when the queue is full. What to return when the queue is not full is not in the specification though. In JavaScript, uninitialized values have a special value called undefined. It makes sense to return that when adding to the queue does not remove any element, so we can add the following two test cases.

TDD purists gonna be triggered.

it("Should return the removed element from a full queue", () => {
  const queue = new RoundQueue(3);
  queue.add(1);
  queue.add(2);
  queue.add(3);

  const result = queue.add(4);

  expect(result).to.equal(1, "removed wrong element");
});

it("Should return undefined when the queue is not full", () => {
  const queue = new RoundQueue(3);

  const result = queue.add(1);

  expect(result).to.equal(undefined, "should not return an element");
});

Source

Cool, so let's just return the element from the node we just removed:

class RoundLinkedQueue {
  // ...

  add(element) {
    const node = {
      data: element,
      next: null,
    };

    let removedElement;

    if (this.length < this.maxLength) {
      if (!this._first) {
        this._first = node;
        this._last = node;
      }

      this._length += 1;
    } else {
      removedElement = this._first.data;
      this._first = this._first.next;
    }

    this._last.next = node;
    this._last = node;

    return removedElement;
  }
}

Source and diff

Look like we are are done with the add method!

Testing removing elements

Removing elements seems like a simpler operation. Our base use case is when the queue is not empty. We remove an element from it and decrease its length by one:

describe("When removing elements", () => {
  it("Should remove the first element of a non-empty queue", () => {
    const queue = new RoundQueue(3);
    queue.add(1);
    queue.add(2);
    queue.add(3);
    const lengthBefore = queue.length;

    const result = queue.remove();

    const lengthAfter = queue.length;

    expect(lengthAfter).to.equal(lengthBefore - 1, "length should decrease by 1");
    expect(result).to.equal(1, "first element should the one being removed");
    expect(queue.first).to.equal(2, "should shift the second element to the head of the queue");
    expect(queue.last).to.equal(3, "should not change the last element");
  });
});

Source

Running the tests will once again give us an error:

Now we add some code just to make the test pass:

class RoundLinkedQueue {
  // ...

  remove() {
    const removedElement = this.first;

    this._first = this._first.next;
    this._length -= 1;

    return removedElement;
  }
}

Source and diff

The only other use case is when the queue is empty and we try to remove an element from it. When this happens, the queue should throw an exception:

it("Should throw an error when the queue is empty", () => {
  const queue = new RoundQueue(3);

  expect(() => queue.remove()).to.throw("Cannot remove element from an empty queue");
});

Source

Running the test suite as is:

Ouch!

Adding some conditions to test for emptyness and throw the proper error:

class RoundLinkedQueue {
  // ...

  remove() {
    const removedNode = this._first;
    if (!removedNode) {
      throw new Error("Cannot remove element from an empty queue");
    }

    this._first = this._first.next;
    this._length -= 1;

    return removedNode.data;
  }
}

Source and diff

And that's it!

Testing edge cases

There are still some bugs in or code. When we wrote the add method, we included the first and last getters as well. But what happens if we try to access them when the queue is empty? Let's find out! first things first (ba dum tsss!):

describe("When accessing elements", () => {
  it("Should throw a proper error when acessing the first element of an empty queue", () => {
    const queue = new RoundQueue(3);

    expect(() => queue.first).to.throw("Cannot access the first element of an empty queue");
  });
});

Source

Running the tests:

Looks like the error message is not really helpful. In fact, it is a little too low level. Let's make it better:

class RoundLinkedQueue {
  // ...

  get first() {
    if (!this._first) {
      throw new Error("Cannot access the first element of an empty queue");
    }

    return this._first.data;
  }

  // ...
}

Source and diff

Lastly, for the last getter, we will do the same:

it("Should throw a proper error when acessing the last element of an empty queue", () => {
  const queue = new RoundQueue(3);

  expect(() => queue.last).to.throw("Cannot access the last element of an empty queue");
});

Source

First the failing result:

Then fixing the code:

class RoundLinkedQueue {
  // ...

  get last() {
    if (!this._last) {
      throw new Error("Cannot access the last element of an empty queue");
    }

    return this._last.data;
  }

  // ...
}

Source and diff

Aaaaaaand that's about it!

Conclusion

I tried to make this a comprehensive introduction to TDD with the Node.js/JavaScript ecosystem. The data structure we had to implement here was intentionally simple so we could follow the methodology as much as possible.

When doing TDD in real world applications, things are usually not so linear. You will find yourself struggling from time to time with the design choices you make while writing your tests. It can be a little frustrating in the beginning, but once you get the gist of it, you will develop a "muscle memory" to avoid the most common pitfalls.

TDD is great, but as almost everything in life, it is not a silver bullet.

Be safe out there!

T-t-th-tha-that's i-is a-a-all f-f-fo-f-folks!

Did you like what you just read? Why don’t you buy me a beer (or a coffee if it is before 5pm 😅) with tippin.me?

Building a contextual logger

This is the continuation of my previous article about logging context. Check it out to better understand the purpose of what we will build.

TL;DR

The code we are going to build on this story is on my Github. If you just want to check out the final version, you can get it at the master branch.

Intro

Last time we walked through a way of managing context through concurrent requests using pino and cls-hooked. Now let’s build a wrapper around pino that will automatically deal with this for us.

And now, it’s tiiiiiiiime!

What do we want to achieve?

We need to build a logger that will have base “global” context through cls-hooked, but will also allow us to augment such context when actually calling the logger methods.

To improve reusability and interoperability, we want to maintain the original default pino API, so we already have a good set of test cases to cover. Also, we need to provide a way for our application interact with the context.

How will we write our code?

We are going to implement this wrapper TDD style. However the tests we will write are not “unit” tests in a strict sense, because they will include pino itself and make assertions about the generated log data. This is possible because pino accepts a custom WritableStream as its destination.

As testing framework, we will use ava. Keep in mind that while ava transpiles test files by default, it doesn’t do that for the actual code without properly setting babel. To avoid adding more complexity to this solution, all code (including tests) will not use ES modules or any features that are not available in Node.js 10.9.0.

If you want do follow along the implementation, please check out the instructions in the Github repository:

I tried to make this the sequence as natural as possible, only eliminating some inner loops and struggles that happen in a regular coding session.

Implementation steps

Initial setup

yarn init -y
yarn add pino cls-hooked
yarn add --dev ava

A nice feature of pino accepts a custom WritableStream as its destination. This will make our lives easier when testing our custom logger.

Ensuring methods for log levels

For simplicity, lets stick with pino default log levels: trace, debug, info, warn, error and fatal.

The simplest way to achieve that is:

The logger.js is currently just a factory function that return plain pino instances. The logger.test.js file generates one test case for each available method to make sure we don’t break anything later.

parse-json-stream.js is a utility that will parse the log output stream and return plain Javascript objects to make it easier to run assertions against the log output.

stream-to-generator.js is there for convenience: ava doesn’t play well with stream-based APIs. To make tests more concise, we convert the logging stream to a generator that yields promises to the next log entry.

The later two are not important in the context of what we are trying to achieve, they are here only for reference. The remaining snippets won’t include them.

Keeping context on logger method call

Also, notice that pino allow us to pass local context to a log entry by prepending an object to the argument list. This is a behavior we want to keep.

So, lets add a test case that covers this scenario:

Since so far we are just creating a pino instance, the test will pass.

Adding CLS awareness

Now we start touching CLS. First we need to create namespace and expose it to the world:

Preventing CLS context sharing between instances

For some reason, we might want to have multiple loggers in a given application. When doing that, it’s important to not mix the namespaces of both. However, the way we implemented above, all instances will have the same namespace '@@logger', which could cause strange behavior laters.

The easiest way to fix this would be to have a counter variable that would increment whenever we call createLogger and append the counter value to the namespace name.

While counters are not the most safe bet to generate unique names, since they are reset when the application restarts, they work in this case because all logger instances would be recreated anyway when the server restarts. Also, this value is not exposed anywhere, it serves only for the purpose of creating different namespaces, so we are fine.

Sometimes less is more!

Here’s what’s changed:

Applying CLS context to logs

This one is a big leap, so bear with me. First, let’s see the changes in the code, then let’s discuss it:

Sorry, I couldn’t break this into smaller changes :/

The test code has nothing special about it, just notice that we must run our logging and assertion within the logger.cls.run method callback.

Things start to get interesting on the actual code though. We are leveraging Javascript Proxy to intercept log method calls and patch their arguments.

So, in line 52 we create a proxy for our logger object, whose handler is named a loggerObjectHandler— lines 34–43. The handler defines a get trap, that will intercept only the calls for the log methods — trace, debug, etc. What it does is wrap those methods into yet another proxy, whose handler is named logMethodHandler — lines 11–32.

The loggerMethodHandler gathers the current active context on CLS, excluding some irrelevant properties from it — lines 14–15. Then, based on the current argument list, it checks whether we have or not a local context on the method call. If we don’t, then we simply need to prepend the CLS context to the argument list — lines 20–23. Otherwise, we need to merge the local context into the CLS context — lines 24–28. Finally, we call the original method with the proper arguments — line 30.

Propagating changes to child loggers

A nice feature from pino is that it allows us to create child loggers through the .child() method. A child logger maintains all properties from its parent, but can also accept additional context. So, we need to make our child generation CLS aware too:

Again, the new tests are self-descriptive. Let’s focus on the implementation. First we extracted the wrapper creation into its own function, named createWrapper — lines 47–52. This allows us to create a wrapper for the child loggers as well.

Next, we define a childMethodHandler which will intercept the calls to .child() — lines 18–25. This handler will call createWrapper on the newly created child logger, passing the CLS context from the parent as a parameter. This will guarantee that parent and children (and children of children) all have the same context.

Lastly, we change the implementation of loggerObjectHandler to include the proxy for the .child() method as well — lines 30–45 — including some internal refactoring on the conditionals.

Further improvements

Seems like our code works so far, but it might not be optimal. An issue that is easy to spot is that we are creating new proxies on the fly for every call on the child and log methods. While this might not be an issue with the former — because we wouldn’t call .child() very often — that’s not true for the latter.

To prevent this problem, we could create the proxies for the desired methods by the time we create the logger itself and put them as properties of the logger object. When we call the methods, the loggerObjectHandler would just check to see if there is a proxy set for the current method. If so, it returns the proxy, otherwise, it returns the original property:

Integrating with our web application

So now we have our logger factory. Now we need to integrate it with our application. From the final example from the previous article, we could refactor to:

Outro

The code above is pretty much the same I have successfully used in production some times and have saved me and the teams I worked with a lot of time while debugging.

If you have any suggestions that could improve it, you are more than welcome.

Did you like what you just read? Buy me a beer with tippin.me.

How to make sense out of your Node.js web app logs

Logging might be one of the most difficult things to do right when building a real world application. Log too little and you will be staring at your screen trying to make sense of them (or the charts generated from them). Log too much and you will end up lost in a swamp of useless information, still having no clue if everything is OK or if you have a problem.

Logs without the right amount of context look like…

Speaking specifically of the Node.js/Javascript ecosystem, the top 3 logging libraries — Winston, Bunyan and Pino — can help you to manage context way better than the good ol’ console.log could ever do.

For this article I will use Pino, but the ideas can be easily replicated for both Bunyan and Winston (or any other mainstream logging utility).

Use log levels wisely

Pino has 6 default log levels, with increasing severity: trace, debug, info, warn, error and fatal. Each one of these levels maps to an integer from 10 to 60. This makes it easy to analyze your logs later using tools like jq:

jq 'select(.level > 40)' # gets ERROR and FATAL logs

While Pino allows you to define custom log levels, I have never seen a use case where they would be necessary, so I tend to stick with the default ones.

Usually, for production, it is recommended to ignore trace and debug levels, unless you are explicitly trying to debug some production issue.

Pino has a configuration option that allows you to define the minimum required level for the log entry to be generated. You can use environment variables to avoid having to make a deploy just to change the log level:

const logger = pino({
  level: process.env.LOG_LEVEL || 'info'
});

Rules of thumb

• Use trace for internal logging that has a potentially high throughput.
• Use debug for eventual debugging sessions you might need, but remember to remove them after you are finished.
• Use info for regular application workflow logs.
• Use warn for expected and frequent error conditions (like user input validation).
• Use error for expected but infrequent error conditions (like network failures, database timeouts).
• Use fatal for unexpected error conditions.

Embrace request IDs

While we are still developing the application, running unit/integration tests, manually triggering some request to see if everything is running smoothly, it’s all good. The events being produced happen in a more or less predictable order, so it’s easy to follow.

However, once the production version is launched, things can go really crazy. Your app will most certainly process concurrent requests. If you have a few asynchronous steps — like querying a database or calling some external services — the order of each event will be completely unpredictable. In this case, if you are manually inspecting the logs (we all have done this at some point 😅), you can become very frustrated trying to find a thread of execution.

Some frameworks — like Hapi — already take care of this for you. But if you like me still rely on good ol’ express, you have to do it yourself. Defining a middleware that does that is as simple as:

function setRequestId(generateId) {
  return (req, res, next) => {
    req.id = generateId();
    next();
  };
}

Then use it:

let i = 0;
const generateId = () => i++;
app.use(setRequestId(generateId));</span>

Of course, this naive implementation would not work if you ever restart your server, since the counter would be reset to 0. For a real world application, it’s recommended to use a more robust ID generator, such as uuid or, my personal choice, cuid.

If you use a micro-services architecture (or want to be prepared to), you can leverage distributed tracing simply by allowing your services to forward and receive a given request ID:

function setDistributedRequestId(generateId) {
  return (req, res, next) => {
    const reqId = req.get('X-Request-Id') || generateId();
    req.id = reqId;
    res.set('X-RequestId', reqId);
    next();
  };
}

Now we can create another middleware that logs incoming requests:

function logIncomingRequests(logger) {
  return (req, res, next) => {
    logger.trace({ req, requestId: req.id}, 'Incoming request');
    next();
  }
}

And use it:

app.use(logIncommingRequests(pino()))</span>

The generated log entry would look like:

{"level":30, "time":1533749413556, "pid":15377, "hostname":"henrique-pc", "msg":"Incoming request", "req":{"method":"GET", "url":"/", "headers":{"host":"localhost:4004", "user-agent":"curl/7.61.0", "accept":"*/*"}},
"requestId":1, # <---- notice here!
"v":1}

So far, so good. We could even use express-pino-logger to further integrate the logger with our express app. The major problem here is that the request ID is tightly coupled with our web layer. Unless you define all your business logic within express handlers — something I urge you to please don’t — you won’t be able to access the request ID value in other layers.

Oh, I could store the IDs in-memory or in a Redis cache and then retrieve it when logging something in other layers!!!

Yeah, nice try. I thought that myself too, but it doesn’t work. The reason is that you can’t know which request you are currently processing when you have concurrent accesses. Or can you?

Meet Continuation Local Storage

Imagine that each request is an isolated “thread” of connected execution paths (function calls) that is discarded when the result of the original call is returned.

While Javascript doesn’t spawn real threads for handling user requests, it emulates this by registering callbacks that will be called in the proper sequence when the results of the function calls are available.

Luckily for us, Node.js provides a way to intercept the hops through this execution “thread”. Continuation Local Storage (or CLS for short) leverages this capability to keep data available within a given “thread”.

When you set values in continuation-local storage, those values are accessible until all functions called from the original function — synchronously or asynchronously — have finished executing. This includes callbacks passed to process.nextTick and the timer functions (setImmediate, setTimeout, and setInterval), as well as callbacks passed to asynchronous functions that call native functions (such as those exported from the fs, dns, zlib and crypto modules).

Me when I first discovered CLS…

Redefining our request ID middleware, we would have something like:

import { createNamespace } from 'cls-hooked';
import cuid from 'cuid';
const loggerNamespace = createNamespace('logger');

function clsRequestId(namespace, generateId) {
  return (req, res, next) => {
    const reqId = req.get('X-Request-Id') || generateId();

    res.set('X-RequestId', reqId);

    namespace.run(() => {
      namespace.set('requestId', reqId);
      next();
    });
  };
}

app.use(clsRequestId(loggerNamespace, cuid));

Breaking it down:

• A namespace is roughly the CLS equivalent of a table from a relational database or a collection/key space from a document store. To create one, we simply need to identify it as a string.
• Our “high order” middleware clsRequestId now needs two parameters: the namespace and the ID generator function.
• namespace.run is the function that creates a new context, bounded to the execution “thread”.
• namespace.set puts the request ID into local storage.
• next will call the next express handler. IMPORTANT: to make this work as expected, next MUST be called inside the namespace.run callback.

Now, whenever we need to access this value, we can use getNamespace from cls-hooked:

import { getNamespace } from 'cls-hooked';
import pino from 'pino';
const logger = pino();

loggerNamespace = getNamespace('logger');

function doStuff() {
  // ...
  logger.info({ requestId: loggerNamespace.get('requestId') }, "Some message");
}

If function doStuff call was ultimately originated in one of the handlers from the express app which registered that clsRequestId middleware, the value will be available.

Putting everything toghether:

Here’s a sample output generated with autocannon:

{"level":30,"time":1533759930690,"msg":"App is running!","pid":4985,"hostname":"henrique-pc","endpoint":"http://localhost:4000","v":1}
{"level":30,"time":1533759933634,"msg":"Before","pid":4985,"hostname":"henrique-pc",**"requestId":"cjkll2awx0000uhwg9qh20e0b"**,"v":1}
{"level":30,"time":1533759933636,"msg":"Before","pid":4985,"hostname":"henrique-pc",**"requestId":"cjkll2awz0001uhwgoyiptfxv"**,"v":1}
{"level":30,"time":1533759935531,"msg":"Middle","pid":4985,"hostname":"henrique-pc",**"requestId":"cjkll2awz0001uhwgoyiptfxv"**,"v":1}
{"level":30,"time":1533759939590,"msg":"Middle","pid":4985,"hostname":"henrique-pc",**"requestId":"cjkll2awx0000uhwg9qh20e0b"**,"v":1}
{"level":30,"time":1533759941222,"msg":"After","pid":4985,"hostname":"henrique-pc",**"requestId":"cjkll2awz0001uhwgoyiptfxv"**,"v":1}
{"level":30,"time":1533759941228,"msg":"Before","pid":4985,"hostname":"henrique-pc",**"requestId":"cjkll2grw0002uhwgzz14qyb6"**,"v":1}
{"level":30,"time":1533759943632,"msg":"Before","pid":4985,"hostname":"henrique-pc",**"requestId":"cjkll2imo0003uhwgf4dutgz3"**,"v":1}
{"level":30,"time":1533759946244,"msg":"Middle","pid":4985,"hostname":"henrique-pc",**"requestId":"cjkll2grw0002uhwgzz14qyb6"**,"v":1}
{"level":30,"time":1533759949490,"msg":"After","pid":4985,"hostname":"henrique-pc",**"requestId":"cjkll2awx0000uhwg9qh20e0b"**,"v":1}
{"level":30,"time":1533759951621,"msg":"Middle","pid":4985,"hostname":"henrique-pc",**"requestId":"cjkll2imo0003uhwgf4dutgz3"**,"v":1}
{"level":30,"time":1533759952464,"msg":"After","pid":4985,"hostname":"henrique-pc",**"requestId":"cjkll2grw0002uhwgzz14qyb6"**,"v":1}
{"level":30,"time":1533759953632,"msg":"Before","pid":4985,"hostname":"henrique-pc",**"requestId":"cjkll2qcg0004uhwgnmgztdr7"**,"v":1}
{"level":30,"time":1533759954665,"msg":"Middle","pid":4985,"hostname":"henrique-pc",**"requestId":"cjkll2qcg0004uhwgnmgztdr7"**,"v":1}
{"level":30,"time":1533759955140,"msg":"After","pid":4985,"hostname":"henrique-pc",**"requestId":"cjkll2imo0003uhwgf4dutgz3"**,"v":1}
{"level":30,"time":1533759957183,"msg":"After","pid":4985,"hostname":"henrique-pc",**"requestId":"cjkll2qcg0004uhwgnmgztdr7"**,"v":1}

If you look closely you will see that, even though the call order of the logger function is non-linear, the requestId for each different request is mantained.

It’s complete maaaagic!

Now, whenever you want to see the logs from a single request in isolation, you can again use jq and run:

jq 'select(.requestId == "cjkll2qcg0004uhwgnmgztdr7")' <log_file>

The output will be:

{
  "level": 30,
  "time": 1533759953632,
  "msg": "Before",
  "pid": 4985,
  "hostname": "henrique-pc",
  "requestId": "cjkll2qcg0004uhwgnmgztdr7",
  "v": 1
}
{
  "level": 30,
  "time": 1533759954665,
  "msg": "Middle",
  "pid": 4985,
  "hostname": "henrique-pc",
  "requestId": "cjkll2qcg0004uhwgnmgztdr7",
  "v": 1
}
{
  "level": 30,
  "time": 1533759957183,
  "msg": "After",
  "pid": 4985,
  "hostname": "henrique-pc",
  "requestId": "cjkll2qcg0004uhwgnmgztdr7",
  "v": 1
}

Further improvements

While the structure presented in this story works, it’s not practical for everyday use. It would be very tedious having to manually get the namespace and retrieve all values you need like in the example code above:

const namespace = getNamespace('logger');                                                 logger.info({ requestId: namespace.get('requestId') }, 'Before')</span>

Next time we will build a wrapper around pino to handle all of this transparently.

Bye!

Did you like what you just read? Buy me a beer with tippin.me

Part 2 is now available here:

When testing Redux, here are a few guidelines:

Vanilla Redux

• The smallest standalone unit in Redux is the entire state slice. Unit tests should interact with it as a whole.

• There is no point in testing reducers, action creators and selectors in isolation. As they are tightly coupled with each other, isolation gives us little to no value.

• Tests should interact with your Redux slice same way your application will: through action creators and selectors.

• Avoid assertions like toEqual/toDeepEqual against the state object, as they create coupling between your tests and the state structure.

  • Selectors give you the granularity you need to run spot on assertions.

• Selectors and action creators should be boring, so they won't require testing.

• Your slice is somewhat equivalent to a pure function, which means you don't need any mocking facilities in order to test it.

Redux + redux-thunk

• Dispatching thunks doesn't have any direct effect. Only after the thunk is called we will have the side-effects we need to make our application work.

• Here you can use stubs, spies and sometimes mocks (but don't abuse mocks).

• Because of the way thunks are structured, the only way to test them is by testing their implementation details.

• The strategy when testing thunks is to setup the store, dispatch the thunk and then asserting whether it dispatched the actions you expected in the order you expected or not.

I have created a repo implementing the ideas above.

Intro

As a Software Engineer, I am always finding ways to get better at my craft. It is not easy. Not at all. Coding is hard enough. Writing good code is even harder.

Then there are tests. I think every single time I start a new project — professionally or just for fun — my ideas on how I should test my code change. Every. Single. Time. This is not necessarily a bad thing as different problems require different solutions, but this still intrigues me a little.

The Problem with Tests

As a ~most of the time~ TDD practitioner, I have learned that the main reason we write tests it not to assert the correctness of our code — this is just a cool side effect. The biggest win when writing tests first is that it guides you through the design of the code you will write next. If something is hard to test, there is probably a better way to implement it.

However, after if you have done this for some time, you realize that writing good tests are as hard as writing production code. Sometimes is even harder. Writing tests takes time. And extra time is something that your clients or the business people in your company will not give you so easily.

Testing?! Ain't nobody got time for that! (Photo by Aron Visuals on Unsplash)

And it gets worse. Even if you are able to write proper tests, throughout the lifespan of the product/project you are working on, requirements will change, new scenarios will appear. Write too many tests, make them very entangled and any minor change in your application will take a lot of effort to make all tests pass again.

Flaky tests are yet another problem. When it fails, you have no idea were to start fixing it. You will probably just re-run the test suite and if it passes, you are good to go.

Schrödinger's tests: sometimes they fail, sometimes they pass, but you cannot know for sure (Picture by Jie Qi on Flickr)

But how do you know if you are writing good tests? What the hell is a good test in the first place?

Schools of Testing

There is an long debate between two different currents of thoughts known as London School and Detroit School of Testing.

Summarizing their differences, while Detroit defends that software should be built bottom-up, with emphasis on design patterns and the tests should have as little knowledge as possible about the implementation and have little to no stubbing/mocking at all, London advocates that the design should be top-down, using external constraints as starting point, ensuring maximum isolation between test suites through extensive use of stubs/mocks, which has a side effect of having to know how the subject under test is implemented.

This is a very brief summary — even risking being wrong because of terseness — but you can find more good references about this two decades old conundrum here, here and here.

Testing in the Real World

So which one is right, Londoners or Detroitians? Both of them and neither of them at the same time . As I learnt throughout the almost five years I have been a professional Software Engineer, dogmatism will not take you very far in the real world, where projects should be delivered, product expectations are to be matched and you have bills to pay.

What you really need is to be able to take the best of both worlds and use it in your favor. Use it wisely.

We live in a world where everybody seems obsessed with ~almost~ perfect code coverage, while the problem of Redundant Coverage is rarely mentioned — it is not very easy to find online references discussing this. If you abuse tests, you may end up having a hard time when your requirements suddenly change.

In the end we are not paid to write tests, we are paid to solve other people's problems through code. Writing tests is expensive and does not add perceivable value to the clients/users. One can argue that there is value added by tests, but in my personal experience it is very hard to make non-technical people buy that.

What we as Software Engineers should strive for is to write the minimum amount of tests that yields enough confidence in code quality and correctness — and "enough" is highly dependent on context.

Redux Testing According to the Docs

Redux is known to have an outstandingly good documentation. In fact this is true. There is not only API docs and some quick examples, as there are also some valuable best practices advice and even links to more in depth discussions regarding Redux and its ecosystem.

However, I believe that the "Writing Tests" section leaves something to be desired.

Testing Action Creators

That section in the docs start with action creators.

export function addTodo(text) {
  return {
    type: 'ADD_TODO',
    text
  }
}

Then we can test it like:

import * as actions from '../../actions/TodoActions'
import * as types from '../../constants/ActionTypes'

describe('actions', () => {
  it('should create an action to add a todo', () => {
    const text = 'Finish docs'
    const expectedAction = {
      type: types.ADD_TODO,
      text
    }
    expect(actions.addTodo(text)).toEqual(expectedAction)
  })
})

While the test is correct and passes just fine, the fundamental problem here is that it does not add much value. Your regular action creators should be very boring, almost declarative code. You do not need tests for that.

Furthermore, if you use helper libraries like redux-act or Redux's own @reduxjs/toolkit — which you should — then there is absolutely no reason at all to write tests for them, as you would be testing the helper libs themselves, which are already tested and, more important, are not even owned by you.

And since action creators can be very prolific in a real app, the amount of test they would require is huge.

But how can we know for sure our plain-old action creators do not contain silly errors like typos on them?

Bear with me. More on that later.

Testing reducers

In Redux, a reducers is a function which given a state and an action, should produce an entirely new state, without mutating the original one. Reducers are pure functions. Pure functions are like heaven to testers. It should be pretty straightforward, right?

The docs gives us the following example:

import { ADD_TODO } from '../constants/ActionTypes'

const initialState = [
  {
    text: 'Use Redux',
    completed: false,
    id: 0
  }
]

export default function todos(state = initialState, action) {
  switch (action.type) {
    case ADD_TODO:
      return [
        {
          id: state.reduce((maxId, todo) => Math.max(todo.id, maxId), -1) + 1,
          completed: false,
          text: action.text
        },
        ...state
      ]

    default:
      return state
  }
}

Then the test:

describe('todos reducer', () => {
  it('should return the initial state', () => {
    expect(reducer(undefined, {})).toEqual([
      {
        text: 'Use Redux',
        completed: false,
        id: 0
      }
    ])
  })

  it('should handle ADD_TODO', () => {
    expect(
      reducer([], {
        type: types.ADD_TODO,
        text: 'Run the tests'
      })
    ).toEqual([
      {
        text: 'Run the tests',
        completed: false,
        id: 0
      }
    ])

    expect(
      reducer(
        [
          {
            text: 'Use Redux',
            completed: false,
            id: 0
          }
        ],
        {
          type: types.ADD_TODO,
          text: 'Run the tests'
        }
      )
    ).toEqual([
      {
        text: 'Run the tests',
        completed: false,
        id: 1
      },
      {
        text: 'Use Redux',
        completed: false,
        id: 0
      }
    ])
  })
})

Let's just ignore the fact that the suggested test case "should handle ADD_TODO" is two tests bundled together — which might freak-out some TDD zealots. Even though in this case I believe it would be best to have different test cases — one for an empty list and the other for a list with some initial values — sometimes this is just fine.

The real issue with those tests is that they are tightly coupled with the internal structure of the reducer. More precisely, the tests above are coupled to the state object structure through those .toEqual() assertions.

While this example is rather simple, it is very common for the state of a given slice in Redux to change over time, as new requirements arrive and some unforeseen interactions need to occur. If we write tests like the ones above, they will soon become a maintenance nightmare. Any minimal change in the state structure would demand updating several test cases.

So how exactly are we supposed to write those tests?

Testing Redux the right way

Disclaimer: I am not saying this is the best or the only way of testing your Redux application, however, I recently concluded that doing it the way I suggest below yields the best cost-benefit that I know of. If you happen to know a better way, please reach out to me through the comments, Twitter, e-mail or smoke signs.

Here is a popular folder structure for Redux applications that is very similar to the ones that can be found in many tutorials and even the official docs:

src
└── store
    ├── auth
    │   ├── actions.js
    │   ├── actionTypes.js
    │   └── reducer.js
    └── documents
        ├── actions.js
        ├── actionTypes.js
        └── reducer.js

If you are like me and like to have test files colocated with the source code, this structure encourages you to have the following:

src
└── store
    ├── auth
    │   ├── actions.js
    │   ├── actions.test.js
    │   ├── actionTypes.js
    │   ├── reducer.js
    │   └── reducer.test.js
    └── documents
        ├── actions.js
        ├── actions.test.js
        ├── actionTypes.js
        ├── reducer.js
        └── reducer.test.js

I have already left actionTypes tests out as those files are purely declarative. However, I already explained why action creators should be purely declarative, and therefore should not be tested as well. That leaves us with testing the only reducer itself, but that does not seem quite right.

The problem here is what we understand as being a "unit" in Redux. Most people tend to consider each of the individual files above as being a unit. I believe this is a misconception. Actions, action types and reducers must be tightly coupled to each other to function properly. To me, it does not make sense to test those "components" in isolation. They all need to come together to form a slice (e.g.: auth and documents above), which I consider being the smallest standalone piece in Redux architecture.

For that reason, I am fond of the Ducks pattern, even though it has some caveats. Ducks authors advocates everything regarding a single slice (which they call a "duck") should be placed in a single file and follow a well-defined export structure.

I usually have a structure that looks like this:

src
└── modules
    ├── auth
    │   ├── authSlice.js
    │   └── authSlice.test.js
    └── documents
        ├── documentsSlice.js
        └── documentsSlice.test.js

The idea now is to write the least amount of tests possible, while having a good degree of confidence that a particular slice works as expected. The reason why Redux exists in the first place is to help us manipulate the state, providing a single place for our application state to lie in.

In other words, the value Redux provides us is the ability to write and read a state from a centralized place, called the store. Since Redux is based on the Flux Architecture, its regular flow is more or less like this:

The Flux Architecture by Eric Eliott on Medium

Redux Testing Strategy

In the end of the day, what we want to test is that we are correctly writing to — through dispatching actions — and reading from the store. The way we do that is by given an initial state, we dispatch some action to the store, let the reducer do its work and then after that, we check the state to see if the changes we expect were made.

However, how can we do that while avoiding the pitfall of having the tests coupled with the state object structure? Simple. Always use selectors. Even those that would seem dumb.

Selectors are you slice public API for reading data. They can encapsulate your state's internal structure and expose only the data your application needs, at the granularity it needs. You can also have computed data and optimize it through memoization.

Similarly, action creators are its public API for writing data.

Still confused? Let's try with some code using @reduxjs/toolkit:

Here is my auth slice:

import { createSlice, createSelector } from '@reduxjs/toolkit';

export const initialState = {
  userName: '',
  token: '',
};

const authSlice = createSlice({
  name: 'auth',
  initialState,
  reducers: {
    signIn(state, action) {
      const { token, userName } = action.payload;

      state.token = token;
      state.userName = userName;
    },
  },
});

export const { signIn } = authSlice.actions;

export default authSlice.reducer;

export const selectToken = state => state.auth.token;
export const selectUserName = state => state.auth.userName;
export const selectIsAuthenticated = createSelector([selectToken], token => token !== '');

Nothing really special about this file. I am using the createSlice helper, which saves me a lot of boilerplate code. The exports structure follows more or less the Ducks Pattern, the main difference being that I don't explicitly export the action types, as they are defined in the type property of the action creators (e.g.: signIn.type returns 'auth/signIn').

Now the test suite implemented using jest:

import reducer, { initialState, signIn, selectToken, selectName, selectIsAuthenticated } from './authSlice';

describe('auth slice', () => {
  describe('reducer, actions and selectors', () => {
    it('should return the initial state on first run', () => {
      // Arrange
      const nextState = initialState;

      // Act
      const result = reducer(undefined, {});

      // Assert
      expect(result).toEqual(nextState);
    });

    it('should properly set the state when sign in is made', () => {
      // Arrange
      const data = {
        userName: 'John Doe',
        token: 'This is a valid token. Trust me!',
      };

      // Act
      const nextState = reducer(initialState, signIn(data));

      // Assert
      const rootState = { auth: nextState };
      expect(selectIsAuthenticated(rootState)).toEqual(true);
      expect(selectUserName(rootState)).toEqual(data.userName);
      expect(selectToken(rootState)).toEqual(data.token);
    });
  });
});

The first test case ('should return the initial state on first run') is only there to ensure there is no problem in the definition of the slice file. Notice that I am using the .toEqual() assertion I said you should not. However, in this case, since the assertion is against the constant initialState and there are no mutations, whenever the state shape changes, initialState changes together, so this test would automatically be "fixed".

The second test case is what we are interested in here. From the initial state, we "dispatch" a signIn action with the expected payload. Then we check if the produced state is what we expected. However, we do that exclusively using selectors. This way our test is more decoupled from the implementation

If your slice grows bigger, by using selectors when testing state transitions, you gain yet another advantage: you could use only those selectors that are affected by the action you dispatched and can ignore everything else. Were you asserting against the full slice state tree, you would still need to declare those unrelated state properties in the assertion.

An observant reader might have noticed that this style of testing resembles more the one derived from Detroit School. There are no mocks, stubs, spies or whatever. Since reducers are simply pure functions, there is no point in using those.

However, this slice is rather too simple. Authentication is usually tied to some back-end service, which means we have to manage the communication between the latter and our application, that is, we have to handle side effects as well as the loading state. Things start to get more complicated.

Testing a More Realistic Slice

The first step is to split our signIn action into three new: signInStart, signInSuccess and signInFailure. The names should be self-explanatory. After that, our state needs to handle the loading state and an eventual error.

Here is some code with those changes:

import { createSlice, createSelector } from '@reduxjs/toolkit';

export const initialState = {
  isLoading: false,
  user: {
    userName: '',
    token: '',
  },
  error: null,
};

const authSlice = createSlice({
  name: 'auth',
  initialState,
  reducers: {
    signInStart(state, action) {
      state.isLoading = true;
      state.error = null;
    },
    signInSuccess(state, action) {
      const { token, userName } = action.payload;

      state.user = { token, userName };
      state.isLoading = false;
      state.error = null;
    },
    signInFailure(state, action) {
      const { error } = action.payload;

      state.error = error;
      state.user = {
        userName: '',
        token: '',
      };
      state.isLoading = false;
    },
  },
});

export const { signInStart, signInSuccess, signInFailure } = authSlice.actions;

export default authSlice.reducer;

export const selectToken = state => state.auth.user.token;
export const selectUserName = state => state.auth.user.userName;
export const selectError = state => state.auth.error;
export const selectIsLoading = state => state.auth.isLoading;
export const selectIsAuthenticated = createSelector([selectToken], token => token !== '');

The first thing you might notice is that our state shape changed. We nested userName and token in a user property. Had we not created selectors, this would break all the tests and code that depends on this slice. However, since we did have the selectors, the only changes we need to do are in the selectToken and selectUserName.

Notice that our test suite is still broken, but that is because we fundamentally changed the slice. It is not hard to get it fixed though:

describe('auth slice', () => {
  describe('reducer, actions and selectors', () => {
    it('should return the initial state on first run', () => {
      // Arrange
      const nextState = initialState;

      // Act
      const result = reducer(undefined, {});

      // Assert
      expect(result).toEqual(nextState);
    });

    it('should properly set loading and error state when a sign in request is made', () => {
      // Arrange

      // Act
      const nextState = reducer(initialState, signInStart());

      // Assert
      const rootState = { auth: nextState };
      expect(selectIsAuthenticated(rootState)).toEqual(false);
      expect(selectIsLoading(rootState)).toEqual(true);
      expect(selectError(rootState)).toEqual(null);
    });

    it('should properly set loading, error and user information when a sign in request succeeds', () => {
      // Arrange
      const payload = { token: 'this is a token', userName: 'John Doe' };

      // Act
      const nextState = reducer(initialState, signInSuccess(payload));

      // Assert
      const rootState = { auth: nextState };
      expect(selectIsAuthenticated(rootState)).toEqual(true);
      expect(selectToken(rootState)).toEqual(payload.token);
      expect(selectUserName(rootState)).toEqual(payload.userName);
      expect(selectIsLoading(rootState)).toEqual(false);
      expect(selectError(rootState)).toEqual(null);
    });

    it('should properly set loading, error and remove user information when sign in request fails', () => {
      // Arrange
      const error = new Error('Incorrect password');

      // Act
      const nextState = reducer(initialState, signInFailure({ error: error.message }));

      // Assert
      const rootState = { auth: nextState };
      expect(selectIsAuthenticated(rootState)).toEqual(false);
      expect(selectToken(rootState)).toEqual('');
      expect(selectUserName(rootState)).toEqual('');
      expect(selectIsLoading(rootState)).toEqual(false);
      expect(selectError(rootState)).toEqual(error.message);
    });
  });
});

Notice that signInStart has less assertions regarding the new state, because current userName and token do not matter to it. Everything else is much in line with what we have discussed so far.

Another subtlety might go unnoticed. Even though the main focus of the tests is the reducer, they end up testing the action creators as well. Those silly errors like typos will get caught here, so we do not need to write a separate suite of tests to prevent them from happening.

The same thing goes for selectors too. Plain selectors are purely declarative code. Memoized selectors for derived data created with createSelector from reselect should not be tested as well. Errors will get caught in the reducer test.

For example, if we had forgotten to change selectUserName and selectToken after refactoring the state shape and left them like this:

// should be state.auth.user.token
export const selectToken = state => state.auth.token;

// should be state.auth.user.userName
export const selectUserName = state => state.auth.userName;

In that case, all test cases above would fail.

Testing Side-Effects

We are getting there, but our slice is not complete yet. It lacks the part that orchestrates the sign-in flow and communicates with the back-end service API.

Redux itself deliberately does not handle side effects. To be able to do that, you need a Redux Middleware that will handle that for you. While you can pick your poison, @reduxjs/toolkit already ships with redux-thunk, so that is what we are going to use.

In this case, the Redux docs has a really good example, so I took it and adapted it to our use case.

In our authSlice.js, we simply add:

// ...
import api from '../../api';

// ...
export const signIn = ({ email, password }) => async dispatch => {
  try {
    dispatch(signInStart());
    const { token, userName } = await api.signIn({
      email,
      password,
    });
    dispatch(signInSuccess({ token, userName }));
  } catch (error) {
    dispatch(signInFailure({ error }));
  }
};

Notice that the signIn function is almost like an action creator, however, instead of returning the action object, it returns a function that receives the dispatch function as a parameter. This is the "action" that will be triggered when the user clicks the "Sign In" button in our application.

This means that functions like signIn are very important to the application, therefore, they should be tested. However, how can we test this in isolation from the api module? Enter Mocks and Stubs.

Since this is an orchestration component, we are not interested in the visible effects it has. Instead, we are interested in the actions that were dispatched from within the thunk according to the response from the API.

So we can change the test file like this:

import configureMockStore from 'redux-mock-store';
import thunk from 'redux-thunk';
// ...
import api from '../../api';

jest.mock('../../api');

const mockStore = configureMockStore([thunk]);

describe('thunks', () => {
    it('creates both signInStart and signInSuccess when sign in succeeds', async () => {
      // Arrange
      const requestPayload = {
        email: 'john.doe@example.com',
        password: 'very secret',
      };
      const responsePayload = {
        token: 'this is a token',
        userName: 'John Doe',
      };
      const store = mockStore(initialState);
      api.signIn.mockResolvedValueOnce(responsePayload);

      // Act
      await store.dispatch(signIn(requestPayload));

      // Assert
      const expectedActions = [signInStart(), signInSuccess(responsePayload)];
      expect(store.getActions()).toEqual(expectedActions);
    });

    it('creates both signInStart and signInFailure when sign in fails', async () => {
      // Arrange
      const requestPayload = {
        email: 'john.doe@example.com',
        password: 'wrong passoword',
      };
      const responseError = new Error('Invalid credentials');
      const store = mockStore(initialState);
      api.signIn.mockRejectedValueOnce(responseError);

      // Act
      await store.dispatch(signIn(requestPayload));

      // Assert
      const expectedActions = [signInStart(), signInFailure({ error: responseError })];
      expect(store.getActions()).toEqual(expectedActions);
    });
  });

So unlike reducers, which are easier to test with Detroit School methodology, we leverage London School style to test our thunks, because that is what makes sense.

Because we are testing implementation details, whenever code changes, our tests must reflect that. In a real-world app, after a successful sign-in, you probably want to redirect the user somewhere. If we were using something like connected-react-router, we would end up with a code like this:

+import { push } from 'connected-react-router';
 // ...
 import api from '../../api';

 // ...
     const { token, userName } = await api.signIn({
       email,
       password,
     });
     dispatch(signInSuccess({ token, userName }));
+    dispatch(push('/'));
   } catch (error) {
     dispatch(signInFailure({ error }));
   }
 // ...

Then we update the assert part of our test case:

+import { push } from 'connected-react-router';
 // ...

 // Assert
 const expectedActions = [
   signInStart(),
   signInSuccess(responsePayload),
+  push('/')
 ];
 expect(store.getActions()).toEqual(expectedActions);
 // ...

This is often a criticism against redux-thunk, but if you decided to use it, that is a trade-off you have to deal with.

Conclusion

When it comes to the real world, there is no single best approach for writing tests. We can and should leverage both Detroit and London styles to effectively test your applications.

For components that behave like pure functions, that is, given some input, produce some deterministic output, Detroit style shines. Our tests can be a little bit more coarse-grained, as having perfect isolation does not add much value to them. Where exactly we should draw the line? Like most good questions, the answer is "It depends".

In Redux, I have concluded that a slice is the smallest standalone unit that exists. It makes little to no sense to write isolated tests for their sub-components, like reducers, action creators and selectors. We test them together. If any of them is broken, the tests will show us and it will be easy to find out which one.

On the other hand, when our components exist solely for orchestration purposes, then London-style tests are the way to go. Since we are testing implementation details, tests should be as fine-grained as they get, leveraging mocks, stubs, spies and whatever else we need. However, this comes with a burden of harder maintainability.

When using redux-thunk, what we should test is that our thunk is dispatching the appropriate actions in the same sequence we would expect. Helpers like redux-mock-store eases the task for us, as it exposes more of the internal state of the store than Redux native store.

T-th-tha-that's a-all f-fo-fo-folks!

A few weeks ago I wrote about my journey into Terraform. You can read more about this quest here:

Now I’ll try to summarize some learnings I had during the process.

Foreword

It’s ironic that the Portuguese translation of Earth is “Terra”.

Before really started exploring this infrastructure-as-code world, I’ve heard a lot of blabbering about Terraform.

— Terraform is really easy, you just write some code and bam!, there is your infrastructure…

— Terraform uses declarative syntax, it can’t be that hard…

— Terraform is cloud-agnostic, write your infrastructure code once and run it on AWS, GoogleCloud, Azure, Bluemix or wherever…

— Terraform can prevent children from starving in sub-Sahan Africa…

Of course that’s all bullshit!

Don’t get me wrong, I still think Terraform is a fantastic tool once you get to know it in further details, but the learning curve can be very steep, specially if you don’t have a good understanding of how the underlying provider works.

Here are some things I wish I knew before diving into this quest.

Terraform is not mature yet

Yo, easy there… You ain’t grown up enough for this!

Terraform is still under very active development. You can see how serious I am about that by visiting its releases and issues page on Github. By the time I started writing this post, the latest “stable” version available was 0.9.11 and there were 742 issues open only in the main repo (there are also individual repos for each major cloud provider). I myself must have opened a handful of them in the last few weeks.

Because its major version is still 0 , it means you can expect significant breaking changes until v1 is launched. In fact, that’s the most frustrating part when you are trying to get started with it is that if you find that cool article on the internet about how to do X with Terraform, there are high chances that it will be outdated, even if it is just a few months old.

Beware of this before you start putting all your infrastructure in it.

Terraform is not cloud-agnostic

In the end, we are still all a bunch of cloud-believers…

I don’t remember where exactly I heard or read this, but this is probably the most widespread fake news™ about Terraform.

While it does provide support for multiple providers — since full-featured ones, as AWS, to more specific services, such as Github — it’s not like you can describe your infrastructure with generic code that will be translated into provider-specific resources.

In fact, Terraform has resources that maps more or less 1-to-1 to the underlying provider resources, often keeping known jargons as well. For example, an AWS Classic Load Balancer is named aws_elb in Terraform, while the closer equivalent on Microsoft Azure is called azurerm_lb . As you might expect, the configuration parameters for each resource also change, so they are not interchangeable whatsoever.

Therefore, cloud platform migration will continue to suck for the time being, but Terraform can make this task a little (yeah, just a little) less brittle.

Terraform won’t hide the complexity of underlying providers

If you don’t understand how AWS works, Terraform will not make your life easier. Indeed, it might make it worse, because you’ll have to deal with both AWS and Terraform quirks.

AWS has regional and global services. For instance, EC2 is regional — which means that an auto-scaling group in North Virginia has nothing to do with one in São Paulo. Therefore, they can have the same name (which are ASG identifiers). IAM, in its turn, is global, which means that when you define a role, it can be used anywhere. Then there is S3. S3 is a hybrid: while it has regional scope, its namespace is global, which means you can’t have buckets with the same name, even across different regions.

Terraform ain’t gonna help you with that sort of things. When you run teraform plan, it will tell you it’s all ok. Then you will happily proceed to terraform apply and instead of running smoothly as you’d expect, it’s going to blow up on your face and tell you how stupid you are.

“Avoid code duplication” is so old-fashioned

No, I’m not and advocate of the RCP design pattern (Reuse by Copy & Paste), but, as far as I know, it’s very hard to keep your code generic with Terraform.

For example, if you are nesting modules, to make a parameter of the innermost one externally configurable, you must lift it up to every module in the middle:

# A/main.tf

variable "b" {}

variable "c" {}

module A {
  source="/path/to/module/A"
  b="${var.b}"
  c="${var.c}"
}

# ...

# B/main.tf

variable "b" {}
variable "c" {}

module B {
  source="/path/to/module/B"
  var_c="${var.c}"
}

# ...

# C/main.tf

variable "c" {}

some_provider "some_resource" C {
  some_attribute="${var.c}"
}

You have to declare the variable c 3 times and b 2 times to make it work.

Sometimes you are better off shamelessly copying a resource or module and changing it a little bit then trying to extensively parametrize it in order to make it more general. You’ll thank me later 😉.

If you know a better way of doing this, for the sake of God, Budah, Krishna, Goku, etc., please let me know.

Terraform is full of dirty hacks

HCL (HashiCorp Configuration Language) is the name of the description language used by Terraform (HashiCorp is the company behind it). It has some, let’s say, peculiar ways of solving certain problems.

Warming a pizza with Terraform feels like…

One example: suppose that you want to conditionally create a resource, like this:

variable "custom_sg" {
  description = "Custom security groups for the instance"
  default = ""
}

resource "aws_security_group" "default_sg" {
  count = "${custom_sg == "" ? 1 : 0} # this is a bit odd, but ok
  # More params bellow... they are not relevant
}

resource "aws_instace" "example" {
  securit_groups = ["${custom_sg != "" ? custom_sg : aws_security_group.default_sg.id}"]
  # More params bellow... they are not relevant
}

The code above should work, right? WRONG!

Whenever you use the count parameter in a resource, Terraform will assume it is list of resources, even if the only possible values are 0 and 1 . So the code above you crash right in front of your eyes (the good news is that it fails on plan stage).

Well, since your aws_security_group.default_sg is a list, you cannot access its params directly. Instead, you have to point to individual items on that list. Terraform let’s you do that using a syntax like resource.name.<n>.param , where n is a number.

So I can just declare it like this:

aws_security_group.default_sg.0.id

Nice try, but you can’t! Because the way HCL is implemented, it has to parse the whole template, even if the value is not used. So in the case that you set default_sg , there is no item in aws_security_group.default to be referenced by 0, so Terraform will once again beat you to the ground and then laugh on your crying-baby face.

The “official” workaround is this:

resource "aws_instace" "example" {
  securit_groups = ["${custom_sg != "" ? custom_sg : join("", aws_security_group.default_sg.*.id)}"] # WTF dude?
  # More params bellow... they are not relevant
}

Seriously? Where the heck did this come from? That * means all elements in the list. When none is present, it returns an empty list. That join is just a regular List -> String function. When the list is empty, it returns an empty string. When it has one element, it returns the parameter I want.

There are lots of dirty little hacks like this. The place to find them is the issues page.

Don’t use nested/in-line resources

No, seriously… Don’t do that… this gave me some pain in the arse.

In case you have no idea what I’m talking about, Terraform allows you to define some resources within its “parent” as well as a standalone resource with a reference to it.

The reason why I recommend doing so is because, at least for us, a common use case is the need to extend security groups, route tables and other resources that support in-line resources. And the only way we can do that from outside the module definition is with standalone resources pointing to the resource within the module.

I have defined a VPC module, which contains the basic definition for a VPC. This makes the creation of VPCs on multiple regions a piece of cake. However, our main infrastructure is located in N. Virginia, so its configurations should be a little bit different from the other ones (not different enough to justify code duplication, though).

Initially I had something like this in our VPC module definition:

# ...
resource "aws_route_table" "public" {
  vpc_id = "${aws_vpc.cluster_vpc.id}"
  route {
    destination_cidr_block = "0.0.0.0/0"
    gateway_id = "${aws_internet_gateway.default_ig.id}"
  }
}
# ...

Then I would create our VPCs by reusing the module:

module "us-east-1" {
  source = "../../modules/vpc"
  aws_region = "us-east-1"
  vpc_cidr_block = "10.0.0.0/16"
}

module "us-east-2" {
  source = "../../modules/vpc"
  aws_region = "us-east-2"
  vpc_cidr_block = "10.1.0.0/16"
}
# ...

However, for us-east-1, I needed to create a VPC peering between another VPC we have in this region. To make this work, beyond creating the peering connection, I need to modify the route table, adding an entry that will properly route the traffic.

# ...
resource "aws_route" "peer_vpc" {
  route_table_id = "${module.us-east-1.public_route_table_id}"
  destination_cidr_block = "172.16.0.0"
  vpc_peering_connection_id = "pcx-xxxxxx"
}
# ...

This should work, right?

The behavior I observed when mixing both styles was that if the standalone resources didn’t exist, they would be created. However, once created, if I ran terraform apply again, they would be deleted. If I tried one more time, they would be created and so on…

The gotcha is that you can’t mix both. I had to dig through some Terraform issues on Github to learn that.

Fortunately, this is now clear in the documentation, as it’s stated for example in aws_route_table resource:

NOTE on Route Tables and Routes: Terraform currently provides both a standalone Route resource and a Route Table resource with routes defined in-line. At this time you cannot use a Route Table with in-line routes in conjunction with any Route resources. Doing so will cause a conflict of rule settings and will overwrite rules.

The solution was to extract the in-line route within the module to its own resource:

# ...
resource "aws_route" "internet" {
  route_table_id = "${aws_route_table.public.id}"

  destination_cidr_block = "0.0.0.0/0"
  gateway_id = "${aws_internet_gateway.default_ig.id}"
}
# ..

This way I can amend the route table outside the module without pulling my hair out.

Conclusion

It has been great to work with Terraform. Sometimes I think it would feel better to burn alive, but still great.

From the time I started until now, I noticed improvements in the documentation, which might help to make the learning curve less steep.

Since I’m still learning, I probably got some things wrong. So there might be room to improve this article in the future as I discover new/better patterns.

Did you like what you just read? Why don’t you buy me a beer with tippin.me?

The tale of a joy/pain-ful lonely journey into the infrastructure-as-code world

Once upon a time…

I might have said this a couple of times, but the best and the worst part of working in a startup environment is the diversity of activities you enroll in. One day you are debugging some bug in a Angular/React front-end page, the other you are pulling your hair off with some obscure infrastructure problem.

That is good because you sort of get an intensive crash course in many disciplines, being forced to learn things very fast — like front-end, back-end, infrastructure, design, business, billing, you name it — that would take you a lot of time (if at all) in a mid-size/large company. On the other hand, having to deal with such broad aspects together (sometimes, simultaneously more than one), means that you will not be a specialist on any of those subjects and the constant context-switching may drive your productivity low.

At Revmob, we are currently on an effort to strike a balance between flexibility and specialization in the tech team, with well defined squads and responsibilities. However, we still do not have an ops team, neither someone whose primary focus is developing and maintaining infrastructure, so basically everybody is responsible for it.

But, as we say here in Brazil, literally translated, “a dog with two owners starves to death” (I believe the closest saying in English would be “too many cooks spoil the broth”). With the lack of standards and, at some extent, of knowledge, whenever there was a problem with our infrastructure, we used to go full panic mode, basically dropping and recreating everything.

This is the poor infra-dog :(

We have relied on AWS Elastic Beanstalk for a while and, whilst it reduces the effort to setup a basic web or worker-based standard infrastructure, once you need more flexibility, you end up struggling against the multiple levels of abstraction, resources that are hard to find, the complexity of CloudFormation stacks, configuration files using the .ebextensions directory inside projects leading to lots of code duplication, etc.

However, our main pain point was that the warm up time for a new instance being too high (about 7~8 minutes) and we verified that most of that time was spent on Beanstalk setup than on our applications. We managed to make some tweaks, by creating a custom AMI based on some Beanstalk-ready default images, so the warm up time went down to about 4~5 minutes. Still, this was not good enough for our highly elastic environments (oh, the irony!), as certain unpredictable high traffic spikes (very frequent in mobile advertising networks) could not be properly handled.

We needed to find a way around this problem.

Beyond that, we were facing the need for our first multi-region application. Since this would require a lot of work to setup by hand, we felt motivated to explore alternatives that could solve both problems.

I had heard about Terraform a couple of times and our CTO told us that the ops team in his previous company used it to manage infrastructure. So we decided to give it a shot.

If you are interested in a more technical article, check this out:

Then the journey begins…

I was chosen/volunteered to be in charge of exploring this brave new world. The initial idea was that I would spend one sprint (which means 1 week for us) in order to make the setup needed to run two new applications, including the dreaded multi-region one.

We thought it would be hard to do pairing since none of us had the slightest idea of how that thing worked, so it was decided that I by myself would be my own team for that sprint.

I started by doing the good ol’ hello world. The installation is quite easy, just download a binary file and put it in your PATH . Then I followed all the steps of the guide. Piece of cake.

Now let’s get to the real work!

Troubles in paradise

I found a nice article series about Terraform, written by some company, other than HashiCorp — the creator of the whole thing — and started to read it. While it gave me some cool insights, when I started to try the code out, it simply did not work. I tried to move forward with the series, but ended up stumbling on the same problem.

I got kind of pissed and completely removed it from my bookmarks and never looked back. Later I found out that those articles were based on a previous version of Terraform, which was not compatible with the most recent I was using.

Then I decided to dive into the official documentation and continue by trial and error. That kind of worked for a while, however I was going in baby steps.

Differently from software development, where you can easily setup an automated test suite, put them to run on watch mode and get instant feedback when you make any changes, working with Terraform seemed to be some orders of magnitude slower. Many times I caught myself just staring at my screen to see what was going to happen after I changed a resource. But at least I felt I was moving forward.

At some point, however, I hit a wall. Then I went pray to our lord Google.

Oh, mighty Google, please save this poor siner’s soul

Nonetheless, the all-powerful was not very merciful at me. Each link of “how to do X with Terraform” led me to a completely different approach, which probably did not even work. Some articles, only a few months old, were already outdated (it seems like Terraform 0.9.x was a completely new beast from 0.8.x).

By leaps and bounds I could reach a somewhat satisfactory result for the simpler application I had. It was kind of nice to see all that code being translated into actual Load Balancers, Auto-Scaling Groups, VPCs, Security Groups, CloudWatch metrics and alarms, and so on.

It was already Friday an I was happy that I was done. Now I just needed to change some input parameters and Terraform would automagically reproduce the setup I had for the other application, then I would just need to make it multi-regional. Easy-peasy.

That’s what I thought.

Yeah, not so fast, hasty!

It turned out that Terraform does not work quite like I thought it would. I had gotten it all wrong. I had not understood that modules play a central role in Terraform code reuse patterns, so I just ignored them altogether. It was too late, the sprint was over and I had failed.

During the grooming meeting on the same Friday, I gave the status to my boss and he asked me if I wanted more time. I told him I would get everything done until the next Wednesday (yeah, I was that stupid). He agreed to allow me to continue my quest during the following week.

Also, in the meantime, since we are only two in my squad so far, I sort of left my squad-mate alone for the sprint and would leave him by himself again. He got sick of waiting for my return and joined another squad for the next sprint.

I feel you, SpongeBob

Taking a step back

During the weekend I was reflecting on what went wrong and I concluded that I was in frenzy mode . I needed to take it slower, gain more muscle and agility in order to win the challenge.

I focused the first days on understanding the internals of Terraform and to review some infrastructure-related and AWS specific concepts. For example, I had to remember a few things of that Computer Networks classes I had never payed too much attention during college.

I would probably never have to write a Route Table by hand nor partition a VPC into several subnets my entire life, I don’t need this bull$*!7.

I realized that while I was complaining about Beanstalk, it had spoiled me a a lot. But it wasn’t long before I could find that information in a dusty corner in my mind.

Oh, so that’s how I do to calculate the number of bits I need in my subnet mask?

After those mental push ups, I was back on track. I had finally understood why I needed modules, how to partition Terraform state and how to use it remotely. My code started to get reusable.

Of course, the screen staring wasn’t entirely over, so obviously I couldn’t get everything done by Wednesday as I promised.

The epic final battle

On Thursday morning I made a promise to myself.

I won’t drag this $*!7 with me another sprint. I will finish it until tomorrow.

Time to suit up!

So it begun. I was back on frenzy mode, but this time a little more conscious. My strategy was to deploy both two applications in a single region and, after making sure everything was running smoothly, I would replicate one of them across AWS 14 regions.

I was a little bit more savvy on Terraform than before, so my main struggle became the AWS quirks. One of the problems we had with Beanstalk finding specific resources through its console (have you tried to find the load balancer of an environment?), so I decided to use meaningful names for all resources I could.

Then came the name collisions. Luckily this was already solved by Terraform, allowing me to use name prefixes for certain resource type, generating a new name every time, while keeping its meaningfulness. I was also trying to use CodeDeploy, something we had never used before. This gave me a bit of work as well.

When I had figured out the problems with AWS, I felt the need for refactoring the infrastructure code to make it more manageable. I was ready to throw everything away and rebuild my infrastructure when I found out that Terraform allows you to move state.

That moment was the first time I loved Terraform…

This feature saved me some time, as I could keep what I had created and just move the state around. I was getting close.

Before the setup was 100% functional for a single region, I got into a fight with PM2 as well (I’m a bit feisty, you might be thinking). Again it was something I had never used and, as it was late in the night on a Friday, the tiredness started to consume me. PM2 was able to throw a few strong punches against me.

PM2 looked like Apollo Creed…

But, as Rocky Balboa himself, I managed to defeat the dreaded adversary and was ready to move on to the next challenge.

Time to go multi-regional!

It was about 5 a.m. of Saturday when I finished the first multi-regional setup. Sleep was a long lost. I wanted to finish everything before going home. I had to make some minor tweaks and after a few minutes — bam — there was my first multi-regional application deployed and ready to go.

Just to be sure, I ran a little test suite on Postman I had setup during development. The first time it failed for Japan, but I realized that the infrastructure was not fully set yet. One more try and everything ran smoothly \o/. It was 5:53 a.m.

At that moment, I started to loudly play “We are the Champions” at the Office.

Of course, there was no one there to listen… So I put if off, asked for my Uber and went home to get the sleep of the righteous.

What would I do different?

I’ve made several mistakes during those two weeks. I probably could’ve stressed less, suffered less, slept better and drunk more water during the process.

I believe our assumption that would be better to make this a single person job was wrong. There are so many things involved, it’s easy to screw up without someone helping you. I would definitely suggest people who are starting with Terraform to do pairing.

Furthermore, it got pretty lonely. Those who know me will probably say that I’m not the most sociable person they know, so trust me. I barely spoke to my teammates for two weeks, even though I was sitting right next to them. I couldn’t ask for insights — since nobody knew squad about what I was doing — or even just complain about my fate.

I also think that I aimed too high. I had to learn/remember so many things at once: Terraform, CodeDeploy, PM2, Computer Networks, AWS quirks and how to setup a full-blown multi-regional application. It was just too much. I should’ve started small and make improvements with time.

The end (?)

That was my story. I tried to summarize it as much as possible, but failed miserably.

I will probably write another post from a more technical perspective within the next few days, sharing the little I learned from this endeavor. Keep posted.

Did you like what you just read? Buy me a beer with tippin.me.