Black Box Programming

Sometimes you need to zoom in instead of zoom out

This might be counterintuitive, but hear me out. I think that a large part of programming is about zooming in rather than zooming out. So often, we are told to “look at the bigger picture” or “see the system as a whole”, but that’s not always the best advice. Sure, when you are building a new system, or trying to figure out how a new service will fit into the overall system architecture, you need to look at the entire system. However, once you are actually building that system, you need to zoom in. This isn’t always easy to do, and there are tradeoffs to doing it, but it’s often necessary to get work done.

I don’t know when this occurred to me, but I found myself coming up with this answer on the spot the other day when talking with some engineers. Really it comes down to productivity, a topic on which I’ve watched a ton of YouTube content, read a few books, and listened to a podcast or two. Ironically, that was often more for my personal life than my professional one. I’ve applied different strategies a few times for work productivity, but nothing ever quite stuck. That’s partly due to primarily working at startups for the past few years, as they are inherently prone to changing direction rather quickly. It wasn’t until I was helping someone address an “analysis paralysis” problem that I realized that all of that knowledge had slightly transformed my approach to solving problems.

I often start with getting the big picture thinking out of the way, which is basically just a high level view of “this is generally how I think it should work”. Then I dig deeper. I’ll start down a code path of building an API call and figuring out what data I need. I’ll code it up with some functions (sometimes just as comments) that I think we’ll need.

// controllers/books.js
const findBooks = async (req, res) => {
  const books = await models.books.find();

  res.status(200).send(books);
};

Disclaimer: Please don’t take these code examples or file structure as industry standards. They are greatly simplified to illustrate a point.

Then I’ll realize I need to build the database models functions before I can continue. I’ll go build those, commit it, and then go back to the API call (whether that be a REST controller or a GraphQL resolver).

// models/books.js
import { driver } from '../lib/neo4j-utils.js';

const find = async () => {
  const session = driver.session();
  const result = session.run(
    `
      MATCH (b:Book)
      RETURN b
    `
  );
  session.close();

  const books = result.records.map(
    record => record.get('b').properties
  );
  return books;
};

Disclaimer: You may not need a separate models structure if your database has a standard Object-Relational Mapping (ORM) layer.

Then I realize I need a helper to manage the pagination logic. So I code that up and add the missing pagination logic to the models function I created earlier.

// controllers/books.js
import models from './models/index.js';
import { paginationHelper } from './helpers.js';

const findBooks = async (req, res) => {
  const { limit, page } = req.query;

  const books = await models.books.find({ limit, page });

  const paginatedBooks = paginationHelper(books, { limit, page });

  res.status(200).send(paginatedBooks);
};

// models/books.js
import { driver } from '../lib/neo4j-utils.js';

const find = async ({ limit, page }) => {
  const session = driver.session();
  const result = session.run(
    `
      MATCH (b:Book)
      RETURN b
      SKIP $page * $limit 
      LIMIT $limit
    `,
    { limit, page }
  );
  session.close();

  const books = result.records.map(
    record => record.get('b').properties
  );
  return books;
};

Finally, I can hook that up to a route and test it to ensure it’s all working and returning the data that the UI needs to display. Maybe I’ll even be hooking it up to the UI if I’m building the feature end-to-end. This might sound pretty standard, but what I didn’t quite explain is my thought process through it. When I was starting on the API call, I wasn’t thinking about the models, or the pagination helper, or even how the UI was going to display the data. I was only thinking about the data that needed to be returned in isolation, and any inputs I needed for that particular piece. I was effectively thinking about it as a “black box”, which is a very common pattern in computer programming.

This all might sound obvious to some, but I don’t think it always is. As an engineer, it can be easy to get caught up always thinking about the big picture (particularly whenever you are a full-stack developer tasked with the feature end-to-end). While it is generally good to keep that in mind, it can cause analysis paralysis. You don’t know where to start, or you do a little bit here and a little bit there. You work on something for hours and by the end of the day you have a bunch of half-baked code. None of it works. You can’t commit any of it to git and tomorrow you’ll have to figure out how to finish all of these pieces and get them in enough of a working state to test it. Then you’ll have this huge git commit and PR that will take awhile for someone to look at.

The irony of me talking about how to avoid analysis paralysis is not lost on me. But I think it’s precisely because of my experience having that problem that helps me help others overcome it.

— Me, addressing my friends reading this going “huh?”

If you take a beat and focus on one piece at a time, you can actually wholly finish a few pieces by the end of the day and commit them. You think of them as a black box. You ignore everything else and only consider the inputs and the outputs. You know you will have time to integrate that into the big picture later. Set all of that aside and get the one function working. My example might be a simplistic one, but extrapolate that out to a function that needs to call three other complicated functions and two external services, and you end up trying to boil the ocean. The beauty of this is realized the next day. You jump on your computer and you know that you finished all of the complicated functions yesterday. You know they will provide the right result if you provide the right input. They “just work”. You don’t even need to think about them until there is a bug. Now you get to focus on the function that is doing the orchestration as its own black box. I know this doesn’t account for issues encountered in the integration testing phase, but it’s often much easier to fix a problem in existing code than to create it from nothing.

Let’s think about this from another angle: unknowns. When you are creating something new, there are often unknowns. Whether it be in how something is going to work, how it is going to look, or what scale it will support. Treating some unknowns as black boxes can help you avoid the paralysis of worrying about some unknown you have yet to figure out and focus on what you do know. In the previous example, you could code the orchestration and API call layer without needing to understand how the external services are going to work. Sure, that could change how the overall function will work in the future, but if you know you can figure that out later, it can let you keep moving now. This really helps me when I don’t know exactly how I want something to work. I know I need to figure it out, but moving onto the other problems I need to solve (and already know how to solve) reduces the mountain of work I need to do. It relieves some of the stress of seeing the big pile of cans at the end of the road. I get a chance to remove the cans I can easily handle, and ignore (for the time being) the ones I can’t. Sometimes, your brain will continue to work on the unknown problem that you don’t know how to solve while you are solving the ones you do (even though you aren’t explicitly thinking about it, kind of like taking a break and walking away). This isn’t to say that we shouldn’t be aware of the unknowns before we start, but we shouldn't let them stop us from solving the problems we can.

At this point, I probably still haven’t convinced you that zooming in is better than zooming out. You are probably thinking “what about X?”, and that is an excellent point (whatever it is because I know there are multiple but I didn’t spend all time thinking of them, see what I did there?). I think most of the time when engineers are being told to zoom out is earlier in their careers when they aren’t seeing the whole problem. They only think about the one small thing and then have to completely rewrite it after a code review because they forgot about these other three things that won’t work with that design. Honestly, when you are more junior, you don’t know enough about the rest of the system yet to understand how the piece you are working on fits into the bigger picture, so you are encouraged to zoom out and think about the bigger picture. That works until you get to the point where you cannot write anything without spending multiple days thinking of all of the implications of various solutions. That’s when you need to zoom back in just to get something done. Get one piece done and be comfortable with coming back to modify it when you figure out other parts of the system. Over time, you will learn and get better at being able to account for other parts of the system even when focused on one small part, without it affecting your overall productivity.