DEV Community: Erik

How to Create a Sparkly-Spoiler Effect like the one in Threads Mobile App

Erik — Sat, 18 Apr 2026 20:29:11 +0000

If you use Meta’s Threads app on a mobile device, you may have noticed an interesting effect: the spoiler tag that hides text behind a sparkly veil. In this article, I will show you how to implement this effect in the browser using HTML, CSS, and JavaScript.

NOTE: You can scroll to the bottom of this article to find the completed HTML, CSS, and JavaScript code if you don’t want to read the whole article.

The Spoiler Effect on Threads in a Nutshell

If you’re unfamiliar with the “spoiler” tag, it is a Threads feature in which users can selectively hide part of their post until a user is ready to see it. Look at my example below where I am about to post “Highlight the word “hello” to mark as a spoiler!”

Notice how I have highlighted the word “hello” and see a button that says “Mark spoiler.” I click that and then post the thread. See how it looks on my desktop browser:

The word “hello” is now obscured by a gray bar. If someone reading my post wants to see the word I’ve hidden, they will have to click on the gray bar.

The above screenshot is what a reader will see after clicking the gray bar; they can now see the word that I’ve hidden.

In a nutshell, that is how “Mark spoiler” works in Threads; the author selects some text to selectively hide and the user must intentionally perform an action to reveal the hidden text. The intended utility of this feature is for when the user wants to talk about a plot twist or conclusion in a story without giving away–or “spoiling”–the surprise for readers who haven’t yet gotten to that part. It protects readers from passively reading a spoiler by forcing them to deliberately reveal the hidden text.

Note, however, this spoiler bar is a little more fun on the mobile app. The next screenshot shows the same post but from the mobile Threads app:

Notice how instead of a gray bar, it’s now a sparkling obfuscation. How cool! This sparkly spoiler effect is what this article will teach you how to do.

Actual Spoiler Alert!

I am going to be talking about a plot twist from Star Wars: Episode V – The Empire Strikes Back. Even though the movie came out in 1980, it’s possible you haven’t seen it yet and I would hate to ruin the surprise for you. If you plan on watching this movie and have no clue about its plot, this article is definitely going to spoil it for you!

Continue at your own risk!

How to Achieve this Effect

First, we will plan out how to implement this effect before getting into the technical details. Here are the things we need to do:

Mark text as spoiler
Create the effect
Apply that effect to appropriately tagged text
Remove the effect when a user clicks

Let’s begin!

Mark text as spoiler

First, let’s create an HTML file called “spoiler_text.html” and fill it out with some standard HTML boilerplate code:

<body>
    <p>
        The biggest plot twist of all time is:
        <span class="spoiler">
            Darth Vader is Luke Skywalker's father!
        </span>
    </p>
</body>

This is just a basic HTML page in which we’ve added some text about the biggest plot twist of all time. Right now, the page is pretty bare bones and should look like this in your browser:

Since the background and text in Threads is more of a dark theme, let’s apply that to our HTML page. This isn’t part of the spoilers effect, it’s just something that will make it a little more pleasant to look at later. Let’s create a CSS file called style.css and make the page a little easier to look at. Here’s what I came up with:

:root {
  --bg: #0f1115;
  --text: #f4f7fb;
}

body {
  min-height: 25vh;
  display: grid;
  place-items: center;
  background: var(--bg);
  color: var(--text);
}

First, I create a :root rule to store some CSS variables. As of right now, we only have two variables, bg and text, so we could just use the values instead of variables in the body rule. However, later in this project we’re going to have a few more variables and I think it’s best to keep them all in one place, so we’ll start putting our variables in the :root rule early.

Next, I set the min-height, display, and place-items values for the web page. The specific values I’ve added in the above CSS will make the text appear in the middle and slightly below the very top of the browser window. Finally, I use the bg and text variables to set the background and color attributes for the webpage. Now, just below the <title> tag in the HTML file, I add this line:

<link rel="stylesheet" href="style.css" />

And now the browser should look like this:

Feel free to adjust the bg and text variables so that the page you’re looking at is aesthetically pleasing to you.

Now, we need to mark the spoiler text somehow. For now, I’m going to put the spoiler part in a span tag of class “spoiler.” Doing so won’t actually make anything change visually yet, we’re just preemptively marking the spoiler text before creating the effect. The body tag of the HTML file should now look like this:

<body>
    <p>
        The biggest plot twist of all time is:
        <span class="spoiler">
            Darth Vader is Luke Skywalker's father!
        </span>
    </p>
</body>

Now, let’s create the effect!

Create the Effect

To start creating the effect, we’ll take a small step by just writing a little CSS to blur out the tagged text. In our style.css file, we’ll need to make a rule for the spoiler class. We’ll make use of the built-in CSS blur function and make the cursor a pointer. Here’s how my css for the spoiler class looks:

.spoiler {
  filter: blur(7px);
  cursor: pointer;
  position: relative;
}

With this CSS code, we set the filter attribute on the spoiler class to blur(7px) and set the cursor attribute to pointer. As you might guess from the blur function, using it to set a filter value makes the element blurry. The number you pass to the function defines the radius of the blur; the higher the number, the larger the blur effect. We set the cursor attribute to pointer so that when the user hovers their mouse over the effect, the cursor changes to a pointer and lets them know the effect will do something if they click on it.

Additionally, we set the position attribute to relative so that the span doesn’t interfere with any of the other elements in the page

With the new CSS, the page in the browser should now look like this:

The above screenshot shows that our spoiler text is now blurred out thanks to the filter: blur(7px) line in our CSS.

Before we can finally build the sparkle animation and the toggle feature, we have to make a few adjustments to the structure of our HTML. We should actually put two more span elements inside of this spoiler span: one to wrap the content, and one to contain the blur effect. This step will make implementing the toggle effect along with the sparkle animation a little easier.

I will create one span class called spoiler-content, and one called spoiler-blur. The spoiler-content span will wrap the text, while the spoiler-content span won’t wrap anything, it will just sit inside the spoiler span. Here’s what the HTMLin the body tag looks like now:

<p>
    The biggest plot twist of all time is:
    <span class="spoiler" data-hidden="true">
        <span class="spoiler-content">
            Darth Vader is Luke Skywalker's father!
        </span>
        <span class="spoiler-blur" />
    </span>
</p>

And the CSS now has to be adjusted as such:

.spoiler {
  cursor: pointer;
  position: relative;
}

.spoiler[data-hidden="true"] .spoiler-content {
  filter: blur(7px);
}

With these updates, I’m making use of the data-hidden attribute by setting it to “true” on the spoiler span and then creating a CSS rule that applies the blur to any spoiler-content span inside of the spoiler span. We will later use this attribute when we create the toggling feature of this effect.

Now, we’re ready to start adding the sparkle effect.

Adding the Sparkle Effect

To add the sparkle effect, we’re going to make use of two things: CSS key frames and some JavaScript. The general strategy will be like this:

Create a CSS spark class to make each individual sparkle-blur span covering the spoiler text
Append several spark elements as children to the spoiler-blur span with JavaScript
1. We will create a function for generating random numbers
2. We will set a few custom properties on the element with random values (from the random function mentioned above) that will be used in the CSS animations/keyframes to create the overall “sparkle effect”
Create a twinkle CSS keyframe that will make the sparks appear to be twinkling
Create a drift CSS keyframe that will make the sparks appear to be moving

First, we’ll create the spark class with just a few of its necessary properties. I defined mine like this:

.spark {
  position: absolute;
  width: 4px;
  height: 4px;
  border-radius: 50%;
  background: var(--spoiler-dot);
}

The position: absolute line will allow each individual sparkle to take up space in its overlay (the spoiler span) without disrupting the position of other elements. The three lines for width, height, and border-radius make the sparkles into perfect circles. If you prefer squares, you can remove the border-radius line, if you like vertical lines, you can play with the width and height properties. The background uses a CSS variable that you can set in the :root rule, I set mine to a white-ish effect like so: --spoiler-dot: rgba(255,255,255,0.85);.

This element defines the style for each individual “sparkle” in our sparkly-spoiler effect. We’ll need to make several sparkle elements of random sizes and append them to random places on the spoiler-blur span element. Create a JavaScript file and call it sparkle.js then import it into the HTML’s body like this:

<script src="sparkle.js"></script>

NOTE: You must put this line inside the body tag of the HTML, it won’t work if you put it in the head tag.

Now, in sparkle.js, we’ll start by making a convenience function for returning random values, since we’ll rely on randomness to get the sparkle effect. I wrote my random function to take a rscale variable and multiply it by a number generated by the Math.random function:

function random(rscale) {
    return Math.random() * rscale;
}

Next, we’ll make a function called buildSparkles that takes in a span element, and the number of sparkles we want to create. Each sparkle we create will be created with a starting 2-dimensional coordinate (an x and y representing the left and top properties), an ending 2-dimensional coordinate, a duration of time in which we want the sparkle to move between those two coordinates, a duration of time for which the sparkle should “twinkle”, and a scale variable to control size; all of these values will be randomly created for each sparkle.

Once each sparkle element is built, we’ll append it to spoiler-blur span we passed to the function. Here’s how I wrote my function:

function buildSparkles(spoilerBlurSpan, sparkleCount) {
    for (let i = 0; i < sparkleCount; i++) {
        const s = document.createElement("span");
        s.className = "spark";
        const x0 = random(100);
        const y0 = random(100);
        const x1 = Math.max(0, Math.min(100, x0 + random(18)));
        const y1 = Math.max(0, Math.min(100, y0 + random(14)));
        s.style.left = `${x0}%`;
s.style.top = `${y0}%`;
        s.style.setProperty("--x0", "0px");
        s.style.setProperty("--y0", "0px");
        s.style.setProperty("--x1", `${x1 - x0}px`);
        s.style.setProperty("--y1", `${y1 - y0}px`);
        s.style.setProperty("--dur", `${random(60)}s`);
        s.style.setProperty("--twinkle", `${random(1.2)}s`);
        s.style.setProperty("--scale", `${random(1.5)}`);

        spoilerBlurSpan.appendChild(s)
    }
}

In the function signature, we take spoilerBlurSpan as the first argument. This will be a span somewhere on the web page that has been marked as containing spoiler text. We also take sparkleCount, which is how many sparkles we want to put on one span. Then, we enter the for loop.

Each iteration of the for loop creates one spark element. The first two lines create a span element and assign it the class name spark.

Next, we set x0, y0, x1, and y1; these variables are basically two sets of random x, y coordinates. Later, we will create a CSS animation called drift that will make the sparkle move between those two coordinates.

The lines setting left and top are putting span elements at the randomly generated x and y coordinates we just generated. The next four setProperty lines are assigning the properties and initial values of x0, y0, x1, and y1 on the sparkle span we just created. The need for these properties will become clear in a bit when we create the CSS animations.

Finally, we use the last three lines to set properties called dur, twinkle, and scale. We have to set these properties and their initial values to be used by the CSS animation we’re about to create. The dur property is going to be used to control the duration of time a sparkle spends moving between its two x,y coordinates. The twinkle property is going to control the “speed” at which each sparkle twinkles (switches between low and high opacity). Finally, the scale property is just going to impact the size of each twinkle.

After setting all these properties, we then use the built-inappendChild method to add the newly created sparkle span to the spoilerBlurSpan we got in the function signature.

Finally, we end the sparkle.js file with a few more lines:

function wireSpoiler(spoilerBlurSpan) {
    buildSparkles(spoilerBlurSpan, 75);
}

document.querySelectorAll(".spoiler-blur").forEach(wireSpoiler);

We create one more function called wireSpoiler that takes a spoilerBlurSpan as an argument so that it can send it to the spoilerBlurSpan function along with the number of sparkles to make–in the above example, 75.

Finally, we use querySelectorAll to get all spans on the web page of the class spoiler-blur and pass it to the wireSpoiler method. This ensures that every single spoiler span on our page will get the sparkle-effect, even though we only have one spoiler span on our page.

Now, we move to the CSS.

First, lets create the animations using @keyframes. We’ll create two, drift and twinkle. Add this CSS to the bottom of your style.css file:

@keyframes drift {
  from {
    transform: translate(var(--x0), var(--y0)) scale(var(--scale));
  }
  to {
    transform: translate(var(--x1), var(--y1)) scale(var(--scale));
  }
}

@keyframes twinkle {
  from { opacity: 0.25; }
  to   { opacity: 0.95; }
}

The drift keyframe makes use of the CSS function translate which moves an element to a given location. Since this will be used in an animation property later, we give it a from and to property to define the two positions the element should be moved between. In this case we are using the variables x0 and y0 to define the sparkle‘s starting point, and x1 and y1 as their ending point. Remember, we randomly generated these values in the JavaScript file earlier, so each spark we attach to the spoiler-blur element will have its own starting and ending point.

The twinkle keyframe just defines an opacity value for the spark elements to oscillate between. Opacity is the level of an element’s transparency, so going between a low and high value gives the illusion that the element is twinkling.

Now, we update the spark element’s CSS to include these keyframes as animations. The element’s entire CSS rule should look like this:

.spark {
  position: absolute;
  width: 4px;
  height: 4px;
  border-radius: 50%;
  background: var(--spoiler-dot);
  animation:
    drift var(--dur) linear infinite,
    twinkle var(--twinkle) ease-in-out infinite alternate;
}

The change here is the addition of animation to the spark class and adding drift and twinkle. For drift, we pass the randomly-generated dur value we defined in the JavaScript; this value defines how long it takes the spark to move from its starting coordinates to its ending coordinates. We also pass the arguments linear and infinite to the animation, ensuring the movement is in a straight line and repeats as long as the page is loaded.

Similarly, we pass the randomly-generated twinkle variable to the twinkle keyframe animation, this variable defines the amount of time it takes the spark to go from its starting opacity to its ending opacity. The alternate argument ensures the opacity value doesn’t “start over” once it reaches its ending opacity and instead eases back to the starting value.

With this final addition, we should have a nice sparkly veil over our spoiler content. Here’s what mine looks like:

Note that since we are randomly generating a lot of values, your example might not look exactly like mine, and if you refresh the page, it’ll look different again. But as long as you see sparkles twinkling and drifting over your blurred-out spoiler text, you’ve done everything right so far!

The last thing to do is add the toggle effect!

Adding the Toggle Effect

The point of the spoiler tag is to prevent people from seeing something they’re not ready to see. But for users that are willing to risk spoiling a piece of media, we have to add the ability to remove the sparkly veil and reveal the hidden text. We will add that effect now.

The strategy for how we will do this is as follows:

Add a function that sets the spoiler span’s data-hidden attribute to the opposite of what it currently is (if it’s true, set it to false and vice versa)
Add this function to the “click” event listener on the spoiler span
Update the CSS of attributes inside of a spoiler span with a data-hidden="false" attribute

Part one and two are going to happen simultaneously inside of the wireSpoiler function. The first thing we’ll do is get the spoiler-blur element’s parent element (which should be the spoiler span). This is accomplished via this one JavaScript line:

let spoilerSpan = spoilerBlurSpan.parentElement;

Now, just below that line, we’ll create a function called toggle. This function will get the data-hidden attribute of the spoiler and set a variable called hidden to the opposite value. Then, we’ll use the setAttribute method to set data-hidden to the value of the new hidden variable. This is what is meant by “toggling.”

Finally, we’ll use the addEventListener method to add the toggle method to that element’s click event. Altogether, the wireSpoiler function should now look like this:

function wireSpoiler(spoilerBlurSpan) {
    buildSparkles(spoilerBlurSpan, 75);
    let spoilerSpan = spoilerBlurSpan.parentElement;

    function toggle() {
        const hidden = spoilerSpan.getAttribute("data-hidden") === "true";
        spoilerSpan.setAttribute("data-hidden", String(!hidden))
    }

    spoilerSpan.addEventListener("click", toggle);
}

In the above code added a few lines to the wireSpoiler function. First, we got the spoilerBlur element’s parent element, which should be the spoiler span. Then, we create a toggle function, the first line of which grabs the data-hidden attribute’s value and checks if it is true. Since we wrote this line in the HTML:

<span class="spoiler" data-hidden="true">

The value will be the string value "true" when the web page initially loads, meaning our hidden variable’s value will be the boolean value true.

On the next line, we set the attribute to the string representation of whatever the opposite of that value is by using !hidden. The ! operator negates whatever variable is next to it. So !true (read “not true”) becomes false and !false (read “not false”) becomes true.

Finally, outside of the toggle function, we use the addEventListener method to add the toggle function to the click event, meaning this function will run any time the spoiler span is clicked.

As of right now, clicking the spoiler tag will just change its data-hidden attribute between true and false. You can view the element in your browser and watch the data-hidden value in the HTML change every time you click it. To actually toggle the blur effect on and off, we need to update the CSS and tell elements inside of spoiler elements with data-hidden attributes of true to behave a certain way.

We do this by making a block of rules for all spark elements inside of a spoiler span whose data-hidden attribute is false. We’ll do this by adding the following to our CSS file:

.spoiler[data-hidden="false"] .spark {
  animation-play-state: paused;
  display: none;
}

The code above targets only those spark elements inside of a spoiler element with the data-hidden value of "false". Then, we set the animation-play-state to paused, this means the sparkle will stop where it is. We also set the display attribute to none, which essentially hides the sparkle.

Since clicking the spoiler tag will transition its data-hidden value to false if it’s already true, the new rules we wrote above will kick in on all the spark spans, essentially hiding them. It will also unblur the text inside of spoiler-content since we set the filter attribute to blur(7px) only in the rule for a data-hidden="true"element.

With this final update, you should now be able to toggle the blur on and off in the web page! Refresh your browser and click the blurry spoiler text to see the effect in action.

On Your Own

Now that you have the basic ideas down for implementing the spoiler effect, there are lots of things you can do to expand upon and improve the code. For starters, the code above is inefficient because it sets the toggle function on the spoiler span for every sparkle, how would you fix that? You can also experiment with setting individual spoiler tags, each needing their own click to unblur, or set all spoiler tags to unblur when any of them are clicked. There are also tons of things you can do to change the animation style and sparkle appearance. Play around with it!

Conclusion

In this article, you learned how to mimic the spoiler effect from Threads and used HTML, CSS, and a little JavaScript to make it happen. We used keyframes and learned a little bit about CSS animations. You now know the basic strategies behind conditionally hiding and showing text, you can use this effect on your own website for a variety of reasons.

Final Code

At the end of this article, I had three files called spoiler_text.html, style.css, and sparkle.js. Here are the contents of my files:

index.html:

<!doctype html>
<html>
<head>
    <title>Sparkly Spoiler Effect</title>
    <link rel="stylesheet" href="style.css" />
</head>
<body>
    <p>
        The biggest plot twist of all time is:
        <span class="spoiler" data-hidden="true">
            <span class="spoiler-content">
                Darth Vader is Luke Skywalker's father!
            </span>
            <span class="spoiler-blur" />
        </span>
    </p>
    <script src="sparkle.js"></script>
</body>
</html>

style.css:

:root {
  --bg: #0f1115;
  --text: #f4f7fb;
  --spoiler-dot: rgba(255,255,255,0.85);
}

body {
  min-height: 25vh;
  display: grid;
  place-items: center;
  background: var(--bg);
  color: var(--text);
}

.spoiler {
  cursor: pointer;
  position: relative;
}

.spoiler[data-hidden="true"] .spoiler-content {
  filter: blur(7px);
}

.spark {
  position: absolute;
  width: 4px;
  height: 4px;
  border-radius: 50%;
  background: var(--spoiler-dot);
  animation:
    drift var(--dur) linear infinite,
    twinkle var(--twinkle) ease-in-out infinite alternate;
}

.spoiler[data-hidden="false"] .spark {
  animation-play-state: paused;
  display: none;
}

@keyframes drift {
  from {
    transform: translate(var(--x0), var(--y0)) scale(var(--scale));
  }
  to {
    transform: translate(var(--x1), var(--y1)) scale(var(--scale));
  }
}

@keyframes twinkle {
  from { opacity: 0.25; }
  to   { opacity: 0.95; }
}

sparkle.js:

function random(rscale) {
    return Math.random() * rscale;
}

function buildSparkles(spoilerBlurSpan, sparkleCount) {
    for (let i = 0; i < sparkleCount; i++) {
        const s = document.createElement("span");
        s.className = "spark";
        const x0 = random(100);
        const y0 = random(100);
        const x1 = Math.max(0, Math.min(100, x0 + random(18)));
        const y1 = Math.max(0, Math.min(100, y0 + random(14)));

        s.style.left = `${x0}%`;
        s.style.top = `${y0}%`;
        s.style.setProperty("--x0", "0px");
        s.style.setProperty("--y0", "0px");
        s.style.setProperty("--x1", `${x1 - x0}px`);
        s.style.setProperty("--y1", `${y1 - y0}px`);
        s.style.setProperty("--dur", `${random(60)}s`);
        s.style.setProperty("--twinkle", `${random(1.2)}s`);
        s.style.setProperty("--scale", `${random(1.5)}`);

        spoilerBlurSpan.appendChild(s)
    }
}

function wireSpoiler(spoilerBlurSpan) {
    buildSparkles(spoilerBlurSpan, 75);
    let spoilerSpan = spoilerBlurSpan.parentElement;

    function toggle() {
        const hidden = spoilerSpan.getAttribute("data-hidden") === "true";
        spoilerSpan.setAttribute("data-hidden", String(!hidden))
    }
    spoilerSpan.addEventListener("click", toggle);
}

document.querySelectorAll(".spoiler-blur").forEach(wireSpoiler);

How to Start Contributing to Open Source Software

Erik — Thu, 17 Oct 2024 20:27:52 +0000

I am pretty open with my enthusiasm for open-source software (OSS) and, as a result, often receive questions from others on how they can get started contributing to projects. I’ve received enough questions to notice some trends in where people are starting from and what kind of problems they’re having. As such, I wanted to write this article to help anyone who is wanting to get started contributing to OSS projects, because getting started is the hardest part! To begin, let’s briefly talk about the advantages of contributing to OSS.

Contribute to the Community

One of the main reasons I like contributing to OSS projects is because I generally like being helpful. I enjoy fixing bugs that trip people up or adding features that help people get their work done more efficiently.

I also like contributing specifically to my field. As a PhD student and possibly future scientist, I have a vested interest in the quality of the software in my field–specifically, structural bioinformatics. I use several tools in this field and often find areas that can be improved, both for myself and others. As an example, consider this minor documentation change I added to the Biopython documentation.

This is admittedly a very small change, however I updated documentation that was incorrect because on two separate occasions, I landed on this page while trying to do something and got errors by following the provided example. In the future when I (or anyone else) arrive at this page, they will not have to go through the trouble of debugging the examples.

Develop or Hone new Skills

This is one of the primary direct benefits of contributing to OSS projects–the chance to work on significant projects with technologies you’re interested in working with. This can be fun for you and it can also help provide “proof” that you know how to work with certain technologies even though your work life might not provide you with exposure to them.

One personal example, doing this PR adding support for some operators to the autowrap project gave me the chance to learn a lot about Cython and work with a small bit of C++. Another example from the Biopython ecosystem is this PR updating the documentation-deployment process. This work was one of my first exposures to working with CI/CD systems, a topic I feel I excel in now. Being able to deliver this feature on a project that impacts many people was a very valuable and fun learning experience.

Develop a Name for Yourself

Another benefit to contributing to OSS projects is making a name for yourself in a certain area. For example, many of the frequent contributors to the CPython project become Python fellows or contributing members. This designation signals a kind of proven competence. Anyone who regularly contributes to CPython must be either an expert in Python, C, or both, right?

Being known as a helpful presence in some software community will garner goodwill for yourself, and that goodwill can go a long way.

Setting up your GitHub Profile

Now that we know why we want to contribute to open-source software, lets finally get to how. The very first thing you need to do, if you haven’t already, is set up your GitHub account. You can also additionally set up a GitLab account if you know the project you want to which you want to contribute is hosted there.

There’s really no beating the official documentation from GitHub on setting up your account, so I wont’ write out all the steps here. Follow the steps in that link so you can start contributing.

If you’re new to Git, feel free to check out my professional git series (part 1, part 2, part 3), which should teach you all you’ll need to know about working with Git and contributing to OSS.

Picking a Project

Perhaps the most important step is the first: picking a project to contribute to. This is where I see a lot of aspiring OSS contributors quit before they even start, but it’s not their fault! Often, the advice to people just finishing learning the basics is to find an OSS project in that language and “just start contributing!” Personally, I find this advice unhelpful because it can send inexperienced learners towards complex projects they’re likely not ready for.

As an example, when I first learned Python, someone suggested that I start contributing to Django. Django is a web development framework for Python and it’s incredibly popular, but it was way over my head at the time as I had just finished an introductory level book on Python development and had absolutely no experience. I didn’t even know how to use Django, let alone contribute to it. My inability to even figure out how to get Django running locally put me off OSS development for a long time. My point here is, don’t just browse GitHub and start trying to make sense of the most popular project written in your language of choice, find a project that really interests you.

I believe the trick to being a frequent contributor to a project is caring enough about it to stick with it. You will likely never make meaningful contributions to a project in just a few hours of work, you will likely spend days or weeks on fixing bugs and adding features; and that’s after you’ve spent a bunch of time figuring out how the software works as is. So, you need to find a project that you are interested in enough to work on for extended periods of time.

Caring about a project requires one of two things: investment in the problem being solved by the project, or an interest in the work itself. To illustrate the first point, consider Biopython. Most of the contributors to that project are bioinformatics researchers. For the second point, consider the fact that many contributors to open source programming languages (like CPython, Rust, and more) are language enthusiasts that contribute because they like the kinds of challenges language design presents.

Since finding a project to contribute to is so important, take your time doing it. There’s no reason you have to decide today what project you want to contribute to. Browse different topics in GitHub and see what sparks your interest the most.

Figuring out the Development Workflow

Once you’ve found a project to contribute to, the next step is figuring out how the project’s community builds their software. If you want to contribute to their work, you have to respect their process. Often, this will be explained in a file called CONTRIBUTING.md or something similar.

The kinds of things you want to look out for is how the community prefers to receive pull requests. For some, maintainers want all pull requests to be associated with a corresponding GitHub issue. For example, this pull request to the OBOFoundry home page removes a page from their website, but I only made this PR because one of the maintainers created issue 2590 in the “Issues” tab asking for that page to be removed.

The most common workflow I’ve encountered involves forking the main repository into your own profile, creating a branch based on some issue, making your changes, writing tests for the changes, and making a pull request to the main repo. Development workflows frequently include a CI pipeline with tests and checks your changes have to pass.

More often than not, the final step in the development workflow is a review process. This typically includes a project maintainer looking over your changes and making sure you’ve solved the problem and that your code code is in line with project’s standards. This is usually a back-and-forth process, especially for your first contribution, in which a reviewer may ask you for changes multiple times. Don’t get discouraged or take these reviews personally! This is someone else’s project after all, you are a guest (though a helpful one!).

Project’s sometimes have guidelines for tagging reviewers (mentioning their GitHub username so they’re notified that you’re awaiting their review) so your work can move forward. Don’t get impatient and tag reviewers constantly, but don’t be too shy to bump your PR if the guidelines allow for it.

The main takeaway here is to set yourself up to be a productive member of the project community by adhering to their process and not just diving in headfirst writing code and making changes that no one wants. Remember, software engineering is a collaborative process, even (or especially!) in open source development.

Conclusion

Contributing to OSS can be beneficial to you, project maintainers, and the general public alike. It can also be a lot of fun! Remember to carefully choose the projects you want to contribute to and respect the development process. Feel free to reach out if you have any questions.

An Introduction to Software Architecture

Erik — Tue, 26 Sep 2023 06:08:40 +0000

Software architecture is the concept of how a software project is structured; a holistic view of the entire system including class hierarchies, interface design, even deployment patterns. A system’s architecture impacts nearly every facet of interaction with that system, from end users to the developers that build and maintain it. When you understand your system’s architecture, you’ll develop an intuition for estimating the time it will take to build new features, where bugs may be hiding in the code, and how to influence the performance of your system. In order to give you the tools to study the architecture of the projects you’ll one day be contributing to, this article lays a foundation for the most important concepts related to the topic, namely architectural patterns, reusability, quality attributes, and tradeoffs.

It is difficult to find a helpful and exact definition for software architecture that accurately communicates its importance and prevalence. One simple definition of software architecture might be “the structure of a software system.” But that definition lacks the gravitas the concept deserves.

More colloquially, you will often hear people define software architecture as “the stuff that’s expensive to change later.” Such a definition, while true, doesn’t really tell us much about what it is. For example, the programming language in which a system is written is expensive to change, but so is the application’s project management software (for example, moving from internal documents to a professional project management system like Jira is very expensive). These are both expensive things to change, but only the programming language is a facet of software architecture.

In truth, software architecture means different things to different people and there is no universal definition that all developers will agree on. To some, software architecture is similar to class design in that it models the relationships between classes throughout the entire system.

Others extend the concept to include not only class relationships, but also the infrastructure upon which the system runs such as the type of database a system might use or the type of web server.

An official definition for software architecture exists in a document from the International Organization for Standardization (ISO) in conjunction with the International Electrotechnical Commission (IEC) and the Institute of Electrical and Electronics Engineers (IEEE) called ISO/IEC/IEEE 42010:2022 – Systems and software engineering – Architecture Description. This standardization document describes software architecture as:

[The] fundamental concepts or properties of an entity in its environment and governing principles for the realization and evolution of this entity and its related life cycle processes

ISO/IEC/IEEE 42010:2022 – Systems and software engineering – Architecture Description

“Environment” in that definition is later defined in the same document as:

[The] context of surrounding things, conditions, or influences upon an entity

ISO/IEC/IEEE 42010:2022 – Systems and software engineering – Architecture Description

Personally, I like this definition and I think it serves the purpose of defining architecture well enough for newcomers to the topic. For the purposes of the rest of this article, however, we will only learn about two facets of software architecture: architectural patterns and components, and reusability. These are the most significant and impactful parts of software architecture as well as the easiest to explain. Let’s go over what each one means.

Architectural Patterns and Components

Architectural components are layers of functionality within a system that are responsible for a specific system-wide task like accessing a datastore or routing application requests. Architectural patterns refer to the ways in which these different components work together to achieve the system’s purpose.

The idea of architectural patterns is somewhat similar to that of design patterns; like design patterns, architectural patterns are made up of components accomplishing specific tasks that are pieced together to accomplish broader tasks in the most effective way possible. Another way in which architectural patterns are like design patterns is that there are many well-documented patterns to choose from, each with their own strengths and weaknesses. However, unlike design patterns, architectural patterns have system-wide impacts.

As an example, consider the factory method design pattern which creates instances of objects based on a given context. This pattern’s components work together to accomplish a very useful task, but outside of that task, it has little—if any—impact throughout the system. While architectural patterns are also made up of multiple components, each with a specific responsibility, the scope of that responsibility is much larger and less specific than those of a design pattern’s responsibility.

It’s useful to think of the components of an architectural pattern as self-contained collections of code, interfaces, and possibly infrastructure. This is a level of abstraction higher than the idea of components in a design pattern because design pattern components only consist of code. As an example of an architectural component, consider the model-view-controller (MVC) pattern. In this architectural pattern, the model component (more often referred to as the model layer) consists of not only class definitions, but also the underlying datastore—like a database—of the application. It’s important to understand this idea of components at the architectural level because components are what make up architectural patterns and influence their strengths and weaknesses.

Reusability

On the topic of software architecture, reusability refers to a component’s suitability for handling a specific system-wide task. To use the MVC architecture example again, the architectural component called the controller layer is responsible for handling application routing tasks. Most of the time, MVC is used in web applications, so the controller layer is responsible for handling HTTP/HTTPS requests from browsers, sending them to the appropriate code hosted on a web server, and sending the response back to the browser. Once the controller layer is built, we don’t have to write it again. This means that no matter how many features we have, the controller layer is reusable for handling all the application routing needs of that feature. This saves the development team weeks of time by sparing them the task of writing HTTP routing code every time they want to introduce a new feature into a system.

By now we should be starting to understand what the concept of software architecture is getting at. Now, let’s talk about why software architecture is worth knowing about.

Why Architecture Matters

Software architecture is the “big picture” of a software system and therefore permeates every aspect of the code. Consider an obvious metaphor: the architecture of a building. The building’s architecture dictates everything about the building that makes it useful; how many people it can hold, where the emergency exits are, how to move from one area to another, the functions of different parts of the building, how future contractors might make upgrades to the building, even how the building affects the environment around it.

Additionally, a building’s architect has to make decisions about what attributes of the building are most important, which are nice to have, and which are not important at all. For example, an architect may be told to make a building with the primary goal of keeping operational costs low, even if the building isn’t very aesthetically pleasing. In this case, the architect will decide that, even though floor-to-ceiling windows on every floor would make for a very nice-looking building, it would be terrible for operational costs because the sun will heat up the building during the day and the people inside will have to turn up the air conditioner. In this case, the architect will have to opt for smaller windows throughout the building and perhaps even exterior walls of concrete to absorb the heat.

Likewise, software architecture dictates everything about the system that makes it useful, and software architects have to make decisions that sacrifice one thing for another. In software architecture, these “things” that are sacrificed are called quality attributes, and the decisions about which quality attributes to prioritize are called tradeoffs. Let’s talk about both of these concepts, starting with quality attributes.

Quality Attributes

Quality attributes are the non-functional attributes of a software system. In this case, an attribute is non-functional in that it doesn’t do anything. For example, suppose we’re building a calculator app. Being able to do addition is an example of a functional attribute. On the other hand, how quickly the calculator can produce the results of adding two numbers together is a non-functional attribute. Saying the calculator is fast or slow is a comment on one of the calculator’s quality attributes. Aside from the power and modernity of the physical infrastructure upon which a project runs, nothing impacts a system’s quality attributes more than its architecture. Let’s define some of the most important quality attributes, what they are, and how architectural decisions affect them.

Reliability

Reliability is a quality attribute that defines a system’s ability to perform its tasks under some given conditions for some given amount of time. For example, suppose we’re building a web server upon which someone to host their website. Say one of our users has their homepage, index.html, on our server. We expect that any time someone in the world sends and HTTP request to our server requesting that index.html file, our server will properly send the page back to that person. Occasionally, due to any number of reasons, our server will fail to send the requested index.html page and will instead send the user a 404 – Not Found response code. This is considered a failure.

The probability of our server correctly returning the requested index.html page at any given time under some given condition is a measure of reliability. For example, suppose we send one thousand requests for the index.html page over the span of fifteen minutes and our server correctly sends back the index.html page 991 times, the reliability of our server with at one thousand requests over fifteen minutes is thus 99.1%.

One way in which software architecture affects reliability in this case is how we decide to handle request failures on the server. When the server receives a request for index.html and for some reason cannot find it, instead of sending back a failure 404 message to the user, we could tell the server to try again up to five times. A very simplified version of how this might look in the code is as follows:

def handle_request(file_name, retries=0):
  try:
    resource = open(file_name)
    # File was found, send it to the requester
    send(resource)
  except:
    # File not found, try again
    retries += 1
    if retries < 5:
      print(f"Resource not found, retry number {retries}")
      handle_request(file_name, retries)
    else:
      # Couldn't find file after 5 attempts,
      # send 404 error message
      send(404)

In the code above, we wrote a function that handles a request for a resource like index.html and sends it back to the user (note, the send method is not defined in this code, this is just for example purposes). We put the open method inside of a try block and use the except block to handle any failure to find the resource. This will help alleviate any random failures that might happen by simply telling the server to try again. However, we also prevent any infinite loops from happening by telling the server to quit trying to find the requested resource after five tries. If the max number of retries is reached, the server sends the 404 response code. Now if we were to test the server again with a thousand requests for index.html, we may see improvements; perhaps the server eventually sends the requested resource 999 times, increasing our reliability measure to 99.9%.

Availability

Availability is closely related to reliability in that it measures the probability that a system will be available to perform its task at any given time. This measure is directly impacted by the system’s reliability, but also takes things like maintenance downtime into account. Availability over some period of time is measured as that period of time, minus the amount of downtime and then divided by the period of time being measured. For example, if we measure daily availability and find that the system is unavailable for about five minutes every day, the availability of the system is 1440 (the number of minutes in a day) minus 5 (the number of minutes the system is unavailable), all divided by 1440, or 99.65%.

You may occasionally hear people refer to their system availability in terms of nines. The system in our previous example has “two nines” of availability because the number 99.65 has two nines in it. These numbers are often used in service level agreements (SLAs), contracts that system developers have with their customers agreeing on how much availability they can expect from the system. Five nines (99.999%) of availability is considered very high availability as it allows for five minutes and 15 seconds of downtime a year.

Software architecture impacts the system’s availability by defining how system maintenance is conducted. For example, if an application’s database goes offline for some reason, can the application still be accessed while we figure out how to get the database back up and running, or does the whole system have to come offline? If we wrote our components to be reliant on the database, the whole application will likely be unavailable while the database is down. If it takes us five minutes and 15 seconds to get the application back online, we can’t have any more downtime for the rest of the year if we want maintain five nines availability.

Scalability

Scalability refers to how a system’s performance and cost increases and decreases with demand. For example, imagine our system has a baseline number of 500 users at any given time, but for some reason, one day we have two-thousand users for a couple of hours. In order for the system to perform identically for all users, we’ve programmed it to spin up a duplicate server every additional 500 users to handle the extra demand. Also, we’ve programmed the system to spin down those servers as the spike in demand tapers off. This means that both cost and performance increase and decrease based on the demands placed on the system.

Scalable systems are created by writing modular code. As a senior engineer or the architect on your team, you enforce code modularity by ensuring that the components that rely on infrastructure can handle that infrastructure being duplicated. For example, hardcoding the IP address of our servers in the code would not be conducive to scalability since new servers would have different IP addresses.

Security

Security is a quality attribute that refers to a system’s ability to protect data from unauthorized access, prevent users from accessing a higher level of privilege in the system than they need, and much more. Security is a broad topic and there’s a ton of information to know about it, so this section will cover only a very small subset of how it applies to software architecture.

When you’re architecting a system, you have to think about how data will be stored and if it’s allowed to travel networks unencrypted. For example, when transmitting data from a web server to a user’s browser, the data is accessible to people (network administrators, people using packet sniffers, and so on). In fact, it’s good practice to assume that any data traveling over a network is being read by nefarious people. We might not care who sees some data if it’s something innocuous like a person’s shoe size, but often we transmit data like names or credit card numbers over a network and we definitely don’t want people reading that, so we encrypt it.

**_Encryption_**, like security, is also a broad topic but I wanted to briefly define what it is in case you’ve not heard of it before. Basically, encryption is the process of scrambling data so that it’s not human-readable, and the only way to unscramble it is with a special decoder called an encryption key. Generally speaking, only the machine receiving encrypted data has the appropriate encryption key, so even if that data is captured by nefarious people, it is no good to them because they can’t read what it says.

The system’s data encryption policy is an important architectural decision because data is much bigger when it’s encrypted than when not. As a consequence, encrypted data takes slightly longer to travel from point a to point b in a network. It’s not a huge difference, but as your system grows, the amount of data flowing through the system will impact the system’s performance. Figuring out what data needs to be encrypted is just one small facet of security minded software architecture.

Maintainability

The last quality attribute we’ll discuss is maintainability. A system’s maintainability is a qualitative measure of how easily developers can add features, fix bugs, or tweak performance. Maintainability isn’t something we can measure with a formula; it is more of a general feeling about how easy the system is to work with from a developer’s perspective. Many of the topics you learn as you become a more senior developer are techniques for writing maintainable code. For example, making smart decisions about class design makes future work easier to do because intuitive class design lowers cognitive complexity. Likewise, using design patterns where appropriate along with naming their components descriptively helps developers navigate the system easily, reducing the time needed to add a new feature or fix an overlooked bug.

Software architecture impacts the maintainability of the system because every architectural decision influences how code must be written. As an example, a microservices architecture pattern is useful for separating business concerns into individually deployable components. This means that, as a developer, if I get tasked with writing a feature for the marketing team, I know exactly which component I need to work on. However, if I’m working in a system built on the MVC architectural pattern, I may have to work with several components to deliver the required feature.

Architectural Tradeoffs

Now that know what quality attributes are and have seen some examples, let’s talk about architectural tradeoffs. As a senior developer or software architect, you’ll often be faced with building features that have competing priorities. Consider the example we used when talking about reliability. Adding the try … except blocks to the server’s code allowed us to increase the server’s reliability, but it most likely will make the server a little slower. This is because exception handling is very slow when exceptions are raised, meaning that every now and then, a user will have a slightly slower experience than if we got rid of the server’s exception handling altogether.

Deciding which is more important, the speed of the request response or the reliability of the server, is an architectural tradeoff you will have to make. This decision will be influenced by many factors such as the kind of application you’re building, the needs of the users you’re building it for, and much more. There are no universally correct answers to architectural tradeoff questions, they always depend on the context in which you’re working.

One of the tradeoffs you’ll have to think about quite often is security. As we discussed in the quality attribute section, something like encryption can have a significant impact on your system’s performance. As an architect, you’ll have to decide what needs to be encrypted and what doesn’t. If you’re working in the defense industry, you’ll likely have to encrypt every bit of data that travels over a network, causing your system to perform slowly. However, if you’re writing web-based browser games that don’t require a login, you likely don’t need to encrypt anything. The kinds of tradeoffs you make will depend on the industry in which you’re working and the type of system you’re building.

One thing that will impact your architectural decisions and tradeoffs is the due date of your projects. For example, if you have a year to build a back-office application for a small company, you will likely have maintainability as a high priority so that you can troubleshoot problems easily and add new features to the system upon request. Conversely, if you’ve been tasked with building the same system in a month, you will not take the time to consider maintainability. This is because building maintainable systems requires extra care and planning, time you don’t have with such a quick due date.

Yet another decision about architecture you’ll have to make is whether to build a component yourself or use a third-party component. Third party components, sometimes called commercial off the shelf (COTS) software, are components built by other organizations that you usually have to pay for. COTS software is supposed to require very little work for a developer, and sometimes includes 24-hour support (that is, you can call someone for help if the COTS software isn’t doing what you want it to). One example of a homegrown vs. COTS solution is the decision on how to monitor your system’s performance. Quite often, it’s necessary to watch your system’s performance such as CPU or RAM usage, network latency, and much more. Some teams find it best to write their own software for monitoring system performance while others buy performance monitoring software from other vendors. The decision on which way to go will depend on a lot of things such as staffing (you need more developers to build and maintain such software) and cost (some performance monitoring applications are very expensive), and whether or not the COTS software can easily service your needs.

Finally, perhaps the most important consideration when making architectural tradeoffs is cost. You and/or your company have a finite amount of money set aside to build whatever system you’re working on. How you use that money to accomplish the goals of the system often includes making several architectural tradeoffs. For example, do you want to pay for a database management system like Microsoft SQL Server so that you can have 24-hour support ready to help you if something goes wrong, or do you prefer using an open-source option like MySQL or Postgres for free, but which do not have a helpdesk for you to call? The answer to that question will depend on many factors. For example, are you a lone developer for the company you’re building this system for, and the company needs the application to be running all the time? If so, you might want to opt for the SQL Server option since you may need help one day. Conversely, are you building an open-source project for people to share recipes with each other? If so, your budget is probably small, and your users won’t lose millions of dollars if your database stops working. In such a case, self-hosting a free database system like MySQL should be fine.

The responsibility for making the decisions on these tradeoffs will fall on you more often as you become more experienced in whatever tech community you’re a part of, be it an open-source organization or a company. Understanding the overall goals your system and how your organization plans to use and support it will be an important step towards making good architectural tradeoffs.

Conclusion

This article introduced an important concept in your path to becoming an experienced and professional-grade programmer: software architecture. Software architecture can be defined in a number of ways, but it’s ultimately the process of developing components and fitting them together in the most efficient way relevant to our system’s goals. We learned about those components and how they form architectural patterns. Then, we learned about quality attributes and how they’re impacted by software architecture. Learning about quality attributes is a prerequisite to learning about architectural tradeoffs, the decisions you must make when developing systems. Hopefully now you have an understanding for what software architecture is and why it’s important.

How to Debug Code (with Python Examples)

Erik — Mon, 25 Sep 2023 04:01:06 +0000

Often in your programming career, you will inadvertently write flawed code that introduces some fault into your codebase. These faults are called bugs and the activity of fixing bugs is called “debugging.” Of course, as developers, we try to write correct code every time, but writing bugs is simply a fact of life for programmers. The ability to diagnose and fix bugs is the mark of an experienced and talented developer and is foundational for advanced troubleshooting. In this article, we’ll go over some techniques for effective debugging and introduce you to PDB, the Python debugger. At the end of this article, you’ll be ready to not only troubleshoot and fix your own code, but to diagnose and debug existing code as well.

Fundamentals of Debugging

As a developer, you will spend a lot of time debugging. In fact, several sources estimate that as little as 35% or as much as 75% of a developer’s time is spent debugging code that’s already been written rather than writing new code (here’s a pretty good source that puts that number between 35% and 50%). Since so much of our time as developers will be spent debugging, we should put effort into building this skill. As with all skills, you will get better at debugging the more you do it. There are, however, some fundamental techniques to troubleshooting and debugging software that you should know and build upon. This section will introduce those fundamentals, starting with confirming the bug.

Confirm the Bug

An important part of debugging code is understanding the broader context and knowing what the code is intended to do. This is important because your first step in fixing a bug is confirming that it truly is a bug. For example, consider a bug report that says, “the area_of_cube function isn’t returning the proper values unless you pass it 6.” In order to start debugging, we first decide to try the function out. We load up a console and try a few values:

>>> from formulas import area_of_cube
>>> area_of_cube(4)
96
>>> area_of_cube(5)
150
>>> area_of_cube(6)
216

It looks right, so we go and double check the formula for finding the surface area of a cube and find that it is the length of an edge squared and then multiplied by six. That means that the numbers we got back from our console test are correct. However, we remember that the bug report says the function only works when you pass it six. Now we wonder if the user reporting the bug is confusing area with volume, since the formula for the volume of a cube is the edge length raised to the third power (or cubed). This means that a cube with edges of length 6 will have both a surface area and volume of 216, and we think the user reporting this bug meant to use the volume function and erroneously reported the area function as having a bug.

At this point, we have to track down the user reporting the bug and see if they really meant to use the area function when they were reporting the bug or if they meant to use the volume function. This is what confirming the bug is about, making sure a reported bug truly is a bug. This is an important step in debugging because it could potentially save you from fixing code that isn’t actually broken. This example may seem silly, but you will be surprised how often reported bugs in the software are actually due to misuse.

Reproduce the Bug

Once you’ve confirmed a bug report is genuine, the next step in fixing a bug is reproducing it. Many times, bug reports will say something isn’t working but when you go to check it out yourself, it seems like it’s working fine. In order to properly diagnose the problem, you have to first see the problem happening. As an example, imagine you get a bug report from a coworker that says something like “The greet_user function isn’t working. The function doesn’t print anything to the console when called.” You start your console to test it out and see no problem:

>>> from greeters import greet_user
>>> greet_user('Erik')
Good evening, Erik

In the above console session, you imported the reportedly buggy greet_user method and called it by passing Erik as an input parameter. The function then printed “Good evening, Erik” as expected. You haven’t yet reproduced the bug, so now you have to go look at the code and see if you can figure out what’s going on. You open up the greeters.py file and look for the greet_user method and see this:

import datetime

def greet_user(name):
  now = datetime.datetime.now()
  hour = now.hour
  if hour < 12:
    print(f'Good morning, {name}')
  elif hour > 12 and hour < 18:
    print(f'Good afternoon, {name}')
  elif hour > 18:
    print(f'Good evening, {name}')

As you look over this code, you see that the method extracts the hour of the day and prints an appropriate greeting: “good morning” if the hour is before 12, “good evening” if it’s after 12 but before 18, and “good evening” if it’s after 18. The bug report said the function isn’t printing anything, how could that be possible?

Upon further inspection, you notice that the method’s if statement only provides instructions for what to do if the hour variable is 0 to 11, 13 to 17, or 19 to 23. If hour is 12 or 18, the code does nothing! This means there’s 2 hours of the day in which this code will not print anything to the console. The person who reported the bug must have been trying to use it during one of those times. In order to test your theory, you decide to manually set the hour variable to 12 to see if that reproduces the bug:

def greet_user(name):
  now = datetime.datetime.now()
  # hour = now.hour
  hour = 12
  if hour < 12:
    print(f'Good morning, {name}')
    . . .

Now, you rerun the same commands from your console session earlier:

>>> from greeters import greet_user
>>> greet_user('Erik')

But this time, you don’t see any output. You’ve successfully reproduced the bug and you now know why the bug is happening. In order to fix this bug, we simply have to update the conditionals to use greater-than-or-equal-to comparisons:

import datetime

def greet_user(name):
  now = datetime.datetime.now()
  # hour = now.hour
  hour = 12
  if hour < 12:
    print(f'Good morning, {name}')
  elif hour >= 12 and hour < 18:
    print(f'Good afternoon, {name}')
  elif hour >= 18:
    print(f'Good evening, {name}')

Now, with your now = 12 line still active, you can see that the code now works. This is why reproducing the problem is important, some of the trickiest bugs only show up under specific conditions.

Challenge Assumptions

Another technique for debugging especially tricky bugs is challenging your assumptions. Sometimes, a bug makes no sense and no matter how many things we test, we cannot seem to figure out why a bug is happening. When this happens, the next step is to ask ourselves if the things we generally take for granted still apply. This means making sure environment variables are what you think they are, verifying any authentication functions are still working, making sure you’re using versions of software that support what you’re trying to do, and so on.

As an example, imagine you receive a bug report saying there’s something wrong with the length_check method; it’s throwing an error any time someone tries to use it. You find the method in your codebase to see it’s defined as the following:

def length_checker(max, values):
  if (count := len(values)) > max:
    print(f'Cannot have more than {max} values, you have {count}')

This method takes two inputs, max and values, and prints an error if the number of items in the values parameter is above the max. Everything seems to look fine to you, so you start a console session to test the method out:

>>> from checkers import length_checker
>>> length_checker(1, [2,2])
Cannot have more than 1 values, you have 2
>>> length_checker(2, [1,2])
>>>

In the above example, you call length_checker and pass it a max value of one and a list with two items. Doing so causes the function to print an error to the console as expected. In the next line, we call length_checker with a max value of two and again pass it a two-item list. This time, nothing is printed to the console, as we expected. This function seems to be working perfectly, why did someone report a bug?

It’s time to challenge our assumptions. The error report came from someone else on a different computer, perhaps there’s some difference between their computer and our own that is causing this function to fail. You decide to start by checking the Python version of your computer and the bug reporter’s. You run python --version in your command terminal and see that you’re currently using Python 3.10. The bug reporter runs the same command and lets you know they’re using Python 3.7.

Could the error be because the bug reporter is on an older version of Python than you? You take a closer look at the function and realize that it uses the walrus operator (the := syntax). You vaguely remember reading about this being added to Python a few years ago, so you look up when it was added and find out that the walrus operator wasn’t part of Python until Python 3.8! No wonder the function isn’t working for the bug reporter, that version of Python doesn’t even know what to do with the := syntax. To fix this bug, you simply ask the reporter to update their Python version to 3.8 or higher, and the problem is solved.

Challenging your assumptions is one of the easiest steps to forget when it comes to advanced troubleshooting and debugging. Whenever you find yourself working on a bug that doesn’t make sense, ask yourself if you’ve tested literally every facet of the program—from the operating system to the software versions, and every line of code up to and including the error. Doing so will often show you that one of your assumptions is wrong and that’s why you’re encountering the bug.

Now that we’ve talked about the fundamentals of debugging, it’s time to talk about a built-in Python tool that makes troubleshooting easier: the debugger.

PDB: The Python Debugger

The Python debugger (called PDB) is a tool built into the Python programming language that allows us to stop a script at run time so we can check the value of variables at that given moment in the script’s execution. In this section we’ll learn how to set breakpoints, check the value of variables, and step into functions and methods. We’ll start with setting breakpoints.

Setting Breakpoints and Checking Values

When using a debugger, a breakpoint refers to a line in the code upon which we want to pause execution of the program. Before we see how to set a breakpoint with PDB, let’s write a script in which to use the debugger:

def add(x, y):
  sum = x + y
  return sum

a = 1
b = 2
c = add(a, b)
print(f"c is {c}")

This code is very simple. First, we define a method called add which takes two parameters and returns their sum. Then, we create two variables, a and b, and a third c which is the return value of add when passed a and b variables. Finally, we print the value of c to the console.

Now, lets set our first breakpoint to see how using the debugger works. Just above the line a = 1, write breakpoint(), and then run the script. You should see the following output in the console:

> path/to/script.py(6)<module>()
-> a = 1
(Pdb)

There are three things to note in the output above. First, the line preceded with the > symbol is telling us what file we’re in (your output may vary depending on what you named this file and where you saved it). The next line, -> a = 1, shows us where in the Python script we’ve stopped. Notice that this is the line just below where we set our breakpoint. The line a = 1 hasn’t run yet, we’ve paused the script just before its execution. Finally, you see (Pdb) and a prompt. We can type debugger commands in this prompt.

NOTE: You don’t also write import pdb; pdb.set_trace() instead of breakpoint(), but this is considered the “old way”. In fact, if you’re using Python 3.7 or below (which you shouldn’t because they’re all deprecated), you’ll have to use this. When I first wrote this article, I was using the set_trace way exlusively, so if you see one in the article, it’s because I missed it and you can easily just drop in breakpoint().

The first command we’ll run is next, type next into the console and press enter. You should see the following output:

> path/to/script.py(7)<module>()
-> b = 2
(Pdb)

Notice that we’re on the next line of the script. The next command executes the line in which we were paused and then pauses on the next line. So, when we typed next just now, Python ran the line a = 1 and stopped on the line you see in the console now, b = 2.

One of the main benefits of using the debugger is seeing the value of variables at a given time. Since we’re paused just after the a = 1 line, we can type the variable name a into the debugger console and see what the value is:

(Pdb) print(a)
1

This is useful for when we want to confirm that a variable holds the value we think it does. If we want, we can also change the value of a here as well, check it out:

(Pdb) a = 5
(Pdb) print(a)
5

Manually setting variable values at runtime like this is helpful for experimenting when debugging; sometimes you want to see how different values will cause a script to behave.

NOTE: If you have a variable that happens to share a name with a debugger command, you can prefix the variable name with an exclamation mark (!) to have the word interpreted as the script variable instead of the debugger command.

Now, let’s stop pausing the script and allow it to finish running. We do this with the continue command which will let the program run until it reaches the next breakpoint or until it’s done executing. Run continue and you should see the following output:

(Pdb) continue
c is 7

Notice two things here. First, the output says c is 7. This is because when we paused the script, we set the value of a to 5, so when we passed a and b to the add function, we actually passed 5 and 2 instead of 1 and 2 like the script appears to say. Also notice that your console is probably back to your normal command prompt instead of the debugger. This is because the script has finished executing.

Now that we know how to set breakpoints, move around in a script, check the value of a variable at a given time, and change the value of a variable, lets look at stepping into functions.

Stepping Into Functions

When we set a breakpoint in a script, the debugger does not leave the scope of where it’s pausing the script. As an example, move your breakpoint from the previous section to just over the c = add(a, b) line. Now, run the script, you should see the following output:

> path/to/script.py(8)<module>()
-> c = add(a, b)
(Pdb)

Notice that the debugger is paused just before the add function is called. Type next and hit enter in the debugger console, you should see the following output:

> path/to/script.py(9)<module>()
-> print(f"c is {c}")
(Pdb)

Notice that the debugger paused at the next line after the function call instead of going into the add function itself like we might expect. This is because the default behavior of the debugger is to step over function calls so that you, as the developer, can stay inside of the context in which you’re debugging. If you want to step into a function, you have to use the step command. Run the script again, once you get here:

> path/to/script.py(8)<module>()
-> c = add(a, b)
(Pdb)

Type step and then press enter. Now, you’ll be in the add function and you should see the following in your console:

--Call--
> path/to/script.py(1)add()
-> def add(x, y):
(Pdb)

Notice a couple of new things about the debugger’s output now. First, at the top of the output is a line that says --Call--, this lets us know that we’ve left the context in which we were originally debugging because a function or method call was made. Also notice that the line containing the file location no longer says <module> but says add(). Again, this lets you know that you are currently debugging a function inside of the script.

Now you know how to use the debugger to move around a script, check and change values of variables, and step into functions. These operations exist in virtually every programming language debugger out there, so the commands you learned in this chapter will likely be the same or similar for any other debugger you might use in the future.

Conclusion

In this article you learned the foundational skills of debugging software. We started by learning about three important steps to debugging: confirming the bug, reproducing it, and challenging your assumptions. Then, we learned about PDB, Python’s built-in debugging tool. We learned how to pause a script at run time so we can check the values of variables, then we learned how to control a script’s execution with the next and continue commands. Finally, we learned how to step into functions while debugging with the step command. All of these skills will serve you well in your future work as a programmer and will go a long way in advanced troubleshooting.

If you liked this article about debugging, you may like my primer on TDD with Python.

Python Exception Handling and Customization

Erik — Sun, 17 Sep 2023 23:47:05 +0000

Like bugs, exceptions are inevitable when developing software, especially as the complexity of that software increases. Sometimes exceptions are surprising, other times we can anticipate them coming. How a program responds to the occurrence of exceptions is called exception handling, and as programmers, we can define and customize exception handling. In this chapter, we’ll learn what exceptions are, how to handle them, and how to make our own.

What are Exceptions?

Exceptions can be thought of as unplanned events in the execution of a program that disrupt that execution. When a runtime error occurs, a specific exception is raised. If an exception is raised and the program does not have any code defining how to handle that exception, the exception is said to be uncaught, and it will terminate execution of the program. Exceptions are not unique to Python; every programming language has exceptions and a means of handling them. You’ve actually most likely seen lots of exceptions already, but just for clarity, let’s raise a few exceptions on purpose.

First, in a Python console, try to divide any number by zero and press enter. You should see something similar to the following:

>>> print(3 / 0)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ZeroDivisionError: division by zero

When we tried to divide by zero, Python raised an exception. In this case, the exception raised was ZeroDivisionError. ZeroDivisionError is the name of the exception raised when a program tries to divide by zero. There are other kinds of exceptions as well, let’s raise another. In the Python console, try to add a number to a string:

>>> print('hi' + 2)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: can only concatenate str (not "int") to str

This time, Python raised a TypeError. TypeError is the kind of exception raised when a program tries to do something to an object that the object’s class doesn’t support. Notice also that there’s a message along with the exception: can only concatenate str (not “in”) to str. Exceptions can, and often do, have different messages for different ways in which they can be raised. For example, let’s raise another TypeError, this time by trying to divide a number by a string:

>>> print(20 / 'hi')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unsupported operand type(s) for /: 'int' and 'str'

Again, Python raises a TypeError in this situation, but notice that the message is different. The message says unsupported operand type(s) for /: ‘int’ and ‘str’. This will be relevant later when we write our own exceptions.

Now that we have an intuition about what exceptions are, lets learn how to handle them.

Handling Exceptions

An exception will terminate the execution of a program if left unhandled. For example, consider the following script:

def divide(x, y):
  return x / y

print(divide(4, 2))
print(divide(2, 0))
print(divide(9, 3))

In this script, we have a method called divide that takes two parameters, divides them, and returns the result. Next, we have three lines calling the divide function and printing the result. Notice that the second line passes zero as a second parameter; let’s see what happens when we try to run this script:

2.0
Traceback (most recent call last):
  File "path/to/script.py", line 5, in <module>
    print(divide(2, 0))
  File "path/to/script.py", line 2, in divide
    return x / y
ZeroDivisionError: division by zero

Notice that the program printed 2.0 to the console, meaning the first call to the divide function worked. Then, the program raised an exception, ZeroDivisionError, on the second line because we tried to divide by zero. Notice also that the program did not run the final line, print(divide(9, 3)). This is because the ZeroDivisionError was an uncaught exception. In order to make the program continue running, we have to write code specifically for handling that exception. We do this in Python with try except blocks.

Using try and except

In Python, we use the keywords try and except to handle exceptions. The try block contains the operation that may raise an exception, and the except block contains the code that defines what we want the program to do when an exception is raised. As an example, let’s rewrite the divide function from earlier to handle ZeroDivisionError exceptions:

def divide(x, y):
  try:
    return x / y
  except ZeroDivisionError:
     return "Cannot divide by zero"

Notice that the line return x / y is now in a try block. This tells Python to run the code in this block until the last line, or until an exception is raised. If an exception is raised, Python checks for an except block that matches the type of exception. In our case, we wrote except ZeroDivisionError, so if a ZeroDivisionError is raised, Python will run the code inside of the except block. To see this is action, run the script with the updated divide function. You should see the following output in the console:

2.0
Cannot divide by zero
3.0

This time, when the script got to the line that said print(divide(2, 0)), the program did not print the exception message to the console, it instead ran the code in the except block, which we can see in the console when it printed Cannot divide by zero. Notice also that even though an exception was raised, the program continued execution, as we can see by the 3.0 printed to the console when Python ran the last line of our script. This is an example of how exception handling is useful, we can anticipate problems and handle them without terminating the program.

Note: When a program encounters an error or otherwise fails in some way but does not terminate its execution, it is said to fail gracefully.

Handling Other Types of Exceptions

The divide function can now fail gracefully when a ZeroDivisionError exception is raised, but what about other exceptions? What if someone tries to pass a string to the function? As an example, update the last lines in the previous script to look like the following:

print(divide(4, 2))
print(divide(2, 0))
print(divide('hi', 2))
print(divide(9, 3))

If you try to run the script now, you’ll see the following output:

2.0
Cannot divide by zero
Traceback (most recent call last):
  File "path/to/script.py", line 14, in <module>
    print(divide('hi', 2))
  File "path/to/script.py", line 7, in divide
    return x / y
TypeError: unsupported operand type(s) for /: 'str' and 'int'

Notice this time that Python raised a TypeError when we tried to divide a string with an integer. We never wrote any code to handle this exception, so Python terminated the program once this exception was raised. If we want to add code to handle this exception, we have two options.

First, we could add another except block to the divide function for that specific exception. The function would then look like the following:

def divide(x, y):
  try:
    return x / y
  except ZeroDivisionError:
     return "Cannot divide by zero"
  except TypeError:
    return "Can only divide number types"

This would handle both ZeroDivisionError and TypeError exceptions. But there are yet more exceptions, we cannot anticipate them all. Instead of writing an except block for each exception type that could possibly beraised, we can write an except block that catches all exceptions besides the ZeroDivisionError one. This approach would have the function looking like this:

def divide(x, y):
  try:
    return x / y
  except ZeroDivisionError:
     return "Cannot divide by zero"
  except:
    return "An error occurred"

With the divide function written this way, there is no exception that can be raised which would terminate execution of our program. We have an except block dedicated specifically to catching ZeroDivisionError exceptions and informing the user they cannot divide by zero, and we have an except block that catches any other kind of exception that could possibly be raised and simply alerts the user that an error occurred. These are the basics of handling exceptions in Python but there’s one more thing to learn about, finally blocks.

Using finally

When a line of code raises an exception, Python immediately leaves that line and goes to the except block (or terminates the program) meaning that nothing under the line raising the exception gets executed. To get an idea why this is a problem, consider the following script:

def add(x, y):
  try:
    print(x + y)
    print("Add function completed")
  except:
    print("An error occurred")

add(2, 4)
add('hi', 9)

Here, we’ve defined an add function that tries to add its parameters together and print the result to the console, then prints that it has completed. An except block catches any exceptions and prints that an error has occurred. Under the function we call the function twice, once with numbers and once with a string and a number, forcing an exception. Let’s see what happens:

6
Add function completed
An error occurred

Notice that when the exception was raised, we didn’t see Add function completed printed to the console. This is because once the error was raised, the Python interpreter left the try block and went to the except block, leaving the print statement unexecuted. If we want to execute code regardless of if an error is raised or not, we have to use a finally block. The finally keyword is used like so:

def add(x, y):
  try:
    print(x + y)
  except:
    print("An error occurred")
  finally:
    print("Add function completed")

We’ve added a finally block and put the line printing Add function completed in it. This will ensure that this line runs every time the function is called, regardless of whether there is a caught exception. Run the script again and you should see the following output:

6
Add function completed
An error occurred
Add function completed

This time, the message Add function completed prints to the console every time. This is how finally blocks work.

Now we know how to handle exceptions with try, except, and finally. There is one other thing you should know about exception handling before we move on to customized exceptions. Handling exceptions is computationally slow. This is for a variety of reasons that are much too technical to be relevant to this chapter. The main point is that using exception handling to handle errors may be slower than other techniques (like if statements). Exception handling is best used when we don’t want errors to terminate program execution. Otherwise, it’s often better to use if statements or to just let the program execution terminate. With that caution out of the way, lets talk about customized exceptions.

Customized Exceptions

In the previous sections, we raised two different types of exceptions, ZeroDivisionError and TypeError. These exception classes are just two of many other built-in exception classes in Python. Other exception classes include ImportError, ModuleNotFoundError, KeyError, and many more. In addition to these built-in exception classes, we can also make our own custom exception classes. Customizing exception classes is useful in large projects because it aids in debugging, and it helps you define how errors are handled in your system.

Creating a Custom Exception Class

To create a customized exception, we simply write a class that inherits Python’s Exception class. Write the following:

class MyError(Exception):
  pass

As you can see, writing a custom exception is just a matter of making a subclass of Exception. Now, to raise this exception, we use the raise keyword. Consider the following script:

for i in range(1, 5):
  print(i)
  if i == 3:
    raise MyError

In this script, we make a simple loop that prints its iteration number but raises our customized MyError exception if the number is 3. If you run this script, you should see the following output:

1
2
3
Traceback (most recent call last):
  File "path/to/script.py", line 8, in <module>
    raise MyError
__main__.MyError

Notice that in the output, we see that our custom exception was raised. Suppose we did not know about this script but found a bug report saying “some method is throwing a MyError exception.” Since this is a custom (that is, non-built-in) exception class, we can search our codebase for any line that says raise MyError and figure out what’s going on. When developing large and complex software systems, saving time by searching for customized exceptions is very helpful.

Editing the Exception Message

We can further customize our exception classes by overriding their constructor methods and defining what message they print to the console. For example, suppose we are writing a human resources application for tracking employee salaries. We might have an Employee class like the following:

class Employee:
  def __init__(self, name, salary):
    self.name = name
    self.salary = salary

Here we have a basic Employee class that takes a name and salary attribute in its constructor. Now suppose we wanted to ensure that an employee’s salary is between 20 thousand and 500 thousand and we want to not only raise an exception when someone attempts to make an Employee object with a salary attribute outside of that range, we also want to log such an event to a database. First, we have to create a custom exception, we’ll call it SalaryError. Then, we’ll override its constructor to print why an exception was raised and to log the exception to a database. Check it out:

class SalaryError(Exception):
  def __init__(self, salary):
    self.message = f"Salary must be between 20k and 500k, you put {salary}"
    print("Logging the following to the database:")
    print(f"Attempted to create employee with salary {salary}")
    super().__init__(self.message)

In this code, we create a new class called SalaryError and inherit the Exception class, creating a custom exception. Then, we override the constructor by telling it to expect an input called salary. Then, we set the exception message to inform that a salary attribute must be between 20,000 and 500,000 and put what the attempted salary was.

Next, we print that we are logging the error to the database (in this example, we don’t actually log anything to a database since setting up the necessary connections for that would take away from the focus of this article). Finally, we use super() to fill out the base Exception class’s information so that the class will behave as a standard Python exception.

In order to implement this exception, we’ll tweak the Employee constructor to raise the exception if the salary attribute falls out of the desired range:

class Employee:
  def __init__(self, name, salary):
    if salary < 20000 or salary > 500000:
      raise SalaryError(salary)
    self.name = name
    self.salary = salary

Here, we’ve rewritten the Employee constructor to check if the salary range is within the 20 to 500 thousand range. If it is, we continue with constructing the object; if not, we raise the SalaryError exception. To see how this works, write the following code:

bob = Employee('Bob', 19000)

This code tries to instantiate an Employee object with a salary of 19,000, just below the required range. If you try to run this code, you should see the following output in the console:

Logging the following to the database:
Attempted to create employee with salary 19000
Traceback (most recent call last):
  File "path/to/script.py", line 16, in <module>
    bob = Employee('Bob', 19000)
  File "path/to/script.py", line 11, in __init__
    raise SalaryError(salary)
__main__.SalaryError: Salary must be between 20k and 500k, you put 19000

We can see here that the Employee constructor raised the SalaryError exception and printed out the appropriate error message. It also alerts the user that the error is being logged to the database.

This is the benefit of writing custom exceptions. Not only do we help our future debugging efforts by making exceptions searchable, we also can craft descriptive and helpful error messages. Additionally, custom exceptions let us specify the behavior of our program when encountering errors, allowing us to do things like report errors to a database.

Conclusion

In this article, we improved our debugging and quality assurance skills by learning about exceptions and exception handling in Python. We started by learning about handling exceptions, allowing us to define how the occurrence of certain errors affect the behavior of our programs. Then, we learned how to write our own custom exceptions and how doing so can improve our future troubleshooting efforts. These skills come in handy especially as your software system grows in complexity and usage, since this typically leads to interesting program states that need to be specifically handled.

For more Python-specific information on exceptions, check out chapter 8 in the Python docs.

If you’re trying to get better at Python, try one of my other language-specific tutorials like the delegation/decorator pattern series starting here: Delegate and Decorate in Python: Part 1 – The Delegation Pattern.

Test-Driven Development with Python: a Primer

Erik — Sat, 16 Sep 2023 21:12:05 +0000

Making sure the software we build works the way we (and our customers) want it to work is called, unsurprisingly, software testing. Software testing is an enormous topic; indeed, there are entire books, courses, conferences, academic journals, and more about the topic. One can even make a career out of testing software. We couldn’t possibly scratch the surface of the complexity involved in software testing in this article, so we’ll only focus on a topic most relevant to us as programmers: test-driven development.

What is Test-Driven Development?

To put it simply, test-driven development, or TDD, is a programming style in which a component’s tests are the primary mechanism guiding its development–this is often accomplished by writing tests before writing feature code. This is in contrast to the way software is traditionally built, where a feature is built end-to-end and then testing is added later as an afterthought.

There are several reasons for taking a test-first approach to building software. Most importantly, it forces you to think before you start writing code. Pondering how you might test something you’ve not written yet will necessarily make you imagine how you intend to build it, what you want it to do, and how other parts of the code might interact with it. Writing tests first also helps flush out unanswered questions you hadn’t yet thought about (for example, should a math function gracefully handle bad datatype inputs or fail loudly? What kind of data should a function expect as input? And so on).

One practical reason for writing tests before application code is that after a while, you have a fairly comprehensive test suite that reports on the functionality of most—if not all—of your system. This is a lifesaver as your projects get more complex and the possibility of a minor change in one part of the code breaking a feature in another area increases.

Now you may be wondering what it means to “write” a test. In the context of TDD, the word test is almost always shorthand for unit test—a program that checks the functionality of one small piece of functionality (a unit). If that sounds like it means we write code to test code, it’s because we do! In Python, unit tests are classes and methods that we build in order to verify the performance of other classes and methods that we build. Let’s first learn how to write unit tests before diving in to the TDD workflow.

Getting Started with Unit Testing in Python

As their name suggests, unit tests test one unit of code—that could be a function’s return value, the attributes of a class, or even a single unit of functionality like the ability to login to a website. We don’t have to get hung up on the semantics right now, let’s just start writing. Create a file called add_fucntion.py and write this function in it:

def add(x, y):
  return x + y

The above function simply takes two arguments and returns their sum (adds them together). Now, suppose we wanted to test this function to make sure it works. We might open an interactive Python shell and make sure it returns the values we expect based on the values we pass to it:

>>> from add_function import add
>>> add(5, 5)
10
>>> add(4, -2)
2
>>> add(True, "hello")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "path/to/add_function.py", line 2, in add
    return x + y

In the above session, we imported the add function from add_function. We passed 5 and 5 to the function to see if we got 10 as expected; we pass a 4 and a -2 to make sure the function handles signed and unsigned integers appropriately; and finally, we pass the Boolean True and string hello to the function to make sure it throws an error. All inputs behave as expected, so we can be pretty sure the add function behaves as expected.

This is fine for this function since it’s small and simple, but what if we have much larger functions or functions that change a lot? Are we going to manually test every function in a Python shell like this? As you might expect, the answer is no!

This is where unit testing comes in, we’ll write a program that will run all these different lines and more. Python has two popular testing frameworks: pytest and unittest. The unittest module is part of the standard library, meaning we don’t have to do anything special to install it, so we’ll use it for the examples in this chapter.

To get started with unittest, make a file in the same directory as add_function.py and call it test_add_function.py. Inside this new file we first need to import the unittest module and the file with the add function in it. To do that, make the top of the test_add_function.py file look like this:

import unittest

from add_function import add

Now, with the add function and unittest module imported, it’s time to write our first test. With unittest, we start by first defining a test class. Test classes are usually a collection of tests for one conceptual level of functionality; in this case, the add function. Let’s call ours AddTest:

class AddTest(unittest.TestCase):

Notice that the test class inherits the TestCase class from unittest. This gives us access to the assert methods we’ll use in a few minutes. Next, we’ll define a test method. Test methods are what make up test classes and we usually write one for each of the different ways our code might be used. For example, the most likely use of our add function is probably to add positive numbers, so let’s write a test method for that:

class AddTest(unittest.TestCase):
  def test_positive_addition(self):

In the above code, all we did was add the test method test_positive_addition to the AddTest test class. Now we’re ready to write the actual assertions. In this case, we’ll pass the add function two numbers, save its return value in a variable called actual, and compare that with a variable called expected (the number that should be returned). Then, we’ll use the assertEqual method from unittest to test that the two values are the same. Here’s how your entire test_add_funciton.py file should like by now:

import unittest

from add_function import add
class AddTest(unittest.TestCase):
  def test_positive_addition(self):
    expected = 30
    actual = add(10, 20)
    self.assertEqual(actual, expected)

Notice inside of the test method test_positive_addition we create two variables, expected with the value of 30, and actual with the return value of add when passed 10 and 20. Finally, we call the assertEqual class method and pass it actual and expected. As you might guess from its name, this method checks if the two values passed to it are equal.

You’ve now written your first unit test! It’s time to actually run the test. Open a terminal and, from whatever folder your add_function.pyand test_add_function.py files are in, run the following command:

python -m unittest

This runs the unittest program which will find any file that starts with the word test and runs the assertions inside. You should see output similar to the following:

.
----------------------------------------------------------------------
Ran 1 test in 0.000s

OK

You might not see it, but the single dot at the top of the terminal output represents a test method that passed. Had the test failed, you would have seen an F. This segues nicely into an important point: you should always make sure your tests can fail. We actually never saw our test fail so how can we be 100% certain that our unit test is working?

To make sure our unit test is working the way we expect, let’s change the expected variable to 40 and see what happens. Make this change in your code:

expected = 40

And run python -m unittest in your console once again. You should see something like the following output:

F

==================================================================

FAIL: test_positive_addition (test_add_function.AddTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/path/to/test_add_function.py", line 9, in test_positive_addition
    self.assertEqual(actual, expected)

AssertionError: 30 != 40
----------------------------------------------------------------------

Ran 1 test in 0.001s
FAILED (failures=1)

This is very interesting output, not only do we see that our test is testing the right thing (which we can see by the FAILED message when we tried to see if 20 + 10 could equal 40), but we also see what it looks like when a test fails. Notice that the output includes the file name and test method that failed as well as the line number of the assertion that caused the failure. This is super helpful when running big test suites (collections of automated tests like this one) with hundreds or thousands of tests.

Notice too that the failure message also tells us what failed. Specifically, it tells us that 30 != 40, which is exactly what we put into the test to force a failure. Go ahead and change the expected variable back to 30.

Now that we know how to write unit tests, let’s write a couple more. First, let’s test the method’s ability to handle negative numbers. We’ll also write this test specifically to fail. Check it out:

def test_negative_addition(self):
    expected = 0
    actual = add(10, -20)
    self.assertEqual(actual, expected)

Here we’ve defined another test method under the AddTest test class. This time, we are using negative numbers to see if the add function handles numbers lower than zero. We’ll test this by passing 10 and -20 to the add function, but instead of setting the expected variable to -10, we set it to 0. We do this because we want to make sure the unit test we just wrote is testing the right thing, and the best way to do that is by forcing an assertion failure. Once again, go ahead and run python -m unittest in your terminal, you should see something like the following:

F.

==================================================================
FAIL: test_negative_addition (test_add_function.AddTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "path/to/test_add_function.py", line 14, in test_negative_addition
    self.assertEqual(actual, expected)

AssertionError: -10 != 0
----------------------------------------------------------------------

Ran 2 tests in 0.001s
FAILED (failures=1)

Notice this time that the first line of output has a . followed by an F. This means that one of the test methods failed while another passed. As we hoped, the rest of the output tells us that our test_negative_addition method failed because -10 does not equal 0. Go ahead and change the expected variable to -10 so that the tests will pass.

Let’s write a slightly different test now. In the previous examples, we asserted two values were equal. This is of course an important kind of test to run but that’s not the only thing functions ever do. Remember earlier we tested what would happen by calling the add function by passing it True and the string hello and we saw that it raised a TypeError? We can actually test for that behavior too, check it out:

def test_bad_datatypes(self):
    with self.assertRaises(TypeError):
      add(True, "hello")

In the above test method, we start by using the with keyword in combination with the unittest assert method assertRaises and pass it TypeError. Finally, inside the with block, we call the add function and pass True and the string hello. We have to use the with block because otherwise the method will error (even though it’s the error we’re testing for).

If you run this, it should pass. For the sake of time, I did not show you how to make this test intentionally fail. I leave that as an exercise to the reader.

Aside from assertTrue and assertRaises, unittest has many other assertion methods such as assertAlmostEqual for approximating, assertIn for checking the presence of a single element in a list, assertFalse for checking things you expect to be untrue, and many more.

Now that we’ve learned the basics of writing unit tests in Python with unittest, let’s talk about the flow of TDD.

The TDD Flow

As I mentioned at the beginning of this article, test-driven development often consists of writing tests before writing actual code, but that’s just the beginning. The TDD workflow follows a three-phase cycle: write a failing test, write just enough code to make that test pass, refactor. You repeat this cycle until the code is doing what it’s meant to do, is thoroughly tested, and is as well designed as it can be. To understand the TDD workflow, we’ll start with analyzing the requirements and then repeating the cycle a couple of times.

The Requirements

Imagine we are building a cash register application for our favorite coffee shop. This application currently consists of a Drink class, representing the many different drinks the coffee shop serves, an Order class that records a customer’s orders, and an OrderItem class representing an individual item in an order. These three simple classes are in a file called coffee_cash_register.py and look like this:

class Drink:
  def __init__(self, name, price):
    self.name = name
    self.price = price

class OrderItem:
  def __init__(self, drink):
    self.drink = drink

class Order:
  def __init__(self, items):
    self.items = items

  def add_item(self, item):
    self.items.append(item)

The above class hierarchy consists of a Drink class which contains just a constructor and name and price attributes, an OrderItem class which simply contains a drink attribute, and an Order class that contains a list of OrderItem instances and a method for adding more OrderItem objects to that list.

We now want to build a class called CashRegister that serves as an way to access the Order class and allows an employee to create orders and build receipts. Since we are doing this the TDD way, our first step is to write a failing test for the CashRegister class.

Write a Failing Test

To write our first test for CashRegister, we first have to think about what kind of attributes and features the CashRegister class should have. Let’s start by thinking about what a real-world cash register does. It allows an employee to build an order by adding items a customer wants to purchase or removing items if one was accidentally added to the order. It should also be able to calculate the total price for the entire order, so the employee knows how much to charge the customer.

This thought process has revealed three things the CashRegister class must be able to do, and thus three test methods. Create a file called test_cash_register.py and write the following code:

import unittest

from coffee_cash_register import *

class CashRegisterTest(unittest.TestCase):
  def test_add_item_to_order(self):
    cr = CashRegister()

  def test_remove_item_from_order(self):
    cr = CashRegister()

  def test_calculate_total(self):
    cr = CashRegister()

Now remember, you want to write just enough of the test that the test should fail. We currently have three test methods, each creating a new instance of the CashRegister class—a class we haven’t even created yet. Run python -m unittest in the directory you’ve created these files and see that you have three failures with messages like name CashRegister is not defined. We’ve successfully completed the first part of our TDD cycle, writing a failing test.

Get the Test to Pass

Now that we have our test class and three test methods created and failing, it’s time to get them to pass. Since our tests failed because CashRegister wasn’t defined, let’s go ahead and create that class. In coffee_cash_register.py, add the following class:

class CashRegister:
  def __init__(self):
    pass

Here, we’ve created the CashRegister class with a very basic constructor. If you run the tests again, you’ll see that they pass. Believe it or not, that means we’re actually done with the second part of the TDD cycle: getting the test to pass. Let’s move on to the next step.

Refactor

Before we return to the start of the TDD cycle, we need to refactor the code we’ve written. This includes both the test code and the application code; but right at this moment in our particular example, the only thing that needs refactoring is our test code. Notice how we instantiated a CashRegister object at the beginning of each test method. In order to save us from retyping the same line for every test method, let’s just make cr a class attribute:

class CashRegisterTest(unittest.TestCase):
  cr = CashRegister()

 def test_add_item_to_order(self):
    pass

  def test_remove_item_from_order(self):
    pass

  def test_calculate_total(self):
    pass

Notice that we defined an attribute for the test class called cr and instantiated it with CashRegister(). Now we don’t have to write this line at the beginning of every test method. Notice also that we replaced the line inside each test method with pass, this is simply because since we no longer need to instantiate a CashRegister object, there is nothing to write in the test methods. If we leave them empty, our subsequent test runs will fail.

Now, let’s repeat the TDD cycle a couple more times to really get a feel for how this works.

Repeat

We’ve now completed one iteration of the TDD cycle. The next step is to write another failing test. Actually, we’ll just add to one of our existing tests and get it to fail. Let’s write the test code for the test method test_add_item_to_order. Again, we have to start by thinking about how this might work. We decide that adding an item to an order will most likely be done by creating a method in CashRegister that takes an instance of Drink, turns it into an OrderItem, and adds it to an Order object’s items attribute. We write the test method as such:

def test_add_item_to_order(self):
    drink = Drink("Latte", 3.49)
    self.cr.add_item_to_order(drink)

Now, if we run the tests, we should see a failure from this test method because the CashRegister object has no attribute add_item_to_order. The next step in the cycle is to get the test to pass, which right now means building that method for the CashRegister class. We might start by writing a method in CashRegister that would look something like:

def add_item_to_order(self, drink):
    item = OrderItem(drink)

But now that we’ve written this line and are wondering how to add it to an order, we realize that neither the method nor the class actually has an Order object to add the OrderItem to. Now we have to think about how we plan to instantiate the idea of an Order into our CashRegister app. Do we make it a class attribute of the CashRegister class? Do we make CashRegister inherit Order? Neither of these approaches sound quite right since a cash register in real life is mostly independent from the orders it builds. We decide instead to pass an instance of Order to the add_item_to_order method. Check it out:

def add_item_to_order(self, drink, order):
    item = OrderItem(drink)

In the method above, we’ve updated the argument list to take in an order parameter—an instance of the Order class. This will allow us to add the OrderItem instance to an actual Order instance. We also need to update our test code to account for the new method parameter:

def test_add_item_to_order(self):
    drink = Drink("Latte", 3.49)
    order = Order([])
    self.cr.add_item_to_order(drink, order)

Now that we’ve instantiated an empty Order object, we can call the add_item_to_order method without breaking anything. This means we’ve completed the “get test to pass” portion of the TDD cycle once again! The next step should be to refactor the code but, in this case, there is not really much to refactor, so we start the cycle over once again.

We’re now starting iteration three of the TDD cycle and once again, our first task is to get the test to fail. The best way to accomplish that at this point is to write our first assertion. Since the add_item_to_order method is supposed to add an Item object to an Order object’s items attribute, and since we are instantiating an Order object with an empty items attribute in our test, we can assert that this method is working as intended by checking that the length of order.items is greater than 0. Check it out:

def test_add_item_to_order(self):
    drink = Drink("Latte", 3.49)
    order = Order([])
    self.cr.add_item_to_order(drink, order)
    self.assertTrue(len(order.items) > 0)

Notice the line we added, self.assertTrue(len(order.items) > 0). We figure if the add_item_to_order method is working like it should, the order.items list will have one item in it. If we run this test, it should fail with a message like False is not True. This fails because, in our application code, we never add the OrderItem object to the Order object’s items attribute. We’re again done with the first part of the TDD cycle. Let’s complete the second part of the TDD cycle by fixing the add_item_to_order method:

def add_item_to_order(self, drink, order):
    item = OrderItem(drink)
    order.add_item(item)

We make this test pass by adding a call to the Order object’s add_item method which adds the item variable to the order variable’s items attribute. If you run the tests again, you should see that this one now passes. It’s time for the refactor stage.

Now that we think about it, the test we wrote is a little weak. The length of the order variable’s items attribute is only a side effect of what we really wanted to do which was add a LineItem containing the drink variable to the order variable’s items attribute. By only testing the length of items, the add_item_to_order method could add anything at all to the items attribute and our test would say that it’s passing. For the refactor stage, let’s be a bit more explicit about what we’re testing:

def test_add_item_to_order(self):
    drink = Drink("Latte", 3.49)
    order = Order([])
    self.cr.add_item_to_order(drink, order)
    item = order.items[0]
    self.assertIsInstance(item, OrderItem)
    self.assertEqual(item.drink, drink)

We’ve rewritten the test method to be more explicit. First, we extract the first object out of the order variable’s items attribute after calling the add_item_to_order method. We expect that the item should be an OrderItem object, so we use the unittestassertIsInstance method to make sure it is. Then, we use the assertEqual method to make sure that the item variable’s drink attribute is the Drink instance we created earlier in the test. This is a much more explicit test of the functionality we want because not only will it tell us if the method incorrectly created the OrderItem, it will let us know if we somehow accidentally appended the wrong kind of object to the Order instance’s items list.

It looks like this part of the class is pretty thoroughly tested now and we’re confident that the CashRegister class’s add_item_to_order method works as expected. You’ve now completed your first TDD workflow! Practice your understanding of this flow by repeating the steps for the other two test methods. I’ll give you a hint, to write a failing test for test_remove_item_from_order, you’ll probably have to call a method you haven’t written yet.

Conclusion

This article gently introduces the world of software testing by exploring the concept of test-driven development (TDD). We started by defining what TDD is and why it’s helpful. We highlighted the fact that TDD not only helps us think about the software we build, but it also has the side effect of leaving us with a collection of tests that we can always check our software against. Then, we took a quick detour to talk about how to use Python’s unittest testing framework so we could learn what it means to “write” tests. We did this so we could practice the TDD workflow with an example. In our example we ran through the TDD cycle three times to fully build out one thorough test method for our CashRegister class.

To learn more about TDD, I suggest you start trying to use the flow in your daily programming work. At first, you will likely be less productive than you are used to, but as you practice more, you may be more productive than you’ve ever been! If you are primarily a Python programmer, I suggest you install pytest and learn how to use that instead of unittest. The pytest package is much more popular in Python projects, we started with unittest so we could skip the tedium of installing a new package.

Understanding INNER, OUTER, LEFT, and RIGHT Joins in SQL

Erik — Mon, 08 Aug 2022 22:53:59 +0000

One of the first and most common things newcomers to SQL struggle with is how each JOIN is different from the other. In this article, I’ll explain the differences between the inner vs outer JOIN and left vs right JOIN in SQL using examples from each.

Introduction to Joining Tables

In SQL, we use JOIN in our queries to create result sets made up of multiple tables. For example, it’s common to have countries and states tables when capturing address information, so an address record may look something like this:

street_address city state_or_province country_id
123 Main St Houston Texas 1
558 Maple AveToronto Ontario 2

Here, the numbers in the country_id column of the addresses table refer to the IDs of their respective country in the countries table, which may look like this:

id country_name
1 United States
2 Canada

Now, say we want to write a query that displays a full address, something like 123 Main St, Houston, TX United States. Since the street, city, and state names are recorded in the addresses table, and the country name is recorded in the countries table, we have to join the two tables to get the result set that we want. The relevant SQL would probably look something like this:

SELECT
    a.street_address,
    a.city,
    a.state_or_province,
    c.country_name
FROM
    addresses a
JOIN
    countries c
ON
    a.country_id = c.id

The SQL code above shows us something like the following results:

street_address city state_or_province country_name
123 Main St HoustonTexas United States
558 Maple Ave Toronto Ontario Canada

Notice here that we have values from the street_address, city, and state_or_province columns of the addresses table, and values from the country_name column of the countries table. We were able to join these two tables together by matching the country_id value in addresses with the id column of countries.

There are a few kinds of joins in SQL, each one behaving slightly differently. This article aims to help you understand the difference between the different kinds of joins in SQL.

Download and Build the Sample Database

If you’d like to follow along with this article, go ahead and clone the code to build the example database . To build the database in pgAdmin, first run the command in create_database.sql then make sure to change connections to the newly created employee_database database and run the commands in build_and_seed_tables.sql.

You should now have a database with four tables: addresses, countries, departments, and employees. Let’s quickly discuss the data model before moving on to the actual point of this article.

The addresses and countries tables are just like the ones in the examples above; each address record references a country record. Then, we have an employees table, each record of which references an address record as well as a department record. The department records are pretty simple; the departments table simply has an id and name column and refers to what department an employee works.

Ok, now on to actual joins!

Right vs. Left Joins

Let’s start by talking about the difference left and right joins. It’s helpful to think about all the tables in a SQL query as a horizontal list of tables. The leftmost table is the table in the FROM clause, the next table to the right of the leftmost table is whatever table we are joining. So for example, if we write the query

SELECT * FROM employees e JOIN addresses a ON e.address_id = a.id

Then the employees table is the leftmost table while the addresses table is the next one to the right, kind of like this:

If we take it a step further and join the countries table to the addresses, the next table to the right would be countries. In other words, the following SQL query:

SELECT *
FROM employees e
JOIN addresses a ON e.address_id = a.id
JOIN countries c ON a.country_id = c.id

can be visualized as the following:

Notice again that whenever a table is joined to another, the already-existing table (for lack of a better word) is the left table and the joining table is the right table. So in our example above, our base table is employees so it’s on the left; then we joined addresses to it, so it’s to the right of employees. Finally, we then joined countries to addresses, putting addresses to the right of countries and thus the rightmost table.

This visualization is helpful in understanding how left and right joins work. For example, query all of the employees records and LEFT JOIN the addresses table and see what happens. The query:

SELECT e.first_name, a.street_address, a.city
FROM employees e
LEFT JOIN addresses a ON e.address_id = a.id

The results:

"Bob" "123 Main St" "Houston"
"Jordan" "220 C 30" "Izamal"
"Nicole" "3030 Burgos" "Xalapa"
"Erik" "1010 2nd St" "Omaha"
"Lina" "1010 2nd St" "Omaha"
"Sandra", NULL, NULL

Notice here that all records for employees, the leftmost table, appear as results in this query, including a record with no associated addresses record. The records in the rightmost table, addresses, only appear if they can be joined to the leftmost table. There are records in addresses that cannot be joined to employees because no employees record has a corresponding address_id.

Conversely, if we use RIGHT JOIN to join the addresses table, we will see results that include all of the addresses table–the rightmost table–despite not being able to be joined to the employees table. For example, the query:

SELECT e.first_name, a.street_address, a.city
FROM employees e
RIGHT JOIN addresses a ON e.address_id = a.id

The results:

"Bob" "123 Main St" "Houston"
"Jordan" "220 C 30" "Izamal"
"Nicole" "3030 Burgos" "Xalapa"
"Erik" "1010 2nd St" "Omaha"
"Lina" "1010 2nd St" "Omaha"
null "558 Maple Ave" "Toronto"
null "99 Jackson Rd" "Flin Flon"
null "821 Carol" "Nuuk"

Notice this time that we have three addresses records in the result set that have null values in their respective addresses column. Also notice that the employee record with no associated address record is not in this result set. This is because by doing a RIGHT JOIN, we essentially tell the database to give us all results of the rightmost table, and join only the leftmost table’s records if they are associated with the rightmost.

Conversely, we can switch which table is right and left by referring to the addresses table in the FROM clause and then joining the employees table. For example, let’s SELECT from addresses and LEFT JOIN the employees table. The query:

SELECT e.first_name, a.street_address, a.city
FROM addresses a
LEFT JOIN employees e ON e.address_id = a.id

The result:

"Bob" "123 Main St" "Houston"
"Jordan" "220 C 30" "Izamal"
"Nicole" "3030 Burgos" "Xalapa"
"Erik" "1010 2nd St" "Omaha"
"Lina" "1010 2nd St" "Omaha"
null "558 Maple Ave" "Toronto"
null "99 Jackson Rd" "Flin Flon"
null "821 Carol" "Nuuk"

The results when we LEFT JOIN the employees table to the addresses table are the same as when we RIGHT JOIN the addresses table to the employees table. That’s because in this case, the leftmost table is addresses so by left joining the employees table to it, we are telling the database to give us all the addresses records and then join the employees table records if they exist.

Now, let’s take it a step further and see what happens when we chain JOIN commands together. First, let’s SELECT FROM the addresses table and RIGHT JOIN the countries table so we can see how many of the countries records are not associated to addresses. The query:

SELECT
    a.street_address,
    a.city,
    a.state_or_province,
    c.country_name
FROM addresses a
RIGHT JOIN countries c
ON a.country_id = c.id

The results:

"1010 2nd St"   "Omaha" "NE"    "United States"
"123 Main St"   "Houston"   "TX"    "United States"
"3030 Burgos"   "Xalapa"    "Ver"   "Mexico"
"220 C 30"  "Izamal"    "Yuc"   "Mexico"
"99 Jackson Rd" "Flin Flon" "MB"    "Canada"
"558 Maple Ave" "Toronto"   "ON"    "Canada"
"821 Carol" "Nuuk"  "SQ"    "Greenland"
NULL    NULL    NULL    "Iceland"

As we can see from the result set of our previous query, the only country in our database that isn’t related to an address record is Iceland.

Knowing this, let’s see what happens when we SELECT FROM the employees table, LEFT JOIN the addresses table to it, and then RIGHT JOIN the countries table to that. Do you think we’ll see all the countries? Let’s find out! The query:

SELECT
    e.first_name,
    e.last_name,
    a.street_address,
    a.city,
    a.state_or_province,
    c.country_name
FROM employees e
LEFT JOIN addresses a
ON e.address_id = a.id
RIGHT JOIN countries c
ON a.country_id = c.id

The results:

"Lina"  "Mazin" "1010 2nd St"   "Omaha" "NE"    "United States"
"Erik"  "Whiting"   "1010 2nd St"   "Omaha" "NE"    "United States"
"Bob"   "Robertson" "123 Main St"   "Houston"   "TX"    "United States"
"Nicole"    "Nicholson" "3030 Burgos"   "Xalapa"    "Ver"   "Mexico"
"Jordan"    "Mays"  "220 C 30"  "Izamal"    "Yuc"   "Mexico"
NULL    NULL    NULL    NULL    NULL    "Canada"
NULL    NULL    NULL    NULL    NULL    "Greenland"
NULL    NULL    NULL    NULL    NULL    "Iceland"

These are interesting results that show us something about the way RIGHT and LEFT joins behave. Notice that we only get 5 rows of results that include records from the addresses table even though there are 7 records in that table. This is because there are only 5 addresses that are associated to employees records, and when we LEFT JOINed the addresses table, we told the database that we only want records from the addresses table if they’re associated with an employee record. However, when we RIGHT JOIN the countries table, we tell the database we want all records from the countries table, even if there’s no associated addresses record in the result set.

Now that we’ve talked all about the difference between RIGHT and LEFT, let’s talk about INNER and OUTER joins.

Inner vs. Outer Joins

The difference between INNER and OUTER joins are very similar to the differences between LEFT and RIGHT joins. However, in this case, it’s more helpful to think of tables as Venn diagrams. You know, the ones where two circles meet in the middle and the overlapping section is some shared attribute between the circles?

We still want to think of tables being joined from left to right, but with INNER joins, we have to think about how records might overlap.

NOTE: The default JOIN operation in Postgres (and every other RDBMS I've ever worked with) is the INNER JOIN. That means if you write something like SELECT * FROM A JOIN B ON A.b_id = B.id, it is implied that you mean to do an inner join and the SQL SELECT * FROM A INNER JOIN B ON A.b_id = B.id is exactly the same thing.

Let’s see how INNER JOIN works when we join addresses to employees. The query:

SELECT e.first_name, a.street_address
FROM employees e
INNER JOIN addresses a
ON e.address_id = a.id

The results:

"Bob"   "123 Main St"
"Jordan"    "220 C 30"
"Nicole"    "3030 Burgos"
"Erik"  "1010 2nd St"
"Lina"  "1010 2nd St"

Notice this time that we have five results, despite there being six employees records in the database. This is unlike the LEFT JOIN from the previous section where the employee record with no associated address record was still in the result set. In this way, INNER JOIN means we only want results from the leftmost table if the rightmost table can be joined to them. That’s why you often see Venn diagrams used to explain joins. In this case, INNER JOIN is this Venn diagram:

So, the main difference between LEFT and INNER joining that we’ve seen so far is that if you want all records of the leftmost table–whether they have associated records in the rightmost table or not–you want to use a LEFT join. If you only want records from the leftmost table if a record from the right most table can be joined to it, you want to use INNER JOIN.

So what about OUTER? The OUTER JOIN is a special case because even though it seems antimonious with INNER JOIN, OUTER joins need to to be specified with either RIGHT, LEFT, or FULL. For example, the following SQL will result in a syntax error:

-- Doesn't work!
SELECT e.first_name
FROM employees e
OUTER JOIN addresses a
ON e.address_id = a.id

We have to specify if we want the leftmost or right most table to be outer-joined. Let’s see how LEFT OUTER JOIN behaves. The SQL:

SELECT
    e.first_name,
    a.street_address
FROM employees e
LEFT OUTER JOIN addresses a
ON e.address_id = a.id

The results:

"Bob"   "123 Main St"
"Jordan"    "220 C 30"
"Nicole"    "3030 Burgos"
"Erik"  "1010 2nd St"
"Lina"  "1010 2nd St"
"Sandra"    NULL

Here we specified that we want the leftmost table to be outer-joined and therefore got all results from employees regardless of whether a record from the addresses table can be joined to it or not.

The RIGHT OUTER JOIN on the other hand will include all records from the rightmost table regardless of whether a record from the leftmost table can be joined to it. The SQL:

SELECT
    e.first_name,
    a.street_address
FROM employees e
RIGHT OUTER JOIN addresses a
ON e.address_id = a.id

The results:

"Bob"   "123 Main St"
"Jordan"    "220 C 30"
"Nicole"    "3030 Burgos"
"Erik"  "1010 2nd St"
"Lina"  "1010 2nd St"
NULL    "558 Maple Ave"
NULL    "99 Jackson Rd"
NULL    "821 Carol"

See here that all the addresses records were included in the result set, even if there was no employees record that could be joined to it. Notice also that the value from employees that has no associated addresses record is not included in the result set.

Did you notice that the results for LEFT OUTER JOIN and RIGHT OUTER JOIN are the exact same results we get from LEFT JOIN and RIGHT JOIN respectively? If so, good on you! LEFT JOIN and LEFT OUTER JOIN are exactly the same thing; same with RIGHT JOIN and RIGHT OUTER JOIN. Whenever you use RIGHT or LEFT in your JOIN, the OUTER is implied and you actually don’t have to write it (though some people do because they say it adds clarity).

There’s one more kind of OUTER join: FULL. The FULL OUTER JOIN will include all results of both the left and rightmost tables, regardless of whether they can be joined to each other. Check it out, the SQL:

SELECT
    e.first_name,
    a.street_address
FROM employees e
FULL OUTER JOIN addresses a
ON e.address_id = a.id

The results:

"Bob"   "123 Main St"
"Jordan"    "220 C 30"
"Nicole"    "3030 Burgos"
"Erik"  "1010 2nd St"
"Lina"  "1010 2nd St"
"Sandra"    NULL
NULL    "558 Maple Ave"
NULL    "99 Jackson Rd"
NULL    "821 Carol"

See how this time we have all records from each table. The leftmost table, employees, includes even the record with no associated addresses record. Likewise, the result set includes three addresses records with no associated employees record. That’s because the FULL join is both a LEFT and RIGHT join put together.

Conclusion

Nearly everyone struggles with the different kinds of joins when they’re first learning SQL, so let’s review what we learned:

LEFT, RIGHT, and FULL joins are the same as LEFT OUTER, RIGHT OUTER, and FULL OUTER joins
- LEFT will include all records from the leftmost table, even if there are no records in the rightmost table that can join to them
- RIGHT will include all records from the rightmost table, even if there are no records in the leftmost table that can join to them
- FULL includes records from both the rightmost and leftmost tables, even if the records from one table have no joining records in the other
INNER JOIN is the default join
- INNER will only include records where both the leftmost and rightmost tables have associated records
- Use this when you want only full results; records from one table with no associated records in the other table will not show up in the result set

Now that you know how to use joins, try out some of them in the example database using the departments column. As always, feel free to tweet me at @erikwhiting4 or send me an email at erik@erikwhiting.com if you have any questions. Good luck!

Is there a Better Label for "Self-Taught" Developers?

Erik — Sun, 08 May 2022 15:22:08 +0000

If you hang out in software development spaces online long enough, you’ll start to notice there’s a notion of “self-taught” developers. I think it’s time to rethink that label.

What is a Self-Taught Developer?

Before I talk about why I disagree the concept of developers being self-taught, let me first introduce the characteristics most often associated with the label. It’s best to start by talking about what a self-taught developer is not.

The term self-taught developer is used to describe developers who did not learn their skills from someone else. The following people are typically not considered self-taught:

College students/graduates
Coding bootcamp graduates
Someone who was trained on the job

This implies that the only people who have the supposed right to call themselves self-taught developers are people who picked up their skills by setting out on their own, reading articles, watching videos, digging through documentation, or asking questions online.

But wait, what’s the difference?

Self-Taught vs. Not

On the surface, it seems like the differentiating factor between self-taught developers and developers who aren’t is the presence of someone else doing the teaching. Linguistically, this makes sense, you’re self-taught because you looked up all the information on your own.

However, the sources from which a self-taught person gets this information is produced by other people. There are scores of people online writing articles, answering questions, and sharing code, free to be consumed by anyone with an internet connection.

Why aren’t internet content creators considered teachers? Why does learning Python from a blogger preserve a person’s self-taught label, but learning Python from a professor does not?

Proposing a Better Label: Community-Taught Developer

You may be wondering why I care about this. After all, the label isn’t hurting anyone, and it is indeed harder to navigate the programming world on your own; why rob people who figured it out the distinction of calling themselves self-taught?

I don’t actually want to trivialize the effort required to figure out the programming world with minimal direction. However, a person can learn nearly everything they need to know to be a professional developer from online content produced by many people freely writing articles, making videos, and answering questions on Stack Overflow and other websites.

The entire community of software developers, over time, have produced a giant corpus of software engineering knowledge. Any time you read language documentation, watch a YouTube video, or read a how-to article about something related to programming, you are consuming the knowledge of the software development community. I believe the concept of “self-taught” developers discredits that community, or at least makes it seem less important than it is.

In this way, I believe that most people who call themselves self-taught developers could more accurately describe themselves as “community-taught” developers. This title still indicates that the learner took on most of the initiative to figure out a learning path for themselves, but it also honors the cumulative efforts of the larger community.

Am I Bitter?

You may be thinking I’m biased and/or bitter about the idea of people calling themselves self-taught developers because I myself am a university-educated developer. Furthermore, I’m one of those bloggers I was just talking about. Am I just looking for my own kudos? Am I jealous of the self-taught label?

Perhaps, but consider my learning path and how I arrived at these thoughts about giving credit to the community for my own competence.

I learned C++ from this website before I ever took a computer-based class in college. I read every word and did all the exercises. Whenever I was stuck, I typed my problem into a search engine and found an answer, usually on Stack Overflow.

I did all of this on my own, but I don’t feel right calling that “self teaching.” The authors of that website put in countless hours to create a comprehensive and well-written tutorial. Likewise, the people answering questions on Stack Overflow are not usually gaining anything from doing so. Without them, I never would have gotten as far as I did in this field. Calling myself self-taught, I believe, trivializes their efforts.

Conclusion

It’s still a monumental task to take on the responsibility of learning everything you need to know about software development on your own. This article wasn’t intended to minimize those efforts. However, the greater community of developers, writers, and OSS maintainers that put in time to share their knowledge deserve some of the credit, hence why I propose a new title: community-taught developers.

How to Implement Many-to-Many Relationships in Relational Databases

Erik — Sun, 20 Feb 2022 08:27:30 +0000

Nearly any data model of non-negligible complexity will include at least one (if not several) many-to-many relationships. This relationship emerges when, given two tables A and B, instances of A could have many instances of B, and instances of B could have many instances of A. Successfully implementing this relationship is a bit of stumbling block for people new to working with or building databases. However, the strategy for implementing this relationship in relational databases is important to understand. Believe it or not, it’s also not all that complicated.

Once you’ve implemented this relationship a few times, the strategy becomes rather intuitive. However, I remember this concept being one of the hardest ones for me and my classmates to finally understand when it was introduced in second year of undergrad. Hopefully, this article will help you internalize the many-to-many relationship like I eventually did.

What is the Many-to-Many Relationship?

The many-to-many (m2m) relationship is a relationship that occurs in data modeling when two entities are related but neither belong to the other and both can have many instances of each other. Here are some examples of many-to-many relationships:

a pet can have many owners, and owners can have many pets
a language can be spoken by many people, and a person can speak many languages
a class can have many students, and a student can be in many classes
a venue can have many visitors, and a visitor can visit many venues

To better understand the many-to-many relationship, let’s briefly talk about the one-to-many relationship to see how the m2m differs:

a country can have many cities, but a city can only be in one country
a car company can make many car models, but each car model is made by only one company

In each of these examples, note that there is a sense of one entity belonging to another.

Modeling One-to-Many in the Database

One-to-many relationships are easy to model in the database. For example, consider the country-city example. The country India has many cities such as Mumbai, Hyderabad, and Kolkata, but Kolkata only belongs to the country of India.

So, in the database, we probably have a Country table that looks like this:

Country table with Id and Name columns

And the City table plus relationship to the Country table would look like this:

Country/City relationship

This is simple enough, and we can query for all cities in a country by simply writing something like SELECT City.Name FROM City JOIN Country ON Country.Id = City.CountryId WHERE Country.Name = 'India' and we’d get a result set including Mumbai, Hyderabad, Kolkata, and more.

But this kind of modeling is easy and we’ve only used it as an example to better understand our next section: modeling the many-to-many relationship.

Modeling Many-to-Many in the Database

Now let’s consider the more complex many-to-many relationship. Specifically, let’s consider the instance of many people speaking many languages. For example, we might have a Person table and Language table like so:

Person and Language tables

Now, suppose we have two people: Erik and Lina. Erik speaks English and Spanish, Lina speaks English and Arabic. This is what we mean by many-to-many relationships; Lina speaks many languages–English and Arabic–and English is spoken by many people–Erik and Lina. So how do we relate the Person and Language tables to list out all of the languages a person speaks, or all of the people that speak a language?

The answer is to create a new table, the associative table. The associative table sits between the two entities in a many-to-many relationship and actually has a one-to-many relationship with each. That is, given tables A and B and an associative table C, A has many Bs through C and B has many As through C; put another way, A has many Cs, B has many Cs, but C has one A and one B.

Note: The associative table are also sometimes called association, bridge, intermediary, join, junction, crosswalk, or linking table (and there’s probably many more names for it). I learned the term “associative” table in undergrad, and I think that’s its official name in relational theory, so that’s what we’ll stick with in this article.

Let’s make this more concrete by creating an associative table for the Person and Language relationship. It’s common practice to name association tables as an amalgamation of the tables they join, so we’ll call our associative table PersonLanguage. I’d create such a table with the following SQL:

CREATE TABLE PersonLanguage(
    Id INT IDENTITY(1,1) PRIMARY KEY,
    PersonId INT FOREIGN KEY REFERENCES Person(Id),
    LanguageId INT FOREIGN KEY REFERENCES Language(Id)
)

Notice that we specified two foreign keys on this table, PersonId and LanguageId, pointing to the Person and Language tables respectively. The table structure now looks like this:

Now, there will be a record in the PersonLanguage table for every language an individual Person speaks. As a result, there will also be a record in the PersonLanguage table for every person that speaks a Language. If we want to see the list of languages Lina speaks, we could run the following query:

SELECT l.name
FROM PersonLanguage pl
JOIN Person p ON p.id = pl.PersonId
JOIN Language l ON l.Id = pl.LanguageId
WHERE p.name = 'Lina'

and we’d get results like:

Results

Likewise, if we want a list of all the people that speak English, we could run a query like this:

SELECT p.name
FROM PersonLanguage pl
JOIN Person p ON p.id = pl.PersonId
JOIN Language l ON l.Id = pl.LanguageId
WHERE l.name = 'English'

This would give us the following results:

Results

Extending the Associative Table

The above examples show the most basic implementation of building associative tables to break up many-to-many relationships. The example PersonLanguage associative table above only has two columns (besides its own ID column) for pointing to Person and Language records.

We can actually add columns to this table as well and capture interesting information. For example, what if we wanted to record which language was a person’s mother tongue? In other words, if Erik grew up speaking English and learned Spanish later, but Lina grew up speaking Arabic and learned English later, how would we capture that?

Thanks to the associative table, the answer is pretty easy. We just have to add a column to PersonLanguage that indicates if the Language record captured is the Person record’s primary language. In T-SQL, I’d run the following ALTER statement:

ALTER TABLE PersonLanguage ADD PrimaryLanguage BIT

and set the PrimaryLanguage value to 1 for languages the person learned from birth. By capturing this data on the associative table, we can now write a query to generate a report of every person and their mother tongue like so:

SELECT p.name AS 'Person', l.Name AS 'Primary Language'
FROM PersonLanguage pl
JOIN Person p ON p.id = pl.PersonId
JOIN Language l ON l.Id = pl.LanguageId
WHERE pl.PrimaryLanguage = 1

and we’d get results like:

Results

That is just one example of how to extend the functionality of our database with associative tables.

Conclusion

In this brief article, we learned how to break up many-to-many relationships in relational databases. The general approach is to place an associative table between the two tables and have a one-to-many relationship with them. Then we saw how we can use associative tables to capture nuanced data and create interesting reports. This is an important concept to understand in any kind of software or data engineering work that uses relational databases. As always, I hope this was helpful to you and don’t be afraid to contact me with any questions you might have!

Improving Browser Automation Tests: How to Add Microsoft Edge and Edgedriver to Linux CI Systems

Erik — Tue, 28 Sep 2021 23:09:27 +0000

Lately, the Microsoft Edge browser has been growing in popularity, recently unseating Firefox as the 3rd most popular web browser and approaching Safari in the number two spot. This means that any web application with a potentially wide user base should include MS Edge tests in its automated test suite. Setting this up to run automatically, particularly in *nix based CI servers, can be quite a hassle when compared to setting up Chrome and Firefox browser tests. In this article, I’ll show you the steps to get the Edge browser and web driver running in a Linux-like CI system.

Why should I run tests in multiple browsers

If you write web applications, you may only ever test them in the browser of your choice. For example, I use Chrome almost exclusively, and when I write new features for an application, I never really check and see if those features work the same in Firefox, Safari, Edge, or anything else.

For the most part, you can get away with this, but each browser is a little different and if you’re only testing through one browser, you only know how your web apps look and behave for people using that browser. There are subtle differences between browsers and I’ve received reports of browser-specific bugs from users before that I could’ve caught if I’d just verified my app through something other than Chrome.

Likewise, you should probably add some variety to your automated test suite and not rely solely on Chrome and Chromedriver for your web-based automated tests. I’m not saying you need to run every feature test in each major browser (though you certainly could) but any test that interacts with or verifies the status or behavior of:

The UI
Cookies
Session data

should be tested in at least two popular browsers. These things have the most variability between different browsers, so it’s good practice to verify these attributes with multiple browsers and web drivers.

Now that we know why we want to test features with multiple browsers, let’s get to the main point of this article: running automated browser tests with Microsoft Edge and Edgedriver.

Edge-Specific Challenges

The reason an article like this is needed is because most CI pipelines run in Linux-like environments, and Microsoft Edge isn’t really “native” to this operating system. Additionally, many popular CI systems (like CircleCI, TravisCI, GitHub Actions) don’t have out-of-the-box support for Edge and Edgedriver like they do for Chrome and Chromedriver or Firefox and Geckodriver.

Therefore, we have to add some specific steps in our build files, which we’ll go over in the next section.

Add MS Edge and Edge Driver to Server

Our general strategy moving forward will be like so:

Install Microsoft GPG
Add MS Edge “dev” distro to sources list
Install the development version of MS Edge (the browser)
In the development version of Edgedriver (the Edge-specific web driver)
Make the driver accessible to the OS

Before I show you the script, there’s one more thing you need: the current Edgedriver development version. We’re going to save ourselves some work by only using the stable dev version of the Edge browser, which we can do by simply specifying “stable” in our script (which you’ll see in a moment) but there is no such easy way to get the latest stable Edgedriver. That’s why we need to go find the specific version.

To get the version number, go to this page and look at the version number under “Dev.” As of writing this, the number says 95.0.1020.5. We’ll use this number inside of a URL in our build script. Finally, the script will look like this:

curl https://packages.microsoft.com/keys/microsoft.asc | gpg --dearmor > microsoft.gpg
sudo install -o root -g root -m 644 microsoft.gpg /etc/apt/trusted.gpg.d/
sudo sh -c 'echo "deb [arch=amd64] https://packages.microsoft.com/repos/edge stable main" > /etc/apt/sources.list.d/microsoft-edge-dev.list'
sudo rm microsoft.gpg
sudo apt update
sudo apt install microsoft-edge-dev
wget https://msedgedriver.azureedge.net/95.0.1020.5/edgedriver_linux64.zip -P ~/
unzip ~/edgedriver_linux64.zip -d ~/
rm ~/edgedriver_linux64.zip
sudo mv -f ~/msedgedriver /usr/local/share/
sudo chmod 777 /usr/local/share/msedgedriver
sudo ln -s /usr/local/share/msedgedriver /usr/local/bin/msedgedriver

This script is currently a build step for a personal project of mine and is working fine in GitHub Actions. If you’re using a different CI, the exact syntax of how to lay out the different commands may be different, but the actual commands and their order are correct regardless of CI tool.

Notice in the wget command, we have the Edge driver version number in the URL and we’ve specified that we’re using the 64 bit Linux build.

Issues with Local vs Remote Driver

There is one issue I was running into when trying to run my tests in a remote CI server. I don’t remember the specifics of the issue, nor do I know if it will be the same in all Selenium ports (I use Python) but basically the issue is this: When you’re using the dev version of the browser and driver for edge, you have to tell Selenium you’re doing so or the tests will not find the driver.

My approach to fixing this issue is to set an environment variable in the CI script such as RUNNING_IN_CI=True or something similar and then check for that variable in the test. If it’s not there, that means I’m running tests locally and I don’t need to specify that I’m using the dev version. If that variable is present however, I need to set the following values (this is specific to LuluTest browser automation framework, but your implementation of Selenium will likely be similar):

options_hash = BrowserOptions({
    'driver_type': browser,
    'headless': True,
    'browser_binary_location': shutil.which('microsoft-edge-dev'),
    'webdriver_location': shutil.which('msedgedriver'),
    'operating_system': 'LINUX'
})

Here I’m using Python’s shutil library to find the path of Microsoft Edge and Edgedriver in the CI system, as well as some other housekeeping things. This is necessary if you’ve used a build script like mine above.

That should do it!

Join my OSS Browser Automation Project!

Thanks for reading this article, I hope it helped you. If you like browser automation and open-source software, consider checking out my OSS project LuluTest and adding some contributions. I’m looking for developers, testers, and writers alike! I’ve tried to make the project as beginner friendly and inclusive as possible so with #hacktoberfest 2021 coming up, maybe you can knock out some of your PRs with me.

Professional Version Control with Git: Pt 3 – Rebase and Bisect

Erik — Sat, 10 Jul 2021 19:39:18 +0000

Part 1 - The Basics
Part 2 - Collaboration

Hello everyone and welcome to part 3 of the professional Git series here at Erik’s Code Space. In part 1, we learned the basics and got our skills good enough to start version controlling our own projects. In part 2, we learned about the collaboration tools available in git and got our skills good enough to start contributing to open source projects. In this part, we’re going to learn about the rebase and bisect commands, two commands that help us with troubleshooting. Without further ado, let’s begin.

Keeping History Clean

In part 1, we learned about how our commit messages become the official “history” of our projects. The git history is supposed to keep a timeline-like record of the growth and change of our repository, so we try to keep our commit messages accurate and helpful, so then when someone runs a git log command against our repo, they get an accurate idea of how the project has been evolving lately.

Sometimes though, we need to rewrite history, or get rid of it completely. Maybe we made a bad commit, made a typo in our commit message, or need to combine two commits into one. That’s where git rebase comes in! Let’s get started; make a directory called rebase_and_bisect, cd into it, and run the git init command. If you’re using a Linux or Mac terminal, or GitBash for Windows, the following commands will do this:

$> mkdir rebase_and_bisect
$> cd rebase_and_bisect
$> git init

Now make a file called poem.txt and commit it to the repository:

$> touch poem.txt
$> git add .
$> git commit -m "Create file"

Now we’ll populate that file with a beautiful poem I’ve written. We will add one line at a time and commit that changes before writing the next line. I will share the commands to do this below, without the command prompt (the $>) so that you can copy and paste them into your terminal, saving you some time. (Please note, the following commands will probably not work with PowerShell or CMD, please use GitBash if you’re on Windows):

echo "Roses are red" > poem.txt
git commit -am "Add first line"
echo "JavaScript is yellow" >> poem.txt
git commit -am "Add second line"
echo "I'm learning git" >> poem.txt
git commit -am "Add third line"
echo "and I wanna say hello!" >> poem.txt
git commit -am "Add fourth line"

You might have to manually press “enter” one more time, but you should now have a four line poem in poem.txt.

There’s one more piece of housekeeping we have to do before we get on with the rest of this tutorial, setting up our git editor. The git editor is the text processing program that opens when you do things with git that require editing text. I believe the default text editor on GitBash is vim, which isn’t user friendly. Personally, I like nano, so in order to set my default git text editor to nano, I would run the following script:

$> git config --global core.editor "nano -w"

If you want to use a different program, you have to pass the path to that program in place of where I put nano -w. For a more exhaustive list of how to set specific text editors, click this link.

With this out of the way, let’s learn about rebase.

Rebase

We learned in part 1 that commit messages make up the history of our project. When someone runs git log, they see all the commit messages made on the project. Our goal is to write concise yet descriptive messages of the work we’ve done so that the collection of commit messages show an accurate picture of where our project came from and how it’s evolving.

Sometimes though, we make mistakes with our commit messages. We could make a typo, duplicate commits, or commit something we really didn’t mean to. Once we’ve made the commit though, the damage is already done, right? Wrong! With git rebase, we can rewrite history!

Making Bad Commit Messages

One of the main uses of using git rebase is rewriting commit messages. We’ll open up poem.txt in our text editor of choice and sign our name at the bottom. Then we’ll commit the changes but, oh no! We make a typo:

$> git commit -am "Signing my nammee"

Run git log to see the typo in our list of otherwise good commit messages. We’re now going to use rebase to fix this. We’re going to use the interactive version of rebase to do this, and we’re going to go back one commit when rebasing. So the command I want you to run is:

$> git rebase <strong>-i</strong> HEAD<strong>~1</strong>

The -i is for “interactive” and the HEAD~1 means “go back one commit from the latest version.” As soon as you hit enter, you should see something like this:

Let’s talk about what we’re looking at real quick. This text file is called git-rebase-todo and lives inside the local .git directory for this project. The first line is the commit we’ve rolled back to, the one with the typo. Below that is a few lines to help users figure out what to do from here, and is pretty helpful once you get the hang of rebasing. For now though, I’ll just walk you through the next few steps.

So the goal here is to fix the typo from our last commit message. That means that we want to “reword” the commit message, so on the first line, delete pick and type reword. Then, save the file and exit it (if you’re using nano, you save by pressing “Ctrl + o” then press enter and exit with “Ctrl + x”).

Once you’ve closed the file, another one will open containing the commit message for that particular commit. Mine looks like this:

Scroll over to nammee and change it to name, save, and exit. You’ll be returned to the terminal with a message about successfully rebasing. Run git log again and what do you notice? The typo from our last commit message is gone and our history is saved! Here’s what mine looks like:

#### Did you get Stuck in Vim?

If you missed the part about setting your default text editor and your files opened in a text editor that doesn’t seem to make sense, you might be in vim. Don’t panic, I’ll walk you through getting out of this. First, check and see that you really are in vim. If you are, you’ll notice that all the lines below the commented-out portion (the portion preceded by #s) are all tildes (~). In my GitBash, vim looks like this:

Operating vim is a little weird if you’re not used to it. When it first opens, you’re in “normal” mode and you can’t edit text the way you think you can. To enter “insert” mode, just type the letter i. Once you’ve done this, you can edit the text like you would in a normal program, using the arrows to maneuver around and the backspace and delete buttons the way you’d expect.

Once you’ve changed “pick” to “reword,” you have to go back into “normal” mode by pressing the Esc key. Once you’ve done this, you can save the file by first pressing the key combination for colon (:) which is Shift + ;. You’ll notice now that your cursor is now at the bottom left of the screen. Type wq! and then press enter. This is the command for saving and exiting (or writing and quitting). Repeat the process when it opens the commit message file and you should be good to go!

Consolidate Commits

Sometimes, I forget to do a whole task before I commit it. As an example, say I wanted to remove my name from poem.txt file. I open it up, delete my name, save, the file, then run:

$> git commit -am "Removed name"

But then I realize that I didn’t also delete the empty line my name was on. So I go in and delete that line, which was really part of the name-deleting work so I make a silly commit message like:

$> git commit -am "Finish removing name"

Now my history looks a little weird because I have two commits for deleting my name. There’s nothing really wrong with this except that I want my git history to be clean. In order to do that, I decide I want to put the last two commits together. So I need to do another interactive rebase, this time going back two commits, so I run:

$> git rebase -i HEAD~2

This time, when the git-rebase-todo file opens, I have the last two commits listed. This means I want to squash the most recent commit into the last one, so I change pick to squash on the second line:

On the next screen, I will pick the commit message I want to keep. I’m going to remove the Finish removing name message by commenting it out, the file will look like this:

Notice that I added the # symbol in front of the line underlined in red. From here, save and exit this file again to return to the terminal. Run git log once again and what do you notice? We’ve successfully merged the last two commits into a single commit, keeping our history clean!

There’s a ton more you can do with rebase and I encourage you to read the commented lines detailing the commands in the git-rebase-todo file. Play around with it some to get a feel for what you can do with this useful tool.

Bisect

Git’s bisect command is pretty cool because it helps us locate the source of a bug by searching for the specific commit a bug was introduced. The way it works is we first identify a “bad” commit, or a commit in which a bug exists, then we identify a commit in which the bug does not exist, or a “good” commit. Then, git will checkout old commits between the good and bad ones and ask us if the bug still exists.

Then, by process of elimination, we eventually pinpoint the exact commit that broke introduced the problem. It might sound confusing, so let’s just get hands on with it. I’m going to give you another list of terminal commands to run. One of which will intentionally introduce a bug, which we’ll then pinpoint with the bisect tool. Run the following commands:

echo "This poem is about JavaScript" >> poem.txt
git commit -am "Add description of poem"
sed -i -e 's/yellow/orange/g' poem.txt
git commit -am "Replace 'yellow' with 'orange'"
echo "This is for my git tutorial" >> poem.txt
git commit -am "Add reason for poem"

Now, take a look at the contents of poem.txt and note the bug, “yellow” seems to have been replaced with “orange.” See for yourself:

$> cat poem.txt
Roses are red
JavaScript is <strong>orange</strong>
I'm learning git
and I wanna say hello!
This poem is about JavaScript
This is for my git tutorial

We’re now going to use bisect to identify the commit that introduced this bug. The first thing we have to do is identify a good commit–where the bug doesn’t exist–and a bad commit–where the bug does exist. Run git log to find these commits. Please note that your commit hashes will not match mine, so for the rest of the article, you cannot copy/paste my example commands, you’ll have to use your relevant commit hashes.

My git log looks like this:

Since I know that the bug currently exists, I’m going to call my most recent commit (b33c81d) the bad one, and I know that the second line still said “yellow” when I signed my name, so I’m going to call that commit (00b7e71) the good one.

Now, I need to start the bisect wizard by running git bisect start, then identify the good and bad commits with git bisect good [hash] and git bisect bad [hash]. Take a look:

Git then picks a commit between the good and bad ones. From here, we’re supposed to check and see if the bug still exists, if it does, we tell it with bad if not, we use good. So check the file for the bug:

$> cat poem.txt
Roses are red
JavaScript is yellow
I'm learning git
and I wanna say hello!
This poem is about JavaScript

The word “yellow” is still there, so we run git bisect good. Git then picks another commit between this one and the commit we marked as bad. Take a look:

When we check the contents of the poem.txt file, we see that we called JavaScript “orange” instead of “yellow.” So we mark this one as bad with git bisect bad. With this information, git now believes it’s found the commit that introduced the bug. It correctly identifies the commit in which we replaced “yellow” with “orange” (for me, commit d5ebcc3) as the offending commit.

We’ll now exit the bisect wizard by running git bisect reset which returns us back to the most recent commit. Now that we know which commit introduced the “yellow -> orange” bug, we’ll use rebase to get rid of it.

Using Rebase to Fix Bug

Since we know the commit that introduced the bug, we should check and see what was going on with that commit. We can compare the commit with it’s previous commit to see what changes were made by using the git diff command and passing in two hashes.

For me, the hash for the bad commit was d5ebcc3. If I want to reference the commit before that without looking up its hash, I can append ~1 to the hash as something of a relative way of referencing the previous commit. In other words, if I want to compare d5ebcc3 and the commit prior to it with diff, I would run the following command:

$> git diff d5ebcc3<strong>~1</strong> d5ebcc3

The output then shows me what changed in that commit, see my screenshot:

Since I can see from this diff that nothing other than changing the word “yellow” to “orange” happened in this commit, I can safely assume that the whole commit can be removed from the project’s git history. I will use rebase to expunge any record of that commit ever taking place. To do this, I need to do another interactive rebase, this time to the commit before d5ebcc3. In order to this, I run the following command:

$> git rebase -i d5ebcc3~1

This brings us back to git-rebase-todo file we saw earlier. This time, I’m going to drop the offending commit. Not only will that remove that commit’s message from the project’s history, it’ll also get rid of the changes made in that commit. Make your file look like mine (but don’t change the commit hashes):

Save and exit the file and you’ll be returned to the terminal with the message “Successfully rebased and updated refs/heads/main.” First, let’s make sure the bug is fixed. cat the contents of poem.txt and make sure JavaScript is yellow again:

$> cat poem.txt
Roses are red
JavaScript is yellow
I'm learning git
and I wanna say hello!
This poem is about JavaScript
This is for my git tutorial

Great! The bug has been fixed, now run git log and note that the offending commit has been stricken from the record:

And now we know how to use bisect to locate bugs and rebase to get rid of them!

Conclusion

That concludes part 3 of the Professional Version Control with Git series. In part 1, we learned the basics of using git for our own projects, then in part 2 we learned about the tools for collaboration. Finally, in this part, we learned how to use rebase to keep our history clean, and bisect to help us track down bugs. That concludes the Professional Version Control with Git series, but keep an eye out for more content about version control and collaboration tools. Don’t hesitate to contact me with any questions you have by emailing ewhiting@erikscode.space or hitting me up on Twitter, @ErikWhiting4. Thanks, and see you next time!

Professional Version Control with Git: Pt 2 - Collaboration

Erik — Sat, 10 Jul 2021 19:38:08 +0000

Part 1 - The Basics
Part 3 - Rebase and Bisect

Welcome back, this article is part 2 of Erik’s Code Space’s series on professional version control with Git. In part one, we learned about the basics of making commits, branching, and merging. In this section, we’re going to learn about the collaboration tools available to us with Git through GitHub. Let’s get started!

In this part of the Git series, we’re going to learn about the tools that exist to help us collaborate with other developers. We’re going to learn about three general concepts: publishing, cloning , and making pull requests.

Publishing is when we put our own projects out into the world to be viewed, downloaded, and tweaked by anyone. This is what people mean when they tell you to have your own side projects up for potential employers to see.
Cloning is when you copy someone else’s project onto your own computer so that you can work on or use it. In Git, “cloning” is almost synonymous with “downloading.” Instead of downloading binaries of a program though, we get the actual source code.
Pull Requests (sometimes called Merge Requests) are what you create when you’ve made changes to a project and you want to merge it into the main project. Recall that in part one we learned that merging meant adding events to the true history of a project. When you collaborate with other developers, your changes typically need to be reviewed before they’re merged into a project.

Setting Up Your GitHub Profile

If you haven’t already, you should head over to GitHub and sign up; don’t worry, it’s free. Signing up should be pretty straightforward basic stuff. You don’t have to worry about stuff like your About Me, profile picture, and other stuff if you don’t want to. Just finish setting up and get to your homepage.

Once you’ve set up your profile, we need to get your local Git configuration aligned with your profile information. Open up your command line of choice so we can start on this. The commands you should run are the following:

$> git config --global user.name "username"
$> git config --global user.email "your_eamil@example.com

The next thing we need to do is get SSH enabled. Personally, I hate this step and any time I have to redo this step, I have to go and relearn to do it. The official docs for doing this are really great if you want a much more precise guide to it than I’m about to give you, they can be found here.

Generating an SSH Key

If you’ve decided to stick with me, the first thing we have to do is generate an SSH key. If you’re on Linux, you should already have the ability to do this from the terminal with ssh-keygen. If you’re on Windows, the best way to do this is with Git Bash, which should have come with your installation of Git. If not, go back to the download page and download it again. You can find Git Bash by typing bash into your search box and clicking this icon:

From the Bash terminal, run ssh-keygen -t rsa -b 4096 -C "your_email@example.com". The resulting prompt will ask you what path and filename to save the key into, put something like “~/.ssh/gittutorial” or something you’ll be able to remember. The next question will ask you to make a passphrase; this is optional and is just for providing a bit of extra security. You can just press enter to go with no passphrase.

Now we need to add our key to the ssh-agent. To be honest with you, I don’t really understand the point of this step, but I know it’s necessary, so let’s power through it together. First, you need to make sure the ssh-agent is running. It probably is but just in case, run the following command:

$> eval `ssh-agent -s`

Note that the characters around ssh-agent are backticks (Shift + ~). If after running this command you see something like “Agent pid 5278”, you’re good to go. If not, please refer to this article on the GitHub docs.

Once we know the agent is running, we simply add our newly generated key with ssh-add ~/.ssh/gittutorial. You will be prompted again for your passphrase, just retype it or press enter if you didn’t make one.

Adding SSH Key to GitHub Account

Now we need to add your public SSH key to the GitHub profile. On GitHub, click your profile thumbnail in the top right of the screen and click “settings.” From here, go to “SSH and GPG keys” in the left menu. Click “New SSH key” at the top. In the next screen, give your SSH key a title. I typically name the keys after whatever computer that key is for.

Now, head back to your terminal and run cat ~/.ssh/gittutorial.pub and copy the output (if you have a program called clip installed, you can run clip < ~/.ssh/gittutorial.pub). Back in your browser, paste the contents into the “key” field. Check carefully for an errant line break, if your key ends with your email, but your cursor is on the next line, hit backspace to get rid of that character.

Click “Add SSH Key” when you’re done and that should be the end of that. Everything we just did is so that we can clone and push repositories remotely. Getting this out of the way will save us some time for the rest of this article.

Make a New Repository

Now that our profile is ready to go, it’s time to make a new repository. As a reminder, a “repository” is another name for a project’s source code. In part one, we likened it to a company with our commits being the company’s timeline. The “repository” is where all our code will live. From now on, I will call the repository the “repo.”

From the GitHub homepage, in the left menu, you should see a button that says “New” with a book icon. Click it! The next screen will ask you to name the repo, I’m going to go with “ecs-git-tutorial.”

The next selection is the choice to make the repo public or private. I typically keep my repositories private until they’re ready to be used or viewed, but I will keep this one public and I invite you to do the same. Finally, click the option about generating a README and then click “Create repository.”

The next screen will be pretty boring, it’s the home page and README of your mostly empty repository. You’ve officially created a repository on GitHub. It may be a small step but congratulations anyway!

Cloning The Repository

As I mentioned earlier, cloning a repository means to copy it’s contents into your local computer. This is a fairly straightforward process, so let’s jump right into it.

On your repo’s main page, there should be a green button that says “Code” with a down arrow. Click it and select the SSH option. Copy the address in the input bar by clicking the clipboard icon on the right of the Clone pop up.

Go back to your command prompt and type git clone git@github.com:user/project replacing the part after “clone” with the thing you just copied. For example, for me to clone the repository I just created, I would run:

$> git clone git@github.com:erik-whiting/ecs-git-tutorial.git

Yours will be slightly different. After you hit enter, you’ll see a few outputs and finally you’ll get your prompt back. Run dir or ls to see the contents and notice that there’s now a folder called ecs-git-tutorial. Go ahead and cd into it and run ls again. Notice that the README is in there. That’s the same README you see on your repository’s homepage on GitHub. We’ve successfully cloned the repo onto our local machine!

Now that you know how to do this, you know how to clone just about any project that’s maintained on GitHub. For example, one of the most popular repos for GitHub is rich, a project that gives you “rich text and beautiful formatting in the terminal.” If you wanted to look through the code of that project on your own IDE, you know how to do that now: with git clone.

Making Changes and Pushing

Most of the time you clone a project from GitHub, it’s so that you can make your own contributions and push them. Let’s do that right now to get a feel for it. Make a new file called “tutorial.txt” and put something simple in there like:

Hi, I'm a text file

Add the file to the repository with git add tutorial.txt and make a simple commit, something like git commit -m "First commit". Now we’ve got this tutorial.txt file and commit history on our local computer only. We now want to publish our changes by pushing them. To push your changes to the repo, run the following command:

$> git push

Once again, you’ll see a few outputs in the terminal. If everything went according to plan, we just successfully pushed the tutorial.txt file to GitHub. Go back to your browser and refresh your repo’s homepage. You should now see a new file called “tutorial.txt” with “First commit” and how long ago you committed the file. mine looks like this:

Pretty cool right? We’ve gotten the changes we made on our local machine up to GitHub where anyone can see them. This means anyone with access to GitHub can now clone our work and collaborate with us!

Typically, pushing changes directly to the main branch is considered very bad practice. We generally use a workflow involving “pull requests,” which we’ll learn about next.

Creating the Pull Request

Most of the time when we collaborate on projects with multiple developers, or even when we’re working on our own projects, we don’t want to do work on the main branch. Instead, we follow a workflow that involves creating a new branch locally, doing our work, publishing the branch in the repository, and creating a pull request for it. Let’s go through this workflow together.

From your terminal, create a new branch called “edit-tutorial.txt.” If you forgot how to do that, the following command will create and checkout the branch:

$> git checkout -b edit-tutorial.txt

Now, open up the tutorial.txt file and add another line to it, something like “This line was added from the edit-tutorial.txt branch.” From your terminal, run git status to make sure you see that tutorial.txt has been modified. Go ahead and commit these changes with:

$> git commit -am "Add line to tutorial.txt"

After committing, run git push. Notice that we get an error that says

fatal: The current branch edit-tutorial.txt has no upstream branch. To push the current branch and set the remote as upstream, use

git push –set-upstream origin edit-tutorial.txt

What happened here? Well, we tried to publish our changes to a branch that doesn’t exist upstream, or in our repo on GitHub. Right now, the only branch on GitHub should be main. We have to create the branch on GitHub in order to publish our local branch. Doing so is as easy as running the command suggested by the error message though, so simply run:

$> git push --set-upstream origin edit-tutorial.txt

The terminal will again output some information. Once it’s done, go back to the repo’s homepage. You should see a message about a branch having new pushes with a button to “Compare & pull request” like this:

Just in case you don’t see that though, you can click “Pull Requests” in the menu, and click “New pull request” in the next screen. In the next screen, you’d have to change the “compare” dropdown to “edit-tutorial.txt” and click “Create pull request” to create the pull request or PR.

In the next screen, you’re given the option to title and write a comment about the PR. The default title is the last commit message, but I typically like to name them something more official sounding. I also like to add a little description to the PR, but it’s not required. Here’s what my PR looks like:

Now, before finishing the creation of the PR, scroll down on this page and notice that you can now see the incoming changes. I prefer “Split” view to “Unified” (those options are in the top right of the screen relative to the file comparison section of the screen). On the left you’ll see the code that exists in the main branch, on the right you’ll see the changes made highlighted in green (deletions would be highlighted in red). This is usually a good time to look over the changes you’ve made to make sure you didn’t miss anything important.

Go ahead and click the “Create pull request” button to finish creating the PR. You’ll then be taken to the homepage of the PR

Merging the Pull Request

Typically, the repository owner will be doing this next step. From the pull request’s page, the reviewer can look at the changes made in your branch. They can leave comments, tag your username with the @ symbol to notify you, and even comment directly on lines of code.

Let’s pretend we’re the reviewers of this PR. We look through the proposed changes and we decide everything is good to go. The next step is to click the “Merge pull request” button, which should be enabled because there should not be any conflicts with the base branch. If there were conflicts, the button would be disabled.

After clicking the “Merge pull request” button, it’ll change to “Confirm merge.” Click it again to confirm and note the new message “Pull request successfully merged and closed.” We can click the “Revert” button if we need to undo this merge, but since everything is fine, let’s go ahead and delete the branch.

**Note: Some teams prefer to delete branches once they’ve been merged, others don’t. Make sure to always adhere to the repository’s norms.

Go back to your repo’s homepage (by clicking “ecs-git-tutorial” in the top right) and click the tutorial.txt file. You should now see the line “This line was added from the edit-tutorial.txt branch” we just added. We’ve successfully merged our changes!

Resync With Main

Back in your terminal, checkout the main branch again by running:

$> git checkout main

You’ll get a message that says:

Switched to branch 'main'
Your branch is up to date with 'origin/main'

but that’s not true. After checking out main, open tutorial.txt in a text editor. What do you notice? You’ll see that the line we added in the PR isn’t in our local copy of main. What gives?

Well, we actually just need to sync our local main branch with the remote main branch on GitHub. First, we can sync our data by running git fetch which will update our local repository with notifications of any changes made to remote (NOTE! The fetch command does not update your local branches, it only updates your repository with current information about the repo like PRs made or if your branches are behind).

Run git status and you should now see a message about being behind by a few commits. Mine looks like this after running the fetch command:

$> git status
On branch main
Your branch is behind by 'origin/main' by 2 commits, and can be fast-forwarded.
  (use "git pull" to update your local branch)

nothing to commit, working tree clean

To pull the changes from the merged PR onto our local main branch, we simply have to run git pull. Run that in your terminal and open the tutorial.txt file again. You should notice this time that the “This line was added from the edit-tutorial.txt branch” line is in there now.

And that concludes our lesson on merging pull requests!

Conclusion

This concludes part two of our professional git series. We now know how to initialize a repository, make and commit changes, make new branches, publish on GitHub, and create pull requests. That’s a ton of useful information and if your next job is anything like mine, this is stuff you will do multiple times a day, so it’s good you know how to do it now.

Also, this is all the information you need to know in order to participate in open source software (OSS) on GitHub. You now have all the tools you need to start contributing to public projects (if you want to, that is)!

In the next and final part of this series, we will learn rebasing and bisecting, two git commands that are not so common, but are super useful when you need them. Until next time, let me know if you have any questions by emailing me at ewhiting@erikscode.space, or tweet at me @ErikWhiting4 (I just got a new Twitter, give me a follow!)