Tag: software testing

Testing Sorely Needed

I’m working on an implementation of the dgemm matrix multiply function for the BLAS interface [1] for a Parallel Computation course. While trying to introduce a feature to take advantage of the processor’s cache hierarchy, I ran into a reoccurring issue in my software development career. This time it bothered me more than usual.

In order to complete the feature, four components would need to be added or changed for the program to function correctly. For someone who isn’t habituated to test their code after incremental changes, four changes is basically nothing. “Watch me perform dependency inversion while thrashing every class in the codebase,” says the programmer who had it coming. [2] But for someone who now expects more from himself as a software developer, this was now uncomfortable.

“If I think this through, map it out on paper, and act cautiously, I can probably get it on my first try,” is what I use to say. I knew better this time, and I was correct. It didn’t work the first time.

The problem isn’t that it didn’t work the first time. The problem was that I had made too many changes at once and now it was impossible to pinpoint which piece was failing just by looking at the result of my Frankenstein-memory-access-matrix-multiply.

Three of the four components were mostly self-contained and should have been unit tested. These were also the most complex changes, so they DEFINITELY should have been unit tested. What were the two obstacles that prevented me from unit testing this code?

First, was inexperience. I should own up to my own shortcomings before criticizing the code of other people. Even though I’ve been using C/C++ longer than any other language, there’s a lot I still need to learn. Just in this codebase alone: make files, extern ‘C’ code, and static inline functions [3] all showed me how little I truly know. Furthermore, I’ve never properly tested C/C++ software before with unit and integration testing. So there was no way I was going to slap a testing framework into this codebase without first receiving many well deserved battle scars.

Second, was code structure. Two things seem almost inescapable in software.

  1. If you don’t start a program with the intention of testing, as the program grows, it will probably get too difficult to ever test.
  2. Even when you start out testing a codebase, as the project grows and becomes more complex, it will eventually also get too difficult to test.

This code is strictly academic, for learning purposes only. They thankfully included some handy debugging functionality to the project, but its creators didn’t imagine people would need to properly test this code. From the complexity of how the code is linked together to the static inline functions that make some important components impossible to access, this is not convenient to test.

I write this the day of my well learned lesson. In the coming weeks, I hope to learn a number of things about software testing in general, testing C/C++, and understanding some of the internals that have given me so much grief today.

References:

  1. http://www.netlib.org/blas/
  2. Yea, I’ve done that. Not proud of it. I got what I deserved.
  3. Static inline functions actually made a lot of sense in this context since we’re trying to optimize the code to be wicked fast. Why declare a C function as static inline?

First Real Software Testing Experience

In the spring of 2020, I participated in a graduate software engineering course focused on the software principles of modularity. We read many canonical articles and put their messages into practice by refactoring an Android mobile game. My team and I implemented very few automated tests in our test suite, mostly because we spent so much time attempting to refactor the system. But when we did make tests, they became outdated in a week’s time because of how rapidly we were making changes.

This summer, I set my sights on building a website for Toastmasters International using Python and Django. The tutorial I used to learn Django was Vitor Freitas’s. Vitor’s dedication to sharing knowledge with others is astounding and his Django tutorial is phenomenal. What surprised me most throughout the course was his emphasis on testing. Looking back, the tutorial was as much an intro to software testing as it was an intro to Django.

Once my project diverted from the tutorial material, most of my testing involved copying and pasting test classes to fit all my new views. Later, I learned that I could test whole models. And I was most proud of how I tested user permissions for some of the views.

Though there wasn’t much to my test suites, I had gotten far enough along to develop an eye for what could be improved in my system:

First, I was repeating myself a lot! I’m sure the Don’t Repeat Yourself (DRY) principle also applies to test suites. I attempted to utilize the “setup” method of the Django TestCase class as best I could, and even used some inheritance to make the concrete test classes less repetitive. Still, I couldn’t shake the feeling that there wasn’t an “authoritative and unambiguous representation” [1] for components of my test suite. The finest example was user permission tests. Instead of directly testing the mixins that I created by inheriting from UserPassesTestMixin, I repeatedly tested the views that used these permission mixins.

Second, the dependency chains created by my models made it more complex to test individual components. If every Club could have Meetings, and Meetings could have Performances, and Performances could have Evaluations, then I was creating Clubs, Meetings, and Performances simply to test the Evaluation components of my system. Surely, this problem lies in a failure of mine to properly include abstractions between my components and utilize Dependency Inversion. [2]

As my first real experience in software testing, the following are just a few of my big takeaways for why I’ll be taking testing more seriously in my projects from now on.

  1. I was much more confident to make changes when I knew I had tests backing me up. With each added test, I was less worried about making mistakes because my failed test cases were my safety harness and light in my tunnel.
  2. While learning C++, I remember relying on the compiler to be my test suite. This Django project has really taught me that you can’t rely on the compiler errors to know you’ve done something wrong. The Django template language and many components within Django rely on strings. On several occasions I had features that simply weren’t working and couldn’t believe that uninstantiated variables in these string types didn’t throw helpful errors. All the more reason for better test suites.
  3. I even got into the habit of using test cases that always failed as a reminder that I hadn’t yet implemented a certain feature. With tons of links on each webpage, it was incredibly easy to forget a hyperlink here and there. Testing then became a habit that was more congruent with the David Allen “Getting Things Done” mindset I’ve been trying to develop. I let my testing suite be my reminder of unfinished tasks so I could free up space in my head for the task at hand.

You can find more info about my Toastmasters Feedback project here.

References:

  1. Orthogonality and the DRY Principle, A Conversation with Andy Hunt and Dave Thomas, Part II by Bill Venners, March 2003.
  2. Robert C. Martin, “The Dependency Inversion Principle”, C++ Report, May 1996.