Category Archives: Code

Bug Hunting Adventures #15: Hex String Entanglements

“Don’t be drawn into any web of entanglement created by others.”
― Steven Redhead, Life Is Simply A Game

The following routine is from a real-world project. It’s supposed to convert binary data into a printable C-string of hexadecimal digits. Even though the developer diligently wrote some unit tests, he got complaints from his fellow coders a few days later. Can you find what’s wrong with it?


const char*
to_hexstring(const void* data, size_t data_len, char* out, size_t out_len) {
    if (data == NULL || data_len == 0 || out == NULL || out_len == 0) {
        return NULL;
    }
    //  Only proceed if output buffer is large enough:
    //  Two chars per byte + trailing string terminator.
    if (out_len < (data_len * 2) + 1) {
        return NULL;
    }
    const char* p = (const char*) data;
    for (size_t i = 0; i < data_len; ++i) {
        char hexbyte[2 + 1];
        snprintf(hexbyte, sizeof(hexbyte), "%02X", *p++);
        strcat(out, hexbyte);
    }
    return out;
}

const char*

to_hexstring(const void* data, size_t data_len, char* out, size_t out_len) {

if (data == NULL || data_len == 0 || out == NULL || out_len == 0) {

return NULL;

}

// Only proceed if output buffer is large enough:

// Two chars per byte + trailing string terminator.

if (out_len < (data_len * 2) + 1) {

return NULL;

}

const char* p = (const char*) data;

for (size_t i = 0; i < data_len; ++i) {

char hexbyte[2 + 1];

snprintf(hexbyte, sizeof(hexbyte), "%02X", *p++);

strcat(out, hexbyte);

}

return out;

}

Unit tests:


TEST(to_hexstring, some_tests) {
    const unsigned char TEST_DATA[] = { 0x00, 0x11, 0x22, 0x33, 0x1F };
    char hexstring[sizeof(TEST_DATA) * 2 + 1];

    // Typical case, output buffer large enough.
    hexstring[0] = '\0';
    EXPECT_STREQ("001122331F", to_hexstring(TEST_DATA, sizeof(TEST_DATA), 
        hexstring, sizeof(hexstring)));

    // Convert just a single byte.
    hexstring[0] = '\0';
    EXPECT_STREQ("00", to_hexstring(TEST_DATA, 1, hexstring,
        sizeof(hexstring)));

    // Convert two bytes.
    hexstring[0] = '\0';
    EXPECT_STREQ("0011", to_hexstring(TEST_DATA, 2, hexstring,
        sizeof(hexstring)));

    // Error: output buffer too small.
    hexstring[0] = '\0';
    EXPECT_STREQ(NULL, to_hexstring(TEST_DATA, sizeof(TEST_DATA),
        hexstring, 1));

    // Error: output buffer too small.
    hexstring[0] = '\0';
    EXPECT_STREQ(NULL, to_hexstring(TEST_DATA, 1, hexstring, 1));

    // Error: output buffer still too small.
    hexstring[0] = '\0';
    EXPECT_STREQ(NULL, to_hexstring(TEST_DATA, 1, hexstring, 1 + 1));
}

TEST(to_hexstring, some_tests) {

const unsigned char TEST_DATA[] = { 0x00, 0x11, 0x22, 0x33, 0x1F };

char hexstring[sizeof(TEST_DATA) * 2 + 1];

// Typical case, output buffer large enough.

hexstring[0] = '\0';

EXPECT_STREQ("001122331F", to_hexstring(TEST_DATA, sizeof(TEST_DATA),

hexstring, sizeof(hexstring)));

// Convert just a single byte.

hexstring[0] = '\0';

EXPECT_STREQ("00", to_hexstring(TEST_DATA, 1, hexstring,

sizeof(hexstring)));

// Convert two bytes.

hexstring[0] = '\0';

EXPECT_STREQ("0011", to_hexstring(TEST_DATA, 2, hexstring,

sizeof(hexstring)));

// Error: output buffer too small.

hexstring[0] = '\0';

EXPECT_STREQ(NULL, to_hexstring(TEST_DATA, sizeof(TEST_DATA),

hexstring, 1));

// Error: output buffer too small.

hexstring[0] = '\0';

EXPECT_STREQ(NULL, to_hexstring(TEST_DATA, 1, hexstring, 1));

// Error: output buffer still too small.

hexstring[0] = '\0';

EXPECT_STREQ(NULL, to_hexstring(TEST_DATA, 1, hexstring, 1 + 1));

}

Solution

Circular Adventures VIII: The Eternal Quest For Mod-Like Behavior

“A circle is a round straight line
with a hole in the middle.”
— Mark Twain

In circular situations, we often wish to employ the mod operator to achieve “wrap-around” at both ends of a linear system to mimic a circular system:


0 1 2 3 4 5 6 7 (n = 8)
          ^
          i

0 1 2 3 4 5 6 7 (n = 8)

Achieving clockwise wrap-around (from n to 0) usually poses no problem:


size_t circular_increment(size_t index, size_t n) {
    return (index + 1) % n;
}

size_t circular_increment(size_t index, size_t n) {

return (index + 1) % n;

}

The % operator is the compiler vendor’s approximation of the mod operator. It ensures that values which exceed n wrap-around nicely:


assert(circular_increment(5, 8) == 6);
assert(circular_increment(6, 8) == 7);
assert(circular_increment(7, 8) == 0);
assert(circular_increment(0, 8) == 1);
:

assert(circular_increment(5, 8) == 6);

assert(circular_increment(6, 8) == 7);

assert(circular_increment(7, 8) == 0);

assert(circular_increment(0, 8) == 1);

Alas, most operator % implementations are not so useful when it comes to counter-clockwise wrap-around (from 0 to n):


size_t circular_decrement(size_t index, size_t n) {
    return (index - 1) % n;
}

size_t circular_decrement(size_t index, size_t n) {

return (index - 1) % n;

}

If index – 1 becomes negative, we likely get negative results — which is far from what we desire:


assert(circular_decrement(2, 8) == 1);
assert(circular_decrement(1, 8) == 0);
assert(circular_decrement(0, 8) == 7); // Fail! -1 returned instead of 7.

assert(circular_decrement(2, 8) == 1);

assert(circular_decrement(1, 8) == 0);

assert(circular_decrement(0, 8) == 7); // Fail! -1 returned instead of 7.

To readers of this column (part I in particular) this is of course old hat. They know that you can’t trust the % operator when negative operands are involved. Circular coders often work around this shortcoming like so:


size_t circular_decrement(size_t index, size_t n) {
    if (index == 0)
        return n - 1;
    else
        return index - 1;
}

size_t circular_decrement(size_t index, size_t n) {

if (index == 0)

return n - 1;

else

return index - 1;

}

Not nice, but no big deal either, you probably think. But what if you need a function that advances an index in either direction, depending on the sign of the offset:


size_t circular_advance(size_t index, int offset, size_t n) {
    size_t new_index;
    if (offset >= 0) {
        new_index = (index + offset) % n;
    } else {
        offset = -offset;
        if (index >= offset) {
            new_index = index - offset;
        } else {
            new_index = n - offset + index;
        }
    }
    return new_index;
}

size_t circular_advance(size_t index, int offset, size_t n) {

size_t new_index;

if (offset >= 0) {

new_index = (index + offset) % n;

} else {

offset = -offset;

if (index >= offset) {

new_index = index - offset;

} else {

new_index = n - offset + index;

}

return new_index;

}

Now this looks disgusting, this is a big deal! And all just because our wretched % operator doesn’t behave like it should. Remember, if it did, we could simply write:


size_t circular_advance(size_t index, int offset, size_t n) {
    return (index + offset) % n;  // if only '%' was 'mod'... sigh!
}

size_t circular_advance(size_t index, int offset, size_t n) {

return (index + offset) % n; // if only '%' was 'mod'... sigh!

}

Did you notice that our convoluted circular_advance function actually contains a bug? No? Here’s a hint. The bug is in this line:


new_index = n - offset + index;

new_index = n - offset + index;

What happens if the offset is larger than n? Answer: the expression n – offset will underflow producing a wrong result. That’s why we would have to write this, at least if we wanted to be 100% correct:


new_index = n - (offset % n) + index;

new_index = n - (offset % n) + index;

which makes the code even uglier. But do we really need to support such large offsets? If we don’t (which is almost always the case in circular scenarios) and can live with offsets whose value is less than or equal to n, we can simplify circular_advance significantly. Here’s how.

We know that adding a multiple of n to the dividend x doesn’t change the result of the modulo operation:


x mod n ≡ (x + i*n) mod n

x mod n ≡ (x + i*n) mod n

If we include an arbitrary offset, we get this:


(x + offset) mod n ≡ (x + offset + i*n) mod n

(x + offset) mod n ≡ (x + offset + i*n) mod n

This allows us to turn negative arguments to operator % into positive arguments by adding i*n, without changing the result. Further, if we ignore cases where the offset value is greater than n, we only have to add 1*n to enforce a positive argument to operator %. Here’s an example:


(1 - 3) mod 8 ≡ (1 - 3 + 8) mod 8
-2 mod 8 ≡ 6 mod 8
6 ≡ 6

(1 - 3) mod 8 ≡ (1 - 3 + 8) mod 8

-2 mod 8 ≡ 6 mod 8

6 ≡ 6

Since all operands are positive, we can use 6 % 8 to compute -2 mod 8. Equipped with this knowledge, we can eradicate the ugliness and circular_advance boils down to this:


size_t circular_advance(size_t index, int offset, size_t n) {
    return (index + offset + n) % n;
}

size_t circular_advance(size_t index, int offset, size_t n) {

return (index + offset + n) % n;

}

If we apply the same technique to the ring buffer’s original ring_buffer::size method from the last installment:


size_t size() const {
    return head_ >= tail_ ?
        head_ - tail_ :
        BUFSIZE - (tail_ - head_);
}

size_t size() const {

return head_ >= tail_ ?

head_ - tail_ :

BUFSIZE - (tail_ - head_);

}

we get a simple branch-free solution:


size_t size() const {
    return (head_ - tail_ + BUFSIZE) % BUFSIZE;
}

size_t size() const {

return (head_ - tail_ + BUFSIZE) % BUFSIZE;

}

which can be optimized easily by the compiler, especially if BUFSIZE is a base-2 number.

So here’s the takeaway. To get efficient, branch-free mod-like behavior for positive divisors n:

If n is a base-2 number, use bitwise-AND instead of the % operator:


a mod n ≡ a & (n - 1)

a mod n ≡ a & (n - 1)

If there’s only clockwise wrap-around (ie. a >= 0), % works just fine


a mod n ≡ a % n

a mod n ≡ a % n

If a is either positive or negative, but |a| is <= n, add n to the operand before applying the % operator:


a mod n ≡ (a + n) % n

a mod n ≡ (a + n) % n

More circular adventures…

Let’s Unit-Test Everything!

“More than the act of testing, the act of designing tests is one of the best bug preventers known. The thinking that must be done to create a useful test can discover and eliminate bugs before they are coded – indeed, test-design thinking can discover and eliminate bugs at every stage in the creation of software, from conception to specification, to design, coding and the rest.”
– Boris Beizer

Some programmers are definitely hard to convince that unit testing is a good idea. Especially die-hard embedded C coders go to great lengths to “prove” that for their project, unit testing is unthinkable.

One frequently presented argument against unit testing is that the project is real-time and/or deeply embedded and that the code is so low-level and hardware-dependent that it’s impossible to unit test it.

To me, this argument doesn’t hold water. You just need to introduce the right abstractions. Obviously, you can’t unit-test the precision of a timer with a one microsecond resolution, still, you can test against its (mocked) interface. And you can definitely test the effects of an expired timer, when an “timer expired” callback method is executed. It’s just a matter of isolating hardware (or in general, things that are difficult to unit test) from business logic. If the abstractions are right, unit testing is not just possible, it’s a breeze! As a bonus, you usually get lots of portability and reuse for free.

Another excuse for not unit testing is lack of time. “We can’t do unit testing because our schedule is so tight. Boohoo!”. That’s a fake lament, too. Granted, implementation might initially take longer, but implementation isn’t everything: software integration and system testing contribute major parts to the overall schedule, and both phases can be significantly reduced by employing unit testing, as many studies have demonstrated (refer to Roy Osherove’s book “The Art of Unit Testing, 2nd Edition”, for instance). The later you find a bug, the more expensive it is to fix. And while unit testing obviously can’t find all bugs it finds a lot of bugs and it finds them early in the development life-cycle.

In order to lead by example, I’m going to do something outrageous: I’m going to unit-test one of the most simple pieces of software ever written, the famous “Hello, World!” program. Even though it might sound crazy, I chose “Hello, World” because most developers would think that it’s a) impossible to unit-test and b) hard to get wrong in the first place.

Let’s start by taking a look at the venerable implementation by Kernighan and Ritchie as shown in their famous “The C Programming Language” book:


#include <stdio.h>

main()
{
    printf("hello, world\n");
}

#include <stdio.h>

main()

{

printf("hello, world\n");

}

Actually, “Hello World” does two things: apart from printing the famous words, it (implicitly) returns an exit code of zero to the operating system. My first refactoring makes the original version’s behavior more explicit:


int main() {
    printf("hello, world\n");
    return 0;
}

int main() {

printf("hello, world\n");

return 0;

}

Since calling main from a unit test is not possible, let’s do a another quick modification and encapsulate the “Hello World” functionality in a function of its own:


int hello_world() {
    printf("hello, world\n");
    return 0;
}

int main() {
    return hello_world();
}

int hello_world() {

printf("hello, world\n");

return 0;

}

int main() {

return hello_world();

}

Now, if this isn’t an example of “Design for Testability”, I don’t know what is ;-) Contrary to the original code, this new design allows calling/observing the behavior from a unit test. While verifying the return code is easy, ensuring that the right string is printed is a bit more involved and requires some creativity.

My initial idea was to redirect stdout to a file during the test and later check if the file contents are what I expect:


const char* const TEST_FILE = "/tmp/test_hello_world_output.txt";
if (freopen(TEST_FILE, "w", stdout) != NULL) {
    int exit_code = hello_world();
    fclose(stdout);

    assert(exit_code == 0);
    assert(file_content_matches_string(TEST_FILE, "hello, world"));
}

const char* const TEST_FILE = "/tmp/test_hello_world_output.txt";

if (freopen(TEST_FILE, "w", stdout) != NULL) {

int exit_code = hello_world();

fclose(stdout);

assert(exit_code == 0);

assert(file_content_matches_string(TEST_FILE, "hello, world"));

}

Even though this approach works, I rather prefer the more traditional mock-based approach, which uses the preprocessor to redirect calls to printf to a mocked version of printf:


#define printf mock_printf

static size_t mock_printf_calls = 0;
static const char* mock_printf_arg1 = NULL;
static int mock_printf_return = 0;

void mock_printf_reset() {
    mock_printf_calls = 0;
    mock_printf_arg1 = NULL;
    mock_printf_return = 0;
}

int mock_printf(const char *__restrict text, ...) {
    ++mock_printf_calls;
    mock_printf_arg1 = text;
    return mock_printf_return;
}

#define printf mock_printf

static size_t mock_printf_calls = 0;

static const char* mock_printf_arg1 = NULL;

static int mock_printf_return = 0;

void mock_printf_reset() {

mock_printf_calls = 0;

mock_printf_arg1 = NULL;

mock_printf_return = 0;

}

int mock_printf(const char *__restrict text, ...) {

++mock_printf_calls;

mock_printf_arg1 = text;

return mock_printf_return;

}

The test itself is straightforward:


mock_printf_reset();
int ret = hello_world();
assert(ret == 0);
assert(mock_printf_calls == 1);
assert(strcmp(mock_printf_arg1, "hello, world!\n") == 0);

mock_printf_reset();

int ret = hello_world();

assert(ret == 0);

assert(mock_printf_calls == 1);

assert(strcmp(mock_printf_arg1, "hello, world!\n") == 0);

As you can see, it’s quite simple to create your own mocks, even in plain old C, where no fancy mocking frameworks are available.

But of course, unit testing is impossible if your insist on writing monolithic code with functions comprising 100+ lines that directly do I/O or manipulate device registers. It’s also impossible if you prefer delving through bug reports from field testing when you instead could have found and fixed that off-by-one error from the comfort of your home office, ten minutes after you introduced it.

Unit testing is no panacea, it’s by no means a substitute for system testing — it’s yet another rung on the software quality ladder. Let’s be honest: the real reason for not doing unit testing is not a technical one — it’s pure laziness.

(You can find the source code, tests and makefiles here.)

Simplified III: The Goat Problem

“The solution often turns out more beautiful than the puzzle.”
— Richard Dawkins

The goat problem, or Monty Hall problem, named after the host of the game show “Let’s Make a Deal” confuses the heck out of people. Here’s the original formulation from Marilyn vos Savant:

Suppose you’re on a game show, and you’re given the choice of three doors: Behind one door is a car; behind the others, goats. You pick a door, say No. 1, and the host, who knows what’s behind the doors, opens another door, say No. 3, which has a goat. He then says to you, “Do you want to pick door No. 2?” Is it to your advantage to switch your choice?

As counter-intuitive as it might seem, the correct answer is — yes, it is!

Everyone (unless they knew this problem already) including software developers and physicists that I confronted with this answer, shook their heads in disbelieve. They are, however, in good company. According to this Wikipedia article:

Many readers of vos Savant’s column refused to believe switching is beneficial despite her explanation. After the problem appeared in “Parade”, approximately 10,000 readers, including nearly 1,000 with PhDs, wrote to the magazine, most of them claiming vos Savant was wrong (Tierney 1991). Even when given explanations, simulations, and formal mathematical proofs, many people still do not accept that switching is the best strategy (vos Savant 1991a). Paul Erdős, one of the most prolific mathematicians in history, remained unconvinced until he was shown a computer simulation demonstrating vos Savant’s predicted result (Vazsonyi 1999).

Fascinating, isn’t it? The problem (or rather the solution) is so hard to grasp, yet it’s so easy to follow this simplified™ explanation:

– If you initially picked a goat and you switch, you’ll win.
– If you initially picked a car and you switch, you’ll lose.
– It’s twice as likely that you initially picked a goat.
– Thus, it’s twice as likely to win when you switch!

Still not convinced? I wrote a little simulation in Python. Here’s the result after one million experiments:


$ ./goat_problem.py 1000000
games played: 1000000, games won (stayed): 33.30%, games won (switched): 66.70%

$ ./goat_problem.py 1000000

games played: 1000000, games won (stayed): 33.30%, games won (switched): 66.70%

The first number is the total number of experiments. The second number is the percentage of games won when the contestant stayed, while the third number shows the percentage of games won when the contestant switched.

Circular Adventures VII: A Ring Buffer Implementation

“The Limited Circle Is Pure”
— Franz Kafka

In systems programming, ring buffers are ubiquitous. Like regular queues, they’re first-in first-out data structures but contrary to classic queues they’re fixed in size and don’t grow dynamically. This is especially important in real-time contexts where time and space determinism is paramount.

In today’s circular episode I want to show you my go-to ring buffer implementation. I like it, because it’s short and decently efficient. I deliberately used only basic C++ syntax, so it should compile fine with even compilers from the previous millennium (i. e. C++98):


template<typename T, int N>
class ring_buffer {
public:
    ring_buffer() { clear(); }
    size_t capacity() const { return N; }
    bool empty() const { return head_ == tail_; }
    size_t size() const {
        return head_ >= tail_ ?
            head_ - tail_ :
            BUFSIZE - (tail_ - head_);
    }
    void add(const T& item) {
        buffer_[head_] = item;
        advance(head_);
        if (head_ == tail_) {
            advance(tail_); // Drop oldest entry, keep rest.
        }
    }
    const T& remove() {
        assert(!empty());
        size_t old_tail = tail_;
        advance(tail_);
        return buffer_[old_tail];
    }
    void clear() { tail_ = head_ = 0U; }

private:
    static const size_t BUFSIZE = N + 1U;
    void advance(size_t& value) { value = (value + 1) % BUFSIZE; }

    T buffer_[BUFSIZE];
    size_t head_;
    size_t tail_;
};

template<typename T, int N>

class ring_buffer {

public:

ring_buffer() { clear(); }

size_t capacity() const { return N; }

bool empty() const { return head_ == tail_; }

size_t size() const {

return head_ >= tail_ ?

head_ - tail_ :

BUFSIZE - (tail_ - head_);

}

void add(const T& item) {

buffer_[head_] = item;

advance(head_);

if (head_ == tail_) {

advance(tail_); // Drop oldest entry, keep rest.

}

const T& remove() {

assert(!empty());

size_t old_tail = tail_;

advance(tail_);

return buffer_[old_tail];

}

void clear() { tail_ = head_ = 0U; }

private:

static const size_t BUFSIZE = N + 1U;

void advance(size_t& value) { value = (value + 1) % BUFSIZE; }

T buffer_[BUFSIZE];

size_t head_;

size_t tail_;

};

You can specify the data type of ring buffer elements via template parameter T and the total number of elements to track via template parameter N, respectively. Here’s a little usage scenario:


ring_buffer<int, 10> rb;
assert(rb.capacity() == 10);

rb.add(123);
rb.add(42);
rb.add(23);
assert(rb.size() == 3);

while (!rb.empty()) {
    cout << rb.remove() << endl;
}
assert(rb.size() == 0);

ring_buffer<int, 10> rb;

assert(rb.capacity() == 10);

rb.add(123);

rb.add(42);

rb.add(23);

assert(rb.size() == 3);

while (!rb.empty()) {

cout << rb.remove() << endl;

}

assert(rb.size() == 0);

A common problem that you face when implementing a ring buffer is discriminating between empty and full states, as in both cases the head and tail index point to the same location. I solved it by allocating one extra element, which might waste a little bit of space but makes the code easier on the eye and probably faster compared to the alternative approach which requires you have to maintain an extra boolean flag.

When the buffer becomes full during an add operation (head == tail), the tail index is immediately advanced by one, thus dropping the oldest element. As this all happens within the add method, from a ring_buffer user’s point of view the head index is only ever equal to the tail index when the ring buffer is empty.

Contemporary compilers will replace the potentially costly modulo operation in the advance helper method with an AND operation for buffer sizes that are base-2 numbers. However, keep in mind that the advance method uses the internal buffer size (BUFSIZE), which is one greater than the requested ring buffer size (N), so this is most likely an efficient ring buffer:


ring_buffer<double, 15> rb;     // Internal buffer has 16 entries.
                                // index % 16 reduced to index & 15.

ring_buffer<double, 15> rb; // Internal buffer has 16 entries.

// index % 16 reduced to index & 15.

while this isn’t:


ring_buffer<double, 16> rb;     // Internal buffer with 17 entries.

ring_buffer<double, 16> rb; // Internal buffer with 17 entries.

[Update 2020-05-29: Part VIII of this series shows how to optimize/simplify ring_buffer::size().]

More circular adventures…

The Happy Path to Modern C++ (C++11 done!)

I’ve finally found time and motivation to resume work on the “Happy Path to Modern C++” project. I’m sure it now contains enough code snippets to give you a good overview of C++11. As a matter of fact, I can’t think of an easier way to learn C++11, provided you’re already well-versed in C++ and C++98, in particular.

Here’s the project on GitHub.

“Dèyè mòn, gen mòn”, as the Haitians say: “Behind mountains there are more mountains”. The next goal is adding plenty of C++14 snippets. Again, contributions are highly appreciated!

Approxion

Code – People – Everything

Category Archives: Code

Bug Hunting Adventures #15: Hex String Entanglements

Circular Adventures VIII: The Eternal Quest For Mod-Like Behavior

Let’s Unit-Test Everything!

Simplified III: The Goat Problem

Circular Adventures VII: A Ring Buffer Implementation

The Happy Path to Modern C++ (C++11 done!)