Detecting arithmetic overflow in C with NASM

by

This snippet is about detecting whether the carry flag is set or cleared. I work on Mac OSX, so my snippet supports Mac only.

First, we need a routine that does the job:

func.s:

global _read_carry_flag

_read_carry_flag:
    mov al, 0
    jnc end 
    mov al, 1
end:
    ret

(Try nasm2 -f macho64 func.s for compiling into an object file.)

main.c:

#include <stdbool.h>
#include <stdint.h>
#include <stdio.h>

#define A (2 * 1000 * 1000 * 1000)
#define B (1 * 1000 * 1000 * 1000)

extern bool read_carry_flag();

int main(int argc, char* argv[])
{
    int32_t a = A;
    int32_t b = B;
    int32_t ret = a + a + b;

    printf("%d\n", read_carry_flag());

    a = A;
    b = 1;
    ret = a + b;

    printf("%d\n", read_carry_flag());
    return 0;
}

(Try gcc -o prog main.c func.o for obtaining a process image.)

I would like to hear about possible improvements/extensions to the idea.

Answer

Wrong flag

I believe you should be looking at the overflow flag instead of the carry flag, since all of your operands are signed values. On x86, the overflow flag is set if signed addition overflows. The carry flag is set if unsigned addition overflows.

Not reliable

As @Edward pointed out, it doesn’t seem reliable to use this kind of function because you never know how the compiler is going to rearrange your code. Even without the compiler rearranging your code, the results could be confusing. From your own example:

ret = a + a + b;
overflow = read_overflow_flag();

Here, you are only detecting overflow over the second of the two additions. If the first addition overflowed but the second didn’t, you wouldn’t catch it. In other words, the assembly might look like this:

add %ecx, %edx, %edx  // ret = a + a  <- overflow not detected here
add %ecx, %ecx, %ebx  // ret = ret + b
seto %al              // overflow = overflow flag

To do it correctly, I would suggest you use a function that adds two numbers together and updates a cumulative overflow:

// If the add overflows, 1 will be added to *pOverflow.
int32_t add32_with_overflow(int32_t x, int32_t y, int *pOverflow);

int32_t ret      = 0;
int     overflow = 0;

ret = add32_with_overflow(a,   a, &overflow);
ret = add32_with_overflow(ret, b, &overflow);
printf("Overflow = %d\n", overflow);

Attribution
Source : Link , Question Author : coderodde , Answer Author : Community

Leave a Comment