Improving Go Error Handling

Last time I mentioned that I had a way to improve error-handling in Go but I didn't get into the details. The idea is to use the compiler to eliminate all the boilerplate error-handling code but without the problems of full-blown templates.

But first, let's try to understand why Go is designed as it is. An obvious, and major influence was experience with C so I will look at my experience with error-handling in C. (If you are not familiar with C then you may be inclined to skip the following section but please at least skim it.)

C Error-Handling

C (and C++)...
are less forgiving

Generally, error-handling in C is poor to non-existent. This has probably caused more aggravation for users than any other software deficiency of the last few decades. That's not to say that C programmers are of a lower standard (probably the contrary :), just that most of the commonly used software was written in C (and C++) and these languages are less forgiving if you do not know what you are doing.

Here I share some of my experience with C, which is typical.

I started programming in C in the early 1980's and one of the biggest tediums (tedia ?) was having to write masses of error-handling code, and making sure the code worked. As an example, consider this code to copy a file. (I know there are simpler ways to copy a file but this sort of code was like a lot of C code I was writing at the time.)

bool copy_file(const char *in_name, const char *out_name) {
  FILE *fin, *fout;
  const size_t BUF_SIZE = 1024;
  char * buf;
  size_t count;

if ((fin = fopen(in_name, "r")) == NULL)
  {
return false;
}
  if ((fout = fopen(out_name, "w")) == NULL)
  {
  fclose(fin);
return false;
  }
  if ((buf = malloc(BUF_SIZE)) == NULL)
  {
fclose(fin);
fclose(fout);
return false;
}

  for (;;)
{
  if ((count = fread(buf, 1, BUF_SIZE, fin)) < BUF_SIZE)
{                                // WARNING: see below
  if (feof(fin))
break;

  fclose(fin);
  fclose(fout);
  free(buf);

  return false;
  }

  if (fwrite(buf, 1, count, fout) < count)
    {
fclose(fin);
fclose(fout);
free(buf);
return false;
  }
  }

  fclose(fin);
  fclose(fout);
  free(buf);

  return true;
}

Listing 1. C function to copy a file.

“most C
error-handling code
is never tested”

An obvious problem with Listing 1 is that the large amount of error-handling tends to obscure the essence of the code.  (Compare it with the same code without error-handling, in Listing 3 below.)  This sort of code is hard to write, hard to read and hard to modify.  Further, it is difficult to verify that error-handling code is correct; in fact most C error-handling code is never tested and causes all sorts of chaos in the field when the actual errors do occur.

WARNING: I should point out that Listing 1 (and Listing 2), despite attempts at thorough error-handling has at least two problems.  Can you spot them? Furthermore, production code should have more informative error handling - such as trying to diagnose, and inform the user why different errors occurred. For example, if the input file could not be opened was that because it did not exist, was not accessible or some other reason?

Part of the complexity of Listing 1 is due to all the fclose/free/return statements in the error-handling which are repetitive and error-prone (remember DRY). It would be quite easy to forget a call to free()  and cause a memory leak. In fact the code I would typically write (if the coding standards in effect allowed use of goto :) would be more like this:

bool copy_file(const char *in_name, const char *out_name)
{
  bool retval = false;
  FILE *fin = NULL, *fout = NULL;
  const size_t BUF_SIZE = 1024;
  char * buf = NULL;
  size_t count;

  if ((fin = fopen(in_name, "r")) == NULL)
  goto handle_error;
  if ((fout = fopen(out_name, "w")) == NULL)
  goto handle_error;
  if ((buf = malloc(BUF_SIZE)) == NULL)
goto handle_error;

  for (;;)
  {
  if ((count = fread(buf, 1, BUF_SIZE, fin)) < BUF_SIZE)
  {
  if (feof(fin))
  break;
  else
goto handle_error;
  }
  if (fwrite(buf, 1, count, fout) < count)
  goto handle_error;
  }
  retval = true;  // indicate success

handle_error:
  if (fin != NULL) fclose(fin);
  if (fout != NULL) fclose(fout);
  if (buf != NULL) free(buf);

  return retval;
}

Listing 2. Using goto to avoid repeated code.

Note that the Go language neatly addresses this problem with the defer statement as I mention later.

Now look at the same function (Listing 3) without any error-handling code.

void copy_file(const char *in_name, const char *out_name)
{
  FILE *fin = fopen(in_name, "r");
  FILE *fout = fopen(out_name, "w");
  const size_t BUF_SIZE = 1024;
  char *buf = malloc(BUF_SIZE);
  size_t count;

  while ((count = fread(buf, 1, BUF_SIZE, fin)) > 0)
  fwrite(buf, 1, count, fout);

  free(buf);
  fclose(fout);
  fclose(fin);
}

Listing 3. No error handling.

This is plainly much simpler than the previous versions (and fixes the major bug). This is why many examples you see in textbooks omit the error-handling code to make it easier to understand.

Unfortunately, a lot of production code is actually written like this!  Moreover, even more (95% or more) has inadequate error-handling to some extent.  Why is that?

1. Blind following of example code from textbooks, as I just mentioned.
2. Lack of awareness that special values indicate errors, due to C's use of "in-band" signalling. I discuss how Go addresses this below.
3. Lack of awareness that some functions even return errors. For example, in Listing 1, the final fclose(fout) may return an error. (If buffered data can't be written to disk for some reason then fclose() will return an error.).  Unfortunately, Go does not really address this problem.
4. The attitude (laziness?) of many C programmers. For example, many security threats have been caused by buffer overflow problems. Most C programmers are aware of the dangers of strcpy() but neglect using safer functions like strncpy().
5. Poor reasoning. Sometimes you can ignore errors but often there may be subtleties of which you are unaware. For example, I have seen a lot of software that assumes you can write a file in current directory which is not always true
6. Code changes made without full understanding of the existing code.
7. Finally, occasionally, despite the best of intentions, errors are ignored by accident or oversight.

How Go Improves on C

Go has several facilities, such as the error type and multiple return values, that make error-handling simpler and safer than in C. (To be honest the standard C library's error-handling strategy, especially use of the global errno variable, could not be much worse.)

The error type

Go eschews the common C error-handling pattern of "in-band signalling". That is, in C, when a function returns a value of a certain type it reserves a special value of that type to indicate a failure. When returning an integer, -1 is often used, or when returning a pointer, NULL is used. You can see this above in the calls to fopen() and malloc() which can return NULL pointers. A different example in the above code is fwrite(), which indicates an error by returning a written count less than the requested count.

The are a few problems with in-band signaling:

1. sometimes there is no spare value that can be used as the error value
2. you often want to return more information than just that something went wrong
3. it is easy to ignore the error return value and continue blithely
4. it may not even occur to the uninitiated that there is a special error return value

An example of the problem (1) can be seen in the code above - fread() will return zero on error to indicate that nothing could be read, but zero is also returned when you attempt to read at the end of file, which, in general, is not an error condition. One way this is handled in the C standard library is to check errno after the call (remembering to ensure it is zero before the call); in the case of fread() you need to do a further call to ferror() or feof() to distinguish between an error and reading at EOF.

(2) The C standard library uses the global errno variable to indicate more about the nature of the error.  However, this is a poor strategy which has been discussed at length elsewhere.

(3) and (4) are common in C and the cause of countless bugs.

Go addresses these problems using the error type (and allowing functions to have multiple return values - see below). A function that may encounter an error returns a value of type error as well as its normal return value(s). If the error value returned is not nil then it indicates there was an error and the value can be further inspected to determine the exact nature of the problem.

Defer

Although it is rarely mentioned when talking about error-handling in Go, understanding how to use the defer statement is critical.

First using defer avoids lots of repetitive cleanup code (as seen in Listing 1 above).  Moreover, I believe it greatly reduces the chances of accidentally forgetting cleanup code, and makes it easier to visually inspect code to check that cleanup is done.  I have seen countless bugs in C code (and even created a few myself early on in my career) where some resource is allocated (eg file opened, resource handle allocated, mutex locked, etc) but then never released/closed/freed/unlocked causing a resource leak or worse.

In C it is easy to forget to free something because the allocate and free functions are necessarily called at different places (classic example of the DIRE principle). The problem is also often due to complex control flows or later code changes such as someone adding an early return from a function. Using defer in Go to free a resource immediately after it has been (successfully) allocated very neatly avoids these problems.

Though the creators of Go may deny it, I think that defer is inspired by C++ destructors (which inspired with/using statements of other languages).  As well as being called in normal return circumstances, destructors are called when an exception is thrown in C++ (during stack-unwinding); similarly defer statements are called when the code panics.

The addition of defer to Go is especially useful for error-handling but there are a few pitfalls for the unwary:

• Deferred functions are called at the end of the function not the enclosing block.
          (I expected behavior like C++ destructors which are called at the end of the block.)
• Only defer freeing of a resource after checking that it was successfully allocated.
• It’s common to see a deferred Close such as this:

if file, err = os.Open(fileName); err != nil { return err }
defer file.Close()
…

The problem with this code is that it ignores the error-return value from Close(). Generally, it should be written like this:

if file, err := os.Open(fileName); err != nil { return err }
defer func() {
if err = file.Close(); err != nil { return err }
}
…

Multiple Return Values

A function in Go can return more than one value. I believe that a reason, probably the main reason, this was added to the language is to allow a function to return a value and an error condition without resorting to C's in-band signalling due to the problems discussed above (especially problem 3).

Go forces you to explicitly say you are ignoring an error by using the blank identifier (a single underscore).  For example:

i,  _  := strconv.Atoi(str)

which converts a string into an integer. In this case if there is an error during the conversion then it is ignored, and i retains the zero value. However, there are some problems with this system.

First, even if you know an error will not occur, you cannot use the return value in an expression.  I often want to use the return value of strconv.Atoi() in an expression knowing that the string represents a valid integer (or just being happy with a zero value, if not).  It is tedious and error-prone to have to assign the return value to a temporary variable, which is why I usually wrap Atoi() in my own function which returns a single value.

A bigger problem is that you can ignore easily ignore an error return value when you are not interested in any of the the returned values. It is all too common to see Go code that ignores the error return value of functions like os.file.Close().

For example, this generates a compile error:

file := os.Open(fileName)   // compile error: multiple-value in single-value context

If you know the error will not occur, in a particular circumstance, then you can explicitly state that you do not want to use the error-return value like this:

when a function
only returns
an error in Go, it is
too easy to ignore it

file, _  := os.Open(fileName)

However, you can call Close() like this:

file.Close()         // compiles OK!!

whereas having to do something like this would be preferable (if we know that Close() cannot return an error):

_ = file.Close()   // ignore error from Close()


A Go Example

As an example, of how Go error-handling compares with C here is the same function as in Listing 1 but written in Go (or a language like Go), using defer and multiple return values.  Note that this is not real Go code as the standard file handling functions are different and Go has no need for anything like malloc().

func copy_file(in_name string, out_name string) error
{
  var err error
  var fin, fout *file
  if fin, err = fopen(in_name, "r"); err != nil {
  return err
  }
  defer fclose(fin)
  if fout, err = fopen(out_name, "w"); err != nil {
return err
}
  defer fclose(fout)

  var buf []byte
  if buf, err = malloc(1024); err != nil {
  return err
  }
  defer free(buf)

  for {
  if eof, err := fread(buf, fin); err != nil {
  return err
  }
    if eof {
  break
  }
  if err = fwrite(buf, fout); err != nil {
  return err
  }
  }
  return nil
}

Listing 4. Error handling in a "GO" like language

Comparing Listing 4 with Listing 1 you can see that there are some improvements, but it is still far from ideal when you compare it with Listing 3. You might argue that the only way to get anything like Listing 3 is to have exceptions added to the language - I agree that exceptions have advantages, but they also have negatives - so we take a slight detour to consider why Go does not have exceptions.

Exceptions

exceptions
simply can't be
... ignored!     

There are two major advantages with exceptions, plus a disadvantage.

Advantages

A. If you search on the Internet for the advantages of exceptions you find all sorts of things mentioned (eg the first hit at Google I get is to the Java documentation on exceptions). What they fail to mention is the most important one - exceptions simply can't be (accidentally or intentionally) ignored! This was the first thing that struck me when I first read of exceptions in

* Note that nowadays these sort of C bugs are sometimes detected in some way and the software terminated by the operating system but when I started using C on operating systems without hardware memory protection (eg MSDOS, AmigaDOS, etc) an ignored error could cause mayhem, eg: behave erratically, even corrupt its own data and save it to disk. It might also trash other running software, or bring down the OS!

Stroustrup's The C++ Programming Language, 2nd Edition as I had spent many painful years tracking down bugs of this nature in C code* - if an error occurs you throw an exception and the program stops, unless steps are taken to catch it.

B. Of course, the other major advantage of exceptions is reduced complexity (see Listing 3). The error-processing code does not obscure the "normal" control flow.  This makes the software more understandable and even easier to get right. And of course, it relieves the tedium of writing lots of similar, uninteresting code.

Disadvantages

From the above advantages you can see that exceptions are great to (A) ensure that errors are not ignored and (B) to make "normal" code easier to write and understand.  In other words they are great when:

A. exceptions are thrown and not caught (error causes program to terminate)
B. exceptions are never thrown (no error encountered)

The real problems occur when:

C. exceptions are thrown and caught

The reason is that exceptions are often thrown when you are not expecting it. At the point they are caught it is easy for the software to be left in an inconsistent state causing memory/resource leaks and even more serious bugs. In fact there are many examples of simple, seemingly innocuous, C++ code where an exception causes that most heinous of coding-crimes: undefined behavior.

This is not too serious if exceptions are used properly and sparingly but the last decade has revealed a new problem - the gross overuse and misuse of exceptions for normal control flow as seen in a lot of Java code.

To elaborate, you can write safe code using exceptions but it is hard. The trouble is you have to always be thinking about what exceptions could be thrown in addition to the actual problem that you are trying to solve. And the human brain is not good at multi-tasking.

Furthermore, exceptions and concurrency do not mix well. (Eg: see the section on Mismatch With Parallel Programming in Exception Handling Considered Harmful).  Go makes writing concurrent code easy, so it seems better to avoid exceptions in the language.

Go

So Go does not have exceptions due to their major problem (C above). Go attempts in other ways to obtain their benefits (A and B), but does does not do so effectively. My proposal is to enhance the Go compiler's support for error handling to obtain the benefits that exceptions give to A and B.

The Proposal

My proposal relies on the compiler generating some hidden code. (Alternatively, this could be done by some sort of preprocessing of Go code.)

In summary, I propose these changes to Go:

• If a function is called which returns one or more values, the last of which is of type error, and
• if that last returned (error) value of the function is not assigned or used AND
• the calling function also has a (last) error-return value THEN
• the compiler will automatically add code to check the error return value AND
• if the called function returns an error then the calling function should return the same error

Returning tto our original example, Listing 5 has the same copy function as Listing 4 but with no error-handling, which is proposed to be implicitly added by the compiler.

func copy_file(in_name string, out_name string) error
{
  fin := fopen(in_name, "r")
  defer fclose(fin)
  fout := fopen(out_name, "w")
  defer fclose(fout)
  buf := malloc(1024)
  defer free(buf)

  fmt.Println("Copying", in_name, "to", out_name)
  for fread(buf, fin) != eof {
  fwrite(buf, fout)
  }

}

Listing 5 Copy function with implicit error-handling.

the compiler would generate code equivalent to:

func copy_file(in_name string, out_name string) error

{

  fin, err := fopen(in_name, "r")
  if (err != nil) {
  return err
  }

  defer fclose(fin)

  fout, err := fopen(out_name, "w")
  if (err != nil) {
  return err
  }

  defer func() {
    err := fclose(fout)
  if (err != nil) {
  return err
  }

  }

  buf, err := malloc(1024)
  if (err != nil) {
  return err
  }

  defer free(buf)

  _, err = fmt.Println("Copying", in_name, "to", out_name)

  if err != nil {

  return err

}

  for {

  tmp, err := fread(buf, fin)
  if (err != nil) {
  return err
  }
    if tmp == eof {

      break;

    }

  err = fwrite(buf, fout)
    if err != nil {
  return err
  }

  }

}

Listing 6 Same function showing compiler-generated error-handling.

In this way errors can be propagated up the call stack without the need for explicit error-handling code at each level. Even accidentally forgetting to check the error return value of a function like close() will automatically be handled.

Of course, at any level you can override the compiler generated code, if it's error-handling is insufficient. This is done by simply using the error return value.

Or, if you know the error condition won't occur then you can explicitly assign the error to the blank identifier to ignore it.

A further advantage is that functions which return a result and an error can be used in expressions like this:

calc(strconv.Atoi(str))

instead of:

tmp, _ := strconv.Atoi(str)
calc(tmp)

which avoids the use of error-prone temporaries and has the added advantage that if Atoi() unexpectedly does get an error that it will be detected and returned.

Auxiliary Proposal

In addition, I have a related, but independent, proposal.

• If a function just returns an error then you must use it (or assign to the blank identifier)

That is, this code should generate a compile-time error:

file.Close()          // compile error: error return cannot be ignored

To explicitly ignore the error you must do this:

_ = file.Close()   // ignore error return value from Close

Combined with the above main proposal this makes code much safer. In fact a lot of existing code that ignores the return value of Close() will now be safer without any code changes! Of course, without my main proposal this proposal would mean modifying a lot of existing code, such as most uses of fmt.Printf().

Summary

Two of the biggest problems I found in decades of programming in C were:

1. code that ignored error return values, often with severe consequences
2. having to write lots of boilerplate error-handling code

Exception handling (as first implemented in C++) was a great boon in addressing both of these problems.  However, exception handling introduced problems of it's own (as mentioned above) and the creators of Go chose not to add exception handling to the language, which I endorse.

Unfortunately, Go's approach to error-handling is not that much better than C. It attempts to address problem 1 but does not do so convincingly.  For example, it is easy to accidentally ignore an error-return value from a function that only returns one value.

Problem 2 has been alleviated somewhat by the introduction of the defer statement, but it is still tedious and error-prone - the sort of thing that a computer can do. Many people, including the Go creators, have debated this subject at length but their proposed strategies can be as tedious as the problem and are not generally applicable (at least until generics are added to Go. :).

My proposal addresses both the above problems without using exceptions. It relieves the tedium of writing a lot of very similar code, and makes the code easier to scan, and hence less likely to have bugs.

The benefits are obvious from comparing Listing 4 and Listing 5 above.

It also has the added advantage that functions that return two values, like strconv.Atoi(), can be used in expressions when the error return value is not needed. Again this can make the code simpler, by avoiding error-prone temporaries, and easier to read.