How to Develop Securely

An anonymous reader writes "This column explains how to write secure applications; it focuses on the Linux operating system, but many of the principles apply to any system. In today's networked world, software developers must know how to write secure programs, yet this information isn't widely known or taught. This first installment of the Secure programmer column introduces the basic ideas of how to write secure applications and discusses how to identify the security requirements for your specific application. Future installments will focus on different common vulnerabilities and how to prevent them."

Comments

huh?

[Anonymous Coward]

A hacker is a computer expert or enthusiast. So I'm a hacker?

Re:huh?

[the_olo]

Read this, kid. http://www.catb.org/~esr/jargon/html/H/hacker.html

Article Summary:

[Farley Mullet]

There are basically two parts to the article:

1. An unfortunately much needed reminder to use common sense, and;
2. A way too long discussion of what exactly "Free" and "free" and open source software is.

Re:Article Summary:

[amarodeeps]

I disagree. The part about Free and Open-source software was very topical. It basically concerned the tendency of some to believe that FLOSS (Free/Libre and Open Source Software, as the author calls it) is somehow more inherently secure, and it debunks this while qualifying the debunking with some intelligent critique--such as the fact that when many eyes _are_ actually looking at the source code, there is a greater possibility that secure vulnerabilities will be found, as well as the difficulties inherent in comparing the meaning of security in the case of a very widely used OS like MS Windows with OpenBSD. Yes, this was a lot of common sense for the most part, but you should know from reading Slashdot that people are often lacking in that...

How to develop securely in 4 words

[squiggleslash]

Use strncpy() not strcpy().

(Well, ok, perhaps it's more complicated than that, but that's a good start)

Re:How to develop securely in 4 words

[__past__]

And in next month's installment, we will learn about strlcpy(3).

After that, maybe we can look into programming languages that actually have a string type, and don't tend to make every bug exploitable by default.

Re:How to develop securely in 4 words

[Circuit Breaker]

Ahhhm.

Use strlcpy(). strncpy is almost as bad as strcpy(), as it doesn't guarantee a terminating nul.

Re:How to develop securely in 4 words

[Curien]

How about (radical idea coming) we just all use the functions correctly! strcpy is not inherently insecure, it just doesn't check anything /for/ you. strncpy doesn't guarantee a terminating null character, so you have to (gasp!) check for it.

The main problem with strlcpy is that it's not standard, hence it may not be available on your target platform.

Re:How to develop securely in 4 words

[__past__]

strlcpy is 27 lines of portable C in the FreeBSD version, including K&R-style prototype, blank lines and comments. Hardly too much to include if autoconf tells you that it's missing, and well worth it - not because code using str(n?)cpy is inherently broken, but because it's just easier to work with.

Re:How to develop securely in 4 words

[cpeterso]

and don't forget snprintf() instead of sprintf(). I've heard that sprintf() is a more common cause of buffer overflows than strcpy(). sprintf() is often used to format user input.

Re:How to develop securely in 4 words

[stevey]

That's a good start, and all the 'n' functions are worthy - but it's worth thinking a level higher and being careful not to trust user/network/other programs as a source of input.

A really good read is the Secure Programming Howto, but even that is just a start, security is a process not a product...

Re:How to develop securely in 4 words

[klaxor]

After that, maybe we can look into programming languages that actually have a string type, and don't tend to make every bug exploitable by default.

Yeah, whatever. Why not teach people to write correct code instead. And BTW, what happens when your String class runs out of memory? (Yes, I've seen code which reads an entire file into a String....) So I guess it's better to segfault than risk buffer overrun?

After programming in C++ for a number of years, I've stopped using the string types, and actually gone back to using regular C-style strings. Here's why:

• The String types dynamically allocate memory. Hence, an operation such as this:

  string1 = string2 + string3;

  can fail. This doesn't leave you with any options; either you wrap every string assignment in try/catch blocks, or you just hope it works. Compare this with the following in C:

  char string1[80];
  char string2[40];
  char string3[40];
  
  ...
  
  strcat(string1,string2);
  strcat(string1,string3);
  

  The above code can't possibly fail. There's no need for try/catch, or needless error checking - the destination array will always be large enough.
• With the datasets I've been using, I quite frequently run out of memory. Since a string allocates memory on the fly, any memory allocation failure will happen in the middle of the algorithm, rather than before it - at which point recovery is pretty much impossible. By using safeguards such as the above and allocating memory before I begin processing, I can alert the user to a memory allocation problem before the processing has begun. Thus, my users can be relatively certain that a long operation really will succeed, rather than running out of memory at the last minute.
• malloc has some notable bugs in it - it often hangs when allocating numerous small chunks of memory; when it does, no amount of exception handling will bring control back to your program. Because of this, it's actually better to allocate a large chunk of memory at the outset and parcel it out as the program needs it, rather than running the risk of hanging the machine with a large number of calls to malloc.

When it comes down to it, every computer has limited memory. The difference between those programs written using a C++ String and a C-style string is that a program written uses c-style strings does not allow the programmer to forget that memory is limited. A C++ programmer, OTOH, will write as if there's no limit to memory, and hopes that someone won't process a particularly large file, or his program will crash. Unlike the C++ programmer, the C programmer must choose an algorithm that can work within a given space requirement, and hence, the size of the data processed becomes irrelevant - it simply takes more time, rather than segfaulting, or worse, hanging the machine.

I could go on about how my university taught students to allocate memory but not to free it, but I'll spare you. I think the real problem is that the illusion of infinite memory is just that - an illusion. While the String types have some nice features, everything done with a String type can be done with C-style strings, and if it can't, it's usually trivial to implement. With C-style strings, I can formally prove the correctness of an algorithm, but not with the String types, simply because their behavior is not well defined for cases in which memory allocation fails. Even if these string types had a standard behavior, you're still left with an algorithm that becomes useless when memory allocation fails.

And do you know for certain that the String type isn't immune to buffer overflow? If your program crashed because of excess input, would it escape back to a root-level shell? When it comes down to it, security requires more thought than just using some nifty-yet-formally-incomplete classes.

Re:How to develop securely in 4 words

[Anonymous Coward]

I've used both C & C++ style string handling. Just to nitpick your example: it most certainly CAN fail.
Look at what you wrote:


char string1[80];
char string2[40];
char string3[40];

...

strcat(string1,string2);
strcat(string1,string3 );


If string1 has any existing contents, you may run out of space. strncpy()/strcat() is probably what you meant.

Re:How to develop securely in 4 words

[__past__]

Why not teach people to write correct code instead.

Why not do both?

And BTW, what happens when your String class runs out of memory? (Yes, I've seen code which reads an entire file into a String....) So I guess it's better to segfault than risk buffer overrun?

Yes, in most cases, it is. Normally, I'd rather have a possible DoS than somebody writing in my stack.

The String types dynamically allocate memory...

This has little to do with the problem I was talking (well, at least thinking...) about. Of course, if you need to run in bounded space, dynamical allocation is not for you. Still, bounds checking would be nice.

malloc has some notable bugs in it - it often hangs when allocating numerous small chunks of memory; when it does, no amount of exception handling will bring control back to your program. Because of this, it's actually better to allocate a large chunk of memory at the outset and parcel it out as the program needs it, rather than running the risk of hanging the machine with a large number of calls to malloc.

Even without bugs, mallocing lots of small chunks tends to be rather slow - which is indeed something many programmers tend to forget, treating malloc as an essentially cost-free operation. I have actually seen C code that became a lot faster after using a garbage collector - the mark-and-sweep phase was less overhead than what was gained by making allocation to little more than a pointer increment.

When it comes down to it, security requires more thought than just using some nifty-yet-formally-incomplete classes.

Of course. I'd still claim that some programming environments make it easier to introduce critical bugs than others, but nothing will prevent a non-thinking coder from messing things up big time. Look at all the PHP guys, they don't have buffer overflows (except in the PHP interpreter itself, of course), but they still manage to keep all their nifty web forums a frequent topic on the security mailing lists.

Re:How to develop securely in 4 words

[YellowElectricRat]

char string1[80];
char string2[40];
char string3[40];
...
strcat(string1, string2);
strcat(string1, string3);

The above code can't possibly fail. There's no need for try/catch, or needless error checking - the destination array will always be large enough.

Can't possibly fail, huh? Too bad if your char arrays don't turn out to be \0 terminated... Then you're in big shit. Hell, even strncat won't help here if 'string1' doesn't have a \0 terminator.

Moral of the story - even if something "can't possibly fail", someone will still find a way to make it fail.

Re:How to develop securely in 4 words

[Anonymous Coward]

The above code can't possibly fail. There's no need for try/catch, or needless error checking - the destination array will always be large enough.

Of course it can fail. It fails for the same reason the String classes can fail -- because you've run out of memory. The difference is that you fail when you run out of stack memory, not heap/virtual memory.

For added interest, while your program may have up to 2 or 3 GB of virtual memory, limited by swap space, your stack is typically a more limited resource (by default, just 1 MB on Windows systems, though you can change this when the program is linked).

Additional fun comes when your program generates an out-of-stack exception -- this is NOT a C/C++ exception, it's an OS level exception outside the runtime. Those are more trouble to catch, and require the try/catch and needless error checking you seem to disdain.

If I'm writing a program that uses extensive memory resources, I have far more of those resources available to me if I use the String classes and the heap than if I declare local stack variables. And I'll use those classes rather than the C string equivalents, because I trust the classes to clean up after themselves more than I trust my own code, or my co-workers.

Re:How to develop securely in 4 words

[kruntiform]

malloc has some notable bugs in it - it often hangs when allocating numerous small chunks of memory

That is surprising. Whose malloc are you talking about?

Re:How to develop securely in 4 words

[squiggleslash]

27 lines?

char *strlcpy(char *to,char *from, size_t size)
{
to[--size] = '\0';
return strncpy(to, from, size);
}

Probably just as easy to put it in a macro, as long as you're willing to define a couple of globals for to and size (in case those are expressions with side-effects)

Re:How to develop securely in 4 words

[klaxor]

Windows, GNU, and Solaris. I haven't tested the malloc implementation in Linux, but given the fact that 4 different compilers (Visual C++, DJGPP, Borland, and gcc on Solaris) produce the same problem, I'm inclined to think that the problem lies in some hereditary UNIX code that was incorporated into Win32 and Solaris. The fact that this bug has also been reported in Solaris further reinforces this.

It is rather unfortunate, though, because malloc usually works fine as long as you :

1. Don't allocate small chunks of memory (less than 4 bytes at a time), and,
2. Don't make more than a few hundred calls allocating such small amounts of memory.

malloc seems to work just fine for allocating large chunks (greater than 64 bytes), though. Methinks there's a bug in malloc's list algorithm.

Re:How to develop securely in 4 words

[Anonymous Coward]

You are ready to start programming SML or Ocaml, for example :

1. it is fast.
2. it is completely safe.
3. programs can be proven correct (in fact SML was designed with theorem provers in mind).

Re:How to develop securely in 4 words

[Anonymous Coward]

And for the same reason, in C++, when you want to get a C string from a C++ string type, don't use mystring.data() but mystring.c_str()

Re:How to develop securely in 4 words

[radsoft]

Yes, too true. This is exactly the type of situation that leads to exploits: people that dabble where we need dedicated pros.

Let's try to make some sense out of that snippet. There might be many ways, but...

char string1[80], string2[40], string3[40];

ASSERT(sizeof(string1) >= sizeof(string2) + sizeof(string3));

// Guarantee 0 termination
// And remember: '80' and '40' are magic numbers
memset(string1, 0, sizeof(string1));
memset(string2, 0, sizeof(string2));
memset(string3, 0, sizeof(string3));

// Failsafe: force 0 termination on string1 anyway;
// also, strcat call is always more expensive
strcpy(string1, string2);

// This has to work
strcat(string1, string3);

Re:How to develop securely in 4 words

[Anonymous Coward]

Sigh. FUD and bugs.

The code you show can fail, it's just that you'll fail due to overflowing the stack rather than failing due to allocate memory from the heap. As you say, "every computer has limited memory". If the point is to fail before you concatenate the strings, you can always do this in C++ with:


string1.reserve(string2.length() + string3.length());


Of course, if your VM system lies (er, sorry, the polite Linux-y way to say this is "overcommit", isn't it?), then your program is likely to die at random times during severe memory strain anyway, and there's nothing you can do to fix it. Get a better operating system (or turn overcommit off, if that's an option).

Also, C++ standard strings have defined behavior when memory allocation fails. Specifically, standard strings use the standard allocator, which throws bad_alloc when memory allocation fails. You may not like this behavior, but it is specified.


try {
string3 = string1 + string2;
}
catch(const bad_alloc& x) { // Oops, allocation failed. Recover here.

}


Anyway, the point is that retreating to C, whacking out big hunks of the stack, and pretending that there's no way this can ever fail doesn't help, either. You've changed the failure mode without fixing the real problem. If the normal state of the system is that your processing runs out of memory, you need to rework your program to run successfully in that environment rather than throwing up your hands, backing off from dynamic allocation, and expecting the problem to go away.

Re:How to develop securely in 4 words

[Karellen]

Bollocks.

If you're in a low memory situation, and the platform you're running on has a dynamically allocated stack, then it's entirely possible to run out of stack space as well as out of heap.

So, your example above can fail (given a pathological case) with an error that's harder to check for and trap than a malloc/new failure. (What is behaviour on out-of-stack? Is it a catchable signal?)

On top of that, any algorithm that uses a limited amount of stack space to deal with large files by operating on them in a stream-like fashion, only processing a small amount of data at a time, can be coded to use a small amount of heap space dealing with small chunks of data at a time. What's more, the heap-based algorithm will be more flexible, allowing you to deal with (for example, in the case of `cat') an occasional extra-long line with relatively little extra code or complexity. The example you give is limited to dealing with 80 characters.

Malloc buggy? Which version? I've not found such a bug in a long time in glibc's malloc() implementation. If your vendor is shipping a crappy malloc implementation and you've gotta work around it, well, I really feel for you there 'cos that's the sort of thing you shouldn't have to deal with, but my advice would be to find yourself a new C library that works.

C++'s std::string behaviour undefined in the absense of sufficient memory? Dude, you need to read your manuals more carefully. If it can't allocate memory, it'll fail by throw()ing std::bad_alloc. If you don't want it doing that, you can create your own string type using the std::basic_string<>() template, supplying your own memory allocator that allocates memory however you want it to, and does whatever you want in the case of failure.

K.

Re:How to develop securely in 4 words

[HuguesT]

Your example can fail if you insert a bug like so:

#define VLONG_STRING 80
#define LONG_STRING 60
#define SHORT_STRING 40

char string1[LONG_STRING]; /* oops, wrong one */
char string2[SHORT_STRING];
char string3[SHORT_STRING];

string1 can overload. While this is obvious written like that when attention is drawn to it, in the middle of a real program people do write things ike that all the time. This is precisely the problem C++ strings were designed to fix. You don't need to allocate them beforehand because programmers are likely to get it wrong some of the time.

As for your problem of running out of memory in the middle of your program instead of at the beginning, lookup the string::reserve() member function. If you can't reserve() it will throw there and then.

If you want to implement strings with limited storage so that it doesn't eat all your heap, you can use your own allocator that does precisely that.

Finally I don't believe you can prove the correctness of something involving stack allocation like in your example. You seem to assume stack space is infinite, and it isn't. Try using your example in a recursive framework for example. At any rate you can quote Knuth on the correctness of algorithms:

"Beware of bugs in the above code; I have only proved it correct, not tried it."

So there you go. All the best.

Re:How to develop securely in 4 words

[HuguesT]

Not quite good enough:

strlcpy(to,from,0); /* fails horribly */

Re:How to develop securely in 4 words

[Homburg]

The String types dynamically allocate memory. Hence, an operation such as this:

string1 = string2 + string3;

can fail. This doesn't leave you with any options; either you wrap every string assignment in try/catch blocks, or you just hope it works.

Well, you can do:

string1.reserve(string2.size() + string3.size());

Then

string1 = string2 + string3

is guaranteed to succeed.

With C-style strings, I can formally prove the correctness of an algorithm, but not with the String types, simply because their behavior is not well defined for cases in which memory allocation fails. Even if these string types had a standard behavior, you're still left with an algorithm that becomes useless when memory allocation fails.

What algorithm doesn't become useless when memory allocation fails? If you don't have enough memory to process your data, you don't have enough memory. Whether you do

if( ! (buffer = malloc(SIZE_I_GUESS_IS_BIG_ENOUGH) )
{
fprintf(stderr, "Could not allocate memory");
exit(1);
}
some_algorithm();

or

try
{
some_algorithm();
}
catch( std::bad_alloc a )
{
std::cerr << "Could not allocate memory\n";
std::exit(1);
}

is surely irrelavent. Using C strings doesn't gain you anything, but does mean you have to remember to check for terminating nulls and buffer lengths and deal with memory allocation all the time. I'd rather have all the capacity checks etc. in one place (in the string class), so that if there are buffer overflow bugs, they only have to be fixed once, rather than scattered throughout the application.

If you have to ask

[SHEENmaster]

then you don't know enough about programming to be considered an expert by the rest of us.

There's often an obsession attached to the definition by those of us that consider ourselves hackers. If you stay up until 4 am with school the next morning, working on a program for no profit or wage, you can consider yourself a hacker.

Exceptions include VB programmers, or |-|@X0Z, and those that break into computers using exploits written by others, where targets are chosen because of apparent vulnerability, aka script kiddies.

Breaking into computers is NOT an essential part of hacking, that's a misunderstanding by the media. Then again, is "ya'll" a valid contraction in the south-east?

Re:If you have to ask

[The Analog Kid]

According to one of my Linux Admin books, if you break into systems, your a cracker, however to the media, as stupid as it is, though they like to think their so smart with technology, have screwed with the meaning of hacker.

Re:If you have to ask

[soft_guy]

It would be y'all, not ya'll. Its a contraction of "you all". When making a contraction, you should put the apostrophe in the place of the missing letters.

Re:If you have to ask

[Richard_at_work]

Language changes, live with it. Words can have their meanings altered over a period of time, again - live with it.

Wargames, Security Development, etc

[SHEENmaster]

There are a lot of exceptions to the "if you break into systems" rule.

apt-get install flawfinder

[delirium of disorder]

http://packages.debian.org/testing/utils/flawfinde r.html

How to develop securely in 2 words

[Detritus]

Ditch C.

Use a safer language, one that has bounds checks and restricts the use of pointers. It wouldn't solve everything but it would eliminate a large class of exploitable bugs.

Re:How to develop securely in 2 words

[Hellkitten]

Ditch C.

Understand C

C isn't the problem, people that use C without knowing what they are doing is. No programming language can prevent stupid programmers for making mistakes that can potentially be exploited. But C has the advantage is that those stupid programmers very often don't manage to get a compiling / working program at all, sparing us the security risk

Re:How to develop securely in 2 words

[Detritus]

I understand C and can use it reasonably safely. But I have much more experience with it and systems programming than the average programmer.

C is a dangerous tool. Some substantial percentage of programmers are "stupid", ignorant or inexperienced. They are not going to disappear just because you call them names. They are going to write programs and make their share of mistakes. Society is going to have to live with the results.

Re:How to develop securely in 2 words

[Hellkitten]

They are not going to disappear just because you call them names. They are going to write programs and make their share of mistakes.

And replacing C with something else will only cause them to make different mistakes, possibly just as vulnerable to beeing exploited. The language isn't the problem the programmers are, giving them a tool that prevents them from making one set of mistakes will simply mean they make other mistakes that the tool allows. I believe the less forgiving the language is (make a mistake and it crashes) the more likely programmers are to try and understand what they are doing, but if they can simply draw a pretty interface and write a few functions and get a program that works (most of the time) they might not.

Re:How to develop securely in 2 words

[Anonymous Coward]

SML fits perfectly to this definition :
functional, no pointers, completely safe, yet powerful.

Not about developing software

[Frans Faase]

The article is not about developing secure software, it merely talks about security issues to consider when developing software, but it doesn't give any actual answers about how to do it. It only helps identifying the problem, not providing an answer for it.

Security problems caused by tacks growing down

[Frans Faase]

Many security problems are caused by once clever design design to let stacks grow down. The reason why buffer-overruns are dangeroes, lies in the fact that it allows you to overwrite the return address of the calling procedure. If stacks would have grown up, this would never have been possible.

When these first processors (8088) were designed, nobody was aware of the possible implications with respect to security. And that is still the problem with many software/hardware engineering issues.

Re:Security problems caused by tacks growing down

[Anonymous Coward]

Sparc stacks grow up, and it isn't a lot harder to exploit a buffer overrun there either, the buffer in question just needs to be in the frame of the function that called whatever code you're exploiting.

Then when you end up writing out of bounds for that buffer you write over the current functions return address and voila.

When the stack grows down you can exploit a buffer on your own frame because it would have a lower address then your frame's pushed return address. So overflowing the bounds of that buffer will hit your return address.

So whether the stack grows up or down may enable or disable SPECIFIC attacks, but neither precludes the problem.

I've always found the whole up/down thing counter-intuitive though, because when I imagine the stack I imagine a picture with 0 at the top and 0xffff (say) at the bottom. I imagine the x86 growing the stack UP because it starts at the bottom and works its way up; even though that means it is actually decrementing the stack pointer. Every time I have to recall which is which I have to translate to which _I_ imagine it to be and which other people imagine it to be. :)

Non-executable stacks are a better answer but they don't solve every vector of buffer overrun based attacks. You could put the code somewhere on the heap and overwrite the return address with the address on the heap, for instance (assuming the data segment is executable). This may seem hard and often might be but depending on how the program works it may NOT be.

On some classes of processors you can't even have a non-executable stack. (sparc v8 I think)

In reality it comes down to the fact that unbounded memory/string operations are NEVER safe, because on some class of hardware or under some type of attack it will MOST LIKELY be exploitable to SOME effect.

It just depends on who is trying to do the exploiting, and on how much effort they want to put into it.

Persistance, access to the binary with the flaw with which to perform experiments, and a reasonably small amount of skill are all that is really required.

Reciprocal Link

[muirhead]

There's an identical article on Newsforge here.

nah, no need to change anything

[DrSkwid]

a cracker is a hacker but a hacker aint a cracker

Just use reserve()

[aspargillus]

If you know the size requirements for your strings (or vectors or whatever std containers) beforehand consider the reserve() member function. Eliminating ill behaviour due to running out of memory (assuming you avoid the creation fo temporaries) then also enables your proofs just as in C. And it speeds things up too.

Everyone Back to the IBM/370!

[Detritus]

Someone once told me that C/370 used activation records instead of a stack, like PL/I. Come to think of it, there's no reason you have to use the hardware stack pointer on the x86, except for hardware interrupts.

Re:Everyone Back to the IBM/370!

[Anonymous Coward]

Yes, it did. It was a great system because all the memory allocation for subroutine parameters was done statically and a lot of the actual values could be set at compile time. Backtraces were easy too.
On the down side, you did have to declare functions as RECURSIVE if you wanted them to be recursive, so that dynamic allocation could be used: but in real life (as opposed to computer science) recursive functions are very, very rare.

Too Many Words

[radsoft]

This was a talking head article. What we don't need is more talking heads telling us and the world what programmers need. What we do need is a better understanding amongst programmers so that they take greater pride in their work. It is no coincidence that the software I represent is generally bug and vulnerability free and has been for years: The people who wrote it are dedicated and took great pride in it.

Writing good software is the same as writing secure software, said a good friend of mine who just happened to be the clever chap who nabbed the fellow behind Melissa and sent the Washington Post to Anaconda after ILOVEYOU. He's right.

A good programmer will never allow a buffer overflow. It's just not done. A good programmer will not assume his program is going to be used as advertised. An IT manager I used to work with began all his own reviews of software by banging on the keyboard with his fists and throwing the keyboard to the floor. The programs were not allowed to crash or do anything stupid. Several developers thought this was unfair, but of course it was not. It was just good thinking.

You have to perhaps develop a feeling for what can go wrong, but getting into particulars here is not what is needed. Good instincts are what is needed.

If you don't have the instincts, and the dedication, then all the advice of all the talking heads in the world will not help you. They'll still make money hand over fist selling their blather to the suckers, but software will still be leaky.