Seeking Multi-Platform I/O Libraries?

An Anonymous Coward asks: "I'm just getting ready to plunge into a new project, and joy of joys have been given complete freedom when it comes to the implementation language - so long as the program will build and run on both x86 Linux and Windows. Now, I don't need a GUI, this is systems stuff only (processing binary executables in fact, so lots of bitfiddling and big nasty algorithms over hairy data structures) so pretty much all I need are standard IO libraries. C is currently at the top of my list..but what other language should I be looking at? I'm happy to learn a new one, and have the go ahead to do it..like I say, they want absolute speed. Can someone suggest a better language? C++ is out, it does come with a speed hit (using C++ properly anyway, not as a souped-up C). If I'm gonna take the speed hit, I may as well consider something like Ocaml which might let me claw the speed back with better algorithms and data structures.."

Comments

-1, Flamebait

[JabberWokky]

C++ is out, it does come with a speed hit

[ in my best announcer voice ]:

Let's get ready to RUMBLE!!!

--
Evan

Re:-1, Flamebait

[RoninM]

The only speed hit relevant to c++ would be using virtual methods, ...

Ideally, perhaps, but not in reality. There's definite, significant overhead in using much of C++'s standard library. Time a loop using IOstreams (std::cout) against one using std::printf(). In GCC 3.0, at least, you should find the C++ significantly slower. (In fact, slower than a similar test in Java.) C++ is a good language with a lot of critics, but just because the criticisms have all been heard before doesn't mean they lack merit. One can say most of the performance problems aren't inherent in the language, and that'd be true, but what difference does it make if all of the implementations are so bad?

Re:-1, Flamebait

[Anonymous Brave Guy]

You're right on the money there, unfortunately. There is no theoretical reason for C++ to be slower than C at all; if you implement the equivalent functionality to, say, virtual function dispatch in C, then it's likely to run with at least the same overhead as if the compiler does it for you in C++.

OTOH, standard library implementations mostly suck. At least they're improving, though, sometimes drastically; check out SGI's latest. And the template implementation actually helps the optimiser a lot in some cases. When you've finished comparing cout and printf, try qsort vs. the C++ library sort and spot the difference.

IMHO, though, the biggest problem is still the optimisers behind the compilers. We've got decades of experience optimising C to high levels. We've got perhaps a quarter of that optimising things like templates, exceptions and so on in a C++ compiler. Again, it's getting better, and sometimes quantum leaps get made as a new implementation technique is discovered, but it's still got a way to go. That is where the really big practical disadvantage lies, at least for now.

Try lots of implementations....

[borgboy]

You will likely find that algorithmic improvements will gain you more speed than IO library efficiency, as long as you avoid VB. Heck, I'd even strongly look at Java with a good JIT. Don't write off anything 'till you've tried it.

Re:Try lots of implementations....

[JanneM]

But given the same algorithmic improvements,you will of course be able to gain additional speed by choosing your language.

If speed were not so critical, I'd suggest Perl, actually. With the speed demands, and the need for cross-platform IO, I think C is probably what you want to use.

/Janne

Re:Try lots of implementations....

[ThePilgrim]

I wonder if you could rip the Perl abstracted I/O layer, its written in C and is cross platform

Re:Try lots of implementations....

[JanneM]

Perl is actually very good for bit fiddling; the pack and unpack operations would be excellent for this type of work. And it is also a very nice language for manipulating complex data structures with the ability to dynamically create and manipulate hashes and arrays.

Perl is no good at real-time tasks, of course. I doubt I would consider Perl for heavily calculation-oriented applications either.

/Janne

Consider Python... Wait! Don't leave!!!

[Phoukka]

I know it's a bit of a stretch, but consider Python. Prototype the heck out of the system in Python, profile the application, then recode the bottlenecks in C. Use SWIG to generate your interfaces. Easier to program, easier to extend, easier to read/maintain. Shorter programming time, too.

You'll be happier, your fellow programmers will be happier, your successor programmers will be happier, and the chewy parts of your code will still be really fast. Think about it.

I have a similar question:

[Cuthalion]

Hi, yes! I have a similar question: What is the best language for What I Want To Do? It needs to be able to handle floating point numbers. I don't want to use perl, because it's slower than C but messier than Smalltalk. Also, should I use vi or emacs to edit my source?

But seriously.. Every language provides standard support for file IO, unless it's totally half-assed*.

If you actually want to get helpful answers, you might provide a little more information. For instance: How much analysis will you be doing on the files? How much data are you dealing with? Is this probably going to be blocking on input all the time, or does run speed actually matter? How large and/or complicated will your program be? Does cost of deployment really matter?

* Or halfway totally assed, or whatever.

Re:I have a similar question:

[sinserve]

Long answer:
What you are asking for does not need an entire language, just a
library.
Get the best library out there, and scan through list of its "binding"
languages, then pick the one you are most comfortable with.

Short answer:
Common Lisp (CLISP + CLOCC + emacs + ilisp) is a killer.

Re:Consider Python... Wait! Don't leave!!!

[Anonymous Coward]

Yes and no. Python is a nice language, and potentially useful. However, where I work we have a "legacy" system written in python with SWIG (interacting with C++).

The problem is that under large workloads (which is normal for us) you end up with python spending more time marshalling and unmarshalling objects. It's a PITA. I blame this mostly on SWIG (which I am NOT a fan of. Don't get me started on what the maintainers consider good development practice.)

Python's a great choice if you can do it all natively. It's also a great language to prototype in and then "translate" to another language like C++ or C or Java. (depending on task and preference.) But I wouldn't do the python+swig thing.

[Note: I'm only posting anonymously to protect my identity. There are certain political factions at work that read /. and would be very unhappy with me discussing this.]

Yes, try O'Caml!

[Tom7]

Yes, I would really recommend O'Caml. Here's why:

If you just write the same program you would have written in C, the speed will be quite good, probably about 20% CPU-slower than C. (And if your program is IO-heavy, you might not notice this at all.)

If you have any sort of limited time or interest (as most projects do), you'll be able to write a much better program in O'Caml than you would in C, because:

- Because it's safe, you won't need to ever spend time tracking down or debugging core dumps or memory leaks. Because it's statically typed, a large percentage of bugs are caught at compile-time.
- If your program is interacting with the network, you won't need to worry about buffer overflows, format string bugs, or most of the common security problems.
- O'Caml has a much richer core language than C, with support for algebraic datatypes, pattern matching, higher-order functions, threads, modules, and objects. You can do a lot of great stuff with these.
- O'Caml has a nicer (though not as nice as, say, SML) module system, which keeps your program from getting unmanageable, and helps isolate faults to a particular module.

And by better, I also mean faster -- development wisdom says that algorithms and data structures are what matter most, not just the instruction-level efficiency of your code.

Of course, if you don't know the language, then it will have a higher startup cost for you. But I think it's worth it; you'll learn a different programming style that can help you think in new ways even when you're writing code in Old School languages. =)

Use OCaml

[__past__]

Scince you mention it yourself, why not really use OCaml. The "speed hit" isn't too big compared with other languages, and optimizing "nasty algorithms over hairy data structures" will definitly work better than in C.

Of course, it has a portable IO lib - just because the corresponding module for more low level stuff is called "Unix" doesn't mean that it isn't available on Windows as well, with some restrictions.

c++ is out?

[Aniquel]

I'm really very curious why you decided that c++ is out. I understand that the common (mis)perception is that c++ is slower - but let me ask this: Have you ever benchmarked it? If not, then I strongly suggest that you don't discount c++ out of hand. It has the cross-platform io facility of which you speak (streams), already has all the (completely debugged) algorithms and advanced data structures. Look, nothing is going to be faster than c (except for hand-tuned assembly) - If you absolutely need every little bit of performance, then don't bother with a language other than c. But, if you're looking for a language nearly as fast, with a complete template and streams library, that's portable, then you ought to seriously consider c++. (btw, I've written extensive projects in c++ (25000+ lines) - There isn't much performance difference, and the benefits to using it far outweigh any other penalties.)

Re:c++ is out?

[4of12]

Right on.

Benchmarking is the key. And, it pays to do it every few years or so, as compilers and hardware and software platforms evolve.

While not related directly to your I/O question, a colleague found that earlier benchmarks we had done for floating point intensive calculations which showed FORTRAN beating C++ by about a factor of two were outdated. Current tests show them comparable in speed (as long as you're not too careless with your C++).

I think I/O in C++ can be reasonably fast for most purposes, but again, as long as your careful about how you do it.

By all means, benchmark!

Re:c++ is out?

[jmv]

My experience shows that in many situations, C++ can actually be much faster than C (not always of course). The reason: templates and inlining. With inlining, not only do you save function calls (which usually aren't that expensive), but the optimizer is free to use common sub-expression elimination across the "call". With templates, you can produce better generic code. Just compare the C qsort to the C++ sort algorithm. In the first case, you go through a function call by pointer (for the comparison operator) which is *very* expensive, while in the second case, the function will be optimized just for the type you need.

Re:c++ is out?

[gkatsi]

Even though you have it right that it is a misconception that C++ is slower than C, you miss one very importatnt point: the supposedly slower features of C++ (like virtual functions) do not have an equivalent in C. In fact, in order to achieve the same functionality in C, you will have to hand code what the compiler already does for you in C++. But we already know that compilers are better than humans in avoiding errors and applying the same solution over and over with good efficiency.

Moreover, because the compiler knows what you're actually trying to do, it can often perform optimizations that are not possible in C. For the example of virtual function calls, the equivalent in C (both in terms of functionality and efficiency) is calls using function pointers. The difference is that in C++ the compiler often knows the dynamic type of an object (if it's an actual object and not a pointer or reference) and can optimize away the virtual function call and replace it with a static call (or even inline the function). The C compiler is unable to do that.

So yes, there are features in C++ that have a performance penalty, but they have no equivalent in C, so the comparison is invalid.

As for ocaml or other FP languages, I think it's a good idea to try them. Besides the productivity and maintainability gains, you may also have actual efficiency benefits. Again, because the compiler knows what you're trying to do in a high(er) level language, sometimes it can perform obscure but very effective optimizations that can beat what an average or even good C programmer can do.

Re:c++ is out?

[kraf]

> It has the cross-platform io facility of which you speak

My work experience is that c++ is not easily portable.
All c++ compilers I've worked with on various unixen had some kind of brain damage that made most of the advanced c++ features (like templates) near unusable.

Use Logo!!!! (not...)

[walt-sjc]

If you know C best, use C. If you know Java best, use Java. Ditto for Perl.

Really.

The better you know a language, the faster you will be able to write your app, the more optimized it will be, fewer bugs, etc. This is common sense.

(I was going to have a really smart-assed comment on Logo, but I'll reserve that for later....)

more than just a language performance question...

[CaptainAbstraction]

This is more than just a language question. It looks like you're starting to get the standard responses already for Java, C++, etc.

But all of these opinions presume that you're fairly experienced in these languages. Ignore them.

Language experience/familiarity is THE factor here, so don't discount it. Someone who has been eating and breathing Java would likely produce speedier code than someone who is just learning C, for example.

Your employer/client wants SPEED. This project involves hairy and complicated bit fiddling. I would suggest NOT using this project to learn a new language, for the risks outweigh the rewards in this situation.

If you choose to use a new langauge for this critical job, you're setting yourself up for disappoint. Do not forget that you're going to have to go through the all the growing pains associated with a new langauge. You're going to spend weekends tracking down (and learning from) all the newbie mistakes one makes with a new langauge. You are going to encounter new and unfamiliar bugs at all levels - logical design, physical design, semantic, syntactic.

Do you really want to spend your nights and weekends figuring out what the heck is throwing some particular JAVA exception seamingly at random? Why your C++ function template specialization is being ignored?

Learning a new language is exhilarating, but that will quickly turn to FRUSTRATION when you run into that weekend-long show-stopper bug.

With your product being measured by performance, and with deadlines looming... When it comes down to crunch-time, I think the choice is OBVIOUS!!

Choose a different, fun project to learn a new language. But for this product you're delivering, I would encourage you to stick with the tools you know and love.

Best,
Captain Abstraction

Use C

[mccalli]

Looking for an IO library standard across platforms?

#include <stdio.h>

Says it all really.

Cheers,
Ian

Re:Use C

[josepha48]

I'd add:
#include <stdlib.h>

Yeah, that does say it all. I have been working on such a thing. I have been attempting to do a cross platform library. So that it will at least be source compatible.

This is really difficult to do. If all you are doing is memcpy, file io, printf, then it is possible. If you get into sockets then it gets a little more machine dependant. Use log on and off, is even worse.

One option is to pick a cross platform C API. glib may work. I think there is a port to windows and if not it still should work under cigwin. Its speed is not that bad and it gives you things like sockets and linked lists and all the things you'd need for a daemon process or simple none gui program.

Java

[alanjstr]

Ok, so Java isn't the greatest at performance, but it is cross-platform.

Don't forget the Apache Portable Runtime

[Anonymous Coward]

Apache 2.0 is based on an excellent platform independent IO library (and many other cross platform data types, data structures, etc), the Apache Portable Runtime. It's written in C, and it's fast.

http://apr.apache.org/

wow

[sinserve]

Your "speed" priority, and the binary processing bit, got me almost sold, and then
I saw O'Caml!!

You quiche eating wanker, how COULD you forget assembly? Isn't that what programming is
all about? And WHY are you comparing C to O'Caml, a fine assembly macro language, to
shity ML dialect used by equally hard-wanking mathematicians and abstractly thinking
creatures? If these wankmaticians knew how the world operated, they would not
have invented recursion let alone APPROVED of inductions as a sane, corner stone
princible in their so called "art". Induction is only possible as long as the
the "counter" register can hold your index, and recurssion is the crackwhore narcessistic
twin sister of iteration (there is nothing she does, iteration can't do with
a well placed label and a jump.)

Listen to me son, read Quine, Boole and DeMorgan, get the manual to your processor,
and "script" at the level of the ONE TRUE ABSTRACTION LAYER.

Re:wow

[popeyethesailor]

Are you Mel by any chance?

Using C++ properly????

[PD]

How can you use it improperly? C++ is an object capable language, not a strict object oriented language. If you want to use objects, then fine. If not, then please don't.

Object oriented development is a tremendous thing, useful for many things, and a marvel of overcoming complexity through abstraction.

BUT, OOP is not the solution for everything. There are many problems that don't need an object structure, and should be written another way. Above all, drop the notion that C++ should be used only a certain way to be proper. The latest cool feature of C++, the Standard Template Library, isn't even object oriented - it's GENERIC, because that type of programming just was the right thing to do for that library.

Re:Consider Python... Wait! Don't leave!!!

[Circuit Breaker]

Have you actually tried using Python? If you have, it's probably for not enough time or using the wrong tools

1. Using indentation instead of braces kills the religious "coding convention" wars before they have a chance to start. It's easy to read, it makes what you read and what the parser read consistent (Never chased a mismatched indentation/braces case, have you?), and it just plain works. Where did that function start? Any editor worth its while can tell you that, most of them already have a macro that does this. If you ever used Scintilla/SciTE you'd probably never go back to "find matching" only style editors unless you were forced to - collapsing functions makes a lot of sense even in the curly brace world (more so in Python's indentation world).

2. There are add-ons that can enforce that, but that would be missing the point. The Python interpreter and language specification goes to some length to catch this kind of errors, and although it's a long way from e.g. C or Java, it caters for the common cases. Typos in long variable names may create annoying bugs, but ones that are _always_ easy to identify and fix. True, they wait for run time rather than compile time; personally, the number of bugs of this kind that I get is consistently low enough for this not to matter (and, since Python code tends to be an order of magnitude shorter than any other language except Lisp or APL, it's more than worth it. Plus, there's a Lint for Python if you insist). Variable declarations are NOT free documentation. "Object my_object = new Blah();" is not more informative than "my_object = Blah()". It's the variable's name that's the documentation, rarely it's type.

3. Oh jesus. C++, Java, SmallTalk, LISP and just about any other language does this too. What language are you using? Plus, try scintilla and you'll be amazed at what a GOOD language sensitive editor can do (for any of the above languages).

TCL

[WetCat]

Try TCL.
For me, using TCL my performance increased by 60%
(especially when using its [Incr TCL] OO Extension)
TCL works on most unices, Windows, Mac, VMS, Palm Pilot...
Tk graphical library is so successful that other languages
(perl, prolog, python) are using it.

No, thanks.

[Rick the Red]

I'm not seeking Multi-Platform I/O Libraries. Thanks for asking.

One Word:

[brunes69]

QBasic.

What speed hit?

[SIGFPE]

Last time I checked people were writing faster readable code in C++ than in C.

A smart C++ programmer can use template metaprogramming in a library like Blitz++ to automatically build code optimised for the job. To write the equivalent code in C is possible but it's much more laborious and harder to maintain.

There are good reasons not to use C++. Performance isn't one of them.

Re:What speed hit?

[SIGFPE]

These traits that you point out aren't necessarily C++ problems. Yeah...some people get carried away with Russian-doll like hierarchies of C++ - but some people don't. Similarly there's no reason for C++ to have much of an memory overhead compared to C. If you use virtual functions you might get a tiny performance hit and memory hit but plenty of C code uses tables of pointers to functions. I think the problems you're seeing are due to the way programmers who like bloat are drawn to C++ rather than being a C++ problem inherently.

Re:What speed hit?

[RoninM]

You're clearly confused. The performance of ld.so has little to do with C++ and nothing to do with deep levels of inheritance. There IS an issue with virtual functions (vtables) and relocations, which is probably what you're trying to reference, but this isn't really an issue with C++ and the problem can be addressed by lazy binding of vtables. It's also completely moot when we're talking about a low-level C++ application: the chances of him needing to do heavy dynamic linking or having a vast framework of objects with significant numbers of virtual functions is so slim that it doesn't even bear mentioning in this discussion.

General purpose vs. best of breed

[Kirruth]

You can choose to use a general-purpose language which has a good spread of capabilities, or you can go with a best of breed language in the area you are trying to work in.

For general projects, I use a mix of Python and C++. I'd say the best of breed languages for text would be Perl, math would be Haskell, and for getting down to the metal would be Assembler.

For what you are trying to do, the no-brainer choice would be souped-up C, i.e. C which uses a few C++ features to make your life easier.

Use K

[Jayson]

K is a high-performance data processing language. It is a high-level language with very fast performance (it even beats out well written C code). Many people after switching to K have noticed 100x decrease in code side (yes, 2 orders of magnitude) and sometimes even more. It has very high-performance I/O facilities and was explicitly made for muching data. It is cross-platform and runs on NT, Solaris, Linux, FreeBSD, and AIX (you can probably get a build for other systems, too, since the guy who write it is very nice about that).

Some of theK programming maxims are that memmap is better than read/write (the native file I/O is memmap), operating over bulk data is better than scalar data (the language is built around bulk operators), and terse code is good.

There is a warning, though. K is very elite and may be too elite for you (it was for me at first), but it is very eay to learn.

Kylix / Delphi

[Micah]

No one's mentioned Borland's tools, but I think they'd fit the bill. Borland has great compiler technology, and it will compile and run cleanly across Linux and Windows (possibly with a few {$IFDEF}s). It has an I/O library that's as capable as C's (maybe a bit more wordy sometimes). Developing and debugging in Kylix is *much* quicker, in my experience, than using gcc/gdb. It's truly compiled, the compiler is lightning fast, and the integrated debugger is quite a bit more efficient than gdb based solutions.