Slashdot Mirror
Domain: kx.com
Stories and comments across the archive that link to kx.com.
Comments · 76
-
He didn't include K.K is a high-performance array language. It is based on APL and Lisp. It really shines when crunching obscene amounts of data. This seems like something that would be perfect for the language. The proof of K's speed lies in KDB, a database written entirely in K. On TPC benchmarks is spanks Oracle and other leading databases (including some amazing scaling across processors: simple table scans with 2.5 billion rows take 1 second and multi-dimensional aggregations take 10-20 seconds).
There is a quick and dirty intro to K over at Kuro5hin.
Some more links for more inforation:
Kernigan's benchmark test
more examples
Kx: the people who make K and KDB -
Influence by little known languages.I was wondering if you have looked at little known languages for ideas in Perl? It seems that Perl is agressively going after areas that are heavily explorer in other language paradigms. Have you seen or been influenced by any smaller language (aside from the common choices of Scheme, Python, Ruby, Tcl, and the such)?
Seeing Perl6's introduction of more powerful map-like and reduce-like operators in the hypers, I would like to know if you have used and been influenced by the modern APL derivatives: Dr. Iverson's APL successor, J, is more well-known; but Arthur Whitney's K mixes more functional, Lisp-inspired abilities into the language.
-
It's called APL
and it has been around for years (the second oldest programming language still in use?). There is a modern day version called K that will crush C based systems when full SIMD support is implemented (is already does a good job of thrashing most languages). Collection based languages used to be much bigger that they are today for some reason. Current programming paradigms go into one direction (OO) while we keep having to deal with data in the other direction (bulk operations).
-
Use KK is a high-performance data processing language. It is a high-level language with very fast performance (it even beats out well written C code). Many people after switching to K have noticed 100x decrease in code side (yes, 2 orders of magnitude) and sometimes even more. It has very high-performance I/O facilities and was explicitly made for muching data. It is cross-platform and runs on NT, Solaris, Linux, FreeBSD, and AIX (you can probably get a build for other systems, too, since the guy who write it is very nice about that).
Some of theK programming maxims are that memmap is better than read/write (the native file I/O is memmap), operating over bulk data is better than scalar data (the language is built around bulk operators), and terse code is good.
There is a warning, though. K is very elite and may be too elite for you (it was for me at first), but it is very eay to learn. -
Re:What about K?
>I don't understand what that does, but it is pretty fast (can't do values above 9 though).
It simply archives values so they need not be recomputed.
Tack[3]10
8189
On a 400 MB machine running a K research license it overflows stack at (3;11). The free license has a limitation on memory usage (100MB) and should poop out earlier.
>I assume that a C program could be written using the same approach and it would still be faster.
Tack was written within an hour of when the problem (stack overflow) was presented - look through the k@listbox archives under "stack error" and "[Fwd: More interesting challenge (was Re: Challenge problem)]".
i would be interested in a C program which actually was faster though, however long that took you. Life is too short for me to try. Anyway machines are much better C programmers than i am, and they are dirt cheap with no need for medical benefits. C is far too cumbersome for my pocket, i am happy to carry the Swiss bankers K'nife (AND a real Swiss Army Knife:) though.
-gheil -
oops, use this link instead
I screwed up the link in the previous post, try this one:
here
K was invented way before OSS really got off. kdb is free for non-commercial use and their license is no more anal than say MySQL's was before recently while the quality is way above. I hope K will get used, even if only for writing kdb extensions/stored procs. You have no idea how useful a FUNCTIONAL database can be (as opposed to relational) -
Re:What about K?
-
What about K?I agree that most language are the same C derived POS. C was different and inventive when it was created. Lisp, APL, Prolog, and Smalltalk were all different when they were created. It seems like as time went on we started narrowing our field of vision and implementing the same languages: C to C++ to Java, what kind of intereting steps are those? At least Smalltalk to Self was a very interesting pushing of the boundries. Today, almost nobody pushes anything, except how similar their langauge is to C and why that it good. Even Python and Perl don't attempt to explore any new concepts, they are happy being a Frankenstein of older languages that people seemed to have forgetten about; name three new features of either language, just try to name one!
My sole exception to this is a language called K. Yes, it has its roots in APL and has added to the APL model from languages such as Lisp and Scheme, but it has some very interesting new features of its own.
K is very very very fast to write and the run. It blazes in both categories. There is a full relational database that is written in K, called KDB. It crushed Oracle on the TPC-B and TPC-D benchmarks in both speed and storage size, requiring only a few percent above the dataset size in overhead. It has native clustering and replication that allowed it to run on a 50 cpu Linux cluster loaded with 2.5 billion stock trades and quotes and have simple table scans (such as, select max price from trade) take under a second and multi-dimensional aggregations (such as, 100 first desc select sum size*price by sym from trade) take only 10 seconds. Starting the database cluster took a tenth of a second. It is SQL92 compliant, has an extended ultra-powerful query language called KSQL that makes writing queries very simple, and the stored procedure languages are K and C.
In bwk's language benchmarks, even though this is not the K strong point, the sum of the execution times were: K at 32 seconds, Perl at 95, Java at 300, and TCL above 1400. The lines of code to implement were: K at 9 lines, awk at 95, Perl at 96, TCL at 105, Scheme at 170, VB at 200, and Java at 350.
Yes, K can look like line noise, but unlike Perl, you get alot from this. First you get extreme code density and see the entire problem on the screen at once. I came from a Scheme background and Perl hurt my eyes, so I was very skeptical, but after my roommate persuaded me to look at K harder, I realized that this high code density made it very easy debug and write code. It is rumored that KDB is written in 26 files of code, each file consisting of a single screen of code, labeled a to z. Try doing that in any other language. The language is exceptionally regular. It is so logical and consistent that it takes a little getting used to. You never have to remember any baroque language rules. Anything that makes sense, you can do. Also, even though it looks difficult, it is extremely easy to learn because K is directly translatable to English, in fact there is a K program that will do this automatically. For example to split a line by tabs you could write:cut:{1_'(&x=*x)_ x:"\t",x}
And this is read:cut gets function, 1 drop each, where x equals first x quantity, cut x. When X gets tab join x.
It may take a little getting used to, but with a month of K, my roommate and I were able to converse this way when describing K and you could see the picture developing in your head. It was amazing.
A unique feature of K is what is called the K tree. Unification is a very strong idea in K, so it unifies the idea of object, variables, attributes, namespaces, and dictionaries. A dictionary is a native K type. Each variable lives in a dictionary (somwhat like Python). These dictionaries are joined hierarchically and can be removed and added dynamically. All variables are on the K tree, too, so a new namespace is really just a dictionary on the K tree! This means that you can rearrange the K tree and change what functions get called. This is the most reflective language that I have ever seen (Python, Scheme, and CLisp come in a very close behind). All variables have attributes. All attributes are is a special dictionary attached to the variables (the language is so regular that this is really a namespace with a blank name so to refer to the attributes of a variable you say ns.var..attrib). And, of course, each attribute is just a variable so each of those can have attributes, too.
This interesting K tree leads to a very elegant GUI. Each variable can have an attribute named c (for class), and this can have certain values like `table, `check, `radio, `button, and others (the backtick ` is how you make a symbol). Lets take radio for an example. Then you would have another attribute o (for option) with possible values:r..o:`zero`one`two`three`four
These four lines would create a radio box with five choices, zero through four, and everytime you evaluated r whatever the radio was set to, r would evaluate to. Basically, each variable has a direct on-screen representation (they default to `data) and is directly manipulable.
r:r..o[1]
r..c:`radio
`show$`r
K also has the ideas of dependencies and triggers in the language, so if a..d:"1+b" then refering to a will dynamically calculate 1+b, but only when necessary (if you refer to a multiple times but b does not change between those references, a will only be calculated once and stored; K figures out the dependency graph for you). There are also triggers. If b..t:"a:b-1" then whenever b is assigned or modified then a will get the appropriate value. This trigger can be anything, such as a network operation or a gui command.
The language has some other unique features like an interesting callback oriented interprocess communication system and an on-the-fly optimizing vm.
Of course since it inherits some background from APL it has bulk operators, called adverbs, that modify functions in every conceivable way (much more powerful than APL or Perl). One of the signs of a good K programmer is one who knows how to do this and doesn't use any loops (KDB, the relational database, is written without any loops).
From functional languages K inherits higher-level functions and projections. Both which are very standard practices especially when combined with the bulk operators. b f[a;;c;]'d takes the four argument function f, fixes the first and third arguments projecting a function of two arguments, then applies it to each down the list of argument in b and d.
When you use K you truly are standing on the shoulders of giants. The person who wrote it, Arthur Whitney, has this amazing ability to identify the important pieces of a problem and simplify away the rest. The performance in K and KDB is incredibly; the simplicity and power of the language and the database is incredibly.
K runs on various flavors of Unix and NT, so people should take an open mind (I didn't have one at first and was very skeptical) and really try the language and try a new style of programming. Your code and thoughts on developing will never be the same.
-j -
What about K?I agree that most language are the same C derived POS. C was different and inventive when it was created. Lisp, APL, Prolog, and Smalltalk were all different when they were created. It seems like as time went on we started narrowing our field of vision and implementing the same languages: C to C++ to Java, what kind of intereting steps are those? At least Smalltalk to Self was a very interesting pushing of the boundries. Today, almost nobody pushes anything, except how similar their langauge is to C and why that it good. Even Python and Perl don't attempt to explore any new concepts, they are happy being a Frankenstein of older languages that people seemed to have forgetten about; name three new features of either language, just try to name one!
My sole exception to this is a language called K. Yes, it has its roots in APL and has added to the APL model from languages such as Lisp and Scheme, but it has some very interesting new features of its own.
K is very very very fast to write and the run. It blazes in both categories. There is a full relational database that is written in K, called KDB. It crushed Oracle on the TPC-B and TPC-D benchmarks in both speed and storage size, requiring only a few percent above the dataset size in overhead. It has native clustering and replication that allowed it to run on a 50 cpu Linux cluster loaded with 2.5 billion stock trades and quotes and have simple table scans (such as, select max price from trade) take under a second and multi-dimensional aggregations (such as, 100 first desc select sum size*price by sym from trade) take only 10 seconds. Starting the database cluster took a tenth of a second. It is SQL92 compliant, has an extended ultra-powerful query language called KSQL that makes writing queries very simple, and the stored procedure languages are K and C.
In bwk's language benchmarks, even though this is not the K strong point, the sum of the execution times were: K at 32 seconds, Perl at 95, Java at 300, and TCL above 1400. The lines of code to implement were: K at 9 lines, awk at 95, Perl at 96, TCL at 105, Scheme at 170, VB at 200, and Java at 350.
Yes, K can look like line noise, but unlike Perl, you get alot from this. First you get extreme code density and see the entire problem on the screen at once. I came from a Scheme background and Perl hurt my eyes, so I was very skeptical, but after my roommate persuaded me to look at K harder, I realized that this high code density made it very easy debug and write code. It is rumored that KDB is written in 26 files of code, each file consisting of a single screen of code, labeled a to z. Try doing that in any other language. The language is exceptionally regular. It is so logical and consistent that it takes a little getting used to. You never have to remember any baroque language rules. Anything that makes sense, you can do. Also, even though it looks difficult, it is extremely easy to learn because K is directly translatable to English, in fact there is a K program that will do this automatically. For example to split a line by tabs you could write:cut:{1_'(&x=*x)_ x:"\t",x}
And this is read:cut gets function, 1 drop each, where x equals first x quantity, cut x. When X gets tab join x.
It may take a little getting used to, but with a month of K, my roommate and I were able to converse this way when describing K and you could see the picture developing in your head. It was amazing.
A unique feature of K is what is called the K tree. Unification is a very strong idea in K, so it unifies the idea of object, variables, attributes, namespaces, and dictionaries. A dictionary is a native K type. Each variable lives in a dictionary (somwhat like Python). These dictionaries are joined hierarchically and can be removed and added dynamically. All variables are on the K tree, too, so a new namespace is really just a dictionary on the K tree! This means that you can rearrange the K tree and change what functions get called. This is the most reflective language that I have ever seen (Python, Scheme, and CLisp come in a very close behind). All variables have attributes. All attributes are is a special dictionary attached to the variables (the language is so regular that this is really a namespace with a blank name so to refer to the attributes of a variable you say ns.var..attrib). And, of course, each attribute is just a variable so each of those can have attributes, too.
This interesting K tree leads to a very elegant GUI. Each variable can have an attribute named c (for class), and this can have certain values like `table, `check, `radio, `button, and others (the backtick ` is how you make a symbol). Lets take radio for an example. Then you would have another attribute o (for option) with possible values:r..o:`zero`one`two`three`four
These four lines would create a radio box with five choices, zero through four, and everytime you evaluated r whatever the radio was set to, r would evaluate to. Basically, each variable has a direct on-screen representation (they default to `data) and is directly manipulable.
r:r..o[1]
r..c:`radio
`show$`r
K also has the ideas of dependencies and triggers in the language, so if a..d:"1+b" then refering to a will dynamically calculate 1+b, but only when necessary (if you refer to a multiple times but b does not change between those references, a will only be calculated once and stored; K figures out the dependency graph for you). There are also triggers. If b..t:"a:b-1" then whenever b is assigned or modified then a will get the appropriate value. This trigger can be anything, such as a network operation or a gui command.
The language has some other unique features like an interesting callback oriented interprocess communication system and an on-the-fly optimizing vm.
Of course since it inherits some background from APL it has bulk operators, called adverbs, that modify functions in every conceivable way (much more powerful than APL or Perl). One of the signs of a good K programmer is one who knows how to do this and doesn't use any loops (KDB, the relational database, is written without any loops).
From functional languages K inherits higher-level functions and projections. Both which are very standard practices especially when combined with the bulk operators. b f[a;;c;]'d takes the four argument function f, fixes the first and third arguments projecting a function of two arguments, then applies it to each down the list of argument in b and d.
When you use K you truly are standing on the shoulders of giants. The person who wrote it, Arthur Whitney, has this amazing ability to identify the important pieces of a problem and simplify away the rest. The performance in K and KDB is incredibly; the simplicity and power of the language and the database is incredibly.
K runs on various flavors of Unix and NT, so people should take an open mind (I didn't have one at first and was very skeptical) and really try the language and try a new style of programming. Your code and thoughts on developing will never be the same.
-j -
What about K?I agree that most language are the same C derived POS. C was different and inventive when it was created. Lisp, APL, Prolog, and Smalltalk were all different when they were created. It seems like as time went on we started narrowing our field of vision and implementing the same languages: C to C++ to Java, what kind of intereting steps are those? At least Smalltalk to Self was a very interesting pushing of the boundries. Today, almost nobody pushes anything, except how similar their langauge is to C and why that it good. Even Python and Perl don't attempt to explore any new concepts, they are happy being a Frankenstein of older languages that people seemed to have forgetten about; name three new features of either language, just try to name one!
My sole exception to this is a language called K. Yes, it has its roots in APL and has added to the APL model from languages such as Lisp and Scheme, but it has some very interesting new features of its own.
K is very very very fast to write and the run. It blazes in both categories. There is a full relational database that is written in K, called KDB. It crushed Oracle on the TPC-B and TPC-D benchmarks in both speed and storage size, requiring only a few percent above the dataset size in overhead. It has native clustering and replication that allowed it to run on a 50 cpu Linux cluster loaded with 2.5 billion stock trades and quotes and have simple table scans (such as, select max price from trade) take under a second and multi-dimensional aggregations (such as, 100 first desc select sum size*price by sym from trade) take only 10 seconds. Starting the database cluster took a tenth of a second. It is SQL92 compliant, has an extended ultra-powerful query language called KSQL that makes writing queries very simple, and the stored procedure languages are K and C.
In bwk's language benchmarks, even though this is not the K strong point, the sum of the execution times were: K at 32 seconds, Perl at 95, Java at 300, and TCL above 1400. The lines of code to implement were: K at 9 lines, awk at 95, Perl at 96, TCL at 105, Scheme at 170, VB at 200, and Java at 350.
Yes, K can look like line noise, but unlike Perl, you get alot from this. First you get extreme code density and see the entire problem on the screen at once. I came from a Scheme background and Perl hurt my eyes, so I was very skeptical, but after my roommate persuaded me to look at K harder, I realized that this high code density made it very easy debug and write code. It is rumored that KDB is written in 26 files of code, each file consisting of a single screen of code, labeled a to z. Try doing that in any other language. The language is exceptionally regular. It is so logical and consistent that it takes a little getting used to. You never have to remember any baroque language rules. Anything that makes sense, you can do. Also, even though it looks difficult, it is extremely easy to learn because K is directly translatable to English, in fact there is a K program that will do this automatically. For example to split a line by tabs you could write:cut:{1_'(&x=*x)_ x:"\t",x}
And this is read:cut gets function, 1 drop each, where x equals first x quantity, cut x. When X gets tab join x.
It may take a little getting used to, but with a month of K, my roommate and I were able to converse this way when describing K and you could see the picture developing in your head. It was amazing.
A unique feature of K is what is called the K tree. Unification is a very strong idea in K, so it unifies the idea of object, variables, attributes, namespaces, and dictionaries. A dictionary is a native K type. Each variable lives in a dictionary (somwhat like Python). These dictionaries are joined hierarchically and can be removed and added dynamically. All variables are on the K tree, too, so a new namespace is really just a dictionary on the K tree! This means that you can rearrange the K tree and change what functions get called. This is the most reflective language that I have ever seen (Python, Scheme, and CLisp come in a very close behind). All variables have attributes. All attributes are is a special dictionary attached to the variables (the language is so regular that this is really a namespace with a blank name so to refer to the attributes of a variable you say ns.var..attrib). And, of course, each attribute is just a variable so each of those can have attributes, too.
This interesting K tree leads to a very elegant GUI. Each variable can have an attribute named c (for class), and this can have certain values like `table, `check, `radio, `button, and others (the backtick ` is how you make a symbol). Lets take radio for an example. Then you would have another attribute o (for option) with possible values:r..o:`zero`one`two`three`four
These four lines would create a radio box with five choices, zero through four, and everytime you evaluated r whatever the radio was set to, r would evaluate to. Basically, each variable has a direct on-screen representation (they default to `data) and is directly manipulable.
r:r..o[1]
r..c:`radio
`show$`r
K also has the ideas of dependencies and triggers in the language, so if a..d:"1+b" then refering to a will dynamically calculate 1+b, but only when necessary (if you refer to a multiple times but b does not change between those references, a will only be calculated once and stored; K figures out the dependency graph for you). There are also triggers. If b..t:"a:b-1" then whenever b is assigned or modified then a will get the appropriate value. This trigger can be anything, such as a network operation or a gui command.
The language has some other unique features like an interesting callback oriented interprocess communication system and an on-the-fly optimizing vm.
Of course since it inherits some background from APL it has bulk operators, called adverbs, that modify functions in every conceivable way (much more powerful than APL or Perl). One of the signs of a good K programmer is one who knows how to do this and doesn't use any loops (KDB, the relational database, is written without any loops).
From functional languages K inherits higher-level functions and projections. Both which are very standard practices especially when combined with the bulk operators. b f[a;;c;]'d takes the four argument function f, fixes the first and third arguments projecting a function of two arguments, then applies it to each down the list of argument in b and d.
When you use K you truly are standing on the shoulders of giants. The person who wrote it, Arthur Whitney, has this amazing ability to identify the important pieces of a problem and simplify away the rest. The performance in K and KDB is incredibly; the simplicity and power of the language and the database is incredibly.
K runs on various flavors of Unix and NT, so people should take an open mind (I didn't have one at first and was very skeptical) and really try the language and try a new style of programming. Your code and thoughts on developing will never be the same.
-j -
What about K?I agree that most language are the same C derived POS. C was different and inventive when it was created. Lisp, APL, Prolog, and Smalltalk were all different when they were created. It seems like as time went on we started narrowing our field of vision and implementing the same languages: C to C++ to Java, what kind of intereting steps are those? At least Smalltalk to Self was a very interesting pushing of the boundries. Today, almost nobody pushes anything, except how similar their langauge is to C and why that it good. Even Python and Perl don't attempt to explore any new concepts, they are happy being a Frankenstein of older languages that people seemed to have forgetten about; name three new features of either language, just try to name one!
My sole exception to this is a language called K. Yes, it has its roots in APL and has added to the APL model from languages such as Lisp and Scheme, but it has some very interesting new features of its own.
K is very very very fast to write and the run. It blazes in both categories. There is a full relational database that is written in K, called KDB. It crushed Oracle on the TPC-B and TPC-D benchmarks in both speed and storage size, requiring only a few percent above the dataset size in overhead. It has native clustering and replication that allowed it to run on a 50 cpu Linux cluster loaded with 2.5 billion stock trades and quotes and have simple table scans (such as, select max price from trade) take under a second and multi-dimensional aggregations (such as, 100 first desc select sum size*price by sym from trade) take only 10 seconds. Starting the database cluster took a tenth of a second. It is SQL92 compliant, has an extended ultra-powerful query language called KSQL that makes writing queries very simple, and the stored procedure languages are K and C.
In bwk's language benchmarks, even though this is not the K strong point, the sum of the execution times were: K at 32 seconds, Perl at 95, Java at 300, and TCL above 1400. The lines of code to implement were: K at 9 lines, awk at 95, Perl at 96, TCL at 105, Scheme at 170, VB at 200, and Java at 350.
Yes, K can look like line noise, but unlike Perl, you get alot from this. First you get extreme code density and see the entire problem on the screen at once. I came from a Scheme background and Perl hurt my eyes, so I was very skeptical, but after my roommate persuaded me to look at K harder, I realized that this high code density made it very easy debug and write code. It is rumored that KDB is written in 26 files of code, each file consisting of a single screen of code, labeled a to z. Try doing that in any other language. The language is exceptionally regular. It is so logical and consistent that it takes a little getting used to. You never have to remember any baroque language rules. Anything that makes sense, you can do. Also, even though it looks difficult, it is extremely easy to learn because K is directly translatable to English, in fact there is a K program that will do this automatically. For example to split a line by tabs you could write:cut:{1_'(&x=*x)_ x:"\t",x}
And this is read:cut gets function, 1 drop each, where x equals first x quantity, cut x. When X gets tab join x.
It may take a little getting used to, but with a month of K, my roommate and I were able to converse this way when describing K and you could see the picture developing in your head. It was amazing.
A unique feature of K is what is called the K tree. Unification is a very strong idea in K, so it unifies the idea of object, variables, attributes, namespaces, and dictionaries. A dictionary is a native K type. Each variable lives in a dictionary (somwhat like Python). These dictionaries are joined hierarchically and can be removed and added dynamically. All variables are on the K tree, too, so a new namespace is really just a dictionary on the K tree! This means that you can rearrange the K tree and change what functions get called. This is the most reflective language that I have ever seen (Python, Scheme, and CLisp come in a very close behind). All variables have attributes. All attributes are is a special dictionary attached to the variables (the language is so regular that this is really a namespace with a blank name so to refer to the attributes of a variable you say ns.var..attrib). And, of course, each attribute is just a variable so each of those can have attributes, too.
This interesting K tree leads to a very elegant GUI. Each variable can have an attribute named c (for class), and this can have certain values like `table, `check, `radio, `button, and others (the backtick ` is how you make a symbol). Lets take radio for an example. Then you would have another attribute o (for option) with possible values:r..o:`zero`one`two`three`four
These four lines would create a radio box with five choices, zero through four, and everytime you evaluated r whatever the radio was set to, r would evaluate to. Basically, each variable has a direct on-screen representation (they default to `data) and is directly manipulable.
r:r..o[1]
r..c:`radio
`show$`r
K also has the ideas of dependencies and triggers in the language, so if a..d:"1+b" then refering to a will dynamically calculate 1+b, but only when necessary (if you refer to a multiple times but b does not change between those references, a will only be calculated once and stored; K figures out the dependency graph for you). There are also triggers. If b..t:"a:b-1" then whenever b is assigned or modified then a will get the appropriate value. This trigger can be anything, such as a network operation or a gui command.
The language has some other unique features like an interesting callback oriented interprocess communication system and an on-the-fly optimizing vm.
Of course since it inherits some background from APL it has bulk operators, called adverbs, that modify functions in every conceivable way (much more powerful than APL or Perl). One of the signs of a good K programmer is one who knows how to do this and doesn't use any loops (KDB, the relational database, is written without any loops).
From functional languages K inherits higher-level functions and projections. Both which are very standard practices especially when combined with the bulk operators. b f[a;;c;]'d takes the four argument function f, fixes the first and third arguments projecting a function of two arguments, then applies it to each down the list of argument in b and d.
When you use K you truly are standing on the shoulders of giants. The person who wrote it, Arthur Whitney, has this amazing ability to identify the important pieces of a problem and simplify away the rest. The performance in K and KDB is incredibly; the simplicity and power of the language and the database is incredibly.
K runs on various flavors of Unix and NT, so people should take an open mind (I didn't have one at first and was very skeptical) and really try the language and try a new style of programming. Your code and thoughts on developing will never be the same.
-j -
What about K?I agree that most language are the same C derived POS. C was different and inventive when it was created. Lisp, APL, Prolog, and Smalltalk were all different when they were created. It seems like as time went on we started narrowing our field of vision and implementing the same languages: C to C++ to Java, what kind of intereting steps are those? At least Smalltalk to Self was a very interesting pushing of the boundries. Today, almost nobody pushes anything, except how similar their langauge is to C and why that it good. Even Python and Perl don't attempt to explore any new concepts, they are happy being a Frankenstein of older languages that people seemed to have forgetten about; name three new features of either language, just try to name one!
My sole exception to this is a language called K. Yes, it has its roots in APL and has added to the APL model from languages such as Lisp and Scheme, but it has some very interesting new features of its own.
K is very very very fast to write and the run. It blazes in both categories. There is a full relational database that is written in K, called KDB. It crushed Oracle on the TPC-B and TPC-D benchmarks in both speed and storage size, requiring only a few percent above the dataset size in overhead. It has native clustering and replication that allowed it to run on a 50 cpu Linux cluster loaded with 2.5 billion stock trades and quotes and have simple table scans (such as, select max price from trade) take under a second and multi-dimensional aggregations (such as, 100 first desc select sum size*price by sym from trade) take only 10 seconds. Starting the database cluster took a tenth of a second. It is SQL92 compliant, has an extended ultra-powerful query language called KSQL that makes writing queries very simple, and the stored procedure languages are K and C.
In bwk's language benchmarks, even though this is not the K strong point, the sum of the execution times were: K at 32 seconds, Perl at 95, Java at 300, and TCL above 1400. The lines of code to implement were: K at 9 lines, awk at 95, Perl at 96, TCL at 105, Scheme at 170, VB at 200, and Java at 350.
Yes, K can look like line noise, but unlike Perl, you get alot from this. First you get extreme code density and see the entire problem on the screen at once. I came from a Scheme background and Perl hurt my eyes, so I was very skeptical, but after my roommate persuaded me to look at K harder, I realized that this high code density made it very easy debug and write code. It is rumored that KDB is written in 26 files of code, each file consisting of a single screen of code, labeled a to z. Try doing that in any other language. The language is exceptionally regular. It is so logical and consistent that it takes a little getting used to. You never have to remember any baroque language rules. Anything that makes sense, you can do. Also, even though it looks difficult, it is extremely easy to learn because K is directly translatable to English, in fact there is a K program that will do this automatically. For example to split a line by tabs you could write:cut:{1_'(&x=*x)_ x:"\t",x}
And this is read:cut gets function, 1 drop each, where x equals first x quantity, cut x. When X gets tab join x.
It may take a little getting used to, but with a month of K, my roommate and I were able to converse this way when describing K and you could see the picture developing in your head. It was amazing.
A unique feature of K is what is called the K tree. Unification is a very strong idea in K, so it unifies the idea of object, variables, attributes, namespaces, and dictionaries. A dictionary is a native K type. Each variable lives in a dictionary (somwhat like Python). These dictionaries are joined hierarchically and can be removed and added dynamically. All variables are on the K tree, too, so a new namespace is really just a dictionary on the K tree! This means that you can rearrange the K tree and change what functions get called. This is the most reflective language that I have ever seen (Python, Scheme, and CLisp come in a very close behind). All variables have attributes. All attributes are is a special dictionary attached to the variables (the language is so regular that this is really a namespace with a blank name so to refer to the attributes of a variable you say ns.var..attrib). And, of course, each attribute is just a variable so each of those can have attributes, too.
This interesting K tree leads to a very elegant GUI. Each variable can have an attribute named c (for class), and this can have certain values like `table, `check, `radio, `button, and others (the backtick ` is how you make a symbol). Lets take radio for an example. Then you would have another attribute o (for option) with possible values:r..o:`zero`one`two`three`four
These four lines would create a radio box with five choices, zero through four, and everytime you evaluated r whatever the radio was set to, r would evaluate to. Basically, each variable has a direct on-screen representation (they default to `data) and is directly manipulable.
r:r..o[1]
r..c:`radio
`show$`r
K also has the ideas of dependencies and triggers in the language, so if a..d:"1+b" then refering to a will dynamically calculate 1+b, but only when necessary (if you refer to a multiple times but b does not change between those references, a will only be calculated once and stored; K figures out the dependency graph for you). There are also triggers. If b..t:"a:b-1" then whenever b is assigned or modified then a will get the appropriate value. This trigger can be anything, such as a network operation or a gui command.
The language has some other unique features like an interesting callback oriented interprocess communication system and an on-the-fly optimizing vm.
Of course since it inherits some background from APL it has bulk operators, called adverbs, that modify functions in every conceivable way (much more powerful than APL or Perl). One of the signs of a good K programmer is one who knows how to do this and doesn't use any loops (KDB, the relational database, is written without any loops).
From functional languages K inherits higher-level functions and projections. Both which are very standard practices especially when combined with the bulk operators. b f[a;;c;]'d takes the four argument function f, fixes the first and third arguments projecting a function of two arguments, then applies it to each down the list of argument in b and d.
When you use K you truly are standing on the shoulders of giants. The person who wrote it, Arthur Whitney, has this amazing ability to identify the important pieces of a problem and simplify away the rest. The performance in K and KDB is incredibly; the simplicity and power of the language and the database is incredibly.
K runs on various flavors of Unix and NT, so people should take an open mind (I didn't have one at first and was very skeptical) and really try the language and try a new style of programming. Your code and thoughts on developing will never be the same.
-j -
What about K?I agree that most language are the same C derived POS. C was different and inventive when it was created. Lisp, APL, Prolog, and Smalltalk were all different when they were created. It seems like as time went on we started narrowing our field of vision and implementing the same languages: C to C++ to Java, what kind of intereting steps are those? At least Smalltalk to Self was a very interesting pushing of the boundries. Today, almost nobody pushes anything, except how similar their langauge is to C and why that it good. Even Python and Perl don't attempt to explore any new concepts, they are happy being a Frankenstein of older languages that people seemed to have forgetten about; name three new features of either language, just try to name one!
My sole exception to this is a language called K. Yes, it has its roots in APL and has added to the APL model from languages such as Lisp and Scheme, but it has some very interesting new features of its own.
K is very very very fast to write and the run. It blazes in both categories. There is a full relational database that is written in K, called KDB. It crushed Oracle on the TPC-B and TPC-D benchmarks in both speed and storage size, requiring only a few percent above the dataset size in overhead. It has native clustering and replication that allowed it to run on a 50 cpu Linux cluster loaded with 2.5 billion stock trades and quotes and have simple table scans (such as, select max price from trade) take under a second and multi-dimensional aggregations (such as, 100 first desc select sum size*price by sym from trade) take only 10 seconds. Starting the database cluster took a tenth of a second. It is SQL92 compliant, has an extended ultra-powerful query language called KSQL that makes writing queries very simple, and the stored procedure languages are K and C.
In bwk's language benchmarks, even though this is not the K strong point, the sum of the execution times were: K at 32 seconds, Perl at 95, Java at 300, and TCL above 1400. The lines of code to implement were: K at 9 lines, awk at 95, Perl at 96, TCL at 105, Scheme at 170, VB at 200, and Java at 350.
Yes, K can look like line noise, but unlike Perl, you get alot from this. First you get extreme code density and see the entire problem on the screen at once. I came from a Scheme background and Perl hurt my eyes, so I was very skeptical, but after my roommate persuaded me to look at K harder, I realized that this high code density made it very easy debug and write code. It is rumored that KDB is written in 26 files of code, each file consisting of a single screen of code, labeled a to z. Try doing that in any other language. The language is exceptionally regular. It is so logical and consistent that it takes a little getting used to. You never have to remember any baroque language rules. Anything that makes sense, you can do. Also, even though it looks difficult, it is extremely easy to learn because K is directly translatable to English, in fact there is a K program that will do this automatically. For example to split a line by tabs you could write:cut:{1_'(&x=*x)_ x:"\t",x}
And this is read:cut gets function, 1 drop each, where x equals first x quantity, cut x. When X gets tab join x.
It may take a little getting used to, but with a month of K, my roommate and I were able to converse this way when describing K and you could see the picture developing in your head. It was amazing.
A unique feature of K is what is called the K tree. Unification is a very strong idea in K, so it unifies the idea of object, variables, attributes, namespaces, and dictionaries. A dictionary is a native K type. Each variable lives in a dictionary (somwhat like Python). These dictionaries are joined hierarchically and can be removed and added dynamically. All variables are on the K tree, too, so a new namespace is really just a dictionary on the K tree! This means that you can rearrange the K tree and change what functions get called. This is the most reflective language that I have ever seen (Python, Scheme, and CLisp come in a very close behind). All variables have attributes. All attributes are is a special dictionary attached to the variables (the language is so regular that this is really a namespace with a blank name so to refer to the attributes of a variable you say ns.var..attrib). And, of course, each attribute is just a variable so each of those can have attributes, too.
This interesting K tree leads to a very elegant GUI. Each variable can have an attribute named c (for class), and this can have certain values like `table, `check, `radio, `button, and others (the backtick ` is how you make a symbol). Lets take radio for an example. Then you would have another attribute o (for option) with possible values:r..o:`zero`one`two`three`four
These four lines would create a radio box with five choices, zero through four, and everytime you evaluated r whatever the radio was set to, r would evaluate to. Basically, each variable has a direct on-screen representation (they default to `data) and is directly manipulable.
r:r..o[1]
r..c:`radio
`show$`r
K also has the ideas of dependencies and triggers in the language, so if a..d:"1+b" then refering to a will dynamically calculate 1+b, but only when necessary (if you refer to a multiple times but b does not change between those references, a will only be calculated once and stored; K figures out the dependency graph for you). There are also triggers. If b..t:"a:b-1" then whenever b is assigned or modified then a will get the appropriate value. This trigger can be anything, such as a network operation or a gui command.
The language has some other unique features like an interesting callback oriented interprocess communication system and an on-the-fly optimizing vm.
Of course since it inherits some background from APL it has bulk operators, called adverbs, that modify functions in every conceivable way (much more powerful than APL or Perl). One of the signs of a good K programmer is one who knows how to do this and doesn't use any loops (KDB, the relational database, is written without any loops).
From functional languages K inherits higher-level functions and projections. Both which are very standard practices especially when combined with the bulk operators. b f[a;;c;]'d takes the four argument function f, fixes the first and third arguments projecting a function of two arguments, then applies it to each down the list of argument in b and d.
When you use K you truly are standing on the shoulders of giants. The person who wrote it, Arthur Whitney, has this amazing ability to identify the important pieces of a problem and simplify away the rest. The performance in K and KDB is incredibly; the simplicity and power of the language and the database is incredibly.
K runs on various flavors of Unix and NT, so people should take an open mind (I didn't have one at first and was very skeptical) and really try the language and try a new style of programming. Your code and thoughts on developing will never be the same.
-j -
More justification of OO being a phony.
Each instance of an object in a program is a model. You think about what it is conceptually that you are modeling and derive classes for those concepts. In practice these objects usually have hierarchial and recursive relationships.
Indeed I don't beleive 9 out of 10 developers using an OO language are actually writing OO code. It's modular at best. To use it effectively is non-trivial.
First, but what is the purpose of making modeling easier? Isn't the goal of almost all new language to make programming easier in some respect.
Second, I am not one one of the "objects as real-world entities" people. I believe strongly that to effectively partition work -- to make programming more tractable -- many times you will create work partitions and create objects that have no correlation to the real-world. A contrived example would be a deck of cards. Imagine that you make each card an object, then you have a deck object, and a dealer object. In the real-world the dealer would shuffle a deck, but in our code this responsibility would fall on the deck, but the dealing mechanics would probably rest with the dealer.
I am the kind of programmer that believes in active objects. I am the kind of person that makes everything an object. I make all my constants objects (for type checking and I cannot stand it when people use integers for this) and slowly through most development I have done, these once data-only objects gain methods and no longer become just place-holding constants.
I think that to effectively use the OO paradigm, you need to really abuse it. I use dynamic dispatch for almost all my flow control; try finding even an 'if' statement in my code. I even have the_empty_tree_node objects that don't do anything except return so my tree recursions don't need a base case. This abuse is non-trivial; you are correct. I have spent alot of time abusing C++ and Java for it to come more naturally to me.
However, I don't think it will ever come as naturally to me as functional programming. I am not saying that I do not use OO techniques in my functional code. You can see a sort of dynamic dispatch mimicked by function passing and you can see pseudo-object in functions that return different functions, each operating on the same lexically captured bindings. I am just saying that OO is over-hyped and I feel that the functional paradigm effectively captures OO benefits.
APL's style of data oriented processing has also shown to be much faster and require far less code. Why develop a library when it only takes a single line implement the functionality? Also when you language supports dynamically evaluating a string code reuse in terms of modifying a string then evaluating is phenominal (plus the on-the-fly compiler gets to optimize the hell out of it). -
Re:APL - A mathematical programming language
I'll second that.
Morgan Stanley has a GPLed implementation for most Unices called A+.
For a reference, pick up APL: the Language and its Usage, Learning APL: An Array Processing Language,
or Ken Iverson's A Programming Language, from which the name of the language is derived.
A modern variant which uses ASCII characters is K. Free as in beer, for FreeBSD, AIX, Linux, Win32. -
Re:OT: question about hyper operators.
Yes, I see your point about wanting to stop evaluation periodically; laziness is a very important concept to allow. POE also sound very cool.
I do not use nearly as much Perl as I should, so I cannot comment as to specific Perl usages. However, you should never run out of memory with hyper operators, since it is too easy for the language to play swap games itself. This is my opinion, at least.
I too use very large data sets at work tied to databases. I do massive text processing tasks working as a core developer at one of the largest search engines on the web. For this taks we use a database called KDB.
<PLUG>
KDB is entirely based around bulk data operations. This database is written in K. It is the fastest database I know of, doing 50,000 transactions a second on a 100 million records.
</PLUG> -
Re:OT: question about hyper operators.
Yes, I see your point about wanting to stop evaluation periodically; laziness is a very important concept to allow. POE also sound very cool.
I do not use nearly as much Perl as I should, so I cannot comment as to specific Perl usages. However, you should never run out of memory with hyper operators, since it is too easy for the language to play swap games itself. This is my opinion, at least.
I too use very large data sets at work tied to databases. I do massive text processing tasks working as a core developer at one of the largest search engines on the web. For this taks we use a database called KDB.
<PLUG>
KDB is entirely based around bulk data operations. This database is written in K. It is the fastest database I know of, doing 50,000 transactions a second on a 100 million records.
</PLUG> -
Re:OT: question about hyper operators.
Yes, I see your point about wanting to stop evaluation periodically; laziness is a very important concept to allow. POE also sound very cool.
I do not use nearly as much Perl as I should, so I cannot comment as to specific Perl usages. However, you should never run out of memory with hyper operators, since it is too easy for the language to play swap games itself. This is my opinion, at least.
I too use very large data sets at work tied to databases. I do massive text processing tasks working as a core developer at one of the largest search engines on the web. For this taks we use a database called KDB.
<PLUG>
KDB is entirely based around bulk data operations. This database is written in K. It is the fastest database I know of, doing 50,000 transactions a second on a 100 million records.
</PLUG> -
OT: question about hyper operators.
First, let me admit a bias: I am madly in love with an APL like language called K. It has destroyed some unreasonable language biases I had in the past by being simply an amazing language. I have become used to the hyper operator concept, even though in K functions are called verbs and the hyper decorations are called adverbs.
My question is about how hyperness is applied to hyperness and how you make hyperness apply to only one side of the operator. Here are how you do things like this in K; are these possible in the new Perl? If not, these would be monumental omissions.
' is read each
\: is read each left
Given the lists x:1 2 3 and y:10 20 30 and value z:100
x+'y is 11 22 33
x+\:z is 101 102 103
z+/:y is 110 120 130
but these can all be done implicitly, so you really do not need the decorations: x+y, x+z, and z+y are fine. This allows you walk down only a single list, while aggregating results.
x+\:y is (11 21 31; 12 22 32; 13; 23 33)
x+/:y is (11 12 13; 21 22 23; 31 32 33)
Then you can also walk down the left then right sides of list by \:/:(left each right each).
You can walk down a list unarily:
There are other adverbs and they can be combined to modify each other arbitrarily. This winds up being an incredibly powerway to write programs. It removes the programer from the burden of flow control and compacts code enormously. Think of removing all the loops from your code and replacing them with a couple charcters, instead. -
Everybody writes C in every language, anyways.
What difference does changing languages to most of you mean? 99% of everybody I see write a program in any language, writes C. I have seen hundreds of self-proclaimed "real Scheme hackers" use a dozen local variables in function; complex state is their definition of incredible code. I have seen even more "Perl gurus" write C with line noise, not understanding the unique features of perl that make it almost a concatenative language at times. I cannot count the number of C++ or Java twirps that have never seen the simple "TheEmptyTreeNode" as control-flow (or even worse, the ones that think it is poor style) -- these are usually the people that have never thought of the idea of removing explicit control flow from their code. Has everybody forgotten about the idea of constant data and as little state as possible, keeping with the Knuth maxim of code being as simple as possible (but no simplier). Has C wedged everybody's mind into thinking that every segment of code must be littered with control statements. Here is a challenge next time you code: write the code with as few control statements as possible, using every idiom you know and trick you can think of to remove all the if, switch, for, while, do, and related from your code and try to treat everything generically.
As much as I have a love-hate relationship with a certain APL derivative, they have the best programmers I have ever seen. They don't worry about performance, except when needed (and they have some extreme views of what is truly fast). They use a few big generic data structures and operate on them uniformly; they do not litter the landscape with a thousand different structures (that are all mostly similar and should be handled the same anyways). They write very terse code, where one screen of text can tell you the entire picture; nobody has to page through 30 pages of code to understand even the most complex operations. Why cannot more programmers be like this? -
Views on other minimalist projects: pinky and j/kThe Q:
Have you seen other projects that aim towards minimalization and what do you think of the direction they have choosen to follow? What projects (commercial, univerisity, or open) do you see as heading in a very good direction?
The Elaboration:
Specifically, on the hardware side, the Pinky 1 bit processor with 4 instructions seems near your chip design philosophy: small and smaller. Have you had a chance to look at the ideas behind it?The pinky processor is a chip that's designed to be extremely parallelizable, that is, it is designed to work extremely well in groups. Developed by Dave Taylor, they are extremely simple chips made to be small and efficient, and fast.
And on the software side, do you see any programming languages headed in The Right Direction? Such as the APL branches of J and K, where the community sees code concision playing a very important role in understanding and maintainability. [A slight warning: I am now a convert of the K and KDB religion, so I am definitely biased. If Arthur Whitney can out code Oracle, it must say something about K; my fragile little mind is still disturbed by some of his golf scores.]
The Thanks:
Also, thanks for Fourth. I arrived there via excursions into PostScript and Joy and try to use it often.
-j -
why aren't you using kdb from kx systems?the kdb faq, from kx Systems.
- What is Kdb ?
Kdb is an extremely fast RDBMS extended for time-series analysis.
- Does Kdb support SQL92, ODBC and JDBC ?
Yes.
- Is Kdb a read-only RDBMS ?
No. Kdb is very fast for OLTP (online transaction processing). For example, it runs over 50,000 ATM-style transactions per second logged to disk with full recovery on a single cpu. This was against a database of over 100,000,000 accounts, tellers and branches. Kdb can do batch updates at several hundred thousand records per second per cpu.
- Is Kdb a memory resident RDBMS ?
No. Kdb has minimal memory requirements and is very fast from disk. For example, it ran the gigabyte TPC-D (an industry standard decision support benchmark) queries and updates on a 200MHZ PC with 64 megabytes of memory, an ultrawide SCSI controller and four disk drives many times faster than the best published results at a fraction the cost.
- What about time series ?
Kdb handles much more than just SQL92 tables. Online analytical processing (OLAP) on multi-dimensional arrays is done with our extended SQL language, KSQL. For example, on the 35 megabyte OLAP APB-1 benchmark queries , Kdb ran 12,000 queries per minute with no precalculation.
- 6. Since Kdb is so fast, does it require more storage ?
No. Kdb is simple and will often store just the raw data . For example, in TPC-D, the published results required storage between 3 and 10 times the raw data. The Kdb factor is a little over one. Some OLAP tools require (for fast queries) massive precalculations. For example, in APB-1 some expanded the 35 megabytes of input data to many gigabytes. Kdb aggregates relations (extended with time series fields) so fast that precalculation is often obviated. Certainly when the raw data is less than a few gigabytes.
- Is there a parallel version ?
Yes. Although Kdb can handle much larger databases than other database products without requiring parallel processing, there is a parallel version for the largest applications. Kdb scales.
--
http://kx.com
taylor:{+/y**\1.0,x%1+!-1+#y} - What is Kdb ?
-
why aren't you using kdb from kx systems?the kdb faq, from kx Systems.
- What is Kdb ?
Kdb is an extremely fast RDBMS extended for time-series analysis.
- Does Kdb support SQL92, ODBC and JDBC ?
Yes.
- Is Kdb a read-only RDBMS ?
No. Kdb is very fast for OLTP (online transaction processing). For example, it runs over 50,000 ATM-style transactions per second logged to disk with full recovery on a single cpu. This was against a database of over 100,000,000 accounts, tellers and branches. Kdb can do batch updates at several hundred thousand records per second per cpu.
- Is Kdb a memory resident RDBMS ?
No. Kdb has minimal memory requirements and is very fast from disk. For example, it ran the gigabyte TPC-D (an industry standard decision support benchmark) queries and updates on a 200MHZ PC with 64 megabytes of memory, an ultrawide SCSI controller and four disk drives many times faster than the best published results at a fraction the cost.
- What about time series ?
Kdb handles much more than just SQL92 tables. Online analytical processing (OLAP) on multi-dimensional arrays is done with our extended SQL language, KSQL. For example, on the 35 megabyte OLAP APB-1 benchmark queries , Kdb ran 12,000 queries per minute with no precalculation.
- 6. Since Kdb is so fast, does it require more storage ?
No. Kdb is simple and will often store just the raw data . For example, in TPC-D, the published results required storage between 3 and 10 times the raw data. The Kdb factor is a little over one. Some OLAP tools require (for fast queries) massive precalculations. For example, in APB-1 some expanded the 35 megabytes of input data to many gigabytes. Kdb aggregates relations (extended with time series fields) so fast that precalculation is often obviated. Certainly when the raw data is less than a few gigabytes.
- Is there a parallel version ?
Yes. Although Kdb can handle much larger databases than other database products without requiring parallel processing, there is a parallel version for the largest applications. Kdb scales.
--
http://kx.com
taylor:{+/y**\1.0,x%1+!-1+#y} - What is Kdb ?
-
kx.com & kdbkx.com:
"50,000 ATM-style transactions per second logged to disk with full recovery on a single cpu. This was against a database of over 100,000,000"
"on the 35 megabyte OLAP APB-1 benchmark queries, Kdb ran 12,000 queries per minute with no precalculation."
"TPC/B benchmark that achieves 25,000 transactions per second with full recoverability and TCP/IP overhead on an (167 Megahertz) UltraSparc I."
and you can eat it too:"K is also small. The entire runtime system, written in C, fits in 300,000 bytes."
-j
--
http://kx.com
taylor:{+/y**\1.0,x%1+!-1+#y} -
Re:NoteThere is also a derivative language of J, called (wait for it) "K", which also uses ASCII syntax. (www.kx.com)
It's not free, or even cheap
On the plus side, you can download nice PDF versions of their reference manual and user guide for free, and hopefully the syntax is close to J.
-
j & kI haven't seen anyone mention j or k , both of which are derivitives (more or less) of APL and both of which are highly functional in approach.
and how often nowadays do you see a programming environment that can ship on a floppy disk? The entirety of K was under 200k the last I checked, and it has a smoking database engine. J also is amazingly fast.