Domain: kx.com
Stories and comments across the archive that link to kx.com.
Comments · 76
-
Re:More accurate
Kdb+ from http://kx.com/ a column database.
-
Re: Web Searches For These Suck
You should try Q and K.. http://kx.com/software-downloa...
-
Re:okay...?
KDB anyone?
-
Re:Wall Street uses a lot of PerlActually, it's even better, Wall Street uses tons and tons of the relatively obscure language K (and its related database product KDB or Q). This language looks like so: example xml parser in K (notice that the code is up until the lone "\", then the rest is just documentation).
The code above, and a lot of code like it, is actually used in production work at Wall Street. K actually has a lot of great merits, being a very efficient APL descendant, but it is still kind of fun that Wall Street does use a language far less readable than Perl as a core computation language for a lot of these kinds of models
:) -
Re:I really want a copy of this...
You haven't heard of kx then: http://kx.com/q/d/q1.htm
-
Re:Are relations obsolete?It depends on the application. I have done quite a bit of work with Jean-Claude Wippler metakit database. It is essentially a column based store with some clever underpinnings to make it relational as well. We mainly used it to track molecules and assay results coming off of our plate-readers. The nice thing is that this streaming data can be saved to disk and used in our LIMS system. What it does really well is scan and sort columns, the throughput is really quite amazing. It gets pretty slow for relational queries, but has the benefit that results of queries are persistant in the sense you can chain them together in a relational algebra fashion.
When I was getting interested in this, I also came across kdb a relational database based on the K programming language. kdb is also a column based store with one huge benefit, the interpreter that analyzes the columns,K fits inside a level 1 cache. The throughput of kdb is immense which is why it is mainly used to track and analyze financial data and other streaming and real-time data. It used to be you could try it out for free, I'm not sure what the state is currently, but I suggest at least giving it a try, it's not cheap but it certainly changed my view of databases.
Just my two cents. -
Parallel Programming is EASY (APL, J, K langauges)
I've been programming using parallel techniques for more than 25 years using APL.
I was so imnpressed with APL that I implemented an APL like derivative language
called Simmunity which is a hybrid compiler for an APL like syntax. Simmunity
can be targeted at a parallel processor implemented in FPGA's that provides
multiple simultaneously executing vector/matrix operations in parallel...
(stay tuned for more info)
The quintessential parallel database programming language available is clearly
KxSystems Q and KDB+
See the faq at http://kx.com/faq/ and tutorial at http://kx.com/q/d/primer.htm
Ken Iverson invented APL and then the J language which anyone interested in
really discussing parallel programming should look at closely. The Connection
Machine LISP had many of the APL operators/functions and certainly Map/Reduce
that it added to provide a convienent parallel expression langauge
Most programmers think in terms of sequential code execution with threads and
processes. APL incourages the programmer to conceive of programming using
vector and matrix operations to process strings and numbers and manipulate
data like a spreadsheet. An application that people might be familiar with
is Lotus Improv which provided naturally parallel expressions based on a subset
of APL operations.
Cheers, Simbuddha -
Parallel Programming is EASY (APL, J, K langauges)
I've been programming using parallel techniques for more than 25 years using APL.
I was so imnpressed with APL that I implemented an APL like derivative language
called Simmunity which is a hybrid compiler for an APL like syntax. Simmunity
can be targeted at a parallel processor implemented in FPGA's that provides
multiple simultaneously executing vector/matrix operations in parallel...
(stay tuned for more info)
The quintessential parallel database programming language available is clearly
KxSystems Q and KDB+
See the faq at http://kx.com/faq/ and tutorial at http://kx.com/q/d/primer.htm
Ken Iverson invented APL and then the J language which anyone interested in
really discussing parallel programming should look at closely. The Connection
Machine LISP had many of the APL operators/functions and certainly Map/Reduce
that it added to provide a convienent parallel expression langauge
Most programmers think in terms of sequential code execution with threads and
processes. APL incourages the programmer to conceive of programming using
vector and matrix operations to process strings and numbers and manipulate
data like a spreadsheet. An application that people might be familiar with
is Lotus Improv which provided naturally parallel expressions based on a subset
of APL operations.
Cheers, Simbuddha -
Re:Given that...
Can you compare kdb+ ( of which you're probably aware, but if not - http://kx.com/ ) to Vertica ?
-
see also KDB used by most financial institutions
See also KDB from kx.com.
It's the database used by most large financial institutions (Fidelity, Merrill Lynch, UBS, etc); and stores all it's variables on the global K stack.
For thy types of database work stock traders do, it's orders of magnitude faster than the Oracles, etc of the world.
http://kx.com/
http://www.kuro5hin.org/?op=displaystory;sid=2002/ 11/14/22741/791 -
Re:What would be cool...
We're doing something like this at NYU:
NYU Query by Humming
There were talks last summer with AOL about expanding our database massively (millions of songs) for some sort of music downloading service, and a mobile phone application was discussed. Essentially the phone would do the signal processing on the recording of the voice, send a small string to the server to query, and the server would return a list of matches. Phones can do this now, but it's often horribly slow as FFT libraries are in their infancy on phones and differ greatly across models (at least that was the state of affairs a year ago... surely things have changed since).
This isn't the same as an iPod query-by-humming application, though, where the searching would have to be performed on the device itself. I've never developed for the iPod so I can't speak to its speed compared to phones, but the searching part probably wouldn't be too bad. Our database is kdb running on a wimpy P4 2.2 with 512mb ram (!) and though the data set is quite small, searches are very fast and it should scale well. (Heck, k and kdb binaries are so tiny (~200kb) they could just run on the device!) -
Re:database?
use q (kx.com) to minimize access times.
http://kx.com/q/d/kdb+.htm -
Re:ugh...
try q, http://kx.com/q/d/q.htm
-
Re:Multi-Terabyte Data Warehouse and MySQL
www.kx.com,
Celko's article
http://www.intelligententerprise.com/010327/celko_ online.jhtml
read about tick
http://www.kx.com/products/kdb+tick.php
q is used for terabyte datawarehouses on wall street. realtime algorithmic trading.
milan's page.
http://homepage.hispeed.ch/milano/k4kdb+.htm -
bad bad bad
when will people learn that SQL was created in a time of ram scarcity? Oracle was designed back when machines had 4 or 8 MB of RAM __tops__. New developers should experiment with vector based systems instead of row based systems. Before anyone boohaahaas this as a toy, I recommend you look at www.kx.com customer's list, all the top Ibanks and bond houses in the __world__ use it. Lehman brothers has a 50 TB bond datbase that uses this technology.
http://www.kx.com/
http://www.kx.com/a/kdb/document/contention.txt
read up on http://www.kx.com/ they have a almost fully compliant sql engine written in 200K of code. The interpreter fits in a couple lines of cache.
This is the original J interpreter (written by Arthur Whitney), it looks like line noise, but use the 'indent' command and you will see its beauty:
typedef char C;typedef long I; typedef struct a{I t,r,d[3],p[2];}*A; #define P printf #define R return #define V1(f) A f(w)A w; #define V2(f) A f(a,w)A a,w; #define DO(n,x) {I i=3D0,_n=3D(n);for(;it=3Dt,z->r=3Dr,mv(z->d,d,r); R z;} V1(iota){I n=3D*w->p;A z=3Dga(0,1,&n);DO(n,z->p[i]=3Di);R z;} V2(plus){I r=3Dw->r,*d=3Dw->d,n=3Dtr(r,d);A z=3Dga(0,r,d); DO(n,z->p[i]=3Da->p[i]+w->p[i]);R z;} V2(from){I r=3Dw->r-1,*d=3Dw->d+1,n=3Dtr(r,d); A z=3Dga(w->t,r,d);mv(z->p,w->p+(n**a->p),n);R z;} V1(box){A z=3Dga(1,0,0);*z->p=3D(I)w;R z;} V2(cat){I an=3Dtr(a->r,a->d),wn=3Dtr(w->r,w->d),n=3Dan+wn; A z=3Dga(w->t,1,&n);mv(z->p,a->p,an);mv(z->p+an,w->p ,wn);R z;} V2(find)true V2(rsh){I r=3Da->r?*a->d:1,n=3Dtr(r,a->p),wn=3Dtr(w->r,w->d) ; A z=3Dga(w->t,r,a->p);mv(z->p,w->p,wn=3Dn>wn?wn:n); if(n-=3Dwn)mv(z->p+wn,z->p,n);R z;} V1(sha){A z=3Dga(0,1,&w->r);mv(z->p,w->d,w->r);R z;} V1(id){R w;}V1(size){A z=3Dga(0,0,0);*z->p=3Dw->r?*w->d:1;R z;} pi(i){P("%d ",i);}nl(){P("\n");} pr(w)A w;{I r=3Dw->r,*d=3Dw->d,n=3Dtr(r,d);DO(r,pi(d[i]));nl() ; if(w->t)DO(n,P("p[i]))else DO(n,pi(w->p[i]));nl();}
C vt[]=3D"+{~=3D'a'&&a'9')R 0;z=3Dga(0,0,0);*z->p=3Dc-'0';R z;} verb(c){I i=3D0;for(;vt[i];)if(vt[i++]=3D=3Dc)R i;R 0;} I *wd(s)C *s;{I a,n=3Dstrlen(s),*e=3Dma(n+1);C c; DO(n,e[i]=3D(a=3Dnoun(c=3Ds[i]))?a:(a=3Dverb(c))?a :c);e[n]=3D0;R e;}
main(){C s[99];while(gets(s))pr(ex(wd(s)));}
Here is another excellent page:
http://www.kuro5hin.org/story/2002/8/30/175531/763 -
bad bad bad
when will people learn that SQL was created in a time of ram scarcity? Oracle was designed back when machines had 4 or 8 MB of RAM __tops__. New developers should experiment with vector based systems instead of row based systems. Before anyone boohaahaas this as a toy, I recommend you look at www.kx.com customer's list, all the top Ibanks and bond houses in the __world__ use it. Lehman brothers has a 50 TB bond datbase that uses this technology.
http://www.kx.com/
http://www.kx.com/a/kdb/document/contention.txt
read up on http://www.kx.com/ they have a almost fully compliant sql engine written in 200K of code. The interpreter fits in a couple lines of cache.
This is the original J interpreter (written by Arthur Whitney), it looks like line noise, but use the 'indent' command and you will see its beauty:
typedef char C;typedef long I; typedef struct a{I t,r,d[3],p[2];}*A; #define P printf #define R return #define V1(f) A f(w)A w; #define V2(f) A f(a,w)A a,w; #define DO(n,x) {I i=3D0,_n=3D(n);for(;it=3Dt,z->r=3Dr,mv(z->d,d,r); R z;} V1(iota){I n=3D*w->p;A z=3Dga(0,1,&n);DO(n,z->p[i]=3Di);R z;} V2(plus){I r=3Dw->r,*d=3Dw->d,n=3Dtr(r,d);A z=3Dga(0,r,d); DO(n,z->p[i]=3Da->p[i]+w->p[i]);R z;} V2(from){I r=3Dw->r-1,*d=3Dw->d+1,n=3Dtr(r,d); A z=3Dga(w->t,r,d);mv(z->p,w->p+(n**a->p),n);R z;} V1(box){A z=3Dga(1,0,0);*z->p=3D(I)w;R z;} V2(cat){I an=3Dtr(a->r,a->d),wn=3Dtr(w->r,w->d),n=3Dan+wn; A z=3Dga(w->t,1,&n);mv(z->p,a->p,an);mv(z->p+an,w->p ,wn);R z;} V2(find)true V2(rsh){I r=3Da->r?*a->d:1,n=3Dtr(r,a->p),wn=3Dtr(w->r,w->d) ; A z=3Dga(w->t,r,a->p);mv(z->p,w->p,wn=3Dn>wn?wn:n); if(n-=3Dwn)mv(z->p+wn,z->p,n);R z;} V1(sha){A z=3Dga(0,1,&w->r);mv(z->p,w->d,w->r);R z;} V1(id){R w;}V1(size){A z=3Dga(0,0,0);*z->p=3Dw->r?*w->d:1;R z;} pi(i){P("%d ",i);}nl(){P("\n");} pr(w)A w;{I r=3Dw->r,*d=3Dw->d,n=3Dtr(r,d);DO(r,pi(d[i]));nl() ; if(w->t)DO(n,P("p[i]))else DO(n,pi(w->p[i]));nl();}
C vt[]=3D"+{~=3D'a'&&a'9')R 0;z=3Dga(0,0,0);*z->p=3Dc-'0';R z;} verb(c){I i=3D0;for(;vt[i];)if(vt[i++]=3D=3Dc)R i;R 0;} I *wd(s)C *s;{I a,n=3Dstrlen(s),*e=3Dma(n+1);C c; DO(n,e[i]=3D(a=3Dnoun(c=3Ds[i]))?a:(a=3Dverb(c))?a :c);e[n]=3D0;R e;}
main(){C s[99];while(gets(s))pr(ex(wd(s)));}
Here is another excellent page:
http://www.kuro5hin.org/story/2002/8/30/175531/763 -
bad bad bad
when will people learn that SQL was created in a time of ram scarcity? Oracle was designed back when machines had 4 or 8 MB of RAM __tops__. New developers should experiment with vector based systems instead of row based systems. Before anyone boohaahaas this as a toy, I recommend you look at www.kx.com customer's list, all the top Ibanks and bond houses in the __world__ use it. Lehman brothers has a 50 TB bond datbase that uses this technology.
http://www.kx.com/
http://www.kx.com/a/kdb/document/contention.txt
read up on http://www.kx.com/ they have a almost fully compliant sql engine written in 200K of code. The interpreter fits in a couple lines of cache.
This is the original J interpreter (written by Arthur Whitney), it looks like line noise, but use the 'indent' command and you will see its beauty:
typedef char C;typedef long I; typedef struct a{I t,r,d[3],p[2];}*A; #define P printf #define R return #define V1(f) A f(w)A w; #define V2(f) A f(a,w)A a,w; #define DO(n,x) {I i=3D0,_n=3D(n);for(;it=3Dt,z->r=3Dr,mv(z->d,d,r); R z;} V1(iota){I n=3D*w->p;A z=3Dga(0,1,&n);DO(n,z->p[i]=3Di);R z;} V2(plus){I r=3Dw->r,*d=3Dw->d,n=3Dtr(r,d);A z=3Dga(0,r,d); DO(n,z->p[i]=3Da->p[i]+w->p[i]);R z;} V2(from){I r=3Dw->r-1,*d=3Dw->d+1,n=3Dtr(r,d); A z=3Dga(w->t,r,d);mv(z->p,w->p+(n**a->p),n);R z;} V1(box){A z=3Dga(1,0,0);*z->p=3D(I)w;R z;} V2(cat){I an=3Dtr(a->r,a->d),wn=3Dtr(w->r,w->d),n=3Dan+wn; A z=3Dga(w->t,1,&n);mv(z->p,a->p,an);mv(z->p+an,w->p ,wn);R z;} V2(find)true V2(rsh){I r=3Da->r?*a->d:1,n=3Dtr(r,a->p),wn=3Dtr(w->r,w->d) ; A z=3Dga(w->t,r,a->p);mv(z->p,w->p,wn=3Dn>wn?wn:n); if(n-=3Dwn)mv(z->p+wn,z->p,n);R z;} V1(sha){A z=3Dga(0,1,&w->r);mv(z->p,w->d,w->r);R z;} V1(id){R w;}V1(size){A z=3Dga(0,0,0);*z->p=3Dw->r?*w->d:1;R z;} pi(i){P("%d ",i);}nl(){P("\n");} pr(w)A w;{I r=3Dw->r,*d=3Dw->d,n=3Dtr(r,d);DO(r,pi(d[i]));nl() ; if(w->t)DO(n,P("p[i]))else DO(n,pi(w->p[i]));nl();}
C vt[]=3D"+{~=3D'a'&&a'9')R 0;z=3Dga(0,0,0);*z->p=3Dc-'0';R z;} verb(c){I i=3D0;for(;vt[i];)if(vt[i++]=3D=3Dc)R i;R 0;} I *wd(s)C *s;{I a,n=3Dstrlen(s),*e=3Dma(n+1);C c; DO(n,e[i]=3D(a=3Dnoun(c=3Ds[i]))?a:(a=3Dverb(c))?a :c);e[n]=3D0;R e;}
main(){C s[99];while(gets(s))pr(ex(wd(s)));}
Here is another excellent page:
http://www.kuro5hin.org/story/2002/8/30/175531/763 -
Kdb+
Kdb+ by KX Systems (http://www.kx.com/ is by far and away the best thing for this. Its main use is to store tick data from financial markets, and is excellent at this (if expensive).
From how you descibed your needs, this would probably bit the bill.. -
raw speed, you say?
Check out K Database. I wonder if anything beats it. And yes, it has quite powerful SP language.
Two disadvantages, though: it wants lots of RAM, and it costs much. -
Coding speed
If it was about coding speed, the question would be: "Which non-special character APL-family-language to use, J (http://www.jsoftware.com/) or Q (http://www.kx.com/)(formerly known as K)
-
Re:a little easy but...
Ok, a few pointers - unfortunately the ones I am familiar with offhand are fairly academic languages. And now that I look for them, it appears that other languages do not provide the same sort of lists of 'training problems'.
Therefore, a great way to learn a language that would also help others would be to take the k language idiom list and build a similar one implementing the problems in whatever language you are trying to learn. Maybe even post the resulting idiom lists for various languages on sourceforge... Heck, if someone starts a project, I might even contribute solutions for a couple of languages. ;-)
APL idiom library link
K language idiom list in MS word format -
Re:Don't underestimate optimizations
kx have an interesting product.. 10-100x as fast as you "legacy " database, by which they mean oracle et al of course, and specialized for time-series (which is a big deal if you deal with those).
I don't think oracle is quite the fastest general-purpose SQL(semi)compliant RDBMS there is - it trades speed in favor of integrity.
Also, if databases were ueber-efficient at executing SQL queries, there'd be no great need to use server side stored procedures to speed things up.
And, last time I checked, google didn't run oracle, and for a reason. -
Re:Subject rejected - looks too much like ASCII ar
Look at this: XHTML parser using K programming language
Perl is really clean language :) -
Re:Subject rejected - looks too much like ASCII ar
Look at this: XHTML parser using K programming language
Perl is really clean language :) -
Can't see the forest for all the trees?
it's easier [...] to debug perl than many languages simply because you have less lines you have to look at to figure out whats going on.
Take a look at some K code (there are examples in the user manual) and then come back and say that. If K is too exotic, then try looking at some macro-heavy LISP code -- it has the same problem just slightly less so.
Code density can be good when you're trying to see the big picture (fewer screenfulls of code is a good thing in this case), but it can work against you when you're trying to understand the little details.
since you get to look at regex patterns to figure out what's going on (looks to hard to manage? get over it. Regex is a small, orthogonal set of commands).
Regular expressions are nothing more than a hack to make up for the fact that generalized LR parsers were quite inefficient up until a few years ago. Just compare a reasonably complex regular expression to the BNF form of a grammar for parsing the same input to see how much easier GLR is to use -- you can see some examples of just how easy GLR parsing is to use here. And it can actually handle more general patterns with nesting, etc. I really think regexes are really just a question of premature optimization -- with GLR you just start out with an incredibly readable and simple grammar, and if it proves to be slow (i.e. if there are lots of points of ambiguity along certain parse trees) you can optimize it towards a purely LR(k) grammar. -
Ja, jaThere are even more impressive results with Kdb by Kx Systems.
Financial organizations are very conservative but even Deutsche Bank are migrating to Linux some of their less important processes.
In all the cases the future of the financial industry is in cheap linux clusters.
-
Ja, jaThere are even more impressive results with Kdb by Kx Systems.
Financial organizations are very conservative but even Deutsche Bank are migrating to Linux some of their less important processes.
In all the cases the future of the financial industry is in cheap linux clusters.
-
How about K?
The K Programming Language is language agnostic. Quote: "One of the hardest things for many people to get over at first is the way K looks. Even the strongest K enthusiast will freely admit that K tends to look like line noise." [A Shallow Introduction to the K Programming Language"]
-
Re:Speaking of Databases....
I stand corrected, K is not S or R. I just downloaded the demo version from here:
http://www.kx.com/download/download.htm
Strangely, the whole install comes in at only 308k. Suffice it to say that I can't imagine there being much in the way of statistical models in there.
Kdb's niche is in handling huge amounts of time series data, such as every tick of every stock on an exchange. A regular db isn't designed for such frequent small updates, and also can't optimise tasks like correlation. What you really want to do to compute a correlation is step over two arrays in parallel. With a normal database, you'd have to fetch large chunks of data from each table and then running your calculation in main memory.
Here's a good overview of array databases generally, as well as Splus and Kdb in particular:
http://www.cs.nyu.edu/shasha/papers/jagtalk.html
As far as I can tell, the thing K gets used for most is technical analysis, the strategy Malkiel criticizes but that remains popular. If you want to find all the cases where stock broke through its three-month high on a Friday after 4:00, Kdb is the way to go. Unfortunately, when I looked a few years ago, there didn't seem to be any way for a private individual to get historical intraday data in any volume except to log it yourself from a subscription quote service.
Here's the Splus homepage, btw. -
Re:Speaking of Databases........90s at a few dotcom startups and it was presented to him as a way of interfacing with the kdb highspeed relational database.
Well, thanks (sortof) for supporting my contention that "K is NOT S" (though now I'm curious what 'S' is, wish that guy had supplied a link), if you had clicked the link in my initial post, it would've taken you...well...here, smart guy! ;) And if you wanted to find out more about K you could've stripped out "product_info/high-performance-database-kdb-plus.h tm" from the URL and it would've taken you here.
But, since I am at work and don't have time to read the entire site before I get home I'm hoping some nice poster (or even a nasty one! I'm not being picky) will chime in with some useful info on whether 'K' is- Procedural
- Object Oriented
- Aspect Oriented
- or some programming style with which I am not yet acquainted
-
Re:Speaking of Databases........90s at a few dotcom startups and it was presented to him as a way of interfacing with the kdb highspeed relational database.
Well, thanks (sortof) for supporting my contention that "K is NOT S" (though now I'm curious what 'S' is, wish that guy had supplied a link), if you had clicked the link in my initial post, it would've taken you...well...here, smart guy! ;) And if you wanted to find out more about K you could've stripped out "product_info/high-performance-database-kdb-plus.h tm" from the URL and it would've taken you here.
But, since I am at work and don't have time to read the entire site before I get home I'm hoping some nice poster (or even a nasty one! I'm not being picky) will chime in with some useful info on whether 'K' is- Procedural
- Object Oriented
- Aspect Oriented
- or some programming style with which I am not yet acquainted
-
Re:Speaking of Databases........90s at a few dotcom startups and it was presented to him as a way of interfacing with the kdb highspeed relational database.
Well, thanks (sortof) for supporting my contention that "K is NOT S" (though now I'm curious what 'S' is, wish that guy had supplied a link), if you had clicked the link in my initial post, it would've taken you...well...here, smart guy! ;) And if you wanted to find out more about K you could've stripped out "product_info/high-performance-database-kdb-plus.h tm" from the URL and it would've taken you here.
But, since I am at work and don't have time to read the entire site before I get home I'm hoping some nice poster (or even a nasty one! I'm not being picky) will chime in with some useful info on whether 'K' is- Procedural
- Object Oriented
- Aspect Oriented
- or some programming style with which I am not yet acquainted
-
Speaking of Databases....
A friend of mine today told me about an amazing database oriented programming language known as "K". I've been in IT for many a year and NEVER caught wind of it, in any way. My friend said that he first encountered it in the late 90s at a few dotcom startups and it was presented to him as a way of interfacing with the kdb highspeed relational database. Typically, it's supposedly used amongst people who write programs for re-insurers (insurance companies that insure insurance comanies...meta-insurance companies if you will) to run just about every financial model conceivable. Why? I asked. Because it is FAST.
It is able (again, according to him for I know nothing of which he speaks) to predict market performance with scary accuracy. This is a claim of which I'm extremely dubious because I do NOT believe (as Burton Malkiel points out) that past market can in any way predict future performance. However, I digress...
But supposedly these k prediction models currently in us base themselves not solely on data from past market performance, but every form of economic condition imaginable.
But my real question is WHAT THE BLINKIN' HELL IS 'K'??? My bud is not a programmer, so my asking questions like "Is it procedural? OO? Aspect Oriented?" went nowhere. Ever see the look a dog gives you when you make a funny noise with your mouth that confuses him?...not to be condescending toward my buddy just 'cuz he doesn't know what those things are. But tust me for that is exactly the look he made. Which is the look I made when confronted with another single character programming language of which I was unaware.
So, again I ask, WHAT DA HECK IS 'K'??? -
Numbers
Fundamentally market data is just a stream of numbers, and once you have access to a stream, it's just a matter of deciding what you want to do with those numbers. There are plenty of Open Source apps for dealing with large blocks of numerical data, for example graphing it, running statistical algorithms over it, and so on, for example Octave and GNUPlot. There is even an open source library for quantitiative finance. And don't underestimate what you can do with just Perl/Tk. Postgres can take care of all your market history, and it's datatypes and query parser are sophisticated enough for data mining, or look at KDB.
The problem you have is twofold. All this stuff is quite low level; you could build something as good as Reuters Dealing/3000 or a BridgeStation out of it, theoretically, but now we're talking about money, we're really talking about time. To integrate QuantLib with Octave with GNUPlot will take a substantial amount of work on your part, altho' once it was done, you could process a feed almost as well as any commercial trading desktop.
The second problem is getting the feed. If you subscribe to say a Reuters data feed for real time streaming quotes, then the cost of a Reuters terminal is really negligible; you're paying for access to the feed. If you take the feed without the terminal, you still need libraries (like SSL) to actually use it in your own application, or you need something like Tibco eFinance to translate it into XML for you, and you also need something that can format messages back to your counterparty in a format they will accept, say FIX or FpML, - this is probably the easiest part to develop yourself, FIX handles all the instrument classes you're interested in. What you need is access to a feed that comes in a useful format, and which can be sourced for in a contract that doesn't involve taking a physical terminal anyway. -
Don't believe the hype
First: yeah, right because XP (et al) re-writes OO: pair programming, early delivery, RAD, iterative development etc are different ways of running a life cycle, not different ways of structuring your model of a domain.
Second: this reminds of when the K boys did a big rant about "I prove OO is flawed because if you have a class Person and derive from it Customer and Staff classes then you break stuff when a staff member quits his job and walks into the shop and buys stuff as you need to get the object to mutate classes". They claimed to instead invent "OO++" (they called it that). The correct OO answer is that you've got a poor design, you need to revisit it (and aspects or attributes or roles as concepts may help you think about this), but that doesn't break or replace OO (ie straw man argument).
Now meta-programming, such as the (now rather old but still a head-fuck for those who program in one language only) Meta Object Protocol is the direction that I see code structure moving: more Lisp-like structures and flexibility to change your object protocol on the fly, losing strong typing as a fundamental mechanism for OO, these are ways to let you manipulate the larger level structure of your code whilst keeping the lowest level of syntax constant . You let people write their OO code as they like, but as an over-coordinator you can suddenly change the way inheritance works, or the way method-dispatch works to get different effects. It's what I like about Perl (which is making me realise what all those Lisp hackers were raving about for so long, but I prefer the pragmatic approach of perl over the rather purist lisp).
--
T -
Well, yeah... that is kinda my point
However, even Perl's utility isn't born out of anything special in the language. There are other laguages are clearer, simpler, more concise, and more powerful. However, it is all the intangibles that make Perl (and C) like VHS: market penetration, large libraries, and perceived quality is good enough. If you try to evaluate Perl on an single metric, it is clearly inferior to almost any competitor, it is only when you view th language and all that it provide holistically that you get a clear picture of why people use it, and many of these aggregate factors are not even techniccal or unique to Perl.
-
for concision, clarity, and simplicity...
read http://kx.com/a/k/readme.txt and be amazed
-
Why? Aren't there still better choices?
I am assuming you want pure reporting speed and that there will be a couple of batch additions and updates a day. Is this a correct assuption of the usage pattern, or am I off?
In that case there is a whole class of databases -- are better than traditional sytle -- of column oriented databases that will smoke Oracle or PostgreSQL. Fastest amoung them is probably KDB. It doesn't seem like there is that much information in the database, so maybe even a memory resident database would perform better.
I still don't understand, why PostreSQL? It is a cost issue? It certainly isn't a decision based on performance. -
Re:It still contains an outdated syntax
No, the most modern language is K. Seriously. If you are writing a language it should be required knowing about. It's amazing.
-
Brian W. Kernighan's scripting language shootoutBWK wrote a paper on this: Timing Trials, or, the Trials of Timing: Experiments with Scripting and User-Interface Languages. It compared C, Awk, Perl, Tcl, Java, Visual Basic, Limbo, and Scheme. It tested various areas of the language, such as graphics, text processing, and array manipulation.
Although K really isn't a scripting langauge (neither is C), results were done for it, too (being faster and having less code). There is also a shallow introduction to K on Kuro5hin.org.
-
Brian W. Kernighan's scripting language shootoutBWK wrote a paper on this: Timing Trials, or, the Trials of Timing: Experiments with Scripting and User-Interface Languages. It compared C, Awk, Perl, Tcl, Java, Visual Basic, Limbo, and Scheme. It tested various areas of the language, such as graphics, text processing, and array manipulation.
Although K really isn't a scripting langauge (neither is C), results were done for it, too (being faster and having less code). There is also a shallow introduction to K on Kuro5hin.org.
-
There is just better languages, thoughIf you want to deal with vectorized data, then use a vector language. Language, such as K are made for that and are much better than Perl at vector operations. It really doesn't make sense to use Perl for it. K is a smaller language, it is simplier, and if you are used to mathematical notation it is probably easier to learn.
I even use K as my favorite general purpose language, I like it so much. There is even a simple introduction written about it at Kuro5hin.
-
Executable line noise
I mean, take a look at this for example. Ouch. I mean, perl can be bad, C can get hairy, but sheesh. As far as hairy goes this is chewbacca overdosed on rogaine.
-
Re:He didn't include K.Actually, I have never bothered to look at that code before (from the readme, I think), but I can tell very quickly what it does.
K is directly translatable to English (and there is a program that does it). It does this because each symbol is given a short English name (sometimes two). All K is parsed form right to left, so there is no baroque precedence rules (3*2+1 is 9 not 7).
Taking from the code sample:
v*(2*!:'x+1)-x:1_!#p
This is read as:v times quantity 2 times enumerate each x plus one end, minus x, where x gets 1 drop enumerate count p
Break down the right side, what x is, a list from 1 to one less the length of the list p (i.e., if p was 5 elements long, x would be a list form 1 to 4). Now the part in the parenthesis is a list, where each list element is another list, but of numbers from 0 to 2*(x[i]+1) by 2s. So the first element of this list is a list with just 2 in it. The next list element is a list of 2,4, etc... Well, then you multiple all these values by v.This follows in Dr. Iverson's tradition of executable notation (see the J website for more information). What you are writing is a formula to manipulate data. It only took me a couple weeks to get used to the notation. After that my roommate and I were to speak across the room to each other in K.
It really is all just a matter of getting used to things.
-
too far off topic, but...I was only trying to impress the point of operating of bulk data and how it related to the test this person performed with C, Java, etc...
Concerning what you have to say about OLTP processing, you might want to read Dennis Sasha's High Volume Transaction Processing paper. You are right on, as far as the architecture is concerned. There is a entire different set of concerns with KDB, but there are always tradeoff. This is too far offtopic to really go into any detail, though.
It is interesting to see somebody else with as KDB experience. Not a common system (but we can hope it grows in the future).
-
He didn't include K.K is a high-performance array language. It is based on APL and Lisp. It really shines when crunching obscene amounts of data. This seems like something that would be perfect for the language. The proof of K's speed lies in KDB, a database written entirely in K. On TPC benchmarks is spanks Oracle and other leading databases (including some amazing scaling across processors: simple table scans with 2.5 billion rows take 1 second and multi-dimensional aggregations take 10-20 seconds).
There is a quick and dirty intro to K over at Kuro5hin.
Some more links for more inforation:
Kernigan's benchmark test
more examples
Kx: the people who make K and KDB -
He didn't include K.K is a high-performance array language. It is based on APL and Lisp. It really shines when crunching obscene amounts of data. This seems like something that would be perfect for the language. The proof of K's speed lies in KDB, a database written entirely in K. On TPC benchmarks is spanks Oracle and other leading databases (including some amazing scaling across processors: simple table scans with 2.5 billion rows take 1 second and multi-dimensional aggregations take 10-20 seconds).
There is a quick and dirty intro to K over at Kuro5hin.
Some more links for more inforation:
Kernigan's benchmark test
more examples
Kx: the people who make K and KDB -
He didn't include K.K is a high-performance array language. It is based on APL and Lisp. It really shines when crunching obscene amounts of data. This seems like something that would be perfect for the language. The proof of K's speed lies in KDB, a database written entirely in K. On TPC benchmarks is spanks Oracle and other leading databases (including some amazing scaling across processors: simple table scans with 2.5 billion rows take 1 second and multi-dimensional aggregations take 10-20 seconds).
There is a quick and dirty intro to K over at Kuro5hin.
Some more links for more inforation:
Kernigan's benchmark test
more examples
Kx: the people who make K and KDB -
He didn't include K.K is a high-performance array language. It is based on APL and Lisp. It really shines when crunching obscene amounts of data. This seems like something that would be perfect for the language. The proof of K's speed lies in KDB, a database written entirely in K. On TPC benchmarks is spanks Oracle and other leading databases (including some amazing scaling across processors: simple table scans with 2.5 billion rows take 1 second and multi-dimensional aggregations take 10-20 seconds).
There is a quick and dirty intro to K over at Kuro5hin.
Some more links for more inforation:
Kernigan's benchmark test
more examples
Kx: the people who make K and KDB -
He didn't include K.K is a high-performance array language. It is based on APL and Lisp. It really shines when crunching obscene amounts of data. This seems like something that would be perfect for the language. The proof of K's speed lies in KDB, a database written entirely in K. On TPC benchmarks is spanks Oracle and other leading databases (including some amazing scaling across processors: simple table scans with 2.5 billion rows take 1 second and multi-dimensional aggregations take 10-20 seconds).
There is a quick and dirty intro to K over at Kuro5hin.
Some more links for more inforation:
Kernigan's benchmark test
more examples
Kx: the people who make K and KDB