Slashdot Mirror
Ask Slashdot: Changing Career From OLTP To OLAP Dev
First time accepted submitter xby2_arch writes "After spending over 12 years writing OLTP applications (Java EE/JDBC/ORMs), I decided to dabble in the OLAP world. I had decent DB skills, considering most of my previous projects had involved data modeling and coding using Stored Procs, etc. Yet I hadn't designed or implemented any dimensional databases. Luckily for me, I had enough relevant domain knowledge to land a developer job in a data warehousing project. The work was enjoyable enough that it motivated me to spend that extra time and effort I needed to cope with the different dynamics of coding in the OLAP realm. In my past life, data volumes weren't the primary concern (instead, transaction volumes were), here, everything was about data. ETL/Integrations present another set of problems you generally skirt in a typical web/app-tier developer role. All in all, it turned out to be a non-trivial, yet worthwhile transition. I am certain that there are plenty of seasoned developers out there who plan to make a similar move (or have made already), who see data as the next chapter in their careers evolving toward becoming Enterprise Architects. I want to hear what's holding them back, or what helped them move forward. What should be considered a prerequisite to make this switch, and what are the risks, etc.?"
Please stand by while the trolls try to figure out how to respond to your question.
In the meantime, you should expect about 50 posts by PHP developers asking what a stored proc is, and suggesting you move to RoR if you want an ORM.
The first thing we did to strategize mission-critical web-readiness to expedite wireless users was leverage our dot-com ROI and aggregate robust e-markets. Being able to synthesize cutting-edge channels enabled us to unleash extensible users, which in-turn enabled us to orchestrate turn-key mindshare.
Utilizing our synergistic functionalities, we were then focused towards mesh visionary markets and envisioneering collaborative initiatives with our partners. The net result was the incubation of plug-and-play experiences to transform vertical vortals and utilize cutting-edge deliverables.
Plus, I have a large cock. That helped a lot.
You know, we can always check Wikipedia for definitions:
https://en.wikipedia.org/wiki/OLTP
https://en.wikipedia.org/wiki/OLAP
It might have been nice if the editors had included these links in the summary, but it is not as though you do not know how to use Wikipedia.
Palm trees and 8
So... you're saying you've already made the switch from OLTP to OLAP and you'd like to take this opportunity to gloat about it, but you'd still like to hear from other developers what they think the prerequisites are for making such a move and what has held them back from doing all the cool stuff you're doing? Or am I missing the question?
Breakfast served all day!
Moving from one specialized type of programming to a closely related type of specialized programming is pretty straightforward. Apply for such a position and you wont suffer compared with other candidates. Or, if your current employer needs something new done, do that new thing. You're not talking about a major career change here. Programming is programming. Even moving from something like standalone application GUI programming for windows in C# to back-end web service programming in C++ on Unix isn't that big a deal. If you can program, you'll pick up the new tech/language/idioms as needed and notice the striking similarity in the work you actually end up doing.
I'm thinking of making the leap from C# 3.0 to C# 4.0. Does anyone have any advice?
OLTP: Online Transaction Processing: you buy a ticket, you want it immediatly. The seller types it in the computer and prints your ticket, a database checked if there were free seats and immediatly reserves one. "Immediatly" is the important word, the customer is not waiting. OLAP: Online Analytical Processing: How many seats from the US to EMEA have been sold by that kind of sellers with such produyt code. Managements wants the results by the end of the month, it is OK if the query runs a couple of hours/days. Many real life systems contain two databases, one tuned for speed (containing only the current tickets) and one for reference later (containing all tickets sold in the past n months/years). The difference between them is database tuning and SQL tuning. All the rest (such as path to architect: yeah, the more different systems you know, the more chance you will have to design new ones) is hype.
/. refugees on Usenet: news:comp.misc
... the next chapter in their careers evolving toward ...
Whats a "career"? We don't have those around here in "IT related fields". I suppose if you live in silicon valley there is a chance of upward mobility, or maybe TLA .gov jobs, but for everyone else, its just luck that got us in a good spot in a downsizing economy in a downsizing company and downsizing department where they haven't axed us yet.
"Career" in general would be a more entertaining "ask /." topic. Work in the above plus the "ha ha noobs don't realize than even the concept of my job didn't exist when I was their age" and plenty of ageism whining and funny stories about nepotism and stuff like that.
"Science flies us to the moon. Religion flies us into buildings." - Victor Stenger
While it's a fair point that Wikipedia is an obvious starting point, I would have to say that the article on OLAP seems to suggest that OLTP is a synonym for OLAP, and the article on OLAP is short but dense. I'm left with an impression that there's a distinction being made between two approaches for constructing end-user interfaces to databases, but what the distinction between OLTP and OLAP is still unclear to me, and I don't have any idea why the distinction would be so significant as to constitute a career transition.
The difference between them is database tuning and SQL tuning. All the rest (such as path to architect: yeah, the more different systems you know, the more chance you will have to design new ones) is hype.
That's like saying web coding is the same as application coding and the main difference is just languages used. There is a huge difference in the goals and methodology of OLTP and OLAP. DB and SQL tuning is a small part of the larger scheme of things. Take for instance your example of tickets; you can get an OLTP to show how many seats from US to EMEA by seller. It may not be very easy especially as you add conditions (by quarter, by day of week of travelled, by day of week sold, etc). In OLAP these questions become important and you have to design your DB structure to accommodate them and other queries. It also matters how this information is reported by tool as not all tools can take advantage of DB features. After all that then DB and SQL tuning comes into play.
Well, there's spam egg sausage and spam, that's not got much spam in it.
The prerequisites to making the switch is first and most importantly having an appropriate business case for OLAP. The second prerequisite is that you've tried doing analytics in a traditional RDMS, perhaps jumped on to the NoSQL bandwagon, and you've failed at it (i.e. success for a little while but then your data eventually brings your queries down to its knees). Don't worry, failure isn't necessarily wrong, it's just you and your team needed the experience before you could make the next leap.
The risks are a knowledge jump in to an OLAP mindset from a traditional SQL mindset. Invest in you and your fellow developer's knowledge. Push back on management and sales when they want more immediate results and let them know that it will take 3-5 months to replace your current system. Do your proper technology evaluations. Learn FoodMart and Adventureworks and let them guide you down the path of good fact and dimension design. Don't snub your nose at Microsoft as they absorbed the company in the 80's that basically pioneered this stuff and made billions, but also don't take their stuff too literally as there are several products out there and some that do things better.
Read The Data Warehouse Toolkit thoroughly and practice using Mondrian which is an open source Java OLAP engine that can sit on top of PostgreSQL, MySQL, and others. Find a good ETL tool rather than trying to write your own at first and don't be afraid to force your internal users to use this tool to create their facts. Don't worry if you don't get it the first time, but keep trying and keep discussing with your fellow developers as it takes a team to work out all the kinks. Later on you'll probably end up seeing how you did things wrong, but hopefully you can get most things right in the beginning.
From my limited experience, the OLAP community is small and/or behind walled gardens, the tools are poor and closed source, and potential employers are only interested if you have experience in *their* BI tools (Pentaho, Microstrategy, Cognos, etc). Microsoft appears to be the only one trying to establish a theoretical basis for BI, but their efforts are starting to show age despite their being so much more that can be done in the field. Finally, you will be misunderstood by the majority of Rails/PHP/Web developers: The same one who think Key-Value stores and NoSQL are the height of modern technology.
That said, BI can be technically satisfying. If you get down to the SQL/MDX you will appreciate what a database can do; which allows questions to be phrased succinctly. I have seen too much code written in procedural languages (Javascript being the worst of them) that are many lines long and run atrociously slow, that can be restated in SQL (or MDX) simply, and run a 1000x faster. I love that fact there are no loops!
From a business perspective, you have much more exposure to management and other departments: You will have improved visibility in the company, and your worth will be inflated - as you will be the one that satisfies management's appetite for more information to help make decisions.
If it's on the main page the target audience is pretty general, so you really shouldn't have to check Wikipedia. You should ALWAYS tell people what acronyms mean before using them exclusively. And if you have time enough to be an arrogant *&$# while posting links to Wikipedia you could also take the time to spell out what the acronyms mean. At the very least give the long form, e.g. Online Transaction Processing (OLTP), Online Analytical Processing (OLAP). See it's not that fucking hard.
And since I am taking the time to rant (and because any technical article in Wikipedia gets hijacked by propellor heads who like to inject as much as of their industry specific double speak so that they sound important and a layman can't get at least an understandable overview in the summary):
OLTP - Online Transaction Processing. An application/database designed for larger, equal, or nearly equal volumes of database inserts, updates, and deletes, as there are reads. The database is generally more (and often highly) normalized, meaning that there is less data duplication across tables and/or within tables.
OLAP - Online Analytical Processing. Essentially a data warehouse designed for larger volumes of reads than there will be inserts, updates, or deletes (often relatively very, very few deletes). Less normalized meaning that there may be duplicated data across and/or within tables in order to increase query speed at the cost of possible consistency issues due to the data duplication.
Actually i matured 5 years or so ago, after spending some 10 years in the delusion of thinking that the fields i was in were more 'elite' than the others. they were different fields than programming. but, the core concept of 'our work is tougher and more elite - other people are not as elite as us' nonsense was the same.
i grew over it.
Read radical news here
OLTP - Online Transaction Processing. An application/database designed for larger, equal, or nearly equal volumes of database inserts, updates, and deletes, as there are reads. The database is generally more (and often highly) normalized, meaning that there is less data duplication across tables and/or within tables.
OLAP - Online Analytical Processing. Essentially a data warehouse designed for larger volumes of reads than there will be inserts, updates, or deletes (often relatively very, very few deletes). Less normalized meaning that there may be duplicated data across and/or within tables in order to increase query speed at the cost of possible consistency issues due to the data duplication.
Mark this informative comment the hell up!
And smash my Karma if you want to, but the (four letter) acronyms really didn't help the article much.
A=Analytical: the problems of creating useful reports from vast amounts of mainly read-only data.
T=Transactional: the problems of highly concurrent read-write transactions.
Wikipedia simply states that the terms are derived from each other, the techniques they embody are very different.
If you only have a small amount of data and a small number of transactions then you don't care about the difference, but once you have a large number of one or other (or worse both) then it is a very significant distinction.