Back in April 2008, Julian Hyde, founder of Mondrian, extends a basic API to control the contents of member caches within the Mondrian OLAP engine. Its name, quite unsurprisingly, is CacheControl.There is a basic implementation available, yet that feature remains mostly unknown to most of the community.
Jump to December 2010. We are brainstorming on Mondrian 3.3 (today this feels like ages ago). We come up with those crazy ideas about enterprise integration, real time analytics and cool APIs / SPIs. One of these crazy ideas is: Wouldn't it be sweet to update Mondrian's member cache and do real time OLAP?
That's when we remember the old CacheControl API.
Showing posts with label bi. Show all posts
Showing posts with label bi. Show all posts
Thursday, October 27, 2011
Monday, November 1, 2010
olap4j 1.0 - The long road to LTS
Last summer, we at olap4j announced that we would release olap4j 1.0 on October 31st 2010. Now is November 1st and olap4j is still not out of the door. Here's why.
Our reference implementation, the Mondrian project, suffered from delays in the release dates and had to be pushed back. All this kept us very busy and scrambling to make the best of it for a while. But fear not! This very week, the Mondrian team is running QA tests on a 3.2.1-GA build and as soon as it gets the green light, we will be able to put the finishing touches to olap4j 1.0. Why are these projects tied so closely? one might ask. Legitimate question.
You see, every API is a better API if there is at least one reference implementation existing in the wild. For many reasons actually. It allows us to architect, develop and test in a real environment, with real constraints and real data. Several APIs have failed in the past because in the end they were overly complicated for the end user. Some were so overly complicated that in the end, they failed to deliver what an API is supposed to be; a simple interface to a given system. Having a reference implementation mitigates the risk by allowing us to release by little increments and release often, but most importantly, release testable code that works. Another huge advantage is that for every part of the API, there is at least one fully functioning implementation out there, freely available as open sourced software, which implementers can refer to. This is a huge advantage in terms of both project sustainability and project adoption.
So stay tuned, because olap4j 1.0, despite some delays, is right around the corner!
Our reference implementation, the Mondrian project, suffered from delays in the release dates and had to be pushed back. All this kept us very busy and scrambling to make the best of it for a while. But fear not! This very week, the Mondrian team is running QA tests on a 3.2.1-GA build and as soon as it gets the green light, we will be able to put the finishing touches to olap4j 1.0. Why are these projects tied so closely? one might ask. Legitimate question.
You see, every API is a better API if there is at least one reference implementation existing in the wild. For many reasons actually. It allows us to architect, develop and test in a real environment, with real constraints and real data. Several APIs have failed in the past because in the end they were overly complicated for the end user. Some were so overly complicated that in the end, they failed to deliver what an API is supposed to be; a simple interface to a given system. Having a reference implementation mitigates the risk by allowing us to release by little increments and release often, but most importantly, release testable code that works. Another huge advantage is that for every part of the API, there is at least one fully functioning implementation out there, freely available as open sourced software, which implementers can refer to. This is a huge advantage in terms of both project sustainability and project adoption.
So stay tuned, because olap4j 1.0, despite some delays, is right around the corner!
Tuesday, September 21, 2010
olap4j tutorial
Behold! I finally had some time to put the finishing touches to my tutorial for olap4j. It took a year but here it is. Sorry for the delay. I know, I know. I promised it a year ago, but a lot of stuff has been going on since then. Two new jobs, two moving to a different city (and once more next month...). So without further delay, enjoy!
http://olap4j-demo.googlecode.com/svn/trunk/doc/Olap4j_Introduction_An_end_user_perspective.pdf
http://olap4j-demo.googlecode.com/svn/trunk/doc/Olap4j_Introduction_An_end_user_perspective.pdf
Tuesday, June 29, 2010
Pentaho’s Road To Profitability - Take 2
Tom Barber, a collaborator of mine, one of the most active community member of the Pentaho project, posted a very interesting blog entry this week. He is looking back at the path of Pentaho Corporation, a commercial open source business intelligence company, and their latest strategies for growth. He notes that in the past months, Pentaho has put a lot of effort in marketing initiatives and hired many big wigs in their marketing staff. In his opinion, this is somewhat against the ideals of commercial OSS development.
The commercial open source business model is still very young and has not encountered many great challenges up to now. Nor has it sailed in very troubled waters. Yet some signs are already warning people to be very careful with the years to come. Sun Microsystems (now Oracle) was surviving thanks to donations. Compiere recently made the news for all the bad reasons. Red Hat is doing pretty well though. In a nutshell, anything is still possible; good or bad.
Tom’s wish was that Pentaho would rather focus on paying skilled engineers rather than sales people. Now, as much as somewhat agree with the general idea, one must keep in mind that there is no correlation between the number of talented people getting paid to be on a project and it’s success. Some notoriously successful projects depend almost entirely on it’s community base. Mozilla Corporation to name only one. Others employ thousands of employees, yet achieve mediocre results. One could also argue that an effective marketing campaign will in fact boost people’s awareness, thus getting more talented people to join the community base. There is no tested and fail proof recipe so far. As I said earlier, everything goes.
I worked for the past months for a commercial open source company, and the same questions and uncertainties were part of every day discussions. How can a software company making no revenues on licensing be profitable? SQL Power Group, my employer, sponsors the only open source data modeling tool that works cross-platforms and offers, for free, the majority of the functionality included in widely known proprietary tools like ErWin. There are on a weekly basis between five hundred and a thousand downloads of SQL Power Architect. This is a lot for an OSS project in such a narrow niche. Yet a very small proportion of those are actually paying for support/consultancy, or making donations.
There are a lot of factors in play. First off, people willingly using OSS have overall better technical skills than others. To be fair, not every OSS is of acceptable quality, but you get to try as many as you want. Proprietary software is not better in any way, but you won’t get to try many of them, short of having very good contacts and/or deep pockets. This is one reason why reaching the tipping point of adoption is very hard for an OSS company. The percentage of OSS users ready to pay instead of figuring things by themselves is low. Way low. Waaayyy loooww. If you paid for software, you want support (and vice versa). You feel obliged to use the software because you already invested much in it. (Yes, money is time, time is money.) If you didn’t pay for the software and you hit a snag, you are far more likely to stop using it altogether. By abandoning it, you feel like you are exercising damage control, since you save time (thus money). You didn’t pay so why invest time? I’m fully aware that once you do the math, you realize that it’s the exact opposite. People usually don’t.
What does all that have to do with Tom? Well first thing first. Starting next week, I’ll be an employee of Pentaho Corporation. Yup. Am I a big wig? Nope. Am I a marketing guy? Well, I’m good looking, but not good enough with oxymoron. Am I a marketer? I’m having trouble making this blog post interesting, so nope, not a marketer either. I’m just some skilled dude who got his hands dirty for years and now decided that he would devote some serious time to make things better.
Look at it this way. In Tom’s vision of Pentaho, my ass is on the line. I deeply respect his insights on the market and software development in general. Yet here I am, proud as a peacock on a Sunday brunch, about to be part of this fantastic project. Am I worried? Hell no. I’m very enthusiastic about Pentaho having enough budget for marketing initiatives. This is exactly what OSS companies need; market awareness. People need to know you exist and that you are just as good as all those Fortune 100 companies. Even better than these. It has nothing to do with being a sell out. Marketing is not bad for your grassroots values, unless you make it so. Then again, you have only yourself to blame.
OSS companies need to reach the critical mass of contributors and adoption needed to survive. There is simply no other way. OSS users are picky, grumpy, bitchy and unforgiving. I know that for a fact. You want to thrive in that market share? You need two things. Proper marketing and skilled engineers. I’ll be doing my part in the later, and I’m very confident that the good people at Pentaho have picked skilled people to cover the former.
We are competing against giants, so let’s give them a ride for their money.
The commercial open source business model is still very young and has not encountered many great challenges up to now. Nor has it sailed in very troubled waters. Yet some signs are already warning people to be very careful with the years to come. Sun Microsystems (now Oracle) was surviving thanks to donations. Compiere recently made the news for all the bad reasons. Red Hat is doing pretty well though. In a nutshell, anything is still possible; good or bad.
Tom’s wish was that Pentaho would rather focus on paying skilled engineers rather than sales people. Now, as much as somewhat agree with the general idea, one must keep in mind that there is no correlation between the number of talented people getting paid to be on a project and it’s success. Some notoriously successful projects depend almost entirely on it’s community base. Mozilla Corporation to name only one. Others employ thousands of employees, yet achieve mediocre results. One could also argue that an effective marketing campaign will in fact boost people’s awareness, thus getting more talented people to join the community base. There is no tested and fail proof recipe so far. As I said earlier, everything goes.
I worked for the past months for a commercial open source company, and the same questions and uncertainties were part of every day discussions. How can a software company making no revenues on licensing be profitable? SQL Power Group, my employer, sponsors the only open source data modeling tool that works cross-platforms and offers, for free, the majority of the functionality included in widely known proprietary tools like ErWin. There are on a weekly basis between five hundred and a thousand downloads of SQL Power Architect. This is a lot for an OSS project in such a narrow niche. Yet a very small proportion of those are actually paying for support/consultancy, or making donations.
There are a lot of factors in play. First off, people willingly using OSS have overall better technical skills than others. To be fair, not every OSS is of acceptable quality, but you get to try as many as you want. Proprietary software is not better in any way, but you won’t get to try many of them, short of having very good contacts and/or deep pockets. This is one reason why reaching the tipping point of adoption is very hard for an OSS company. The percentage of OSS users ready to pay instead of figuring things by themselves is low. Way low. Waaayyy loooww. If you paid for software, you want support (and vice versa). You feel obliged to use the software because you already invested much in it. (Yes, money is time, time is money.) If you didn’t pay for the software and you hit a snag, you are far more likely to stop using it altogether. By abandoning it, you feel like you are exercising damage control, since you save time (thus money). You didn’t pay so why invest time? I’m fully aware that once you do the math, you realize that it’s the exact opposite. People usually don’t.
What does all that have to do with Tom? Well first thing first. Starting next week, I’ll be an employee of Pentaho Corporation. Yup. Am I a big wig? Nope. Am I a marketing guy? Well, I’m good looking, but not good enough with oxymoron. Am I a marketer? I’m having trouble making this blog post interesting, so nope, not a marketer either. I’m just some skilled dude who got his hands dirty for years and now decided that he would devote some serious time to make things better.
Look at it this way. In Tom’s vision of Pentaho, my ass is on the line. I deeply respect his insights on the market and software development in general. Yet here I am, proud as a peacock on a Sunday brunch, about to be part of this fantastic project. Am I worried? Hell no. I’m very enthusiastic about Pentaho having enough budget for marketing initiatives. This is exactly what OSS companies need; market awareness. People need to know you exist and that you are just as good as all those Fortune 100 companies. Even better than these. It has nothing to do with being a sell out. Marketing is not bad for your grassroots values, unless you make it so. Then again, you have only yourself to blame.
OSS companies need to reach the critical mass of contributors and adoption needed to survive. There is simply no other way. OSS users are picky, grumpy, bitchy and unforgiving. I know that for a fact. You want to thrive in that market share? You need two things. Proper marketing and skilled engineers. I’ll be doing my part in the later, and I’m very confident that the good people at Pentaho have picked skilled people to cover the former.
We are competing against giants, so let’s give them a ride for their money.
Wednesday, May 5, 2010
Monkey Business
Last week we worked on a guerilla-marketing video for Wabit. I'll let you be the judge of that.
Monday, April 12, 2010
mdx4j - MDX query language parser
Last week I launched a spin off of the olap4j parser. Mdx4j wraps olap4j's MDX parser and makes it available to code, without the need of an olap connection.
Why?
Although olap4j contains a SPI parser, we don't want to promote any particular MDX syntax. I therefore packaged it as a separate project so that everyone can have a piece of the pie!
http://code.google.com/p/mdx4j/
Why?
Although olap4j contains a SPI parser, we don't want to promote any particular MDX syntax. I therefore packaged it as a separate project so that everyone can have a piece of the pie!
final String query =
"SELECT{} ON COLUMNS FROM CUBE";
final MdxParser parser =
Mdx4jParserFactory.createMdxParser();
final ParseTreeNode tree =
parser.parseSelect(query);
http://code.google.com/p/mdx4j/
Wednesday, September 30, 2009
All in one BI tool for the non-geeks
A colleague of mine once asked me if I knew a program that can connect to almost all relational databases and offers MsAccess like features to build queries. Sure thing says I. Wabit.
So he downloads it and installs it in 5 minutes. It's free and open source. No hassle. He then creates his connections and manages to do everything he needs to fulfil his duties as a business analyst. Pretty kewl story, heh? Short too. But that's a good sign because as a developer on this project, I can confirm first hand that this is exactly what we aimed for. Making business intelligence easy and painless.
The Wabit is more than that. It's also an OLAP data warehouse browsing and reporting tool. It creates charts in 10 seconds and features a template engine for easy corporate branding. Version 1.0 will feature a server repository for multi-user collaboration and incremental saves, scheduling and fine grained security. The enterprise server is not open sourced though, but the Wabit client is a fully featured platform. You can still save all your queries and reports as an XML file for easy import and export and share it with your fellow co-workers.
The Wabit approaches 1.0 now. We need to reinforce the community around it and we need more feedback. The Wabit works on all platforms with a Java JVM, so whatever your background is, I'm sure that we can make good use of your comments or contributions. You are a GUI designer or a BI consultant or even just the regular Java developer, we have need of your help.
Wabit on Google Code
Wabit homepage
So he downloads it and installs it in 5 minutes. It's free and open source. No hassle. He then creates his connections and manages to do everything he needs to fulfil his duties as a business analyst. Pretty kewl story, heh? Short too. But that's a good sign because as a developer on this project, I can confirm first hand that this is exactly what we aimed for. Making business intelligence easy and painless.
The Wabit is more than that. It's also an OLAP data warehouse browsing and reporting tool. It creates charts in 10 seconds and features a template engine for easy corporate branding. Version 1.0 will feature a server repository for multi-user collaboration and incremental saves, scheduling and fine grained security. The enterprise server is not open sourced though, but the Wabit client is a fully featured platform. You can still save all your queries and reports as an XML file for easy import and export and share it with your fellow co-workers.
The Wabit approaches 1.0 now. We need to reinforce the community around it and we need more feedback. The Wabit works on all platforms with a Java JVM, so whatever your background is, I'm sure that we can make good use of your comments or contributions. You are a GUI designer or a BI consultant or even just the regular Java developer, we have need of your help.
Wabit on Google Code
Wabit homepage
Wednesday, September 9, 2009
Olap4j vs. Oracle and Ruby
During my monthly checkup of this blog analytics data (thank you Google Analytics), I discovered a new trend. More and more, people are searching for information on olap4j's compatibility.Here are the interesting keywords used and the number of occurrences for the last month.
- "olap4j ruby" - 28 occurrences
- "olap4j oracle" - 3 occurrences
Friday, August 28, 2009
Creating Mondrian Schemas with Power*Architect
Since I don't have time to write much software myself these days, I figured I'd share this gem with you all. SQL Power, the Canadian Business Intelligence Authority (that's their tag line these days...), sponsors many open source projects. One of them is called Power*Architect; a marvellous cross-platform data modelling tool.
As far as I know, there are close to none "enterprise ready" data modelling tools that work on Linux and Mac. I also suspect none are free, whatever the platform. Visio is certainly not one of them.
Why is it so wonderful? Well, to start with, it can retro/forward engineer most JDBC compatible databases. That's a big plus. And it gets better. You can also use it to create a Mondrian schema. Yep. The team at SQL Power published a tutorial for that last week.
I do have to disclose that I will be working on their projects starting in October. I'm not trying to sell it to you; it's free anyways. One thing is for sure though. I can't wait to get my hands in there. So I encourage everyone to grab a copy here and fill as many bug reports as you can. It's not 1.0 yet, so community contributions are a must. Having worked with the team for three weeks back in July, I can guarantee that each and every reported bug and suggested feature is closely studied by the development team.
As far as I know, there are close to none "enterprise ready" data modelling tools that work on Linux and Mac. I also suspect none are free, whatever the platform. Visio is certainly not one of them.
Why is it so wonderful? Well, to start with, it can retro/forward engineer most JDBC compatible databases. That's a big plus. And it gets better. You can also use it to create a Mondrian schema. Yep. The team at SQL Power published a tutorial for that last week.
I do have to disclose that I will be working on their projects starting in October. I'm not trying to sell it to you; it's free anyways. One thing is for sure though. I can't wait to get my hands in there. So I encourage everyone to grab a copy here and fill as many bug reports as you can. It's not 1.0 yet, so community contributions are a must. Having worked with the team for three weeks back in July, I can guarantee that each and every reported bug and suggested feature is closely studied by the development team.
Thursday, August 20, 2009
olap4j - A comprehensive tutorial
I've been very busy lately with the new job comming up and many other changes in my personal life, but fear not; I'm cooking something up for you people. I'm working on a comprehensive guide to olap4j. Many people have expressed a need for a more step-by-step introduction on olap4j, what it is, and how to unleash it's raw power. In the next few weeks, I should be able to finally put some more time on it and release a first final draft. Until then, take care!
UPDATE: I finally finished the tutorial. See http://devdonkey.blogspot.com/2010/09/olap4j-tutorial.html
UPDATE: I finally finished the tutorial. See http://devdonkey.blogspot.com/2010/09/olap4j-tutorial.html
Thursday, May 21, 2009
Connect Microsoft SQL Server from olap4j
Browsing my Google Analytics statistics, I realized there is a lot of people out there that are searching for ways to connect Microsoft SQL Server with olap4j.
Here is a nice example.
Update : Some useful links
Here is a nice example.
// We must use the XMLA driver.
Class.forName("org.olap4j.driver.xmla.XmlaOlap4jDriver");// This code is for Java 5. With Java 6, you can directly
// unwrap the underlying connection with the .unwrap() call.
OlapConnection connection =
(OlapConnection) DriverManager.getConnection(// This is the SQL Server service end point.
"jdbc:xmla:Server=http://example.com/olap/msmdpump.dll"// Tells the XMLA driver to use a SOAP request cache layer.
// We will use an in-memory static cache.
+ ";Cache=org.olap4j.driver.xmla.cache.XmlaOlap4jNamedMemoryCache"// Sets the cache name to use. This allows cross-connection
// cache sharing. Don't give the driver a cache name and it
// disables sharing.
+ ";Cache.Name=MyNiftyConnection"// Some cache performance tweaks.
// Look at the javadoc for details.
+ ";Cache.Mode=LFU;Cache.Timeout=600;Cache.Size=100",// XMLA is over HTTP, so BASIC authentication is used.
"username",
"password" );
// We can execute a query. MDX of course.
CellSet set = connection.createStatement().executeOlapQuery(
"SELECT {} ON COLUMNS FROM CUBE");Update : Some useful links
Wednesday, May 20, 2009
PAT 0.3 - Integration and nifty features…
Pentaho Analysis Tool sprint 2 is now over and released since a few weeks. We're currently in the process of wrapping up the third sprint. What's new? What new features will there be?
Nothing very spectacular really. Yet there are a few nice features worth mentioning.
Our initial intent for the third sprint was to make the GUI nicer, but we're still trying to figure out if XUL would be a better generation tool than SWT. We are waiting for developments between the GWT-Mosaic team and Nick Baker of Pentaho and hope that all those nice gents would team up and cook something up to their talent. Fingers crossed here. :)
Also worth mentioning, we started talks to review the Query model currently residing in olap4j. This is a soon to be major component as it will provide developers and GUI enthusiasts with a properly abstracted API to build queries against a data warehouse. Olap4j is very great API for low level stuff, but it still needs an abstraction layer for the common folks. Building queries should not require in-depth knowledge of MDX, for you cannot expect all business analysts to master MDX anyways. Anyone who wants to participate in the process is warmly invited to manifest himself. We are really looking for input on this, whatever your background is. (As long as it's related to BI I guess...)
Nothing very spectacular really. Yet there are a few nice features worth mentioning.
- Integration with the Pentaho BI server
PAT can now run embedded in the Pentaho User Console and be configured remotely. It only supports XMLA connections for this first draft, but don't worry; more compatible connection types are to be supported in the medium term. This also means that you can seamlessly use any XMLA provider. Sql Server? Essbase? Mondrian? You choose. All that thanks to olap4j. If you want specific details on this, Gretchen Moran has written a bunch of documentation on this. Congratulations to Paul Stoellberger and Gretchen Moran for this feature. - Create multiple queries at once
This was a requirement that was passed to us by Pentaho's engineering team. People want to build and use more than one query at a time without concurrency issues. That was not properly supported by the age old JPivot application. Now and then we encountered some problems with multiple queries, so this was something pretty high on our features list. The backend supported that since sprint 1 , but there was higher priorities for the GUI components. - OlapTableModel first draft
The Java API doesn't include a proper TableModel for OLAP data, so we're planning to write one. We still have lots of things to figure out on this subject, but we're planning to mock a draft specification before the sprint 3 deadline.
Our initial intent for the third sprint was to make the GUI nicer, but we're still trying to figure out if XUL would be a better generation tool than SWT. We are waiting for developments between the GWT-Mosaic team and Nick Baker of Pentaho and hope that all those nice gents would team up and cook something up to their talent. Fingers crossed here. :)
Also worth mentioning, we started talks to review the Query model currently residing in olap4j. This is a soon to be major component as it will provide developers and GUI enthusiasts with a properly abstracted API to build queries against a data warehouse. Olap4j is very great API for low level stuff, but it still needs an abstraction layer for the common folks. Building queries should not require in-depth knowledge of MDX, for you cannot expect all business analysts to master MDX anyways. Anyone who wants to participate in the process is warmly invited to manifest himself. We are really looking for input on this, whatever your background is. (As long as it's related to BI I guess...)
Tuesday, November 11, 2008
Of OLAP and the importance of open standards
In these times of economical crisis, many companies will turn to business intelligence (BI) as a source of wisdom and counsel. Millions of dollars will be invested in an effort to understand the extend of their respective problems and find solutions based on accurate and decision oriented datasets.
Since I have a fairly good amount of experience with work in heterogeneous environments and tackling data integration challenges, I thought I'd pitch in my two cents.
The root of the problem is this. The Microsoft OLAP toolkit does not integrate so well with anything else than .NET technologies. SAS offers a Java API, yet it is not ready for production. (I worked with it for two years, and believe me, they are still a fairly long way to production quality code.) As a matter of fact, most software vendors in the OLAP world distribute some API to integrate their technologies, but you often end up with black boxes of questionable quality, flexibility or performance. Some even go as far as to obfuscate their libraries... this really doesn't help in the end.
Some vendors like Oracle went for the all-in-the-box solution. They offer a "complete" solution that can fit every possible need. Then again, what they are telling you is: if we don't have it, you probably don't need it."Probably"? You got to be kidding. Since when does software vendors know what you need and what your future will be? Better switch probably for hopefully.
In the best case, in order to meet your needs, you'll hack your way through at the expense of your project specifications. The final result can be nothing but deceiving. Your celebration will be bitter and probably short lived, I fear.
About the importance of collective work
You have a brand new application. Hooray! This is where the production phase kicks in.
What if you need to move your datamart to another OLAP server? What if there are not enough connections licenses to allow both production connections and all of your maintenance personnel on the OLAP server and they are forced to take turns to debug? What if the CEO decides to migrate to a new platform? What if [insert random but oh so frequent unforeseen event here]? Your thousand dollar code is now rendered useless; you can start crying now, you deserve it. In your quest for more money making, you've created a monster that was expensive and will continue to pump the money out of your institution pockets.
If you were good enough in systems design, you thought about a data layer. The data layer still remains to be rewritten entirely and it often represents at least a third of the overall effort. Close, but no cigar. This might sound like a catastrophic scenario, but it is oh so frequent.
Many people got tired of all this non-sense we decided to work together. We decided that enough time and money was wasted on individual efforts that were ruined in the end. It was time to agree on standards and share the product of our collective effort.
Take Hibernate for example. It is now a de facto standard when it comes to data mappers. For the Java version alone, it represents 859 thousand lines of code worth 12.8 million dollars in work hours. Think you can top that with your in-house data layer in times of economical crisis?
OLAP is a world in itself. You can't take relational paradigms and apply them to the multidimensional world. The .NET toolbox does have very nice libraries to do some neat OLAP stuff, then again, you're locked-in with SSAS. This is a no-no.
On the Java side, things are even worse. There is currently a big void in the Java OLAP market. No OLAP standard emerged at all. Thanks to the selfishness of the big players of the industry, the JOLAP initiative was a total failure. It never reached the final version, so the JSR-69 specification died quietly.
We at Olap4j tried to fill that gap with an open initiative. Everyone can pitch in. And I mean EVERYONE.
You know the expression vendor lock-in? I hope you do, I *really* do, or else you'll learn it the hard way. Olap4j aims at solving exactly this problem. You can develop applications on it's API and switch the underlying OLAP engine without rewriting a single line of code. Not bad heh? Olap4j is more than a database driver. It is an open API built right on top of the JDBC industry standard where everyone collaborates to specify a common base onto which to build.
It even includes transformation libraries and testing facilities.
So far, it has two implementations ready to use. The Mondrian driver allows you to run the much acclaimed Mondrian open source OLAP engine as an in-process data provider.
There is also the XML/A generic driver that can connect to pretty much anything that talks XML/A, whether it's over HTTP or anything else you fancy using. This particular driver allows you to build applications that can switch to and from any of these OLAP engines :
The Olap4j project is gaining momentum and we truly hope to see it become the standard in the Java world.
Since I have a fairly good amount of experience with work in heterogeneous environments and tackling data integration challenges, I thought I'd pitch in my two cents.
Why developers and project managers will have a hard time
The root of the problem is this. The Microsoft OLAP toolkit does not integrate so well with anything else than .NET technologies. SAS offers a Java API, yet it is not ready for production. (I worked with it for two years, and believe me, they are still a fairly long way to production quality code.) As a matter of fact, most software vendors in the OLAP world distribute some API to integrate their technologies, but you often end up with black boxes of questionable quality, flexibility or performance. Some even go as far as to obfuscate their libraries... this really doesn't help in the end.
Some vendors like Oracle went for the all-in-the-box solution. They offer a "complete" solution that can fit every possible need. Then again, what they are telling you is: if we don't have it, you probably don't need it."Probably"? You got to be kidding. Since when does software vendors know what you need and what your future will be? Better switch probably for hopefully.
In the best case, in order to meet your needs, you'll hack your way through at the expense of your project specifications. The final result can be nothing but deceiving. Your celebration will be bitter and probably short lived, I fear.
About the importance of collective work
You have a brand new application. Hooray! This is where the production phase kicks in.
What if you need to move your datamart to another OLAP server? What if there are not enough connections licenses to allow both production connections and all of your maintenance personnel on the OLAP server and they are forced to take turns to debug? What if the CEO decides to migrate to a new platform? What if [insert random but oh so frequent unforeseen event here]? Your thousand dollar code is now rendered useless; you can start crying now, you deserve it. In your quest for more money making, you've created a monster that was expensive and will continue to pump the money out of your institution pockets.
If you were good enough in systems design, you thought about a data layer. The data layer still remains to be rewritten entirely and it often represents at least a third of the overall effort. Close, but no cigar. This might sound like a catastrophic scenario, but it is oh so frequent.
Many people got tired of all this non-sense we decided to work together. We decided that enough time and money was wasted on individual efforts that were ruined in the end. It was time to agree on standards and share the product of our collective effort.
Take Hibernate for example. It is now a de facto standard when it comes to data mappers. For the Java version alone, it represents 859 thousand lines of code worth 12.8 million dollars in work hours. Think you can top that with your in-house data layer in times of economical crisis?
About Java OLAP
OLAP is a world in itself. You can't take relational paradigms and apply them to the multidimensional world. The .NET toolbox does have very nice libraries to do some neat OLAP stuff, then again, you're locked-in with SSAS. This is a no-no.
On the Java side, things are even worse. There is currently a big void in the Java OLAP market. No OLAP standard emerged at all. Thanks to the selfishness of the big players of the industry, the JOLAP initiative was a total failure. It never reached the final version, so the JSR-69 specification died quietly.
We at Olap4j tried to fill that gap with an open initiative. Everyone can pitch in. And I mean EVERYONE.
What makes Olap4j so kewl
You know the expression vendor lock-in? I hope you do, I *really* do, or else you'll learn it the hard way. Olap4j aims at solving exactly this problem. You can develop applications on it's API and switch the underlying OLAP engine without rewriting a single line of code. Not bad heh? Olap4j is more than a database driver. It is an open API built right on top of the JDBC industry standard where everyone collaborates to specify a common base onto which to build.
It even includes transformation libraries and testing facilities.
I want to kick the tires and use it right now
So far, it has two implementations ready to use. The Mondrian driver allows you to run the much acclaimed Mondrian open source OLAP engine as an in-process data provider.
There is also the XML/A generic driver that can connect to pretty much anything that talks XML/A, whether it's over HTTP or anything else you fancy using. This particular driver allows you to build applications that can switch to and from any of these OLAP engines :
- Hyperion Essbase
- Microsoft SQL Server Analysis Services
- Infor
- Mondrian
- Palo
The Olap4j project is gaining momentum and we truly hope to see it become the standard in the Java world.
Monday, August 11, 2008
Data integration challenges tackled
Data integration in business environments can be a painful task. I mean REAL painful. The volume of data is huge, it does not cross-validate, it is dispersed in many heterogeneous formats, yadi yada. You know the song. Some day, I stumbled on Pentaho Data Integration (PDI).This was a real breakthrough.First thing first, it's not subject to "vendor lock-in". It can read most data formats out there and can write it back to pretty much anything. This is a huge plus because gives it the ability to be used by a plenitude of user types and environments. Being written in Java also gives it an edge as an enterprise tool, for it is platform agnostic.
But the real advantages are not those trivial specifications. My love for PDI has much deeper roots. Simply put : it's powerful. Creating an integration process is a trivial matter. Drag and drop. Link. Execute. Those three simple steps will cover most of your business needs. Really, I mean it. Never again will I write a snippet of code to read a CSV file and write it's content in a database. Mark my words; NEVER! This is a waste of time and a developer who lives with his times should know that.
What about the real juicy stuff ?
As you suspected, there is much more to PDI than meets the eye. It can be clustered, it can use a database based repository for all processes, there are automatic documentation generation tools and is supported by a huge community. Many tutorials exist to address most business needs and challenges. It's well made, very stable and easily expandable with plugins for power users.
I strongly recommend to give it a try. The next version should be released soon and it will include many great new features. I met Matt Casters last June and had the chance to see for myself all the new functionalities that will make it to the next release. We're talking about visual performance bottleneck exploration and some more neat stuff you won't find anywhere else.
Cheers, and have a good time integrating !!
Tuesday, July 29, 2008
Pentaho on the iPhone
I successfully added the iPhone extension to my Pentaho platform today and I was more then impressed with the ease with which we can enable the whole platform to work seamlessly on those nifty little phones.Oh yeah, I bought an iPhone too...
I'm slowly discovering the fun of having a cellular phone in my pocket. This is something that I never experienced before; never had a cell phone. I have to say that I'm glad it's a good phone, and sexy too.
The bottom line is : get one.
For those interested, here's the wiki page that says it all. Thanks to Will Gorman, senior developer at Pentaho, who put this all up.
Tuesday, July 15, 2008
Olap4j and XML/A - One more step towards a true olap systems integration API
I've been working for a month now on some enhancements to the Olap4j project to make it more powerful and compatible. The good news is, I succeeded. The previous version, 0.9.5, lacked some basic functionalities which you would expect from a production ready XML/A driver.For one, it's HTTP proxy didn't support cookies. his was a big problem since the myriad of requests required to populate Olap4j's meta data objects each created a new user session on the web service back-end. This was a no-no, but now it's fixed and kicking ass.
I also worked on a SOAP query cache. This is was a big piece of software engineering, since I'm not used to thread safe coding. Thread safe thignys are usually in the lower levels of BI application servers and those issues are tackled from the start. Thanks to Java's java.util.concurrent package, this was a breeze.
Those changes are not part of any release nor in the SVN yet. I'm still waiting for peer review before the whole commit, but for people eager to see what it looks like, I've created a neat little package for y'all.
Now I can move back to my next release of the University of Montreal's Pentaho platform... all work and no play makes Luc a dull boy.
Cheers !
Subscribe to:
Posts (Atom)