Back in April 2008, Julian Hyde, founder of Mondrian, extends a basic API to control the contents of member caches within the Mondrian OLAP engine. Its name, quite unsurprisingly, is CacheControl.There is a basic implementation available, yet that feature remains mostly unknown to most of the community.
Jump to December 2010. We are brainstorming on Mondrian 3.3 (today this feels like ages ago). We come up with those crazy ideas about enterprise integration, real time analytics and cool APIs / SPIs. One of these crazy ideas is: Wouldn't it be sweet to update Mondrian's member cache and do real time OLAP?
That's when we remember the old CacheControl API.
Showing posts with label olap. Show all posts
Showing posts with label olap. Show all posts
Thursday, October 27, 2011
Tuesday, April 12, 2011
olap4j 1.0 is here
Dear olap4j community,
It is with immense pride that we deliver to you today version 1.0 of olap4j. Yup. And you should be proud as well. Thanks to you and to the concerted efforts of countless collaborators, advisers, testers, integrators and developers worldwide, we were able to reach this historic milestone.
If you would allow me to exagerate blatantly for a second, olap4j 1.0 is what I believe the greatest and bestest (yes, you've read it right, bestest) version to have ever been granted to humanity.
At this point, I encourage you to go read the official press release instead of reading my futile efforts at proper journalism. Our fine folks really know how to best express what olap4j represents, why it is important and why it *might* be the bestest thing on Earth. I've also resisted the urge to copy / paste their fabulous text. That said, I'll do my part and get into the more technical details.
You will first notice that the distribution files have changed. The XML/A driver is now separated from the core API. This is a good thing for the project, as it will allow us to fix issues and release more often. Those of you who are using the JDK 1.4 compatible distribution will still get the XML/A driver in the same library. The four binary libraries are available on our Maven repository and Sourceforge project.
Among other changes, the top levels of the connection's metadata have also received some upgrades. A new metadata element was introduced at the top; the Database. We have also removed everything that was marked for deprecation as of 1.0. We therefore encourage people to test carefully while upgrading.
The reference implementation, Mondrian, is compatible with olap4j 1.0, but only for version 3.3.0.14192 and upwards. This means that if you are using both Mondrian and olap4j, the only compatible build is not certified nor tested anymore than any regular CI release. On production systems, you should abstain from upgrading to Mondrian 3.3.X or risk the consequences, although the 3.3.X series are very similar to what we released with the 3.2.X cousin. Anyhow, bottom line is:
It is with immense pride that we deliver to you today version 1.0 of olap4j. Yup. And you should be proud as well. Thanks to you and to the concerted efforts of countless collaborators, advisers, testers, integrators and developers worldwide, we were able to reach this historic milestone.
If you would allow me to exagerate blatantly for a second, olap4j 1.0 is what I believe the greatest and bestest (yes, you've read it right, bestest) version to have ever been granted to humanity.
At this point, I encourage you to go read the official press release instead of reading my futile efforts at proper journalism. Our fine folks really know how to best express what olap4j represents, why it is important and why it *might* be the bestest thing on Earth. I've also resisted the urge to copy / paste their fabulous text. That said, I'll do my part and get into the more technical details.
You will first notice that the distribution files have changed. The XML/A driver is now separated from the core API. This is a good thing for the project, as it will allow us to fix issues and release more often. Those of you who are using the JDK 1.4 compatible distribution will still get the XML/A driver in the same library. The four binary libraries are available on our Maven repository and Sourceforge project.
Among other changes, the top levels of the connection's metadata have also received some upgrades. A new metadata element was introduced at the top; the Database. We have also removed everything that was marked for deprecation as of 1.0. We therefore encourage people to test carefully while upgrading.
The reference implementation, Mondrian, is compatible with olap4j 1.0, but only for version 3.3.0.14192 and upwards. This means that if you are using both Mondrian and olap4j, the only compatible build is not certified nor tested anymore than any regular CI release. On production systems, you should abstain from upgrading to Mondrian 3.3.X or risk the consequences, although the 3.3.X series are very similar to what we released with the 3.2.X cousin. Anyhow, bottom line is:
- If you're only using olap4j and the XML/A driver, upgrade at will!
- If you're using Mondrian too, we suggest waiting for Mondrian 3.3 official.
- If you like to live on the edge, grab olap4j 1.0.0.445 and Mondrian 3.3.0.14192. They are both available in our Maven repository.
olap4j's maven repository cheat sheet
url:
http://repository.pentaho.org/artifactory/
group:
org.olap4j
artifacts:
olap4j
olap4j-xmla
olap4j-jdk14
olap4j-tck
Thursday, February 3, 2011
Mondrian SPI SegmentCache
Fellow Mondrian developers and users,
One month has already passed since the new year festivities, and while most of you have been trying to renew your gym membership or hold on to your new year resolutions the best you could, so did the Mondrian team. Our resolutions, although not requiring personal sacrifices, are none the less starting to bear fruit.
For you see, our resolution for the year was to provide Mondrian developers and integrators means to achieve better understanding, scalability and control. We have many ideas on how to reach those goals. Some of them are still in their infancy, yet some of them have already been committed to the source. Last month, we worked on the first phase. We added means for system architects to externalize and share a pluggable segment cache. What does this mean exactly? Let's take a step back in order to better understand.
Internally, Mondrian splits the tuples in segments. A typical segment could be described as a measure crossjoined by a series of predicates. As an example, a textual representation of a segment contents could be:
Once those segments are populated, Mondrian keeps those in a collection of weak references in local memory. All required segment references are pinned down during the resolving of a particular query, but as soon as the query is done executing, the references are returned to their weak state, thus ready to be garbage collected if needed. This simple mechanism allows Mondrian to answer just about any query, as long as the memory allocated is big enough to answer that particular query. This works really well in fact, since in most small deployments, the maximum amount of memory is never reached. And if it ever gets filled, old segments will be evicted to make some room for the new ones.
Now, there are obvious gotchas. First off, what if it takes a long time for a segment to be populated by the RDBMS. This means that if a particular segment ever gets picked up by the garbage collector, the MDX query sent to Mondrian *might* take longer to execute, whether it was in the segment cache or not. This is not acceptable, simply because this makes all performance predictions impossible.
This is where the SegmentCache SPI comes in. It is essentially a pluggable cache for segments. The algorithm behind the segment loader becomes this:
There are two assumptions that are made towards the implementation. The first obvious one is that the cache must assume that many Mondrian instances might access the cache concurrently, form different threads. We therefore recommend using the Actor Pattern or anything similar in order to enforce thread safety.The second is that SegmentCache implementations will be instantiated very often. We therefore recommend using a facade object which relays calls to the actual segment cache code. Update: This was redesigned so that a singleton is created and used throughout Mondrian's internals.
As for the storage of the SegmentHeader and SegmentBody objects, we tried to make it as simple and flexible as possible. Both objects are fully serializable and are immutable. They are also specially crafted to use dense arrays of primitive data types. We also tried to make extensive use of Java native functions when copying the data to / from the cache within Mondrian internals.
The bottom line is that from now on the Mondrian community will be free to implement segment caches to fit their needs. We will be rolling out a few default implementations and examples, obviously. One neat implementation could be one which pages the segments to a super fast array of SSD drives. Another one could be to store the segments in Terracota or ehCache or Infinispan, or just about any scalable caching system there is out there. So if any of you out there are interested in implementing this SPI for your business and would like to either share your experiences or contribute those implementations, don't hesitate to contact us. Or me directly.
There is more goodness to come, but that's it for now. Stay tuned!
One month has already passed since the new year festivities, and while most of you have been trying to renew your gym membership or hold on to your new year resolutions the best you could, so did the Mondrian team. Our resolutions, although not requiring personal sacrifices, are none the less starting to bear fruit.
For you see, our resolution for the year was to provide Mondrian developers and integrators means to achieve better understanding, scalability and control. We have many ideas on how to reach those goals. Some of them are still in their infancy, yet some of them have already been committed to the source. Last month, we worked on the first phase. We added means for system architects to externalize and share a pluggable segment cache. What does this mean exactly? Let's take a step back in order to better understand.
Internally, Mondrian splits the tuples in segments. A typical segment could be described as a measure crossjoined by a series of predicates. As an example, a textual representation of a segment contents could be:
Measure = [ Sales ]In the case above, the segment would represent the Sales data of all males in California, for all products. It is a lot more effective to deal with those data structures. If Mondrian was to internally represent each data cell individually, the unique identifier of that cell would be of a greater size than the data itself, thus creating a whole lot more problems in terms of data efficiency. This is therefore why Mondrian deals with groups of cells, which it loads in batches, rather than individually. There is a lot of voodoo magic and heuristics in the background trying to figure out how best to group those segments and how to reduce the number of segments to load, ultimately reducing the number of SQL queries to be executed. Mondrian will group all segments with the same predicates but with a different measure into a segment group. Mondrian will also tend to remove as many predicates as it possibly can in order to optimize the data payload. Lets say that a segment covers all products except a single one, Mondrian will still include the product in the segment but filter it out when a specific query requires it.
Predicates = {
[ Products = * ],
[ State = California ],
[ Gender = Male ] }
Data = [ 1346.34, 234.00, ... ]
Once those segments are populated, Mondrian keeps those in a collection of weak references in local memory. All required segment references are pinned down during the resolving of a particular query, but as soon as the query is done executing, the references are returned to their weak state, thus ready to be garbage collected if needed. This simple mechanism allows Mondrian to answer just about any query, as long as the memory allocated is big enough to answer that particular query. This works really well in fact, since in most small deployments, the maximum amount of memory is never reached. And if it ever gets filled, old segments will be evicted to make some room for the new ones.
Now, there are obvious gotchas. First off, what if it takes a long time for a segment to be populated by the RDBMS. This means that if a particular segment ever gets picked up by the garbage collector, the MDX query sent to Mondrian *might* take longer to execute, whether it was in the segment cache or not. This is not acceptable, simply because this makes all performance predictions impossible.
This is where the SegmentCache SPI comes in. It is essentially a pluggable cache for segments. The algorithm behind the segment loader becomes this:
- Lookup segments in local cache and pin those required.
- Optimize / group segments
- Lookup segments from the SPI cache
- Load the segments found from the SPI cache
- Populate the remaining unloaded segments from the RDBMS
- Put the segments which come from the RDBMS into the SPI cache
- Pin all loaded segments
- Resolve the query
- Unpin all segments in the local cache
Future<Boolean> contains(SegmentHeader header);
Future<SegmentBody> get(SegmentHeader header);
Future<List<SegmentHeader>> getSegmentHeaders();
Future<Boolean> put(
SegmentHeader header,
SegmentBody body);
void tearDown();
 |
| Figure 1. Mondrian Segment Loader Architecture |
There are two assumptions that are made towards the implementation. The first obvious one is that the cache must assume that many Mondrian instances might access the cache concurrently, form different threads. We therefore recommend using the Actor Pattern or anything similar in order to enforce thread safety.
As for the storage of the SegmentHeader and SegmentBody objects, we tried to make it as simple and flexible as possible. Both objects are fully serializable and are immutable. They are also specially crafted to use dense arrays of primitive data types. We also tried to make extensive use of Java native functions when copying the data to / from the cache within Mondrian internals.
The bottom line is that from now on the Mondrian community will be free to implement segment caches to fit their needs. We will be rolling out a few default implementations and examples, obviously. One neat implementation could be one which pages the segments to a super fast array of SSD drives. Another one could be to store the segments in Terracota or ehCache or Infinispan, or just about any scalable caching system there is out there. So if any of you out there are interested in implementing this SPI for your business and would like to either share your experiences or contribute those implementations, don't hesitate to contact us. Or me directly.
There is more goodness to come, but that's it for now. Stay tuned!
Monday, November 1, 2010
olap4j 1.0 - The long road to LTS
Last summer, we at olap4j announced that we would release olap4j 1.0 on October 31st 2010. Now is November 1st and olap4j is still not out of the door. Here's why.
Our reference implementation, the Mondrian project, suffered from delays in the release dates and had to be pushed back. All this kept us very busy and scrambling to make the best of it for a while. But fear not! This very week, the Mondrian team is running QA tests on a 3.2.1-GA build and as soon as it gets the green light, we will be able to put the finishing touches to olap4j 1.0. Why are these projects tied so closely? one might ask. Legitimate question.
You see, every API is a better API if there is at least one reference implementation existing in the wild. For many reasons actually. It allows us to architect, develop and test in a real environment, with real constraints and real data. Several APIs have failed in the past because in the end they were overly complicated for the end user. Some were so overly complicated that in the end, they failed to deliver what an API is supposed to be; a simple interface to a given system. Having a reference implementation mitigates the risk by allowing us to release by little increments and release often, but most importantly, release testable code that works. Another huge advantage is that for every part of the API, there is at least one fully functioning implementation out there, freely available as open sourced software, which implementers can refer to. This is a huge advantage in terms of both project sustainability and project adoption.
So stay tuned, because olap4j 1.0, despite some delays, is right around the corner!
Our reference implementation, the Mondrian project, suffered from delays in the release dates and had to be pushed back. All this kept us very busy and scrambling to make the best of it for a while. But fear not! This very week, the Mondrian team is running QA tests on a 3.2.1-GA build and as soon as it gets the green light, we will be able to put the finishing touches to olap4j 1.0. Why are these projects tied so closely? one might ask. Legitimate question.
You see, every API is a better API if there is at least one reference implementation existing in the wild. For many reasons actually. It allows us to architect, develop and test in a real environment, with real constraints and real data. Several APIs have failed in the past because in the end they were overly complicated for the end user. Some were so overly complicated that in the end, they failed to deliver what an API is supposed to be; a simple interface to a given system. Having a reference implementation mitigates the risk by allowing us to release by little increments and release often, but most importantly, release testable code that works. Another huge advantage is that for every part of the API, there is at least one fully functioning implementation out there, freely available as open sourced software, which implementers can refer to. This is a huge advantage in terms of both project sustainability and project adoption.
So stay tuned, because olap4j 1.0, despite some delays, is right around the corner!
Tuesday, September 21, 2010
olap4j tutorial
Behold! I finally had some time to put the finishing touches to my tutorial for olap4j. It took a year but here it is. Sorry for the delay. I know, I know. I promised it a year ago, but a lot of stuff has been going on since then. Two new jobs, two moving to a different city (and once more next month...). So without further delay, enjoy!
http://olap4j-demo.googlecode.com/svn/trunk/doc/Olap4j_Introduction_An_end_user_perspective.pdf
http://olap4j-demo.googlecode.com/svn/trunk/doc/Olap4j_Introduction_An_end_user_perspective.pdf
Wednesday, July 21, 2010
From the olap4j team
I posted this message today on the olap4j mailing list. In the interest of reaching a broader audience, I will copy it here as well.
Cheers!
Dear olap4j community members,
As we previously discussed on this mailing list, we are planning to make the final push towards a 1.0 specification. In order to perform those much needed changes and still maintain compatibility as much as possible, the olap4j team proposes the following transition plan.
A proposed updated functional specification document, as well as a detailed list of API changes will be sent in the following days, right after the 0.9.8 release. We strongly encourage the users of olap4j to express any concerns or ideas that might arise.
Sincerely yours
Luc Boudreau, for the olap4j development team
Cheers!
Dear olap4j community members,
As we previously discussed on this mailing list, we are planning to make the final push towards a 1.0 specification. In order to perform those much needed changes and still maintain compatibility as much as possible, the olap4j team proposes the following transition plan.
- 4th week of July - Release of olap4j 0.9.8
A first initial release, coded 0.9.8, will be performed during the days to come. This release is mostly a wrap-up of the unofficial releases we have put in the Maven repository. Notable changes include compatibility with SAP BW, contextual drill-through for the Query Model, along with various other compatibility fixes. - Month of August
During the month of August, the olap4j community will have a chance to comment the proposed changes to the 1.0 final draft. We will provide an updated functional specification document as well as a complete list of the changes that will be required. Should you judge that some items are still missing, or that some should be modified or removed altogether, you are encouraged to let us know. The mailing list is the best place to hold those discussions, or you can also use our forums. - September 1st - Release of 0.9.9
September 1st is the date that marks the end of our discussions. After that, all the changes that we agreed upon will be implemented in the API, as well as the Mondrian and XML/A implementations of the driver. The 0.9.9 release will include those changes, but will still maintain retro-compatibility. Some API calls will be marked for deprecation, new ones will be present as well. The 0.9.9 release will be the last available before 1.0. Everything that is marked for deprecation will be removed as of 1.0, so users will have a chance to convert their code base progressively. - October 31st - Release of 1.0
We are planning to release olap4j 1.0 on October 31st. All methods that have been marked as deprecated, whether by the 0.9.9 release or any other previous 0.X release will be removed.
A proposed updated functional specification document, as well as a detailed list of API changes will be sent in the following days, right after the 0.9.8 release. We strongly encourage the users of olap4j to express any concerns or ideas that might arise.
Sincerely yours
Luc Boudreau, for the olap4j development team
Wednesday, May 5, 2010
Monkey Business
Last week we worked on a guerilla-marketing video for Wabit. I'll let you be the judge of that.
Monday, April 12, 2010
mdx4j - MDX query language parser
Last week I launched a spin off of the olap4j parser. Mdx4j wraps olap4j's MDX parser and makes it available to code, without the need of an olap connection.
Why?
Although olap4j contains a SPI parser, we don't want to promote any particular MDX syntax. I therefore packaged it as a separate project so that everyone can have a piece of the pie!
http://code.google.com/p/mdx4j/
Why?
Although olap4j contains a SPI parser, we don't want to promote any particular MDX syntax. I therefore packaged it as a separate project so that everyone can have a piece of the pie!
final String query =
"SELECT{} ON COLUMNS FROM CUBE";
final MdxParser parser =
Mdx4jParserFactory.createMdxParser();
final ParseTreeNode tree =
parser.parseSelect(query);
http://code.google.com/p/mdx4j/
Wednesday, September 30, 2009
All in one BI tool for the non-geeks
A colleague of mine once asked me if I knew a program that can connect to almost all relational databases and offers MsAccess like features to build queries. Sure thing says I. Wabit.
So he downloads it and installs it in 5 minutes. It's free and open source. No hassle. He then creates his connections and manages to do everything he needs to fulfil his duties as a business analyst. Pretty kewl story, heh? Short too. But that's a good sign because as a developer on this project, I can confirm first hand that this is exactly what we aimed for. Making business intelligence easy and painless.
The Wabit is more than that. It's also an OLAP data warehouse browsing and reporting tool. It creates charts in 10 seconds and features a template engine for easy corporate branding. Version 1.0 will feature a server repository for multi-user collaboration and incremental saves, scheduling and fine grained security. The enterprise server is not open sourced though, but the Wabit client is a fully featured platform. You can still save all your queries and reports as an XML file for easy import and export and share it with your fellow co-workers.
The Wabit approaches 1.0 now. We need to reinforce the community around it and we need more feedback. The Wabit works on all platforms with a Java JVM, so whatever your background is, I'm sure that we can make good use of your comments or contributions. You are a GUI designer or a BI consultant or even just the regular Java developer, we have need of your help.
Wabit on Google Code
Wabit homepage
So he downloads it and installs it in 5 minutes. It's free and open source. No hassle. He then creates his connections and manages to do everything he needs to fulfil his duties as a business analyst. Pretty kewl story, heh? Short too. But that's a good sign because as a developer on this project, I can confirm first hand that this is exactly what we aimed for. Making business intelligence easy and painless.
The Wabit is more than that. It's also an OLAP data warehouse browsing and reporting tool. It creates charts in 10 seconds and features a template engine for easy corporate branding. Version 1.0 will feature a server repository for multi-user collaboration and incremental saves, scheduling and fine grained security. The enterprise server is not open sourced though, but the Wabit client is a fully featured platform. You can still save all your queries and reports as an XML file for easy import and export and share it with your fellow co-workers.
The Wabit approaches 1.0 now. We need to reinforce the community around it and we need more feedback. The Wabit works on all platforms with a Java JVM, so whatever your background is, I'm sure that we can make good use of your comments or contributions. You are a GUI designer or a BI consultant or even just the regular Java developer, we have need of your help.
Wabit on Google Code
Wabit homepage
Wednesday, September 9, 2009
Olap4j vs. Oracle and Ruby
During my monthly checkup of this blog analytics data (thank you Google Analytics), I discovered a new trend. More and more, people are searching for information on olap4j's compatibility.Here are the interesting keywords used and the number of occurrences for the last month.
- "olap4j ruby" - 28 occurrences
- "olap4j oracle" - 3 occurrences
Thursday, August 20, 2009
olap4j - A comprehensive tutorial
I've been very busy lately with the new job comming up and many other changes in my personal life, but fear not; I'm cooking something up for you people. I'm working on a comprehensive guide to olap4j. Many people have expressed a need for a more step-by-step introduction on olap4j, what it is, and how to unleash it's raw power. In the next few weeks, I should be able to finally put some more time on it and release a first final draft. Until then, take care!
UPDATE: I finally finished the tutorial. See http://devdonkey.blogspot.com/2010/09/olap4j-tutorial.html
UPDATE: I finally finished the tutorial. See http://devdonkey.blogspot.com/2010/09/olap4j-tutorial.html
Thursday, May 21, 2009
Connect Microsoft SQL Server from olap4j
Browsing my Google Analytics statistics, I realized there is a lot of people out there that are searching for ways to connect Microsoft SQL Server with olap4j.
Here is a nice example.
Update : Some useful links
Here is a nice example.
// We must use the XMLA driver.
Class.forName("org.olap4j.driver.xmla.XmlaOlap4jDriver");// This code is for Java 5. With Java 6, you can directly
// unwrap the underlying connection with the .unwrap() call.
OlapConnection connection =
(OlapConnection) DriverManager.getConnection(// This is the SQL Server service end point.
"jdbc:xmla:Server=http://example.com/olap/msmdpump.dll"// Tells the XMLA driver to use a SOAP request cache layer.
// We will use an in-memory static cache.
+ ";Cache=org.olap4j.driver.xmla.cache.XmlaOlap4jNamedMemoryCache"// Sets the cache name to use. This allows cross-connection
// cache sharing. Don't give the driver a cache name and it
// disables sharing.
+ ";Cache.Name=MyNiftyConnection"// Some cache performance tweaks.
// Look at the javadoc for details.
+ ";Cache.Mode=LFU;Cache.Timeout=600;Cache.Size=100",// XMLA is over HTTP, so BASIC authentication is used.
"username",
"password" );
// We can execute a query. MDX of course.
CellSet set = connection.createStatement().executeOlapQuery(
"SELECT {} ON COLUMNS FROM CUBE");Update : Some useful links
Tuesday, July 15, 2008
Olap4j and XML/A - One more step towards a true olap systems integration API
I've been working for a month now on some enhancements to the Olap4j project to make it more powerful and compatible. The good news is, I succeeded. The previous version, 0.9.5, lacked some basic functionalities which you would expect from a production ready XML/A driver.For one, it's HTTP proxy didn't support cookies. his was a big problem since the myriad of requests required to populate Olap4j's meta data objects each created a new user session on the web service back-end. This was a no-no, but now it's fixed and kicking ass.
I also worked on a SOAP query cache. This is was a big piece of software engineering, since I'm not used to thread safe coding. Thread safe thignys are usually in the lower levels of BI application servers and those issues are tackled from the start. Thanks to Java's java.util.concurrent package, this was a breeze.
Those changes are not part of any release nor in the SVN yet. I'm still waiting for peer review before the whole commit, but for people eager to see what it looks like, I've created a neat little package for y'all.
Now I can move back to my next release of the University of Montreal's Pentaho platform... all work and no play makes Luc a dull boy.
Cheers !
Subscribe to:
Posts (Atom)