Custom Heated Build Platform for Prusa Mendel 3D Printer

A lot of things going on at Pentaho have kept me away from this blog in the past months, but in the meanwhile, in my spare time, I've been playing with a couple of things. I had time to pimp up my 3D printer's heatbed for one thing.

The top layer is a piece of pyroceramic glass. It can withstand huge temperatures and supports very well abrupt temperature changes. This also means that it doesn't change shape at all when heated. This is the most critical requirement of a high end build platform for a 3D printer. If the base isn't perfectly flat (I mean down to 0.05mm precision)the prints will come out wrong.

Programmatic roles in Mondrian

I often get asked how to use custom programmatic roles in Mondrian. It is actually very simple to get started. All you need is Mondrian and its dependencies, along with the following code.

It happens in three parts.

First you need to create a MondrianServer instance.

MondrianServer server =
    MondrianServer.createWithRepository(
        new DynamicContentFinder("path/to/datasources.xml"),
        new IdentityCatalogLocator());

The first argument must implement RepositoryContentFinder and provides a "piece of XML" to the Mondrian server. This tells it which datasources, catalogs and schemas to use. You can either take a look at some examples or refer directly to our DTD.

The second argument is used to translate the catalog URLs provided by the RepositoryContentFinder into actual paths on a file system or HTTP or whatever is required. This allows to decouple the contents of the XML, which can be shared by multiple Mondrian server instances, from the physical paths on actual machines. Notice that behind the scenes, Mondrian uses Apache VFS, so anything is possible here.

Entry roleToken =
    server.getLockBox().register(
        new RoleImpl() {
            // Override whatever here.
            // You could also implement Role directly.
        });

This next part registers the Role implementation to use. You can either override our default implementation, or implement the API directly. Registering an object with a server gives you a token back. You will have to use this token when you create your olap4j connection, like so,

OlapConnection connection =
    server.getConnection(
        "Catalog Name",
        "Schema Name",
        roleToken.getMoniker(),
        new Properties());

 And voilà.

A Machine That Goes Bing!

Some of you might remember the famous sketch from Monthy Python's The Meaning of Life featuring the machine that goes Bing!

Although its sole purpose is to go Bing!, the concept of such a machine appealed to me for no particular reason. Besides, I had a few electronic parts that I needed to test so I figured that a machine that goes Bing! was just as good as any.

Oracle Essbase Sample.Basic on olap4j & Saiku Analytics

The olap4j project has had a regain in momentum lately, when community contributor Mark Semsel joined the project and offered to do some work on making olap4j compatible with Oracle Essbase. This is a very interesting milestone. We can scratch one more backend vendor from our to-do list.Who knows what's next.. too soon to whisper SAS?

Stay tuned!

Unified DIFF from perforce changelist spec

Because this snippet is very useful when dealing with Perforce, I thought I'd share it real quick.

Ever wondered how to obtains a DIFF of a specific changelist spec? Try this.

($1 is the P4CLIENT name and $2 is the changeset spec)

P4CLIENT=$1 p4 opened -c $2 | sed -e 's/#.*$//' | P4CLIENT=$1 p4 -x - diff

Robot 02

Last month, I built my second robot, which I cleverly nicknamed Robot 02. Its name stands for... well the name says it all, really.

Pictured above, Robot 02 is my second attempt at building an autonomous robot capable of navigating around a room. I wanted it to go around obstacles and detect when it gets stuck so that it can act accordingly.

A tale of Mondrian and real time analytics

Back in April 2008, Julian Hyde, founder of Mondrian, extends a basic API to control the contents of member caches within the Mondrian OLAP engine. Its name, quite unsurprisingly, is CacheControl.There is a basic implementation available, yet that feature remains mostly unknown to most of the community.

Jump to December 2010. We are brainstorming on Mondrian 3.3 (today this feels like ages ago). We come up with those crazy ideas about enterprise integration, real time analytics and cool APIs / SPIs. One of these crazy ideas is: Wouldn't it be sweet to update Mondrian's member cache and do real time OLAP?

That's when we remember the old CacheControl API.

olap4j 1.0 is here

Dear olap4j community,

It is with immense pride that we deliver to you today version 1.0 of olap4j. Yup. And you should be proud as well. Thanks to you and to the concerted efforts of countless collaborators, advisers, testers, integrators and developers worldwide, we were able to reach this historic milestone.

If you would allow me to exagerate blatantly for a second, olap4j 1.0 is what I believe the greatest and bestest (yes, you've read it right, bestest) version to have ever been granted to humanity.

At this point, I encourage you to go read the official press release instead of reading my futile efforts at proper journalism. Our fine folks really know how to best express what olap4j represents, why it is important and why it *might* be the bestest thing on Earth. I've also resisted the urge to copy / paste their fabulous text. That said, I'll do my part and get into the more technical details.

You will first notice that the distribution files have changed. The XML/A driver is now separated from the core API. This is a good thing for the project, as it will allow us to fix issues and release more often. Those of you who are using the JDK 1.4 compatible distribution will still get the XML/A driver in the same library. The four binary libraries are available on our Maven repository and Sourceforge project.

Among other changes, the top levels of the connection's metadata have also received some upgrades. A new metadata element was introduced at the top; the Database. We have also removed everything that was marked for deprecation as of 1.0. We therefore encourage people to test carefully while upgrading.

The reference implementation, Mondrian, is compatible with olap4j 1.0, but only for version 3.3.0.14192 and upwards. This means that if you are using both Mondrian and olap4j, the only compatible build is not certified nor tested anymore than any regular CI release. On production systems, you should abstain from upgrading to Mondrian 3.3.X or risk the consequences, although the 3.3.X series are very similar to what we released with the 3.2.X cousin. Anyhow, bottom line is:

• If you're only using olap4j and the XML/A driver, upgrade at will! 
• If you're using Mondrian too, we suggest waiting for Mondrian 3.3 official.
• If you like to live on the edge, grab olap4j 1.0.0.445 and Mondrian 3.3.0.14192. They are both available in our Maven repository.

That's it for now. As usual, drop by our forums or mailing list if there is anything we can help with. And again, congratulations to everyone for the hard work.

olap4j's  maven repository cheat sheet
url:
    http://repository.pentaho.org/artifactory/
group:
    org.olap4j
artifacts:
    olap4j
    olap4j-xmla
    olap4j-jdk14
    olap4j-tck

Welcome!

I was searching the internet for some technical documents and stumbled on this website where the documents were behind a password prompt. What struck me is the weird password prompt, which was not the standard Firefox BASIC authentication prompt. Using curl, I discovered this gem

<SCRIPT language="JavaScript">
<!--hide
var password;
var pass="[redacted]";

password=prompt('Enter password:',' ');

if (password==pass)
  alert('Welcome!');
else
  window.location="errorframe.asp";
   
//-->
</SCRIPT>

Welcome indeed.

Mondrian SPI SegmentCache

Fellow Mondrian developers and users,

One month has already passed since the new year festivities, and while most of you have been trying to renew your gym membership or hold on to your new year resolutions the best you could, so did the Mondrian team. Our resolutions, although not requiring personal sacrifices, are none the less starting to bear fruit.

For you see, our resolution for the year was to provide Mondrian developers and integrators means to achieve better understanding, scalability and control. We have many ideas on how to reach those goals. Some of them are still in their infancy, yet some of them have already been committed to the source. Last month, we worked on the first phase. We added means for system architects to externalize and share a  pluggable segment cache. What does this mean exactly? Let's take a step back in order to better understand.

Internally, Mondrian splits the tuples in segments. A typical segment could be described as a measure crossjoined by a series of predicates. As an example, a textual representation of a segment contents could be:

Measure = [ Sales ]
Predicates = {
          [ Products = * ],
    [ State = California ],
    [ Gender = Male ] }
Data = [ 1346.34, 234.00, ... ]

In the case above, the segment would represent the Sales data of all males in California, for all products. It is a lot more effective to deal with those data structures. If Mondrian was to internally represent each data cell individually, the unique identifier of that cell would be of a greater size than the data itself, thus creating a whole lot more problems in terms of data efficiency. This is therefore why Mondrian deals with groups of cells, which it loads in batches, rather than individually. There is a lot of voodoo magic and heuristics in the background trying to figure out how best to group those segments and how to reduce the number of segments to load, ultimately reducing the number of SQL queries to be executed. Mondrian will group all segments with the same predicates but with a different measure into a segment group. Mondrian will also tend to remove as many predicates as it possibly can in order to optimize the data payload. Lets say that a segment covers all products except a single one, Mondrian will still include the product in the segment but filter it out when a specific query requires it.

Once those segments are populated, Mondrian keeps those in a collection of weak references in local memory. All required segment references are pinned down during the resolving of a particular query, but as soon as the query is done executing, the references are returned to their weak state, thus ready to be garbage collected if needed. This simple mechanism allows Mondrian to answer just about any query, as long as the memory allocated is big enough to answer that particular query. This works really well in fact, since in most small deployments, the maximum amount of memory is never reached. And if it ever gets filled, old segments will be evicted to make some room for the new ones.

Now, there are obvious gotchas. First off, what if it takes a long time for a segment to be populated by the RDBMS. This means that if a particular segment ever gets picked up by the garbage collector, the MDX query sent to Mondrian *might* take longer to execute, whether it was in the segment cache or not. This is not acceptable, simply because this makes all performance predictions impossible.

This is where the SegmentCache SPI comes in. It is essentially a pluggable cache for segments. The algorithm behind the segment loader becomes this:

• Lookup segments in local cache and pin those required.
• Optimize / group segments
• Lookup segments from the SPI cache
• Load the segments found from the SPI  cache
• Populate the remaining unloaded segments from the RDBMS
• Put the segments which come from the RDBMS into the SPI cache
• Pin all loaded segments
• Resolve the query
• Unpin all segments in the local cache

But wait! There is more! The SegmentCache SPI is trivial to implement.

Future<Boolean> contains(SegmentHeader header);
Future<SegmentBody> get(SegmentHeader header);
Future<List<SegmentHeader>> getSegmentHeaders();
Future<Boolean> put(
  SegmentHeader header,
  SegmentBody body);
void tearDown();

Figure 1. Mondrian Segment Loader Architecture

There are two assumptions that are made towards the implementation. The first obvious one is that the cache must assume that many Mondrian instances might access the cache concurrently, form different threads. We therefore recommend using the Actor Pattern or anything similar in order to enforce thread safety. The second is that SegmentCache implementations will be instantiated very often. We therefore recommend using a facade object which relays calls to the actual segment cache code. Update: This was redesigned so that a singleton is created and used throughout Mondrian's internals.

As for the storage of the SegmentHeader and SegmentBody objects, we tried to make it as simple and flexible as possible. Both objects are fully serializable and are immutable. They are also specially crafted to use dense arrays of primitive data types. We also tried to make extensive use of Java native functions when copying the data to / from the cache within Mondrian internals.

The bottom line is that from now on the Mondrian community will be free to implement segment caches to fit their needs. We will be rolling out a few default implementations and examples, obviously. One neat implementation could be one which pages the segments to a super fast array of SSD drives. Another one could be to store the segments in Terracota or ehCache or Infinispan, or just about any scalable caching system there is out there. So if any of you out there are interested in implementing this SPI for your business and would like to either share your experiences or contribute those implementations, don't hesitate to contact us. Or me directly.

There is more goodness to come, but that's it for now. Stay tuned!

olap4j 1.0 - The long road to LTS

Last summer, we at olap4j announced that we would release olap4j 1.0 on October 31st 2010. Now is November 1st and olap4j is still not out of the door. Here's why.

Our reference implementation, the Mondrian project, suffered from delays in the release dates and had to be pushed back. All this kept us very busy and scrambling to make the best of it for a while. But fear not! This very week, the Mondrian team is running QA tests on a 3.2.1-GA build and as soon as it gets the green light, we will be able to put the finishing touches to olap4j 1.0. Why are these projects tied so closely? one might ask. Legitimate question.

You see, every API is a better API if there is at least one reference implementation existing in the wild. For many reasons actually. It allows us to architect, develop and test in a real environment, with real constraints and real data. Several APIs have failed in the past because in the end they were overly complicated for the end user. Some were so overly complicated that in the end, they failed to deliver what an API is supposed to be; a simple interface to a given system. Having a reference implementation mitigates the risk by allowing us to release by little increments and release often, but most importantly, release testable code that works. Another huge advantage is that for every part of the API, there is at least one fully functioning implementation out there, freely available as open sourced software, which implementers can refer to. This is a huge advantage in terms of both project sustainability and project adoption.

So stay tuned, because olap4j 1.0, despite some delays, is right around the corner!

olap4j tutorial

Behold! I finally had some time to put the finishing touches to my tutorial for olap4j. It took a year but here it is. Sorry for the delay. I know, I know. I promised it a year ago, but a lot of stuff has been going on since then. Two new jobs, two moving to a different city (and once more next month...). So without further delay, enjoy!

http://olap4j-demo.googlecode.com/svn/trunk/doc/Olap4j_Introduction_An_end_user_perspective.pdf

From the olap4j team

I posted this message today on the olap4j mailing list. In the interest of reaching a broader audience, I will copy it here as well.

Cheers!

Dear olap4j community members,

As we previously discussed on this mailing list, we are planning to make the final push towards a 1.0 specification. In order to perform those much needed changes and still maintain compatibility as much as possible, the olap4j team proposes the following transition plan.

• 4th week of July - Release of olap4j 0.9.8
  A first initial release, coded 0.9.8, will be performed during the days to come. This release is mostly a wrap-up of the unofficial releases we have put in the Maven repository. Notable changes include compatibility with SAP BW, contextual drill-through for the Query Model, along with various other compatibility fixes.


• Month of August
  During the month of August, the olap4j community will have a chance to comment the proposed changes to the 1.0 final draft. We will provide an updated functional specification document as well as a complete list of the changes that will be required. Should you judge that some items are still missing, or that some should be modified or removed altogether, you are encouraged to let us know. The mailing list is the best place to hold those discussions, or you can also use our forums.


• September 1st - Release of 0.9.9
  September 1st is the date that marks the end of our discussions. After that, all the changes that we agreed upon will be implemented in the API, as well as the Mondrian and XML/A implementations of the driver. The 0.9.9 release will include those changes, but will still maintain retro-compatibility. Some API calls will be marked for deprecation, new ones will be present as well. The 0.9.9 release will be the last available before 1.0. Everything that is marked for deprecation will be removed as of 1.0, so users will have a chance to convert their code base progressively.


• October 31st - Release of 1.0
  We are planning to release olap4j 1.0 on October 31st. All methods that have been marked as deprecated, whether by the 0.9.9 release or any other previous 0.X release will be removed.

A proposed updated functional specification document, as well as a detailed list of API changes will be sent in the following days, right after the 0.9.8 release. We strongly encourage the users of olap4j to express any concerns or ideas that might arise.

Sincerely yours

Luc Boudreau, for the olap4j development team

Pentaho’s Road To Profitability - Take 2

Tom Barber, a collaborator of mine, one of the most active community member of the Pentaho project, posted a very interesting blog entry this week. He is looking back at the path of Pentaho Corporation, a commercial open source business intelligence company, and their latest strategies for growth. He notes that in the past months, Pentaho has put a lot of effort in marketing initiatives and hired many big wigs in their marketing staff. In his opinion, this is somewhat against the ideals of commercial OSS development.

The commercial open source business model is still very young and has not encountered many great challenges up to now. Nor has it sailed in very troubled waters. Yet some signs are already warning people to be very careful with the years to come. Sun Microsystems (now Oracle) was surviving thanks to donations. Compiere recently made the news for all the bad reasons. Red Hat is doing pretty well though. In a nutshell, anything is still possible; good or bad.

Tom’s wish was that Pentaho would rather focus on paying skilled engineers rather than sales people. Now, as much as somewhat agree with the general idea, one must keep in mind that there is no correlation between the number of talented people getting paid to be on a project and it’s success. Some notoriously successful projects depend almost entirely on it’s community base. Mozilla Corporation to name only one. Others employ thousands of employees, yet achieve mediocre results. One could also argue that an effective marketing campaign will in fact boost people’s awareness, thus getting more talented people to join the community base. There is no tested and fail proof recipe so far. As I said earlier, everything goes.

I worked for the past months for a commercial open source company, and the same questions and uncertainties were part of every day discussions. How can a software company making no revenues on licensing be profitable? SQL Power Group, my employer, sponsors the only open source data modeling tool that works cross-platforms and offers, for free, the majority of the functionality included in widely known proprietary tools like ErWin. There are on a weekly basis between five hundred and a thousand downloads of SQL Power Architect. This is a lot for an OSS project in such a narrow niche. Yet a very small proportion of those are actually paying for support/consultancy, or making donations.

There are a lot of factors in play. First off, people willingly using OSS have overall better technical skills than others. To be fair, not every OSS is of acceptable quality, but you get to try as many as you want. Proprietary software is not better in any way, but you won’t get to try many of them, short of having very good contacts and/or deep pockets. This is one reason why reaching the tipping point of adoption is very hard for an OSS company. The percentage of OSS users ready to pay instead of figuring things by themselves is low. Way low. Waaayyy loooww. If you paid for software, you want support (and vice versa). You feel obliged to use the software because you already invested much in it. (Yes, money is time, time is money.) If you didn’t pay for the software and you hit a snag, you are far more likely to stop using it altogether. By abandoning it, you feel like you are exercising damage control, since you save time (thus money). You didn’t pay so why invest time? I’m fully aware that once you do the math, you realize that it’s the exact opposite. People usually don’t.

What does all that have to do with Tom? Well first thing first. Starting next week, I’ll be an employee of Pentaho Corporation. Yup. Am I a big wig? Nope. Am I a marketing guy? Well, I’m good looking, but not good enough with oxymoron. Am I a marketer? I’m having trouble making this blog post interesting, so nope, not a marketer either. I’m just some skilled dude who got his hands dirty for years and now decided that he would devote some serious time to make things better.

Look at it this way. In Tom’s vision of Pentaho, my ass is on the line. I deeply respect his insights on the market and software development in general. Yet here I am, proud as a peacock on a Sunday brunch, about to be part of this fantastic project. Am I worried? Hell no. I’m very enthusiastic about Pentaho having enough budget for marketing initiatives. This is exactly what OSS companies need; market awareness. People need to know you exist and that you are just as good as all those Fortune 100 companies. Even better than these. It has nothing to do with being a sell out. Marketing is not bad for your grassroots values, unless you make it so. Then again, you have only yourself to blame.

OSS companies need to reach the critical mass of contributors and adoption needed to survive. There is simply no other way. OSS users are picky, grumpy, bitchy and unforgiving. I know that for a fact. You want to thrive in that market share? You need two things. Proper marketing and skilled engineers. I’ll be doing my part in the later, and I’m very confident that the good people at Pentaho have picked skilled people to cover the former.

We are competing against giants, so let’s give them a ride for their money.

Monkey Business

Last week we worked on a guerilla-marketing video for Wabit. I'll let you be the judge of that.

mdx4j - MDX query language parser

Last week I launched a spin off of the olap4j parser. Mdx4j wraps olap4j's MDX parser and makes it available to code, without the need of an olap connection.

Why?

Although olap4j contains a SPI parser, we don't want to promote any particular MDX syntax. I therefore packaged it as a separate project so that everyone can have a piece of the pie!

final String query =
"SELECT{} ON COLUMNS FROM CUBE";

final MdxParser parser =
Mdx4jParserFactory.createMdxParser();

final ParseTreeNode tree =
parser.parseSelect(query);

http://code.google.com/p/mdx4j/

All in one BI tool for the non-geeks

A colleague of mine once asked me if I knew a program that can connect to almost all relational databases and offers MsAccess like features to build queries. Sure thing says I. Wabit.

So he downloads it and installs it in 5 minutes. It's free and open source. No hassle. He then creates his connections and manages to do everything he needs to fulfil his duties as a business analyst. Pretty kewl story, heh? Short too. But that's a good sign because as a developer on this project, I can confirm first hand that this is exactly what we aimed for. Making business intelligence easy and painless.

The Wabit is more than that. It's also an OLAP data warehouse browsing and reporting tool. It creates charts in 10 seconds and features a template engine for easy corporate branding. Version 1.0 will feature a server repository for multi-user collaboration and incremental saves, scheduling and fine grained security. The enterprise server is not open sourced though, but the Wabit client is a fully featured platform. You can still save all your queries and reports as an XML file for easy import and export and share it with your fellow co-workers.

The Wabit approaches 1.0 now. We need to reinforce the community around it and we need more feedback. The Wabit works on all platforms with a Java JVM, so whatever your background is, I'm sure that we can make good use of your comments or contributions. You are a GUI designer or a BI consultant or even just the regular Java developer, we have need of your help.

Wabit on Google Code
Wabit homepage

Olap4j vs. Oracle and Ruby

During my monthly checkup of this blog analytics data (thank you Google Analytics), I discovered a new trend. More and more, people are searching for information on olap4j's compatibility.Here are the interesting keywords used and the number of occurrences for the last month.

• "olap4j ruby" - 28 occurrences
• "olap4j oracle" - 3 occurrences

Oracle; I can understand. Olap4j is picking up momentum and is more widely adopted. We support both Microsoft Analysis Services and Mondrian via XMLA. Oracle does have an XMLA server, Hyperion Essbase, although we never tested it with olap4j. If one of you reading this post happens to be a Oracle wizard, please contact us so we can have a chat. The more OLAP servers we support the better.Ruby; now that's intriguing. Ruby can run in a JVM thanks to the JRuby project. Would olap4j work well with JRuby? probably. Are there are any OLAP API for Ruby? Google says no. Digging further in the analytics data didn't reveal to me the actual intent of those who are searching for "olap4j ruby" keywords. What a mystery... I therefore send out there a general call to anyone interested in using olap4j inside JRuby, for we might have common interests.

Creating Mondrian Schemas with Power*Architect

Since I don't have time to write much software myself these days, I figured I'd share this gem with you all. SQL Power, the Canadian Business Intelligence Authority (that's their tag line these days...), sponsors many open source projects. One of them is called Power*Architect; a marvellous cross-platform data modelling tool.

As far as I know, there are close to none "enterprise ready" data modelling tools that work on Linux and Mac. I also suspect none are free, whatever the platform. Visio is certainly not one of them.

Why is it so wonderful? Well, to start with, it can retro/forward engineer most JDBC compatible databases. That's a big plus. And it gets better. You can also use it to create a Mondrian schema. Yep. The team at SQL Power published a tutorial for that last week.

I do have to disclose that I will be working on their projects starting in October. I'm not trying to sell it to you; it's free anyways.  One thing is for sure though. I can't wait to get my hands in there. So I encourage everyone to grab a copy here and fill as many bug reports as you can. It's not 1.0 yet, so community contributions are a must. Having worked with the team for three weeks back in July, I can guarantee that each and every reported bug and suggested feature is closely studied by the development team.

olap4j - A comprehensive tutorial

I've been very busy lately with the new job comming up and many other changes in my personal life, but fear not; I'm cooking something up for you people. I'm working on a comprehensive guide to olap4j. Many people have expressed a need for a more step-by-step introduction on olap4j, what it is, and how to unleash it's raw power. In the next few weeks, I should be able to finally put some more time on it and release a first final draft. Until then, take care!

UPDATE: I finally finished the tutorial. See http://devdonkey.blogspot.com/2010/09/olap4j-tutorial.html