The Dev Donkey Blog

Robot 02

2011-12-15T22:30:00.000-05:00

Last month, I built my second robot, which I cleverly nicknamed Robot 02. Its name stands for... well the name says it all, really.

Pictured above, Robot 02 is my second attempt at building an autonomous robot capable of navigating around a room. I wanted it to go around obstacles and detect when it gets stuck so that it can act accordingly.

The hardware

Hardware wise, the robot is quite simple. Its base chassis is a Dagu Rover 5 robot plafrorm. I've discovered that marvelous piece of hardware while searching on the net for robot platforms. This one can use both tracks and wheels, and since it has four independent motors, it can be fitted with Mecanum wheels for ultimate traction control.

Each of the four motors has a dual channel optical encoder built-in. These allow the robot to detect the speed and direction of rotation of each wheel.

Each motor takes approximately 4 amps when stalling, so the Arduino board cannot possibly drive these 4 babies. To drive them, I'm using a Dagu 4 channel DC motor controller.

This board takes care of driving the motors. It has two separate power inputs at the top left and right corners, one driving the logical components, the other driving the motors themselves. On the top side of the board are pictured the connections to the encoders. We have to provide a 5V power supply to each, and we get back two signal wires per encoder (one per optical lens, if you remembered these are dual encoders).

On the sides are the power wires running to the motors.

At the bottom, a ribbon cable sends a mix of digital and analog signals to the Arduino board. It consists of four pins per motor channel. One ground, one digital input that we set to high to reverse the direction, one analog output which varies between 0 and 4.5 volts according to the number of amps the motor uses, and finally one Pulse Width Modulated (PWM) input to control the speed.

Right beside the DC board a HMC6352 compass. From my experience a compass works much better when it isn't surrounded by a whole bunch of other electronic components and wires. Those interfere easily with the compass readings, and since the robot needs to update frequently its heading, we need fast and accurate results. I've mounted the compass on cable headers which I've hotglued to a plastic board spacer. This setup allows me to update the heading more than 5 times a second with reliable readings. Although, I resorted to doing multiple readings and averaging the results. This gives somewhat more reliable readings since it reduces the noise from possible flukes.

At the top of the robot are a pair of opposed ultrasonic sensors fixed to a servo motor. These are required to figure out how close the robot is to obstacles in its environment. I used two LV-MaxSonar-EZ0 ultrasonic sensors. These have a range between 6 and 150 inches and a beam width of about 5 inches, so they are perfectly suited for the size of my robot. These little babies can operate in digital and analog mode. The digital mode gives slower but more accurate readings. In digital mode, the component will send a digital pulse each time the echo comes back. After you do the math and factor in the speed of sound, you get the reading you need. In analog mode, the sensor is much faster but less accurate. The analog pin sends a variable voltage which represents the distance of the echo, but an analog output suffers from the noise of the other components of the robot (especially the servo). These range finders also support serial communication, but I haven't experimented with that yet.

From my experience, I get better results by using the digital mode and I programmed my robot to do multiple readings, which it then averages, to reduce the noise even further. Accuracy and speed are key here.

The SG-5010 servo can rotate 180 degrees in both direction, but we only really need to spin it in an arc of 135 degrees total, thanks to the twin-mount. Since it scans two directions at a time, scanning all around is done pretty quickly. Again, speed is key when dealing with a single threaded micro-controller.

While building my previous robot, I've learned that creating a connection hub with a prototyping board that sits between your components and your micro controller makes your life that much easier. Micro controllers have a limited number of 5V, 3V and ground pins, and this robot requires about a dozen of each. The robot is already clogged with wires; without that board, the robot would looks like a spaghetti ball.

Good prototyping boards are hard to find in Montreal. If like me you don't know where to get good and cheap ones, you can buy them off the old Chinese chap I've met while I was staying in Toronto last year. He runs a little electronics store and specializes mostly in Arduino and other small PMCs. He also makes these neat prototyping boards that you can order directly form his web site. Sadly, as I write this post, I realize that his website is offline for maintenance until December 30th. You can always call him directly to pass your order. He has two models, a 2"x1" and a 2"x4", selling for about 3 and 5 dollars respectively. Trust me, they are really worth their price.

The board will also allow you to mount all your digital and analog inputs on neat wire headers for your micro controller. One thing you don't want is a bunch of loose wires. They get easily disconnected by the moving parts tugging at them every now and then.

The brain is an Arduino Mega 2560. It is pictured above as the blue board with a USB connection and a 9V input. This micro controller is an impressive one. 53 digital (about 10 are PWM compatible) and 15 analog inputs, with 5 serial RX/TX pairs. It also supports SDA/SCL, SPDI and some other mostly shady communication protocols. At 16Mhz, it can run up to 256kb of code with 32kb of memory. This is plenty enough for a robot of that size since it currently use about a third of the inputs and barely a quarter of the processing power.

Sitting right on top of the micro controller is a WiFly shield. Its RN-131G chip uses very little power and has a plethora of features, from wifi network scan to hardware HTTP server. I haven't had time to use it as part of the robot yet, but my plan is to have it serve an HTML page with status information and a manual override of the robot controls.

Power-wise, I use 12 Ni-Mh 2700mAh AA batteries. Its quite a lot of batteries, but then again, that robot is power hungry. The Hong Kong reseller DealExtreme sells these for really cheap (about 14$, shipping included).

The software

Software wise, the robot uses a classical "robot loop". It consists of the three following phases.

Scan the environment.
Analyze and decide what to do next.
Act.
Goto 1.

This loop has to be non-blocking, meaning that if the robot has decided to go forward, the loop must still be executed so that the robot can run the scan phase, thus detecting any problems it should act upon.

I'm programming it in C++ on Eclipse and the AVR plugin. If you are interested the source code of the robot is available on Google code. Arduino boards come with a simple IDE, but I find it to be a bit childish and doesn't offer the productivity tools that you would normally expect from a contemporary IDE. It doesn't have code completion, doesn't let you checkin your files in a SCM, on top of which is a very bad pseudo-file management kinda scheme. I know the ins and outs of Eclipse, so to me this was a no brainer. The effort required to setup Eclipse for AVR development is a bit painful, but nowadays there are some pretty decent tutorials that can help you get there. It's definitely worth the effort though.

As I'm usually working on software only (I'm a software engineer by day...) having to interface with hardware was a brand new experience for me. For one thing, hardware is far from being as precise as simple arithmetics. Most of the components I'm using for this robot have quite a big margin of error when performing measurements. The range finders can give wrong results at random. The compass is easily affected by its surroundings. The servo has to physically turn and you have to account for the time it needs. All these little things add up pretty quickly.

If you fail to factor in all these little things, your robot will quickly behave erratically. From my experience, it feels like each mistake the robot makes is exponentially affecting the final behavior, as each individual error amplifies the others. Make sure that your code can detect these errors and perform the necessary corrections as it goes along.

For this iteration (because I do plan to build many more robots) I haven't really used any fancy path finding algorithm. It uses a heuristic algorithm which goes like this.

Scan the surroundings and map the distance available per polar heading, in increments of 45 degrees.
Pick a direction towards which there is a minimum cleared distance and that is closest to the front of the robot (try to turn as little as possible).
Turn the robot either left or right, whichever is the quickest to the desired heading.
Go forward.
While going forward, scan by alternating between the front and either front-left or front-right (at random, in a proportion of 66% to the side where the distance available is the smallest)
Check the polar heading of the robot. If the robot is not headed properly, goto 3.
If it failed to correct the heading for more than X seconds, the robot is most likely stuck. Back up a little and goto 1.
If a collision is imminent, back up a little, and goto 1.
Goto 4.

I'm planning on implementing something more fancy eventually, but for now it does the job.

In a near future, I'll spend time finishing the HTTP server. It turns out that because the robot's brain has to run in a non blocking loop, HTTP connections are working only scarcely, if at all. This is a problem I still have to resolve. I have a spare ATMega328 micro controller so it will become dedicated to HTTP and will communicate with the brain through SPDI. I've always wanted to learn SPDI anyways.

My girlfriend wants us to build a cute robot body for it, so I'll have to spend some time redesigning the wiring in order to make it much more compact. Right now it is in a bulky prototype form, allowing me to change this and that as I tweak it, but soon enough, I won't need to make any changes to the hardware anymore.

I'll also be spending time perfecting the software, so if you want to collaborate on it, I've made it freely available on Google Code.

If you want more information or need any help with your own design, ask away and maybe we can all benefit from our experiments.

A tale of Mondrian and real time analytics

2011-10-27T13:30:00.001-04:00

Back in April 2008, Julian Hyde, founder of Mondrian, extends a basic API to control the contents of member caches within the Mondrian OLAP engine. Its name, quite unsurprisingly, is CacheControl.There is a basic implementation available, yet that feature remains mostly unknown to most of the community.

Jump to December 2010. We are brainstorming on Mondrian 3.3 (today this feels like ages ago). We come up with those crazy ideas about enterprise integration, real time analytics and cool APIs / SPIs. One of these crazy ideas is: Wouldn't it be sweet to update Mondrian's member cache and do real time OLAP?

That's when we remember the old CacheControl API.

A year and a half had passed since the release of the API. After some PR effort, a critical mass of integrators started using it. This gave us incentive to push for that feature and take it over the final mile. We spent some time testing, fixing what we've found

In the process, Will Gorman, longtime Mondrian contributor and VP Engineering at Pentaho, writes an example Pentaho XAction script to demonstrate how someone can use the API, in Javascript, run by Rhino, within a JVM.

It starts by a Mondrian schema.

<resources>
    <catalog>
      <url>
        <location>solution:steel-wheels/analysis/steelwheels.mondrian.xml</location>
        <mime-type>text/xml</mime-type>
      </url>
    </catalog>
</resources>

To which we add two simple steps, the first of which establishes an OLAP connection.

<action-definition>
      <component-name>MDXLookupRule</component-name>
      <action-type>OLAP Connection</action-type>
      <action-resources>
        <catalog type="resource"/>
      </action-resources>
      <action-outputs>
        <prepared_component type="mdx-connection" mapping="shared_olap_connection"/>
      </action-outputs>
      <component-definition>
        <jndi><![CDATA[SampleData]]></jndi>
        <location><![CDATA[mondrian]]></location>
      </component-definition>
    </action-definition>

The second contains the juicy part. This is where the magic happens. We use the Javascript step to update Mondrian's caches.

<action-definition>
      <component-name>JavascriptRule</component-name>
      <action-type>JavaScript to clear portions of the cache</action-type>
      <action-inputs>
        <shared_olap_connection type="mdx-connection"/>
      </action-inputs>
      <action-outputs>
        <rule-result type="string"/>
      </action-outputs>
      <component-definition>
        <script>
            <![CDATA[
                function cacheControlImpl() {
                    // get access to the necessary cache control API objects
                    var conn = shared_olap_connection.shareConnection().getConnection();
                    var salesCube = conn.getSchema().lookupCube("SteelWheelsSales", true);
                    var schemaReader = salesCube.getSchemaReader(null).withLocus();
                    var parent = schemaReader.getMemberByUniqueName(Packages.mondrian.olap.Id.Segment.toList("Product", "All Products", "Classic Cars", "Autoart Studio Design"), true);
                    var hierarchy = parent.getHierarchy();
                    var member = hierarchy.createMember(
                    parent,
                    parent.getLevel().getChildLevel(),
                    "1927 Ford Model T",
                    null);
                    var cacheControl = conn.getCacheControl(null);
                    // must have mondrian.rolap.EnableRolapCubeMemberCache=false set in
                    // pentaho-solutions/system/mondrian/mondrian.properties
                    // We could also set it explicitly here by calling:
                    //     Packages.mondrian.olap.MondrianProperties.instance().EnableRolapCubeMemberCache.set(false);
                    var addCommand = cacheControl.createAddCommand(member);
                    cacheControl.execute(addCommand);
                    return "Flushed: " + parent;
                }
                cacheControlImpl();
            ]]>
        </script>
      </component-definition>
    </action-definition>

We're done. The complete script is available for download.

CacheControl contains methods to perform changes on the member cache, but the cell cache as well. Using CacheControl to modify the members caches updates both the member caches and flushes the cell data's cached regions as well. (This also affects any pluggable SegmentCache you are using by flushing out the regions touched by the members you modify.)

As mentioned in the script above, you must turn Mondrian's ROLAP member cache off. This means Mondrian must wrap objects as it resolves your query in a thin wrapper. That operation itself takes some time to perform, but allows Mondrian to optimize the members at runtime. Usually, we only wrap them once and cache them, but this leaves out the updates performed by CacheControl. This is a technical debt we pay, but fear not, Mondrian 4.0 will address that issue and remove that layer altogether.

The future

Our next step will be to expose that functionality through olap4j and make it easily available to ETL tools out there. There is, at the time this article is written, someone working on adding olap4j connections to the Kettle project. I can't specify any release date, but trust me, this opens a whole lot of possibilities. (A Kettle step for a Mondrian cache update would be pretty cool. Contributions are welcome, as always, wink wink.)

Our final step (i mean long term here...) will be to bubble up those events to the olap4j driver, using the CellSetListener. This will allow user interfaces to get notified when new data arrives.

Full circle.

olap4j 1.0 is here

2011-04-12T00:01:00.006-04:00

Dear olap4j community,

It is with immense pride that we deliver to you today version 1.0 of olap4j. Yup. And you should be proud as well. Thanks to you and to the concerted efforts of countless collaborators, advisers, testers, integrators and developers worldwide, we were able to reach this historic milestone.

If you would allow me to exagerate blatantly for a second, olap4j 1.0 is what I believe the greatest and bestest (yes, you've read it right, bestest) version to have ever been granted to humanity.

At this point, I encourage you to go read the official press release instead of reading my futile efforts at proper journalism. Our fine folks really know how to best express what olap4j represents, why it is important and why it *might* be the bestest thing on Earth. I've also resisted the urge to copy / paste their fabulous text. That said, I'll do my part and get into the more technical details.

You will first notice that the distribution files have changed. The XML/A driver is now separated from the core API. This is a good thing for the project, as it will allow us to fix issues and release more often. Those of you who are using the JDK 1.4 compatible distribution will still get the XML/A driver in the same library. The four binary libraries are available on our Maven repository and Sourceforge project.

Among other changes, the top levels of the connection's metadata have also received some upgrades. A new metadata element was introduced at the top; the Database. We have also removed everything that was marked for deprecation as of 1.0. We therefore encourage people to test carefully while upgrading.

The reference implementation, Mondrian, is compatible with olap4j 1.0, but only for version 3.3.0.14192 and upwards. This means that if you are using both Mondrian and olap4j, the only compatible build is not certified nor tested anymore than any regular CI release. On production systems, you should abstain from upgrading to Mondrian 3.3.X or risk the consequences, although the 3.3.X series are very similar to what we released with the 3.2.X cousin. Anyhow, bottom line is:

If you're only using olap4j and the XML/A driver, upgrade at will!
If you're using Mondrian too, we suggest waiting for Mondrian 3.3 official.
If you like to live on the edge, grab olap4j 1.0.0.445 and Mondrian 3.3.0.14192. They are both available in our Maven repository.

That's it for now. As usual, drop by our forums or mailing list if there is anything we can help with. And again, congratulations to everyone for the hard work.

olap4j's maven repository cheat sheet
url:
    http://repository.pentaho.org/artifactory/
group:
    org.olap4j
artifacts:
    olap4j
    olap4j-xmla
    olap4j-jdk14
    olap4j-tck

Mondrian High Cardinality Dimensions Demystified

2011-03-01T12:34:00.000-05:00

Performance is the buzz word in Analysis these days. Real time, MapReduce, scalability. Mondrian is no exception in this trend. As the worldwide pool of data grows, our analysis software must follow in its wake. I often get questions related to Mondrian performance, and most of the time, people expect a magic answer to their performance issues. The answer to solve them all.

Mondrian, like other ROLAP engines, are notoriously hard to fine tune for performance. There are many pieces of technology collaborating together, and the final performance will be that of the weakest link. In Mondrian's case, the performance must therefore be taken into account at all levels:

We must perform efficient internal processing
We use the leanest internal data structures
We must generate 'scalable' SQL queries
Many many other things...

The two first points are easy enough to understand. Good software is lean and efficient. There is no black magic here. The third, on the other end, sounds like nothing more than a marketing buzzword, eh? Well... there is more to it. As I said earlier, the total performance will be that of the weakest link. It follows that Mondrian, being a ROLAP engine, must at all times issue the best SQL queries possible to the underlying RDBMS, or suffer the consequences. Why 'scalable', and what does it mean here?

When designing a data warehouse, the number of records in your fact / dimensions tables will greatly influence the overall system performance. Again, no voodoo dolls here, just plain commonsense. Some dimensions are huge. We're talking millions of members. We call those High Cardinality Dimensions (HCD). They are in fact a common occurrence. A popular website's data warehouse and their user's dimension table is very likely to contain millions of rows. A reseller will have a products dimension table just as big, or more. A common way to safeguard the performance of those tables is to use table partitioning. In short, this means your RDBMS will split the table in logical sections. When queried, the RDBMS will only scan the partitions relevant to the query, therefore leaving alone most of the data. This all sounds good in theory, but there is a very important implied requirement here. The SQL queries must take this into account. Issuing a SELECT COUNT(*) on the table will require the RDBMS to scan all partitions, thus eliminating the advantage gained by partitioning the table.

So. Today's article will cover Mondrian's means to leverage the table partitioning of the underlying RDBMS. We will only cover the case of HCD. Not the fact table. I might write something later, but in the meanwhile, there are some very good articles out there on the subject.

Paradigm shift. We are a user. We are seated in front of an OLAP exploration tool. It is connected to a Mondrian instance. We drag the customer's dimension in the grid along with the regions and sales measure. We press the execute button. The query takes 10 seconds to return. We are frustrated. We blame the guy responsible for the Mondrian instance.

Paradigm shift. We are the guy who administers the Mondrian instance. Looking at the SQL logs, we see the following:

select
r.region_name, c.customer_name, sum(f.sales)
from
sales_fact f, regions r, customers c
where
f.region_id = r.id and f.customer_id = c.id
group by
r.region_name, c.customer_name

Remember my previous post about segments and internal cell data representation within Mondrian? The query above is due to Mondrian's decision to create a single segment based on the query the user issued. This query looks great, right? No. Well, it depends. It's fine IF none of the tables here are partitioned. If they were, the query would scan all partitions. This is where Mondrian HCD capabilities come in. Mondrian schemas have an attribute on the CubeDimension element class to instruct it to switch to HCD mode when dealing with that particular dimension. It is detailed here. The end result is that the single segment is split in smaller segments, each covering a single tuple. The SQL generated becomes:

select
r.region_name, c.customer_name, sum(f.sales)
from
sales_fact f, regions r, customers c
where
f.region_id = r.id and f.customer_id = c.id
and r.region_id = 42 and c.customer_id = 1
group by
r.region_name, c.customer_name

select
r.region_name, c.customer_name, sum(f.sales)
from
sales_fact f, regions r, customers c
where
f.region_id = r.id and f.customer_id = c.id
and r.region_id = 42 and c.customer_id = 2
group by
r.region_name, c.customer_name

(...)

select
r.region_name, c.customer_name, sum(f.sales)
from
sales_fact f, regions r, customers c
where
f.region_id = r.id and f.customer_id = c.id
and r.region_id = 42 and c.customer_id = N
group by
r.region_name, c.customer_name

We are now assured that each query touches one and only one partition. It might seem less than optimal to issue so many queries, but when you get down to benchmarking it, the performance is quite good. Not only do small queries take a very small amount of time to execute on the RDBMS side, they are very lightweight and can be handled by Mondrian extremely efficiently. As a matter of fact, in most cases where I tested this, it was more efficient to issue all of those small queries than the cross-partition big wooping query.

But we can do better. By changing the value of the mondrian.result.highCardChunkSize property, we can batch X tuples per segment, compared to a single one. Setting it to 5 would result in queries looking similar to this:

select
r.region_name, c.customer_name, sum(f.sales)
from
sales_fact f, regions r, customers c
where
f.region_id = r.id and f.customer_id = c.id
and r.region_id = 42
and c.customer_id in (1,2,3,4,5)
group by
r.region_name, c.customer_name

The chances that the customer_id predicate overlaps over two different partitions are quite small. Even if it was to happen, it would still touch no more than two partitions as compared to all of them previously. There are more ways to optimize Mondrian for HCDs, but that's it for today.

Welcome!

2011-02-13T12:38:00.000-05:00

I was searching the internet for some technical documents and stumbled on this website where the documents were behind a password prompt. What struck me is the weird password prompt, which was not the standard Firefox BASIC authentication prompt. Using curl, I discovered this gem

<SCRIPT language="JavaScript">

</SCRIPT>

Welcome indeed.

Mondrian SPI SegmentCache

2011-02-03T16:26:00.000-05:00

Fellow Mondrian developers and users,

One month has already passed since the new year festivities, and while most of you have been trying to renew your gym membership or hold on to your new year resolutions the best you could, so did the Mondrian team. Our resolutions, although not requiring personal sacrifices, are none the less starting to bear fruit.

For you see, our resolution for the year was to provide Mondrian developers and integrators means to achieve better understanding, scalability and control. We have many ideas on how to reach those goals. Some of them are still in their infancy, yet some of them have already been committed to the source. Last month, we worked on the first phase. We added means for system architects to externalize and share a pluggable segment cache. What does this mean exactly? Let's take a step back in order to better understand.

Internally, Mondrian splits the tuples in segments. A typical segment could be described as a measure crossjoined by a series of predicates. As an example, a textual representation of a segment contents could be:

Measure = [ Sales ]
Predicates = {
          [ Products = * ],
    [ State = California ],
    [ Gender = Male ] }
Data = [ 1346.34, 234.00, ... ]

In the case above, the segment would represent the Sales data of all males in California, for all products. It is a lot more effective to deal with those data structures. If Mondrian was to internally represent each data cell individually, the unique identifier of that cell would be of a greater size than the data itself, thus creating a whole lot more problems in terms of data efficiency. This is therefore why Mondrian deals with groups of cells, which it loads in batches, rather than individually. There is a lot of voodoo magic and heuristics in the background trying to figure out how best to group those segments and how to reduce the number of segments to load, ultimately reducing the number of SQL queries to be executed. Mondrian will group all segments with the same predicates but with a different measure into a segment group. Mondrian will also tend to remove as many predicates as it possibly can in order to optimize the data payload. Lets say that a segment covers all products except a single one, Mondrian will still include the product in the segment but filter it out when a specific query requires it.

Once those segments are populated, Mondrian keeps those in a collection of weak references in local memory. All required segment references are pinned down during the resolving of a particular query, but as soon as the query is done executing, the references are returned to their weak state, thus ready to be garbage collected if needed. This simple mechanism allows Mondrian to answer just about any query, as long as the memory allocated is big enough to answer that particular query. This works really well in fact, since in most small deployments, the maximum amount of memory is never reached. And if it ever gets filled, old segments will be evicted to make some room for the new ones.

Now, there are obvious gotchas. First off, what if it takes a long time for a segment to be populated by the RDBMS. This means that if a particular segment ever gets picked up by the garbage collector, the MDX query sent to Mondrian *might* take longer to execute, whether it was in the segment cache or not. This is not acceptable, simply because this makes all performance predictions impossible.

This is where the SegmentCache SPI comes in. It is essentially a pluggable cache for segments. The algorithm behind the segment loader becomes this:

Lookup segments in local cache and pin those required.
Optimize / group segments
Lookup segments from the SPI cache
Load the segments found from the SPI cache
Populate the remaining unloaded segments from the RDBMS
Put the segments which come from the RDBMS into the SPI cache
Pin all loaded segments
Resolve the query
Unpin all segments in the local cache

But wait! There is more! The SegmentCache SPI is trivial to implement.

Future<Boolean> contains(SegmentHeader header);
Future<SegmentBody> get(SegmentHeader header);
Future<List<SegmentHeader>> getSegmentHeaders();
Future<Boolean> put(
SegmentHeader header,
SegmentBody body);
void tearDown();

Figure 1. Mondrian Segment Loader Architecture

There are two assumptions that are made towards the implementation. The first obvious one is that the cache must assume that many Mondrian instances might access the cache concurrently, form different threads. We therefore recommend using the Actor Pattern or anything similar in order to enforce thread safety. ~~The second is that SegmentCache implementations will be instantiated very often. We therefore recommend using a facade object which relays calls to the actual segment cache code.~~ Update: This was redesigned so that a singleton is created and used throughout Mondrian's internals.

As for the storage of the SegmentHeader and SegmentBody objects, we tried to make it as simple and flexible as possible. Both objects are fully serializable and are immutable. They are also specially crafted to use dense arrays of primitive data types. We also tried to make extensive use of Java native functions when copying the data to / from the cache within Mondrian internals.

The bottom line is that from now on the Mondrian community will be free to implement segment caches to fit their needs. We will be rolling out a few default implementations and examples, obviously. One neat implementation could be one which pages the segments to a super fast array of SSD drives. Another one could be to store the segments in Terracota or ehCache or Infinispan, or just about any scalable caching system there is out there. So if any of you out there are interested in implementing this SPI for your business and would like to either share your experiences or contribute those implementations, don't hesitate to contact us. Or me directly.

There is more goodness to come, but that's it for now. Stay tuned!

olap4j 1.0 - The long road to LTS

2010-11-01T12:11:00.000-04:00

Last summer, we at olap4j announced that we would release olap4j 1.0 on October 31st 2010. Now is November 1st and olap4j is still not out of the door. Here's why.

Our reference implementation, the Mondrian project, suffered from delays in the release dates and had to be pushed back. All this kept us very busy and scrambling to make the best of it for a while. But fear not! This very week, the Mondrian team is running QA tests on a 3.2.1-GA build and as soon as it gets the green light, we will be able to put the finishing touches to olap4j 1.0. Why are these projects tied so closely? one might ask. Legitimate question.

You see, every API is a better API if there is at least one reference implementation existing in the wild. For many reasons actually. It allows us to architect, develop and test in a real environment, with real constraints and real data. Several APIs have failed in the past because in the end they were overly complicated for the end user. Some were so overly complicated that in the end, they failed to deliver what an API is supposed to be; a simple interface to a given system. Having a reference implementation mitigates the risk by allowing us to release by little increments and release often, but most importantly, release testable code that works. Another huge advantage is that for every part of the API, there is at least one fully functioning implementation out there, freely available as open sourced software, which implementers can refer to. This is a huge advantage in terms of both project sustainability and project adoption.

So stay tuned, because olap4j 1.0, despite some delays, is right around the corner!

olap4j tutorial

2010-09-21T18:17:00.000-04:00

Behold! I finally had some time to put the finishing touches to my tutorial for olap4j. It took a year but here it is. Sorry for the delay. I know, I know. I promised it a year ago, but a lot of stuff has been going on since then. Two new jobs, two moving to a different city (and once more next month...). So without further delay, enjoy!

http://olap4j-demo.googlecode.com/svn/trunk/doc/Olap4j_Introduction_An_end_user_perspective.pdf

From the olap4j team

2010-07-21T05:33:00.000-04:00

I posted this message today on the olap4j mailing list. In the interest of reaching a broader audience, I will copy it here as well.

Cheers!

Dear olap4j community members,

As we previously discussed on this mailing list, we are planning to make the final push towards a 1.0 specification. In order to perform those much needed changes and still maintain compatibility as much as possible, the olap4j team proposes the following transition plan.

4th week of July - Release of olap4j 0.9.8
A first initial release, coded 0.9.8, will be performed during the days to come. This release is mostly a wrap-up of the unofficial releases we have put in the Maven repository. Notable changes include compatibility with SAP BW, contextual drill-through for the Query Model, along with various other compatibility fixes.

Month of August
During the month of August, the olap4j community will have a chance to comment the proposed changes to the 1.0 final draft. We will provide an updated functional specification document as well as a complete list of the changes that will be required. Should you judge that some items are still missing, or that some should be modified or removed altogether, you are encouraged to let us know. The mailing list is the best place to hold those discussions, or you can also use our forums.

September 1st - Release of 0.9.9
September 1st is the date that marks the end of our discussions. After that, all the changes that we agreed upon will be implemented in the API, as well as the Mondrian and XML/A implementations of the driver. The 0.9.9 release will include those changes, but will still maintain retro-compatibility. Some API calls will be marked for deprecation, new ones will be present as well. The 0.9.9 release will be the last available before 1.0. Everything that is marked for deprecation will be removed as of 1.0, so users will have a chance to convert their code base progressively.

October 31st - Release of 1.0
We are planning to release olap4j 1.0 on October 31st. All methods that have been marked as deprecated, whether by the 0.9.9 release or any other previous 0.X release will be removed.

A proposed updated functional specification document, as well as a detailed list of API changes will be sent in the following days, right after the 0.9.8 release. We strongly encourage the users of olap4j to express any concerns or ideas that might arise.

Sincerely yours

Luc Boudreau, for the olap4j development team

Pentaho’s Road To Profitability - Take 2

2010-06-29T19:33:00.000-04:00

Tom Barber, a collaborator of mine, one of the most active community member of the Pentaho project, posted a very interesting blog entry this week. He is looking back at the path of Pentaho Corporation, a commercial open source business intelligence company, and their latest strategies for growth. He notes that in the past months, Pentaho has put a lot of effort in marketing initiatives and hired many big wigs in their marketing staff. In his opinion, this is somewhat against the ideals of commercial OSS development.

The commercial open source business model is still very young and has not encountered many great challenges up to now. Nor has it sailed in very troubled waters. Yet some signs are already warning people to be very careful with the years to come. Sun Microsystems (now Oracle) was surviving thanks to donations. Compiere recently made the news for all the bad reasons. Red Hat is doing pretty well though. In a nutshell, anything is still possible; good or bad.

Tom’s wish was that Pentaho would rather focus on paying skilled engineers rather than sales people. Now, as much as somewhat agree with the general idea, one must keep in mind that there is no correlation between the number of talented people getting paid to be on a project and it’s success. Some notoriously successful projects depend almost entirely on it’s community base. Mozilla Corporation to name only one. Others employ thousands of employees, yet achieve mediocre results. One could also argue that an effective marketing campaign will in fact boost people’s awareness, thus getting more talented people to join the community base. There is no tested and fail proof recipe so far. As I said earlier, everything goes.

I worked for the past months for a commercial open source company, and the same questions and uncertainties were part of every day discussions. How can a software company making no revenues on licensing be profitable? SQL Power Group, my employer, sponsors the only open source data modeling tool that works cross-platforms and offers, for free, the majority of the functionality included in widely known proprietary tools like ErWin. There are on a weekly basis between five hundred and a thousand downloads of SQL Power Architect. This is a lot for an OSS project in such a narrow niche. Yet a very small proportion of those are actually paying for support/consultancy, or making donations.

There are a lot of factors in play. First off, people willingly using OSS have overall better technical skills than others. To be fair, not every OSS is of acceptable quality, but you get to try as many as you want. Proprietary software is not better in any way, but you won’t get to try many of them, short of having very good contacts and/or deep pockets. This is one reason why reaching the tipping point of adoption is very hard for an OSS company. The percentage of OSS users ready to pay instead of figuring things by themselves is low. Way low. Waaayyy loooww. If you paid for software, you want support (and vice versa). You feel obliged to use the software because you already invested much in it. (Yes, money is time, time is money.) If you didn’t pay for the software and you hit a snag, you are far more likely to stop using it altogether. By abandoning it, you feel like you are exercising damage control, since you save time (thus money). You didn’t pay so why invest time? I’m fully aware that once you do the math, you realize that it’s the exact opposite. People usually don’t.

What does all that have to do with Tom? Well first thing first. Starting next week, I’ll be an employee of Pentaho Corporation. Yup. Am I a big wig? Nope. Am I a marketing guy? Well, I’m good looking, but not good enough with oxymoron. Am I a marketer? I’m having trouble making this blog post interesting, so nope, not a marketer either. I’m just some skilled dude who got his hands dirty for years and now decided that he would devote some serious time to make things better.

Look at it this way. In Tom’s vision of Pentaho, my ass is on the line. I deeply respect his insights on the market and software development in general. Yet here I am, proud as a peacock on a Sunday brunch, about to be part of this fantastic project. Am I worried? Hell no. I’m very enthusiastic about Pentaho having enough budget for marketing initiatives. This is exactly what OSS companies need; market awareness. People need to know you exist and that you are just as good as all those Fortune 100 companies. Even better than these. It has nothing to do with being a sell out. Marketing is not bad for your grassroots values, unless you make it so. Then again, you have only yourself to blame.

OSS companies need to reach the critical mass of contributors and adoption needed to survive. There is simply no other way. OSS users are picky, grumpy, bitchy and unforgiving. I know that for a fact. You want to thrive in that market share? You need two things. Proper marketing and skilled engineers. I’ll be doing my part in the later, and I’m very confident that the good people at Pentaho have picked skilled people to cover the former.

We are competing against giants, so let’s give them a ride for their money.

Monkey Business

2010-05-05T05:38:00.000-04:00

Last week we worked on a guerilla-marketing video for Wabit. I'll let you be the judge of that.

mdx4j - MDX query language parser

2010-04-12T05:49:00.000-04:00

Last week I launched a spin off of the olap4j parser. Mdx4j wraps olap4j's MDX parser and makes it available to code, without the need of an olap connection.

Why?

Although olap4j contains a SPI parser, we don't want to promote any particular MDX syntax. I therefore packaged it as a separate project so that everyone can have a piece of the pie!

final String query =
"SELECT{} ON COLUMNS FROM CUBE";

final MdxParser parser =
Mdx4jParserFactory.createMdxParser();

final ParseTreeNode tree =
parser.parseSelect(query);

http://code.google.com/p/mdx4j/

All in one BI tool for the non-geeks

2009-09-30T07:44:00.000-04:00

A colleague of mine once asked me if I knew a program that can connect to almost all relational databases and offers MsAccess like features to build queries. Sure thing says I. Wabit.

So he downloads it and installs it in 5 minutes. It's free and open source. No hassle. He then creates his connections and manages to do everything he needs to fulfil his duties as a business analyst. Pretty kewl story, heh? Short too. But that's a good sign because as a developer on this project, I can confirm first hand that this is exactly what we aimed for. Making business intelligence easy and painless.

The Wabit is more than that. It's also an OLAP data warehouse browsing and reporting tool. It creates charts in 10 seconds and features a template engine for easy corporate branding. Version 1.0 will feature a server repository for multi-user collaboration and incremental saves, scheduling and fine grained security. The enterprise server is not open sourced though, but the Wabit client is a fully featured platform. You can still save all your queries and reports as an XML file for easy import and export and share it with your fellow co-workers.

The Wabit approaches 1.0 now. We need to reinforce the community around it and we need more feedback. The Wabit works on all platforms with a Java JVM, so whatever your background is, I'm sure that we can make good use of your comments or contributions. You are a GUI designer or a BI consultant or even just the regular Java developer, we have need of your help.

Wabit on Google Code
Wabit homepage

For the geeks out there, here are some of the technical details.

The Wabit is a Java thick client. It requires a Java 5 JVM or better. Java 6 is recommended. There are OS specific installers available, or a generic package that is platform agnostic. They are available on the Wabit Google Code site.

In order to connect to your data sources, the Wabit requires a JDBC or olap4j compliant database driver.The Wabit comes with some of them, like MySQL and SQL Server. One could also use Sun's ODBC driver and access OS provided datasources.

Edit
I forgot to mention that it can query against SQL Stream real time queries as well.

Units of work are separated in workspaces. One can open/save his workspace XML files and work on multiple workspaces at any given time. A workspace includes queries, reports, images, charts and templates.

I'm pretty sure you can lean the rest by yourself from there. There are no video tutorials yet available, but it is on the TO DO pile.

Olap4j vs. Oracle and Ruby

2009-09-09T07:06:00.000-04:00

During my monthly checkup of this blog analytics data (thank you Google Analytics), I discovered a new trend. More and more, people are searching for information on olap4j's compatibility.Here are the interesting keywords used and the number of occurrences for the last month.

"olap4j ruby" - 28 occurrences
"olap4j oracle" - 3 occurrences

Oracle; I can understand. Olap4j is picking up momentum and is more widely adopted. We support both Microsoft Analysis Services and Mondrian via XMLA. Oracle does have an XMLA server, Hyperion Essbase, although we never tested it with olap4j. If one of you reading this post happens to be a Oracle wizard, please contact us so we can have a chat. The more OLAP servers we support the better.Ruby; now that's intriguing. Ruby can run in a JVM thanks to the JRuby project. Would olap4j work well with JRuby? probably. Are there are any OLAP API for Ruby? Google says no. Digging further in the analytics data didn't reveal to me the actual intent of those who are searching for "olap4j ruby" keywords. What a mystery... I therefore send out there a general call to anyone interested in using olap4j inside JRuby, for we might have common interests.

Creating Mondrian Schemas with Power*Architect

2009-08-28T05:09:00.000-04:00

Since I don't have time to write much software myself these days, I figured I'd share this gem with you all. SQL Power, the Canadian Business Intelligence Authority (that's their tag line these days...), sponsors many open source projects. One of them is called Power*Architect; a marvellous cross-platform data modelling tool.

As far as I know, there are close to none "enterprise ready" data modelling tools that work on Linux and Mac. I also suspect none are free, whatever the platform. Visio is certainly not one of them.

Why is it so wonderful? Well, to start with, it can retro/forward engineer most JDBC compatible databases. That's a big plus. And it gets better. You can also use it to create a Mondrian schema. Yep. The team at SQL Power published a tutorial for that last week.

I do have to disclose that I will be working on their projects starting in October. I'm not trying to sell it to you; it's free anyways. One thing is for sure though. I can't wait to get my hands in there. So I encourage everyone to grab a copy here and fill as many bug reports as you can. It's not 1.0 yet, so community contributions are a must. Having worked with the team for three weeks back in July, I can guarantee that each and every reported bug and suggested feature is closely studied by the development team.

olap4j - A comprehensive tutorial

2009-08-20T09:52:00.000-04:00

I've been very busy lately with the new job comming up and many other changes in my personal life, but fear not; I'm cooking something up for you people. I'm working on a comprehensive guide to olap4j. Many people have expressed a need for a more step-by-step introduction on olap4j, what it is, and how to unleash it's raw power. In the next few weeks, I should be able to finally put some more time on it and release a first final draft. Until then, take care!

UPDATE: I finally finished the tutorial. See http://devdonkey.blogspot.com/2010/09/olap4j-tutorial.html

Connect Microsoft SQL Server from olap4j

2009-05-21T05:38:00.000-04:00

Browsing my Google Analytics statistics, I realized there is a lot of people out there that are searching for ways to connect Microsoft SQL Server with olap4j.

Here is a nice example.


// We must use the XMLA driver.
Class.forName("org.olap4j.driver.xmla.XmlaOlap4jDriver");

// This code is for Java 5. With Java 6, you can directly
// unwrap the underlying connection with the .unwrap() call.
OlapConnection connection =
(OlapConnection) DriverManager.getConnection(

// This is the SQL Server service end point.
"jdbc:xmla:Server=http://example.com/olap/msmdpump.dll"

// Tells the XMLA driver to use a SOAP request cache layer.
// We will use an in-memory static cache.
+ ";Cache=org.olap4j.driver.xmla.cache.XmlaOlap4jNamedMemoryCache"

// Sets the cache name to use. This allows cross-connection
// cache sharing. Don't give the driver a cache name and it
// disables sharing.
+ ";Cache.Name=MyNiftyConnection"

// Some cache performance tweaks.
// Look at the javadoc for details.
+ ";Cache.Mode=LFU;Cache.Timeout=600;Cache.Size=100",

// XMLA is over HTTP, so BASIC authentication is used.
"username",
"password" );


// We can execute a query. MDX of course.
CellSet set = connection.createStatement().executeOlapQuery(
"SELECT {} ON COLUMNS FROM CUBE");

Update : Some useful links

How to: Install Analysis Services by Using Setup

SSAS Clients Architecture

Olap4j latest builds

PAT 0.3 - Integration and nifty features…

2009-05-20T10:51:00.000-04:00

Pentaho Analysis Tool sprint 2 is now over and released since a few weeks. We're currently in the process of wrapping up the third sprint. What's new? What new features will there be?

Nothing very spectacular really. Yet there are a few nice features worth mentioning.

Integration with the Pentaho BI server
PAT can now run embedded in the Pentaho User Console and be configured remotely. It only supports XMLA connections for this first draft, but don't worry; more compatible connection types are to be supported in the medium term. This also means that you can seamlessly use any XMLA provider. Sql Server? Essbase? Mondrian? You choose. All that thanks to olap4j. If you want specific details on this, Gretchen Moran has written a bunch of documentation on this. Congratulations to Paul Stoellberger and Gretchen Moran for this feature.

Create multiple queries at once
This was a requirement that was passed to us by Pentaho's engineering team. People want to build and use more than one query at a time without concurrency issues. That was not properly supported by the age old JPivot application. Now and then we encountered some problems with multiple queries, so this was something pretty high on our features list. The backend supported that since sprint 1 , but there was higher priorities for the GUI components.

OlapTableModel first draft
The Java API doesn't include a proper TableModel for OLAP data, so we're planning to write one. We still have lots of things to figure out on this subject, but we're planning to mock a draft specification before the sprint 3 deadline.

Our initial intent for the third sprint was to make the GUI nicer, but we're still trying to figure out if XUL would be a better generation tool than SWT. We are waiting for developments between the GWT-Mosaic team and Nick Baker of Pentaho and hope that all those nice gents would team up and cook something up to their talent. Fingers crossed here. :)

Also worth mentioning, we started talks to review the Query model currently residing in olap4j. This is a soon to be major component as it will provide developers and GUI enthusiasts with a properly abstracted API to build queries against a data warehouse. Olap4j is very great API for low level stuff, but it still needs an abstraction layer for the common folks. Building queries should not require in-depth knowledge of MDX, for you cannot expect all business analysts to master MDX anyways. Anyone who wants to participate in the process is warmly invited to manifest himself. We are really looking for input on this, whatever your background is. (As long as it's related to BI I guess...)

Pentaho Analysis Tool - Final push for sprint 2

2009-04-13T09:18:00.000-04:00

I've been really busy lately working on sprint 2 of Pentaho Analysis Tool. We almost reached all our goals for this sprint and are hoping to wrap it all up in the next two weeks. There is still time to add a few late requirements in this sprint so if anyone has a very special wish, now is the time to express it.

About Pentaho Analysis Tool (PAT)

PAT is an attempt to replace the good ol' JPivot application, widely used in the Java world, as a web based browser for OLAP data. There are quite a few similar projects out there, yet none of them quite make the cut in terms of enterprise requirements.

Ad-hoc connections

ACL management

User defined connections saved for later use

Saved queries

Multiple queries editing at once

[ insert even more entreprise software mumbo jumbo here]

We're writing a Google Web Toolkit (GWT) front-end and a Spring based backend as a core. All data manipulations are possible thanks to the Olap4j API. There was talk of a Json bridge later on in development, but this requirement is not part of any sprint planning for now.

The project is hosted on Google code and all the project management is done in the Jira tracker. If you have any further questions about our project or want to chat for whatever reasons, we can be reached via the mailing list or ##pentaho.pat on freenode.

Of easy and painless systems monitoring

2009-03-25T05:47:00.000-04:00

I'm not a systems administrator. I only have 8 servers to babysit and it used to be enough to be a time consuming problem. You might not be a systems administrator either, nor have many machines / services / websites to monitor, yet the fact remains that as IT professionals we need to keep a close eye on what's going on. I'm not talking about 99.999% uptime here, but a 1% downtime is enough to make a lot of customers, clients and managers angry; especially since outages have a way to happen exactly when it should not.

What are your options? How much does it cost? What can you monitor? These are all questions I'll try to shed a light on. The solution I'm proposing today is one I used myself for years. I'm not legally obliged to 5-9 availability, yet this is what I achieved with a total cost of 0. Yep, z.e.r.o. zero. El zilcho.

I'm not saying this will work for anybody, neither am I pretending to be an expert on the issue at hand, but I learned a lot in a few years on the subject so here it is.

Innovation comes from the needs

First thing first. These are the questions you absolutely need to answer before putting your automated monitoring solution together.

If any abnormality is discovered, who will be reached and how ?

Will you setup rounds of monitoring among your partners or employees? Do you have cell phones that can receive emails? Are you using pagers? If you can't answer to these questions, stop now. Monitoring is not for you apparently. You either don't have the resources or the availability to do so.

What needs to be monitored?

There is a whole list of things that can be monitored. As a matter of fact, pretty much anything can. Room and server temperatures, machine responses to arbitrary queries,query execution times, memory and disk usage, network throughput... the list long. As I said, I'll show you how to monitor all these things, but you need to know what is critical and what is not. You can try to monitor everything, sure, but keep in mind that the whole point of automated systems monitoring is being reliable and saving time. Therefore, you'll fail in the later. Make a list.

The tools of the trade

There are lots of monitoring systems out there. Most are expensive, lots are hard to setup. I'm not interested in either one of those categories. I guess you're not. My tools stack is this :

Zenoss Core : Monitoring platform.

SNMP : Monitoring communications standard I use.

Zenoss Core is a free (as in free speech or free beer) open source monitoring platform. It's distributed under the GPL license and available for download in a plethora of package formats. RPM, VMWare virtual appliance, source tarball, zip files, whatever. This is the core of my monitoring solution.

I suggest installing it on a cheap commodity PC with CentOS. Everyone has one of those old machines lying around somewhere in a closet or a warehouse. Find one, plug it, setup the OS of your choice (preferably CentOS, since it's Red Hat binary compatible and FOSS). My preferred way to monitor is through a network via SNMP (I'll get to that later on), so you need a network card on it. If you want it to send messages to pagers, you'll obviously need a modem. That's all the hardware you need.

Getting Zenoss up and running is pretty straight forward and well documented. CentOS is a RPM based system, so one or two command lines usually does the trick.If you did choose CentOS, as I strongly suggested, checkout chapter 5 of the installation guide. Installation should be a trivial matter and takes somewhere around two minutes. Literally.

SNMP - Simple Network Management Protocol

I mentioned SNMP as a monitoring communication standard. What is SNMP exactly? It's a simple way of querying appliances and getting information in the form of a numeric tree. This is where you need to do your homework. It is vital that you understand at least a little what SNMP is and how it works. Don't be afraid though. It's widely used, documented and simple enough to be understood in a 30 minutes research. Skip your favorite TV program tonight, and read a little. 30 minutes of reading will save you hours of problems down the road; your call.

SNMP is supported by a wide variety of devices. Windows Servers, Linux servers, routers, switches, printers, UPCs, name it. It's easy to activate and allows you to monitor pretty much anything.

On the downside, SNMP might not be the most secure monitoring solution, but it does the job fairly well. You can secure it further through firewalls and other such devices later on.

Installation on a Windows machine is easy. Go to the Add/Remove Programs utility, open the list of Windows components, look in the Management and Monitoring Tools and activate Simple Network Management Protocol. You can Google for it if you need more help. It should appear in the services management utility under SNMP Service, which you can fire-up by typing 'services.msc' in the Run dialog.

Linux servers need to install NET-SNMP. Again, I won't go through the details, since the installation process depends on your Linux flavor, but it's usually a trivial matter. Google for it if you need help. It's usually included in your distro base packages. Zenoss offers a nice howto that might be of great help.

Whatever platform you setup, make sure to allow network traffic on the port 161 in UDP. This is the usual port that SNMP agents use. It can be configured to be something else in your respective SNMP agent configuration.

Putting it all together

The hardest part is over. You can go grab a beer and celebrate. Don't celebrate too much though, because you're not done yet. We still need to fire up Zenoss, configure what we want to monitor and setup alerting rules. I'll go through the basic steps very quickly, but there are tons of documentation out there. I suggest using the Administration Guide as a helper.

Zenoss runs by default on HTTP port 8080. Navigate to

http://[your-server]:8080

As of version 2.3, the default login is 'admin' and the default password is 'zenoss'. You're now taken to the dashboard. Pretty slick heh? Select 'Add device' from the menu on the left.

Fill the following fields.

Device name : This is either a DNS or IP that points to the machine you want to monitor.

Device class path : Choose this value carefully. In the drop-down list, you need to select what describes best the device you want to monitor. If it's a Linux server, choose /Server/Linux. Guess what is the best choice for Windows servers?

SNMP community : This is the name of your SNMP community that you specified in your SNMP agent configuration. It's usually something like private or public, but some people like to use secret values.

Everything else can be ignored for now. As you master Zenoss, you'll find those extra fields most useful. Everything comes in good time, so keep focusing on the task at hand for now : press 'Add device'.

The next screen will slowly fill up with log messages. Zenoss is currently discovering your device and has a lot of values to checkup, so let it finish it's job. If everything goes well, you'll see a link to navigate to the device at the end of the log window.

If you get error messages, you did something bad. Start by checking the troubleshooting section of the Zenoss howtos. Most issues can be solved with the information provided there. If the problem persists, there is still your good friend Google.

By default, Zenoss will monitor disk usage, memory usage, CPU usage, system load, network throughput and a few other things. Once you have a first device setup, you can tweak the alerting rules in the 'Settings' section of your Zenoss installation.

That's it for now. Get familiar with Zenoss and devices discovery. This is your homework for this week. Next up, I'll tell you how to monitor IP services, temperatures and even more sexy stuff.

Cheers!

RESOURCES

Zenoss How-To's

Zenoss Installation guide

Zenoss Administration guide

CentOS installation guide

Evading (D)DOS attacks with Apache HTTPD

2009-03-16T06:13:00.000-04:00

Just a quick tech-tip. Ever wondered how to prevent your HTTPD server from being knocked off the net by a DOS (Denial Of Service) attack? Check this nifty little module.

Mod Evasive

Its pretty easy to setup. Compile the module as you would normally do for HTTPD modules and create a configuration file. There are many options available. Here's an example of how to configure it.

<IfModule mod_evasive20.c>
DOSHashTableSize 3097
DOSPageCount 6
DOSSiteCount 100
DOSPageInterval 2
DOSSiteInterval 2
DOSBlockingPeriod 600
DOSEmailNotify "my-monitoring-contact@domain.com"
DOSWhitelist 192.168.*.*
</IfModule>

More details on the configuration and how each parameter will affect the module behavior can be found out there on the net.

Beware though, before installing this, make sure you won't blacklist some legitimate users. For example, if you have a AJAX application that sends a burst of requests once in a while, it might get blacklisted. Make sure you test it in a development environment so you get the thresholds right.

Of economic opinions and commentators

2009-02-12T06:00:00.000-05:00

There is a lot of blogging being done out there in these times of economical crisis -- yes, it is a crisis. Most of it is utter garbage, mixing opinions with specifically picked facts to serve a given purpose, yet somehow, I still find it important to read it all. There is an old saying that goes something like : "Fool is the one who ignores what he considers not worthy, for wise is the one who can learn from anything."

Most of the wisest things i read were published in my favorite monthly publication, Le Monde Diplomatique. On the counterpart, a hellish lot of garbage can be found pretty much anywhere. Then again, during one of my many news scavenging sessions, I was genuinely surprised to find this little post from a man i never noticed before. I do believe this man has a proper sense of economical and political analysis. Here's an excerpt.

US policymakers have ignored the fact that consumer demand in the 21st century has been driven, not by increases in real income, but by increased consumer indebtedness. This fact makes it pointless to try to stimulate the economy by bailing out banks so that they can lend more to consumers. The American consumers have no more capacity to borrow.

With the decline in the values of their principal assets--their homes--with the destruction of half of their pension assets, and with joblessness facing them, Americans cannot and will not spend.

Why bail out GM and Citibank when the firms are moving as many operations offshore as they possibly can?

(...)

The US government really has only two possibilities for financing its budget deficit. One is a second collapse in the stock market, which would drive the surviving investors with what they have left into “safe” US Treasury bonds. The other is for the Federal Reserve to monetize the Treasury debt.

Monetizing the debt means that when no one is willing or able to purchase the Treasury’s bonds, the Federal Reserve buys them by creating bank deposits for the Treasury’s account.

In other words, the Fed “prints money” with which to buy the Treasury’s bonds.

Once this happens, the US dollar will cease to be the reserve currency.

In addition, China, Japan and Saudi Arabia, countries that hold enormous quantities of US Treasury debt in addition to other US dollar assets, will sell, hoping to get out before others.

The US dollar will become worthless, the currency of a banana republic.

I'll keep on the lookout for more interresting articles on this. I beleive that the current economical difficulties are of enoumous importance to us all. Not only are we at risk of loosing big, decisions will soon be made that will dictate the governance of our everyday life for decades to come. I might don't think much of the last decades of governance we just endured, but I certainly won't fallback to cynicism and apathy.

Comments? More reading suggestions?

Of OLAP and the importance of open standards

2008-11-11T11:35:00.000-05:00

In these times of economical crisis, many companies will turn to business intelligence (BI) as a source of wisdom and counsel. Millions of dollars will be invested in an effort to understand the extend of their respective problems and find solutions based on accurate and decision oriented datasets.

Since I have a fairly good amount of experience with work in heterogeneous environments and tackling data integration challenges, I thought I'd pitch in my two cents.

Why developers and project managers will have a hard time

The root of the problem is this. The Microsoft OLAP toolkit does not integrate so well with anything else than .NET technologies. SAS offers a Java API, yet it is not ready for production. (I worked with it for two years, and believe me, they are still a fairly long way to production quality code.) As a matter of fact, most software vendors in the OLAP world distribute some API to integrate their technologies, but you often end up with black boxes of questionable quality, flexibility or performance. Some even go as far as to obfuscate their libraries... this really doesn't help in the end.

Some vendors like Oracle went for the all-in-the-box solution. They offer a "complete" solution that can fit every possible need. Then again, what they are telling you is: if we don't have it, you probably don't need it."Probably"? You got to be kidding. Since when does software vendors know what you need and what your future will be? Better switch probably for hopefully.

In the best case, in order to meet your needs, you'll hack your way through at the expense of your project specifications. The final result can be nothing but deceiving. Your celebration will be bitter and probably short lived, I fear.

About the importance of collective work

You have a brand new application. Hooray! This is where the production phase kicks in.

What if you need to move your datamart to another OLAP server? What if there are not enough connections licenses to allow both production connections and all of your maintenance personnel on the OLAP server and they are forced to take turns to debug? What if the CEO decides to migrate to a new platform? What if [insert random but oh so frequent unforeseen event here]? Your thousand dollar code is now rendered useless; you can start crying now, you deserve it. In your quest for more money making, you've created a monster that was expensive and will continue to pump the money out of your institution pockets.

If you were good enough in systems design, you thought about a data layer. The data layer still remains to be rewritten entirely and it often represents at least a third of the overall effort. Close, but no cigar. This might sound like a catastrophic scenario, but it is oh so frequent.

Many people got tired of all this non-sense we decided to work together. We decided that enough time and money was wasted on individual efforts that were ruined in the end. It was time to agree on standards and share the product of our collective effort.

Take Hibernate for example. It is now a de facto standard when it comes to data mappers. For the Java version alone, it represents 859 thousand lines of code worth 12.8 million dollars in work hours. Think you can top that with your in-house data layer in times of economical crisis?

About Java OLAP

OLAP is a world in itself. You can't take relational paradigms and apply them to the multidimensional world. The .NET toolbox does have very nice libraries to do some neat OLAP stuff, then again, you're locked-in with SSAS. This is a no-no.

On the Java side, things are even worse. There is currently a big void in the Java OLAP market. No OLAP standard emerged at all. Thanks to the selfishness of the big players of the industry, the JOLAP initiative was a total failure. It never reached the final version, so the JSR-69 specification died quietly.

We at Olap4j tried to fill that gap with an open initiative. Everyone can pitch in. And I mean EVERYONE.

What makes Olap4j so kewl

You know the expression vendor lock-in? I hope you do, I *really* do, or else you'll learn it the hard way. Olap4j aims at solving exactly this problem. You can develop applications on it's API and switch the underlying OLAP engine without rewriting a single line of code. Not bad heh? Olap4j is more than a database driver. It is an open API built right on top of the JDBC industry standard where everyone collaborates to specify a common base onto which to build.

It even includes transformation libraries and testing facilities.

I want to kick the tires and use it right now

So far, it has two implementations ready to use. The Mondrian driver allows you to run the much acclaimed Mondrian open source OLAP engine as an in-process data provider.

There is also the XML/A generic driver that can connect to pretty much anything that talks XML/A, whether it's over HTTP or anything else you fancy using. This particular driver allows you to build applications that can switch to and from any of these OLAP engines :

Hyperion Essbase

Microsoft SQL Server Analysis Services

Infor

Mondrian

Palo

The Olap4j project is gaining momentum and we truly hope to see it become the standard in the Java world.

Chain blog… they cought me.

2008-09-24T04:18:00.000-04:00

If you're reading this, you've been included in this blog chain.

Take a picture of yourself right now.

Don’t change your clothes, don’t fix your hair... just take a picture.

Post that picture with NO editing.

Post these instructions with your picture.

I was caught by three techno blogs I follow, so don't be mad at me, Thanks to Julian, Matt and Nick. I had no webcam at hand but i thought about my phone, so there it is: Luc at the office Wednesday morning, before his coffee.

Cheers!

Data integration challenges tackled

2008-08-11T06:52:00.000-04:00

Data integration in business environments can be a painful task. I mean REAL painful. The volume of data is huge, it does not cross-validate, it is dispersed in many heterogeneous formats, yadi yada. You know the song. Some day, I stumbled on Pentaho Data Integration (PDI).This was a real breakthrough.

First thing first, it's not subject to "vendor lock-in". It can read most data formats out there and can write it back to pretty much anything. This is a huge plus because gives it the ability to be used by a plenitude of user types and environments. Being written in Java also gives it an edge as an enterprise tool, for it is platform agnostic.

But the real advantages are not those trivial specifications. My love for PDI has much deeper roots. Simply put : it's powerful. Creating an integration process is a trivial matter. Drag and drop. Link. Execute. Those three simple steps will cover most of your business needs. Really, I mean it. Never again will I write a snippet of code to read a CSV file and write it's content in a database. Mark my words; NEVER! This is a waste of time and a developer who lives with his times should know that.

What about the real juicy stuff ?

As you suspected, there is much more to PDI than meets the eye. It can be clustered, it can use a database based repository for all processes, there are automatic documentation generation tools and is supported by a huge community. Many tutorials exist to address most business needs and challenges. It's well made, very stable and easily expandable with plugins for power users.

I strongly recommend to give it a try. The next version should be released soon and it will include many great new features. I met Matt Casters last June and had the chance to see for myself all the new functionalities that will make it to the next release. We're talking about visual performance bottleneck exploration and some more neat stuff you won't find anywhere else.

Cheers, and have a good time integrating !!

Pentaho on the iPhone

2008-07-29T10:38:00.000-04:00

I successfully added the iPhone extension to my Pentaho platform today and I was more then impressed with the ease with which we can enable the whole platform to work seamlessly on those nifty little phones.

Oh yeah, I bought an iPhone too...

I'm slowly discovering the fun of having a cellular phone in my pocket. This is something that I never experienced before; never had a cell phone. I have to say that I'm glad it's a good phone, and sexy too.

The bottom line is : get one.

For those interested, here's the wiki page that says it all. Thanks to Will Gorman, senior developer at Pentaho, who put this all up.

Field lessons - Securing a Linux server

2008-07-21T07:31:00.000-04:00

There is this old saying which goes like :

"Linux is safe enough to keep it vanilla. Anything you add weakens it's security."

Okay, this is not an actual popular saying, but since most Linux server I saw in my career were configured in conformity to this piece of wisdom (sic), I decided to share some experience with basic and mandatory security measures to add on a Linux server... I'm just sooo tired of fixing broken servers that have been hacked.

There is a simple suite of programs to install and you'll be at the very least secured against kiddies and the like. Here it goes.

Securing the OS

Most of the time, the piece of software that was hacked was the OS itself. Not because there are awful flaws in Linux (or just any OS as a matter of fact), but because simple rules were not respected. How many of you who have configured servers can certify that they are protected against brute force attacks ? How many are protected against DoS attacks ? Linux, nor any other OS I've seen so far (correct me as you wish...) don't come with DoS or BF detection. Having a secured SSH access is mandatory these days, but what's the point of setting passwords when a simple brute force attack will break it.

Here are some solutions. The Advanced Policy Firewall (APF) is a simple Linux firewall that uses the iptables utility to create firewall rules on your system. Why APF and not iptables alone ? Because it integrated with a DoS detection tool and Brute Force Detection (BFD). The DoS tool will detect any Denial of Service attacks while BFD will monitor incoming connections and ban any IP who breaks easy to setup access throttling rules All these tools are free and compatible with most Linux flavors. Try em out! There are many more available from R-fx Networks, the company that maintains them.

As for the setup instructions, google for them; as always. There are many nifty tutorials out there and I won't copy them here :P

Web Applications Security

What if I told you that there is a generic way of applying a minimum security level to all your web applications at the OS level, thus simplifying the life of anyone who administers web servers. You might get frustrated by the fact that you didn't know this at the time you got hacked. You might even wonder how wonderful this would be for your web hosting server.

Well, I'm doing it.

I'll say it.

Ready ? There it is.

ModSecurity

Okay, this was the hard part. Now it will be much easier. It's a simple Apache HTTPD module that you add to your web server configuration and it will validate all requests against a set of nifty threat detectors. It uses regular expression to protect your applications against overflows, injections and whatever might be dangerous for them.

There is even a console available to monitor many installations and keep an eye out for alerts.

Easy to understand, easy to install. As always, google has all your answers.

The bottom line

The lesson to remember is that these tools take half a day of work to setup and they will save you sooo much trouble in the future that it is worthless to discuss the pertinence of using them. The tools are out there, for free. You'd be a fool not to use them.

CQFD

Olap4j and XML/A - One more step towards a true olap systems integration API

2008-07-15T06:53:00.000-04:00

I've been working for a month now on some enhancements to the Olap4j project to make it more powerful and compatible. The good news is, I succeeded. The previous version, 0.9.5, lacked some basic functionalities which you would expect from a production ready XML/A driver.

For one, it's HTTP proxy didn't support cookies. his was a big problem since the myriad of requests required to populate Olap4j's meta data objects each created a new user session on the web service back-end. This was a no-no, but now it's fixed and kicking ass.

I also worked on a SOAP query cache. This is was a big piece of software engineering, since I'm not used to thread safe coding. Thread safe thignys are usually in the lower levels of BI application servers and those issues are tackled from the start. Thanks to Java's java.util.concurrent package, this was a breeze.

Those changes are not part of any release nor in the SVN yet. I'm still waiting for peer review before the whole commit, but for people eager to see what it looks like, I've created a neat little package for y'all.

Now I can move back to my next release of the University of Montreal's Pentaho platform... all work and no play makes Luc a dull boy.

Cheers !