August 13, 2007
Contributed by Pedro Alves, Pentaho Community Member

Community Build Framework

Warning

This is a deprecated version. Please check the main page here

Like many others, I had a hard time installing and setting up a development environment for pentaho. I had some requirements and I think I managed to get a good approach. This were my premises:

I needed to know how to build a demo solution that connects to an arbitrary database and shows simple pentaho abilities in the least possible time (eg: have a demo set in 2 hours in a client's database)

I need to switch configuration on the fly so that I can switch from different scenarios (eg: from client A to client B)

I don't want to change any original files that could get overwritten in a upgrade

the platform upgrade must be easy to do and not break the rest of the setup

Debug is a "must have"

Must support all kinds of different customization in different projects, from using a different databases (I'm using postgresql , thanks elassad) to different security types.

decided to use not the pre-compiled binaries but the sources from svn, as described here: http://wiki.pentaho.org/display/Pent...o+with+Eclipse

Project pentaho already has a good dev_build.xml ant script, but it wasn't enough for my needs, since I needed to change the svn sources in order to configure a solution. So I built my own structure, and its like this:

MyProjectDir
|-- build.properties
|-- build.xml
|-- pentaho
|-- pentaho-data
|-- pentaho-preconfiguredinstall
|-- pentaho-solutions
|-- project-client
|   |-- patches
|   |   |-- pentaho
|   |   `-- target-preconfiguredinstall
|   `-- solution
|-- target-build
|-- target-preconfiguredinstall
|-- target-solutions
`-- updateall-svn.sh

pentaho* directories are cheched out from svn, and for me are read-only. Feel free to use any svn client you want; I usually work in linux, or on a windows machine with cygwin, so I use the original client. I checked out the sources with:

$ svn co svn://source.pentaho.org/svnroot/pentaho/branches/1.7 pentaho

$ svn co http://source.pentaho.org/svnroot/pentaho-solutions/branches/1.7 pentaho-solutions

$ svn co svn://source.pentaho.org/svnroot/pentaho-data/branches/1.7 pentaho-data

# next line for jboss PCI
$ svn co svn://source.pentaho.org/svnroot/pentaho-preconfiguredinstall/branches/1.7 pentaho-preconfiguredinstall

# next line for tomcat
$ svn co  svn://source.pentaho.org/svnroot/pentaho-tomcat/branches/1.7 pentaho-tomcat

If you are under a proxy you can use http:// instead of svn:// . I'm currently using version 1.7 (this one, or later, is required for tomcat support)

target-* directories are the ones generated by the build script. I don't use target-solutions at all (honestly, I'm not sure what this are for, cause they seem a copy of pentaho-solutions. Guess the point is to pick target* and copy it, w/o any other dependencies, to the desired machine). target-build is the temporary build directory. Since we may need to patch the pentaho source, we'll make an entire copy of pentaho/ to target-build and patch the later, leaving the original source intact.

This leaves us with the "project-client" directory. As I mentioned before, I don't want to write anything under pentaho* directories, so all my changes go to this directory. Here is the structure in more detail:

project-client/
|-- patches
|   |-- target-build
|   |   `-- override.properties
|   `-- target-preconfiguredinstall
|       `-- server
|           `-- default
|               |-- conf
|               |   |-- jboss-minimal.xml
|               |   `-- jboss-service.xml
|               |-- deploy
|               |   |-- PentahoHibernate-ds.xml
|               |   |-- client-ds.xml
|               |   |-- datasource1-ds.xml
|               |   |-- datasource2-ds.xml
|               |   |-- datasource3-ds.xml
|               |   |-- datasource4-ds.xml
|               |   |-- datasource5-ds.xml
|               |   |-- pentaho.war
|               |   |   `-- WEB-INF
|               |   |       |-- applicationContext-acegi-security-jdbc.xml
|               |   |       |-- applicationContext-pentaho-security-jdbc.xml
|               |   |       |-- classes
|               |   |       |   `-- hibernate.cfg.xml
|               |   |       |-- jboss-web.xml
|               |   |       `-- web.xml
|               |   |-- quartz-ds.xml
|               |   |-- sampledata-ds.xml
|               |   |-- sampledata_admin-ds.xml
|               |   |-- shark-ds.xml
|               |   |-- solution1-ds.xml
|               |   |-- solution2-ds.xml
|               |   |-- solution3-ds.xml
|               |   |-- solution4-ds.xml
|               |   `-- solution5-ds.xml
|               `-- lib
|                   `-- postgresql-8.2-505.jdbc3.jar
`-- solution
    |-- Portal.properties
    |-- Portal.url
    |-- Portal_pt.properties
    |-- admin
    .... etc

The idea is very simple: All changes that I would normally do under pentaho* I put them under "patches" directory. My top level ant script just picks them and copies the files to the top level directory. It's recommended to patch pentaho* as least as possible because we are actually overwriting the source files, but the file override.properties is meant to be changed. All other changes are made by patching the final directory, target-preconfiguredinstall. (I use a target-preconfiguredinstall-tomcat for tomcat, but that is configurable)

For my first scenario, those were the files I had to change. I had to change some jboss's config files just because I had to change a port that was unavailable; added my datasource (client-ds.xml) and changed the jboss-web.xml and web.xml to register it; changed all other ds's cause I'm using postgresql instead of hsql; added the jdbc lib; changed the security system from memory to jdbc.

This is scalable. Just change files there and they will be copied to the destination target. My ant file does not allow (yet) to remove files from target to destination, but I also don't think thats a relevant issue.

The solution is under project-client/solution, and override.properties points to this dir. For now, I just copied pentaho-solutions to here, and will start from there by adding new solutions and playing a bit.

Tip: Since project-client/solution/system is a generally untouched directory, it's easier to maintain the svn info there. Just copying the directory from pentaho-solutions/system or making a

svn co svn://source.pentaho.org/svnroot/pentaho-solutions/branches/1.7/system

will allow svn updates later. The same approach can be used for _admin, __samples_ and _test_ directories.
To extend customization to a greater extent, I've implemented some filter tokens for some common changeable attributes, such as database locations and passwords. For instance, instead of writing the database location I write (DB_LOCATION) and the correct value is replaced at compile time.

This is the top level build.xml file. Very simple, indeed:

<!-- ======================================================================
     description: main build file for pentaho  . .                                                    
     ====================================================================== -->
<project name="pentaho-build" default="all" >
	<description>Pentaho BI Platform build helper..</description>

	<property file="build.properties" />

	<condition property="choose.JBoss">
		<equals arg1="jboss" arg2="${server.name}"/>
	</condition>

	<condition property="choose.Tomcat">
		<equals arg1="tomcat" arg2="${server.name}"/>
	</condition>

	<target name="init">
		<echo>Building project ${project}</echo>

	</target>


	<target name="copy-init" depends="init" description="Copy project files">
		<!-- apply Copy dir -->
		<echo>Copying files</echo>
		<copy todir="${pentaho.build.dir}" failonerror="true" verbose="false" overwrite="false">
			<fileset dir="${pentaho.dir}" />
		</copy>

		<!-- apply patches -->
		<echo>Applying patches</echo>
		<copy todir="." failonerror="true" verbose="false" overwrite="true">
			<fileset dir="project-${project}/patches/">
				<include name="**/*.java"/>
				<include name="**/*.xml"/>
				<include name="**/*.conf"/>
				<include name="**/*.txt"/>
				<include name="**/*.properties"/>
			</fileset>
			<filterset>
				<filter token="JAVAC_PATH" value="${javac.path}"/>
				<filter token="SOLUTION_ABSOLUTE_PATH" value="${solution.absolute.path}"/>
				<filter token="DB_LOCATION" value="${db_location}"/>
				<filter token="DB_USER" value="${db_user}"/>
				<filter token="DB_PASSWORD" value="${db_password}"/>
				<filter token="BASE_URL" value="${base_url}"/>
				<filter token="GOOGLE_MAPS_KEY" value="${google_maps_key}"/>
			</filterset>
		</copy>

		<copy todir="." failonerror="true" verbose="false" overwrite="true">
			<fileset dir="project-${project}/patches/">
				<exclude name="**/*.java"/>
				<exclude name="**/*.xml"/>
				<exclude name="**/*.conf"/>
				<exclude name="**/*.txt"/>
				<exclude name="**/*.properties"/>
			</fileset>
		</copy>
	</target>


	<target name="dev-setup-tomcat" depends="" if="choose.Tomcat" >
		
		<ant dir="${pentaho.build.dir}" antfile="dev_build.xml" target="dev-setup-tomcat" inheritAll="false"/>
		<!-- since tomcat replaces some files, we may need to patch again -->
		<echo>Applying patches for the second time</echo>
		<ant target="copy-init" />
	</target>

	<target name="dev-setup-jboss" depends="" if="choose.JBoss" >
		
		<ant dir="${pentaho.build.dir}" antfile="dev_build.xml" target="dev-setup" inheritAll="false"/>
	</target>


	<target name="dev-setup" description="Compiles and builds the entire project" depends="copy-init,dev-setup-jboss,dev-setup-tomcat" >
		
	</target>

	<target name="copy-finish" depends="dev-setup" description="Copy target files">
		<!-- apply patches - Not sure if I need this -->
		<!--copy todir="." failonerror="true" verbose="true" >
			<fileset dir="project-${project}/patches/"/>
		</copy-->
	</target>


	<target name="all" description="Compiles and builds the entire project" depends="copy-finish" />


	<!-- Clean targets -->

	<target name="clean" description="Cleans the solution" depends="copy-init" >
		<delete dir="project-${project}/solution/system/content" failonerror="true"/>
		<mkdir dir="project-${project}/solution/system/content"/>
		<delete dir="project-${project}/solution/system/tmp" failonerror="true"/>
		<mkdir dir="project-${project}/solution/system/tmp"/>
		<delete dir="target-preconfiguredinstall/server/default/deploy/pentaho.war" failonerror="false" />
		<delete dir="target-preconfiguredinstall-tomcat/webapps/pentaho" failonerror="false" />
		<ant dir="${pentaho.build.dir}" antfile="dev_build.xml" target="clean" inheritAll="false"/>
	</target>

	<target name="dist-clean" description="Deletes the target preconfigured install dir" >
		<delete dir="${pentaho.build.dir}" failonerror="false"/>
		<delete dir="target-preconfiguredinstall" failonerror="false"/>
		<delete dir="target-preconfiguredinstall-tomcat" failonerror="false"/>
	</target>

</project>

and this is my build.properties file

#####################################
## SOLUTION
######################################
project = etsa

#accepted values: tomcat or jboss
server.name = tomcat

solution.absolute.path = /home/pedro/tex/pentaho/project-client/solution/
db_location = 127.0.0.1
db_user = pedro
db_password = bar
base_url = http://127.0.0.1:8080/pentaho/


#####################################
## PROJECT DIRECTORIES
######################################
pentaho.dir = pentaho/
pentaho.build.dir = target-build/
javac.path=/usr/lib/java/bin/javac

How to build this? Simple: 'ant dist-clean' to delete all compiled classes and delete target* directories, and 'ant all' to build everything.

You may need to change some entries in override.properties to support tomcat instead of jboss, mainly:

(...)
# PCI target folder, any name or location you wish - Project will be built here

#JBOSS:
#target.server.dir=${pentaho.proj.dir}/../target-preconfiguredinstall
#TOMCAT:
target.server.dir=${pentaho.proj.dir}/../target-preconfiguredinstall-tomcat
(...)
#PCI source PCI from subversion
#JBOSS:
#source.server.dir=${pentaho.proj.dir}/../pentaho-preconfiguredinstall
#TOMCAT:
source.server.dir=${pentaho.proj.dir}/../pentaho-tomcat
(...)

#JBOSS
#dev.deploy.dir=${target.server.dir}/server/default/deploy
#TOMCAT
dev.deploy.dir=${target.server.dir}/webapps/

If we had another project in a directory called 'project-anotherclient', all we had to do was to simply change build.properties file and change project entry from client to another client and with a 'ant clean all' we would have a clean solution with totally different configuration

There is an extra file there, called 'updateall-svn.sh'. Its just a simple bash script to update all sources from svn. It will only work on some systems:

for i in `ls -d p*`; do cd $i && svn update && cd .. ; done |less

At this point, all that was missing was the design studio and knowing how to debug the platform. I'm a netbeans user, so eclipse was totally new for me and I was a bit lost. Since I usually work with tomcat, debugging jboss was also a new area.

I tried, and tried, and tried once again to build designstudio from sources, but I could not do it. there are 3 projects on the svn root, but failed miserable in all of them. So I went to the precompiled plug in; There is a tutorial here that shows how to use jbossIDE to do the debug jboss, but I found that a bit complicated for 2 reasons: 1 - I could not get designstudio plugin to work in the same eclipse instance as jbosside plugin; 2 - I'm used to do debugging just by attaching a debugger to a socket, so using a >100 Mb plugin just for debugging seemed to me a great overhead. So I opted by not using jbosside plugin. This is what I did:

1 - Downloaded and installed (unzipped) the latest eclipse

2 - downloaded design studio pre-compiled plug ins and unzipped it to the eclipse directory (starting with eclipse -clean may be necessary)

3 - copied file pentaho-preconfiguredinstall/bin/run.conf to project-client/patches/target-preconfiguredinstall/bin/run.conf and uncomment the line that says:

JAVA_OPTS="$JAVA_OPTS -Xrunjdwp:transport=dt_socket,address=8787,server=y,suspend=n"

You can also change other attributes as needed, such as memory limits. For tomcat check catalina.sh file for instructions

4 - run eclipse, added a project named project-client, and a java project (with existing ant script) for pentaho/dev_script.xml.

And that was it. If I want to work on my xactions I just use "project client", if I need to debug jboss I just need to start it with 'debug.(bat|sh)' and attach the eclipse debugger to it (click debug, choose 'debug remote application' and point to the correct host, port 8787 as the above example). Breakpoints will work as magic

__________________

This is my 2 cents in appreciation for the great effort pentaho team puts in this project, and their courage to go for an open-source approach.

Thread sponsored by irc channel ##pentaho in freenode

Thanks

Pedro Alves