The Pentaho Metastore

Owned by Former user (Deleted)
Last updated: Aug 26, 2014Version comment
4 min read

Metastore

The Metastore project aims to provide a developer with a way of storing metadata, configuration information, settings and so on in a standardized way.   To make this happen we choose a simple meta-model with the organization described below

Meta-model

The Metastore meta-model is organized as following:

  1. Namespaces: the top level entry of the Metastore is always a namespace.  The namespace Pentaho software works in will always simply be "Pentaho".
  2. Element types: each namespace has a set of element types which are identified by a unique ID and have a name and a description.
  3. Elements: each element type can have a number of elements which are identified by a unique ID and have a name and a set of attributes
  4. Attributes: each attribute has an ID, a value and a set of child attributes.  The values of attributes are typically string values but integers, doubles and dates are supported as well.

So essentially, it's a key/value store with children allowing you to store just about any piece of metadata.

Standard definitions

Common MetaModel Element Type definitions can be found on this page:

Standard MetaStore Element types

Java project

The Java project with the API and reference implementations is to be found over here:

https://github.com/pentaho/metastore

There is a CI build process looking at the project over here:

http://ci.pentaho.com/view/Commons/job/metastore/

This build process is pushing maven artifacts into the pentaho repository, ivy dependency line:

<dependency org="pentaho" name="metastore" rev="TRUNK-SNAPSHOT" />

Java API

Even though the organisation of the meta-model is kept simple (on purpose) it makes sense to cover the main Java interfaces a bit.

IMetaStore

The IMetaStore interface is the top level interface describing all the possible operations which are available. 

Namespace operations

  • getNamespaces() : lists all the namespaces
  • createNamespace() : create a new namespace
  • deleteNamespace() : delete a namespace

Element type operations

  • getElementTypes() : list all the element types
  • getElementTypeIds() : list all the element type IDs
  • getElementType(): find an element with a specific ID
  • getElementTypeByName() : find an element type with a specific name
  • createElementType() : create a new element type in the store
  • updateElementType(): update an element type
  • deleteElementType(): delete an element type
  • newElementType() : generate a new empty element type

Element operations

  • getElements() : list all the elements
  • getElementIds() : list all the element IDs
  • getElement(): find an element with a specific ID
  • getElementByName() : find an element with a specific name
  • createElement() : create a new element in the store
  • updateElement(): update an element
  • deleteElement(): delete an element
  • newElement() : generate a new element
  • newAttribute() : generate a new empty attribute suitable for the store
  • newElementOwner() : generate a new element owner

Metastore convenience properties

Self explanatory, to distinguish one Metastore from another.

  • getName()
  • getDescription()

Metastore password encoder

You can ask the MetaStore to give you the methods for encoding, decoding and verifying passwords or you can set your own methods (right after instantiating) with the following methods:

  • getTwoWayPasswordEncoder()
  • setTwoWayPasswordEncoder()

MetaStoreFactory

The basics

For your convenience we provided you with a MetaStoreFactory to facilitate the specific use-case of serializing POJO.
Consider a class annotated with @MetaStoreElementType and where private members are annotated with @MetaStoreAttribute 
  

MyElement me = new MyElement( NAME, ATTR, ANOTHER, INT, LONG, BOOL, DATE ); 
MetaStoreFactory<MyElement> factory = new MetaStoreFactory<MyElement>( MyElement.class, metaStore, "custom" ); 

Now we can save the element in the metastore simply by calling: 

factory.saveElement( me ); 

Loading the element by name from the metastore simply becomes: 

MyElement verify = factory.loadElement( NAME );  

The unit test class which demos this behavior is: MetaStoreFactoryTest 

The list of available methods in the factory is:

  • public T loadElement( String name ) throws MetaStoreException;
  • public void saveElement( T t ) throws MetaStoreException;
  • public List<T> getElements() throws MetaStoreException;
  • public void deleteElement( String name ) throws MetaStoreException;
  • public List<String> getElementNames() throws MetaStoreException;
  • public IMetaStoreElementType getElementType() throws MetaStoreException;
  • public T loadElement( String name ) throws MetaStoreException;
  •  

public void saveElement( T t ) throws MetaStoreException;
public List<T> getElements() throws MetaStoreException;
public void deleteElement( String name ) throws MetaStoreException;
public List<String> getElementNames() throws MetaStoreException;
public IMetaStoreElementType getElementType() throws MetaStoreException;

One factory calling another

If you have one element referencing another by name we can pass a factory to another factory to handle automatic persistence.

See for example the following :

MetaStoreFactory<X> factoryX = new MetaStoreFactory<X>( X.class, metaStore, "pentaho" );
MetaStoreFactory<Y> factoryY = new MetaStoreFactory<Y>( Y.class, metaStore, "pentaho" );
factoryX.addNameFactory( X.FACTORY_Y, factoryY );

In this situation, if you call factoryX.saveElement() you will also persist any referenced elements of Y which are annotated as followed in class X:

@MetaStoreAttribute( factoryNameKey = FACTORY_Y, factoryNameReference = true )
private List<Y> ys;

//and/or  

@MetaStoreAttribute( factoryNameKey = FACTORY_Y, factoryNameReference = true )
private Y y;

The factory is then also used when you are loading elements.

IMPORTANT NOTE: Do not rely on the referenced classes to be unique instances.  If you want that goal, pass unique lists of elements to the factory using addNameList()

Optionally shared

Since you don't always want elements to be shared and visible for the whole metastore to see, you can make an element optionally shared and as such, optionally 'embedded' in the parent element attributes.
To do this, you simply tell the factory what the name is of the boolean field which expresses the fact that the element is shared.
For example, if you have a "shared" flag in class B:

@MetaStoreAttribute
private boolean shared;

In this case you can reference this in class A as follows:

@MetaStoreAttribute( factoryNameKey = FACTORY_B, factoryNameReference = true, factorySharedIndicatorName = "shared" )  
private B b;

The factory will then automatically determine to either create or update an element for B or embed it in the attributes of the A element.