New year changes

Hey, it’s 2012. With the new year comes changes and new things. For starters, I’m moving to Australia really soon (lots of packaging and preparations to do).
Because of this, I’ll be working less on Turmeric SOA, but I’ll keep trying things in my github repo. I’ll try to keep in touch through the #turmeric-dev channel and the mailing lists.

Back to packaging stuff.

Turmeric SOA Metrics using Jetty-turmeric – Part 1

Last months We’ve been working on implementing monitoring capabilities for Turmeric SOA, on top of Apache Cassandra. Basically we created an error handler, a metrics storage provider and a query provider to read/write metrics in Cassandra.

Along this, we’ve been also working on making initial setup easier for adopters, so, we have come up with, which is basically a jetty web server packed with all the required dependencies to run Turmeric services. This server comes with the ability to run our metric services using jpa or cassandra.

Let’s see how to get the jpa-based services going.

1. Get the jetty-turmeric distribution, here.
2. Get the TMC, here.
3. Get the Query Metrics Service, here.
4. Once you have downloaded the jetty-turmeric zip file, extract its contents on a folder called jetty-turmeric.
5. Open the TMC distribution file. Inside there is a file called monitoringconsole-web.xml and a war file, called mgmt-console-1.0.0.4-SNAPSHOT.war. Extract them both to a folder.
6. Replace the contents of the mgmt-console-1.0.0.4-SNAPSHOT.war/WEB-INF/web.xml file with the monitoringconsole-web.xml file.
7. Rename the mgmt-console-1.0.0.4-SNAPSHOT.war to console.war
8. Edit the file jetty-turmeric/resources/META-INF/soa/services/config/GlobalServiceConfig.xml. Uncomment the following block:

<monitor-config>
        <storage-provider name="DAOMetricsStorageProvider">
            <class-name>org.ebayopensource.turmeric.monitoring.storage.DAOMetricsStorageProvider</class-name>
            <storage-options>
                <option name="persistenceUnitName">metrics</option>
                <option name="metricsDAOClassName">org.ebayopensource.turmeric.monitoring.MetricsDAOImpl</option>
                <option name="storeServiceMetrics">false</option>
            </storage-options>
        </storage-provider>
        <snapshot-interval>60</snapshot-interval>
    </monitor-config>

• Place the console.war in jetty-turmeric/webapps/ folder.
• Go to the command line, and inside of the jetty-turmeric/ folder, run: java -jar start.jar etc/jetty-ssl.xml

Ok, now go to this url: http://localhost:8080/console. You should see the TMC without any data. Wait 1 minute, then reload the page, now you should see something like this:

In the part 2, I’ll explain how to run the cassandra related services.

Regards.

gwt-test-utils to the rescue

One of the problems working with gwt is writing the unit tests. In https://www.ebayopensource.org/index.php/Turmeric/HomePage we have 2 GWT apps: The Policy Admin UI and the Monitoring Console. After a couple of days reading the GWTTestCase test usage (and a bit of pulling my hair in despair), somebody pointed me to this: gwt-test-utils.

Gwt-test-utils is basically a library that let’s you test gwt code without the annoying of using GWTTestCase. So it means No 2 minutes waiting for it to startup and compile all the java code. It can handle eventing in your tests too.

How do you use it?. Ok, in Monitoring Console we use it like this:

1.- Add the dependency in the pom.xml

<dependency>
        <groupId>com.octo.gwt.test</groupId>
	<artifactId>gwt-test-utils</artifactId>
	<version>0.28</version>
	<scope>test</scope>
</dependency>

2.- Create the src/test/resources/META-INF/gwt-test-utils.properties file. In this file, you need to tell gwt-test-utils where is the gwt module file of the app you want to test. For Monitoring Console, this is the content of the file:
 

org/ebayopensource/turmeric/monitoring/Console.gwt.xml = module-file

Take care of not leaving whitespaces after the module-file element.

3.- Now, create a test class extending GwtTest:

public class ConsoleGwtTestBase extends GwtTest {

    @Override
    public String getModuleName() {
        return "org.ebayopensource.turmeric.monitoring.Console";
    }

}

4.- Now, just start creating your objects and test them as a regular Junit Test. Enjoy!

GWT debugging in Linux using IE8 in a VirtualBox Windows VM

Hi,

Recently I’ve had the need to test the Turmeric Monitoring Console in IE8, so I needed to use a windows setup. The turmeric Monitoring Console is a GWT app, so, ideally I wanted to run the GWT app using the eclipse debug mode in my linux setup, and use the IE8 browser from my windows setup to access it. Thing is by default, the gwt eclipse plugin doesn’t allow access from a machine different to localhost (the virtualbox setup has it’s own ip address). So, in order to do this, I did the following steps:

1.- In eclipse, select the turmeric console project, right click and go to Properties -> Run/Debug Settings. You should get something like this:

2.- Select Console.html and click on Edit. In the Edit Configuration View, select the arguments tab.

The key here is the  -bindAddress 0.0.0.0 parameter. It tells the gwt environment to link to any address the machine has (this is because the virtualbox setup will see the host machine through an IP such as: 192.168.1.105).

3.- Now, in the VirtualBox VM, start up your IE8 browser, install the gwt plugin for it, and type this url: http://<ip of your host pc>:8888/Console.html?gwt.codesvr=<ip of your host pc>:9997 . In my case, this would be: http://192.168.1.105:8888/Console.html?gwt.codesvr=192.168.1.105:9997

4.- Enjoy your gwt debugging session now!

Busy

I’ve been busy (don’t we all?) lately. Particularly with this, this, also this and a little bit of that. Hope to be able to blog some interesting stuff soon.

Creating a AWS Consumer with Turmeric

I’m going to show you how to create a consumer for an Amazon Web Service.

First, we need a WSDL, in this example I’ll use this one: http://webservices.amazon.com/AWSECommerceService/AWSECommerceService.wsdl

Then, we need to install Eclipse with the Turmeric Eclipse Plugin

Now the good stuff. First, start by opening Eclipse. Then, go to New -> Others -> Turmeric SOA -> New Consumer from WSDL

 

New consumer from wsdl

 

Now, in the next screen, set the WSDL and other info as the screenshot below:

New consumer from wsdl wizard

Now, press Finish and wait for the plugin to work its magic. Once the plugin finishes to create the required artifacts, your workspace should look something like this:

Turmeric Workspace

Notice the 2 projects? those are the Service (AwsdnAWSECommerceServiceV1) and Consumer (AWSECommerceServiceV1Consumer) projects.

One thing we need to change is the maven dependencies in the projects. This is due to a recently fixed bug in the soa-client code. So, we need to make suere we are using the following maven dependencies in the projects:

 
... 
<dependency> 
      <groupId>org.ebayopensource.turmeric.runtime</groupId> 
      <artifactId>soa-client</artifactId> 
      <version>1.0.0-Beta-RC1</version> 
    </dependency> 
... 

and the maven-turmeric-plugin dependency:

 
... 
<groupId>org.ebayopensource.turmeric.maven</groupId> 
        <artifactId>turmeric-maven-plugin</artifactId> 
        <version>1.0.0-Beta-RC1</version> 
... 

To call the Amazon Web Service in our consumer, We need to have an account in Amazon Webservices to have access to the service. For that, I simply went here http://aws.amazon.com/ and created an AWS Account. Once created, you should have an Access Key ID and a Secret Access Key for your account. Then, We need to send 3 parameters in the SOAP header of the request to the service:

1. AWSAccessKeyId: Your access key
2. Timestamp: the current time at the moment of making the call. It needs to be in the format: YYYY-MM-DDThh:mm:ssZ
3. Signature: This is the tricky one. This parameter is the HMAC-SHA256 signature calculated from the concatenation of the Action and Timestamp parameters. For instance, if I want to call the ItemSearch operation in the webservice, I would have to do this: Signature = HMAC-SHA256(ItemSearch2010-10-16T00:00:00Z)

A more detailed explanation of the soap headers is here: http://docs.amazonwebservices.com/AWSECommerceService/2010-11-01/DG/index.html?AuthJavaSampleSig2.html

An example request to this service would look like this:

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:ns="http://webservices.amazon.com/AWSECommerceService/2010-12-01">
<soapenv:Header    xmlns:aws="http://security.amazonaws.com/doc/2007-01-01/">
   <aws:AWSAccessKeyId>your access key here</aws:AWSAccessKeyId>
   <aws:Timestamp>2011-03-15T15:38:23Z</aws:Timestamp>
   <aws:Signature>some generated signature</aws:Signature>
</soapenv:Header>
  <soapenv:Body>
    <ns:ItemSearch>
      <ns:Request>
        <ns:SearchIndex>Books</ns:SearchIndex>
        <ns:Keywords>Harry%20Potter</ns:Keywords>
      </ns:Request>
    </ns:ItemSearch>
  </soapenv:Body>
</soapenv:Envelope>

Ok, now, the consumer code. Inside the consumer project, create a simple java class called AWSConsumer. Then, make it extend this class: SharedAwsdnAWSECommerceServiceV1Consumer. This class is one of the elements the Turmeric Plugin created for us in the service project. If you take a look at the service project, it should look like this:

 
public class AWSConsumer extends SharedAwsdnAWSECommerceServiceV1Consumer{
	public AWSConsumer(String clientName) throws ServiceException { 
		super(clientName); 
	}
       public static void main(String[] args) throws ServiceException{ 
		//nothing yet 
	}
} 

Service Project layout

Now, to call the service, we need to create the Signature parameter and placing it in the soap header. For this, we need to create a helper class. I created this helper class based on this example: http://docs.amazonwebservices.com/AWSECommerceService/2010-11-01/DG/index.html?TheStructureofaSOAPRequest.html

This is the code for the helper class:

 

package com.amazon.aws.util;

import java.io.UnsupportedEncodingException;

import java.net.URLDecoder;
import java.net.URLEncoder;

import java.security.InvalidKeyException;
import java.security.NoSuchAlgorithmException;

import java.text.DateFormat;
import java.text.SimpleDateFormat;

import java.util.Calendar;
import java.util.HashMap;
import java.util.Iterator;
import java.util.Map;
import java.util.SortedMap;
import java.util.TimeZone;
import java.util.TreeMap;

import javax.crypto.Mac;
import javax.crypto.spec.SecretKeySpec;

import org.apache.commons.codec.binary.Base64;

public class SignedRequestHelper {
	private static final String UTF8_CHARSET = "UTF-8";
	private static final String HMAC_SHA256_ALGORITHM = "HmacSHA256";
	private static final String REQUEST_URI = "/onca/xml";
	private static final String REQUEST_METHOD = "GET";

	private String endpoint = "ecs.amazonaws.com"; // must be lowercase
	private String awsAccessKeyId = "Your access key here";
	private String awsSecretKey = "Your secret key here";

	private SecretKeySpec secretKeySpec = null;
	private Mac mac = null;
	private String currentTimestamp = null;
	private String operation = null;

	public SignedRequestHelper(String operationToSign)
			throws UnsupportedEncodingException, NoSuchAlgorithmException,
			InvalidKeyException {
		this.operation = operationToSign;
		byte[] secretyKeyBytes = awsSecretKey.getBytes(UTF8_CHARSET);
		secretKeySpec = new SecretKeySpec(secretyKeyBytes,
				HMAC_SHA256_ALGORITHM);
		mac = Mac.getInstance(HMAC_SHA256_ALGORITHM);
		mac.init(secretKeySpec);
	}

	private String hmac(String stringToSign) {
		String signature = null;
		byte[] data;
		byte[] rawHmac;
		try {
			data = stringToSign.getBytes(UTF8_CHARSET);
			rawHmac = mac.doFinal(data);
			Base64 encoder = new Base64();
			signature = new String(encoder.encode(rawHmac));
		} catch (UnsupportedEncodingException e) {
			throw new RuntimeException(UTF8_CHARSET + " is unsupported!", e);
		}
		return signature;
	}

	public String getTimestamp() {
		if (currentTimestamp == null) {
			Calendar cal = Calendar.getInstance();
			DateFormat dfm = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss'Z'");
			dfm.setTimeZone(TimeZone.getTimeZone("GMT"));
			currentTimestamp = dfm.format(cal.getTime());
		}
		return currentTimestamp;
	}

	public String getSignature() {
		return this.hmac(this.operation + this.getTimestamp());
	}

	public static void main(String[] args) throws InvalidKeyException,
			UnsupportedEncodingException, NoSuchAlgorithmException {
		SignedRequestHelper helper = new SignedRequestHelper("ItemSearch");
		SortedMap<String, String> sortedParamMap = new TreeMap<String, String>();
		sortedParamMap.put("Action", "ItemSearch");
		String timestamp = helper.getTimestamp();
		System.out.println("timestamp = " + timestamp);
		sortedParamMap.put("TimeStamp", timestamp);
		System.out.println(helper.hmac("ItemSearch" + timestamp));
	}

}

Now, let’s add the headers in the consumer code and do the call in the consumer code:

 

package com.amazon.aws.consumer;

import java.io.UnsupportedEncodingException;
import java.security.InvalidKeyException;
import java.security.NoSuchAlgorithmException;

import javax.xml.namespace.QName;

import org.ebayopensource.turmeric.runtime.binding.objectnode.impl.JavaObjectNodeImpl;
import org.ebayopensource.turmeric.runtime.common.exceptions.ServiceException;

import com.amazon.aws.util.SignedRequestHelper;
import com.amazon.webservices.awsecommerceservice._2010_12_01.Item;
import com.amazon.webservices.awsecommerceservice._2010_12_01.ItemSearch;
import com.amazon.webservices.awsecommerceservice._2010_12_01.ItemSearchRequest;
import com.amazon.webservices.awsecommerceservice._2010_12_01.ItemSearchResponse;
import com.amazon.webservices.awsecommerceservice._2010_12_01.Items;
import com.amazon.webservices.awsecommerceservice._2010_12_01.awsecommerceservice.gen.SharedAwsdnAWSECommerceServiceV1Consumer;

public class AWSConsumer extends SharedAwsdnAWSECommerceServiceV1Consumer{

	public AWSConsumer(String clientName) throws ServiceException {
		super(clientName);
	}
	
	public static void main(String[] args) throws ServiceException, InvalidKeyException, UnsupportedEncodingException, NoSuchAlgorithmException{
		AWSConsumer consumer = new AWSConsumer("AWSECommerceServiceV1Consumer");
		ItemSearch itemSearch = new ItemSearch();
		ItemSearchRequest itemSearchRequest = new ItemSearchRequest();
		itemSearchRequest.setSearchIndex("Books");
		itemSearchRequest.setKeywords("Harry Potter");
		itemSearch.getRequest().add(itemSearchRequest);
		SignedRequestHelper helper = new SignedRequestHelper("ItemSearch");
		
		String signature = helper.getSignature();
		System.out.println("signature="+signature);
		System.out.println("timestamp="+helper.getTimestamp());
		JavaObjectNodeImpl soapHeaderSignatureParam = new JavaObjectNodeImpl(new QName("http://security.amazonaws.com/doc/2007-01-01/", "Signature"), signature);
		consumer.getService().getRequestContext().addMessageHeader(soapHeaderSignatureParam);
		
		JavaObjectNodeImpl soapHeaderAWSAccessKeyIdParam = new JavaObjectNodeImpl(new QName("http://security.amazonaws.com/doc/2007-01-01/","AWSAccessKeyId"),"AKIAI3VIWRGZUL2VCDOA");
		consumer.getService().getRequestContext().addMessageHeader(soapHeaderAWSAccessKeyIdParam);
		
		JavaObjectNodeImpl soapHeaderTimestampParam = new JavaObjectNodeImpl(new QName("http://security.amazonaws.com/doc/2007-01-01/", "Timestamp"),helper.getTimestamp());
		consumer.getService().getRequestContext().addMessageHeader(soapHeaderTimestampParam);
		consumer.getService().getInvokerOptions().setMessageProtocolName("SOAP11");
		consumer.getService().getRequestContext().setTransportHeader("SOAPAction", "http://soap.amazon.com/ItemSearch");
		ItemSearchResponse response = consumer.itemSearch(itemSearch);
		for (Items item : response.getItems()) {
			for (Item books : item.getItem()) {
				System.out.println("book title = "+books.getItemAttributes().getTitle());
			}
		}
	}

}

One more thing. The platform uses a file called ClientConfig.xml. This file is used in the consumer code to configure things like protocol to use, message protocols and the url for the service. This file is found here: /AWSECommerceServiceV1Consumer/meta-src/META-INF/soa/client/config/AWSECommerceServiceV1Consumer/production/AwsdnAWSECommerceServiceV1/ClientConfig.xml

To make the call to the Amazon Web Service, you need to configure your ClientConfig.xml file like this:

<?xml version="1.0" encoding="UTF-8"?>
<client-config-list xmlns="http://www.ebayopensource.org/turmeric/common/config">
  <client-config>
    <service-interface-class-name>com.amazon.webservices.awsecommerceservice._2010_12_01.awsecommerceservice.AwsdnAWSECommerceServiceV1</service-interface-class-name>
    <service-location>https://ecs.amazonaws.com/onca/soap?Service=AWSECommerceService</service-location>
    <client-instance-config>
      <invocation-options>
        <preferred-transport name="HTTP11" />
        <request-data-binding>XML</request-data-binding>
        <response-data-binding>XML</response-data-binding>
        <consumer-id>MyConsumerID</consumer-id>
        <message-protocol>SOAP11</message-protocol>
      </invocation-options>
      <protocol-processor version="1.1" name="SOAP11">
        <indicator>
          <transport-header name="X-EBAY-SOA-MESSAGE-PROTOCOL">SOAP11</transport-header>
        </indicator>
        <class-name>org.ebayopensource.turmeric.runtime.sif.impl.protocolprocessor.soap.ClientSOAPProtocolProcessor</class-name>
      </protocol-processor>
    </client-instance-config>
  </client-config>
</client-config-list>

Now, It’s just a matter of building the projects. For this I use: Run As -> Maven Install option inside Eclipse. Then, just run the consumer with the Run As -> Java program. The output should be like this:

signature=jt4Sdl4ur4EMyyUtdLhla2/4htDNGuJCuI8BBETvkho=
timestamp=2011-03-19T15:37:53Z
Found config file at jar:file:/home/manuelchinea/.m2/repository/com/ebay/kernel/uKernelCore/1.73/uKernelCore-1.73.jar!/config/ukernelcore/logging.propertiesMar 19, 2011 11:07:53 AM org.ebayopensource.turmeric.runtime.common.impl.utils.ParseUtils getFileStream
WARNING: Unable to find resource: META-INF/soa/common/config/GlobalRegistry.xml

Found config file at jar:file:/home/manuelchinea/.m2/repository/com/ebay/kernel/uKernelCore/1.73/uKernelCore-1.73.jar!/config/ukernelcore/ConfigCommandListener.properties
Found config file at jar:file:/home/manuelchinea/.m2/repository/com/ebay/kernel/uKernelCore/1.73/uKernelCore-1.73.jar!/config/ukernelcore/ConfigCommandListener.properties
Found config file at jar:file:/home/manuelchinea/.m2/repository/com/ebay/kernel/uKernelCore/1.73/uKernelCore-1.73.jar!/config/ukernelcore/ConfigCommandListener.properties
Found config file at jar:file:/home/manuelchinea/.m2/repository/com/ebay/kernel/uKernelCore/1.73/uKernelCore-1.73.jar!/config/ukernelcore/ConfigCommandListener.properties
book title = Harry Potter: A Pop-Up Book: Based on the Film Phenomenon
book title = Harry Potter Paperback Box Set (Books 1-7)
book title = Harry Potter and the Sorcerer's Stone: 10th Anniversary Edition (Harry Potter)
book title = Harry Potter and the Deathly Hallows (Book 7)
book title = Harry Potter and the Half-Blood Prince (Book 6)
book title = Harry Potter And The Order Of The Phoenix
book title = Harry Potter and the Prisoner of Azkaban (Book 3)
book title = Harry Potter and the Goblet of Fire (Book 4)
book title = Harry Potter and the Chamber of Secrets (Book 2)
book title = The Unofficial Harry Potter Cookbook: From Cauldron Cakes to Knickerbocker Glory--More Than 150 Magical Recipes for Muggles and Wizards


What hudson can tell us about a project…

I’ve been recently working in the Turmeric project. In there, I’ve done a little bit of everything, but one thing I’ve had to do is to setup several Hudson jobs. The whole CI idea was new to me at the time, but I’ve grown to like it.

Now, I tend to test some Hudson/Jenkins configurations and plugins (I love the Hudson plugins, there are a LOT of them) on a spare machine I have. It’s pretty cool to be able to get information about the number of failing/successful unit tests, checkstyle trends, cobertura reports on a run by run basis.  You can also get the javadoc for a given project, download the generated artifacts and get some motivating words from Chuck Norris and so on.

I’ve also wondered what would look like the Hudson dashboard of other popular projects, what could they tell me about them and what could I learn from them. So, I started to google a little bit, and found these:

Eclipse Hudson server: First, the job list is really bigger than Turmeric’s list (just go in there and see the scrollbar size). Well, this one seems obvious as Turmeric is really close to Release 1.0.0 and Eclipse has been around for many years. Also, checking their node view, they have setups for Linux, Mac and Windows 7. Perhaps We could also add more slaves to test in these environments too. Also, one thing I noticed is that the Views they have created there somewhat reflect the several modules eclipse has, but I can’t really identify many of them, except the WTP, Jetty and Modeling views (Well, I don’t know all the eclipse plugins/modules, just the ones I use/install regularly).

hudson.eclipse.org

 

Jboss Hudson server: Wow!, this one is even bigger!. these guys have vertical and horizontal scrollbars in their main view!. Here I see a more logic view list (They have one view per each project they have, which are a lot of them). If you click on the hibernate tab, you’ll see they have jobs configured to run with different DBs. I think we could add some views in Turmeric, one for each important module (Runtime, Policy, Security, Repository).

http://hudson.jboss.org/hudson/

ASF Hudson server: We have a winner!!!. This setup is really huge. It’s so huge their views are letter ranges like: A-F, G-L. Once you get inside one of those tabs, you’ll probably see a project you’ve used before, such as: Ant, Maven, JMeter. Their nodes view have slaves for Linux, Mac, Windows, Solaris and FreeBSD, so It seems they have all the usual environments covered. Also, that cool console button in the main Dashboard is pretty handy!.

https://hudson.apache.org/hudson/

So, It seems We could do some improvements, like adding one view for each module in the project (Runtime, Policy, Security, Repository, etc) and keep this as a rule of thumb while the project grows in size. (For me was easy to identify the modules/projects in the Jboss/ASF hudson views than in the Eclipse one). Also get a slave or two to run our junit tests in Windows, Solaris and Mac environments. And finally, the console button plugin seems pretty handy to me, as I can go straight to the console of the last run.