The actual test for the TCK are defined in the jsr303-tck jar file. You can checkout the source code for the TCK from svn as well or view the tests in fisheye. The tests are an excellent source for Bean Validation usage examples. Each test is linked to an assertions in the specification via an @SpecAssert annotation. All tested assertions are listed in tck-audit.xml which is part of the TCK.
Running the test harness
The easiest way to run the TCK is configured as a maven project. We recommend to setup a separate maven module to run the TCK. This module won't contain any source code itself. Instead it will run the tests defined in the Bean Validation TCK against the Bean Validation implementation implementation specified in the POM.
You'll need:
- a pom.xml
<?xml version="1.0"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd"> <parent> <artifactId>hibernate-validator-parent</artifactId> <groupId>org.hibernate</groupId> <version>4.0.0.CR2-SNAPSHOT</version> </parent> <modelVersion>4.0.0</modelVersion> <artifactId>hibernate-validator-tck-runner</artifactId> <name>Hibernate Validator TCK Runner</name> <description>Aggregates dependencies and run's the JSR-303 TCK</description> <dependencies> <dependency> <groupId>javax.validation</groupId> <artifactId>validation-api</artifactId> </dependency> <dependency> <groupId>org.hibernate</groupId> <artifactId>hibernate-validator</artifactId> <version>${version}</version> </dependency> <dependency> <groupId>org.slf4j</groupId> <artifactId>slf4j-log4j12</artifactId> <scope>runtime</scope> </dependency> <dependency> <groupId>org.testng</groupId> <artifactId>testng</artifactId> <classifier>jdk15</classifier> </dependency> <dependency> <groupId>org.hibernate.jsr303.tck</groupId> <artifactId>jsr303-tck</artifactId> <version>1.0.0.Beta1-SNAPSHOT</version> </dependency> <dependency> <groupId>org.jboss.test-harness</groupId> <artifactId>jboss-test-harness-jboss-as-51</artifactId> <version>1.0.0-SNAPSHOT</version> </dependency> </dependencies> <properties> <jboss.home>/opt/java/jboss-5.1.0.GA</jboss.home> <validation.provider>org.hibernate.validator.HibernateValidator</validation.provider> <remote.debug/> </properties> <build> <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-dependency-plugin</artifactId> <executions> <execution> <id>copy</id> <phase>generate-test-sources</phase> <goals> <goal>copy</goal> </goals> <configuration> <stripVersion>true</stripVersion> <artifactItems> <artifactItem> <groupId>org.hibernate.jsr303.tck</groupId> <artifactId>jsr303-tck</artifactId> <type>xml</type> <classifier>suite</classifier> <overWrite>false</overWrite> </artifactItem> <artifactItem> <groupId>javax.validation</groupId> <artifactId>validation-api</artifactId> <overWrite>true</overWrite> <outputDirectory>${project.build.directory}/dependency/lib</outputDirectory> </artifactItem> <artifactItem> <groupId>org.hibernate</groupId> <artifactId>hibernate-validator</artifactId> <overWrite>true</overWrite> <outputDirectory>${project.build.directory}/dependency/lib</outputDirectory> </artifactItem> <artifactItem> <groupId>org.slf4j</groupId> <artifactId>slf4j-log4j12</artifactId> <overWrite>true</overWrite> <outputDirectory>${project.build.directory}/dependency/lib</outputDirectory> </artifactItem> </artifactItems> </configuration> </execution> </executions> </plugin> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-surefire-plugin</artifactId> <configuration> <suiteXmlFiles> <suiteXmlFile>${project.build.directory}/dependency/jsr303-tck-suite.xml</suiteXmlFile> </suiteXmlFiles> <argLine>-Xmx128m</argLine> <forkMode>once</forkMode> <systemProperties> <property> <name>validation.provider</name> <value>${validation.provider}</value> </property> </systemProperties> </configuration> </plugin> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-surefire-report-plugin</artifactId> <executions> <execution> <id>generate-test-report</id> <phase>test</phase> <goals> <goal>report-only</goal> </goals> </execution> </executions> <configuration> <outputDirectory>${project.build.directory}/surefire-reports</outputDirectory> <outputName>test-report</outputName> </configuration> </plugin> </plugins> </build> <profiles> <profile> <id>incontainer-debug</id> <activation> <property> <name>debug</name> </property> </activation> <properties> <remote.debug>-Xnoagent -Djava.compiler=NONE -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=5005 </remote.debug> </properties> </profile> <profile> <id>incontainer</id> <activation> <property> <name>incontainer</name> </property> </activation> <build> <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-surefire-plugin</artifactId> <configuration> <suiteXmlFiles> <suiteXmlFile>${project.build.directory}/dependency/jsr303-tck-suite.xml</suiteXmlFile> </suiteXmlFiles> <systemProperties> <property> <name>validation.provider</name> <value>${validation.provider}</value> </property> <property> <name>org.jboss.testharness.standalone</name> <value>false</value> </property> <property> <name>jboss.home</name> <value>${jboss.home}</value> </property> <property> <name>org.jboss.testharness.container.javaOpts</name> <value>-Xms128m -Xmx384m -XX:MaxPermSize=128m -Dorg.jboss.resolver.warning=true -Dvalidation.provider=${validation.provider} ${remote.debug} </value> </property> <property> <name>org.jboss.testharness.container.forceRestart</name> <value>true</value> </property> <property> <name>org.jboss.testharness.runIntegrationTests</name> <value>true</value> </property> <property> <name>org.jboss.testharness.libraryDirectory</name> <value>target/dependency/lib</value> </property> <property> <name>org.jboss.testharness.outputDirectory</name> <value>target</value> </property> </systemProperties> </configuration> </plugin> </plugins> </build> </profile> <profile> <id>write-artifacts-to-disk</id> <activation> <property> <name>dumpArtifacts</name> </property> </activation> <build> <plugins> <plugin> <groupId>org.codehaus.mojo</groupId> <artifactId>exec-maven-plugin</artifactId> <executions> <execution> <id>generate-jsr-303-artifacts</id> <phase>test-compile</phase> <goals> <goal>java</goal> </goals> </execution> </executions> <configuration> <classpathScope>test</classpathScope> <mainClass>org.jboss.testharness.api.TCK</mainClass> <systemProperties> <systemProperty> <key>dumpArtifacts</key> <value>true</value> </systemProperty> <systemProperty> <key>org.jboss.testharness.outputDirectory</key> <value>target/jsr303-artifacts</value> </systemProperty> <systemProperty> <key>org.jboss.testharness.libraryDirectory</key> <value>target/jsr303-artifacts/dependencies</value> </systemProperty> </systemProperties> </configuration> </plugin> </plugins> </build> </profile> </profiles> </project> - and src/main/resources/META-INf/jboss-test-harness.properties
org.jboss.testharness.spi.StandaloneContainers=org.hibernate.jsr303.tck.util.StandaloneContainersImpl
org.jboss.testharness.api.TestLauncher=org.jboss.testharness.impl.runner.servlet.ServletTestLauncher
org.jboss.testharness.testPackage=org.hibernate.jsr303.tck.tests
You can see the full setup for the Hibernate Validator TCK runner here - http://fisheye.jboss.org/browse/Hibernate/validator/trunk/hibernate-validator-tck-runner
Once you have the right setup you can choose between running the tests standalone or in a container. The example above assumes that you run the TCK against JBoss 5.
Standalone
This is the easiest way to run the test harness.
> mvn clean test
In this case the tests are run with the default surefire classpath. All tests of the TCK which are not marked as @IntegrationTest are run.
In container
In order to run against JBoss 5 you need:
- A JBoss 5 installation
- Define jboss.home in your POM
If your setup is complete it is enough to run:
> mvn clean test -Dincontainer
Dumping artifacts
In case you are running against a container the tests are deployed as war/ear files. In order to debug the tests it can be useful to actually inspect the deployed artifacts. To do this just specify the dumpArtifacts property at the command line. This will create all the artifacts which gets deployed in target/jsr303-artifacts.
> mvn clean test -DdumpArtifacts
TCK Audit
Part of the source code of the jsr303-tck is an audit tool which allows to determine the coverage of tests in the TCK in relation to the actual specification. The assertions of the TCK are captured in a file called tck-audit.xml which can be found under /src/main/resources of the source code. More information about the format and the usage of this xml file can be found JSR 299 TCK.
In order to run the audit tool you have to checkout the jsr303-tck project and run:
> mvn clean test -P tck-audit
After the build completes there should be a coverage.html in your target directory. Note, this is no measurements whether an actual implementation passes a TCK. It just indicates how well the TCK tests cover the assertions in the specification. More information about the audit tool can be found here.
What do all the annotations mean?
// $Id: BootstrapCustomProviderDefinedInServiceFileTest.java 17345 2009-08-18 09:35:56Z hardy.ferentschik $
/*
* JBoss, Home of Professional Open Source
* Copyright 2008, Red Hat Middleware LLC, and individual contributors
* by the @authors tag. See the copyright.txt in the distribution for a
* full listing of individual contributors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
* http://www.apache.org/licenses/LICENSE-2.0
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.hibernate.jsr303.tck.tests.bootstrap.customprovider;
import javax.validation.Configuration;
import javax.validation.Validation;
import javax.validation.ValidatorFactory;
import org.jboss.test.audit.annotations.SpecAssertion;
import org.jboss.test.audit.annotations.SpecAssertions;
import org.jboss.testharness.AbstractTest;
import org.jboss.testharness.impl.packaging.Artifact;
import org.jboss.testharness.impl.packaging.ArtifactType;
import org.jboss.testharness.impl.packaging.Classes;
import org.jboss.testharness.impl.packaging.IntegrationTest;
import org.jboss.testharness.impl.packaging.Resource;
import static org.testng.Assert.assertNotNull;
import static org.testng.Assert.assertTrue;
import org.testng.annotations.Test;
import org.hibernate.jsr303.tck.common.TCKValidationProvider;
import org.hibernate.jsr303.tck.common.TCKValidatorConfiguration;
import org.hibernate.jsr303.tck.util.TestUtil;
/**
* @author Hardy Ferentschik
*/
@Artifact(artifactType = ArtifactType.JSR303)
@Classes({
TestUtil.class,
TestUtil.PathImpl.class,
TestUtil.NodeImpl.class,
TCKValidationProvider.class,
TCKValidatorConfiguration.class,
TCKValidationProvider.DummyValidatorFactory.class
})
@Resource(source = "javax.validation.spi.ValidationProvider",
destination = "WEB-INF/classes/META-INF/services/javax.validation.spi.ValidationProvider")
@IntegrationTest
public class BootstrapCustomProviderDefinedInServiceFileTest extends AbstractTest {
@Test
@SpecAssertions({
@SpecAssertion(section = "4.4", id = "a"),
@SpecAssertion(section = "4.4.4.2", id = "a")
})
public void testGetFactoryByProviderSpecifiedProgrammatically() {
TCKValidatorConfiguration configuration = Validation.byProvider( TCKValidationProvider.class ).configure();
ValidatorFactory factory = configuration.buildValidatorFactory();
assertNotNull( factory );
assertTrue( factory instanceof TCKValidationProvider.DummyValidatorFactory );
}
}
@Artifact
Annotation is part of the jboss-test-harness framework and decides which type of artifact should be build when deploying the test into a container. This annotation is irrelevant for when tests are run in standalone mode.
@Classes
Per default only classes from the same package as the test class are included in the artifacts. In order to include other classes the @Classes annotation is used. This annotation is irrelevant for when tests are run in standalone mode.
@Resource(s)
Used to include resource files into artifacts. In case of ArtifactType.JSR303 war files are getting build and the resource files should go somewhere into WEB-INF/classes. This annotation is irrelevant for when tests are run in standalone mode.
@IntegrationTest
Per default tests will be run in standalone as well as in incontainer mode. Marking a test as @IntegrationTest will skip this tests in standalone mode.
@Test
TestNG annotation to mark a method as test.
@SpecAssertion(s)
Used to link a test to the BV specification via the tck-audit.xml file.
Further reading
The harness to run the TCK tests is based on and reuses the infrastructure provided by the JSR 299 TCK.


