Sample custom Ant task

Back in December, I wrote about the steps I took to code a custom Ant task and drop it in my build toolkit. Today I am sharing my plugin project as a sample for anybody interested in trying this out on their own. Remember that this is a SAMPLE and NOT a supported solution. I provide it with no promises or guarantees whatsoever. It is JUST to get you started with your own custom coding. Note also that the sample uses an API that wasn’t available until 3.0.1.3, so you’re going to need to be pointing at the latest build toolkit if you want to get this guy to compile. Click here to download the sample.

Once you’ve imported the project and played with the source, you can follow the directions from the original post to export the plugin and drop it in your toolkit. Recall that the point of this task was to compute some changed files and load them to MVS. This task determines which files have changed by comparing the snapshot that is created for the build workspace when the build runs against another specified snapshot. All of the modified files in change sets that are in the build workspace snapshot but not in the other specified snapshot will be loaded to MVS. The snapshot that we compare against can be specified in several ways. The Ant task takes a “delta policy” that specifies which type of comparison to make.

A delta policy of “BUILD” tells the Ant task that we are specifying another build definition, and we will compare the running build’s snapshot against the snapshot from the last successful team run of the specified build definition. For example, if you are running a build in March, you would specify the January build to compare against. In this way, any files that were changed and “built” (loaded) in January will not be seen as changed or loaded in March. If no build definition is specified with the “BUILD” delta policy, we will load all of the files that have changed since the last time this build ran successfully. That is, if the March build is running with no comparison build definition specified, we would load all of the files that have changed in March since the last time the March build ran successfully.

A delta policy of “SNAPSHOT” tells the Ant task that we are specifying a specific other snapshot to compare against. You might use this if, for example, you need to load all of the files that have changed since the snapshot in Production.

A delta policy of “FULL” will load all of the programs in the build workspace.

The full solution is as follows:

1. Create an Ant with Enterprise Extensions build definition.
2. On the Properties tab, create a new property teamz.build.ant.skipFileAgent with value of true.  This prevents the file agent from loading out all of the files at the beginning of the build.
3. On the Properties tab, create a new property deltaPolicy with value BUILD.
4. On the Properties tab, create a new property deltaBuildDefinitionUUID with value equal to the UUID of the build you are comparing against to determine deltas.
5. On the Jazz Source Control – z/OS tab, specify a Workspace, Load directory, and Data set prefix, business as usual.
6. On the Ant with Enterprise Extensions tab, select “Use an existing build file” and specify a file such as deltaload.xml and include its complete path in USS on the build machine (e.g., “/u/ryehle/deltaload.xml”).
7. Create your build script deltaload.xml on the host. It will look something like this:

<?xml version="1.0" encoding="UTF-8"?>
<project default="all" name="Delta load build">
   <description>Delta load build file</description>
   <taskdef name="loaddeltafiles" classname="net.jazz.ant.samples.tasks.LoadDeltaFiles"/>
   <target description="Load" name="load">
      <startBuildActivity autoComplete="true" buildResultUUID="${buildResultUUID}" label="Delta load" parentActivityId="${teamz.build.ant.buildActivityId}" passwordFile="${passwordFile}" repositoryAddress="${repositoryAddress}" userId="${userId}"/>
     <loaddeltafiles deleteMetadata="false" generateListing="true" deltaPolicy="${deltaPolicy}" buildDefinitionUUID="${deltaBuildDefinitionUUID}" passwordFile="${passwordFile}" prefix="${teamz.scm.dataset.prefix}" repositoryAddress="${repositoryAddress}" userId="${userId}" workspaceUUID="${teamz.scm.workspaceUUID}" snapshotUUID="${snapshotUUID}" buildResultUUID="${buildResultUUID}" />
   </target>
   <target depends="load" description="The Whole Thing" name="all"/>
</project>

I think that about covers it! Feel free to post back with questions in case I’ve missed a step here.

Finding your file in the Jazz SCM

They say there is no silver bullet in software engineering.  I say Rational Team Concert and the CLM solution come pretty darn close, enabling teams to deliver complex products on time, within budget, and with high quality. I love RTC and am a proud user, developer, and promoter of this tool. That being said, there is one thing about this product that just irks me…

Where the heck is my file? Why can I not go in the Search menu, pick my repository, type in part or all of my file name, and see what component, project, and folder my file is in? The answer to this question as I understand it comes in two parts. One, there are several open enhancements around this function that you can view on jazz.net. Two, IBM is sometimes guilty of delivering a Cadillac when all people need 90% of the time is a bicycle. What RTC does offer is a change set search, and the thought here is that maybe I’m looking for Foo.java, but Fred changed it last week to Bar.java, or even worse, he deleted it entirely. My search on Foo.java will get me nowhere. However, by searching for change sets, I will be able to see that my file has been renamed or deleted and link through to the work item that explains when and why this happened. Cool!

To try out the change set search capability in the RTC eclipse client, go to Search->Jazz Source Control->Change sets…  You will then be presented with this wizard:

Search for Change sets

As you can see, this wizard is powerful. You can narrow down by component, location (a specific stream, workspace, or baseline), change creator and date, who suspended the change (if applicable), specifically which file is modified, what the file name begins with, and even what type of change occurred (add, delete, modify, etc). The result is a listing of change sets that matched your criteria:

Change set search results

From here, I can double-click the change set to explore the changes, or I can right-click and choose Related Artifacts->Open to see the work item(s) associated with the change. The bummer though is that if I’m just trying to find my file, I need to start popping open and combing through these change sets til I land on it.

The good news is we have some options. For starters, this highlights the importance of using components to organize your source logically in your streams. If you know roughly what your file does, you should be able to make a good guess at what component it is in. I’ve heard some mainframe users say they can’t really organize their code logically, but they can at least apply a naming convention to their components and projects to organize and locate files in some fashion. You can always load your suspect components/projects to your eclipse workspace and then do a local search to find your file. Now, if you happen to have an enterprise developer license, your options get better…

As part of our enterprise dependency build capability, we scan files for structural information, such as type (main vs include) and logical dependencies. Even if you are not using dependency build, you can still choose to scan your streams and gather this source code data. Users then have the option to run a query on the source code data to locate their file:

Source code data query

You can choose which streams are scanned by right-clicking Source Code Data (in the Team Artifacts view under Source Code Data) and choosing Select Streams for Scanning. You can also specify the fixed delay for stream scanning in the ccm Advanced Properties under Source Code Data Collection from the web UI.

Another option you have with an enterprise developer license is to take advantage of context-aware search. This capability allows you to locate relevant source based on natural language. You can specify your own search terms or allow the tool to select your search terms automatically from the content of your work item. You can also specify which components/projects/folders/files are indexed and how often. Although an enterprise license is required, this capability is available for repositories running on any supported platform, including Windows, Linux, etc. Check out the demo video available here.

Here is an example of the context aware search capability in action in the context of a work item.

Invoking CAS from a work item

The Context Aware Search editor opens with some terms pre-selected from the content of the work item. You can choose to select more, remove some or all, and add your own. You can then see the results that are returned, along with a metric indicating the quality of the hit.

Context aware search results

Lastly, CAS allows you to specify synonyms for search purposes. Many synonyms are pre-configured out of the box, but you can edit these synonyms and add your own. For example, CAS knows that if you search on the term “calculate”, then “compute” should also be considered a hit. These configurations are specific to each project area.

And on a semi-related note… If you’ve read this far, chances are high that you are either (a) my mom or (b) a Rational Team Concert enthusiast who loves to be up on the latest and greatest product features, insider tips, and all that jazz. If you fall into the second category, I hope you’ve booked your flight to Orlando for June 3-7 to hang with the Jazz Development Team at Innovate 2012! I’ll be there signing autographs and spreading the good word on all things EE. If you make it to one of my tracks, be sure to come up and introduce yourself as one of my loyal readers to receive your complimentary high five. See you there!

Getting my MVS files into the RTC repository (and getting them back out again)

A little while back I was asked to deliver some education on zimport and zload. I decided not to let the fact that I knew neither zimport nor zload stop me from saying yes, and set out to educate myself via the usual means of reading, playing, and pestering the super smart people who have the misfortune of sitting around me. I learned some great secrets and troubleshooting tips that I’ll be sharing here today. The documentation for zimport and zload is thorough and complete as far as setup and options, so I will just give a brief introduction here and then focus on the good stuff.

Zimport, which you may also hear referred to as “mass import” or “the mass import tool,” is a one time tool for getting your existing MVS data sets into the RTC SCM. It is executed by a project administrator setting up the SCM, and requires a mapping file to specify the data sets to be imported. It creates zComponent projects, which are the type of project required to take advantage of all the various z capability in RTC. It creates Data Set Definitions based on the characteristics of the imported data sets, and optionally associates the imported files with Language Definitions.

Conversely, zload loads files from your repository workspace back out to your z/OS system. While zimport is JUST for importing from MVS, zload allows you to load to MVS, USS, or both.

So now that we know what we’re talking about, one of the main sources of difficulty and confusion is in the area of code pages and line delimiters. Files are typically stored in the Jazz repository in UTF-8; files on z/OS are typically stored in EBCDIC. When we zimport, we convert from EBCDIC to UTF-8. When we zload, we convert from UTF-8 to EBCDIC. Following is the logic taking place under the covers for zimport and zload:

• If zimport specifies -b for binary, we set the line delimiter to none and do no conversion. Otherwise, we convert and set the line delimiter to platform. We determine which encoding to use for the conversion based on the ZLANG environment variable. If ZLANG is set, use it.If ZLANG is unset, use the system’s default encoding.
• To determine which encoding to use when loading to MVS, we:
  • Check the mvsCodePage versionable property of the file.  If present and non-blank, it is used.
  • If no mvsCodePage, default to the value of the ZLANG environment variable, if ZLANG is set.
  • If the mvsCodePage property is not present, and ZLANG is unset, default to using the system’s default encoding.
• To determine which encoding to use when loading to USS, we:
  • Check the mvsCodePage versionable property of the file.  If present, and non-blank, it is used.
  • If the mvsCodePage property is present but blank:
    • If ZLANG is set, use the value of ZLANG.
    • If ZLANG is unset, default to the system’s default encoding.
  • If the mvsCodePage property is not present, load the file as it is in the repository — no conversion is done.

Now let’s talk about some known issues, limitations, and other gotchas.

Q: So, now that I’ve done my zimport and zload, how can I check my MVS changes back into the repository?

A: You can’t. There is currently no scm command line support for MVS. There is an open story 139129: Command Line support for SCM for this where you can add your +1 if it’s important to you.

Q: So, I have FRED.JOE.COBOL and FRANK.BOB.COBOL, and I want to import them into a single zFolder with a single COBOL Data Set Definition. How do I specify a multi-segment HLQ for the zimport?

A: You can’t. We only consider the first segment to be the HLQ. But, if you take a peak at the status of work item 155055: Support multiple-segment HLQs in zimport you might find some upcoming good news in this department. A couple details on this… If you try to import FRED.JOE.COBOL and FRED.FRANK.COBOL at the same time, it will fail with the error saying A zFolder can only contain zFiles imported from one PDS. But what you can do is import FRED.JOE.COBOL into a COBOL folder first. A JOE.COBOL data set definition will be created. You can then reuse your workspace (or any other workspace that flows to the stream where you’ve delivered FRED.JOE.COBOL) to zimport FRED.FRANK.COBOL into that same COBOL folder, which will continue to use the JOE.COBOL data set definition, rather than creating a FRANK.COBOL data set definition. You can always rename that data set definition when you’re done.

Q: What can I do to avoid so many Data Set Definitions being created by zimport?

A: If your data sets have the same name and all of the same attributes, zimport will use the same data set definition. If you end up with DSDs that you don’t want, you can always export all of your system definitions to capture the attributes of the DSDs you DO want, run the language definition generator to delete the data set definitions, and then run it again to recreate the good data set definitions and associate them with your zFolders.

Q: Daaaaaaang, this import is taking forevvvverrrrrrrr!! What should I do?

A: Go fix yourself a snack and keep waiting. If you kill it in the middle, you’re going to corrupt your metadata. To help speed things along, organize your data sets before the import such that you’re not processing through a ton of members to find the ones you want to import. The performance of the tool is a function of the number of files it looks through to decide what to import, more so than the number of members it actually ends up importing.

Q: My zimport is failing with this message: SEVERE: com.ibm.team.filesystem.client.FileSystemException: Error getting sharing descriptor. What do I do now?

A: Kick yourself for not listening to me above. Your metadata is corrupt. You probably killed a previous import while it was in the middle of working, or maybe it crashed in the middle (e.g. because you ran out of space). You could try changing the value of your SCM_WORK environment variable to another directory, or, if there’s nothing in your current SCM_WORK that you need to save, just delete that directory.

Q: What is up with this warning in my zimport?
CRHTC0632I Created the repository workspace zimport Feb 28, 2012 3:31:10 PM.
CRHTC0621E An error occurred while retrieving the members in PDS RYEHLE.SPF.ISPPROF. Its members will not be imported.
//’RYEHLE.SPF.ISPPROF’: fopen() failed – EDC5061I An error occurred when attempting to define a file to the system. errno=61 errno2=0xc00b0403 last_op=50

A: The tool is scanning every data set under the specified HLQ to figure out what to import. Check out work item 146123: Be smarter about scanning data sets for import to see how (and when!) we’ve gotten smarter.

Build Engines, Build Agents, and all that jazz…

Today I’d like to attempt to clarify a couple points of confusion surrounding the Rational Build Agent that is used for all of the Enterprise Extensions builds (Ant EE build, dependency build, promotion, packaging, and deployment) in RTC.

First off is the notion of the “Build Agent” versus the “Build Engine” and how many of each you might need. The “Build Agent” is the actual instance of the Rational Build Agent running on the build machine that is responsible for executing requested builds. The “Build Engine” is an artifact created in the RTC repository (found in the Team Artifacts view under Builds->Build Engines), owned by a particular team or project area, that specifies the host name and port where the build agent is running (among other things). The build agent itself is multithreaded, meaning it can service multiple build requests at once. So if you have several different build definitions supported by a single agent, requests from each of those definitions can be processed in parallel. However, the build engine is single threaded. This means that if each of your build definitions points to the same build engine, only one single request from all of the definitions can be processed at a time. The build engine blocks as busy until the requested build it is servicing completes. Therefore, if you do not want requests for build definition A held up by requests for build definition B, you could create a dedicated build engine for each build definition or specify a pool of build engines that support your build definitions. Those build engines can then point at different build agents on different build machines, different build agents on the same build machine, or one one single build agent instance that can handle requests from all definitions in parallel. More on the round-robin distribution of build requests and other build agent topics can be found here.

Now that we’ve sorted out one possible bottleneck, I must mention another. When a build is requested, a certain amount of processing occurs on the server before the request is passed over to the build agent running on the build machine. For example, when a dependency build runs, a pre-processor running on the server determines which programs will be rebuilt based on what files have changed. This list of programs is then transferred to the build machine to be built. If you are running with a version prior to 3.0.1.2, you will discover that while your dependency build is chugging through trying to figure out what needs to be rebuilt, NO other builds supported by ANY Rational Build Agent will be processed. This is because the service responsible for handling build requests was waiting for all of the pre-processing for each build request to complete before moving on to the next. Fortunately this has been fixed, and details can be seen in BuildAgentRequestService participants should be multi-threaded (171210).

Another source of confusion is the misleading “Build engine process polls for requests” checkbox in the Overview tab of the Build Engine editor. This tab is common to both the Rational Build Agent engines and the Jazz Build Engine (JBE) engines, but this particular setting only really makes sense for the latter. The JBE, unlike the Rational Build Agent, spins in a loop and continuously asks the server “do you have anything for me now? … how ’bout now? … how ’bout now??” This setting tells the server that the JBE is engaging in this behavior and will be servicing build requests. While the Rational Build Agent does NOT poll the server checking for build requests, you DO still need to have this box checked off in order for your engine to process build requests. Confusing.

Perhaps the most confusing aspect of the Rational Build Agent engine is figuring out what userid your build is running under. This is influenced by a number of factors, including how your build agent is started, whether you are using magic logon, and whether you are using build agent authentication overrides. For this, I point you to an excellent reference available on the jazz.net wiki.

I hope this helps to some extent, and I realize I’ve really only scratched the surface of topics in this area. If there are specific topics you’d like to see addressed, or things that were not fully clarified here, please comment back!

Trying the Enterprise Extensions on your own

Happy 2012! As this is my first blog post of the new year, it seems fitting that it be about beginnings and starting from scratch. This post is a response to two of the most common requests I receive: would you demo the z capabilities in RTC for my team, and could you show me how I can try the z capabilities myself.  I have prepared a demo, broken into two parts, showing how you can quickly install and configure the Jazz Team Server on your own machine and get started with the Enterprise Extensions. The demo is meant to reflect how I set up my own system for the purpose of giving demos and testing capabilities, NOT how you would actually set things up in a production environment.

Part 1 includes:

1. Installing the Jazz Team Server.
2. Running the Jazz Team Server setup wizard.
3. Creating the Money that Matters sample.
4. Connecting to your new project area from the Rational Team Concert eclipse client.
5. Creating a new stream and selecting it for source code data scanning.
6. Importing and sharing the sample mortgage application.
7. Running the system definition generator and delivering the updated sample application.

Part 2 includes:

1. Creating a Rational Build Agent engine.
2. Creating and configuring a dependency build definition.
3. Creating a “builder” id.
4. Requesting a dependency build.
5. Creating a second stream and dependency build definition for promotion.
6. Creating and configuring a promotion definition.

I hope you find these videos helpful, and as always you can find more information out on jazz.net.

Adding a custom Ant task to your build toolkit

When I realized the other day that I was going to need to create a new custom Ant task, I was excited by the potential blog post fodder. Surely it was going to be frustrating and complex, and surely my vast sea of followers would benefit from hearing about my trials, tribulations, and eventual triumph. While coding a new service plugin for the RTC server is easy enough, actually getting the server to pick up and run my new code has at times been a less than straightforward process. I imagined updating the build system toolkit would be a similar challenge.

As it turns out, creating the task and updating the toolkit was ridiculously easy, to the point that I was thinking I wouldn’t even bother writing about it. But, since I didn’t see an example out there that walks me through the steps end to end of creating my task and updating the toolkit, I might as well put it out there and potentially save somebody from some unnecessary fear and trepidation. So here we go.

In my case, I needed to write a custom task that generates a list of changed files to be loaded out to MVS. Here are the steps I followed:

0. Follow the steps in the Extensions Workshop on jazz.net to set up your development environment.

1. Create a new plug-in project to contain the custom Ant task. I cheated and copy-pasted one of the existing plug-in projects in the toolkit, then deleted all the contents and replaced it with my own.

2. Add the build-ant.jar file from com.ibm.team.build.toolkit plugin to the new plug-in project, and add it to the Classpath from the Runtime tab of the Plug-in Manifest Editor.

3. Create a new Java class that extends com.ibm.team.build.ant.task.AbstractTeamBuildTask. If your task does not need to interact with the build data model, you could just extend org.apache.tools.ant.Task instead.

4. Override the doExecute() method in your new task, and add setters for any attributes you need to pass in to your task. For example, if you are going to pass buildDefinitionUUID in from your Ant script, you’ll need a public void setBuildDefinitionUUID(String uuid) method in your task.

5. Once your task is coded, export your plugin project using File->Export->Deployable plug-ins and fragments. Specify a destination directory, choose “Package plug-ins as individual JAR archives” and “Allow for binary cycles in target platform,” and use qualifier replacement if you used “.qualifier” in your plugin version.

6. FTP the exported jar file to the build machine, and place it directly in the toolkit directory (e.g., /usr/lpp/jazz/v3.0/buildsystem/buildtoolkit). There is no need to stop and re-start the build agent. Whoa!!

7. To use your new task, add a taskdef to your Ant script, such as:

<taskdef name="loaddeltafiles" classname="net.jazz.ant.samples.tasks.LoadDeltaFiles"/>.

Finally, use the new task somewhere in the script, e.g.:

<loaddeltafiles deleteMetadata="true" generateListing="true" deltaPolicy="FULL" passwordFile="${passwordFile}" prefix="${teamz.scm.dataset.prefix}" repositoryAddress="${repositoryAddress}" userId="${userId}" workspaceUUID="${teamz.scm.workspaceUUID}" buildResultUUID="${buildResultUUID}" />

8. Marvel at your working task and the simplicity of the process.

Finding the right service API for your process extension

Update: An Extending Rational Team Concert 3.x landing page has been created on jazz.net for all of your process extension needs! A plugin called Yari is described here that may be useful for determining the APIs you need for your process extension.

While Rational Team Concert comes with many process configuration options out of the box, you may sometimes find it necessary to code your own process extension for a specific behavior you require. While this task may at first sound daunting, the Extensions Workshop on jazz.net will help you easily set up your development environment and test and create your first process extension. I cannot emphasize enough how beneficial it is to run through this workshop prior to creating your extension.

What this post will focus on is not the mechanics of creating your first process extension, but rather what to do when the extension you need to create isn’t for triggering a build on work item state change as we see in the workshop. How do you determine what services and methods you need to achieve the behavior you desire? In a perfect world, we can simply RTFM.  Read the “Fine” Manual. Unfortunately, that manual might be incomplete, hard to follow, hard to find, or non-existent. So, we are left needing a different approach to finding the services we require.

Last week I helped somebody code an extension that would create a new stream, seed it with components from an existing stream, and set up the proper flow targets when a work item is saved in a particular state. We started out with the code from the workshop, which gave us a sample of how to verify that the operation is a work item save and that the work item is newly being set to the proper state. Now what? How do we create a stream, seed the components, and set up the flows?

Whenever I code a new extension, I try first to think of a feature in Rational Team Concert that does something similar to what I want to do. In this case, I know that there is a “Duplicate…” action when I right click a stream under Source Control in the Team Artifacts view. This brings up the Stream editor, with the Components and Flow Targets populated to match the stream I am duplicating. Perfect! I see that hitting Save on the editor actually creates my new stream. Now what I need to do is determine what this editor is doing under the covers. In comes one of the handiest tools I have learned about in eclipse in the last several years: Plug-in Spy.

Plug-in Spy

To bring up the Plug-in Spy, I click on the Stream editor to be sure it is active, then type Alt-Shift-F1. I can see that TeamPlacePart2 is the active editor class, so I click on that link in the Plug-in Spy to open the source. Once inside, I know I’m looking for a save method of some sort. I see there is a TeamPlaceSaveable inner class with a doSave method that looks like a likely candidate. This does a save on an instance of TeamPlaceWorkingCopy.

Open Declaration

Next, I select the call to save, right-click, and choose Open Declaration to open the save method in TeamPlaceWorkingCopy. In this method, I see that a NewStreamOperation is being created and executed. I do an Open Declaration on NewStreamOperation, where I see the methods for creating the stream, the components, and the flows. It looks like I’ve got a winner.  Next, I create a breakpoint on the call to NewStreamOperation.run and launch a debug client, following the directions in the Extensions Workshop.

In the debug client, I duplicate a stream and hit Save. This triggers my breakpoint and brings up the Debug perspective. I am then able to step through the code up until the point that an actual service call is being made. I see that the service I’ll want to use is IScmService, and the methods being called to create the stream and populate the components are createStream and updateComponents2. I can continue stepping through to find the appropriate API for setting the flow as well.

In summary, I can determine the APIs I need for my process extension by finding a similar function in the RTC eclipse client, determining what class provides that function by using the Plug-in Spy, and then launching a debug client to step through the client code until the appropriate server APIs are invoked. One thing to note is that you may find that the service or methods you want to utilize are not publicly documented. If you decide to go ahead and use them, you are running the risk that they may change or go away in a future release. It is a good idea to test and verify your process extensions when moving from one version to the next.