This is the second part of my retrospective series I started here.

When I'm asked about my favorite project in my career I always talk about this one. Every part of this project was solid, well established and justifiable. There were no need to make any compromise on anything. We didn't have to worry about licence fees, as long as we used IBM. :-) And what's most satisfying about a project? The sponsors and the users find it useful!

Disclaimer: I've put this together mostly from memory and the few remaining documents I found laying around in my archives. I have no idea if this system is is in use any more, for all I know it probably is.

Background

A few years passed after the Intermed project, Java became a strong player in enterprise development. With the J2EE application servers becoming a viable option, coding your entire stack from scratch was no longer necessary. Still there were quite a few gray areas that had to be worked around.

I was working as a contracted freelancer at IBM at the Hungarian manufacturing site, that produced (and still produces) high-end data storage systems. The manufacturing site ran a high number of IT components: shop floor control systems, order management systems, procurement systems, truck load tracking. All in all it was a smooth running operation, but still faced a problem. There was only a really limited communication between the systems, so monitoring the actual status of an order was quite difficult.

Production monitoring is essential for running this operation. No two orders were the same. During the "free" time the plant manufactured "prebuild" machines, that were configured for the most likely order configurations. These were placed in a buffer, and when an order came in they were reconfigured with the exact specifications. This required a common monitoring application that could be used to match the orders and the available machines. The most time in the build phase was used for testing the configuration. If a "feature" was removed from the machine it didn't require re-testing, and it was easily shipped. It was essential to use pre-tested prebuilds that had minimal difference to the order.

To achieve a transparent order monitoring a Lotus Notes application was created which was loaded with  data from the relevant systems provided real-time reports to oversee the operation. There were some problems with the solution, as it was not as fast as expected. The data load was running so long, that the incoming datafiles stepped on each-other's toes, data was inaccessible during the load, so basically the system was down for the better part of the day. The project was started to overcome these problems.

Project goals

The original project goal was only to create faster data loaders and increase the availability of the data. I've created some data loader plans that loaded the data into a separate DB2 database. The issues with the original application, escalated and a team was assigned to create an entire new application front-end along with the loaders.

Availability

The new application had to be designed to provide availability of data especially for the month/quarter-end periods. It was no coincidence this was exactly the time when the most amount of data poured in from the source systems.

Design

The users of the system were ranging from manufacturing staff to higher management. They were used to the Lotus Notes application, so the new app had to match the design principles and feature set of a standard Notes application.

The users were used to be able to create their custom views, reorder columns, filter data using quite sophisticated tools. They were also keen on grouping using "twisties" , as seen on the image. They even liked to use icons in their views  and highlight rows matching given criteria.

Flexibility

The system had to be designed to be easily modifiable and customizable.  It had to be simple enough to be extended with new fields, views or even business logic in days.

Easy administration

The plant had limited administration options, HelpDesk was (for all I remember) centralized, so deploying any software change to the clients was a long procedure. SCISy being a business critical application had to have rapid delivery, and easy administration features.

Security

The application had to be certified by internal audit. They checked for most known application vulnerabilities at that time.

There was no external public access of the application possible, and the internal regulations were quite lax at the time. Still it had to match some security regulations.

The Team

I'm quite certain I can say, the team was the best team in Hungary for this particular job.

Project management was delegated from Mainz, and they didn't have to do much, except for the regular administration tasks. They were always worried about deadlines, but didn't care much about the actual product.

We've done the technical project management, development planning together was with an old friend of mine, who I could call the Business Owner of the project. We set up milestones, made architectural decisions, planned work together.

The team members each had particular strength, and were assigned their tasks accordingly. We originally had

• a lead developer, who created and maintained the loader logic, database structure, integration tasks
• an eccentric developer, who did insanely rapid UI prototyping, even though I replaced, literally (!) every single line he's done with a more robust solution, his mad skills got us through the first milestones
• an excellent developer, with mean algorithmic skills, who worked on tougher business algorithms, and query optimization
• an excellent junior developer from my company, who learned everything in a matter of minutes, and used the patterns to create the rest of the business logic (he was replaced before completing the project with another, less enthusiastic guy)
• and I was also working as the principal developer, to establish the framework and keep it together

Later the UI developer was replaced by two "standard" developers, who did the application programming after the UI guy left, one of them was brilliant enough to make up for the incompetence of the other. Also a trainee joined us as junior test manager, who tested the application, later I hired him, and managed to mentor him into an excellent solution architect.

Architecture

To help me jog my memory, I actually found the technical specification of the app. The provided images are taken from that document.

Overall

The application was designed in the EJB 2.x era. Several design decisions were taken to overcome the weakness in the Application servers.

The system was broken down two two logical parts Front-end and Middleware.  I keep calling it "Middleware" in the document, as that was the pseudonym we called it at then.

Middleware - Data loader

The data loaders were working on the "Backend" database, and were implemented in MQ Message Broker.  This was an excellent hub that made use of the various input filters of the product. We saved tons of time on using the IDOC, fix format, CSV inputs. It had incoming MQ messages, direct DB connections, text file inputs, with a single output: the backend database.

Since the message broker was used for nothing more than data collector, calling it a Middleware is an overstatement. In fact Message Broker was mostly used as an omnivore ETL solution, that was not available on IBM's SW palette at the time.

Middleware - Front-end

That was not Middleware, nor Front-end. In fact that was the actual application server layer of the front-end application.

The application layer was implemented using web-services. Unfortunately at that time only JAX-RPC  was available, and we could only use the Apache Axis 1.3 library. This was quite problematic as the session management had to be implemented separately. This is now considered a huge potential security issue, as it makes session hijacking possible.

The web services were completely stateless  and nothing was stored about client sessions except for the session-id itself.

The most important part of the application layer were the "datasets" and "views". Datasets were basically Hibernate queries mapped to the underlying Backend database.  Views defined how these datasets should be filtered/aggregated and displayed.

The report services provided access to the data and the report definitions. The data access was a simple variation of the view service itself.

Business logic services were data manipulation functions. Most data was overwritten upon updates, but there were some, that were only modifiable from the application itself. All business logic was implemented in the backend layer, so the client only dealt with display, input, and basic data validation.

Client

Most of the application was designed to use database based views. The tree structure on the left, the layout of the data sets, the views based on the data sets, and even the forms used to display the front-end were stored in the metadata database.

The client was deployed using WebStart, which made it easy to distribute new versions to all clients. The application itself was using Eclipse RPC, and the actual application was an OSGI plugin deployed in it. Using Eclipse enabled really fast Windows based software, that is compared to the "traditional" Swing based UI option.

The most sophisticated part of the application was the list view itself. It was an "owner-draw" component, based on a list component  originally written for Swing. It was extremely fast, supported all the listed features, such as icons, sorting, grouping or highlighting.

The "detail" view of the list items was hand coded in XML, and it used a number of pre-defined UI components that referenced the current DataSet's fields.

The SWT application also incorporated BIRT to create very good looking reports in PDF as well as quite complex Excel sheets. The report definitions were in BIRT format, and used a custom data provider to use of the dataset concept.

Lessons learned

Agile development practices work

The project was quite successful. It delivered, it worked, there were no errors after roll-out.  How is that even possible?

We were what you'd now call an agile team. We had two daily stand-up meetings, albeit informal, when the entire team had their morning and afternoon coffee. We went trough everyone's tasks, and discussed whenever it was required. The Product Owner usually participated, but since we carpooled, we discussed everything in detail.

The scope was flexible. We had a set of required functionality for every targeted milestone, but we had quite a huge playing field on what else to put into it. We always managed to put in some eye-candy for the demos, and that pleased the sponsors.

We delivered often, when the framework was already up and running, the business development started straight away using the available functionality. If something was missing, we worked around it.

The team members knew what to do, and how to do it. If they were stuck at one point, they knew who to ask for help. When their task was ready, they found another one.

Open source can be decieving

In the solution we went for some upcoming open source software solutions. They worked, but all the problems in the project lead back to these.

Hibernate 2.0.something was the ORM layer, which was almost stable. Unfortunately when it was not, it was no use to go to the King (Gavain King, the lead developer behind Hibernate) with a problem statement, a failing test case and a proposed patch, he refused to correct the issues. So the project had to fork that version of Hibernate and patch it for itself. (NB. last time I checked a few years ago, but the very same bug still existed in much later versions)

We also had issues with JAX-RPC. Unlike JAX-WS from a few years later, this was quite a performance hog. Axis 1 was a poor implementation, and we had to take lots of manual work to update our WSDLs at releases.

Eclipse RCP and BIRT do not play well, unless they are the exact version BIRT is made for. This caused upgrade problems. Later versions of BIRT were simply too heavyweight to be used as an embeddable component. Eclipse OSGI is not very friendly with WebStart distribution.

These version dependencies were very heavily built in the core of the framework. So even though it could still be running, the core libraries are extremely obsolete now and I have serious doubts, that anyone spent time to upgrade those since.

Design for extensibility

It was one of my earliest projects, where I designed the architecture of a complex system alone, and that architecture was implemented as intended.

Since the layers of the application were so well defined, it was really easily extensible. Later, when there was a less busy period in my work, it took me two days, to add a web interface, that worked exactly like the client. It was a bit crude, and it was missing some features, but it was usable just as well as the rich client. Why it never made it in production? The rich client used less memory, than Internet Explorer.

Use patterns

Patterns are simple, once you understand one, you can use it, no matter what technology is used to implement it.

I've used the same WebStart/RichClient/WebService architecture pattern to put together two similar projects since.  First for small developments at IBM, and it was used for at least two small projects. This used Swing instead of RCP and used non-generic web-services.

The other at my own company a few years later, that was used to implement a distributed inventory system. This later one is still in use today, and uses RCP but with proper JEE 6 backend (JPA2, JAX-WS) and non-generic web services. This also has a JSF2 based web interface to complement the client.