I was looking for the Christmas decorations on my attic, when I came across a set of compact discs. I got rid of most of these, except for some that seemed important enough to keep for posterity. These disks contained an original Window 2000 Professional edition, and one titled Medex Backup 99/05/13.
Yesterday I took my time to review what's in the backup, and took some time to reconstruct the architecture in my mind and the reasons behind it. It occurred to me, that it might be a good time to go through some of my previous projects since, and sum them up, to see how IT (or at least what I'm part of) changed in the past few years.
Strap yourself in, for a travel back in time to the land of the possibilities and techno blable!
Disclaimer: I've put this together mostly from memory and the flashbacks I had during the installation on my virtual machine. I might have left out some interesting aspects, lessons.
Coming up: "10 year retrospective - SCISy, or "Supply Chain Inventory System" @ IBM"
Project goals
The project was started to provide a centralized management system for a group of pharmacies. It had to manage several aspects of IT, point-of-sale terminals with regulation compliant "black-box" hardware, stock management, order management, NHS regulatory reporting and most importantly financial reporting to the owners.
The original target was to cover twenty-something units, which grew to over 120 by the end of the project. In the meantime the expectations also increased, we had hints, that given a few successful installations several major players might join us, and we can expect 400-1000 customers.
The Team
The original development team consisted of three classmates from the University of Veszprém (two of them dropouts at the time, and yes one was me) the brother of one, from a year higher and a long-haired punk rocker who was a team on his own. I will name none of them, as they currently hold respectable positions. The team was positioned in a huge house next to the Balaton for the better part of the project, then started splitting between Budapest and the base when the deployments started. I myself traveled with my trusted midi-tower (alias laptop) there and back at least a hundred times.
Later we extended the Budapest office with a permanent developer trying to create a new-gen fancy client that has never seen the light, a web designer, who designed some graphics, and some short employed developers, who never learned the ropes.
In Budapest we were co-located with our general manager, and the support department, who was always scurrying there and back between the locations to keep the systems running.
The head of the entire organization was our sponsor, he really wanted this to succeed and we had his full support for the entire time.
- Jump to Constraints
- Jump to Architecture
- Jump to Lessons Learned
Constraints
We were faced with quite a few constraints at the time, mostly due to practical reasons.
No internet
There was only a 14.4 baud modem to serve as a point-to-point connection between the pharmacies, wholesalers, and owners.
The National Health Insurance required their reporting in the form of floppies. These had to be generated as two identical sets of 1.44 MB floppies in a signed envelope. The floppies were so unreliable, sometime even 2 sets were not enough.
Must be cheap
Since the system was to replace already established systems, it had to cost even less than those. It also meant, that could not use any commercial software, that required per site or per computer licenses.
Reuse existing hardware
Just like above. The original concept was to reuse whatever the pharmacy already has, and only upgrade whatever is necessary. The system had to run on clients with i286 (16bit) with as low as 640KB of ram and support a number of different network cards. The only hardware we could not reuse was the "black-box" that stored the transaction details for the tax authority.
- Jump to Constraints
- Jump to Architecture
- Jump to Lessons Learned
Architecture
Overall
Each deployment consisted of a back-office computer, a separate server and at least two POS units. All of these were standard PCs with very limited resources. Every pharmacy was connected to LAN via Ethernet, and used the same IP addresses so remote help and access were possible.
The server contained a modem, that was used to receive software and database updates, connect to suppliers (send orders, receive replies) and remote administration.
The software architecture from outside seemed like a simple two-tiered architecture of it's time. In fact it was much more. The clients were quite dumb, they downloaded their resources, including their business logic from the server. The server was in-fact several servers rolled in one. Both the client and the server used the same virtual machine to run the pre-compiled byte code. (Note: The first versions of Java were in development at that time)
Server
The server had several major roles. It was rolled in one, so there would be less components to maintain.
Database server
This would take up an entire article in itself. Why did we think we should create a database server? At the time of the project there were no freely available suitable database engines on the market. The ones available were either servers requiring regular administration, or low-end DBF libraries.
The database engine we ended up with was an extremely fast, mostly relational but not SQL standard based solution. These days it might be called NoSQL, but it wasn't. The record structure was fixed, so it could be indexed, it supported index based aggregation/statistical functions. Indexes were created on demand. It supported different table types to be even faster. If a table had no dynamic length field it was stored in a smaller, faster format.
There were some interesting features in the engine, but one of the most important aspect was the directional link. The data was structured as a standard relational database, but while programming you could "walk" through the associations to see it's content. This pattern emerged in ORM tools years later.
Client connection manager
The server managed the incoming connections, served the resources for the clients, as well as provided access to the database functions.
The connections to the server were through standard RPC calls. In the begining the client called the server GOD. Later a client could connect to other servers, and the names became configurable.
Application server
The server ran it's own VM. It executed server side code exposed to the client as well as cron jobs. Several services were provided using the application server itself.
Internal servers
The web server was first an internal joke, that using the socket API's we created a web server in just a few lines of code. It was used to serve some static content. A couple of months later this was extended to be used as the reporting front-end for the financial data. The reports were collected using the dial-in modem.
As another joke a telnet daemon was written. This was later used, to remotely control the engine itself and became one of the most administration important tool.
The finger daemon was implemented. It returned information about the attached clients and the running threads.
BOOTP server
The thin-clients used BOOTP to download their bootstrap code. More about the thin clients later.
Client
There were three kind of clients for different usage scenarios WIN32, DOS and DOSLESS, they all ran the same VM and the same bytecode, and used the same display forms. Their resolution was constrained to 80x25 characters. On the WIN32 client tweaks with the font, the background image, the button graphics and the modifiable color set made the difference.
The WIN32 version was used exclusively in the back office, it started as a simple local application, and looked like one. It could, and if my memory serves me right, did at one particular installation, reside on the same machine as the server itself. This had a unique feature not available on the DOS clients, it was able to show the report interface in an embedded Explorer (3.0) component. The report data was downloaded from the application server's web server. It was not only an eye-candy, but the presented data showed the actual monthly report with dig-down options.
The DOS version of the client was quite interesting, however never actually used as is in the pharmacies. It was used mainly for development. The client didn't satisfy with the 640KB, it also paged some of it's data to the unused portion of the video memory! The character set was redefined, so good looking big text was available to print the large characters on the POS. This client was ran in a dosemu under linux, so rebooting didn't take as long, and the other terminals were used for development. As my only adventure in Linux kernel development, I created a Hercules kernel module, so I could display debug text on it, my colleagues later wrote an entire debugger that lived on that.
So what's this DOSLESS version? As we had to go cheap, we created an OS based on Linux kernel networking stack, that enabled the client to run without DOS. This was used in the pharmacies, and was downloaded entirely from the server. The POS terminals had no hard drive installed at all.
POS specifics
Beside running the DOSLESS client, the POS had some issues it had to face. As a cashier it had to drive the cashier drawer. This was simple enough, the drawer was connected to the parallel port, and we printed some character and the drawer opened.
But the parallel port? Isn't that supposed to be for the printer? No. Direct printing is forbidden from the POS. It can only have a single attached printer, and that can only print through the black-box, that registers the amount of sales (and the VAT) for the tax-authority. Since these cannot be transferred, and we didn't want to buy from the existing (expensive) black boxes, we created our own. This hardware development was quite a lengthy and painful process, especially when getting the tax-authority's approval. Fortunately it turned out fine, and there were no faults in any of the units.
The POS had bar code readers as well, but we went for the one attached to the keyboard, so that was quite simple.
Tools
There were several tools created to support this framework, both during development, and later on in production.
Naturally there was a byte-code compiler that converted the language (TigC- or tc-) to byte-code (tcb). Then there was a form editor that was used to do the screen layout and set up the tab order. The VM had a built-in debugger, that utilized the Hercules monitors (hard to come by even then)
Later a debugger was created that allowed remote attach to a running VM (server or client) this saved lot of time, as it doubled as an editor. About this time we moved our development to Windows from Linux.
Windiff was modified to support TigC specifics, and enable us to see modifications better.
As we moved to production, remote administration became essential. Several small utilities were created to enable data backup. Later the server core was refactored to be more modular, and enable the server to upgrade itself.
As a final mention, a separate telnet server was created that we installed on the servers. This was used to get command prompt on the remote server. We used FAR commander remotely to create archives, modify settings, view logs. Since the modem was so low-bandwidth, the daemon monitored changes, and sent ANSII (positional and color) codes about changed areas. It was as if I was on my own computer. (Note: Yes there was remote desktop in WinNT, it was incredibly slow through dial-in)
- Jump to Constraints
- Jump to Architecture
- Jump to Lessons Learned
Lessons learned
Validate the constraints
The hardware reuse was completely unnecessary. It turned out there is no problem scrapping the really old hardware. The customers expected it. They wanted shiny new hardware and clean keyboards.
We practically wasted our time on the DOS and DOSLESS versions of the client. A stripped down Linux, with a 32 bit application in proper graphics mode would have looked nicer, and be easier to create. We wouldn't have to have worried about the 640KB memory limit, as 16MB was standard those days.
Plan for maintenance
During the project the 5 core developers shared a house, and worked really long hours to accomplish the goals. It was uplifting, interesting and extremely productive. In about a year from the start, the first pharmacy migration was completed. Unfortunately the team was more motivated about the framework, than about the product itself.
Developing everything from scratch was a necessity. Unfortunately it introduced some huge maintainability issues. The framework had a few quirks to be ironed out, such as the table spaces were expanding faster than expected, which turned out to be an error in the db layer. This required most of the core team to concentrate on the framework itself.
Plan for expansion
When we reached about 5 deployments, development requests started pouring in. We had no means of expansion. The proprietary language was easy to use for a junior developer. That is, once you understood the concept behind the architecture. But that was too much for our candidates to handle. We started hiring too late and in retrospect with no clue of what we are looking for.
Initial application deployment and migration was time consuming. We were going as fast as we could, but we managed to migrate two pharmacies per month tops. Those had to settle for about a month to start a new one. We got better with practice, but the 1000 possible targets (50% market share) was way out of reach.
Know when to quit
Even though these expansion problems started to emerge, the project closed with a success. The company with the biggest share of the pharmacy information system scene noticed us. Since we, due to the sponsor of the project, were destined to cover at least 10% of the market, and we offered better service for less monthly fee, they made their move. We were offered a non-disrespectful sum for the entire company and an employment opportunity for all the developers. We packed up the sources, took the money and turned down the job offer.
