Bootcamp, ACMS, and Wet-Weekend Projects
In my previous post (https://vmssoftware.com/resources/blog/2024-09-30-application-services/) I talked about the unfortunate incremental decline in the number of ISV solutions available for OpenVMS and what can be done about this situation, with my general thesis being that with a little ingenuity and imagination it is invariably possible to devise excellent alternative solutions, often by leveraging open-source products and open standards, such that the resultant replacement solution if done properly may be in some ways superior (and arguably more cost-effective) than its no-longer supported and proprietary counterpart. In this post I will aim to continue with this general theme of "creative modernisation", based around a talk I gave at the recent Bootcamp event in Boston in which I demonstrated the ACMS product working on VSI OpenVMS x86-64, but before launching into the details of what I presented in this session I should just mention how great it was to catch up with old friends at Bootcamp and to see not only many familiar faces but also a good number of new ones. The event was without question a success, and what struck me most was the level of enthusiasm displayed by all those in attendance, which was absolutely great to see.
As I mentioned, one of my tasks at Bootcamp was to talk about and demonstrate ACMS running on VSI OpenVMS x86-64. ACMS (Application Control and Management System) is a powerful and sophisticated transaction processing monitor software system for OpenVMS that was originally developed by Digital Equipment Corporation in the early 1980's and remains popular with many large OpenVMS users to this day, most of whom rely upon it to run their key business-critical applications. ACMS is very big and very complex, and it is probably fair to say that as a consequence of these facts we had been procrastinating just a little in terms of porting it to x86-64, although it should of course also be noted that ACMS is of limited use to most users without Oracle Rdb and CDD, so until we had some certainty around when Oracle would have these products available for OpenVMS x86-64 there as arguably limited value in starting to port ACMS. Around mid-2023 we decided to bite the bullet and start porting the product, and thanks to some absolutely fantastic work by Anders Johansson we had an initial but incomplete port up and running earlier this year, with the remaining work hopefully soon to be completed such that we can provide customers with a field test version or beta release coincident with the field test availability of Oracle Rdb and CDD.
In general, I have something of an aversion to doing live demonstrations of software, having seen way too many such demonstrations go badly awry over the years, but in a moment of madness I accepted the challenge. What could possibly go wrong?
But it occurred to me that ACMS is not really the most enthralling product to demonstrate, so I decided I needed to come up with a twist or two to make the demonstration a little more interesting (and to increase the risk of disaster). Looking at it rather simplistically, ACMS provides infrastructure for building and administering complex transaction processing systems with large numbers of users and multiple server processes. Accordingly, ACMS applications have over time tended to become the centre of the OpenVMS application universe for those customers using the product, and the challenges they face today are not so much around having these application environments support large numbers of concurrent interactive green-screen users, but rather around how external applications can integrate with the ACMS environment to leverage the functionality these applications provide to the business. ACMS applications are inherently service based, with units of work being called "tasks" (as opposed to "services"). Accordingly, ACMS applications are often well-placed to be integrated with other systems, having well-defined interfaces and ready-made application-server-like characteristics. With these thoughts in mind I set about devising a demonstration that included a few ways this integration can be done, and just for good measure I decided to also throw in a couple of what I like to refer to as wet-weekend projects (ideas I've come up with that may or may not ultimately be useful for anything, but that keep me amused on cold and rainy winter days when I can't think of anything better to do than to write some code, sad but true).
The idea of the demonstration application was to simulate a small chain of coffee shops, using the Coffee Shop Sales data set from https://mavenanalytics.io/data-playground. To do this I created a simple ACMS server application comprising tasks to get store locations, to get product details, and to process product purchases, essentially corresponding to the three tables in the aforementioned Coffee Shop Sales dataset. The original CSV-formatted data files were converted into indexed RMS files to create an RMS-based database for the application, and here is the first twist: Why I hear you ask did I not just load the data into a relational database? The answer to this is simple: I wanted to have some fun incorporating into the demonstration one of those wet-weekend projects affectionately known as “Skunk", which provides facilities similar to those provided by Attunity Connect and the CONNX data integration suite to query indexed RMS files via SQL, and the fact that Skunk uses the PostgreSQL wire protocol meant that my ACMS server code (written in C) could include embedded SQL statements that could be pre-processed using the PostgreSQL embedded SQL pre-processor included in our port of the PostgreSQL client for VSI OpenVMS, and the resultant generated code linked with the PostgreSQL client API to facilitate communication (via Skunk) with the RMS database. And before you ask how I built an ACMS application without Oracle CDD (which requires Rdb), I will not bore you with the details, but it is actually not that difficult to do a kind of a hybrid-build, building the data dictionary and the ACMS task, group, and application definition files on Integrity and doing the rest of the build on x86-64. The OpenVMS x86-64 system used for the demonstration was running VSI OpenVMS 9.2-2 on a VMware virtual instance with 16GB RAM and 4 virtual CPUs.
The fact that Skunk uses the PostgreSQL wire protocol means that it is possible to interact with indexed RMS files using essentially any PostgreSQL client, including the PostgreSQL ODBC driver, and the PostgreSQL psql interactive query tool, as illustrated below (you’ll have to take my word for it that I am indeed querying RMS files here). As things stand, Skunk could best be described as an advanced prototype, with the main areas needing attention being performance (to make better use of RMS indexes) and support for SSL/TLS encryption, neither of which are particularly big jobs. Configuration is currently somewhat clumsy, and could also be improved, but fundamentally, this little wet-weekend solution is shaping up rather well.
But getting back to our coffee shop application!
To exercise the ACMS application, I developed a simple client simulation program on Ubuntu Linux that communicated with the server application via the ACMS ACMSDI gateway using the ACMS TPware client API (see https://vmssoftware.com/products/acmsdi/), which we have substantially enhanced over the past few years to support multiple operating systems (OpenVMS, Windows, Linux, and HP-UX), multiple programming languages, and support for SSL/TLS encryption. The client program selected at random the store location, point of sale within the store, product(s), and product quantity to be purchased, performing task calls to the server to obtain the required information and to perform purchase transactions, with each such transaction being written to the RMS transactions data file via an SQL insert statement. The time between purchases was also selected at random, and the simulation could be run indefinitely or for a specified number of transactions. It should be noted that the aim of the simulation was not to assess the performance or scalability of the OpenVMS x86-64 ACMS environment, but it could with some slight modification be used for this purpose.
In addition to implementing the various services (ACMS tasks) mentioned above, the demonstration application also included functionality to replicate sales data off-platform to a pretend data warehouse and functionality to facilitate the real-time monitoring of sales per store. Replication of sales data to the data warehouse was performed using the PAMS replacement API described in my previous blog post (https://vmssoftware.com/resources/blog/2024-09-30-application-services/), with details of each product purchase being published in JSON format to a RabbitMQ message broker instance hosted by CloudAMQP (https://www.cloudamqp.com/) on AWS, in addition to such details being written to the RMS “database”. These messages were consumed by a Fortran program (hey, why not?) running on another OpenVMS x86-64 system, converted to SQL statements, and inserted into an Oracle Rdb database running on an OpenVMS Integrity system using a close wet-weekend relative of Skunk known as Punk, which provides similar functionality to Oracle Rdb SQL Services but – like Skunk – uses the PostgreSQL wire protocol, thereby providing a rather convenient means of interacting with an Oracle Rdb database from pretty much anywhere. In the present instance, connectivity between the Fortran data warehouse loader program and the database was facilitated using UnixODBC for OpenVMS and the PostrgreSQL ODBC driver, using a simple wrapper API to make it easier to use the C/C++ ODBC API from Fortran.
Real-time monitoring of sales per store was implemented by incorporating into the ACMS application a simple Prometheus agent that maintained metrics relating to total sales revenue per store and revenue per minute. These metrics could be consumed via Prometheus (https://prometheus.io/) and viewed via Grafana (https://grafana.com/), as illustrated below.
While the Prometheus agent was intended to be more of a functional feature of the demonstration ACMS application (allowing the coffee store owners to monitor revenue), it also provided a convenient way to monitor transactions occurring when running the simulation client, however it was also possible to observe these transaction using new features incorporated into the ACMSDI gateway product to query various operational metrics including the number of task calls processed, and to observe via the RabbitMQ management interface messages being published to the RabbitMQ broker by the ACMS application and consumed by the data warehouse loader. In fact, for the demonstration I did not initially start the data warehouse loader, allowing messages to build up in the associated RabbitMQ message queue to make it more clearly obvious that transactions were occurring. And of course, it was also possible to periodically query the RMS transactions file (using Skunk) to get a count of the total transactions processed.
Now, the observant reader will note that I have so far actually only talked about one form of integration with ACMS, namely the use of TPware and the ACMSDI gateway to facilitate communication between our Linux-based store simulation program and the core ACMS application. Sticking with the notion I postulated above that over time ACMS applications have in many cases tended to become the centre of the OpenVMS application universe for those using the product, a logical extension of our “in-house” ACMS-based coffee shop application would be to provide an online presence so that customers could order their preferred beverage via the Internet. To this end, I asked one of my younger colleagues (who is much better at such things than I am) if they could develop a simple web-based application to fulfil this requirement, with the basic design brief being that the web application should be developed on VSI OpenVMS x86-64 using a similar architecture to our WebUI product (CivetWeb, Lua, and JavaScript) and should communicate securely with the backend ACMS application using WSIT (the Web Services Toolkit) using AMQP 0.9.1 via the AWS-hosted CloudAMQP-based RabbitMQ message broker used for the data warehouse interface, leveraging exactly the same set of ACMS tasks as used by the simulation program. The other requirement was (perhaps somewhat obviously) that this web application should run in a public cloud environment and be accessible to anyone via the Internet.
My young colleague did a great job of implementing the web application, designing the user interface so that it could be easily used from a mobile phone such that members of the audience could load the web page on their phones during the demonstration by scanning a QR code, and start placing orders. We could then watch these orders come through in real-time via the Prometheus interface described above, with larger orders being clearly visible as spikes on the "Sales per minute" graph. For the cloud environment, we used DigitalOcean (https://www.digitalocean.com/), which provides a nice environment for setting up and running KVM-based OpenVMS guests for small projects.
For those readers not familiar with the Web Services Integration Toolkit (WSIT), it is an extensible API-level integration technology for OpenVMS originally developed by HP in the 2000’s to facilitate the integration of ACMS-based applications and new or existing 3GL code with Java. We have extended the product in recent years to support the Advanced Message Queuing Protocol (AMQP 0.9.1) as an alternative to the OpenVMS-specific ICC protocol, and we have enhanced the code generation facilities of the product to facilitate the generation of Java, C#, Python, and JavaScript code that can be used in conjunction with RabbitMQ (or another AMQP 0.9.1-compliant message broker) to provide an highly scalable, efficient, secure, and fault-tolerant cross-platform integration solution.
But I digress!
If you are starting to lose track of all the moving parts involved in this demonstration, don’t worry; by this point in proceedings so was I, and it was about now that things started to go a wrong, albeit more due to me not paying attention to what I was doing than for any sort of technical reason.
The story at this point in the demonstration went along the lines of the fictitious CTO of the coffee shop chain deciding that we were going replace the RMS-based database with CockroachDB (see https://www.cockroachlabs.com/) hosted on AWS, and we were to move the data warehouse from Oracle Rdb to Microsoft SQL Server on Rocky Linux. CockroachDB is a relatively new relational database, with its key characteristic being that it is a truly distributed relational database that can be scaled both horizontally (by adding more nodes to the CockroachDB cluster) and vertically (by using more powerful nodes). It also uses the PostgreSQL wire protocol, meaning that from the perspective of the coffee shop demonstration, no code changes are required in order to replace the RMS database and Skunk with CockroachDB; it is just a case of creating a CockroachDB instance, migrating the data, changing the connect string in the coffee shop configuration file to use the new database, and restarting the ACMS application. Given the small number of tables involved, creating the necessary database structures was trivial, and facilities provided by the technology underpinning Skunk could be used to migrate the data from the RMS files into CockroachDB in a matter of minutes. Unfortunately, I bungled this process, and the demonstration pretty much petered out at this point into more of a discussion, but hopefully the audience found it interesting and thought-provoking in terms of the sorts of things that are in fact possible when it comes to modernising OpenVMS application environments and integrating them with other systems.
I should add that conversion of the data warehouse from Rdb to SQL Server would have been similarly straightforward, with the plan being to use a little database conversion tool I developed many years ago to automate the conversion process, whereupon all that would be necessary was to change the ODBC connect string used by the data warehouse loader to point to the new database (again no code changes required).
So, there we have it: a demonstration of ACMS running on VSI OpenVMS x86-64 with a few twists, including the use of cloud and cloud-based services, integrations with new and updated middleware tools, real-time monitoring, and the inclusion of some interesting wet-weekend projects that might one day evolve into viable new solutions for OpenVMS. As I suggested at the start of this post, ACMS applications have for may ACMS users gradually evolved into being the centre of the OpenVMS application universe, with the challenges faced today being not so much around having these applications support large numbers of interactive users, but rather with how external applications can integrate with these ACMS environments and leverage the functionality they provide. For example, in the case of the coffee shop, we moved from having what was essentially a local network-based application to providing an online presence without having to make any changes to the core ACMS application, and we seamlessly included extensions for data warehousing and monitoring.
I think I will go back to avoiding doing live demonstrations, but it was fun to create this demonstration, and hopefully you found at least some of it interesting. And perhaps it has given you some ideas for one or two wet-weekend projects!
Director of Application Services
