Pragmatic MDD: Workflows and more

So the last time we were facing 2 problems: A lot of code in a file for just one transformation (and part of this code could be potentially reusable later) and definition of workflows.

Let's begin with the solution for the first problem. It's an easy one.

We proceed here exactly like before: We define a separate transformation for the setting of meaningful defaults. Since there is not model-to-text transformation, result of the runner is just another model, that gets saved in the output folder.

This changes things slightly - Since the input model of the PojoWithAutoupdate transformation is not our original model anymore (remember, we just moved all the default settings to a separate transformation), but the output of the previous step (the new default setting transformation), we also need to change the setup of PojoWithAutoupdate a bit: From this to this, the model name and model path have changed.

So now we have 2 transformations and somehow they need to be executed one after another.. How can we do that?

But before we answer this question, let's solve one more problem. Since Java wants to have package names consistent with the folder structure where files are, at the very beginning we have to setup all the missing folders and, before that, remove all the old files still remaining there. This is done in one more workflow step. This is for sure not the most elegant way, but it does its job.

So now we have even 3 workflow steps that need to be triggered one after another. And this is exactly what we do here: Just triggering one step after another, passing all the overrides as parameters the same way we did before over the command line.

Pragmatic MDD: Hibernate injection

Ok, now we have a domain model - perhaps not the most exciting one, but it has everything we usually need: inheritance, custom data types, an enum.

So let's continue and go a little bit deeper: How about adding JPA hibernate annotations to our classes? Of course, we don't have to use hibernate, but it really doesn't matter, which ORM we take, the principle is always the same.

How can we add these annotations? The same way we have customized our setup before: By declaring injection points in the primary transformations, overriding these points with our custom logic in secondary transformation and then load this secondary transformation on workflow execution.

Enough theory, how does the code really looks like?

1) The normal POJO m2t transformation - here we added some empty methods, like preClassDeclaration and preAssociationFieldDeclaration

2) Override of these empty methods in hibernate injector - here we add our annotations

3) Loading in the runner happens exactly the same way as before - we don't need to change anything

This solution is ok, but now we have several new problems.

First of all, we have a lot of code to set meaningful defaults for multiplicity, naming and aggregation. This is ok as long as we have only one transformation, but what if we have more? This code should be in its own module, since such logic is not really related to the POJO generation.

And this creates a second problem - if we have 2 different transformations, adding of the default values and POJO generation, how can we trigger them one after another? For sure not manually.

Let's see how to define workflows next time.

Pragmatic MDD: Can we scale?

So till now our model looked really very limited: just 2 classes, no enums, no custom types..

Let's change it.

If you take a look at the original model, there are several things we need to consider:

- AccountState enum in Account class

- RFID data type for tag attribute in BookItem class

- And let's just for fun add one more custom type: Book.ISBN will become a String with length 13

And this is how we will model it in JSON: nothing surprising, just an additional list of custom types.

So how shall we implement it?

First, we extend the UML transformations to see some nice pictures: 3 more lines of code to copy the enums and we are done.

Now let's proceed to the object transformations. Not a big change here too:

- m2m: resolving of custom types and enums

- m2t: additional template for enums

- runner: every enum needs to be written in its own file

And this is already all, now we have a fully working model!

Pragmatic MDD: I miss UML

Let's take a look at our JSON model. It looks kind of like a .... JSON model.

You can't really show it to anyone. You can't discuss it with other people. You can't brag with it in front of your boss. You can't put "UML" on your CV. You basically can't do anything with it. Except of generating code out of it, of course.

But wouldn't it be nice to have an UML model as a basis of any design discussion?

Yes, it would.

I don't think that UML should be a starting point, the source of our pragmatic MDD approach, but as one of many targets, as a result of one of the many different transformations we are going to write, it has for sure its merits.

So how are we going to do it, how will we generate UML out of our JSON model?

Will we write some kind of visual UML editor? Isn't it a completely stupid idea, since there are already so many of them out there?

Yes, indeed, it is. The net is vast and infinite, it has already everything we need to solve our problem.

So let's use this very nice plantuml tool to produce some UML diagrams. The only thing we need to do is to transform the JSON model into the plantuml format.

Here is the code how to do it. It's super easy. Seriously, it couldn't be easier.

Perhaps you noticed, that we are missing the runner module here. This is because of the way we are going to use plantuml.

Here is a simple html website with a fake image tag. In the source of this image tag we are going to put the uml text (result of the transformation) and then we will let plantuml do its magic.

But I would like to mention 2 things:

1) Since we are using an html website, let's switch to JavaScript here. We could perfectly well also continue with CoffeeScript using this example, but it actually complicates things way more than necessary. But in that case, of course, to embed the code in html website, we need to compile our CoffeeScript transformations to JavaScript, which is super simple to do: run coffee -b -c filename.coffee and you are done.

2) Run it in Firefox. Chrome and IE don't work out of the box, but why bother anyway.

Download the code, open the website and what do you see?

How much easier can it get?

Pragmatic MDD: Divide and conquer

So last time we found some problems with our code:

- Everything was in one file and it got quite big

- There was no way to customize code generation – e.g. to set the folder where the generated files should be saved

- We didn't have any tests at all (and that is fine for a throw away code, but a must in case we want to maintain it).

Let's see how we can improve the situation step by step.

File is too big? Let's split it.

Before we saw, that our whole process consists of 2 separate steps - model-to-model transformation (m2m) and model-to-text transformation (m2t). So perhaps we should just split these 2 parts into separate files? But why stop there, why not split it into 3 files: One with all the function definitions required for m2m, one with everything for m2t and one that will actually call our m2m and m2t modules, be responsible for the communication with file system and do whatever else is needed. Let's call this third module "runner".

So this is how it will look like. Is not too bad, I think. Such splitting has also one more advantage - testing of the transformations gets incredibly easy. Since all the functions in m2m and m2t modules are side effect free, we can test all of them separately without creating any mocks or writing some complicated setup. Take a look, how the test modules are written.

For the asserts, we are using chai library, as test framework mocha and tests can be executed locally with following command: mocha . --recursive --compilers coffee:coffee-script/register

So far we fixed 2 problems: File sizing, complexity related to it and testing. But what about the customization?

Let's define a separate setup module containing methods, that we will always need, like input folder, where our model is, output folder, where the generated files should be saved and model name (name of the file with the model). Actually, we don't need to define any functions in this module at all - thanks to the dynamic nature of JavaScript we could just add new methods and override old ones on the fly, but still, since we can be quite sure, that these 3 methods will always be needed, let's define them in this module explicitly (even when their bodies will remain empty). Also notice, that we don't define anything regarding namespace here - since namespace will be needed only for such specific transformations as to POJOs, we shouldn't put it in this common setup file.

Now we should add some implementation to empty methods needed for our case. And this is exactly what we do in custom setup file - we are loading our default setup file with empty method bodies, override the old methods and add a new one to provide the namespace for our POJOs.

So far so good.

But how can we access this custom setup file from the runner, where all this information about folders is needed, without referencing it directly?

Via command line.

Take a look at the lines 4 and 5 in the runner. Here first we are loading the default implementation of the setup file with its empty implementations, but then we also load all the modules, that were passed to us via command line.

So if we execute something like coffee runner.coffee ..\..\UseCases\setup.coffee, the application specific setup file will be loaded after the empty setup file, in this way overriding it's old methods and adding new ones. No need for any direct dependencies here.

Pragmatic MDD: Show me some code

Finally! Finally we can take a look at some code!

As discussed last time, we start with generating POJOs out of the model, the simplest exercise ever, and this is exactly what our implementation is doing (BTW, to run the code, just type coffee pojo.coffee).

But perhaps first I should make a remark concerning CoffeeScript syntax: If you are a JavaScript developer and don't know this syntax, here is a nice comparison of CoffeeScript and ES6 (and you might also notice, that ES6 will already include support for most of the features we are using). If you are coming from Java, I hope you are familiar with Stream API. If you are a .NET developer, notice similarities with LINQ. Otherwise you can always take a look at this tiny overview of the language features on the CoffeeScript homepage.

So let's come back to the code now.

It's so small, I don't think we need to spend too much time discussing the details, but the general structure is probably worth a look.

As we have seen from the theoretical part last time, our code needs to contain 2 parts: Model-to-model transformation(m2m) and model-to-text-transformation(m2t).

This separation is also clearly seen from the code:

- m2m: We transform the inputModel (our original JSON model) into the outputModel in such a way, so that it contains all the necessary information for our templates to generate code out of it. And this is exactly what we are doing in lines 9-22: Copy everything except the associations from the original model and then process the associations to include information about them in the corresponding class definitions (don’t forget to add default names for association ends: since in our model the name “wrote” is defined only on one end, for the other one we need to generate something meaningful, like “getAuthors”).

- m2t: We define code generation templates in form of functions (lines 25-43) and then apply these templates to every class of our transformed model (lines 45-49).

Ok, this was simple enough. Altogether less than 50 lines of code, not too bad (even if it is probably not the most meaningful metric in this case).

But let's see how well does this approach scale: how about making our associations auto-updateable?

Here is the code.

So what has changed?

The m2m part got a little bit bigger - now every association belonging to a class not only has to know about the class it references, but also about the class it belongs to. Of course, we could just find all this information on the fly from the model, but why bother, when we can spare this effort by just coping the needed parts. At the end of the day, this is why we use transformations at all.

And what about the m2t part? Here we got a lot more templates: separate templates for attributes and associations, initialization of empty lists..

And, of course, all these different cases in handling of the references: Many to many, one to many, one to one. Setting of an item, resetting of an item. Adding an item, removing an item. It is a lot of templating here.

Unfortunately I don't think we can do much about it - we want to generate code and need to specify templates. This way or another, it needs to be done.

But there are several other problems here:

- Our file got quite big, about 100 lines. This is not so good, it's better to split it.

- What about the package name for our classes? Right now it is taken from the model, but what if we have several models (or actually the same model, but just splitted into packages between different files). Do we then need to put it in every file? Or do we define one "main" file, containing such information? Not good. We need to make it more flexible, we need to pass the package name to our code in some other way. Same goes for the input folder, where the model can be found, and output folder, where we want to have our generated code saved.

- And what about the testing? Right now, with all this file reading and writing, it would be impossible to write unit tests for such a module. But we need to write tests. We are pragmatic here, but not stupid.

So let's see what we can do about these issues next time.

Pragmatic MDD: Use the model, Luke

So (hopefully) we have decided about our stack: JSON for model definitions, CoffeeScript to do the magic and Node.js to make the magic happen.

But there is one tiny problem: unfortunately we are kind of misusing JSON here. In comparison to UML, we are gaining flexibility (a lot of flexibility), but also gaining verbosity. And I don't think we can do much about it.

We could, of course, like mentioned last time, go for the domain specific languages, but it would complicate things a lot, really a lot. So before going this way, let's give JSON a chance, embrace the ugliness and see, how far it can take us.

Ok, by now you are probably fed up with all these theoretical discussions and want to see some code, right?

Sorry, not so fast. First we need to decide what do we actually implement here, what domain do we model.

Since I'm a very lazy person, I don't want to come up with some fake use case on my own, but will just steal something already existing from the internet: It's a library model

Very original, isn't it? So we have some classes here, some attributes there... A few associations... One enum... I hope, you are as excited as I am...

But, ok, let's quit complaining, take for a start just 2 classes, Book and Author, and model them in JSON.

So here comes our definition. Like promised, it's quite ugly.

By the way, if you are wondering, why do we need to keep associations outside of class definitions, here is why: If we keep them inside the classes (meaning, one end of the association is defined in Book class, another end in Author), we could never be sure, that these 2 ends actually belong to the same association. For example, you could have multiple associations between the same pair of classes (e.g. for Book "writtenBy" and "ownedBy" and for Author "owns" and "wrote") and not know, what is the other end for "wrote". Is it "writtenBy"? Is it "ownedBy"? I hope, you can see the problem.

But don't let me hold you back, feel free to experiment with different ways of model definition. Everything depends on your use case anyway. For example, if you just want to generate POJOs/POCOs, then associations need to be placed inside the classes one way or another. Or to save some typing, you could put name and type of an attribute together, something like "string ISBN". Or encode the type in the name of an attribute like "sISNB" and "iNumberOfPages". Once I actually worked in a company with such coding conventions.

But let's move on.

Now we have our tiny model, so what's next? Let's generate some POJOs out of it!

I know, I know, isn't it kind of stupid to ask from all these hip JavaScript people to go down to the enterprise hell known as JEE? Or to expect from all these poor souls who are already burning in the flames of JBoss, WebLogic and WebSphere to have ever heard anything about CoffeeScript? For whom am I actually writing this? The most honest answer is: No idea.

Of course, we could generate JavaScript domain objects, but I'm just much more familiar with the daily needs (and pains) in life of a Java/C# developer, than a JavaScript one. Anyway, the generated code is just the result, it doesn't really matter for the purpose of this whole undertaking: Defining (or at least trying to define) a set of best practices for a pragmatic MDD approach.

But let's get back to business.

So we have our model now. It's ugly, but it's in place, this is already good news. What do we do next?

Let's think for a second: We want to generate code out of the model, so we need to define text templates for our code somewhere. Good. By the way, this is called with a fancy word "model to text transformation". But wait, wouldn't it be much more convenient for us from the template definition point of view to have all the information about a class in one place? You know, like "this class has these attributes and these associations". Because right now they are separated (for a reason, as described before). It's not so good for this particular transformation...

But we can fix this: before dumping our model into the text files, there should be an additional step, where we copy data that is needed and in the way it is needed from the original model into an intermediary one. This is called "model to model transformation". It reminds me a lot of the role that query is playing in CQRS: we are just defining a new view on the existing data in such a way that is most convenient for what we want to do with this data in the next step.

Congratulations, this is already all the theory you need to know, in just two paragraphs! This was not so bad, wasn't it?

But enough of the theory, let's take a look at the real code next time.

Pragmatic MDD: Choose your weapons

So where do we want to start? What technology shall we use?

Let's decide about the model definition first. When you take a look around and see all the tools out there, you will notice, that 99% of them are from "Draw me some UML" kind. This is not bad - visualization is a magnificent thing, but this approach has some downsides.

First of all, you are limited in your options, since you are writing UML and UML only. Do you need something else? Nobody cares, just go and write some more UML.

Let me give you some examples, how the world could look like.

How about enriching your model with some other data? There might be a domain architect who is designing his business domain and absolutely doesn't care about anything else. He doesn't want to pollute his model with some annotations. Is it an embeddable entity? Is it a single table inheritance? Is the primary key on the table surrogate or natural? Or actually why should he care about any non-business related id at all?

Or how about defining your test data for different environments and scenarios? For example we might have a non-functional test setup to check some performance metrics. We could define a data setup (something like "5000 users, 1000 products", only the stuff we really need for the test) and not care about the rest - about all these mandatory fields, all these missing references, persistence of the data...

And what about business validations? Wouldn't it be nice to define them in some language specific to our domain? And then let the validations be part of our code, always executed, never forgotten, design by contract style?

I guess, you already know where I want to go with this - domain specific languages. But let's leave this idea for now and come back to it some happy day in the future, let's start with something much much simpler...

So how do we define our model if it's not UML? What is the most popular data format nowadays? It's JSON.

It also has a wonderful property of being supported by almost every programming language you choose, but by one in particular it is supported so good, that I just couldn't resist: JavaScript. But why stop at JavaScript, why not take its smaller brother instead: CoffeeScript. With it you can write the same code, but just in a much shorter way. And since I'm a very slow typer I consider it being a big plus.

But wait a second, wouldn't we need to generate files or something of that kind? Isn't JavaScript (or CoffeeScript, which is just compiled to JavaScript at the end of the day) executed in sandbox in the browser so we can't write files? You are right, of course, but Node.js comes to our rescue - it will play the role of an execution engine for us.

By the way, if you have never heard of Node.js, hate JavaScript or prefer XML (I heard, such people still exist..) - no problem, feel free to create your own stack! But if you like my choice, but just have no idea about e.g. Node.js or CoffeeScript - no worries, the code we write will be just sooooo simple. It practically can't get any simpler than that, trust me.

Pragmatic MDD: Introduction

Model driven development is incredibly useful. You actually have a model of your domain. You have a common language the business and IT can speak. Your developers can get rid of some of their monkey work writing boilerplate code. The changes in your model are propagated to all levels of the application architecture - front end, middle tier, database, documentation. You can be sure, that the generated code doesn't have bugs. But even if it has, you have to change it only in one place. The problem is, that there is also a price to pay for all of that.

Model driven development is hard. Or it actually can be easy, everything depends on your requirements. If you just want to have some fancy UML diagrams and generate POJOs/POCOs out of them, then you are in luck, this can be done quite easily. But what if you go one step further? What if, for example, you want to generate a DDL file to map your entities to the database using some ORM? Then you might still be fine with some out-of-the-box tool.

But let's make it a little bit more complicated: how about implementation of different inheritance strategies on the databaselevel? Or support of the business rules (which is a huge topic in itself, of course)? Or to have auto-update of the bidirectional associations between classes? Or something out of a completely different area - how about making it work and maintainable by more than one guy who has spent some years digging into this topic before?

Check out this list. How many modeling tools are already out there? And how many of them are not being developed anymore? The problem with out-of-the-box modeling software is that it can be of only 2 kinds:

• It is simple, easily understandable and - as a result - covers only the standard cases (unfortunately it is only a matter of time until you will have to implement a non-standard one and notice, that the chosen tool is not supporting it, but by that time you are already stuck with it)
• It supports all the transformations and templates you can only think of, all the formats and languages, but is so incredibly complex, that you will spend a lot of time building a working workflow for the current use case and you also will become the only one who can ever understand what you have built.

As I see it, the problem lies in variety of use cases itself - there are just so many of them and each one is a little bit different. So I want to try to go another way - to not build something that other people can use, but to show how simple it is for you to build something for your current use case by yourself - without using any of the fancy tools, but just a few helpful Javascript libraries (you don't have to use them by the way) any software engineer can easily understand.

I don't know what the result of this try-and-error project will be, but I am interested enough in its outcome to spend some time on it and see where it goes. I hope, you are too

.