MPS 2020.3 Help

Generator User Guide Demo6

Generator User Guide Demo 6

This demo is probably the most exciting one of all demos, because here we are going to create a 'real' language in the sense that this language will actually define a couple of higher-level concepts.
We will also see how easily this DSL can be integrated with existing languages - 'jetbrains.mps.sampleXML' and 'generator_demo.demoLang5' (created earlier in the Demo 5).

The core idea

We'll allow for a more convenient syntax for entering XML  Elements  in the test models. Developers will use new syntax for  button  and  label, which will be more compact and intuitive. At the same time, we'll be able to reuse the generator defined in the  demoLang5  language that we built in Demo 5. The new  demoLang6  language will translate the new concepts into XML  Elements, which then can be accepted by the  demoLang5  generator.

Gdf5

New Language

  • create new language 'generator_demo.demoLang6'

  • in language properties dialog add  extended languages  : 'jetbrains.mps.sampleXML' and 'jetbrains.mps.baseLanguage'

  • create a new generator for this language, if it does not exist (see Demo 1 for details)

  • in demoLang6  structure model  create new concept named 'Button' extending the concept 'ElementPart' (from 'jetbrains.mps.sampleXML')

  • in 'Button' concept declare a  property  named 'text' and set an  alias  (we need an alias to make auto-completion menu look nice):

Error1

  • create an editor for the 'Button' concept:

Gdf2

The editor consists of three cells: two constant-cells (shown in bold) and one property-cell (shown as {text }) which will render the actual value of the 'text' property.

  • create a similar concept named 'Label'

  • re-generate the language (Ctrl-F9)

New Test Model

Now let's try out our new DSL:

  • go to the 'test_models' solution

  • clone the 'test5' model into 'test6' (this time DO NOT replace  engage on generation language  
    generator_demo.demoLang5 -> generator_demo.demoLang6, since we still need the  demoLang5  generator to run with  test6)

  • add the 'generator_demo.demoLang6' language to the  used languages  section (new!), so we can use the  demoLang6  language for editing our model.

Gdf3

  • in the 'test6' model open the 'Panel' document and replace the two 'label' elements with our new  button  and  label  concepts:

Gdf6

With the new DSL our 'code' became clearer, shorter and less error prone. For instance, users now no longer can add elements inside labels or buttons.

Generator

We could generate methods and method calls out of  button  and  label  mimicking what has been done earlier for XML  Elements . But this isn't a good idea, because this way we would introduce indirect dependency between the  demoLang6  generator and the  demoLang5  generator (the  demoLang6  generator would have to know about the implementation details of the  demoLang5  generator).

Luckily, we have a far better option. As we are on a higher level of abstraction now (comparing to XML), we can simply reduce our semantically rich concepts into lower-level concepts (XML Elements). We don't care anymore about who and how will transform those lower-level (XML) concepts further into even lower-level concepts.

Reduction Rules

We'll employ reduction rules to convert our new concepts into XML  Elements. from the  sampleXML  language.

  • open the (empty)  mapping configuration  'main' in the demoLang6 generator

  • add a new  reduction rule  applicable to the  Button  concept

  • choose an  in-line template  as the rule consequence (Control + Space ):

Gdf7

The template is going to be pretty straightforward and it will not require any surrounding 'context'. That's why it is not necessary to create a full-blown external template as we did in other demos.

Since we want to use XML  Elements  from the 'jetbrains.mps.sampleXML' language in generator templates, we first have to declare that language among  Used Languages  by the  main@generator  model:

Gendemo61

  • back in the  main  mapping configuration choose  Element  as the template's content node:

Gdf9

  • create the 'button' XML element with a 'text' attribute

  • attach a property macro to the attribute value

  • enter the code in the macro's  value  function so that the value of the  text  property gets propagated into the XML button:

Gdf10

That's it. Our high-level button is being reduced into a 'button' XML element with a 'text' attribute. The XML element will presumably be further transformed down to Java by somebody else.

  • add a similar reduction rule applicable to the  Label  concept

First Test (Failure)

Re-build the generator and generate the 'test6' model.

There will be a bunch of error and warning messages in the MPS messages view.
Click on the 'was input node' error message. MPS will show you the node to, which the generator was trying to apply the rule that failed:

Demo6 error messages

This is a reduction rule in the demoLang5 generator (we are generating the 'test6' model with two generators: demoLang5 and demoLang6, remember?) and it has failed to find a static method declaration through the mapping label  method.

The  method  label is being assigned the generated static methods inside the  insert_Button  and  insert_Label  templates, so you may wonder what went wrong. Why couldn't we find the methods created inside the  DempApp  class and stored in the mapping label?

To answer that question let's take a closer look at the generation process as a whole.

As the first step, the generator creates a  transient model, which will serve as an  output model  for the current generation phase. This model name is 'test6@0'.

Then generator applies our  conditional root rule  and creates the 'DemoApp' class in output model.

Then, at some point, the  insert_Panel  template is invoked and its  LOOP -macro is executed (we can see the panel factory method containing the panel initialization code that we are familiar with from the  insert_Panel  template). The LOOP-macro creates a statement 'component.add()' for each child of 'panel' element and reduces the child to generate an actual argument to the 'component.add()' method call.

Our 'panel' element contains three children - a button, a label and a text, which has been turned into a label by a pre-processing generation script. Both the button and the first label are high-level abstractions introduced by  demoLang6  and their reduction yields corresponding XML  Elements. (We can see it clearly in the first screenshot, where the XML element representing a button is passed as an actual argument to the call to the 'component.add()' method. This is reported as a warning by the generator earlier up in the trace output in yellow color).

This ends the generation of 'test6@1_1' and the generator now creates a new transient model, named 'test6@1_2'. This model becomes a new  output model  and the former  output model  becomes a new  input model.

The generator starts to generate a new model 'test6@2_2' pretending that there was nothing before. The generator has no memory about previous activities with only one exception -  conditional root rules  are never applied twice. Thus, this time, the output 'DemoApp' class is created by copying of 'DemoApp' class from the input model, not by applying a  conditional root rule.

At this stage, the generator attempts to reduce further the XML  Elements  that represent the button and the label. The only rule available is the  reduce_Element  reduction rule from  demoLang5. It will attempt to replace the XML  Elements  with references to static factory method declarations, however, since none have been created, none can be found in the  method  mapping label. The method definitions were not created simply because the XML  Elements  were still represented as the  Button  and  Label   demoLang6  concepts at the time when the  LOOP  macro inside  DempApp  was run.

So this is a timing issue between the  demoLang5  and  demoLang6  generators. The latter should be made run first.

Now that we understand what is happening, the next question is how to fix it.

We are relying on the  demoLang5  generator and we know that the  demoLang5  generator works well providing that the input model is a 'valid XML' (we have tested it in Demo 5).
In Demo 6 our input model is not exactly a 'valid XML'. Moreover, we witnessed that the transient model 'test6@1_1' is not a valid model at all - a method call can't accept an XML element as argument (you can even find a warning message  "child 'Expression' is expected for role 'actualArgument' but was 'Element'"  in the MPS message view).

Now we could probably return to the  demoLang5  generator and make some 'improvements' to allow it to handle 'invalid' input models.
Fortunately, there is a better option - we can divide the generation process in two steps so that the original input model gets transformed into a 'valid XML' model first, and then this 'valid XML' model will be transformed to Java by  demoLang5  generator in a second step.

Dividing Generation Process into Steps

We will certainly get 'valid XML' model from the 'test6' input model if we reduce  button  and  label  (in the 'Panel' document) into XML  Elements . This transformation must happen before  demoLang5  starts transforming XML  Elements  into Java. In other words, the reduction rules specified in the  demoLang6  generator must be applied before any rules in the  demoLang5  generator.

Let's specify this constraint:

  • go to the generator in  generator_demo.demoLang6  and open the generator properties dialog

  • add an  extension  dependency on the  demoLang5  generator

    Gdf14

  • in  Generators priorities  tab add a desired priority rule

Demo6 generator priorities2

This will ensure the  demoLang6  generator runs before the  demoLang5  generator.

Second Test

Re-build the language and preview generated text for the 'test6' model. You should now run into no issues during generation and the generated code will correctly include the new button and label.

Gdf16

Saving Transient Models

Our  demoLang5  and  demoLang6  generators are working together well, because we have divided the generation process into two distinct steps.
In the first step the  demoLang6  generator reduces our high-level concepts (button and label) into corresponding XML elements and produces a 'valid XML' model as output.
In the second step the  demoLang5  generator generates Java from that 'valid XML' model.

Thus there should be at least one transient model between step 1 and step 2.
Let's take a look at what models have actually been created in the course of generation:

  • in  Project Settings  enable the  Save transient models on generation  option:

Screen shot 2017 07 24 at 11 08 30 pm

  • generate the 'test6' model

  • in the project tree find and expand the node named 'transient models' (this node is at the bottom)
    There are (surprisingly) five transient models.
    We can browse the transient models and we will find sometimes subtle and sometimes dramatic differences between them. This will give us a clue about what kind of transformation has taken place on each step.

For instance, open the 'Panel' node in model 'test@1_0' (this model has been generated in very first step).

Gdf18

We can see that high-level  button  and  label  concepts have been replaced with their XML counterparts, but the text 'Hello everybody!' is still there.

In the next model 'test6@2_0' the text 'Hello everybody!' has been replaced with a 'label' element (as a result of running the pre-processing script 'fix_text' in the  demoLang5  generator).

Model 'test6@2_1' contains the generated  DemoApp  class.

And model 'test6@2_3' contains the same class, but the string "MPS!" is replaced with "JetBrains MPS!" (as a result of the post-processing script 'refine_text' in the  demoLang5  generator).

Root Nodes Copying and Reduction Rules

As we can see, the model 'test6@1_0' is identical to the original input model 'test6', except for high-level button and label concepts that have been reduced to XML elements.
This is what we wanted, but it is probably still not clear, how roots in 'test6@1_0' have been created (we don't have any 'root rules' that would create a  Document) and why our reduction rules have been applied.
From our previous experience with reduction rules we remember that we had to create a  COPY_SRC _macro for reduction to take place. But in the _demoLang6  generator we don't have any _COPY_SRC _macros, do we?

This way the input  Documents  (Button, Label and Panel) from 'test6' have been copied to model 'test6@1_0' and reduction has been applied to the  button  and  label  concepts that were inside the 'Panel'  Document.

Returning to the _COPY_SRC _macro - it doesn't 'invoke' a reduction (as it might seem). Instead it merely applies the same copying procedure to the  mapped node  and, if any reduction rules happen to be applicable, then the transformation occurs.

Using Generation Tracer Tool

We already know how to save and browse transient models. Actually, with the option to  Save transient models on generation  activated, MPS not only saves transient models, but also collects a great deal of data regarding the process of transformation.
This data can be viewed using the  Generation Tracer Tool.

For instance, open the 'Panel'  Document  in the transient model 'test6@1_0', select 'label' and choose  Show Generation Traceback  in the popup menu.

Gdf19

The Generation Tracer View will be opened.

Gdf22

In the root of this tree there is the 'label' node, for which we have requested the traceback info (the blue arrow denotes an  output node) and the rest of the tree shows the sequence of events, in reversed order, that led to this output.

Looking at this tree and clicking on its nodes (in order to open them in the editor) we can reproduce the process of transformation to a great level of detail.

For instance, we can see that the output 'label' is created by the template in the  reduce Label  rule.

Gdf20

... and the  reduce Label  rule has been applied to the  input node  'Label' while the 'Panel'  Document  (the root input node) was being copied.

Gdf21

Last modified: 26 February 2021