M interop with XML (and others)

Someone asked a great question on the MSDN forums - 

looking at the current bits i can see a transform from M to XML.
But what seems to be missing is a transform from (XSD+XML) => M Schema + M
Any plans for this?
I think you need to support this scenario as most metadata (or models as you call them)
is now encoded as XML

I like to think of M interop with other data formats in two very interesting and different scenarios. 

One is exactly what Tim asked about - how do we take M data (which we call MGraph) and interoperate with other data formats, such as XML? Truthfully the mapping is pretty straight forward and we've prototyped it enough and have an initial implementation. It needs improved engineering before we declare ship ready, but there you have it.

We also think about interop at the schema level. For example, if we transform an MGraph to XML, then wouldn't it be great to transform the representative schema for that data into XSD? That's something we haven't quite done yet, but again I don't think that it would be hard, just work. We don't really have that booked to implement right now. Do people see a huge need?

Also, there's interop at the grammar level. This is probably not something we talk about much because there you really need to consider functional mapping. In other words, a grammar is just a function from character* to structured data (MGraph). Mapping that from one language like M to another like XSLT is quite a complicated problem and not likely something we'll tackle outside of research interests.

Now, there's another really big set of scenarios, quite orthogonal to the external interop ones above. I call these internal interop, or how to enable embedded DSLs or user-defined data structure and types within M. Sounds quite mystical and abstract :)

It means a couple of things. First, what if you wanted to write your own syntax for values inside of an M program. For example, suppose we didn't have a GUID type nor a textual representation for GUID values. Wouldn't it be great if you could tell M - "hey this is my syntax for my data"? We're thinking about that scenario, but nothing is committed to implementation yet. 

Second, what if you could write your own DSL inside of M that would produce the equivalent of M program constructs, such as types and functions. This is what we call 'meta-programming'. We have lots of dreams to do this, and we're working towards that, but it's most likely a V2 priority not a V1 priority.

As always - I'd love your feedback on this.

Ewald on Dynamic languages

Great post by Tim on why you should learn a dynamic language.

Do people think M is a dynamic language?

What's the purpose of types?

Do programmers always grok the purpose of types (nominal or structural)?

What if I didn't use the word 'type'?

What if I used the word 'pattern'? Or 'constraint'? Or 'rule'?

What if I used the word 'allocator'? Or 'constructor'? 

What does it mean to you?

Structural versus Nominal typing

M has a structural type system. Most programming languages today, and in particular most OO languages, have a strict nominal typing system.

Given that types are used in 2 ways (validation/constraints and value construction), how do programmers generally feel about one versus the other?

Do programmers need both? In which case is one better than the other? 

Do programmers really understand both?

I'd love your thoughts.

M Past and Present

Let's talk a little about M's past and present.

 

At PDC 08, we introduced M as a way to write down data (Mgraph), schema and functions about that structured data (Mschema), and character parsing into structured data (Mgrammar).

 

Frankly, at PDC, we tried to tell a '1 language with 3 parts' story, but in my opinion it was more like a '2 languages with similar concepts and marketing terms'. Truth be told, those technologies kinda grew up separate.

 

It doesn't matter. We showed the world some interesting ideas, and we saw how powerful it could be to bring the pieces together.

 

We've been working hard at that - at integrating the languages. As I type, some of the smartest folks I know are busy rebuilding the internals of the compiler. We hope to be able to bring all of the parts together as close as we can before the next PDC. I can't wait to illustrate some of the key customer scenarios we expect to get with that integration.

 

Our first step to integrate the language will be visible to everyone in our next CTP drop. We deleted mg.exe, the compiler tool for Mgrammar. Now, you can use a single executable and API to compile Mschema or Mgrammar programs. You still have to have separate file extensions (.mg for Mgrammar, .m for Mschema).

 

We almost deleted mgx.exe as well, but we weren't able to get all of that work done.

 

Now - what happens after we integrate the language? I'll talk more about that in the future. And, I'll talk about other innovations we'll be seeing in the language - some of them before PDC 09, but maybe not until after.

 

Stay tuned.

"Reflection" in M

Wouldn't it be great if you could use reflection to inspect your M program, and add additional data to it. For example, suppose you wanted to add additional metadata to a type declaration, or a computed value.

In the upcoming release of the Oslo CTP, we introduce a new thing we call the M catalog, along with an operator in M called 'about'. 

The M catalog is simply a structured representation of an M program, or what we call the M semantic graph. The schema for the catalog is written in M, and the compiler can generate instances of your M program into that schema. Then it just shows up in the database using mx.exe.

 

Right now, the way to create an image with the M catalog schema is to run m.exe with the 'catalog definition' parameter, or 'catdef' for short. 

 

m.exe -catdef -out:catalog.mx

 

This creates an image that contains extent declarations for things like Modules, Types, Extents, and Computed Values. The catalog also has various helper computed values for querying the catalog. For example, you can ask if a particular computed value is installed (IsComputedValueInstalled). Or, you can ask if a type is an intrinsic, collection or entity type (IsIntrinsicType, IsCollectionType, IsEntityType). Or, one of my favorite, you can ask for all of the references to an extent ('ReferencesToExtent').

 

For now, we don't automatically generate instances for a given M program. You have to explicitly ask for it, and you have to reference the catalog definition generated above. We hope to fix all of that in the next milestone and turn it on by default. Here's a simple example for how to create catalog instances for your M:

 

m.exe myM.m -catalog /r:catalog.mx

 

When you run mx with myM.mx and catalog.mx, the M catalog will show up in the database. After you load, take a look at the database. In particular, one super cool thing is that the catalog also maintains the relationship between the SQL created from the M and their corresponding M declarations. For example, take a look at Tables, Functions, and Views in the Language.Catalog.SQL namespace.

 

That's a brief intro to the M catalog. There will be lots of scenarios coming where we use the catalog to query, analyze and understand M.

 

What if you want to access the catalog from your M program? That's easy enough because the catalog is written in M. So, you can reference and query the extents directly, or call one of the computed values.

 

The coolest reason to do this is to add additional data to your M program. This is what some people call metadata - although I'm not very fond of that term (it's all just data). Let's start with an example.

 

Let's start with one of our classic Oslo scenario models, the WIX model. Let's suppose that I am building a super-duper add-in to WIX that will deploy M to a database. The problem is that the WIX model does not include database deployment information. I could update the WIX model to include this, but that's not a great extensibility story for our developers. Instead, I will build my own little model for my specialization, I will reference M content in the catalog, and I will use about to write instances that reference specific values in the catalog.

 

Here's a simple version of my model. It describes additional information that I want to associate with a module in M:

 

module MDeployer

{

    import Language.Catalog;

   

    DatabaseConfigurations : ({

        Id : Integer32 => AutoNumber;

        DatabaseName : Text;

        ServerName : Text;

        Module : Modules;

    } where identity Id, value.Module in Modules)*;

}

 

Now, I want to write a deployment package the says that a particular M module is deployed to a particular server and database. Here's an example, except what do I write to specify the module name?

 

module DeploymentPackage

{

    import MDeployer;

    import MyModel;

   

    MDeployer::DatabaseConfigurations

    {

        { DatabaseName => "Testdatabase", ServerName => "TestServer", Module => ... }

    }

}

 

You could write a query, for example, 'Module => (Modules where value.Name == "MyModel")'. That's not always easy to remember. What's worse is that it is not strongly typed - what if I misspell "MyModel". I wouldn't find out about that until loading into the database. What I really want is a compile-time check that I have referenced the right M declaration.

 

'about' is the magic to make this easy. 'about' is a function that does what the query does, but it's strongly typed and uses the actual identifiers of the M declaration that you want to reference. Here it is:

 

module DeploymentPackage

{

    import MDeployer;

    import MyModel;

   

    MDeployer::Databases

    {

        {

DatabaseName => "Testdatabase",

ServerName => "TestServer",

Module =>  about(MyModel)

   }

    }

}

 

In summary, about allows you to add additional data to your M that is about your M. It references that content, and when loaded to the database, references your M program that is represented in the M catalog.

New stuff coming

I have not been very chatty over the last few months. We've been working very hard to rev a new version of the Oslo CTP. It should be out in the next few weeks.

So, you'll be hearing more from me. I have a number of detailed topics I want to share including new language and toolchain features such as:

i. single m.exe

ii. cycle initialization

iii. computed values in initializers

iv. extern

v. catalog and about

vi. Catalog inferrer

vii. Repository patterns via M modeling

i. sequence ID

ii. Folder

iii. Security views

viii. mx command line options, including Folder

ix. New mgraph API

Also - I suppose I should continue on with my detailed discussion of how M query expressions work.

It should be a fun couple of blogging weeks. Stay tuned.

DSLCon was awesome

I attended Sells' DSLCon a few weeks ago - just now blogging about it.

Fowler's keynote was awesome.  Modeling-is-programming is a key message for Oslo/M. It's our differentiation from other kinds of modeling, e.g., UML. His discussion about DSLs and semantic models to represent domains which can then be executed is spot on.

If you really want to understand Oslo and M, check out Fowler's work on DSLs.

Some of the conference was a bit too much powerpointy for me - I would prefer more code. But all-in-all a great event.

Thank you sells! You are the man.