Hanging out at PDC 09

Hey - I'm at PDC 09. Stop by the booth if you want to chat about all things "M".

"Reflection" in M

Wouldn't it be great if you could use reflection to inspect your M program, and add additional data to it. For example, suppose you wanted to add additional metadata to a type declaration, or a computed value.

In the upcoming release of the Oslo CTP, we introduce a new thing we call the M catalog, along with an operator in M called 'about'. 

The M catalog is simply a structured representation of an M program, or what we call the M semantic graph. The schema for the catalog is written in M, and the compiler can generate instances of your M program into that schema. Then it just shows up in the database using mx.exe.

 

Right now, the way to create an image with the M catalog schema is to run m.exe with the 'catalog definition' parameter, or 'catdef' for short. 

 

m.exe -catdef -out:catalog.mx

 

This creates an image that contains extent declarations for things like Modules, Types, Extents, and Computed Values. The catalog also has various helper computed values for querying the catalog. For example, you can ask if a particular computed value is installed (IsComputedValueInstalled). Or, you can ask if a type is an intrinsic, collection or entity type (IsIntrinsicType, IsCollectionType, IsEntityType). Or, one of my favorite, you can ask for all of the references to an extent ('ReferencesToExtent').

 

For now, we don't automatically generate instances for a given M program. You have to explicitly ask for it, and you have to reference the catalog definition generated above. We hope to fix all of that in the next milestone and turn it on by default. Here's a simple example for how to create catalog instances for your M:

 

m.exe myM.m -catalog /r:catalog.mx

 

When you run mx with myM.mx and catalog.mx, the M catalog will show up in the database. After you load, take a look at the database. In particular, one super cool thing is that the catalog also maintains the relationship between the SQL created from the M and their corresponding M declarations. For example, take a look at Tables, Functions, and Views in the Language.Catalog.SQL namespace.

 

That's a brief intro to the M catalog. There will be lots of scenarios coming where we use the catalog to query, analyze and understand M.

 

What if you want to access the catalog from your M program? That's easy enough because the catalog is written in M. So, you can reference and query the extents directly, or call one of the computed values.

 

The coolest reason to do this is to add additional data to your M program. This is what some people call metadata - although I'm not very fond of that term (it's all just data). Let's start with an example.

 

Let's start with one of our classic Oslo scenario models, the WIX model. Let's suppose that I am building a super-duper add-in to WIX that will deploy M to a database. The problem is that the WIX model does not include database deployment information. I could update the WIX model to include this, but that's not a great extensibility story for our developers. Instead, I will build my own little model for my specialization, I will reference M content in the catalog, and I will use about to write instances that reference specific values in the catalog.

 

Here's a simple version of my model. It describes additional information that I want to associate with a module in M:

 

module MDeployer

{

    import Language.Catalog;

   

    DatabaseConfigurations : ({

        Id : Integer32 => AutoNumber;

        DatabaseName : Text;

        ServerName : Text;

        Module : Modules;

    } where identity Id, value.Module in Modules)*;

}

 

Now, I want to write a deployment package the says that a particular M module is deployed to a particular server and database. Here's an example, except what do I write to specify the module name?

 

module DeploymentPackage

{

    import MDeployer;

    import MyModel;

   

    MDeployer::DatabaseConfigurations

    {

        { DatabaseName => "Testdatabase", ServerName => "TestServer", Module => ... }

    }

}

 

You could write a query, for example, 'Module => (Modules where value.Name == "MyModel")'. That's not always easy to remember. What's worse is that it is not strongly typed - what if I misspell "MyModel". I wouldn't find out about that until loading into the database. What I really want is a compile-time check that I have referenced the right M declaration.

 

'about' is the magic to make this easy. 'about' is a function that does what the query does, but it's strongly typed and uses the actual identifiers of the M declaration that you want to reference. Here it is:

 

module DeploymentPackage

{

    import MDeployer;

    import MyModel;

   

    MDeployer::Databases

    {

        {

DatabaseName => "Testdatabase",

ServerName => "TestServer",

Module =>  about(MyModel)

   }

    }

}

 

In summary, about allows you to add additional data to your M that is about your M. It references that content, and when loaded to the database, references your M program that is represented in the M catalog.

New stuff coming

I have not been very chatty over the last few months. We've been working very hard to rev a new version of the Oslo CTP. It should be out in the next few weeks.

So, you'll be hearing more from me. I have a number of detailed topics I want to share including new language and toolchain features such as:

i. single m.exe

ii. cycle initialization

iii. computed values in initializers

iv. extern

v. catalog and about

vi. Catalog inferrer

vii. Repository patterns via M modeling

i. sequence ID

ii. Folder

iii. Security views

viii. mx command line options, including Folder

ix. New mgraph API

Also - I suppose I should continue on with my detailed discussion of how M query expressions work.

It should be a fun couple of blogging weeks. Stay tuned.

DSLCon was awesome

I attended Sells' DSLCon a few weeks ago - just now blogging about it.

Fowler's keynote was awesome.  Modeling-is-programming is a key message for Oslo/M. It's our differentiation from other kinds of modeling, e.g., UML. His discussion about DSLs and semantic models to represent domains which can then be executed is spot on.

If you really want to understand Oslo and M, check out Fowler's work on DSLs.

Some of the conference was a bit too much powerpointy for me - I would prefer more code. But all-in-all a great event.

Thank you sells! You are the man.

M Data Transformation Part 1

Lots of blogs and content on M spend a bunch of time focused on the modeling and DSL aspects of M. And, lots of folks always ask about data transform. So, I'm going to spend some time on transformation. Clearly, if you're working on a data-oriented platform, transformation is a key enabler.  

Let's start with a couple of principles that M transforms live by:

- Functional. functional programming is the right paradigm for writing transformations because they are compositional and side-effect free

- Compositional. A corollary of functional, building transforms on top of transforms is powerful and enables reuse. It also means that clients/consumers do the same thing regardless of whether they are consuming a graph or a transform over a graph.

- Consistent. Queries are expressions that produce new values. Constraints are also expressions. We wanted the query language to be consistent withe the constraint language.

- Familiar. The syntax should be familiar to folks already writing transforms in T-SQL or in LINQ

- Ease. There are a number of shorthand forms for queries that make writing transforms even easier

OK - let's write some transforms. I'm going to use a very simple data model for my examples. Here's a model for Contacts (aka Outlook):

module Contacts

{

    export People, Addresses, Zips;

    People : 

    {

        Name : Text#128;

        Age : Integer32;

        MyAddresses : Addresses*;

    }* where identity(Name);

    

    Addresses : 

    {

        Id : Integer32 = AutoNumber();

        Street : Text;

        Country : Text;

        Zip : Zips;

    }* where identity(Id);

    Zips : Integer32* { 98052, 44114, 44115};

}

Here are a couple of queries for the projections, i.e., selecting values from a collection. Notice that there is a long syntax and a comprehension syntax that uses value.  

module CollectionQueries

{

    import Contacts;

    

    Q1()

    {

        from z in Zips

        select z

    }

    

    Q2()

    {

        Zips select value

    }

}

These are exactly equivalent. Check out the generated SQL:

create view [Queries].[Q1]

(

  [Item]

)

as

  select [z].[Item] as [Item]

  from [Contacts].[Zips] as [z];

go

create view [Queries].[Q2]

(

  [Item]

)

as

  select [$value].[Item] as [Item]

  from [Contacts].[Zips] as [$value];

go

Now, let's write some projections using entity collections. 

module EntityQueries

{

    import Contacts;

    

    Q1()

    {

        from p in People

        select p.Age

    }

    

    Q2()

    {

        People select value.Age

    }

    

    

    Q3()

    {

        People.Age

    }

        

}

Again, we have a full query syntax version and a comprehension form. There's also a 3rd syntax called a projector. It returns the same results, but is written more like a function of the field name.

That's some very basics around projection. Stay tuned for posts on more complex projections, plus selection, join, and other interesting query language features.

PS. if you want to see lots of examples, check out the set of M sample queries in the SDK. We wrote all of the LINQ samples in M so you can compare.

Enjoy!

Why oslo remix

This post is awesome!

Metadata or data

I get quite irritated (sorry - no patience) when I hear others talk of a very distinct difference between metadata and transactional data. I really don't agree.

The arguments generally go something like this: "Metadata is mostly read-only. Transactional data is written much more frequently. Metadata has different access patterns -- I don't even know what that means :).  

I find that to be hogwash. That describes usage not kinds of data. I do not like categorizing data. It's like nominal typing - limits its broader viability and usability after the fact. Any data at any given time can be more like metadata or more like transactional data. For example:

- To an engineer, bill of materials is transactional data when designing a product. But, to a resource planner, the bill of materials is metadata that drives materials planning, purchasing and manufacturing scheduling. 

- A web page is transactional data during development, but metadata at runtime (unless it is self-modifying code) 

So, it just depends on the usage. Don't categorize the data, just understand the usage.

As for Oslo, I assert thatwe are building a broad set of capabilities to describe, validate, transform, access, and store data. Sure, Oslo's primary scenarios and our investments right now are targeted at data that describes runtimes. However, our ambitions are bigger, and our architecture and designs not limited or miopic in our thinking. If they are - please help us. After all, data is just data.

M == Semantic Model + DSL + values

I want to make sure I'm clear wrt my post from the other day about Fowler's work. Here's a simple formula to translate M concepts into Fowler speak:

MGrammar == DSL

MSchema == semantic model

MGraph == values of DSL/semantic model

Foundational work on DSLs

Over the holidays I've been doing lots of reading and coding. It's been great.

I re-read Martin Fowler's writing on language workbenches. If you want a well written, crisp definition of language oriented programming, read it. Or, if you're struggling with what Oslo is all about, read it. Martin's work is foundational at describing what I believeto be  the future of programming and application building - what some have called language-oriented programming.

Oslo is here to enable this vision. Our language work brings a simple, easy-to-use notation for writing the abstract model that Martin talks about; we call that part of the language MSchema. Our language also enables the writing of DSL grammars over that model; we call that MGrammar.

Pay attention to fact that we do not talk about them as different languages - but as One. That's a key concept  that has yet to be exploited by our blogging community and by us, partly because we haven't yet done the work to integrate MSchema with MGrammar. But just wait. That's our longer term goal - to integrate both abstract model and DSL definition languages making Martin's inspiration a reality in a very easy to use way.

Add in a great tool like Quadrant to do textual and visual editing of both model and DSL, and the horizon of that vision is much closer than you think.

We'll be talking more about the integration of the language in the coming months.

DSL for Banking - interactive DSLs

A few weeks ago I posted an M model for a banking transactions. I forgot to post the DSL.

 

Here's a simple DSL that would be quite easy to use. And, it only took me 10 minutes to write it.

module Banking

{

    language BankingLanguage

    {

        interleave Ignorable = ' ' | '\n' | '\t' | '\r';

        

        syntax Main = t:Transaction+

            => Transactions { t };

            

        syntax Transaction =    

            t:TransactionKind 

                a:Amount 

                Transitions 

                Account 

                AccountNumber => { Kind { t }, Amount {a }};

        token Account = "Account";

        token Transitions = "from" | "to";

        token AccountNumber = Digit+;

        token TransactionKind = "Deposit" | "Withdrawal";

        token Amount = CurrencyQualifier? DecimalNumber;

        token CurrencyQualifier = "$";

        token Digit = '0'..'9';

        token DigitWithoutZero = Digit - '0';

        token DecimalNumber = DigitWithoutZero Digit* '.'? Digit+;

    }

}

Here's some sample input:

Withdrawal 200.00 from Account 5

Deposit $3039483.00 to Account 4932

Now, SpankyJ and I were talking (IM'ing actually).

Wouldn't it be cool if you could write a grammar that defining an iteractive textual experience that you could then deploy to a Intellipad session?

Something like this, where you indicate that the syntax is interactive causing the parser to be re-executed with each parse episode termined by the named token, in this case "loop".

         @{Interactive[loop]}

        syntax Main = t:Transaction+

            => Transactions { t };

            

        syntax Transaction =    

            t:TransactionKind 

                a:Amount 

                Transitions 

                Account 

                AccountNumber loop => { Kind { t }, Amount {a }};

        token loop = "\n" | "\r" | "\r\n";

Then, you could have an iterative, transaction-like experience with this particular DSL where people just type in transactions, and get immediate feedback. 

I suppose you'd want the runtime to be hooked up as well... Hmmm - just thinking out loud :)

Oslo - is that all it is?

I enjoyed Jeremy's post about Oslo.

In the comments, there were lots of follow-up questions about the realness and substance of Oslo, and the target customers of the language. Here's some reflections from one Oslo guy...

What we shipped in our first CTP is very much an early alpha of our bits. And, it represents the lowest level of the platform without many of the services, libraries, and bells-whistles that you'd want as a developer. It's as if we shipped the .Net runtime and C# but without any libraries or tools. So, that may be some cause of confusion.

Why did we do this? We wanted to get out the core of the platform as soon as possible so we could start having real discussions with customers and partners. We also want the developer community to embrace modeling as a key principle in the future for how we build applications.

Ultimately, our goal is to enable declarative, model-driven programming. If I look around, I see people doing this today in the form of XML schemas and dialects, various textual reps, and frameworks that encode a domain. We went down that path as well, using visual designers and XML. But at some point the pain was too much :) We evolved our approach into Oslo by trying a more holistic solution. We wanted an easy-to-use language to write down declarations. We wanted a place to store those declarations, and then query, and even compose that data (i.e., join) into "instructions" that runtimes could execute to produce interesting behavior. And then, the idea of making it very easy to create a textual DSL over the model/declarations seemed like a natural step forward to enable app development in an elegant way. It just all fit together for us - and we wanted to share as soon as we could.

Let me also comment about our target customers. We are all developers on our team, and we care deeply about developer tools and productivity. We do not really talk much about making business users programmers. I have to say that personally, I think having a DSL may make it easier to have interesting conversations with business users about the solutions that we build for them. But I don't think that makes them programmers. It just makes it easy to write things down and share/design it with them.

...end of reflecting for now.

API for MGraph

In M we talk about 3 pieces of technology: MGraph is the data model, MSchema is the type system, and MGrammar is the transformation engine.

MGraph is at the core of all of these. MGraph is a labeled directed graph as defined in the System.Dataflow assembly. Interesting enough, rather than force you to implement a class or interface, we designed MGraph to follow a visitor pattern, like a simplified XML navigator. So, you keep or build whatever data structures you have, and then implement IGraphBuilder to expose those values as an MGraph.

Here is IGraphBuilder:

public interface IGraphBuilder

{

IEqualityComparer<object> NodeComparer { get; }

object DefineNode(object label);

void DefineSuccessors(object protoNode, IEnumerable<object> successors);

object GetLabel(object node);

IEnumerable<object> GetSuccessors(object node);

bool IsNode(object value);

}

I'm only going to talk about the navigation part of IGraphBuilder, not the construction protocol.

A node in the graph is identified by your implementation of GraphBuilder by calling IsNode. If this returns true, then the GetLabel and GetSuccessors methods must accept the same object and produce the label and successors, respectively. The Label is optional.

I recently experimented with building an MGraph over an existing SQL database. I defined a SQLGraphBuilder to take a connection and the name of a SQL object, i.e., table or view.

The SQLGraphBuilder queries the database using that name. I constructs a node for the SQL object with the label being the name of the sql object, and each successor being a row in that object. In turn, each row has a label of the primary key (concatenated if there are multiple) followed by each successor, which is itself a node that has the field name as the label and 1 successor being the value.

Here's the base definition for a node in the graph:

abstract class Node : IEquatable<Node>

{

protected string label;

protected SqlDataReader reader;

public Node(string label, SqlDataReader reader)

{

this.label = label;

this.reader = reader;

}

public string Label { get { return label; } }

public SqlDataReader Reader { get { return reader; } }

public abstract IEnumerable<object> GetSuccessors();

public abstract bool Equals(Node other);

}

So then I have a subtype of this for each kind (SqlObject, Row, and Cell). Here's an example:

class SqlObjectNode : Node

{

public SqlObjectNode(string sqlObjectName, SqlDataReader reader) : base(sqlObjectName, reader)

{

}

public override IEnumerable

{

return new Enumerable(new SqlObjectEnumerator(label, reader));

}

public override bool Equals(Node other)

{

var sqlNode = other as SqlObjectNode;

return sqlNode != null

&& sqlNode.label == this.label;

}

}

This is all quite easy except for the definition of identity via the node comparer. You need to make sure that you think through the identity story so that you can easily identity equivalent graph nodes. In my prototype, I use a concatenation of sqlobject + row id + column name. See the Equals method above as an example.

Now here's the tricky part. Some columns are actually foreign keys. So, I maintain a general foreign key lookup table, and when I hit a column name that is actually a foreign key, in stead of returning a value for that cell, I return a new row representing the values from the related table.

Here's the query to find foreign keys:

const string ForeignKeyQuery =

@"select ssource.name as SourceSchemaName,

tsource.name as SourceTableName,

csource.name as SourceColumnName,

fk.name as ForeignKeyName,

starget.name as TargetSchemaName,

ttarget.name as TargetTableName,

ctarget.name as TargetColumnName

from sys.foreign_keys fk

inner join sys.foreign_key_columns fkc on fk.object_id = fkc.constraint_object_id

inner join sys.tables tsource on fkc.parent_object_id = tsource.object_id

inner join sys.tables ttarget on fkc.referenced_object_id = ttarget.object_id

inner join sys.columns csource on (fkc.parent_column_id = csource.column_id and csource.object_id = tsource.object_id)

inner join sys.columns ctarget on (fkc.referenced_column_id = ctarget.column_id and ctarget.object_id = ttarget.object_id)

inner join sys.schemas ssource on ssource.schema_id = tsource.schema_id

inner join sys.schemas starget on starget.schema_id = ttarget.schema_id

and fk.type = 'F'

order by SourceSchemaName, SourceTableName, ForeignKeyName";

And here's the code to construct the row in the reference table. Notice I'm still working on it :)

StringBuilder query = new StringBuilder(@"select * from " + fk.TargetSqlObjectName + " where ");

int i = 0;

foreach (var targetCol in fk.TargetColumnNames)

{

if (i > 0) { query.Append(" and "); }

query.Append(targetCol + " = @var" + i);

i++;

}

i = 0;

//TODO: fix this hack to support non MARs connections; need to dispose connection + reader

var newConnection = new SqlConnection(connection.ConnectionString);

newConnection.Open();

using (var command = new SqlCommand(query.ToString(), newConnection))

{

foreach (var sourceCol in fk.SourceColumnNames)

{

object foo = reader[sourceCol];

command.Parameters.AddWithValue("@var" + i, foo);

i++;

}

var newReader = command.ExecuteReader();

return new ForeignKeyCellNode(fk.TargetSqlObjectName, rowName, fieldName, newReader);

}

GraphBuilder is fairly straightforward to use. It and Mgraphs are at the base of the modeling platform, so learning it now will only help as we introduce more tools and APIs to consume them.

Louis

I'm at PDC with a great team, in a nice place, at a great hotel (I love the beds at Westin).

 

I was watching Louis CK last night (shh - don't tell me wife. She wouldn't like him).

 

It dawned on me that I have "Louis". My "Louis" is just like the CK version. He is super bright, magnanimous, and serious yet funny in the same way.

 

He says what he thinks, which I love. If there's one thing that kills a team culture, it is ulterior motives or backdoor communications. If "Louis" has them, you either would not know, or agree that he's right.

 

That's one reason I love my team. We say what we think. No one is offended. Everyone wants to do the right thing. And we all just get along.

 

And that's "Louis". He's also able to inspire millions, mentor our team, and make a huge impact/influence on our technologies. Yet he (and us with him) have so much fun doing it. 

Self-funded eval of Oslo's M language

I think this is the first self-funded evaluation of M that I've found on the web.  Great job William.

This is my favorite:

Except for this caveat, I like “M”. It’s not anti-XML (you can represent values as XML if you’d like) but it avoids the “the answer is XML/XSD what is the question” approach to modeling that is sometimes a little too prevalent. “M” is a much better schema language for IT systems than XSD. 

Simple Banking Transaction Model in M

At PDC I meet Alejandro. He and I worked through a very simple example of a bank transaction model.

Here's a warning. Not everything writen here is currently supported by the CTP compiler, such as enumerated values for Transaction.Kind. But you'll get an idea of what we would like to support by the time we ship.

Here's the model. Check out the computed value to test that an instance is in fact a valid Transaction. It uses type ascription to test that t adheres to the shape and constraints of the Transaction type.

Also notice the computed value, ChildrenTransactions, to determine the transactions that are children of another transaction. It uses a shorter syntax of LINQ-style queries that we call compact query syntax. I put the full LINQ syntax in quotes so you can compare.

Finally, notice the constraint to validate that Transaction.Target has been set when the Kind is a "Transfer".

module Banking 

{

    Transactions : Transaction* 

where item.Source in Accounts, 

item.Target in Accounts;

    Accounts : Account*;

    

    ValidTransactionGoodness (t : Entity)

    {

        t : Transaction

    }

    ChildrenTransactions(t : Transaction)

    {

        Transactions where value.Parent == t

    }

    

     type Account

    {

        Id : Integer64 = AutoNumber();

    }  where identity Id;

    type Transaction

    {

        Id : Integer64 = AutoNumber();

        Parent : Transaction?;

        Kind : { "Transfer", "Deposit", "Withdrawal"};

        Source : Account;

        Target : Account?;

        Amount : Decimal28;

    }  where identity Id, 

Kind == "Transfer" ? Target != null : Target == null;

}

wonder twin powers activate

I'm a superhero fan. Not really into reading the comics, but definitely watching the Saturday morning cartoons :)

 

I keep gloating over my team. They are awesome. Well we have a couple of folks that I'll call the Wonder Twins, aka, Zan and Jayna (btw - I did not knew their name before today). Individually, each is an awesome person with great talents. Put them together, and BAM, watch out. They have an amazing multiplier greater than 1+1. I enjoy watching them work. I am often jealous.

 

Having wonder twins is wonder-ful. We have a hard problem, wonder twin powers activate. We need some new bits as soon as possible, wonder twin powers activate.

 

In all seriousness, Zan and Jayna are good friends, and great team members. Yet another reason I love my team!

DSL Hell

Everyone remembers DLL hell. You know -- run my app but there's 15 versions/copies of a DLL. So which one do I choose.  That was hell.

 

Guess what? I hate to tell you, but we are in DSL hell. There's XML dialects all over the place. Those are DSLs (for Mrs. Pinky, that's domain specific languages). And so we have lots of developers having to learn different XML dialects just to work with different runtimes.

 

And, APIs themselves are really just DSLs. You have to know which objects to instantiate, which methods to call, and in what order.

 

So - we're in DSL hell. And it's an ugly hell. Who likes writing code in XML? (besides Don Box)

 

Oslo, and M, are about to change the game. We want developers to build and use natural, textual languages. Imagine users configuring and running applications by writing simple text in a simple language targeted at their domain. Imagine writing a WCF service in 10s lines of code instead of 100s lines of C# + 20 lines of config + ... Imagine a standard language for business processes, such as Purchase Order Processing. Imagine imagine imagine...

One more thing. As a developer, I am pretty sure you will have more fun writing a DSL with Oslo than most programming you do today. I haven't had this much fun in years!

So this is Oslo. Check it out: http://msdn.microsoft.com/oslo

Geeks interacting with Models

We're announcing Oslo to the world as I type.

Someone leaned over and said, "Geeks having been looking to find tools to interact with Models for a long time..."

I don't think she ment Oslo models :)

"the brain"

If I'm pinky, then I'm sure everyone is asking, "where's the brain?"

Microsoft has lots of brains. They're all probably trying to take over the world. Or trying to explain the brain to us mere mortals.

But pure intelligence does not bring about revolution. Most of the time brain people are over confident, arrogant, and un-influential (except to threaten to hurt the Pinky).

My brain is different. "Brain" is very smart. He has answers to just about anything. And he's usually right... usually :). But the thing that differentiates my "Brain", and that makes my team so awesome, is that "Brain" is a nice person. "Brain" will listen intently on what you have to say, explain what you don't understand, and do it again if needed. "Brain" empathizes, cares about people, and helps everyone on the team. Brain is taking over the world 1 person at a time.

"Brain" is one of the reasons I love my team.

My team a' la hide-n-seek

I'm sure anyone that follows MS has seen mini-microsoft. Here's an insider hiding his/her identity to talk about Microsoft.

Well - I'm not hiding anything about me (just ask ;). But, I would love to tell you about my team at Microsoft. And yet, I don't want them to be too upset with me. So, I'll inverse mini and tell you about them but with interesting code names/aliases so as to hide the innocent.

BTW - don't expect dirty. I love these folks. They are the best - to work with, to hang with, to build cool stuff with.

My team rocks!