Wednesday, November 12, 2008

General George C Marshall

About 6 months ago I read a book about General Marshall. To be honest, I didn't realize the magnitude of the man and the role that he played in my freedom.

Someone on my team (yet to be identified) and I were talking, and it dawned on me today. We have a General Marshall. My General Marshall is one of the reasons I came to this division, and one of the reasons I'll never leave (unless he creates a new one).

Let me review some of my notes about General Marshall and quotes that he said to illustrate his qualities. I think they also apply to my version.

- A person of integrity. Here's some quotes:
○ Find the necessary moral courage to do the right thing
○ I want to go right straight down the road, to do what is best, and do it frankly and without evasion
○ I can't afford the luxury of sentiment
- A person of action. More quotes:
○ Man is made for action
○ Do as I say.  I will accept all responsibility.
○ The choice is between acting with energy or losing by default
○ I am not interested in the explanation…I am interested in the result

- A person of candor:
○ I am disappointed in all of you.  You haven't disagreed with a single thing I have done all week
○ I am not a diplomat.  I mean exactly what I say and there is no use trying to read between the lines because there is nothing to read there

- A person of fairness:
○ I am awfully tired of seeing mediocrity placed in high positions, with brilliance and talent damned by lack of rank to obscurity
○ Deliver me from the lazy thinker…give me someone who can and will think for himself

- A person of vision:
○ Avoid trivia
○ Face up to our vast responsibility
○ Go home and think about it

I have deep respect and loyalty to my General. It's another reason that I love my team.

DSL for Banking - interactive DSLs

A few weeks ago I posted an M model for a banking transactions. I forgot to post the DSL.
Here's a simple DSL that would be quite easy to use. And, it only took me 10 minutes to write it.

module Banking
    language BankingLanguage
        interleave Ignorable = ' ' | '\n' | '\t' | '\r';
        syntax Main = t:Transaction+
            => Transactions { t };
        syntax Transaction =    
                AccountNumber => { Kind { t }, Amount {a }};

        token Account = "Account";
        token Transitions = "from" | "to";
        token AccountNumber = Digit+;
        token TransactionKind = "Deposit" | "Withdrawal";
        token Amount = CurrencyQualifier? DecimalNumber;
        token CurrencyQualifier = "$";
        token Digit = '0'..'9';
        token DigitWithoutZero = Digit - '0';
        token DecimalNumber = DigitWithoutZero Digit* '.'? Digit+;

Here's some sample input:

Withdrawal 200.00 from Account 5
Deposit $3039483.00 to Account 4932

Now, SpankyJ and I were talking (IM'ing actually).

Wouldn't it be cool if you could write a grammar that defining an iteractive textual experience that you could then deploy to a Intellipad session?

Something like this, where you indicate that the syntax is interactive causing the parser to be re-executed with each parse episode termined by the named token, in this case "loop".

        syntax Main = t:Transaction+
            => Transactions { t };
        syntax Transaction =    
                AccountNumber loop => { Kind { t }, Amount {a }};

        token loop = "\n" | "\r" | "\r\n";

Then, you could have an iterative, transaction-like experience with this particular DSL where people just type in transactions, and get immediate feedback. 

I suppose you'd want the runtime to be hooked up as well... Hmmm - just thinking out loud :)

Sunday, November 9, 2008

Commenting turned on

With this new blog I forgot to setup commenting. It's on now.

Sometimes I think I'm losing it ...

Saturday, November 8, 2008

Oslo - is that all it is?

I enjoyed Jeremy's post about Oslo.

In the comments, there were lots of follow-up questions about the realness and substance of Oslo, and the target customers of the language. Here's some reflections from one Oslo guy...

What we shipped in our first CTP is very much an early alpha of our bits. And, it represents the lowest level of the platform without many of the services, libraries, and bells-whistles that you'd want as a developer. It's as if we shipped the .Net runtime and C# but without any libraries or tools. So, that may be some cause of confusion.

Why did we do this? We wanted to get out the core of the platform as soon as possible so we could start having real discussions with customers and partners. We also want the developer community to embrace modeling as a key principle in the future for how we build applications.

Ultimately, our goal is to enable declarative, model-driven programming. If I look around, I see people doing this today in the form of XML schemas and dialects, various textual reps, and frameworks that encode a domain. We went down that path as well, using visual designers and XML. But at some point the pain was too much :) We evolved our approach into Oslo by trying a more holistic solution. We wanted an easy-to-use language to write down declarations. We wanted a place to store those declarations, and then query, and even compose that data (i.e., join) into "instructions" that runtimes could execute to produce interesting behavior. And then, the idea of making it very easy to create a textual DSL over the model/declarations seemed like a natural step forward to enable app development in an elegant way. It just all fit together for us - and we wanted to share as soon as we could.

Let me also comment about our target customers. We are all developers on our team, and we care deeply about developer tools and productivity. We do not really talk much about making business users programmers. I have to say that personally, I think having a DSL may make it easier to have interesting conversations with business users about the solutions that we build for them. But I don't think that makes them programmers. It just makes it easy to write things down and share/design it with them.

...end of reflecting for now.

Thursday, November 6, 2008

API for MGraph

In M we talk about 3 pieces of technology: MGraph is the data model, MSchema is the type system, and MGrammar is the transformation engine.

MGraph is at the core of all of these. MGraph is a labeled directed graph as defined in the System.Dataflow assembly. Interesting enough, rather than force you to implement a class or interface, we designed MGraph to follow a visitor pattern, like a simplified XML navigator. So, you keep or build whatever data structures you have, and then implement IGraphBuilder to expose those values as an MGraph.

Here is IGraphBuilder:

public interface IGraphBuilder


IEqualityComparer<object> NodeComparer { get; }

object DefineNode(object label);

void DefineSuccessors(object protoNode, IEnumerable<object> successors);

object GetLabel(object node);

IEnumerable<object> GetSuccessors(object node);

bool IsNode(object value);


I'm only going to talk about the navigation part of IGraphBuilder, not the construction protocol.

A node in the graph is identified by your implementation of GraphBuilder by calling IsNode. If this returns true, then the GetLabel and GetSuccessors methods must accept the same object and produce the label and successors, respectively. The Label is optional.

I recently experimented with building an MGraph over an existing SQL database. I defined a SQLGraphBuilder to take a connection and the name of a SQL object, i.e., table or view.

The SQLGraphBuilder queries the database using that name. I constructs a node for the SQL object with the label being the name of the sql object, and each successor being a row in that object. In turn, each row has a label of the primary key (concatenated if there are multiple) followed by each successor, which is itself a node that has the field name as the label and 1 successor being the value.

Here's the base definition for a node in the graph:

abstract class Node : IEquatable<Node>


protected string label;

protected SqlDataReader reader;

public Node(string label, SqlDataReader reader)


this.label = label;

this.reader = reader;


public string Label { get { return label; } }

public SqlDataReader Reader { get { return reader; } }

public abstract IEnumerable<object> GetSuccessors();

public abstract bool Equals(Node other);


So then I have a subtype of this for each kind (SqlObject, Row, and Cell). Here's an example:

class SqlObjectNode : Node


public SqlObjectNode(string sqlObjectName, SqlDataReader reader) : base(sqlObjectName, reader)



public override IEnumerable


return new Enumerable(new SqlObjectEnumerator(label, reader));


public override bool Equals(Node other)


var sqlNode = other as SqlObjectNode;

return sqlNode != null

&& sqlNode.label == this.label;



This is all quite easy except for the definition of identity via the node comparer. You need to make sure that you think through the identity story so that you can easily identity equivalent graph nodes. In my prototype, I use a concatenation of sqlobject + row id + column name. See the Equals method above as an example.

Now here's the tricky part. Some columns are actually foreign keys. So, I maintain a general foreign key lookup table, and when I hit a column name that is actually a foreign key, in stead of returning a value for that cell, I return a new row representing the values from the related table.

Here's the query to find foreign keys:

const string ForeignKeyQuery =

@"select as SourceSchemaName, as SourceTableName, as SourceColumnName, as ForeignKeyName, as TargetSchemaName, as TargetTableName, as TargetColumnName

from sys.foreign_keys fk

inner join sys.foreign_key_columns fkc on fk.object_id = fkc.constraint_object_id

inner join sys.tables tsource on fkc.parent_object_id = tsource.object_id

inner join sys.tables ttarget on fkc.referenced_object_id = ttarget.object_id

inner join sys.columns csource on (fkc.parent_column_id = csource.column_id and csource.object_id = tsource.object_id)

inner join sys.columns ctarget on (fkc.referenced_column_id = ctarget.column_id and ctarget.object_id = ttarget.object_id)

inner join sys.schemas ssource on ssource.schema_id = tsource.schema_id

inner join sys.schemas starget on starget.schema_id = ttarget.schema_id

and fk.type = 'F'

order by SourceSchemaName, SourceTableName, ForeignKeyName";

And here's the code to construct the row in the reference table. Notice I'm still working on it :)

StringBuilder query = new StringBuilder(@"select * from " + fk.TargetSqlObjectName + " where ");

int i = 0;

foreach (var targetCol in fk.TargetColumnNames)


if (i > 0) { query.Append(" and "); }

query.Append(targetCol + " = @var" + i);



i = 0;

//TODO: fix this hack to support non MARs connections; need to dispose connection + reader

var newConnection = new SqlConnection(connection.ConnectionString);


using (var command = new SqlCommand(query.ToString(), newConnection))


foreach (var sourceCol in fk.SourceColumnNames)


object foo = reader[sourceCol];

command.Parameters.AddWithValue("@var" + i, foo);



var newReader = command.ExecuteReader();

return new ForeignKeyCellNode(fk.TargetSqlObjectName, rowName, fieldName, newReader);


GraphBuilder is fairly straightforward to use. It and Mgraphs are at the base of the modeling platform, so learning it now will only help as we introduce more tools and APIs to consume them.

Sunday, November 2, 2008


I'm at PDC with a great team, in a nice place, at a great hotel (I love the beds at Westin).


I was watching Louis CK last night (shh - don't tell me wife. She wouldn't like him).


It dawned on me that I have "Louis". My "Louis" is just like the CK version. He is super bright, magnanimous, and serious yet funny in the same way.


He says what he thinks, which I love. If there's one thing that kills a team culture, it is ulterior motives or backdoor communications. If "Louis" has them, you either would not know, or agree that he's right.


That's one reason I love my team. We say what we think. No one is offended. Everyone wants to do the right thing. And we all just get along.


And that's "Louis". He's also able to inspire millions, mentor our team, and make a huge impact/influence on our technologies. Yet he (and us with him) have so much fun doing it. 

Saturday, November 1, 2008

Self-funded eval of Oslo's M language

I think this is the first self-funded evaluation of M that I've found on the web.  Great job William.

This is my favorite:

Except for this caveat, I like “M”. It’s not anti-XML (you can represent values as XML if you’d like) but it avoids the “the answer is XML/XSD what is the question” approach to modeling that is sometimes a little too prevalent. “M” is a much better schema language for IT systems than XSD. 

Simple Banking Transaction Model in M

At PDC I meet Alejandro. He and I worked through a very simple example of a bank transaction model.

Here's a warning. Not everything writen here is currently supported by the CTP compiler, such as enumerated values for Transaction.Kind. But you'll get an idea of what we would like to support by the time we ship.

Here's the model. Check out the computed value to test that an instance is in fact a valid Transaction. It uses type ascription to test that t adheres to the shape and constraints of the Transaction type.

Also notice the computed value, ChildrenTransactions, to determine the transactions that are children of another transaction. It uses a shorter syntax of LINQ-style queries that we call compact query syntax. I put the full LINQ syntax in quotes so you can compare.

Finally, notice the constraint to validate that Transaction.Target has been set when the Kind is a "Transfer".

module Banking 
    Transactions : Transaction* 
where item.Source in Accounts, 
item.Target in Accounts;

    Accounts : Account*;
    ValidTransactionGoodness (t : Entity)
        t : Transaction

    ChildrenTransactions(t : Transaction)
        Transactions where value.Parent == t
     type Account
        Id : Integer64 = AutoNumber();
    }  where identity Id;

    type Transaction
        Id : Integer64 = AutoNumber();
        Parent : Transaction?;
        Kind : { "Transfer", "Deposit", "Withdrawal"};
        Source : Account;
        Target : Account?;
        Amount : Decimal28;
    }  where identity Id, 
Kind == "Transfer" ? Target != null : Target == null;