ekeepo

Download Package: http://ekeepo.s3.amazonaws.com/Census Public Data Sets Walk Through Package.zip

I wrote a step-by-step walk through (with screenshots) which shows how to use the Year 2000 Census Public Data Set in the context of AWS services (Amazon EC2, Amazon SimpleDB, and Amazon Simple Queue Service (Amazon SQS)).  The following are the scenarios which show the use of Public Data Sets to solve some business problems:

1. Simple Census Parser - Using Amazon EC2, Visual Studio 2008, and C# to parse data from the Year 2000 Census public data set.  The parsed census data is used to hydrate .NET objects which are then serialized to an XML file.  This scenario shows how easy it is to load, parse, and manipulate data in a Public Data Set.
2. Census Excel Viewer – Using Amazon EC2, Visual Studio 2008 Tools for Office, C#, and Excel 2007 to create an Excel 2007 add-in which parses through the Year 2000 Census Public Data Set (using a library created in the Simple Census Parser) and creates Excel sheets with formatted census data.  This scenario shows how to enable business users (familiar with Excel 2007) to easily interact with a Public Data Set from within Amazon EC2.
3. Simple Census Parser to Amazon SimpleDB – An extension of the Simple Census Parser which serializes the data into Amazon SimpleDB.  This scenario shows how a Public Data Set can be exposed as structured data using Amazon SimpleDB.
4. Census Excel Viewer from Amazon SimpleDB – An extension of the Census Excel Viewer which loads the census data from Amazon SimpleDB (instead of from the Amazon EC2 PDS volume).  This component takes advantage of the Simple Census Parser to Amazon SimpleDB code which structures, organizes, and exposes the census data via the Amazon SimpleDB web services.  This scenario shows how Public Data Set data can be consumed by business users (inside Excel 2007 running on any machine with access to Amazon SimpleDB) by exposing the structured data using Amazon SimpleDB.
5. Parallel Query Processor using Amazon EC2 and Amazon Simple Queue Service (Amazon SQS) – A national restaurant/bar chain which has a presence in the top five US markets is investigating the next five markets to which they will expand.  This company is looking for an easy way to access and process the large data set which is the US Census in order to determine the top 10 US cities/regions with the highest number of people between the age of 20 and 34.  Using Amazon EC2, Amazon SQS, and two C# WinForms applications (Query Manager and Query Processor) to implement a parallel query processor implemented using a map-reduce type architecture, this business can easily access the census data, scale to the hardware requirements for this specific query, and parallelize the processing which will help them make the decision quicker while using less resources.  The Query Manager application uses Amazon SQS to map (or break down the components of the census data which will be processed by the Query Processor).  The Query Processor (which can be run on a number of Amazon EC2 instances) is instructed by the Amazon SQS queue to process a subset of the query and report back to the Query Manager, again using Amazon SQS.

Send me a message if you have done some exciting integration with your data and Amazon Public Data Sets or if you have some ideas about how this type of data integration could be interesting to your business.

The installer for TFS 2010 looks great- very nice UI and even had the dependency validation (similar to SQL Server install with the green checkboxes). I decided to brave the install into a Windows Server 2008 R2 (release candidate) for a variety of reasons (HyperV support, Win7 UI, among others) and ran into a problem with the SharePoint part of the install. After you install the SharePoint there are a couple of command line commands (stsadm.exe ….) that you’ll need to run (see TFS 2010 Install Guide). Well, these fail with the following error if you’ve installed SharePoint (without SP2): “Value does not fall within the expected range.”

See the post at http://blogs.msdn.com/dstfs/archive/2009/05/15/installing-tfs-2010-on-windows-server-2008-r2-rc.aspx for the details of how to get around it. There are a couple of steps to fix it. The solution worked like a charm.

I have now created my first Team Project and I like a few things already:
1. Project Collections – Great features for organizing and working with Team Projects in a group.
2. Team Foundation Administration Console – Centralized place to manage all things TFS.
3. Workflow based Team Builds – Very neat and appropriate use of Windows Workflow Foundation. This will make customizing builds very interesting (lots of possbilities for customizations/extensions).

More to come

The two ekeepo™ core service offerings are

1.       Enable software teams to increase organizational maturity using Visual Studio Team System.

2.       Enable software teams to package their assets for reuse using Microsoft Blueprints.

I’ve spent a pretty significant amount of time over the last 6-8 weeks boiling down the ekeepo™ positioning statements.  Apologies to those people who stopped by the web site and wondered, “what can these people do for me?”  That is exactly the motivation for this post.

Enable software teams to increase organizational maturity using Visual Studio Team System

Around mid 2004 (after I left Microsoft), I had the privilege of starting to work with Visual Studio Team System (VSTS).  It was still in pre-Beta and was very difficult to work with, but VSTS was positioned to solve some of the problems I was very passionate about and had been living through in software teams my entire career.  Even though I wasn’t at Microsoft I had to figure out a way to contribute.  Shortly after I left Microsoft I worked on MSF (Microsoft Solutions Framework) 4 (as an outside vendor) with Sam Guckenheimer, David Anderson, and Randy Miller, among some other very smart and talented people.  I learned an incredible amount about how teams work together, the variance between teams, and the opportunity to improve the situation.  I had worked in about 10 teams before that point, but hadn’t really realized how my sample of teams related to (in similarities and differences) to the rest of the software teams in the world.

The opportunity of being co-author on Sam Guckenheimer’s book (Software Engineering with Microsoft Visual Studio Team System http://www.amazon.com/Software-Engineering-Microsoft-Visual-Development/dp/0321278720) not only was an incredible honor, but gave me a chance to show off all the different aspects of VSTS in context of Sam’s text and experience.

At the same time, I was leading a team at Personify Design which was building TeamLook and TeamSpec.  With these two products, our goal was to bridge the gap between the software team and the business by providing interfaces inside Microsoft Outlook and Microsoft Word to Team Foundation Server (TFS).  While we built these products, I had the opportunity to work with high-tens to low-hundreds of teams who were using VSTS (and our products) to increase organizational maturity.   With a number of teams, I was able to dive deeper and help them via ALM consulting and coaching.  It was quite a bit of fun and validated my understanding of the opportunity around helping teams to build better software, with improved quality, on time, and with higher team morale.

As the number of teams adopting VSTS (2005 and 2008) increases and Microsoft continues to add more value to their VSTS 2010 release (as shown in their CTPs), my objective is to help teams use (all flavors of) Visual Studio Team System and Team Foundation Server to increase software engineering maturity.

Enable software teams to package their assets for reuse using Microsoft Blueprints

The idea of packaging a software asset (or code) for reuse is certainly not new.  In fact, by most it’s considered the holy grail of programming and software platforms.  The ability to reuse code libraries promises to create efficiencies desired by programmers and CTO/CIOs alike.  It turns out that it’s easier said than done.   

Over the last 20 years, the industry has come a long way through a number of technologies which have enabled reuse of different forms (i.e. OBJ files, DLLs, OOP, COM, .NET/CLR/IL, SOA).  The challenge with all of these technologies/models is that consuming technologies can often be difficult given the abstractions defined by their corresponding libraries/services.  With Microsoft Blueprints (http://msdn.microsoft.com/en-us/architecture/blueprints.aspx), these libraries/services can be packaged along with the developer workflow (guidance) which drives the developer through the process of using the technology directly inside Visual Studio in the context of their Visual Studio solution and project.   Since early 2008, I’ve had the pleasure of working on parts (Windows Workflow based guidance and extensions) of the Microsoft Blueprints platform (with Michael Lehman from Microsoft) as well as creating Blueprints using the tools.  Given this experience, I am currently working and plan to work with both Enterprises/SIs and ISVs/Component Vendors to build Blueprints which will package their technologies with corresponding guidance and automation.  Depending on what type of company you’re at, here’s how I can help you package your software assets for reuse:

For Enterprises and Consulting/SIs: I can help create your own Blueprints which speed up the process of building the applications your customers/stakeholders need.  These Blueprints can also help distributed development by reducing the ambiguity in how technology is used in your solutions.  For example, if you build web applications for the healthcare industry, what IP can you package in a Blueprint which developers in your company can reuse when building the next component/feature of a web application?  Wouldn’t it be nice if you packaged in all the requirements for implementing your security model and membership/login components into a Blueprint so that you could feel confident that your project is not missing a key requirement?

For ISVs/Component Vendors: I can help create your own Blueprints which help developers consume your technology for a specific application.  For example, if you are the builder of an advanced calendar control which has three special views, you could create 3 different Blueprints which help developers configure/consume your component in each specific view.  This would provide a competitive advantage by increasing the chance that developers get more value out of your component.

If you have any questions or comments, please post questions and comments to the comments below and they’ll get to me.

I follow Jason Calacanis on Twitter and was inspired for this blog post by his (one of many) interesting entry titled ‘The 120% Solution.’   (http://calacanis.com/2008/12/04/the-120-solution/)  I think he makes some fantastic points, but as I started thinking about it I disagree with one of his points.  Here goes:

I agree that we need to cut the waste and start new companies (doing my part J), but I don’t agree that a 6 day work week is the answer.  I think it’s about productivity, making the right decisions, and in the spirit of being Lean, eliminating 20% of things that are not as important as the other 80%. 

How many productive hours does the average American worker experience in a week?  10, 15 maybe?  I’m sure there are people in the 30-35 hour range, but I think that is more the exception that the rule.  Think of this, how many interruptions did you have this week in your office environment?  How much time did that take out of your week?  One more hour of productivity to 30% of our population would result in a huge benefit to our economy.

In the internet connected and device world which we live today couldn’t we get the productivity increase from a larger number of people knowing how to use tools to save time?  I’m a software guy so I don’t think of hammers and pulleys (although recognize their importance), but imagine if people know a little more about spreadsheets, databases, email, and online services.  Imagine if the user interfaces got more intuitive and these tools became more accessible.  The good news is that we’re making this happen. 

One of the simplest tools on the internet today is a search engine.  You type in words and the result is a bunch of pages.  Most people who are reading this know how to use a search engine, but I bet the number of people in America that still don’t know how to use a search engine to answer a specific question (i.e. How do I register my vehicle in the state of Washington?) is staggering.  This might be a result of users never considering that they can use it for this task.

Becoming more productive in an area is not about working harder, it’s about asking the right question and caring to find the right answer.  I help and work in software teams for a living, so applying this to the software teams, if you’re not going to hit your deadline you shouldn’t say, “the Standish Group says that some 60% of projects don’t meet their delivery date” and therefore it’s fine to be part of that 60%.  If you care, which I hope you do as your job is on the line, you should be asking questions such as: Why did we miss the date?  Was it a lack of quality at a specific point or area in the project that caused us not to be ready?  Was it the fact that the scope increased in the middle of the project?  Was it that we lost code during the construction?  Was it that we couldn’t integrate a component into the larger project?  Did we not define the project well enough and not change quick enough when we realized that we were solving the wrong problem?  Yes, it’s hard to know what question to ask to improve, but the first step is caring.  Whatever the problem was/is, the next question should be: How do we fix it?  Continuous Improvement is much about asking that question over and over and following through.

Michael Lehman posted a great Blueprints 101 video on Channel 9.  http://channel9.msdn.com/posts/MichaelLehman/Microsoft-Blueprints-101-The-SimpleEdit-Blueprint/

In this video he shows how to consume a Blueprint which guides a developer to customize an application (and its parts) which is unfolded into a Visual Studio solution.   He also shows the deep integration with Visual Studio which is core to the Blueprints system which I’ll overview below.

The Blueprints Manager off of the Visual Studio Tools menu shows what Blueprints are registered/installed on your machine.  This is the list of Blueprints that you can unfold.  It also shows the Blueprints that have been unfolded into the currently open Visual Studio Solution.

You’ll also notice that he right clicks on projects in the solution and there is a context menu item which is not usually present, namely the ‘Blueprints’ menu item.  In this Blueprint (the one he unfolded at the beginning of the video) he decided to add two menu items to the main Visual Studio project (BPEdit), namely ‘Add Load/Save Filter…’ and ‘Add Options Dialog…’  As a Blueprints author (when you design your own Blueprint) you can define any actions which you want to expose to the Blueprint consumer.  In this case, for the ‘Add Load/Save Filter’ he’s adding a new C# file which is the Load/Save filter.  The guidance (which is part of the Blueprint) explains how to connect this filter.

Speaking of the guidance, every Blueprint contains, at least, some guidance related to its topic.  I say at least because the smallest Blueprint you can create is only guidance.  The Blueprints guidance can be seen in the Visual Studio ‘Workflow’ Tools (dockable) window.  You’ll see that it can even keep state about what has been accomplished (ie. checked) in an unfolded Blueprint.

In this case, Michael is showing how to build an application that is made from specific type of application (an editor application).  This type of pattern reuse is very powerful for companies/developers which are building applications with a core set of libraries/concepts.  For example, a consulting company which creates social networking web sites might create a Blueprint to unfold the User Management features of the social networking site.  That might include pages, database code, or other parts of the site.  Then the developer can extend the pages to provide a unique look and feel for the specific customer’s brand which needs the social network.

Lights, camera, action!  Welcome to the ekeepo, LLC blog and the first official post.

First, you might be wondering, “What is ekeepo?”  Well, first things first.  It’s pronounced ‘eh-kee-poh’.  If you’ve already pronounced it out loud and you speak either Spanish or French then you may already realized that it’s phonetically similar to the Spanish word ‘equipo’ or the French word ‘équipe’ which translate to ‘team’ in English. 

I founded ekeepo, LLC to reach a new level of excellence with ‘Team Decisions and Patterns.’  I’ve been working in software teams for 15 years and in that time have experienced some incredible events (on both sides of the good to bad spectrum).  Some were missed business opportunities where we built great technology, others were great business situations where we missed a technical opportunity, and sometimes we reached flow where both the business and the technology delivered for the customer need.  One thing is for certain: the people on the team make the magic happen.  The challenge is that similar to the greatest artists or composers in history, methods and tools need to be mastered to reach true creativity and innovation.  Whether the advancement you seek with the software you build is elegance, efficiency, or meaning, two core components of reaching a new level of excellence in your team are:

1.      Understanding and enabling decisions throughout the software team

2.      Observing and reusing patterns in the software team where appropriate

Depending on the size of your software team, the people on the team are constantly making tens, hundreds, or perhaps even thousands of decisions at all layers of abstraction and disciplines (ie. Developers, Testers, Business Analysts, Project Managers, etc…).   For example, “Is what I just observed a defect?” “How should I implement this requirement?” “Are we going to make our deadline given the defect rate we’re seeing in the project?”  “Should I write test cases for this new feature I just implemented?”  “Did we prioritize the customer needs correctly?”  There are so many questions similar to these that influence decisions which we’re constantly making in the software team.  For the last 5 years, I have been working with hundreds of teams worldwide at different levels of involvement and building on top Microsoft’s ALM (Application Lifecycle Management) platform Microsoft Visual Studio Team System (VSTS).  Through ekeepo, I plan to continue to help teams reach new levels of excellence by using ALM tools such as VSTS and TFS.  I plan on doing a bit of blogging around ALM and TFS, but don’t take my word on it.  Subscribe to my feed and watch the blog entries flow J.

The other very important side of ekeepo, is the ‘patterns’ component of building value in your team.  You might argue that there is not a part in software which is not patterns.  You’re right!  Patterns are everywhere.  Patterns are in code, in frameworks and code libraries, methods, documentation, requirements documents, use cases, models, data models and databases, operating systems, etc…  The patterns I’m referring to are the ones in how people interact and how they pass deep knowledge between each other.  One core principle of ekeepo is to help teams ‘observe process instead of predicting process.’  It’s very similar to the idea of refactoring code into a library which is useful instead of building a code library which you think might be useful.  Often we find that the former is the better approach.

One great implementation of team patterns is being driven by Michael Lehman at Microsoft titled Microsoft Blueprints.  I’ve had the pleasure of working for Michael on a small part of this effort and I believe Blueprints will improve the way we package software best practices and guidance.  I plan on building Blueprints and hope to blog about Blueprints, how to build them, and how they’re helping teams around the world.

If you have any questions or comments, I’d love to hear from you!  Find me on LinkedIn or Facebook and drop me a line.  Stay tuned for more!