Documenting architecture

One of my side projects at work right now is documenting the architecture of a product that has already been built but will be going through a re-architecting with a focus on a more robust schema and applying some of the learning we've gone through in discovering exactly how our product is being used and ways in which our users want to extend the platform. SaaS and SOA are two good buzz words we'll be throwing around a lot, although to be honest we've been in the SaaS model for years now, just not following all of the best practises. (examples, check out litwareHR)

So despite documentation being at the heart of the architect's role I find it extremely difficult to find good documentation on how to approach a task like this. I have Craig Larman's book Applying UML Patterns which I've enjoyed, but I still find myself grappling for where to even begin sometimes.

These articles on IBM have been good reads for this and I'd reccommend giving them a read if you are facing similar challenges.

Part1
http://www.ibm.com/developerworks/library/ar-archdoc1/index.html?S_TACT=105AGX20&S_CMP=EDU

Part2
http://www.ibm.com/developerworks/library/ar-archdoc2/index.html?S_TACT=105AGX20&S_CMP=EDU

Part 3
http://www.ibm.com/developerworks/library/ar-archdoc3/index.html?S_TACT=105AGX20&S_CMP=EDU

I'm assuming there will be more of these which I'm looking forward to.

Networks, Assignment 1

I just finished my first assignment in a beginning networking course I'm taking and I am so far pretty impressed with how interesting this stuff is. I have a working knowledge of networking that includes decent understanding of the application layer, high level knowledge of the transport layer and basically just awareness of the link layer. It's pretty rare that in my position as a developer that I need to answer questions about the link layer. (thank you my friends in IT)

Some of the questions are actually kind of fun in that they had me visualizing data flowing through networks in ways I had not before. For example given a link between two hosts X km apart, with a transmission rate of R and a propagation delay of N....

      2.4.d What is the width (in meters) of a bit in the link? Is it longer than a football field?

Kind of useless, but super fascinating at the same time, imagining the physical manifestation of all this work I do day in and day out. Pulling these bits from all over the world is so effortless, so fast and so transparent that it's easy to forget the actual resources behind it.
The football field question actually relates to a pretty interesting concept called bandwidth-delay, which refers to the amount of data that exists "on the wire" or "on the air" at any given moment. Data that has been sent but not yet acknowledged. It's helpful in determining minimum buffer sizes for receivers and transmitters over a given link.

http://en.wikipedia.org/wiki/Bandwidth-delay_product

Another element to this first assignment was to setup apache as a proxy server on your local machine which was a bit surprising. I assumed at first that the assignment meant squid, but no, apparently apache itself can be configured to be a proxy server for a number of protocols including both ftp and http traffic.There are numerous articles out there on using it as a personal ad blocker, or caching server.

For reference this is what I had to do to Httpd.conf to make it work :

LoadModule disk_cache_module modules/mod_disk_cache.so
LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_http_module modules/mod_proxy_http.so
LoadModule proxy_connect_module modules/mod_proxy_connect.so

<IfModule mod_proxy.c>
ProxyRequests On
<Proxy *>
Order deny,allow
Deny from all
Allow from 10.0.1.2/255.255.255.0
</Proxy>
</IfModule>

<IfModule mod_disk_cache.c>
CacheRoot "c:\apachecache\"
CacheDirLevels 5
CacheDirLength 3
</IfModule>
Interestingly if you get that configuration wrong, you actually get a big "It Works!" page shown in your browser for any page you try to visit. Go figure. My mistake at that point was just not having uncommented the right modules, so apache was just serving the It Works page rather than attempting to proxy my request. 

blogquotes prototype is working

I'm supposed to be studying for a challenge exam I'm writing this week in "Advanced" Operating Systems. Instead I spend a good chunk of the day today working on blogquotes in between watching/playing with my daughter.

I can justify this time spent because I did all of this work exclusively from a bash shell using vi to refresh myself on some of the content for the course. In this session I was able to use wget,grep,awk,vi,a shell script and some file permissions. I'm not so sure that will get me through the exam but it was fun and I was finally able to put some time towards my random quote include for the blog.

You can see the quotes being pulled in now in the top right hand corner of this page. So far only my wife and I are using it, as this is very much a proof of concept. It works basically from end to end, but without a lot of features that are going to be necessary as this grows. Paging, searching, caching, tags, some UI polish, and some testing. It's very gratifying though to reach this first phase and actually get something working. Adding features will probably happen a lot quicker now that I actually have a user. ;-)

I also took the opportunity to try out some of Yahoo's client side api's in the YUI. Currently I'm using XHR, Layout and DataTable. I was amazed at how quick it was to basically "assemble" my application. Google's app engine makes the CRUD a total cake-walk, and yahoo's user interface library has no dependencies on server side code but works seamlessly with a JSON backed RPC scheme in Python. It's a whole new world! Now as long as my application doesn't get popular and I have to start paying for resources. ;-)

Some interesting snags while working on this latest revision:
  • Randomly selecting an entity in GQL
  • Django Utils simplejson can't serialize google's db.Model classes, so I had to actually proxy my model class to a simpler structure that I serialize to the client via JSON and XHR
  • No unique id's for google user accounts, all you have is email which isn't exactly something people are going to appreciate me passing on url's for the random inclusion widget. The solution was simple for what I needed, a user preference entity keyed on google's User db type and storing a UUID as the publishingKey, that id now becomes my unique id which won't change even if you change your google account name
  • Google has a very cool AJAX Library SDK for sharing hosting/serving up of all the most popular frameworks like dojo and jquery
I think I'll use google code to start some documentation around bugs and features for this tool rather than simply the blog so look for those details elsewhere.

code samples in blogger are a pain

I can't say I enjoy writing in the blogger post interface, in fact it's pretty frustrating. For a while there I was using google docs to write posts(which I loved), then I would just publish to my blog. That actually worked great until the actual publishing process which doesn't allow you to control the title very effectively and totally messed up my rss feed even if I did fix the title. Then I tried scribefire which again was really promising but it's a cramped UI and again the publishing process was really clunky for my workflow. (things remain drafts for me for weeks at a time)

Anyway, I'm looking at my last post and those code samples are embarrassingly poorly formatted. Not only that but if you check the source the blogger editor is introducing tons of html space entities which drives me nuts considering I'm using whitespace:pre on my blockquotes anyway.

I'm really inclined to just use the tools I have when it comes to this site, primarily so that I focus on writing and not tinkering. Since moving my website from a hosted environment to blogger I have actually started to focus again on my writing and my projects rather than tinkering with a wheel that's been built a thousand times (photo gallery scripts, php and perl cgi trickery for mundane templating etc). So while I will probably end up spending time on this at some point I really just want to find something that "just works" for showing code in blog posts. More to come I'm sure.

Microsoft's add-in framework and the need for diligence

We've recently put Microsoft's managed add-in framework (part of .NET 3.5) into very effective use building a plug-in system for a large asp.net application at work. Essentially the framework in place allows other developers (and our own team for out of stream releases) to develop new functionality for our platform that runs the entire life-cycle for a given widget. In our case for this particular widget we're talking about plugins being responsible for up to 4 asp.net controls in different contexts (for example data collection and reporting as two separate controls) as well as a script injection point where plug-ins are able to extend the scriptability of our platform. 
For us going with the framework gave us a few things we didn't have with our original design for the add-ins. 
  1. Tools to help enforce the pattern
  2. An extra layer of versioning over the somewhat naive approach we started with
  3. Built in discovery, provisioning, and a communication pipeline for serializing types and calls across the contracts that make up the interface between host and plugin
  4. And last but not least support from Microsoft. This is somewhat more minor than the points above, but it helps legitimize our design when we are following the best practices laid out by Microsoft and used by others in similar situations. The documentation and training available also make getting other developers up to speed on the framework that much easier.
There have been numerous challenges in using the framework, but perhaps the most surprising of all for me was the human element and how simple it became over the life of the project to break the pattern by coupling components across or outside the pipeline.

Examples :
  1. Referencing an assembly from both the addin and the host that shared code that should have been passed across the pipeline. 
  2. Bypassing the pipeline completely by calling web services from the addin code (client side or server side calling code) 
  3. Conditional code in the host making decisions based on the type of the addin
  4. Loose coupling based on common knowledge (that shouldn't be common) 
These all basically come down to a breach of contract or an absence of contract for various operations that we needed addins to handle. On some level all of these things can be excused and safely done without compromising the framework if they are done right. It's a slippery slope though and requires a commitment to not be lazy to avoid the temptation to sidestep the pipeline.

In the case of #1 above the shared assembly started off very benign. Essentially some shared utility code for handling urls and some common resource tasks. Why rewrite when that code already existing the main project? Break it off from the project so that it has no dependencies then drop it in. Except that slowly the terrible pain of building contracts, views and adapters for every little interface or interface change drives you towards shortcuts. "Oh I'll just put this code here to test and then fix it later"  Even worse are those cases where you've chosen the path of least resistence in dealing with a bug resulting from unexpected behavior with serialization across the pipeline. It only took a few weeks of not being completely on top of this before I discovered our project was littered with types that were being shared directly between host and addin. Any change meant a recompilation of both projects, completely defeating the purpose.

#2 is a legitimate need in our scenario, and we've found ourselves needing to creating proxy services that wrap our own services just to protect against the inevitable change that will follow. Given that third party developers may be writing code for the platform we have to make an effort to protect from change in all of our interfaces, web service or otherwise. In retrospect I think it would have made more sense to strictly enforce a team division so that no one writing addin code was also writing host code.This probably would have gone a long way to preventing these types of problems.

#3 and #4 are a little more insidious and harder to spot without strict code review. #3 for us isn't technically breaking anything in terms of the interface or future versioning, but adds cruft and generally points to a missing method or property on the interface. The last thing you need as the host is to have case statements littered throughout your code looking for addins. #4 took many forms, and in some cases it's fine. An ok example might be sharing enums, which provided they are defined in the contracts or slightly worse something like a utility class is ok. A not ok example for me was code like this :  extension.GetSetting("Menu_Text");  which in this case has two errors. One "GetSetting" shouldn't really exist because how an addin chooses to configure itself should be transparent to the host. Second this code depends on the addin having a value defined in it's config file for the key "Menu_Text". This is next to impossible to enforce and can of course easily break.

Replacing this with extension.MenuText; should be trivial, and a no-brainer. When we started using the framework back in December we were rolling the supporting code by hand. To give you a sense of what this entails, this is how you would define an extension who's only job is to return MenuText as in the code above :

IExtensionContract.cs
using System.AddIn.Pipeline;
using System.AddIn.Contract;

namespace SimpleExtensionContracts
{
 [AddInContract]
 public interface ExtensionContract : IContract
 {
  string MenuText { get; set; }
 }
}

IExtension.cs
namespace SimpleExtensionContracts.AddInViews
{
    
    [System.AddIn.Pipeline.AddInBaseAttribute()]
    public interface IExtension
    {
        string MenuText
        {
            get;
            set;
        }
    }
}

IExtension.cs
namespace SimpleExtensionContracts.HostViews
{
    
    public interface IExtension
    {
        string MenuText
        {
            get;
            set;
        }
    }
}

IExtensionContractToViewHostAdapter.cs
namespace SimpleExtensionContracts.HostSideAdapters
{
    
    [System.AddIn.Pipeline.HostAdapterAttribute()]
    public class IExtensionContractToViewHostAdapter : SimpleExtensionContracts.HostViews.IExtension
    {
        private SimpleExtensionContracts.ExtensionContract _contract;
        private System.AddIn.Pipeline.ContractHandle _handle;
        static IExtensionContractToViewHostAdapter()
        {
        }
        public IExtensionContractToViewHostAdapter(SimpleExtensionContracts.ExtensionContract contract)
        {
            _contract = contract;
            _handle = new System.AddIn.Pipeline.ContractHandle(contract);
        }
        public string MenuText
        {
            get
            {
                return _contract.MenuText;
            }
            set
            {
                _contract.MenuText = value;
            }
        }
        internal SimpleExtensionContracts.ExtensionContract GetSourceContract()
        {
            return _contract;
        }
    }
}

IExtensionHostAdapter.cs
namespace SimpleExtensionContracts.HostSideAdapters
{
    
    public class IExtensionHostAdapter
    {
        internal static SimpleExtensionContracts.HostViews.IExtension ContractToViewAdapter(SimpleExtensionContracts.ExtensionContract contract)
        {
            if (((System.Runtime.Remoting.RemotingServices.IsObjectOutOfAppDomain(contract) != true) 
                        && contract.GetType().Equals(typeof(IExtensionViewToContractHostAdapter))))
            {
                return ((IExtensionViewToContractHostAdapter)(contract)).GetSourceView();
            }
            else
            {
                return new IExtensionContractToViewHostAdapter(contract);
            }
        }
        internal static SimpleExtensionContracts.ExtensionContract ViewToContractAdapter(SimpleExtensionContracts.HostViews.IExtension view)
        {
            if (view.GetType().Equals(typeof(IExtensionContractToViewHostAdapter)))
            {
                return ((IExtensionContractToViewHostAdapter)(view)).GetSourceContract();
            }
            else
            {
                return new IExtensionViewToContractHostAdapter(view);
            }
        }
    }
}

IExtensionViewToContractHostAdapter.cs
namespace SimpleExtensionContracts.HostSideAdapters
{
    
    public class IExtensionViewToContractHostAdapter : System.AddIn.Pipeline.ContractBase, SimpleExtensionContracts.ExtensionContract
    {
        private SimpleExtensionContracts.HostViews.IExtension _view;
        public IExtensionViewToContractHostAdapter(SimpleExtensionContracts.HostViews.IExtension view)
        {
            _view = view;
        }
        public string MenuText
        {
            get
            {
                return _view.MenuText;
            }
            set
            {
                _view.MenuText = value;
            }
        }
        internal SimpleExtensionContracts.HostViews.IExtension GetSourceView()
        {
            return _view;
        }
    }
}

IExtensionAddInAdapter.cs
namespace SimpleExtensionContracts.AddInSideAdapters
{
    
    public class IExtensionAddInAdapter
    {
        internal static SimpleExtensionContracts.AddInViews.IExtension ContractToViewAdapter(SimpleExtensionContracts.ExtensionContract contract)
        {
            if (((System.Runtime.Remoting.RemotingServices.IsObjectOutOfAppDomain(contract) != true) 
                        && contract.GetType().Equals(typeof(IExtensionViewToContractAddInAdapter))))
            {
                return ((IExtensionViewToContractAddInAdapter)(contract)).GetSourceView();
            }
            else
            {
                return new IExtensionContractToViewAddInAdapter(contract);
            }
        }
        internal static SimpleExtensionContracts.ExtensionContract ViewToContractAdapter(SimpleExtensionContracts.AddInViews.IExtension view)
        {
            if (view.GetType().Equals(typeof(IExtensionContractToViewAddInAdapter)))
            {
                return ((IExtensionContractToViewAddInAdapter)(view)).GetSourceContract();
            }
            else
            {
                return new IExtensionViewToContractAddInAdapter(view);
            }
        }
    }
}

IExtensionContractToViewAddInAdapter.cs
namespace SimpleExtensionContracts.AddInSideAdapters
{
    
    public class IExtensionContractToViewAddInAdapter : SimpleExtensionContracts.AddInViews.IExtension
    {
        private SimpleExtensionContracts.ExtensionContract _contract;
        private System.AddIn.Pipeline.ContractHandle _handle;
        static IExtensionContractToViewAddInAdapter()
        {
        }
        public IExtensionContractToViewAddInAdapter(SimpleExtensionContracts.ExtensionContract contract)
        {
            _contract = contract;
            _handle = new System.AddIn.Pipeline.ContractHandle(contract);
        }
        public string MenuText
        {
            get
            {
                return _contract.MenuText;
            }
            set
            {
                _contract.MenuText = value;
            }
        }
        internal SimpleExtensionContracts.ExtensionContract GetSourceContract()
        {
            return _contract;
        }
    }
}

IExtensionViewToContractAddInAdapter.cs
namespace SimpleExtensionContracts.AddInSideAdapters
{
    
    [System.AddIn.Pipeline.AddInAdapterAttribute()]
    public class IExtensionViewToContractAddInAdapter : System.AddIn.Pipeline.ContractBase, SimpleExtensionContracts.ExtensionContract
    {
        private SimpleExtensionContracts.AddInViews.IExtension _view;
        public IExtensionViewToContractAddInAdapter(SimpleExtensionContracts.AddInViews.IExtension view)
        {
            _view = view;
        }
        public string MenuText
        {
            get
            {
                return _view.MenuText;
            }
            set
            {
                _view.MenuText = value;
            }
        }
        internal SimpleExtensionContracts.AddInViews.IExtension GetSourceView()
        {
            return _view;
        }
    }
}

Yeah, seriously. One interface and one string accessor requires nine class/interfaces and over 200 lines of code (which obviously could be made less with formatting etc).  It's also possible to share the views between addin and host but then you lose part of the more compelling robustness of the framework. If you are interested in where these classes come into play and how the add-in framework actually works check out this link for a good description.
Anyway, I can sympathize with the developers in wanting to speed up the process a bit, but the answer is not to bypass the pipeline. The answer is code generation! Thankfully by the time we realized our mistake Microsoft had released a CTP of their pipeline generator which is a nifty little visual studio addin which picks up the output of the Contracts project and uses reflection to find all of the contracts and generate the necessary projects and files for the pipeline. It literally saved us tons of hours and made the addin framework actually usable. Of couse the code generation is only going to work until we version one side or the other, but at that point we should have solidified those interfaces considerably so it will matter a lot less.

Anyway, long story short, the add-in framework is great, but it's really important for the entire team to understand the goal and be diligent in ensuring that all that extra framework code isn't just being wasted by introducing dependencies.