Recently I was engaged in a large scale Lotus Notes to SharePoint 2010 Migration project for a client in the travel/aviation industry vertical . Lotus Notes is a multi-user collaborative client-server application environment competing with Microsoft SharePoint 2010 suite of products /Microsoft Outlook/ Office workspace 2010. There are numerous legacy Lotus-Notes implementations which sometimes can be a overhead to main overtime and scale. This post summarize the approach taken to migrate such a application into SharePoint 2010.
The legacy Notes Applications in the context was designed and developed few years back was not quite helping accelerating the business outcomes for the client. Therefore migrating those applications onto a single platform which can centrally manage and maintain over time was timely.
The Domino product is an integrated messaging and Web application software platform. It provides a robust environment for creating Web applications that support workflow, RDBMS connectivity, and collaboration. Domino is ideal for creating applications that allow "sometimes connected", multiple users to cooperatively process unstructured information.The term Lotus Notes is now used to refer to the underlying client-server architecture that the product is built upon.
The storage model of Lotus notes storage area a "Note" (These Notes will also be referred to as documents, a term more familiar to users). In each Note we have "items"; and each item has its own "item name". ( and an item may be a list of values) .This iterative structure of one element containing other elements has been referred to as the Notes container model. The "Notes Storage Facility" and the file that creates on the Notes Client is given an extension of .NSF.The Notes data model contains no concept of a "record type". All documents in a database are equivalent as far as Notes in concerned.
Forms is the mechanism that Notes Client (Desktop Client) use to show the user what is in a document.The areas on the form that display the values stored in the items on a document will be called fields.This listing of item values from a set of documents in a column format is a Notes View.
You can program Notes to perform tasks automatically using agents (also known as macros). Agents can help you perform repetitive tasks, such as managing documents and sending memos. Agents can complete almost any action you can do manually in your databases.When you create an agent, you can set options to specify:
- When the agent should run. (ie. a document has been changed)
- Which documents in the database the agent should run on. You can target all or any subset of documents in the database for the agent to run on.
- What actions the agent should complete. You can choose actions from a list or use Notes formulas or programming scripts and languages to create actions.
Tools for the Job
- Visio 2010 Premium – Modelling new the process workflows with business analysts/domain experts
- SharePoint Designer 2010 – Authoring the workflows/ customising list forms etc.
- Visual Studio 2010 – Create Solution Artefacts/ Custom event handlers/ create features, packaging sandbox .wsp solutions
- SPServices jQuery library for SharePoint web services – UI extensibility helpers / Cascaded dropdowns etc.
- Bit Bucket with Mercurial/ VisualHG -Distributed Version Control (DVCS)
- Quest Notes Migrator for SharePoint – For Data Mapping and Migration from Notes fields to SharePoint Lists.
Each Notes Application was mapped to Corresponding SharePoint 2010 Custom Application in its own Site Collection with its own Logical Security boundaries.
The new applications were architected and implemented leveraging the new platform capabilities of SharePoint 2010 and mapped the required data into the new application platform. The Quest Notes migrator did a good job for us by nicely abstracting Lotus notes internals by allowing us to defined Extract-Transform-Load (ETL) operations including user mappings from Domino to AD accounts. Where Transformations were complex Extract-Load-Transform (ELT) approach was used with custom transformation rules with PowerShell / C#
High level process followed for migrating each Notes application to SharePoint 2010 platform involved following steps
- Consult Notes application owners (product owners) in each department and created a prioritised product backlog
- Modelled the new workflows /rules with Visio 2010 premium with business users
- Built & Prototyped the new application with the best tool fit for the job. (Visual Studio 2010 - Author and deploy base level custom site columns/content types/list definitions/list instances/custom event handlers , SharePoint Designer 2010 – forward engineer the Visio workflows / XSLT List form customization )
- Data mapping jobs were created/tested with Quest Notes migrator for SharePoint in the dev environment with a subset of production data
- Repeated steps 3 & 4 with product continues reviews/feedback
- Reverse engineer reusable workflows into features and list form into solution artefacts into SharePoint Solution packages and deploy into production environment as SharePoint features
- Execute the Data migration job on the production environment
Hope this post would benefit those who interested in migrating legacy lotus notes applications to SharePoint 2010.
Improving the quality of your Custom SharePoint 2010 Solutions via continues Integration(CI) with Team foundation Server 2010Posted: March 31, 2011
This is a follow up blog post based on the content of my presentation on “SharePoint2010-ALM” presented at Melbourne SharePoint User Group meeting (#MOSSIG) on 23rd March 2011. Since then I’ve received number of requests from the community for sharing the content and the references. Therefore I thought of writing a little detailed post about this topic and here we go.
Continues integration is a critical process in modern day product development in improving the quality of your solution from 1st Sprint (iteration) of your product development if you are following an agile approach like Scrum. This allows you to early detect and rectify the issues in your solution which increases the chances of success without having to go through painful integration issues and surprises at the last minute in your product development cycle.
How does it improve the quality of my custom SharePoint solution?
Well if you have a mature fully automated continues build (CI build) and a Nightly build you are in the correct path and your SharePoint development team enjoys the following in every sprint.
- It enforces you to promote the code from Dev -> Test-> Staging -> Production in the correct way in the first place. Otherwise automation will not be possible.
- Every code check in by a developer will kick-off an automated code validation, run unit tests (if any) and build the SharePoint solution (.wsp files) in the Team foundation server.
- A Nightly build will do every thing in step 2 plus Add, Deploy and Activate your SharePoint solution packages into your test farm and can run and automated coded UI tests and automated load tests (if any) for you.
here is the sketch workflow.
Why was this approach not so common in SharePoint Development?
The Lack of SharePoint development tooling support with VS2008 contributed significantly. This is also coupled with the natural complexity arises when you develop on a platform rather than a development Framework. When it comes to SharePoint development, you may not necessarily develop with Visual Studio environment. You can develop quite powerful collaborative solutions and branding work with SharePoint Designer as no code solutions. But unless we convert those customisations into solution artefacts we will not be able to cleanly source control and maintain the solutions which breaks the fundamental concept of Application Lifecycle Management (ALM).
SharePoint 2010 has evolved and positioned as a single unified platform for most originations playing a mission critical role for day to day operations. With the introduction of Business connectivity Service (BCS), Client OM, Sandbox solutions, WCF based Service Applications and Claims based authentication SharePoint 2010 has also significantly evolved as application development platform which opens up a number of extensibility options for the developers.
We also have integrated set of tools built into Visual Studio 2010 for SharePoint Development. So the challenge is how to use these tools effectively and efficiently in a team based environment to build your product or solution and deploy into those shared mission critical environments confidently with consistent quality? This post describes how to implement an automated continues build with the help of following tools to validate, build and deploy SharePoint solutions.
- VS2010 (with SP1 adds support for 64 bit unit testing and Intellitrace with historical debugging capabilities for SharePoint Development)
- VS2010 SharePoint Power tools (Sandbox code validation, Visual Web Part as Sandbox Solutions)
- TFS 2010
- CKSDev (productivity tools),
- Pex & Moles (unit testing)
- SPDisposeCheck (build validation)
- Sysinternals PSEXEC (remote deployment)
Step1: We need to get all your customisations as solution artefacts. Reverse engineer your SharePoint designer customisations and featurize them. CKSDev Branding SPI can help you to deploy master pages and CSS as Features. Save all SharePoint Designer authored workflows as Reusable workflows as .wsp from SharePoint Designer and bring it to Visual Studio environment.
Step2: Create a new Team project and add your solution into source control under TFS. For the purpose of this post I am using a simple VS2010 solution name SPHelloWorld. For Code analysis, SPDisposeCheck ruleset needs to be in source control and can be shared across multiple projects with in your repository.
Step3: Enable Code analysis for you solution via project poperties and select SPDisposeCheck RuleSet from your local TFS workspace as shown below.
Step4: Note that Code Analysis now runs SPDisposeCheck rules as part of compilation and both Dispose and Do not Dispose rules are fired when detecting the the poor code.
Step5: Now open the Default Build Template for the project. We are going to extend the default build definition file to deploy our solution to the Test SharePoint Farm remotely. TFS 2010 is using windows workflow foundation 4 based
.xaml template. Drag the activities from the left side bar tool box and assemble the activities as shown below just after packaging is finished.
The high level tasks performed by this segment of the workflow are
- Build the Solution
- Run Code Analysis with SPDisposeCheck on the TFS Server. If this code analysis fails our build would fail at this point.
- Create .wsp solution package
- Check the user specified deployment flag (This is a work initiation parameter)
- Write a log message to the log file
- Map a Admin share drive to the Target SharePoint Server (Test Farm) with the user specified credentials.
- SET the STSADM path on the target server
- Stop V4 Admin service in the target farm. (This uses PSEXEC to remotely execute STSADM on the target server)
- Copy each .wsp file to the target SharePoint server and add, deploy and execute the timer job in user specified sequence.
- Finally turn on the V4 Admin timer service.
If you are interested the customised TFS2010 workflow template for building and deploying .wsp solutions can be downloaded from my SkyDrive here
Now define the additional parameters required for the workflow initiation that needs to be supplied by the user when starting a new Team build.
Now set activity properties for each activiry. As a sample I’ve shown below the properties set for the “Add solution” InvokeProcess workflow activity. Once all properties are set WF4 validates your workflow. Once all validations are passed, checkin your custom build definition template into TFS.
Step6: Author a new Build definition based on our updated workflow template.
Some important parameters that you need to set are shown below.
Step7: It’s time to Queue a new Build. Now assume a scenario where the developer rushes to checkin by overriding the Check-in Policies set by the build master.
and here is the result : TFS build agent decided to fail your build because your code did not pass the code analysis. No .wsp’s produced.
So developer now fix the code with proper disposing as shown below and check-in the code again.
And after few seconds we get this TFS Build status. All green!
Opening the build log gives us the workflow execution log as shown below. Note how STSADM has been invoked remotely via psexec.exe on the TFS build agent.
and see below the end result in the central admin in our test SharePoint Farm.
On a final note “the right tools make all the difference”. But tooling it self may not helps you to build a quality product. The process matters. TFS 2010 support you with your chosen process and allows you to tailor the default process templates if required. You have the chance to select the process template when you create a new Team project as shown below. I highly encourage you to check out the new “Microsoft Visual Studio Scrum 1.0” template. This template has been co-developed by Microsoft in collaboration with the Scrum community. The template guides you with scrum based agile product development. I’ve highlighted a few below.
- Creating effective user stories
- Creating prioritizing grooming product backlog
- Agile estimation and Sprint planning
- Scrum artefacts such as sprint burndown and release burndown chart
And here is the high level process behind the template.
The correct tools coupled with the correct process will take your product/solution to the next level faster and smoother and make you a successful SharePoint developer and hence a successful product/solution delivery Team.
Where to go from here?
- Disposing Best Practices
- Unit Testing SharePoint Solutions
- Unit Testing SharePoint with Pex & Moles
- How to Build SharePoint Projects with TFS Team Build
- Configuring TFS for building SharePoint and SPDisposeCheck (10min) by Jeremy Thake
- Scrum Resources