A bright future, Orinoco & Complex Event Processing (CEP)

[Update: Microsoft announced the offical name StreamInsight and a CTP available in August sometime for download. See here for details.]

CEP has been creeping into the blogsphere over the last few years but has not caught on in mainstream computing but I reckon this is about to change. See this link for a quick introduction,

http://www.thecepblog.com/what-is-complex-event-processing/

CEP’s main  proponents are finance, energy,risk,fraud detection, etc. but I reckon the really big break through is going to be consumer driven. I see here an explosion of data especially from consumer devices just think of all that real time GPS data on an iPhone coupled with either of,

-Mint web site which stores a significant proportion of ones financial health

-Amazon online shopping history

A who’s who of vendors include,

StreamBase

Progress Apama

Truviso

IBM

Aleri/Coral8

Finally why blog about this now, well back in February Microsoft announced a product coded named “Orinoco”

Introducing Microsoft’s Platform for Complex Event Processing

Involved In Software Development – You Need This !!

A website worth browsing around. I really liked its page below as its definitely a skill you need to master in development,

How to detect bullshit

Workflow as the wildcard? Absolutely…

This is a response to the blog post,

http://mdavey.wordpress.com/2008/12/29/further-azure-thoughts/

So the question was,

“Or go for a “wild card” WF implementation?”

I believe this has a reasonably chance of being the future of a lot of development. We just need to answer the when but with WF 4.0 being a complete rewrite and items like Dublin, Velocity and Parallels library in the works then yes its a stretch but their unification could actually work and bears investigating.

Just imagine a team building an app. DDD style and exposing public domain services that then get stitched together by a domain expert via a Workflow GUI like K2’s so getting them involved in the application building process, this has to be very compelling for any DDD guy.

In terms of workflow editors I’d recommend folks look at the one in MS CRM 4.0 which is very interesting and goes another step in the right direction for MS. Just look at the image that MS has for the Azure platform and guess what CRM services (don’t get hung up on the name CRM, MS will end up renaming it to something like xRM and it will compete with the likes of Salesforces force.com offering) sits in as one of the 5 services.

So workflow is going to be huge but it will take lots of time to get into the high end (if ever) of financial trading systems but one place to start I reckon is low volume high value transactions…

Death Knoll for Relational Databases ??

Well the subject line maybe a bit premature but I was investigating the new Windows Azure platform and when reading about the SQL Data Services the first thing that jumped into my head was could this be a small but major step in the decline of relational databases. For years OO databases have been touted as the grim reaper for relational databases but none have really broken into the mainstream, enter SQL Data Services.

As its a cloud service its got great scalability & capacity, its got communication (HTTP) covered with both REST & SOAP and finally its just part of the greater MS ecosystem for development support so you can use all the favorites such as Visual Studio, C#, etc. This all helps to lower MS’s entry into cloud computing as developers use familiar tools to develop applications.

So why a replacement for relational databases well just a few points,

• It follows a more abstract model with Authorities (~Server), Containers (~Database) & Entities (~Rows). Entities just define one instance of data and are stored in a generic name\value property bag.
• There really is no fixed schema in the containers or the Entitites just put in what you like.
• Bound to see ORMs being developed to query\update the entities making it easy to work with for devs and should see more maturity in projects such as Prevayler.
• Has internet scale which easily goes horizontal which can be done but is difficult with regular SQL Server.
• No items such as stored procs, identity columns & referential integrity. This forces people into thinking about their domains, entities, repositories, factories, services, etc. and this whole concept has really matured with the advent of  Domain Driven Design.
• Distributed cache will really help to make this whole thing work and you can see Microsoft are working on their own one called,  Velocity
• Its not just Microsoft look at Amazons SimpleDB service
• Folks should still worry about performance but  focus more on the business as massive performance gains in the  future with the likes of Intels  Dunnington Six cores on one chip.

I guess the really question is can the individual companies and the industry as a whole glue all these things together to bring around the demise of the relational database? Well I reckon there’s a better than 50/50 chance that within the decade it will become evident that they are sliding and look to no further proof of that but all Oracle acquisitions over the past few years.

Grid Computing = Silverlight + Parallel Extensions, nearly….

Using my trusty Google Reader I came across a couple of interesting projects which I hoped could be combined to solve very real world problems.

The Projects

Legion is a Grid Computing framework that uses the Silverlight CLR to execute user definable tasks.

Parallel Extensions

Is a library that allows developers to take advantage of multi-core machines while abstracting away the hard problems of multi-threading programming. It includes,

-Parallel Language Integrated Query (PLINQ)

This is declarative in nature and provides a SQL like syntax for your parallel work.

-Task Parallel Library (TPL)

This is imperative in nature and just makes it easier to write managed code that can take advantage of multiple processors.

The Problem

There are certain types of problems in the real world that are CPU intensive a classic one is determining risk. The Monte Carlo method is one way of tackling this problem through repeated random sampling to work out the risk.

It is used in industries such as finance to price complex financial derivatives and within portfolios to work out the Value at Risk (VAR) . In manufacturing it can be used to determine how many product units should be produced. Take a look at this Microsoft Excel Monte Carlo simulation example for an introductory primer as to whats it all about.

Monte Carlo methods are computationally intensive but naturally parallel and so using grid computing to solve the problem is a good fit.

A Potential Solution

There seems to be a good fit between the Legion project and Parallel Extensions to solve these class of problems but unfortunately the releases of Silverlight 2.0 & Parallel Extensions are just to far apart for now (see here). I hope these will be supported together in the future so opening up a whole new area for the average developer. For now I’d recommend keeping an eye on the Silverlight 2.0 release at MIX 08 this week so to see what threading support will be provided.

Multi-Tenancy & the Singleton Design Pattern

Since before xmas we have been working on a upgrading one of our core products which is built on the MS CRM platform (yes platform..working on a post about that topic).

Its the first time I have worked on a multi-tenant system and one of the first things you run into is not being able to use the standard singleton design pattern in a lot of common places. We use is for configuration information about the system but now that you can have multiple organizations (tenants) effectively sharing the same DLL loaded into memory means you have to think differently.

We took a very simple approach and just changed the existing singleton to have a new private member storing a generic list of configuration objects that can be accessed by passing the organization (tenant) as a parameter.

Anyone know of any better ways to accomplish this ?

22+ months of silence

Yep quite a long silence, I hope to do much better in the new year starting with getting this one post in for 2007.

I have been using my fair share of open source projects throughout the years from Cruise Control & Nant to Spring.Net & Rhino Mocks so I was very happy to see earlier this year the emergence of ALT.NET

It really shows that the .NET community has evolved and maturated over the years just as the Java community had done previously. I really like the core tenets of this group (lifted directly from the site),

1. You’re the type of developer who uses what works while keeping an eye out for a better way.
2. You reach outside the mainstream to adopt the best of any community: Open Source, Agile, Java, Ruby, etc.
3. You’re not content with the status quo. Things can always be better expressed, more elegant and simple, more mutable, higher quality, etc.
4. You know tools are great, but they only take you so far. It’s the principles and knowledge that really matter. The best tools are those that embed the knowledge and encourage the principles (e.g. Resharper.)

There has been some us versus them threads on other blogs\publications but no matter which bias you have I always liked the quote,

Let both sides explore what problems unite us instead of belaboring those problems which divide us

Happy New Year

INDA Waterford

Well made it to the INDA event in Waterford for my talk on .NET Build Systems last Friday despite my laptop of many years blowing up and losing all of my demos. But got most of the demos set up on another laptop and the talk went well.

Thanks goes to Brian Delahunty from Waterford INDA for organizing and looking after everyone at the event.

Finally I can whole heartily  recommend  Dooleys hotel as the place to be if you find yourself in Waterford on a Friday night. You just can’t beat the combination of one  man with his electronic keyboard and a bus load of retired folks from across the pond

Finding the sweet spot

So I’ve being playing with Indigo (WCF) since Beta 1 and now Beta 2 with its go live license. And I have to say I’m impressed with its simplicity and extensibility. Its going to reset the bar for developing the next generation of distributed applications.

You have all the major things like adhering to standards, good interoperability with the likes of Java via its implementation of the WS-* specification and critically its the unification of all the existing (ASMX, COM+, Remoting, MSMQ) .NET technologies for building distributed applicaions.

Its interesting so see some of the things that had to be put in place for all this to work. One lesser known one that caught my eye was about improvements to IIS in handling requests at the kernel level for performance and scalability reasons. But for me the real killer point is it allows multiple applications to listen on one IP port. Heres a link to a good article explaining what was done and why.

In terms of simplicitly one of the test apps I developed was a system so that the client talks to a service via an MSMQ so if the service is not running for some reason the client can still work away and when the service does start up it will start processing the messages in the queue in order. This type of work has now just become plain easy, all you need to do is change the config file and you can point the client to either the actual service or the MSMQ.

With respect to interoperability over the past year the work I’ve done has invloved ensuring that the web services I work on are Basic Profile 1.1 compliant and therefore I have worked a little with Java and specifically the Apache Axis toolkit. It really is good to see that the two stacks work well together. And with the announcement late last year of Service Component Architecture (SCA) and it implementing the WS-* specification as well then the future does look good for the two major platforms.

If you want to play around with WCF then the WinFx SDK now ships as part of the Windows SDK and can be downloaded from here. I reckon its going to be the sweet spot of .NET development for the next 5-10 years so enjoy.

INDA & Build Systems

As I mentioned in my first post I did my first public talk with .NET on Build Systems earlier in the month for INDA (Irish .NET Developers Alliance) in Cork. The one thing that struck me (and everyone else in the room) was I had alot to talk about and not enough time. So I hope to cut down on the slides and put more time into the demos next time. And this is going to be on the 17-02-2006 at a Waterford INDA event.

Thanks goes to Joe Gill and the folks from INDA for giving me the oppurtinuity to do the talk and also to Robert Burke from Micrsoft for the words of encouragement.