Reliable end to end tests with Selenium WebDriver

These are the slides and code from the talk I gave last Tuesday at the Sydney Alt.Net. If 2 links is to much of clicking for you then you can grab both slides and code from here as a zip archive.

Virtual desktops via Virtual Machines

I’ve got recently a very decent laptop (Lenovo W520, 16GB RAM) and one of the fastest SSDs on the market(Vertex 3 240GB). The machine is flying and basically does whatever I ask it to do in no time :).
I’ve always wanted to keep client projects, my personal projects and the rest (Outlook, pictures, etc) fully isolated. The main reason being the fact that every project requires some specific setup that I don’t want to keep forever and re-installing the whole machine after each piece of work doesn’t appeal to me. On top of that I don’t want to limit myself and don’t play with the latest technology just because it might break something on my system and I won’t be able to work.
So I’ve created 2 VMs. One for my personal stuff(VS, Ruby, Erlang, but no Outlook) and one for my current client. The idea is that I will create a separate VM for each new client. Another advantage of this approach is that if something happens to my laptop I can copy the VM to a different machine and be up and running within minutes. I backup all VMs twice a week to an external disk using CrashPlan. A nice side effect of using CrashPlan is that I get versioning so in case of emergency I don’t have to go all the way back to the initial state of the VM.
The VMs are obviously not as fast as the host with IO being the main bottleneck but they are fast enough so I can work comfortably. I can even run 2 VMs at the same time and they both start within seconds and are responsive enough so I can have fully functional VS open in each of them.
Have a look at screenshots from CrystalDiskMark to see the IO performance drop.

I started the whole experiment a week ago so we will see how it goes. Drop me a line if you know how to squeeze more performance out of VirtualBox.

Automated database deployments to AppHarbor

AppHarbor is trying to provide .NET developers with similar goodness that Heroku offers to their Ruby counterparts. At the moment deployment to AppHarbor has at least 6 distinct stages:

• Code gets pushed to AppHarbor
• Code gets compiled
• Unit tests get run
• Config transformations get applied
• Configuration Variables get applied
• Code gets actually deployed to a web server

At least one obvious stage is missing which is the database deployment. In .NET world there is nothing that would be even close to Ruby on Rails migrations but we can do something about that.
There is a small library called DbUp that simply makes sure that a given set of SQL scripts gets executed only once. This the ”up” part of RoR migrations. Now the question is when the database upgrade should be performed. There are basically 2 ways. Either before the new code gets deployed or after. I prefer the former option because as long as the database changes are non-breaking the app is fully functional at all times. If the database upgrade fails then the new version of the application won’t be deployed.
If the upgrade is done after the new version of the application is deployed then it needs to be done at the startup which prevents users from using the application for some period of time. What is more if the database upgrade fails then the new version of the application needs to be rolled back. In most cases the new code won’t work with the old database schema.
At the moment AppHarbor doesn’t have an explicit step when database upgrade can be performed. So to work around this problem I created a simple MsBuild target called UpgradeDatabaseInProduction that gets run when the application is successfully compiled. All the target does is extract production connection string from Web.Release.config and pass it to DbUp console app that performs the upgrade.

<target Name="UpgradeDatabaseInProduction"
        AfterTargets="MvcBuildViews"
        Condition="'$(Configuration)'=='Release'">
  <calltarget Targets="UpgradeDatabase" />
</Target>

<target Name="UpgradeDatabase">
  <xmlpeek XmlInputPath="$(MSBuildThisFileDirectory)Web.$(Configuration).config"
           Query="/configuration/connectionStrings/add[@name='database']/@connectionString">
    <output TaskParameter="Result" ItemName="ConnectionString" />
  </XmlPeek>
  <exec Command="$(MSBuildThisFileDirectory)..\libs\dbconsole-custom\DbUp.Console.exe -cs "@(ConnectionString)" -d $(MSBuildThisFileDirectory)SqlScripts" />
  
  <warning Text="Database upgraded" />
</Target>

This works only because AppHarbor build servers can talk  to AppHarbor database servers. To be honest it’s more of a hack than a proper solution :) but at least I can deploy the whole app (code + database changes) with single git push which is simply priceless.

YOW 2010 - loose thoughts

It doesn’t happen often that nearly every single talk at a conference is great and on top of that half of them are actually funny. That’s YOW 2010 for you summarized in one sentence :).

Justin Sheehy explained how to quickly narrow down the choice of database technologies that might be useful in a particular case. His method is based on a simple matrix of operations requirement (local, single server, distributed, etc) by data model (relational, column families, key/value, etc). Once this is done and there are only a few solutions on the table a more sophisticated and time consuming research can be conducted to choose the right solution. Every single NoSQL solution is different and a generic split SQL/NoSQL doesn’t really make sense. It’s all about tread-offs. It’s amazing how often this simple fact needs to be reminded.

Eric Evans talk was focused on the idea of bounded contexts. In other words a single enterprise model is an anti-pattern and is one of software engineering fallacies. Eric mentioned also a few disadvantages of doing big design upfront (AKA let’s build a great framework that less skilled devs can use) and postponing the initial release for long time. Nothing really new but it was well delivered.

Gregor Hohpe talked about trade-off decisions that Google had to make to be able to reach its current scale. He covered the whole spectrum of optimizations from data access at the disk level to minimize heat generation to skipping some longer than expected running parts of map reduce executions to make sure results are delivered in timely manner. When I asked Gregor if Google uses regular Pub/Sub or transactions he said that if there is a technology out there Google has built something on top it :). Just use the right tool for the job.

Second day started with Erik Meijer explaining coSQL (AKA NoSQL). It was a funny presentation about what NoSQL really is and how it relates to SQL. They both complement each other even in a mathematical sense hence the co part of coSQL. Additionally co is more positive than no and this makes Erik happy :).

Jim Webber talked passionately about how much he hates dislikes ESBs and how rarely ESB is the right tool for the job. His presentation was extremely funny but still full of useful information. The main point was that a custom built system can be cheaper (but not cheap) and less risky to deploy than an out of the box ESB which often requires a substantial up-front cost.

Dave Farley took us to the world of <1ms latency and speed of 100k per second. According to Dave this is achievable on commodity servers. The main enabler seems to be lack of synchronization, keeping as few threads per core as possible, keeping all the data in memory and keeping methods very short. 1 CPU can execute 1 billion instructions a second. That’s a lot and as long as we don’t waste it today hardware should be more than enough for needs of most consumers. The main message was that we underestimate what we can get from today hardware. I suppose this is only partially true because nowadays we rarely deploy apps on real hardware. In most cases all we see is a VM that shares the host with Gazillion of other VMs. This might the main reason why the perception of the current hardware capabilities is skewed.

After the conference there were 2 days of workshops. I spent the first day with Ian Robinson and Jim Webber learning about REST. What I believed constituted a fully blown RESTfull service was actually a very basic RESTfull service that scores only 1 out of 3 points in Richardson maturity model. Each of the levels has its place but obviously the higher you get the more you take advantage of the Web and that’s the whole purpose of using REST. REST is CRUDish as it mostly relies on GET, POST, PUT and DELETE. My initial thought was that this is very limiting but then it turned out that it doesn’t have to be. The same applies to lack of transactions. This can be worked around with proper structure of resources, meaningful response codes and proper use of HTTP idioms. Another important thing to keep in mind is that domain model shouldn’t be exposed directly. What you want to expose instead are resources that represent client – server interactions (use cases). In most cases O(resources) > O(domain classes) – notation by Jim Webber :). The Web is inherently based on polling (request/response) thus REST is not suitable for apps which require low latency. In this case you might want to use Pub/Sub.

The next day I attended a workshop with Corey Haines. This was a true hands-on workshop. I spent at least half a day writing code retreats, code katas and coding dojos. Going back to the very basics was surprisingly refreshing. I spent two 45 minutes long sessions constantly refactoring maybe 15 lines of code until most of if statements were gone and code read properly. You wouldn’t do this at work but the whole point of the exercise was to actually go over the line and try to come up with best possible code without feeling the time pressure.

At last but not least, the attendees were fantastic and every coffee/lunch break was full of valuable conversations.

I had an amazing time and YOW 2010 is the best conference I’ve ever been to.

Tags: YOW, REST, NoSQL, Loose Thoughts, Performance

Microsoft Azure on-premises in 2011

It looks like Microsoft is filling an obvious gap and its customers will be able to deploy Azure on their own machines in 2011. This should significantly speed up the adoption of Microsoft Cloud offering as it introduces an additional checkpoint half way through the migration and lowers the risk of the whole process. If on top of that other Cloud providers deploy Azure to their own data centers then the risk will be even smaller because ”vendor lock in” stops being such a big problem. These are all good changes and I’m really looking forward to how they affect the global market of Cloud Computing.

BTW I recently watched an interesting presentation by Chris Read from ThoughtWorks where he focuses on the Cloud from the Operations perspective. One of the takeaways that very often is not obvious to people is that there is no need to fire the infrastructure guys. You simply give them different, more creative tasks :)

 

Google CDN is not immune to being down

Just a reminder to myself that it’s good to have a fallback procedure when Google is down…not that it happens often :)
This happened to my blog a few days ago:

RubyMine is a real gem

Today I had to fix a piece of custom code inside of Redmine. I have very little experience with Ruby on Rails but I was able to get the app up and running with a debugger attached within 15 minutes.

I downloaded the latest version of RubyMine 2.5 EAP, installed it, pointed it to the folder with the app, selected production configuration and hit Debug. RubyMine analysed my Ruby setup and popped up a window with a notification that I’m missing some gems and the IDE can download and install them for me. I hit Ok and 5 minutes later I was debugging the app. Ruby on Rails experience on Windows is far from being perfect but RubyMine is simply awesome.