YOW 2011 - loose thoughts

Talks I attended at this year conference were focused on Agile, functional programming and low level performance. It turned out to be quite an interesting mix.

Marry Poppendieck talked about ways of supporting continuous design. One of the advices was to create an environment that facilities fair fights. To achieve this you need to hire good, diverse people. Their diversity let them solve complex problems because they can approach them from multiple angles. Once you have a team like that leave them alone and simply keep supporting them.

I haven’t learnt much new at Agile Architecture workshop delivered by Rebecca Wirfs-Brock but it was good to hear that I’m not some kind of weirdo that demands impossible :).

Simon Peyton Jones delivered 2 amazing and full of passion sessions about functional programming. I’ve done a bit of this type of programming before but I was surprised by Haskell type system and its ability to ensure lack of side effects at the compile time. On top of that on my way back to Sydney I had a good chat with Tony Morris who told me about a project where he used functional programming to create a composable data access layer. Composablity and strict control over side effects is enough to push Haskell to the top of my list of languages to play with :).

Inspired by Coderetreat Brisbane organized by Mark Ryall I’ve decided to use Conway’s Game of Life as a problem that I will try to solve every time I learn a new language. It’s worked for CoffeeScript and I hope it will work Haskell.

The end of the conference was filled by performance tips from .NET (Joel Pobar and Patrick Cooney) and Java (Martin Thompson) land. Both sessions emphasized that computation is cheap and memory access is slow. By simply keeping all required data in L1/L2 CPU cache you can cut the execution time by half. Obviously this was presented using a micro benchmark but still it is something that it’s worth keeping in mind.

Functional languages rely on immutable data structures to isolate processing and in this way control side effects. Unfortunately this means that a lot of memory is pushed around and this has a significant influence on the overall performance. Again, it’s all about trade-offs :).

Martin talked about “mechanical sympathy” which is boils down to having at least basic understanding of hardware your software runs on. Without it your software not only doesn’t take full advantage of the underlying hardware but often works against it which has severe impact on the overall performance. It’s one of those advices that most of us won’t use on daily basis as most our infrastructure is in the Cloud but it’s good to keep it in mind.  

We’ve been told multiple times that “free lunch is over” and CPUs are not getting any faster. Martin proved that this is not correct. It’s true that we are not getting more MHz, we are actually getting less MHz than we used to but CPUs get faster because every year engineers design better chips. The message Martin is trying to send is that we should prove/disprove assumptions based on evidence and not beliefs.

All in all it was another good conference and I will try to attend it next year.

YOW 2010 - loose thoughts

It doesn’t happen often that nearly every single talk at a conference is great and on top of that half of them are actually funny. That’s YOW 2010 for you summarized in one sentence :).

Justin Sheehy explained how to quickly narrow down the choice of database technologies that might be useful in a particular case. His method is based on a simple matrix of operations requirement (local, single server, distributed, etc) by data model (relational, column families, key/value, etc). Once this is done and there are only a few solutions on the table a more sophisticated and time consuming research can be conducted to choose the right solution. Every single NoSQL solution is different and a generic split SQL/NoSQL doesn’t really make sense. It’s all about tread-offs. It’s amazing how often this simple fact needs to be reminded.

Eric Evans talk was focused on the idea of bounded contexts. In other words a single enterprise model is an anti-pattern and is one of software engineering fallacies. Eric mentioned also a few disadvantages of doing big design upfront (AKA let’s build a great framework that less skilled devs can use) and postponing the initial release for long time. Nothing really new but it was well delivered.

Gregor Hohpe talked about trade-off decisions that Google had to make to be able to reach its current scale. He covered the whole spectrum of optimizations from data access at the disk level to minimize heat generation to skipping some longer than expected running parts of map reduce executions to make sure results are delivered in timely manner. When I asked Gregor if Google uses regular Pub/Sub or transactions he said that if there is a technology out there Google has built something on top it :). Just use the right tool for the job.

Second day started with Erik Meijer explaining coSQL (AKA NoSQL). It was a funny presentation about what NoSQL really is and how it relates to SQL. They both complement each other even in a mathematical sense hence the co part of coSQL. Additionally co is more positive than no and this makes Erik happy :).

Jim Webber talked passionately about how much he hates dislikes ESBs and how rarely ESB is the right tool for the job. His presentation was extremely funny but still full of useful information. The main point was that a custom built system can be cheaper (but not cheap) and less risky to deploy than an out of the box ESB which often requires a substantial up-front cost.

Dave Farley took us to the world of <1ms latency and speed of 100k per second. According to Dave this is achievable on commodity servers. The main enabler seems to be lack of synchronization, keeping as few threads per core as possible, keeping all the data in memory and keeping methods very short. 1 CPU can execute 1 billion instructions a second. That’s a lot and as long as we don’t waste it today hardware should be more than enough for needs of most consumers. The main message was that we underestimate what we can get from today hardware. I suppose this is only partially true because nowadays we rarely deploy apps on real hardware. In most cases all we see is a VM that shares the host with Gazillion of other VMs. This might the main reason why the perception of the current hardware capabilities is skewed.

After the conference there were 2 days of workshops. I spent the first day with Ian Robinson and Jim Webber learning about REST. What I believed constituted a fully blown RESTfull service was actually a very basic RESTfull service that scores only 1 out of 3 points in Richardson maturity model. Each of the levels has its place but obviously the higher you get the more you take advantage of the Web and that’s the whole purpose of using REST. REST is CRUDish as it mostly relies on GET, POST, PUT and DELETE. My initial thought was that this is very limiting but then it turned out that it doesn’t have to be. The same applies to lack of transactions. This can be worked around with proper structure of resources, meaningful response codes and proper use of HTTP idioms. Another important thing to keep in mind is that domain model shouldn’t be exposed directly. What you want to expose instead are resources that represent client – server interactions (use cases). In most cases O(resources) > O(domain classes) – notation by Jim Webber :). The Web is inherently based on polling (request/response) thus REST is not suitable for apps which require low latency. In this case you might want to use Pub/Sub.

The next day I attended a workshop with Corey Haines. This was a true hands-on workshop. I spent at least half a day writing code retreats, code katas and coding dojos. Going back to the very basics was surprisingly refreshing. I spent two 45 minutes long sessions constantly refactoring maybe 15 lines of code until most of if statements were gone and code read properly. You wouldn’t do this at work but the whole point of the exercise was to actually go over the line and try to come up with best possible code without feeling the time pressure.

At last but not least, the attendees were fantastic and every coffee/lunch break was full of valuable conversations.

I had an amazing time and YOW 2010 is the best conference I’ve ever been to.

Tags: YOW, REST, NoSQL, Loose Thoughts, Performance