tag:blogger.com,1999:blog-1681474838849690002024-02-20T03:38:17.125+11:00Pawel Pabich's blogRandom thoughts on computer things I'm passionate about.Pawel Pabichhttp://www.blogger.com/profile/05213210256288888149noreply@blogger.comBlogger159125tag:blogger.com,1999:blog-168147483884969000.post-88143884583703840562016-01-28T00:37:00.002+11:002020-10-26T10:35:50.974+11:00Time in distributed systems<br />
Time in a program that runs on a single machine and uses local physical clock is already a tricky concept. Just think about time zones, daylight saving and the fact they can change at any time because they are governed by politicians. Distributed programs add one more dimension to this problem space because there isn’t a single clock that decides what time it is. <br />
<br />
The goal of this post is to create a short summary of some of the academic papers written on this topic and in the process to better understand the challenges faced by the builders of distributed systems. I hope I won’t be the only person that will benefit from it :).<br />
<br />
As end users we take the notion of time for granted and rely on our physical clocks to determine whether event A happened before event B. This works great most of the time because in our day-to-day lives we don’t really need high precision time readings. The fact that my watch is running a minute or two late comparing to some other watch doesn’t really matter all that much. Computer systems are different because they operate at much higher speed and even small variations can affect the after-before relationship. The clocks that are used in computers nowadays tend to drift enough that distributed systems can’t really rely on them. At least not as the only source of truth when it comes to time.<br />
<br />
To solve this problem Lasslie Lamport introduced an idea of virtual time that can be implemented using logical clocks (<a href="http://research.microsoft.com/en-us/um/people/lamport/pubs/time-clocks.pdf">Time, Clocks, and the Ordering of Events in a Distributed System</a>). His algorithm implements an invariant called Clock Condition: <em>if event A happened before event B then the value of logical clock when A happened is lower than the value of logical clock when B happened, in short If A –> B then C(A) < C(B)</em>. This invariant provides one of many possible, consistent partial orderings of events in a distributed system. The drawback of the ordering established based on Clock Condition is that C(A) < C(B) doesn’t mean A happened before B. Moreover, as far the end user is concerned B might even have happened before A or might have happened concurrently with A. <br />
<br />
Vector Clocks, introduced by Colin J. Fidge (<a href="http://zoo.cs.yale.edu/classes/cs426/2012/lab/bib/fidge88timestamps.pdf" title="http://zoo.cs.yale.edu/classes/cs426/2012/lab/bib/fidge88timestamps.pdf">Timestamps in Message-Passing Systems That Preserve the Partial Ordering</a>), fill the shortcoming mentioned above by providing an invariant that is a reverse of Clock Condition: <em>if C(A) < C(B) then A –> B</em><strong>. </strong>Vector Clock is built on top of logical clock designed by Lasslie Lamport. The main difference is that instead of using just one logical clock for the whole system it uses one logical clock per process in the system. In this way it tries to capture distributed state of time in the system. Vector Clock stores a list of values so the original invariant should be rewritten to<em> if VC(A) < VC(B) then A –> B where VC(A)[i] <= VC(B)[i] and j exists that VC(A)[j] < VC(B)[j]</em>. E.g. [1,2,<strong>3</strong>] < [1,2,<strong>4</strong>]. If neither A->B nor B->A (eg. VC(A) = [1,2,1] and VC(B) = [2,1,2]) then A and B are considered to be concurrent events. <br />
<br />
Even with Vector Clocks we can end up with concurrent events that from the end user perspective are not concurrent. To mitigate this problem we can make a full circle and employ help from physical clocks or we can simply let the user order the events for us. This is how <a href="http://basho.com/products/#riak">Riak</a> deals with write conflicts. It accepts concurrent writes to the same key but then the conflict needs to be resolved at the read time. <br />
<br />
The first step towards fixing a problem is to realize that there is a problem and this is what Vector Clocks provide because they can detect conflicts. If you don’t think this is a big deal then follow <a href="https://aphyr.com/tags/Jepsen">Kyle Kingsbury’s Jepsen series on distributed systems correctness</a> to see how many mainstream (database) systems can lose data and not know about it.Pawel Pabichhttp://www.blogger.com/profile/05213210256288888149noreply@blogger.com0tag:blogger.com,1999:blog-168147483884969000.post-72540387623562254132015-09-09T09:30:00.001+10:002015-09-09T13:21:28.457+10:00Random lessons learnt from building large scale distributed systems, some of them on Azure. <p>This post is based on more than one project I’ve been involved in but it was triggered by a recent project where I helped build an IoT system that is capable of processing and storing tens of thousands of messages a second. The system was built on top of Azure Event Hubs, Azure Web Jobs, Azure Web Sites, HDInsight implementation of HBase, Blob Storage and Azure Queues.</p> <p>As the title suggests it is a rather random set of observations but hopefully they all together form some kind whole :).</p> <h4>Keep development environment isolated, even in the Cloud</h4> <p>A fully isolated development environment is a must have. Without it developers step on each others toes which leads to unnecessary interruptions and we all know that nothing kills productivity faster than context switching. When you build an application that uses SQL Server as a backend you would not think about creating one database that is shared by all developers because it’s relatively simple to have an instance of SQL Server running locally. When it comes to services that live only in the Cloud this is not that simple and often the first obvious solution seems to be to share them within the team. This is the preferred option especially in places that are not big on automation.</p> <p>Another solution to this problem is to mock Cloud services. I strongly recommend <strong>against</strong> it as it will hide a lot of very important details and debugging them in CI (or UAT or Production) is not an efficient way of working.</p> <p>If sharing is bad then then each developer could get a separate instance of each Cloud service. This is great in theory but it can get expensive very quickly. E.g. the smallest HBase cluster on Azure consists of 6 VMs.</p> <p>The sweet spot seems to be a single instance of a given Cloud service with isolation applied at the next logical level. </p> <ul> <li>HBase – table(s) per developer <li>Service Bus - event hub per developer <li>Azure Search – index(es) per developer <li>Blob Storage – container per developer <li>Azure Queues – queue(s) per developer</li></ul> <p>Each configuration setting that is developer specific can be encapsulated in a class so the right value is supplied to the application automatically based on some characteristic of the environment in which the application is running, e.g. machine name. </p> <h4>Everything can fail</h4> <p>The number of moving parts increases with scale and so they are chances of one or more of them failing. This means that every logical component of the system needs to be able to recover automatically from an unexpected crash of one or more of its physical components. Most of PaaS come with this feature out of the box. In our case the only part we had to take care of was the processing component. This was relatively easy as Azure lets you continuously run N instances of a given console app (Web Job) and restarts them when they crash or redeploys them when the underlying hardware is not healthy.</p> <p>When the system recovers from a failure it should start where it stopped which means that it possibly needs to retry the last operation. In a distributed system transactions are not available. Because of that if an operation modifies N resources then all of them should be <a href="https://en.wikipedia.org/wiki/Idempotence">idempotent</a> so retries don’t corrupt the state of the system. If there is a resource that doesn’t provide such a guarantee then it needs to be modified as the last one. This is not a bullet proof solution but it works most of the time.</p> <ul> <li>HBase – writes are done as <a href="https://en.wikipedia.org/wiki/Merge_(SQL)">upserts</a> <li>Event Hub - checkpointing is idempotent <li>Blob Storage – writes can be done as <a href="https://en.wikipedia.org/wiki/Merge_(SQL)">upserts</a> <li>Azure Queues – sending messages is <strong>NOT</strong> idempotent so the de-duplication needs to be done on the client side</li></ul> <p>Sometimes failure is not immediate and manifests itself in a form of an operation running much longer than it should. This is why it is important to have relatively short timeouts for all IO related operations and have a strategy that deals with them when the occur. E.g. if latency is important then a valid strategy might be to drop/log data currently being processed and continue. Another option is to retry the failed operation. We found that the default timeouts in the client libraries where longer than what were comfortable with. E.g. default timeout in HBase client is 100 seconds. </p> <h4>Don't lose sight of the big picture</h4> <p>It is crucial to have access to an aggregated view of logs from the whole system. I can’t imagine having tail running for multiple components and doing the merge in my head. Tools like <a href="http://getseq.net/">Seq</a> are worth every cent you pay for them. Seq is so easy to use that we used it also for local development. Once the logging is setup I recommend spending 15 minutes a day watching logs. <u>Well instrumented system tells a story and when the story does not make sense you know you have a problem</u>. We have found several important but hard to track bugs just by watching logs. </p> <p>And log errors, always log errors. A new component should not leave your dev machine unless its error logging is configured.</p> <h4>Ability to scale is a feature</h4> <p>Every application has performance requirements but they are rarely explicitly stated and then the performance problems are discovered and fixed when the application is already in Production. As bad as it sounds in most cases this is not a big deal and the application needs only a few tweaks or we can simply throw more hardware at it.</p> <p>This is not the case when the application needs to deal with tens of thousands of messages a second. Here performance is a feature and needs to be constantly checked. Ideally a proper load test would be run after each commit, the same way we run CI builds, but from experience once a day is enough to quickly find regressions and limit the scope of changes that caused them. </p> <h4></h4> <h4>Predictable load testing is hard</h4> <p>Most CI builds run a set of tests that check correctness of the system and it doesn’t matter whether the tests run for 0.5 or 5 or 50 seconds as long as all assertions pass. This means that the performance of the underlying hardware doesn’t affect the outcome of the build. This is not true when it comes to load testing where slower than usually hardware can lead to false negatives which translate to wasted time spent on investigating problems that don’t exist. </p> <p>In ideal world the test would run on isolated hardware but this is not really possible in the Cloud which is a shared environment. Azure is not an exception here. What we have noticed is that using high spec VMs and running load test at the same time of the day in the morning helped keep the performance of the environment consistent. This is a guess but it looks like the bigger the VM the higher the chance for that VM to be the only VM on its host. Even with all those tweaks in place test runs with no code changes would differ by around 10%. Having load testing setup from the beginning of the project helps spot outliers and reduce the amount of wasted time.</p> <h4>Generating enough of correct load is hard</h4> <h4></h4> <p>We started initially with Visual Studio Load Test but we didn’t find a way to fully control the data it uses to generate load. All we could do was to generate all WebRequests up front which is a bit of problem at this scale. We couldn’t re-use requests as each of them had to contain different data.</p> <p><a href="http://jmeter.apache.org/">JMeter</a> doesn’t suffer from this problem and was able to generate enough load from just one large VM. The templating language is a bit unusual but it is pretty powerful and at the end of the day this is what matters the most. </p> <p>JMeter listeners can slow down significantly the load generator. After some experimentation we settled on Summary Report and Save Responses to a file (only failures) listeners. They gave us enough information and had very small impact on the overall performance of JMeter.</p> <p>The default batch file limits JMeter heap size to 512MB which is not a lot. We simply removed the limit and let the JVM to pick one for us. We used 64 bit JVM which was more than happy to consume 4GB of memory.</p> <h4>Don’t trust average values</h4> <p>Surprising number of tools built for load testing shows average as the main metric. Unfortunately average hides outliers and it is a bit like looking at 3D world through 2D glasses. Percentiles is a much better approach as they show the whole spectrum of results and help make an informed decision whether it’s worth investing more in performance optimization. This is important because performance work never finishes and it can consume infinite amount of resources.</p> <h4>Feedback cycle on performance work is looooooooooooong</h4> <p>It is good to start load testing early as in most cases it is impossible to make a change and re-run the whole load test locally. This means each change will take quite a bit of time to be verified (in our case it was 45 minutes) . In such a case it is tempting to test multiple changes at once but it is a very slippery slope. In a complex system it is hard to predict the impact a given change will have on each component so making assumptions is very dangerous. As an example, I sped up a component responsible for processing data which in turn put more load on the component responsible for storing data which in turn made the whole system slower. And by change I don’t always mean code change, in our case quite a few of them were changes to the configuration of Event Hub and HBase.</p> <h4>Testing performance is not cheap</h4> <p>Load testing might require a lot resources for relatively short period of time so generally it is a good idea to automatically create them before the test run and then automatically dispose them once the test run is finished. In our case we needed the resources for around 2h a day. This can be done with Azure though having to use three different APIs to create all required resources was not fun. I hope more and more services will be exposed via Resource Manager API. </p> <p>On top of that it takes a lot of time to setup everything in a fully automated fashion but if the system needs to handle significant load to be considered successful then performance related work needs to be part of the regular day-to-day development cycle.</p> <h4>Automate ALL THE THINGS</h4> <p>I’m adding this paragraph for completeness but I hope that by now it is obvious that automation is a must when building a complex system. We have used Team City, Octopus Deploy and a thousand or so lines of custom PowerShell code. This setup worked great. </p> <h4>Event Hubs can scale</h4> <p>Event Hubs are partitioned and the throughput of each partition is limited to 1k messages a second. This means that Event Hub shouldn’t be perceived as a single fat pipe that can accept any load that gets thrown at it. For example, an Event Hub that can in theory handle 50k messages a second might start throwing exceptions when the load reaches 1k messages a second because all messages happen to be sent to the same partition. That’s why it’s very important to have as even distribution of data that is used to compute PartitionKey as possible. To achieve that try a few hashing algorithms. We used SHA1 and it worked great.</p> <p>Each Event Hub can have up to 20 independent consumer groups. Each consumer group represents a set of pointers to the message stream in the Event Hub. There is one pointer per partition. This means that there can be only one Event Hub reader per partition per consumer group. </p> <p>Let’s consider an Event Hub with 50 partitions. If the readers need to do some data processing then a single VM might not be enough to handle the load and the processing needs to distributed. When this happens the cluster of VMs needs to figure out which VM will run readers for which partitions and what happens when one of the readers or/and VMs disappears. Reaching consensus in a distributed system is a hard problem to solve. Fortunately, Microsoft provided us with Microsoft.Azure.ServiceBus.EventProcessorHost package which takes care of this problem. It can take couple of minutes for the system to reach consensus but other than that it just works. </p> <h4>HBase can scale</h4> <p>HBase is a distributed data base where data is stored in <a href="https://en.wikipedia.org/wiki/Lexicographical_order">lexicographical order.</a> Each node in the cluster stores an ordered subset of the overall data. This means that data is partitioned and HBase can suffer from the same problem as the one described earlier in the context of Event Hubs. HBase partitions data using row key that is supplied by the application. As long as row keys stored at more or less the same time don’t share the same prefix then the data should not be sent to the same node and there should be no hots spots in the system. </p> <p>To achieve that the prefix can be based on a hash of the whole key or part of it. The successful strategy needs to take into account the way the data will be read from the cluster. For example, if the key has a form of PREFIX_TIMESTAMP_SOMEOTHERDATA and in most cases the data needs to be read based on a range of dates then the prefix values needs to belong to a predicable and ideally small set of values (e.g. 00 to 15). The query API takes start and end key as input so to read all values between DATE1 and DATE2 we need to send a number of queries to HBase which equals the number of values the prefix can have. In the sample above that would be 16 queries. </p> <p>HBase exposes multiple APIs. The one that is the easiest to use is REST API called Stargate. This API works well but its implementation on Azure is far from being perfect. Descriptions of most of the problems can be found on the <a href="https://github.com/hdinsight/hbase-sdk-for-net/issues">GitHub page of the SDK project</a>. Fortunately, Java API works very well and it is actually the API that is recommended by the documentation when the highest possible level of performance is required. From the tests we have done Java API is fast, reliable and offers predictable latency. The only drawback is that it is not exposed externally so HBase and its clients need to be part of the same VNet. </p> <p>BTW Java 8 with lambdas and streams is not that bad :) even though the lack of var keyword and checked exceptions are still painful. </p> <h4>Batch size matters a lot</h4> <p>Sending, reading and processing data in batches can significantly improve overall throughput of the system. The important thing is to find the right size of the batch. If the size is too small then the gains are not big and if the size too big then then a change like that can actually slow down the whole system. It’s taken us some time to find the sweat spot in our case.</p> <ul> <li>Single Event Hub message has a size limit of 256 KB which might be enough to squeeze multiple application messages in it. On top of that Event Hub accepts multiple messages in a single call so another level of batching is possible. Batching can also be specified on the read side of the things using EventHostProcessor configuration. <li>HBase accepts multiple rows at a time and its read API can return multiple rows at a time. Be careful if you use its REST API for reading as it will return all available data at once unless the batch parameter is specified. Another behaviour that took as by surprise is that batch represents the number of <strong>cells (not rows)</strong> to return. This means a single batch can contain incomplete data. More details about the problem and possible solution can be found <a href="https://github.com/hdinsight/hbase-sdk-for-net/issues/40">here</a>. </li></ul> <h4>Web Jobs can scale</h4> <p>Azure makes it very easy to scale up or down the number of running instances. The application code doesn’t have to be aware of this process but it might if it is beneficial to it. Azure notifies jobs before it shuts them down by placing a file in their working folder. <a href="https://github.com/Azure/azure-webjobs-sdk">Azure Web Jobs SDK</a> comes with useful code that does the heavy lifting and all our custom code needed to do is to respond to a notification. We found this very useful because Event Hub readers lease partitions for a specific amount of time so if they are not shutdown gracefully the process of reaching distributed consensus takes significantly more time.</p> <h4>Blob Storage can scale</h4> <p>We used Blob Storage as a very simple document database. It was fast, reliable and easy to use.</p> Pawel Pabichhttp://www.blogger.com/profile/05213210256288888149noreply@blogger.com2tag:blogger.com,1999:blog-168147483884969000.post-87761639586418515922015-05-12T22:35:00.001+10:002015-05-12T22:36:11.701+10:00Trying to solve a real world problem with Azure Machine Learning<p> </p> <p>I’ve spent last couple of days playing with <a href="http://azure.microsoft.com/en-US/services/machine-learning/">Azure Machine Learning</a> (AML) to find out what I can do with it. To do so I came up with an idea that might or might not be useful for our recruitment team. I wanted to predict whether a given candidate would be hired by us or not. I pulled historical data and applied a few statistical models to it. The end result was not as good as I expected it to be. I guess this was caused by the fact that my initial dataset was too small.</p> <h2><font size="4">How to start</font></h2> <p>I’ve done some ML and AI courses as part of my university degree but that was ages ago :) and I needed to refresh my knowledge. Initially I wanted to learn about algorithms so I started reading <a href="http://shop.oreilly.com/product/0636920028529.do">Doing Data Science</a> but it was a bit too dry and I needed something that would be focused more on doing than theory. Recently released <a title="http://www.microsoftvirtualacademy.com/ebooks#9780735698178" href="http://www.microsoftvirtualacademy.com/ebooks#9780735698178">Microsoft Azure Essentials: Azure Machine Learning</a> was exactly what I was looking for. The book is easy to read and explains step by step how to use AML.</p> <h2><font size="4">Gathering initial data is hard</font></h2> <p>For some reason I thought that once I have data extracted from the source system it should be easy to use it. Unfortunately this was not true as the data was inconsistent and incomplete. I spent significant amount of time filling the gaps and sometimes even removing data when it did not make sense in the context of the problem I was trying to solve. At the end of the whole process it turned out that I removed 85% of my original dataset. This made me question whether I should be even using it. But this was a learning exercise and nobody was going to make any decisions based on it so I didn’t give up.</p> <p>Data cleansing was a mundane task but at the end of it I had a much better understanding of the data itself and relationship between its attributes. It’s a nice and I would say important side effect of the whole process. Looking at data in its tabular form doesn’t tell the full story so as <a href="https://twitter.com/joevernon/status/546068632628101120">someone mentioned on the Internet it is important to visualize it</a>.</p> <p><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh35nKQZuMo5tp1YDodXLwqGYMT5JzmhTIXdHhhDt7a9sKw6Cg9yjrCRrl3DDzBv_IG5WJuf6GgFgB1w8VzCrFaCllgwxlV5mbSWBuLpgQc7Vu7l1jeUpZZwH6dKdW4JgyELR5Zlz6gnOY/s1600-h/data7.png"><img title="data" style="border-left-width: 0px; border-right-width: 0px; background-image: none; border-bottom-width: 0px; padding-top: 0px; padding-left: 0px; display: inline; padding-right: 0px; border-top-width: 0px" border="0" alt="data" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiX8oPNg8FGB0uyF9S_wcUD4-kO30ANqhoXy1Lj82j43kaYMA7PQ8KmDioz0vzmkYfvxXPb7O75y7Ihh0ohCm1POetslUwwNPWIhguXBl5rPir80IbDKl0vKKcIRVi1NX-PDnicgrhJjMs/?imgmax=800" width="101" height="107"></a></p> <p><a href="https://studio.azureml.net/">AML Studio</a> solves this problem very well. Every dataset can be quickly visualized and its basic statistical characteristics (histogram, mean, median, min, max, standard deviations) are displayed next to the charts. I found this feature very useful. It helped me spot a few mistakes in my data cleansing code. </p> <p><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhHpJiUeqAqivE-r7rTze1in73TCTLzHj1oYc1jIMWnYNvskNXP6nP2BGuSqvasjKUjisoCNwnRDXLDwKSbjdBjJcHFgoCmI3AOpSNb9gIzZGadbLTkEOPH-a4PHSorSxDZdcPKk7-zQrE/s1600-h/visulization8.png"><img title="visulization" style="border-left-width: 0px; border-right-width: 0px; background-image: none; border-bottom-width: 0px; padding-top: 0px; padding-left: 0px; display: inline; padding-right: 0px; border-top-width: 0px" border="0" alt="visulization" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEitxtgbJOKDyi4uc6WL4iUlyGGIbrsVJVVq0EHIRq-_wNmTqFARx1dK9_NJu-Vm1obNYlznNe-UQnLz0Qs_mBpy1gD6Xk6L2Fk8u-gGXP4sKm_nUdzPQP15zcpwP0ddP1aN_gRcKPdlAEM/?imgmax=800" width="223" height="244"></a></p> <h2><font size="4">Trying to predict the future</font></h2> <p>Once the data has been cleaned I opened <a href="https://studio.azureml.net/">AML Studio</a> which is an online tool that lets you design ML experiments. It took me only a couple of minutes to train a Two-Class Boosted Decision Tree model. To do it in AML Studio you need to take the following steps:</p> <ol> <li>Import data <li>Split data into two sets: training set and test set <li>Pick the model/algorithm <li>Train the model using the training set <li>Score the trained model using the test set <li>Evaluated how well the the model has been trained.</li></ol> <p>Each of the steps is represented by a separate visual element on the design surface which makes it very easy to create and evaluate experiments.</p> <p><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj4Pd2BoQc0KkeTw5eS2skhyphenhyphenRLwmCCV1MLRdwQy5xv24zjLio9iuOdbN3owg8fCYD3ZKVeOZtTlH4ABmFKXdDDzz5UMht7ua5sFfnx56iU0KmMl23RmXC1mLT3sFlDuCle4AuYJ8rZSd1Q/s1600-h/design3.png"><img title="design" style="border-left-width: 0px; border-right-width: 0px; background-image: none; border-bottom-width: 0px; padding-top: 0px; padding-left: 0px; display: inline; padding-right: 0px; border-top-width: 0px" border="0" alt="design" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhhpMTh7MWOhti-pU0MyfZEslqQIBo3wyFRpWCMsMeBrLtTd0dFn-oh_Nr3jmD1h-bfBMDx3ksflLfzuhcu018DQFnAqKVFrfx5gLNtRQ5V6LJxlspmyqOfXspu3fbMRX8vNRZD189Yn1c/?imgmax=800" width="244" height="209"></a></p> <p>I tried a few more algorithms but none of them gave me better results. This part was more time consuming than it should be. Unfortunately at the moment there is no API that would allow me to automate the processes of creating and running experiments. Somebody already mentioned that as a possible improvement so there is a chance it will be added in the future: <a href="http://feedback.azure.com/forums/257792-machine-learning/suggestions/6179714-expose-api-for-model-building-publishing-automat">http://feedback.azure.com/forums/257792-machine-learning/suggestions/6179714-expose-api-for-model-building-publishing-automat</a>. It would be great to automatically evaluate several models with different parameters and then pick the one that gives best predictions. </p> <h2><font size="4">But it is Drag & Drop based programming!</font></h2> <p>In most cases Drag & Drop based programming demos well, sells well but has serious limitations when it comes to its extensibility and maintainability. “Code” written in this way is hard or impossible to test and offers poor or no ability to reuse logic so “copy and paste” approach tends to be used liberally. At first glance AML seems to belong to this category but after taking a closer look we can see that there is light at the end of the tunnel. </p> <p>Each AML experiment can be exposed as an independently scalable HTTP endpoint with very well defined input and output. This means that each of them can be easily tested in isolation. This solves the first problem. The second problem can be solved by keeping the orchestration logic outside of AML (e.g. C# service) and treating each AML endpoint as <a href="http://en.wikipedia.org/wiki/Pure_function">a pure function</a>. It this way we let AML do what it is good at without introducing unnecessary complexity of implementing “ifs” visually. </p> <h2></h2> <h2><font size="4">I’ve trained the perfect model, now what ?</font></h2> <p>As I mentioned in the pervious paragraph, the trained mode can be exposed as a HTTP endpoint. On top of that AML creates an Excel file that is a very simple client build on top of the HTTP endpoint. As you type parameters the section with predicted values get refreshed. And if you want to share your awesome prediction engine with the rest of the world you can always upload it to <a href="http://gallery.azureml.net/">AML Gallery</a>. One of the available APIs there is a service that calculates person age based on their photo. </p> <p><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgT0Cpru8J0a2Jsyc0DGJGouwvMDekv3qcFpX2VCAbjp20TXT30F-kvsnQOIG_iIgSdHeWljHunoP9C94OGY4q9_BJOSa_UaDzdaTvrGbwA7g_goVCSrALIYTk0LkCxG4GONg4LhHBxY6s/s1600-h/excel3.png"><img title="excel" style="border-left-width: 0px; border-right-width: 0px; background-image: none; border-bottom-width: 0px; padding-top: 0px; padding-left: 0px; display: inline; padding-right: 0px; border-top-width: 0px" border="0" alt="excel" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhhD1bDBO6Eg6xBcoKkJgcmB4EaWB1M1WAYzvY6V0AcLEA0uipICnZsErYe2ITYekOmujEwtPGAqjz5tCOhT5TiS7UboRIT-dJAJlrq_DpZ2LCkmJNavokHFYYj31IRkR1b8Gw2-lxRY6Y/?imgmax=800" width="244" height="149"></a></p> Pawel Pabichhttp://www.blogger.com/profile/05213210256288888149noreply@blogger.com0tag:blogger.com,1999:blog-168147483884969000.post-16174731319126674892014-06-01T23:14:00.001+10:002014-06-09T22:06:50.313+10:00How to run Selenium tests in parallel with xUnit<p>I’m a big fan of automated testing, unfortunately end to end tests, like the ones that use Selenium, have always been a problem. They are slow and their total execution time very quickly goes beyond of what most teams would accept as a reasonable feedback cycle. On my current project our Full CI build takes nearly 2 hours. Every project of a significant size I know of, has this problem, some of them have 4h+ builds. Assuming you have only one test environment then you can test up to 4 changes a day. Not a lot when daily deployments to Production are the ultimate goal. There are many solutions to this problem but most of them are based on splitting the list of tests into smaller subsets and executing them on separate machines (e.g. Selenium grid). This works but from my experience has two major drawbacks:</p>
<ul>
<li>it is complex to setup and gather results
<li>it can’t be replicated on a single machine, which means
<ul>
<li>I can’t debug easily tests that fail only because they are run together with some other tests
<li>I can’t quickly run all tests locally to make sure my changes do not break anything</li></ul></li></ul>
<p>Another added benefit of running tests in parallel is that it becomes a very basic load test. This isn’t going to be good enough for sites that need to handle thousands of concurrent users, but it might be just what you need when the number of users is in low hundreds.</p>
<p>In the past I used <a href="https://code.google.com/p/mb-unit/">MbUnit</a> with VS 2010 and it worked great. I was able to run 20 Firefox browsers on a single VM. Unfortunately the project seems to be abandoned and it doesn’t work with VS 2013. Until recently, none of the major testing frameworks supported running tests in parallel. As far as I remember even <a href="https://github.com/xunit/xunit">xUnit</a> 2.0 initially wasn’t meant to support this feature but this has changed and now it does.</p>
<p>So where is the catch? After spending some time with xUnit code my impression is that the core of xUnit and newly added parallelism were built with unit testing in mind. There is nothing wrong with this approach and I fully understand why it was done this way but it makes running tests that rely on limited resources (e.g. browser windows) challenging.</p>
<p>The rest of the blog posts focuses on how to overcome those challenges. A sample app that implements most of the described ideas is available on GitHub: <a title="https://github.com/pawelpabich/SeleniumWithXunit" href="https://github.com/pawelpabich/SeleniumWithXunit">https://github.com/pawelpabich/SeleniumWithXunit</a>. The app is based on custom fork of xUnit: <a title="https://github.com/pawelpabich/xunit" href="https://github.com/pawelpabich/xunit">https://github.com/pawelpabich/xunit</a>.</p>
<h4>How to run tests in parallel</h4>
<p>To execute tests in <a href="https://github.com/pawelpabich/SeleniumWithXunit">the sample app</a> in parallel in VS 2013 (tested with Update 2) you need to install <a href="https://github.com/pawelpabich/SeleniumWithXunit/blob/master/CustomBuildOfxUnit/xunit.runner.visualstudio.vsix">xUnit runner for VS</a> and <a href="https://download.mozilla.org/?product=firefox-stub&os=win&lang=en-US">Firefox</a>. Once this is done, compile the app, open Test Explorer (TEST –> Windows), select all tests and run them. You should see two browsers open at more or less the same time.</p>
<p><u>WARNING</u>: If you want to run Selenium tests on a machine which has text size set to any value that is <u>not</u> 100% stick to Selenium 2.37.0 and Firefox 25 as this the only combo that works well in such a setup. This problem should be solved in the next version of Selenium which is 2.42.0 <a title="https://code.google.com/p/selenium/issues/detail?id=6112" href="https://code.google.com/p/selenium/issues/detail?id=6112">https://code.google.com/p/selenium/issues/detail?id=6112</a></p>
<h4>xUnit unit of work</h4>
<p>xUnit executes in parallel test collections. Test Collection is a group of tests that either belong to a single class or single assembly. Tests that belong to a single test collection are executed serially. Collection behaviour can be specified in VS settings or as a global setting in code via <a href="https://github.com/pawelpabich/SeleniumWithXunit/blob/master/SeleniumWithXunit/Properties/AssemblyInfo.cs?#L40">CollectionBehviour</a> attribute. On top of that we have ability to control the level of concurrency by setting <a href="https://github.com/pawelpabich/SeleniumWithXunit/blob/master/SeleniumWithXunit/Properties/AssemblyInfo.cs?#L41">MaxParallelThreads</a>. This threshold is very important as a single machine can handle only limited number of browsers. </p>
<p>At the moment a single test can’t become a test collection which means that a class with a few long running tests can significantly extend the total execution time, especially if it happens to be run at the end of a test run. I added this feature to my fork of xUnit: <a title="https://github.com/pawelpabich/xunit/commit/7fa0f32a1f831f8aef55a8ff7ee6db0dfb88a8cd" href="https://github.com/pawelpabich/xunit/commit/7fa0f32a1f831f8aef55a8ff7ee6db0dfb88a8cd">https://github.com/pawelpabich/xunit/commit/7fa0f32a1f831f8aef55a8ff7ee6db0dfb88a8cd</a> and the sample app uses it. Both tests belong to the same class yet there are executed in parallel.</p>
<h4>Isolate tests, always</h4>
<p>Tests should not share any state but tests that run in parallel simply must not do it. Otherwise it is like asking for trouble which will lead to long and frustrating debugging sessions. The way this can be achieved is to make sure that all state used by a test is created in the constructor of the test and it gets disposed once the test is finished. For this we can use Autofac and <a href="https://github.com/pawelpabich/SeleniumWithXunit/blob/master/SeleniumWithXunit/Infrastructure/BaseTest.cs?#L20-21">create a new LifetimeScope for each test</a> which <a href="https://github.com/pawelpabich/SeleniumWithXunit/blob/master/SeleniumWithXunit/Infrastructure/BaseTest.cs?#L26-29">then gets disposed when the test object gets disposed</a>. xUnit doesn’t support IoC so we need to inject all dependencies using property injection in the constructor which is not a big deal.</p>
<h4>Shared resources, yes, they exist</h4>
<p>Technically speaking each test could launch a new instance of Firefox but this takes quite a bit of time and can be easily optimized. What we need is <a href="https://github.com/pawelpabich/SeleniumWithXunit/blob/master/SeleniumWithXunit/Infrastructure/BrowserPool.cs">a pool of browsers</a> where tests can take browsers from and return them to once they are done with them. In most cases shared resources can be initialized once per test run. xUnit doesn’t have a concept of global initialization so that’s why the test run setup happens <a href="https://github.com/pawelpabich/SeleniumWithXunit/blob/master/SeleniumWithXunit/Infrastructure/BaseTest.cs?#L48-58">in the constructor of the base class</a> that all other tests inherit from. This isn’t a great solution but it works. You might be able to move it to the static constructor as long as the setup code doesn’t use Threads and Tasks because <a href="http://blogs.msdn.com/b/pfxteam/archive/2011/05/03/10159682.aspx">they will cause deadlocks</a>. Lack of global initialization means that there is a no global clean up either, but this can be worked around by <a href="https://github.com/pawelpabich/SeleniumWithXunit/blob/master/SeleniumWithXunit/Infrastructure/TestRun.cs?#L16">subscribing to AppDomain.Unload</a> event and performing clean up there. From experience this works most of the time so I would rather have a proper abstraction in the framework. </p>
<h4>Don't block synchronously </h4>
<p>When MaxParallelThreads is set to a non zero value then xUnit creates a dedicated Task Scheduler with limited number of threads. This works great as long as you use async and await. But if you need to block synchronously you might cause a deadlock as there might be no free threads to execute the continuation. In such a case the safest way is to execute the code on the <a href="https://github.com/pawelpabich/SeleniumWithXunit/blob/master/SeleniumWithXunit/Infrastructure/CustomTaskFactory.cs">Default .NET Task Scheduler</a>.</p>
<h4>I need to know when something goes wrong</h4>
<p>If a test implements IDisposable then xUnit will call Dispose method when the test is finished. Unfortunately this means the test doesn’t know whether it failed or succeeded which is a must have for browser based tests because we want to take a screenshot when something went wrong. That’s why I added <a href="https://github.com/pawelpabich/xunit/commit/77d79aa316dc57950c9f5c110dc72f25111f7127">this feature to my fork of xUnit</a>. </p>
<h4>What about R#, I won’t code without it!</h4>
<p>There is an alpha version of xUnit plugin for R#. It runs xUnit tests fine but it doesn’t run them in parallel at the moment. But this might change in the future so keep an eye on <a title="https://github.com/xunit/resharper-xunit/pull/1" href="https://github.com/xunit/resharper-xunit/pull/1">https://github.com/xunit/resharper-xunit/pull/1</a>.</p>
<h4>Async, async everywhere</h4>
<p>Internally xUnit uses async and await extensively. Every piece of the execution pipeline (Create test object –> Execute Before Filter –> Execute test –> Execute After Filter –> Dispose test) can end up as a separate Task. This doesn’t work well with tests that rely on limited resources because new tests can be started before already finished tests are disposed and their resources returned to a shared pool. In such a case we have two options. Either the new tests are blocked or there will be more resources created which is not always a viable option. This is why in my fork of xUnit <a href="https://github.com/pawelpabich/xunit/commit/6a0965c9e4c371405a95b5f6c66ab9c044e609b8">the whole execution pipeline is wrapped in just one Task</a>.</p>
<h4>What about Continuous Integration</h4>
<p>There is a console runner that can output Team City service messages and this is what I use. It works great, and the only complain I have is that TeamCity doesn’t display well messages coming concurrently from multiple different flows in its web interface. This should be fixed in the future <a title="http://youtrack.jetbrains.com/issue/TW-36214" href="http://youtrack.jetbrains.com/issue/TW-36214">http://youtrack.jetbrains.com/issue/TW-36214</a>.</p>
<h4></h4>
<h4>Can I tell you which tests to execute first?</h4>
<p>In ideal world we would be able to run failed test first (e.g we can discover that via API call to the build server, thanks <a href="http://blog.stangroome.com/">Jason Stangroome</a> for the idea) and then the rest of the tests ordered by their execution time in descending order. xUnit lets us order tests within single test collection but there is no abstraction to provide custom sort order of test collections which is <a href="https://github.com/pawelpabich/xunit/commit/74eb583589f01711137a754d7b90e5b7a79bb1e4">what I added to my fork</a>. The orderer is <a href="https://github.com/pawelpabich/SeleniumWithXunit/blob/master/SeleniumWithXunit/Properties/AssemblyInfo.cs?#L42">specified in the same way a test orderer would be.</a></p>
<h4>It works but it has some sharp edges</h4>
<p>All in all I’m pretty happy with the solution and it has been working great for me over last couple of weeks. The overall execution time of our CI build dropped from 1:41h to 21 min. </p>
<p>The work is not finished and I hope that together we can make it even easier to use and ideally make it part of the xUnit project.I started a discussion about this here (<a title="https://github.com/xunit/xunit/issues/107" href="https://github.com/xunit/xunit/issues/107">https://github.com/xunit/xunit/issues/107</a>). Please join and share your thoughts. It’s time to put the problem of slow tests to bed so we focus on more interesting challenges :). </p>Pawel Pabichhttp://www.blogger.com/profile/05213210256288888149noreply@blogger.com6tag:blogger.com,1999:blog-168147483884969000.post-20888683371177731802013-01-01T20:38:00.001+11:002013-01-01T20:42:01.589+11:00How to run Hadoop on WindowsOne can spend only so much time surfing in 30C+ weather ;). So while my body was recovering from too much sunshine I decided to play with Hadoop to learn first hand what it actually is. <br />
<br />
The easiest way to start is to download <a href="https://ccp.cloudera.com/display/SUPPORT/Cloudera's+Hadoop+Demo+VM+for+CDH4#Cloudera%27sHadoopDemoVMforCDH4-DemoVMwareImage">a preconfigured VMware image</a> from Cloudera. This is what I did and it worked but it did not work well. The highest resolution I could set was 1024x768. I installed the VMware client tools but they did not seem to work with the Linux distribution picked by Cloudera. I managed to figure out how to use vim to edit text files but a tiny window with flaky UI (you can see what is happening inside Hadoop using a web browser) was more that I could handle. Then I thought about getting it working on Mac OS X which is a very close cousin of Linux. The installation process is simple but <a href="http://wiki.apache.org/hadoop/Running_Hadoop_On_OS_X_10.5_64-bit_(Single-Node_Cluster)">the configuration process</a> is not.<br />
<br />
So I googled a bit more and came across <a href="http://www.microsoft.com/sqlserver/en/us/solutions-technologies/business-intelligence/big-data.aspx">Microsoft HDInsight</a> which is Microsoft distribution of Hadoop that runs on <a href="http://www.microsoft.com/web/gallery/install.aspx?appid=HDINSIGHT-PREVIEW">Windows</a> and <a href="https://www.hadooponazure.com/">Windows Azure</a>. <a href="http://www.microsoft.com/web/gallery/install.aspx?appid=HDINSIGHT-PREVIEW">HDInsight</a> worked great for me on Windows 8 and I was able to play with 3 most often used query APIs: native Hadoop Java based map/reduce framework, <a href="http://hive.apache.org/">Hive</a> and <a href="http://pig.apache.org/">Pig</a>. I used Word count as a problem to see what each of them is capable of. Below are links to sample implementations:<br />
<ul>
<li><a href="https://ccp.cloudera.com/display/DOC/Hadoop+Tutorial#HadoopTutorial-Example%3AWordCountv1.0">Java map/reduce framework</a> – run c:\hadoop\hadoop-1.1.0-SNAPSHOT\bin\hadoop.cmd to get into command line interface for Hadoop </li>
<li><a href="http://en.wikipedia.org/wiki/Pig_(programming_tool)">Pig</a> – run C:\Hadoop\pig-0.9.3-SNAPSHOT\bin\pig.cmd to get into <a href="http://wiki.apache.org/pig/Grunt">Grunt</a> which lets you use Pig </li>
<li><a href="https://gist.github.com/4426172">Hive</a> – run C:\Hadoop\hive-0.9.0\bin\hive.cmd to get into Hive command line interface</li>
</ul>
<br />
As far as I know Microsoft is going to to contribute their changes back to the Hadoop project so at some stage we might get Hadoop running natively on Windows in <a href="http://nodejs.org/download/">the same way nodejs is.</a>Pawel Pabichhttp://www.blogger.com/profile/05213210256288888149noreply@blogger.com1tag:blogger.com,1999:blog-168147483884969000.post-70080721223071726562012-10-18T21:26:00.000+11:002012-11-07T20:45:07.190+11:00Quick overview of TFS PreviewI spent last week playing with shiny/new Web technologies and I needed a place where I could store my todo list and keep my code. I could’ve used Git + Trello but I wanted to give <a href="https://tfspreview.com/">TFS Preview</a> a try so I created a project there, converted my todo list into user stories and connected my repo with my Azure account for automated deployments. After a week of using TFS Preview this is what I’ve learnt:<br />
<ul>
<li>It was very easy to configure automated deployments to Azure </li>
<li>The performance of web portal ranged from acceptable to painfully slow. </li>
<li>The UI is decent and it’s easy to execute all the basic tasks like add user story, add bug, look up a build</li>
<li>Burndown chart worked out of the box </li>
<li>Scrum board is simple and does the job </li>
<li>Builds would take up to couple of minutes to start even if the build queue was empty </li>
<li>Total build time was a bit unpredictable, ranged from 60s to 160s for the same code. </li>
<li>Adding support for nUnit was more <a href="https://tfspreview.com/en-us/learn/build/build-with-other-unit-test-frameworks-in-vs/">hassle</a> than I anticipated </li>
<li>Story boarding in PowerPoint is not integrated with TFS so you can’t push files directly from PowerPoint to TFS </li>
<li>There is no Wiki </li>
<li>Build log is as “useful” as it was in TFS 2010 </li>
</ul>
All in all it is a decent start and I hope that <a href="http://blogs.msdn.com/b/bharry/">Brian Harry’s</a> team will keep deploying often so every week will welcome us with something new.<br />
<h3>
Update</h3>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
You actually can specify how many builds you want to keep. For some reason when I clicked on the number of builds to preserve it didn't change from a label into a drop down list. Pawel Pabichhttp://www.blogger.com/profile/05213210256288888149noreply@blogger.com3tag:blogger.com,1999:blog-168147483884969000.post-42443671446621311892012-07-03T20:23:00.000+10:002012-11-06T00:32:41.206+11:00Cloudcast show: the clouds are different down under<p>A few weeks ago I was a guest on <a href="http://www.uhurusoftware.com/blog.aspx">the Uhuru podcast</a> where I talked to <a href="https://twitter.com/#!/msurkan">Micheal Surkan</a> about my experience with deploying to the Cloud and using Cloud based service providers. The discussion was focused on risks associated with using the Cloud and techniques to mitigate them.</p> <p>You can listen to it <a href="http://www.uhurusoftware.com/blog/blog/2012/june/the-clouds-are-different-down-under.aspx"><strong>here</strong></a>. It’s only 10 minutes :). Enjoy.</p> Pawel Pabichhttp://www.blogger.com/profile/05213210256288888149noreply@blogger.com1tag:blogger.com,1999:blog-168147483884969000.post-34386096665357446332012-05-30T20:23:00.000+10:002012-11-06T00:32:41.197+11:00Less\Sass done right in .NETThese are the <a href="http://www.slideshare.net/PawelPabich/lesssass-done-right-in-net"><strong>slides</strong></a> and <a href="https://github.com/pawelpabich/SampleBundleImplementationsForLess"><strong>code</strong></a> from the talk I gave last Tuesday at the <a href="http://sydney.ozalt.net/2012/05/lightning-talks-make-it-yay-for-may.html"><strong>Sydney Alt.Net.</strong></a> You can grab both of them as a zip file from <a href="https://dl.dropbox.com/u/7579883/Blog/Blog_SampleBundleImplementationsForLess.zip"><strong>here</strong></a><strong>.</strong> Enjoy!Pawel Pabichhttp://www.blogger.com/profile/05213210256288888149noreply@blogger.com0tag:blogger.com,1999:blog-168147483884969000.post-10702795263198795432012-05-16T20:22:00.000+10:002012-11-07T19:34:41.539+11:00Entity Framework migrations don't support parallel developmentEF migrations borrowed a lot of ideas from Ruby on Rails migrations and this a good thing. The bad thing is that EF treats database as the source of truth. Unpleasant side effect of this approach is that only one developer can work on database changes at a time. <br />
<u>It seems</u> that every time EF creates a DB migration it stores some kind hash of the current database schema which is then used before to the migration is actually executed to make sure that the database is in the right state. This breaks down very quickly when more than 1 developer works on the model. Sample scenario: <br />
<ol>
<li>Dev1 adds migration for a new property called Master and pushes the change. </li>
<li>Dev2 adds migration for AnotherDev property and applies it to the database </li>
<li>Dev2 pulls Dev1’s changes </li>
<li>Dev2 tries to run the app and gets an exception.</li>
<li>Dev2 needs to delete his migration and rollback corresponding changes in the database </li>
<li>Dev2 applies Dev1’s migration </li>
<li>Dev2 adds again migration for AnotherDev property </li>
<li>Dev2 is ready to push his changes unless someone else introduced a new migration in the meantime. In such case Dev2 goes back to step number 4. </li>
</ol>
I’ve put together <a href="https://github.com/pawelpabich/EFMigrationsLimitations"><strong>a simple app on GitHub</strong></a> that reflects this flow. Each commit is meant to represent a single step.<br />
EF has a chance to become a decent ORM with even better tooling but before this happens EF team needs to understand that developers know what they do and prefer to have more than less rope at their disposal.<br />
I’m happy with the product to be safe by default but there should be a way of disabling those annoying limitations. Without this ability <a href="http://www.hanselman.com/blog/"><strong>Scott Hanselman</strong></a> idea of LEGO size building blocks is….well just an idea when it comes to EF. Pawel Pabichhttp://www.blogger.com/profile/05213210256288888149noreply@blogger.com13tag:blogger.com,1999:blog-168147483884969000.post-26669091295533651222012-05-06T20:21:00.000+10:002012-11-06T00:32:41.194+11:00Quick IO performance test of VirtualBox, Parallels and VMware<p>I’ve been using VirtualBox for over a year and I’m pretty happy with it but I wanted to see if new releases of VMware and Parallels can give me even better IO performance. This is a very simple and coarse test but it clearly indicates that I should give VMware a try.</p> <p>Test setup:</p> <ul> <li>Guest: Windows 7 64bit SP1 </li> <li>Guest: 8 CPUs and 4GB of RAM </li> <li>Guest: Vendor specific additions installed </li> <li>VMware: Workstation 8.0.3 </li> <li>Parallels: Workstation 6.0.13950 </li> <li>VirtualBox: 4.1.12 </li> <li><a href="http://crystalmark.info/download/index-e.html#CrystalDiskMark">CrystalDiskMark 3.0.1c</a> was run 3 times for each app + host </li> </ul> <p><iframe height="225" src="https://docs.google.com/spreadsheet/pub?key=0Asn2NBGAslMCdHowR2hpaDRncTVXMHNsUjNSbjNSS0E&single=true&gid=0&range=A1%3AF9&output=html&widget=true" frameborder="0" width="425"></iframe></p> Pawel Pabichhttp://www.blogger.com/profile/05213210256288888149noreply@blogger.com0tag:blogger.com,1999:blog-168147483884969000.post-83230246592532140682012-03-20T21:21:00.000+11:002013-11-17T23:22:05.876+11:00Good enough softwareSo the first question is what “good enough software” actually means? My current definition is rather simple: <u>whatever makes the business owner happy</u>. I didn’t mention any technical characteristics on purpose because business very often neither cares nor understands them. <br />
Doesn’t it mean that we are doomed to low quality, crappy software that we can’t possibly be proud of? I don’t think so. I’m yet to see healthy, profitable business that thinks its IT delivers features too fast. And quality is a must have when you want to achieve sustainable, high velocity.<br />
The fact that a team can complete tasks quickly doesn’t mean they work on the right tasks. As software engineers we are perfectionists and this is great but at the same time it means that we can become very easily victims of gold plating. To make sure this happens as rarely as possible (I don’t think this can be fully prevented :) ) we need to make sure that every significant activity is driven by the business. It goes without saying that having a prioritised back log is a must have nowadays. But then there are other, smaller tasks that very often are not driven by the business though they should.<br />
One of them is bug fixing. It’s the business that should decide if a particular bug is worth fixing or not. This might be hard to digest for developers and QAs but it is the business that pays our salaries and it should decide what we work on. Our job is to make sure the business is aware of the trade-offs and in this way can make informed decisions. Same logic applies to Definition of Done, build versus buy and so on.<br />
As a side effect of getting the business involved in the process of building software by IT we have a good chance to establish trust between those two which unfortunately is not very common.<br />
Companies rely more and more on IT because it’s the best way to achieve their goals. But we have to keep in mind that if they find a better way they will switch away from IT in no time. So let’s make sure we stay No.1 as long as possible by building good enough instead of perfect software.Pawel Pabichhttp://www.blogger.com/profile/05213210256288888149noreply@blogger.com1tag:blogger.com,1999:blog-168147483884969000.post-61272710555534469712011-12-22T22:20:00.000+11:002012-11-06T00:32:41.215+11:00YOW 2011 - loose thoughts<p>Talks I attended at this year conference were focused on Agile, functional programming and low level performance. It turned out to be quite an interesting mix.</p> <p><a href="https://twitter.com/#!/mpoppendieck">Marry Poppendieck</a> talked about ways of supporting continuous design. One of the advices was to create an environment that facilities fair fights. To achieve this you need to hire good, diverse people. Their diversity let them solve complex problems because they can approach them from multiple angles. Once you have a team like that leave them alone and simply keep supporting them.</p> <p>I haven’t learnt much new at Agile Architecture workshop delivered by <a href="https://twitter.com/#!/rebeccawb">Rebecca Wirfs-Brock</a> but it was good to hear that I’m not some kind of weirdo that demands impossible :). </p> <p>Simon Peyton Jones delivered 2 amazing and full of passion sessions about functional programming. I’ve done a bit of this type of programming before but I was surprised by Haskell type system and its ability to ensure lack of side effects at the compile time. On top of that on my way back to Sydney I had a good chat with <a href="https://twitter.com/#!/dibblego">Tony Morris</a> who told me about a project where he used functional programming to create a composable data access layer. Composablity and strict control over side effects is enough to push Haskell to the top of my list of languages to play with :). </p> <p>Inspired by <a href="http://coderetreat.ning.com/events/global-coderetreat-brisbane-queensland-australia">Coderetreat Brisbane</a> organized by <a href="https://twitter.com/#!/markryall">Mark Ryall</a> I’ve decided to use <a href="http://en.wikipedia.org/wiki/Conway's_Game_of_Life">Conway’s Game of Life</a> as a problem that I will try to solve every time I learn a new language. It’s worked for CoffeeScript and I hope it will work Haskell.</p> <p>The end of the conference was filled by performance tips from .NET (<a href="http://www.twitter.com/joelpob">Joel Pobar</a> and Patrick Cooney) and Java (<a href="http://twitter.com/mjpt777">Martin Thompson</a>) land. Both sessions emphasized that computation is cheap and memory access is slow. By simply keeping all required data in L1/L2 CPU cache you can cut the execution time by half. Obviously this was presented using a micro benchmark but still it is something that it’s worth keeping in mind. </p> <p>Functional languages rely on immutable data structures to isolate processing and in this way control side effects. Unfortunately this means that a lot of memory is pushed around and this has a significant influence on the overall performance. Again, it’s all about trade-offs :).</p> <p>Martin talked about “mechanical sympathy” which is boils down to having at least basic understanding of hardware your software runs on. Without it your software not only doesn’t take full advantage of the underlying hardware but often works against it which has severe impact on the overall performance. It’s one of those advices that most of us won’t use on daily basis as most our infrastructure is in the Cloud but it’s good to keep it in mind. </p> <p>We’ve been told multiple times that “free lunch is over” and CPUs are not getting any faster. Martin <a href="http://gotocon.com/dl/jaoo-melbourne-2011/slides/Folklore_Martin_Thompson.pptx.pdf">proved</a> that this is not correct. It’s true that we are not getting more MHz, we are actually getting less MHz than we used to but CPUs get faster because every year engineers design better chips. The message Martin is trying to send is that we should prove/disprove assumptions based on evidence and not beliefs.</p> <p>All in all it was another good conference and I will try to attend it next year.</p> Pawel Pabichhttp://www.blogger.com/profile/05213210256288888149noreply@blogger.com0tag:blogger.com,1999:blog-168147483884969000.post-68463197196645505142011-11-30T22:19:00.000+11:002012-11-06T00:32:41.219+11:00Healthy team<p>I’ve had recently a few conversations about what constitutes a great software development team and I found myself using phrase ”healthy team” a lot. It’s a rather vague expression so the goal of this blog post is to explain what I mean by it.</p> <h2>People </h2> <p>A healthy team consists of people that share the same vision (the end goal of the project) and the same basic principles (high care factor, working as a team, not leaving broken build overnight, etc.). Even one person that doesn’t share them can cause a fair amount of frustration for the rest of the team and frustration is what needs to be avoided at any price as there is nothing that kills productivity faster. </p> <h2>Skills</h2> <p>In ideal world every team would have only senior members that are highly skilled and experienced. In real life this never happens and to be honest I don’t see any problem with having junior members on a team as long as they want to learn, are capable of learning quick and there are not too many of them. If I had to pick I would say 1 junior for every 3 to 5 senior members is more or less the right ratio. </p> <p>Software development is complex and it’s unreasonable to expect software engineers to be experts in every possible technical area. That’s why I believe every team needs to have at least one expert for each significant horizontal/cross cutting concern. Such a person would ensure consistency and correctness across all vertical features. One person can drive multiple horizontals. Sample horizontals: CSS, deployment, messaging, etc..</p> <p>At the same time all team members, including experts, should work on vertical end to end features and ask experts for help when they struggle with a certain horizontal concern. In this way knowledge related to horizontals flows from experts to the rest of the team. On top of that every team member can influence a horizontal with experience he’s gained working on vertical features. </p> <p>This approach takes initially more time than splitting the team by skills and let people work in isolation on what they know best. But there are a few very valuable advantages of this approach in the long run. Team is more resilient to departures because knowledge is shared and the quality of the final product is higher because the product is created based on feedback from both horizontals and verticals. Obviously, there are certain specialized areas of software engineering which might not work with this approach but it should be just fine for most of us.</p> <h2>Leadership</h2> <p>I’ve worked on a few projects where there was no team leader and this always lead to the same problem. Different team members had different visions of what we were trying to achieve and how we were supposed to achieve it. I know this might be a slippery slope and be against some of Agile principles but at this stage of my career I believe that every team should have a team leader to make sure everybody is going in the same direction and that the team avoids basic/fatal mistakes. </p> <p>A good team leader is experienced, trusts other team members, treats them as partners, listens to them and always is more than happy to answer their questions but at the same time should be able to make the final call when it’s needed. And <strong>no</strong>, I’m not trying to promote absolute dictatorship as way of leading teams :).</p> <h2 /> <h2>Process</h2> <p>A healthy team is a team that is as independent as possible and is not <a href="http://www.pabich.eu/blog/archive/2011/10/25/Good-process-helps-bad-process-destroys.aspx"><strong>hindered by the process it needs to follow.</strong></a> On top of that the team members should be able to use tools that make them most productive and they should feel that they are in the driver’s seat. </p> <h2>Size</h2> <p>I believe that a healthy team is a small team. Ideally 5 to 7 people but no more than 10. More than that and communication becomes challenging which slows down knowledge sharing and increases frustration. </p><p><strong>Update:</strong> I've been asked a few times if 5-7 is the initial size of the team. The short answer is no. Ideally 2-3 people form the initial team and then bring more people when they are needed and team is ready to accommodate them. I didn't write <strong>1</strong>-3 on purpose because I think that it's always good to have someone that you can bounce your ideas off. </p> <h2 /> <h2 /> <h2> </h2> <p>The order of paragraphs matters but all of them deal with important issues and I wouldn’t ignore any of them lightly.</p> <p>The blog post has been updated on 24/01/2012 with comments about the initial size of the team. </p>Tags: <a href="http://www.pabich.eu/blog/tags/Software/default.aspx" rel="tag">Software</a>Pawel Pabichhttp://www.blogger.com/profile/05213210256288888149noreply@blogger.com0tag:blogger.com,1999:blog-168147483884969000.post-46986046997957407722011-10-25T21:15:00.000+11:002012-11-06T00:32:41.208+11:00Good process helps, bad process destroysI hear very often that all a successful project needs is a great team. This is based on an assumption that no matter how bad the rest of the project is a good team always finds its way to successful ending. This is correct only when it comes to areas that the team has control over. Very often process is not one of them and a bad process causes great teams to become mediocre as they can’t operate in their usual way. <br /><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi2XfzZiJ_l8NvykK-RuS7zx87FdM7-F-AtLf6cOxkFQZyUhg78NTOD_XgvlQJAiLXsJ_S46TYICMr6kNkv65tSi7Dt9EVYXB-rxDWIpPqoxCCcE5TMajdmN7z6w3xlw04uVJh6PfHJiOg/s1600/process_vs_people_6.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="326" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi2XfzZiJ_l8NvykK-RuS7zx87FdM7-F-AtLf6cOxkFQZyUhg78NTOD_XgvlQJAiLXsJ_S46TYICMr6kNkv65tSi7Dt9EVYXB-rxDWIpPqoxCCcE5TMajdmN7z6w3xlw04uVJh6PfHJiOg/s400/process_vs_people_6.png" width="400" /></a></div><br /><br />Pawel Pabichhttp://www.blogger.com/profile/05213210256288888149noreply@blogger.com0tag:blogger.com,1999:blog-168147483884969000.post-15088185435849019992011-07-28T20:15:00.000+10:002012-11-06T00:32:41.186+11:00Reliable end to end tests with Selenium WebDriverThese are the <a href="http://www.slideshare.net/PawelPabich/reliable-tests-with-selenium-web-driver"><strong>slides</strong></a> and <a href="https://github.com/pawelpabich/Reliable-e2e-tests-with-Selenium-Webdriver"><strong>code</strong></a> from the talk I gave last Tuesday at the <a href="http://sydney.ozalt.net/"><strong>Sydney Alt.Net</strong>.</a> If 2 links is to much of clicking for you then you can grab both slides and code from here as <a href="https://dl.dropbox.com/u/7579883/Blog/2011_07_Alt_NET_Sydney.zip"><strong>a zip archive</strong></a>.Pawel Pabichhttp://www.blogger.com/profile/05213210256288888149noreply@blogger.com2tag:blogger.com,1999:blog-168147483884969000.post-17648696313240580872011-05-31T20:14:00.000+10:002012-11-06T00:32:41.187+11:00Virtual desktops via Virtual MachinesI’ve got recently a very decent laptop (Lenovo W520, 16GB RAM) and one of the fastest SSDs on the market(Vertex 3 240GB). The machine is flying and basically does whatever I ask it to do in no time :). <br />I’ve always wanted to keep client projects, my personal projects and the rest (Outlook, pictures, etc) fully isolated. The main reason being the fact that every project requires some specific setup that I don’t want to keep forever and re-installing the whole machine after each piece of work doesn’t appeal to me. On top of that I don’t want to limit myself and don’t play with the latest technology just because it might break something on my system and I won’t be able to work.<br />So I’ve created 2 VMs. One for my personal stuff(VS, Ruby, Erlang, but no Outlook) and one for my current client. The idea is that I will create a separate VM for each new client. Another advantage of this approach is that if something happens to my laptop I can copy the VM to a different machine and be up and running within minutes. I backup all VMs twice a week to an external disk using <a href="https://www.crashplan.com/"><strong>CrashPlan</strong></a>. A nice side effect of using <a href="https://www.crashplan.com/"><strong>CrashPlan</strong></a> is that I get versioning so in case of emergency I don’t have to go all the way back to the initial state of the VM.<br />The VMs are obviously not as fast as the host with IO being the main bottleneck but they are fast enough so I can work comfortably. I can even run 2 VMs at the same time and they both start within seconds and are responsive enough so I can have fully functional VS open in each of them.<br />Have a look at screenshots from <a href="http://crystalmark.info/software/CrystalDiskMark/index-e.html"><strong>CrystalDiskMark</strong></a> to see the IO performance drop.<br /><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiduSf6TUyo68rZf-pq8i4JwMtFuwjv4IsqMDTGEE163F3oNwxunPljGIVE6jwQUeGpqi8xHBjCXM0CtFIf86EslE1syomM8eM3orvQkE35qnN6zyo_oTOUvuR81afLflFh-JC4VP86SQ0/s1600/Host_2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="285" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiduSf6TUyo68rZf-pq8i4JwMtFuwjv4IsqMDTGEE163F3oNwxunPljGIVE6jwQUeGpqi8xHBjCXM0CtFIf86EslE1syomM8eM3orvQkE35qnN6zyo_oTOUvuR81afLflFh-JC4VP86SQ0/s320/Host_2.png" width="320" /></a></div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhgp0DvBv38ZLD4uWqqBCM1dzLxqdullMW3LnXibkhumn0-l77lSDOq1SNndb1t4Gc6ZngzyfjBBXDGfwprufcbx9RT4HY0nvulmoFqr6P6A7R-GIBrebexyrV9_7SG9waw_vnvkcQu-1A/s1600/Guest_2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="283" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhgp0DvBv38ZLD4uWqqBCM1dzLxqdullMW3LnXibkhumn0-l77lSDOq1SNndb1t4Gc6ZngzyfjBBXDGfwprufcbx9RT4HY0nvulmoFqr6P6A7R-GIBrebexyrV9_7SG9waw_vnvkcQu-1A/s320/Guest_2.png" width="320" /></a></div><div class="separator" style="clear: both; text-align: center;"><br /></div><div class="separator" style="clear: both; text-align: justify;"><span style="text-align: start;">I started the whole experiment a week ago so we will see how it goes. Drop me a line if you know how to squeeze more performance out of </span><a href="http://www.virtualbox.org/" style="text-align: start;"><strong>VirtualBox</strong></a><span style="text-align: start;">.</span></div>Pawel Pabichhttp://www.blogger.com/profile/05213210256288888149noreply@blogger.com1tag:blogger.com,1999:blog-168147483884969000.post-54879261831094939822011-03-28T21:12:00.000+11:002012-11-06T00:32:41.212+11:00Automated database deployments to AppHarbor<a href="http://www.blogger.com/appharbor.com"><strong>AppHarbor</strong></a> is trying to provide .NET developers with similar goodness that <a href="http://heroku.com/"><strong>Heroku</strong></a> offers to their Ruby counterparts. At the moment deployment to AppHarbor has at least 6 distinct stages:<br /><ul><li>Code gets pushed to AppHarbor </li><li>Code gets compiled </li><li>Unit tests get run </li><li>Config transformations get applied </li><li>Configuration Variables get applied </li><li>Code gets actually deployed to a web server </li></ul>At least one obvious stage is missing which is the database deployment. In .NET world there is nothing that would be even close to Ruby on Rails migrations but we can do something about that.<br />There is a small library called <a href="http://code.google.com/p/dbup/"><strong>DbUp</strong></a> that simply makes sure that a given set of SQL scripts gets executed only once. This the ”up” part of RoR migrations. Now the question is when the database upgrade should be performed. There are basically 2 ways. Either before the new code gets deployed or after. I prefer the former option because as long as the database changes are non-breaking the app is fully functional at all times. If the database upgrade fails then the new version of the application won’t be deployed.<br />If the upgrade is done after the new version of the application is deployed then it needs to be done at the startup which prevents users from using the application for some period of time. What is more if the database upgrade fails then the new version of the application needs to be rolled back. In most cases the new code won’t work with the old database schema.<br />At the moment AppHarbor doesn’t have an explicit step when database upgrade can be performed. So to work around this problem I created a simple MsBuild target called UpgradeDatabaseInProduction that gets run when the application is successfully compiled. All the target does is extract production connection string from Web.Release.config and pass it to DbUp console app that performs the upgrade. <br /><br /><br /><pre class="brush: html"><target Name="UpgradeDatabaseInProduction"<br /> AfterTargets="MvcBuildViews"<br /> Condition="'$(Configuration)'=='Release'"><br /> <calltarget Targets="UpgradeDatabase" /><br /></Target><br /><br /></pre><br /><br /><pre class="brush: csharp;html-script: true"><target Name="UpgradeDatabase"><br /> <xmlpeek XmlInputPath="$(MSBuildThisFileDirectory)Web.$(Configuration).config"<br /> Query="/configuration/connectionStrings/add[@name='database']/@connectionString"><br /> <output TaskParameter="Result" ItemName="ConnectionString" /><br /> </XmlPeek><br /> <exec Command="$(MSBuildThisFileDirectory)..\libs\dbconsole-custom\DbUp.Console.exe -cs "@(ConnectionString)" -d $(MSBuildThisFileDirectory)SqlScripts" /><br /> <!-- This is the only way to have some debug statements visible in the build log --><br /> <warning Text="Database upgraded" /><br /></Target><br /><br /></pre><br />This works only because AppHarbor build servers can talk to AppHarbor database servers. To be honest it’s more of a hack than a proper solution :) but at least I can deploy the whole app (code + database changes) with single git push which is simply priceless. <br /><div class="wlWriterEditableSmartContent" id="scid:0767317B-992E-4b12-91E0-4F059A8CECA8:177106ab-ad8d-4f42-ab94-ea2ecb5e2714" style="display: inline; float: none; margin: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;"><br /></div>Pawel Pabichhttp://www.blogger.com/profile/05213210256288888149noreply@blogger.com3tag:blogger.com,1999:blog-168147483884969000.post-88419492952763162902010-12-11T22:09:00.000+11:002012-11-06T00:32:41.236+11:00YOW 2010 - loose thoughts<p>It doesn’t happen often that nearly every single talk at a conference is great and on top of that half of them are actually funny. That’s <a href="http://www.yowconference.com.au/brisbane/index.html"><strong>YOW 2010</strong></a> for you summarized in one sentence :).</p> <p><a href="http://twitter.com/#!/justinsheehy"><strong>Justin Sheehy</strong></a> explained how to quickly narrow down the choice of database technologies that might be useful in a particular case. His method is based on a simple matrix of operations requirement (local, single server, distributed, etc) by data model (relational, column families, key/value, etc). Once this is done and there are only a few solutions on the table a more sophisticated and time consuming research can be conducted to choose the right solution. Every single NoSQL solution is different and a generic split SQL/NoSQL doesn’t really make sense. It’s all about tread-offs. It’s amazing how often this simple fact needs to be reminded.</p> <p>Eric Evans talk was focused on the idea of bounded contexts. In other words a single enterprise model is an anti-pattern and is one of software engineering fallacies. Eric mentioned also a few disadvantages of doing big design upfront (AKA let’s build a great framework that less skilled devs can use) and postponing the initial release for long time. Nothing really new but it was well delivered.</p> <p><a href="http://twitter.com/#!/ghohpe"><strong>Gregor Hohpe</strong></a> talked about trade-off decisions that Google had to make to be able to reach its current scale. He covered the whole spectrum of optimizations from data access at the disk level to minimize heat generation to skipping some longer than expected running parts of map reduce executions to make sure results are delivered in timely manner. When I asked Gregor if Google uses regular Pub/Sub or transactions he said that if there is a technology out there Google has built something on top it :). Just use the right tool for the job.</p> <p>Second day started with Erik Meijer explaining coSQL (AKA NoSQL). It was a funny presentation about what NoSQL really is and how it relates to SQL. They both complement each other even in a mathematical sense hence the <b>co</b> part of <b>co</b>SQL. Additionally <b>co</b> is more positive than <b>no</b> and this makes Erik happy :).</p> <p>Jim Webber talked passionately about how much he <s>hates</s> dislikes ESBs and how rarely ESB is the right tool for the job. His presentation was extremely funny but still full of useful information. The main point was that a custom built system can be cheaper (but not cheap) and less risky to deploy than an out of the box ESB which often requires a substantial up-front cost.</p> <p>Dave Farley took us to the world of <1ms latency and speed of 100k per second. According to Dave this is achievable on commodity servers. The main enabler seems to be lack of synchronization, keeping as few threads per core as possible, keeping all the data in memory and keeping methods very short. 1 CPU can execute 1 billion instructions a second. That’s a lot and as long as we don’t waste it today hardware should be more than enough for needs of most consumers. The main message was that we underestimate what we can get from today hardware. I suppose this is only partially true because nowadays we rarely deploy apps on real hardware. In most cases all we see is a VM that shares the host with Gazillion of other VMs. This might the main reason why the perception of the current hardware capabilities is skewed.</p> <p>After the conference there were 2 days of workshops. I spent the first day with Ian Robinson and Jim Webber learning about REST. What I believed constituted a fully blown RESTfull service was actually a very basic RESTfull service that scores only 1 out of 3 points in <a href="http://martinfowler.com/articles/richardsonMaturityModel.html"><strong>Richardson maturity model</strong></a>. Each of the levels has its place but obviously the higher you get the more you take advantage of the Web and that’s the whole purpose of using REST. REST is CRUDish as it mostly relies on GET, POST, PUT and DELETE. My initial thought was that this is very limiting but then it turned out that it doesn’t have to be. The same applies to lack of transactions. This can be worked around with proper structure of resources, meaningful response codes and proper use of HTTP idioms. Another important thing to keep in mind is that domain model shouldn’t be exposed directly. What you want to expose instead are resources that represent client – server interactions (use cases). In most cases O(resources) > O(domain classes) – notation by Jim Webber :). The Web is inherently based on polling (request/response) thus REST is not suitable for apps which require low latency. In this case you might want to use Pub/Sub. </p> <p>The next day I attended a workshop with <a href="http://twitter.com/#!/coreyhaines"><strong>Corey Haines</strong></a>. This was a true hands-on workshop. I spent at least half a day writing code retreats, code katas and coding dojos. Going back to the very basics was surprisingly refreshing. I spent two 45 minutes long sessions constantly refactoring maybe 15 lines of code until most of if statements were gone and code read properly. You wouldn’t do this at work but the whole point of the exercise was to actually go over the line and try to come up with best possible code without feeling the time pressure.</p> <p>At last but not least, the attendees were fantastic and every coffee/lunch break was full of valuable conversations.</p> <p>I had an amazing time and <a href="http://www.yowconference.com.au/brisbane/index.html"><strong>YOW 2010</strong></a> is the best conference I’ve ever been to. </p> <div style="padding-bottom: 0px; margin: 0px; padding-left: 0px; padding-right: 0px; display: inline; float: none; padding-top: 0px" id="scid:0767317B-992E-4b12-91E0-4F059A8CECA8:9b712f87-8295-40e2-9767-6aa830e0ee77" class="wlWriterEditableSmartContent">Tags: <a href="http://www.pabich.eu/blog/tags/YOW/default.aspx" rel="tag">YOW</a>, <a href="http://www.pabich.eu/blog/tags/REST/default.aspx" rel="tag">REST</a>, <a href="http://www.pabich.eu/blog/tags/NoSQL/default.aspx" rel="tag">NoSQL</a>, <a href="http://www.pabich.eu/blog/tags/Loose+Thoughts/default.aspx" rel="tag">Loose Thoughts</a>, <a href="http://www.pabich.eu/blog/tags/Performance/default.aspx" rel="tag">Performance</a></div>Pawel Pabichhttp://www.blogger.com/profile/05213210256288888149noreply@blogger.com3tag:blogger.com,1999:blog-168147483884969000.post-41301291114953528152010-11-15T21:56:00.000+11:002012-11-06T00:32:41.204+11:00Microsoft Azure on-premises in 2011<p>It looks like <a href="http://www.microsoft.com" title="" target="_blank"><a href="http://www.microsoft.com" title="" target="_blank">Microsoft</a></a> is filling an obvious gap and its customers will be able to deploy Azure <a href="http://www.infoq.com/news/2010/11/Hyper-V-Cloud"><strong>on their own machines in 2011</strong>.</a> This should significantly speed up the adoption of <a href="http://www.microsoft.com" title="" target="_blank"><a href="http://www.microsoft.com" title="" target="_blank">Microsoft</a></a> Cloud offering as it introduces an additional checkpoint half way through the migration and lowers the risk of the whole process. If on top of that other Cloud providers deploy Azure to their own data centers then the risk will be even smaller because ”vendor lock in” stops being such a big problem. These are all good changes and I’m really looking forward to how they affect the global market of Cloud Computing. </p> <p>BTW I recently watched <a href="http://www.infoq.com/presentations/The-Cloud-Silver-Bullet"><strong>an interesting presentation</strong></a> by Chris Read from ThoughtWorks where he focuses on the Cloud from the Operations perspective. One of the takeaways that very often is not obvious to people is that there is no need to fire the infrastructure guys. You simply give them different, more creative tasks :)</p> <p> </p> Pawel Pabichhttp://www.blogger.com/profile/05213210256288888149noreply@blogger.com0tag:blogger.com,1999:blog-168147483884969000.post-22673426256045812882010-09-06T23:04:00.000+10:002012-11-06T00:32:41.190+11:00Google CDN is not immune to being downJust a reminder to myself that it’s good to have a fallback procedure when Google is down…not that it happens often :)<br />This happened to my blog a few days ago:<br /><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgNdYK1V6LLDqSP32lGms7Go8rGfvxEXh4VfBnnLDUwru6-rQYhUjUdp0rQC_QyXKMvLTDvncwv8ni8HR7jhBUHSXX07DdLianFrDuU1kdyeuSgxPzeK1UoD_U26VM64V_BoDaMI4j4eYA/s1600/timeout_2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="132" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgNdYK1V6LLDqSP32lGms7Go8rGfvxEXh4VfBnnLDUwru6-rQYhUjUdp0rQC_QyXKMvLTDvncwv8ni8HR7jhBUHSXX07DdLianFrDuU1kdyeuSgxPzeK1UoD_U26VM64V_BoDaMI4j4eYA/s400/timeout_2.png" width="400" /></a></div><br />Pawel Pabichhttp://www.blogger.com/profile/05213210256288888149noreply@blogger.com0tag:blogger.com,1999:blog-168147483884969000.post-12022279255316649152010-08-26T23:04:00.000+10:002012-11-06T00:32:41.181+11:00RubyMine is a real gem<p>Today I had to fix a piece of custom code inside of <a href="http://blog.duc.as/2010/08/25/a-day-in-the-life-of-a-metro-veloper/feed/"><strong>Redmine</strong></a>. I have very little experience with Ruby on Rails but I was able to get the app up and running with a debugger attached within 15 minutes. </p> <p>I downloaded the latest version of <a href="http://confluence.jetbrains.net/display/RUBYDEV/RubyMine+EAP"><strong>RubyMine 2.5 EAP</strong></a><strong>,</strong> installed it, pointed it to the folder with the app, selected production configuration and hit Debug. <a href="http://www.jetbrains.com/ruby/"><strong>RubyMine</strong></a> analysed my Ruby setup and popped up a window with a notification that I’m missing some gems and the IDE can download and install them for me. I hit Ok and 5 minutes later I was debugging the app. Ruby on Rails experience on Windows is far from being perfect but <a href="http://www.jetbrains.com/ruby/"><strong>RubyMine</strong></a> is simply awesome.</p> Pawel Pabichhttp://www.blogger.com/profile/05213210256288888149noreply@blogger.com0tag:blogger.com,1999:blog-168147483884969000.post-86009157284040252572010-07-22T23:03:00.000+10:002012-11-06T00:32:41.234+11:00Java Script unit testing with YUI Test and Jack mocking frameworkI strongly believe in unit testing and recently I spent a bit of time trying to apply this technique to Java Script code. <br />The first problem that I had to solve was which framework to use. From what I’ve read it looks like <a href="http://code.google.com/p/jsspec/"><strong>JSSpec</strong></a>, <a href="http://docs.jquery.com/QUnit"><strong>qUnit</strong></a> and <a href="http://developer.yahoo.com/yui/3/test/"><strong>YUI Test</strong></a> get most of the attention nowadays. YUI is the most mature from them and offers by far the most functionality out of the box. On the other hand it is the most complex one to setup but still the whole process takes only a few copy/paste clicks. At the end of the day I decided to go with YUI Test because I wanted to check if I really need its rich capabilities.<br />In C# world to make unit testing easy we use mocking frameworks. In Java Script world mocking frameworks are not needed because Java Script is a dynamic language and every method/object can be overwritten at any time at runtime. Still mocking might take a bit of effort because you have to keep the original method somewhere around to put it back to where it was at the end of a test. Otherwise you end up with state that is shared between tests which is a bad thing. <a href="http://boss.bekk.no/display/BOSS/Jack"><strong>Jack</strong></a> is a mocking framework that helps solve this problem. It’s not perfect but it is good enough for what I wanted to do.<br />Enough introduction, let’s start with the story that I’ve implemented. The link to the complete source code is located at the bottom of this post.<br />There is a simple form and we have to write client side validation logic for it. The rules are as follows:<br /><ul><li>The user can select either one or more predefined reasons or can provide a custom reason. The user can not use both. </li><li>If the form validation succeeds then the user gets a popup with “Correct” message </li><li>If the form validation fails then the user gets a popup with “Wrong” message. </li></ul>This is how the form looks like:<br /><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjx7JMVBR2WavTJZvlqh3NLIBPUMaW7GkbIO4D2_5deyLcbhlksWuH9IG-xv_oLzpm5OUzH1E3ecKNybm1GkF-5rGOgOiJ1_Bexa_fdVwtQ-3yOuiM0ZQeIOPySmGOCOTGrKtxdJdOrDVw/s1600/image_6.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="256" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjx7JMVBR2WavTJZvlqh3NLIBPUMaW7GkbIO4D2_5deyLcbhlksWuH9IG-xv_oLzpm5OUzH1E3ecKNybm1GkF-5rGOgOiJ1_Bexa_fdVwtQ-3yOuiM0ZQeIOPySmGOCOTGrKtxdJdOrDVw/s320/image_6.png" width="320" /></a></div><br /><br />and this is the HTML behind it: <br /><script src="https://gist.github.com/3979072.js"> </script> To be able to run the unit tests we have to have an HTML page that simply loads all required Java Script code and executes it. <br /><script src="https://gist.github.com/3979097.js"> </script> <br /><pre class="code"><span style="color: blue;"> </span></pre><a href="http://11011.net/software/vspaste"></a><a href="http://11011.net/software/vspaste"></a> <br />As you can see the HTML page is very simple. It loads a few files that belong to YUI framework, then it loads code under test from <strong>Form.js</strong> and the actual unit tests from <strong>UnitTests.js</strong>. <br />Below is the content of <strong>UnitTest.js</strong> file. <br /><script src="https://gist.github.com/3979105.js"> </script> and the end result in a web browser:<br /><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh4bc5vMEe0jmmG8vHO5kcT7efvsxFF_wJeFE64jOkZEGAK1QkBY0VWfnSQg9WF9QhcebrEam_h1tULziN7oPalUXCAe-gxgxtLNsgp4IN2PaSKWs-rEMZ24UcNWf93iPa1MsUuhgrad8g/s1600/image_4.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh4bc5vMEe0jmmG8vHO5kcT7efvsxFF_wJeFE64jOkZEGAK1QkBY0VWfnSQg9WF9QhcebrEam_h1tULziN7oPalUXCAe-gxgxtLNsgp4IN2PaSKWs-rEMZ24UcNWf93iPa1MsUuhgrad8g/s320/image_4.png" width="192" /></a></div><br /><br /><a href="http://11011.net/software/vspaste"></a><a href="http://11011.net/software/vspaste"></a>The only thing that requires explanation here is the difference between Validation and Submission tests. The <strong>validate</strong> method is a standalone method that does not have any dependencies hence its unit testing is very simple and boils down to passing different sets of input parameters and asserting the correct results. <br />The unit testing of the <strong>submitForm</strong> method on the other hand is not that simple because the method relies on <strong>getPredefinedReasons</strong> and <strong>getCustomReason</strong> methods that grab data from the DOM and <strong>validate</strong> method that ensures that the user provided data is valid. We are not interested in the way those methods work while unit testing <strong>submitForm</strong> method. They actually gets in the way. What we need to do is to mock them and focus on making sure the <strong>submitForm </strong>method shows correct messages to the user. <br />The mocking framework takes care of that. All we have to do is create an anonymous method that encapsulate all our mocking logic. The mocking framework will make sure that once the test is done the global state gets rolled back to where it was before the test was executed. The way Jack is designed reminds me of <strong>using</strong> and <strong>IDisposable</strong> in C#.<br />As you can see the jQuery based code is encapsulated into getXXX methods which makes easy to mock them. Some people don’t mock jQuery and instead try to recreate enough DOM elements on the test page to satisfy the tests. I don’t like this approach because changes to either HTML or jQuery might force us to change the unit tests which makes them brittle. It is like using L2S in a unit test. It’s not a unit test, it is an integration test. Other approach I’ve seen is to mock jQuery methods one by one. This is a slightly better approach but still changes to jQuery queries can break the tests. It is like trying to mock a sequence of Linq extension methods. It’s way easier to mock the whole method that simply encapsulates the query.<br />If this was a C# code then getXXX methods would be defined on some kind of repository and validate method would belong to a validation component. Both of them would be injected to the main logic that handles the form submission. If this was an ASP.NET MVC app that would be a controller. It was not my intention to structure the code in this way but that’s what I ended up with writing the tests first.<br />You might wonder why I haven’t shown the actual code yet. Well, I did it on purpose. The unit tests should be enough to understand what the client side code does and what its desired behaviour is. If this is still not clear then it means that either the code is not structured properly or that the names are not descriptive enough.<br />And that would be it. The last thing to do is to show the actual code: <br /><script src="https://gist.github.com/3979127.js"> </script> <pre class="code"></pre><div class="wlWriterEditableSmartContent" id="scid:8eb9d37f-1541-4f29-b6f4-1eea890d4876:67e3b872-6b62-4770-8d62-af6e543968c5" style="display: inline; float: none; margin: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;"><br /><div><a href="https://dl.dropbox.com/u/7579883/Blog/JSUnitTesting_2.zip" target="_self">JSUnitTesting.zip</a></div></div><div class="wlWriterEditableSmartContent" id="scid:0767317B-992E-4b12-91E0-4F059A8CECA8:090b9cc2-d8c9-4452-9b6e-16e8c15775a1" style="display: inline; float: none; margin: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;"><br /></div>Pawel Pabichhttp://www.blogger.com/profile/05213210256288888149noreply@blogger.com3tag:blogger.com,1999:blog-168147483884969000.post-23833932723668737912010-06-28T22:59:00.000+10:002012-11-06T00:32:41.192+11:00Blog upgraded from Subtext 2.1 to Subtext 2.5<p>No problems so far and I really like the new dashboard. Good job guys!</p> Pawel Pabichhttp://www.blogger.com/profile/05213210256288888149noreply@blogger.com0tag:blogger.com,1999:blog-168147483884969000.post-84042055133244408482010-06-20T22:58:00.000+10:002012-11-06T00:32:41.233+11:00Generic retry logic in PowerShellI spent recently a bit of time writing PowerShell scripts that deploy a system that I’m working on both locally and to QA. Basically, you get the latest code from TFS, run single PowerShell script and press F5 :). Why? Have a look at <a href="http://www.joelonsoftware.com/articles/fog0000000043.html"><strong>The Joel Test</strong></a>. <br />Anyway, while writing those scripts I needed to implement a basic retry logic in multiple places. It turned out that PowerShell supports closures and that you can pass any part of the script to a function as an argument. Having all of that at my disposal made my task very easy:<br /><br /><pre class="brush: ps">function Execute-Command($Command, $CommandName) {<br /> $currentRetry = 0;<br /> $success = $false;<br /> do {<br /> try <br /> { <br /> & $Command;<br /> $success = $true;<br /> Log-Debug "Successfully executed [$CommandName] command. Number of entries: $currentRetry";<br /> } <br /> catch [System.Exception] <br /> {<br /> $message = 'Exception occurred while trying to execute [$CommandName] command:' + $_.Exception.ToString();<br /> Log-Error $message;<br /> if ($currentRetry -gt 5) {<br /> $message = "Can not execute [$CommandName] command. The error: " + $_.Exception.ToString();<br /> throw $message;<br /> } else {<br /> Log-Debug "Sleeping before $currentRetry retry of [$CommandName] command";<br /> Start-Sleep -s 1;<br /> }<br /> $currentRetry = $currentRetry + 1;<br /> }<br /> } while (!$success);<br />}<br /></pre><br /><br />And this is how you can use it:<br /><br /><pre class="brush: ps">$command = { Get-ChildItem $Folder -Recurse | Remove-Item -Recurse -Force}; <br />$commandName = "Delete content of [$Folder]"; <br />Execute-Command -Command $command -CommandName $commandName;<br /><br /></pre>Pawel Pabichhttp://www.blogger.com/profile/05213210256288888149noreply@blogger.com0tag:blogger.com,1999:blog-168147483884969000.post-57510846752357512792010-06-20T22:54:00.000+10:002012-11-06T00:32:41.214+11:00CloudCamp is coming again to Sydney!<p><strong><a href="http://www.cloudcamp.org/sydney/2010-08-06">This time</a></strong> you need to take half a day off to attend it but I believe it’s well worth your time and I really enjoyed <a href="http://pabich.eu/blog/archive/2009/09/01/couldcamp-ndash-a-bunch-of-loose-thoughts.aspx"><strong>the previous camp</strong></a><strong>.</strong></p> Pawel Pabichhttp://www.blogger.com/profile/05213210256288888149noreply@blogger.com0