Book Review - Exploring Everyday Things with Ruby and R by Sau Sheong Chang

 

Disclaimer: I received a free (electronic) copy of this ebook (Exploring Everyday Things with Ruby and R by Sau Sheong Chang) from O'Reilly as part of the O'Reilly Blogger Review Program, which also requires me to write a review about it. That aside, I would have purchased this book this year anyway, and would have reviewed it on this blog too.

About me and why I read this book

I have been programming professionally for about 8 years, mainly business applications and reporting, so I already have quite some love for data. While I haven't used math much in my day jobs, I liked (and was good at) it in high school, including taking extra classes - so I have learned basic statistics. Refreshing and advancing my data analytics skills is one of my goals this year, and reading this book was part of that plan - I have heard that R is one of the most powerful languages for statistical analysis currently available.

About the book

The book is written assuming basic understanding of programming and sets two goals:

• to awaken the curiosity in the reader to go out and explore things and search for explanation, models, and experiments to validate understanding;
• to show you some basic, but practical R and Ruby.

While the author intended each chapter to be more or less self sufficient, I have found it to be better read sequentially, especially the simulation chapters.

Ruby

I had no trouble with the code examples, even though I have only programmed about half an hour total in my life in Ruby. Beware that the only knowledge you gain about Ruby is the bare minimum required, so you'll have to put aside your thirst for complete understanding of the language and its ecosystem. If you need to have a proper understanding to work in a language (which I don't think is necessary), you are better off either reading a Ruby book first or using your favorite language to obtain the data - the code is easy to port.

Making me curious

I have had a lot of wow/a-ha moments, both about the topics chosen for discussion as well as the math/algorithmic ideas. You may find that you disagree with some of the conclusion the author draws, and it is emphasized during the introduction that the goal of the book is not to convince you about these conclusions, but to demonstrate the journey from question to conclusion in order to equip you with tools to enable you doing the same. This is mostly achieved.
I award extra bonus points for mentioning the limitations of the used analytical tools - I don't think I would trust any book/article/blog post which presents something without its downsides!
Not all examples are exactly everyday (e.g.: an analysis of going to work by car vs. public transportation would have been more everyday than how to simulate the flocking of birds), but they cover a wide breath of topics. The processing and analysis of the data is always challenging enough, plus your general knowledge is expanded.
One thing I was missing is a description of a really important part - being a layman, how do I go about finding which algorithms to use? While it isn't a book about Research 101, a description of the search process would have been great. You can of course always google, but when entering a new topic I find guided search helpful - which are some of the trick keywords, which sites to prefer/avoid, etc. On the other hand, enough methods are described that just properly learning and understanding them would make me a much better statistician already. Once done with that I could just fall back reading through the R packages and methods, hoping that if I have seen a word before it would emerge from my passive knowledge when I'm faced with a matching problem.

The R language

The book does a solid job to help you get started. It demonstrates enough language features to enable to you experiment with it for work projects (e.g.: use MySql as a datasource, create packages, etc.); points out the R component/library hubs to look for community packages; and recommends further learning resources.
The code examples are like most programming book snippets - procedural, (mostly) everything is located in a single method/script. Not a tangled-spaghetti mess that makes one despise it in legacy code, only it makes for a lower signal/noise ratio and requires more effort from the reader. Guess its a genre problem, so if you have read other programming books, you shouldn't have any problems with this one.

Technical comment: the ebook isn't formatted to play nice with the Kindle DX, and while in print the code block might be only broken between left & right pages, on the kindle it makes for awkward read.

The exposed APIs suggest that R is a bit too ceremonial for my taste, but that could be abstracted away for the project that warrants R's use. I have also used a number of visually great .NET UI third party components that were a pain to work with from a programmer's perspective, yet helped us create a great product. Plus things that feel alien first become second nature after enough practice, so it isn't a big deal. I plan to take a look at NumPy as well, and defer the decision whether to dive deeper into R (possibly via using F# 3.0 type providers for R).

Overall

The book hasn't left me in awe, but it didn't feel like a chore to read as some other books. I got the taste of R that I wanted when I picked up my copy to read. On top of that, I have learned about fun things, and it also added books to my reading (wish)list (e.g.: The Grammar of Graphics by Leland Wilkinson, Armchair Economist by Stephen E. Landsburg, and more). This is no definitive guide on R, but to wet your appetite and get you started, it is a good one I can recommend without reservations.

Book Review - Programming Collective Intelligence by Toby Segaran

Disclaimer: I received a free (electronic) copy of this ebook (Programming Collective Intelligence by Toby Segaran) from O'Reilly as part of the O'Reilly Blogger Review Program, which also requires me to write a review about it. That aside, I would have purchased this book this year anyway, and would have reviewed it on this blog too.

About me and why I read this book

I've been programming professionally for ~7.5 years, mainly business applications and reporting, so I already have quite some love for data. While I haven't used math much in my day jobs, I liked (and was good at) it in high school, including taking extra classes - so I have learned basic statistics. Refreshing and advancing my data analytics skills is one of my goals this year, and reading this book was part of the plan.

About the book

The book introduces lots of algorithms that can be used to gain new insight into any kind of data one might come across. The explanations are broken up into digestible chunks, and are supported by great visualizations. While understanding of the previous chunks is required for the later ones, this allowed me to read through most of the book on the train to and from work.

Each of the algorithms is illustrated with real world application examples, and examples where applying them doesn't make sense are brought too. The exercises at the end of the chapters are applied and not purely theoretical - and coming up with exercises from the domain I work with every day was pretty easy! The book is really inspiring, which is great for an introductory book!

In addition to the well written, gradual introduction, the book has a concise algorithm reference at the end, so when one needs a quick refresher, there is no need to wade through the lengthy tutorials.

While the prose and the logic of the explanations are great, I have found the code samples hard to follow: really short, cryptic variable names; leaky abstractions; inconsistent coding style just to name a few. Some code samples are actually incorrect implementations of the given algorithm and there are antipatterns like string sql concatenation in the code without a warning comment to the reader to remind them it's a bad practice.

Nonetheless, it is great to have actual code to play with, just the initial reading and reviewing of it requires some extra effort.

The book claims that you don't need previous Python knowledge to understand the code samples, which I can't confirm (I use Python at my day job), but I wouldn't be surprised if not knowing Python could make understanding the code even more difficult (I've actually learned a few new language features from the samples!). Also, the Python language has come a long way since 2.4, which is the version used in the book - and that old version makes the code feel dated.

The book was written in 2007, but is not dated. First, the foundations of any topic tend to be timeless, and the most recent algorithm the book describes was published in 1990. The Table of Contents is comparable to more recently written ones (though I haven't read other introductory books yet).

In summary: I would recommend it as a great introductory book!

Book Review - Python Testing Cookbook by Greg L. Turnquist

I have been doing (developer) automated testing for years now, but I recently moved from .NET to Python. Recently, at one point I suggested to collegues that we try Concordion, only to learn that there is the doctest module that could be used to achieve similar result (more about that in a later post). Remembering my own advice: When In Rome, Do as the Romans Do, I set out looking for a Python specific book about testing - and the Python Testing Cookbook by Greg L. Turnquist book seemed to be a good fit based on reviews.

---

Overall, I liked the book, and it lived up to my expectations - it provided me with a list of tools and some sample code to get started with each of them.

Beware that it is an entry level book - if, like me, you are already familiar with the testing concepts, and are looking for a book to learn about advanced testing concepts, theories, this book might be too little for you (or just read through the "There is more" sections of the recipies). But it is great for someone new to testing - though discussions with (and code reviews by) someone experienced in testing should accompany the reading.

Despite the below criticisms, which are meant to be rather a companion to the book than an advice against it (i.e.: probably the only book I wouldn't find anything missing from and nothing to criticise about would be written for me, in real time, based on my immediate needs). The fact that the list is short shows how I found the rest of the book valuable, with great advices that go beyond the cookbook format (why you shouldn't use fail(), why there should be one assert per test, etc.). While I don't see eye to eye on each topic with the book, but just as the book is not written in a "my way or the highway" style, I will not get into minor differrences of opinion.

Format of the book

Each chapter explores a topic, with multiple specific recipes. Each recipe is relatively self contained, so if we are in a hurry and need to get to the solution of one specific problem without reading the whole book/chapter, it's possible. However, for those reading whole chapters, this results in a bit of repetition - I had to fight the urge to just gloss over the code I (thought) I had seen before.

Each recepie follows the format of

• stating the problem


• showing code that solves it


• explaining how the code works


• and finally, providing warnings about pitfalls and problems in the code, and some further advice

While this format is easy to follow, it has a few drawbacks.

• until I got used to this style, I often found myself cursing out loud like the code reviewers in this comic while reading the code that will later be explained to be bad/antipattern.


• each recipe has a lot of additional testing insight, potentially unrelated to the tool being demonstrated - but one can miss these, thinking "oh, I know all about doctest, I'll just skip that chapter"


• for people in a hurry, just scanning the cookbook and copying (typing up) the code - there is nothing to indicate in the code that there is an antipattern there, only in the later paragrpahs - which you might not read when in a hurry. Just thinking about the examples where the unit tests have no asserts but only print statements gives me the shivers (and it's even used for illustration in the chapter about Continious Integration!).

What was missing from the book

• About testing legacy code, I was missing two things:
  
  
  • a pointer to Michael Feather's classic book, Working Effectively with Legacy Code
  
  
  • a warning about a mistake I have seen almost everyone (myself included) make when getting started with testing legacy code: don't write tests just because you can - only add cases for the area you are working on and only add enough to cover your current needs. This is hinted at, but I've found it's important to state it explicitly.
  
  
  
  


• Notes about test maintainability
  
  
  • I strongly disagree with the approach of having one test class per physical class, and test methods directly excercising the class's method. I've found these can lead to maintainability problems down the road, so I prefer to introduce helper abstractions (e.g.: assert_roman_numeral_conversion(expected_result, roman_numeral_string) method) for most of my tests, and organize test methods by logical scenarios instead of mirroring code organizational units (on successful login, validating user input, etc.). These abstraction (indirections) makes it easier to update tests after an API change or refactoring. It might sound like an advanced topic, but I think it's a key concept for effective tests, and one that people should be exposed to early (just after they've made the mental jump from throwaway main methods with visual/human assertions to automated tests with automated assertions).
  
  
  • Acceptance Testing - it is notoriously difficult for us programmers to write good acceptance tests that are both maintainable and readable by the customers. I'm rather sure that in the example given in the book, the customers would not be interested in knowing which html tag contains the price in the output.
  
  
  
  

Minor criticisms

• there is an inconsistent level of detail and further pointers. E.g.: while optparse is explained in detail, virtualenv and setuptools are glossed over.


• In addition to the assertless test methods, the other thing that shocked me was the example in the doctest module that - to illustrate that the test works - introduced a bug in the test code. While the fact that test is code and thus can be buggy should be emphasized, but that wasn't the case here. This could leave the reader wondering why exactly we introduced the bug in the test code - aren't we testing the application?


• The book is careful not to fall into the trap of elitist speak that might alienate people, but saying that coupling and cohesiveness are subjective terms is just providing gunpowder to unwinnable arguments(*).

Interesting notes

• This might be a cultural thing (I'm coming from .NET), but I've found it rather surprising that such an entry level book talks about test runners, and write custom test runners. It's useful knowledge, just something that I have not seen mentioned in so much detail in the Java/.NET world so early. Maybe it's got to do with IDEs being more widespread, where running a subset of the tests is easy.

As said, the book lives up to the expectations, so if you would like to get a quick and practical introduction to testing in pytohn - both tools and concepts, I can recommend this one for you.

---

(*) Reminds me of a story from long ago. The team in question has decided to use bind variables for all SQL invocations (I've said it's been some time ago) to prevent SQL Injection. The one programmer wrote a stored procedure that concatenated the SQL command in its body... and argued that this is just a matter of style. At least the procedure was invoked from the application using bind parameters...