While looking at a (relatively small, 1.7 million records) big data example of New York Yellow Cab taxi trips, I am coming to the conclusion that the best place (if as we do you are using Microsoft tools) for initial analysis, including the all important first step of finding outliers/errors, is Azure Machine Learning Studio (Azure ML, as opposed to Excel, Power BI or bespoke analysis using e.g. Kendo UI).
Why Azure ML for initial analysis?
- It loads data quite quickly (e.g. just over a minute to import almost 2 million records from an Azure SQL database). This is currently much quicker than Power BI.
- It automatically produces histograms and box plots of numeric fields (see the images below, and above, where the field FareAmount has been selected). We can tell immediately from the box plot that there are several outliers (and in fact probable errors that will need to be either corrected or removed, in that FareAmount should not have negative values!).
As a “proper” programmer, used to programming in heavy duty, compiled languages like C# (and before that C++ and C), my reaction on discovering during my Data Science journey that R and Python are heavily used by data scientists was: why??
Why would anyone use an interpreted language, which is therefore bound to be slower, and why would anyone go to the trouble of using yet another language when there are perfectly good compiled languages around like C#, F# and VB.net?
The answer seems to be partly that R and Python are free (open source), and also because R and Python have excellent visualisation tools, which the other languages currently lack.
I am delighted to have now completed the Microsoft Professional Program for Data Science. It has been 10 online courses (taking a total of 322 hours) over just more than 11 months and my average (mean) mark over the 10 courses was 96.6%. The final course was a capstone project which involved analysing data from the 2015 earthquake in Nepal, building a model to help predict the degree of damage to buildings (amongst other things to help emergency response teams prioritise rescue efforts) and producing a report on this. This was an extremely practical way to complete the course.
I have created a series of slides (collected together in a Microsoft Sway online document) showing the main stages of my journey. You can see them at https://sway.com/lsUjwGITuGFpsHIM?ref=Link