Spending too much time preparing data and not enough time analyzing and visualizing it? IBM and Datawatch have a combined solution to help.
With the recent IBM and Datawatch partnership, announced at the Gartner Business Intelligence & Analytics Summit, March 14-16, you can use Datawatch Monarch to extend the data access and preparation capabilities of IBM Watson Analytics and IBM Cognos Analytics.
Use this collection of self-service data preparation and analytics tools to take your data from acquire, prepare, and merge … to explore, visualize and analyze … a lot quicker.
- Datawatch Monarch – quickly convert and combine semi-structured data from multiple files and formats and then export the structured data directly to Cognos Analytics and Watson Analytics.
- Cognos Analytics – build interactive dashboards from a collection of data sources and visualizations.
- Watson Analytics – explore your data using cognitive technology and guided analytics.
Here’s a summary of an analytics journey using these tools on a collection of data that included open city 311 data, weather data and demographics.
Acquire, prepare, and merge … explore, visualize and analyze
The city of Boston, MA – like many other cities across the country and around the world – logs hundreds of thousands of 311 service requests each year. These requests come from citizens and city workers reporting the need for every-day, but important, city services for things like snow plowing, street cleaning and pot hole repair.
For this example, we used Datawatch Monarch, Cognos Analytics and Watson Analytics to import, merge and analyze the 311, weather and demographic data acquired from PDFs, XLS files, and web pages. We started out looking just at the combination of 311 and weather data, but this brought us to the idea of adding demographic data to see what other interesting correlations might exist. This combined data gave us the big picture of the city’s primary customers (the citizens) and their related service requests.
Datawatch Monarch made easy work of combining our three disparate data sources – Boston 311 requests, weather data and demographics for Boston’s neighborhood – into one structured data file for analysis. Using Monarch’s core feature of importing and parsing semi-structured data from different file formats, we were able to quickly acquire the data from a range of sources, including PDFs and XLS files and web pages.
First, we imported almost 200,000 records of Boston 311 requests from 2015 (https://data.cityofboston.gov/). Then we matched that data with several hundred thousand weather data points by zip code and date/time stamp. Finally, we joined all of that data with demographic data from a web page for Boston’s 20+ neighborhoods. For the demographic data, we were able to automatically extract and parse that data directly from an HTML table on the following web page: https://en.wikipedia.org/wiki/Boston.
To join the data, we used Monarch’s intuitive and visual interface to arrange the multiple data tables and configure the data join properties. With a couple clicks, we converted and merged all the different input data into one structured data table for analysis.
Dashboard creation with Cognos Analytics
Bringing the data directly from Datawatch Monarch into Cognos Analytics allowed us to quickly get to work building dashboards using the easy-to-use and visually interactive features in Cognos Analytics. We were able to create visualizations and dashboards using simple drag and drop actions and animate the data using smart filters and data players.
Using dashboards we were able to visually combine and compare the three different data sources on a single canvas. We could investigate how long it takes to resolve certain requests based on type of request and neighborhood where the request came from. We could also blend in the weather data by looking at request resolution compared to temperature levels and weather conditions. This allowed us to quickly view what happens to requests during the summer months or during heavy snow conditions.
To further explore the data and run predictive analytics on it, we imported the data directly from Datawatch Monarch into Watson Analytics.
We started exploring the data by using the natural language-based question feature in Watson Analytics. The column renaming and data cleansing we did in Datawatch Monarch enabled us to more easily ask questions in plain English in Watson Analytics like: “What is the number of Service Requests by Neighborhood?”
To do this, we ran a prediction analysis in Watson Analytics with the prediction target set to Resolution Efficiency. Are requests resolved in “less than a week”, “2 to 4 weeks”, “more than 6 months” or other time period?
The results showed a predictive strength of 70% that Resolution Efficiency is impacted by which department the request is assigned to (Department Assigned). So, if your request goes to one department it could be resolved pretty quickly, but if your request gets assigned to a different department it could take longer. The spiral diagram in Watson Analytics visually and interactively summed up the results.
- Using Datawatch Monarch, we went from three separate collections of data, all in different formats and domains, to a single unified data set ready for smart data discovery.
- With Cognos Analytics, we quickly built interactive visualizations and dashboards to investigate and visualize trends across 311 requests, weather and demographics.
- With Watson Analytics, we used smart data discovery, asked questions using NLP and ran predictive analytics to find insights that typical data discovery would not have found.
Use this combination to iterate through your data analysis faster, more efficiently and more broadly by merging and visualizing a wider collection of data and information.
Reposted from the IBM Watson Analytics Community, original Expert Blog “Combining the strengths of Cognos Analytics, Watson Analytics and Datawatch Monarch, insights about 311 calls become clear” by Joseph True, Content Data Scientist | Product Experience & Design | IBM Watson Analytics – 3/25/16.