- Spreadsheets applications allow data to be stored with low
development overheads, but also with low data quality.
Reporting on data from such sources is difficult using
traditional techniques. This case study uses text data
mining techniques to analyse 12 years of data from dam
pump station maintenance logs stored as free text in a
spreadsheet application. The goal was to classify the data
as scheduled maintenance or unscheduled repair jobs.
Data preparation steps required to transform the data
into a format appropriate for text data mining are
discussed. The data is then mined by calculating term
weights to which clustering techniques are applied.
Clustering identified some groups that contained
relatively homogeneous types of jobs. Training a
classification model to learn the cluster groups allowed
those jobs to be identified in unseen data. Yet clustering
did not provide a clear overall distinction between
scheduled and unscheduled jobs.
With some manual analysis to code a target variable
for a subset of the data, classification models were trained
to predict the target variable based on text features. This
was achieved with a moderate level of accuracy.
(from crpit.com)
(local if available)
