We still build the dashboard ourselves when the software gathers the data

In an earlier article I explained why we no longer build one-off dashboards for clients, and why we build the foundations that let their own AI tools produce good ones instead. That holds for most reporting work, though there is one kind of dashboard we still design and build ourselves. It is worth explaining why, because the reasons turn out to be much the same.

Some software exists to collect the data

Some of the systems we build exist to gather information in the first place. The aviation assessment platform we built for a specialist consultancy is a good example. Crews and assessors answer structured surveys and record observations flight by flight, evaluators review each session, and the whole point is to collect that information carefully so that it can be presented, compared and searched for patterns later.

In software like this, the dashboard is part of the product. The people using it know most of their questions before the first session is recorded (how a campaign is progressing, how errors are spread across the phases of a flight, whether a rating differs from one base to another), and they ask those questions again and again. A dashboard built to answer them well is worth the effort, and it would make little sense to have a chat tool draw it afresh each time.

What the dashboard does

At the top are a few headline figures for the people running the surveys: how many sessions are not yet started, under way, awaiting review or reviewed, campaign by campaign, along with how long sessions take on average to move from one stage to the next. Open any figure and you see the sessions behind it.

Below that, each user has their own set of charts and tables, arranged under headings. One set of filters narrows every chart at once, by survey, campaign, dates, and whether you want the crew’s answers, the evaluator’s, or both. Click a bar, a slice or a table cell and a panel lists the sessions behind it, each with a line saying why it is there and a link to that exact place in the session. A chart can be saved as an image for a report.

Administrators can also design new charts themselves, choosing what to count, how to divide it and how to draw it, with a preview that redraws as they go. The choices are built with the subject in mind. Errors can be shown per sector flown, for instance, so that a campaign with more flights does not simply look worse.

Why its figures can be trusted

When I compare it with the foundations we build for AI-generated dashboards, nearly every one is already there.

The data is clean and traceable from the start. Surveys are versioned, so once results exist a survey is superseded rather than edited, and observation sequences can only be recorded in a valid order. Every figure leads back to a session and a question.

Each figure is defined once. What counts as “awaiting review” on the headlines is worked out exactly as it is for the lists of sessions beside them, so the two always agree. If two colleagues see different figures on the same chart, it is because each counts only the sessions they are permitted to see, and the guide says so plainly.

Charts are built from parts known to work. Each headline and each kind of chart is a small component that knows how to fetch its own data and how to drill down. A new kind of chart is a new component, which leaves the dashboard itself alone.

The guidance is kept up to date. The written guide for administrators is changed in the same piece of work as the page it describes, so it does not drift from what the software actually does.

It is honest about uncertainty. The analysis pages, which ask whether something happened more often in some circumstances than others, will say “Too few to say” when there are not enough sessions, and “No difference shown. That is not evidence that there is none.” when nothing stands out. Automated scans put forward possible findings, but nothing reaches anyone else until an analyst has reviewed it, written a note and published it.

There is AI in it too, and it works the way I described the learning loop in the earlier article. A language model can tag observers’ written comments so that they can be counted, but no tag can be used in an analysis until a person has checked a sample of texts and agreed with the model often enough. We also keep a set of generated test data with known patterns planted in it, so that we can check a chart or an analysis finds what we know is there, and the software never mixes test data with real sessions without saying so.

Where the difference lies

This platform can have a dashboard built by hand because its data is shaped at the moment it is collected. We designed the questions, the structure and the definitions alongside the client, so every figure had an agreed meaning before anyone asked for a chart, and the set of questions worth asking is fairly well known.

Most organisations’ data is not like that. It sits in a finance system, a CRM and a good many spreadsheets, each set up by different people for different purposes, and the questions people want to ask of it are many and change from week to week. Nobody could build a dashboard by hand for each of those questions, which is exactly where AI earns its place, provided it has the same foundations to work from that this platform has by design.

So I do not see the two approaches as rivals. In both, drawing the charts is the easy part, and what makes them worth believing is the work underneath. If you are building software that collects data of its own, or trying to make sense of data that was never prepared for it, I would be glad to hear what you are working on.