From dirty data to dependable datasets in Nigeria

A Nigerian distributor combines an order spreadsheet, a payment export and a delivery log. One file records an amount as “₦15,000”; another stores “15000”. An order number has lost its leading zero. Two customer names look similar. A delivery address includes a landmark that does not fit neatly into a street field.

This is a fictional scenario, but it captures the questions a cleaning pipeline must answer. Which differences are formatting? Which records describe the same event? Which gaps need a person to investigate? A file can look tidy while still giving the wrong answer.

For teams working with Nigerian business data, useful cleaning starts with understanding how a record was created and what it represents. This guide sets out Rust7’s recommended approach: preserve the source, make changes explicit, and release data only when it meets the requirements of its intended use.

1. Define “clean” for the actual task

A sales report, a delivery-routing system and an AI evaluation dataset need different things. A state-level market analysis may tolerate a missing street number. A delivery task needs much more precise location information. A language model evaluation may need the original wording of a customer message, including spelling variation.

Write a short data specification before transforming anything. Define the unit of observation—an order, order line, payment, customer or delivery attempt. List required fields, accepted formats, currency and time conventions, permitted uses, and the conditions that should stop a release.

For an order table, an order identifier might need to be unique. For an order-line table, the same identifier should repeat across products. A blanket “remove duplicate order IDs” instruction could delete valid business activity.

2. Keep the source and measure the problems

Keep an unchanged source copy under the project’s agreed access and retention controls. Give each incoming batch an identifier and record its origin, extraction time, reporting period and file checksum. Preserve source row references so a cleaned value can be traced back to its input.

Then profile the batch: count rows, inspect data types, measure missing required fields, identify repeated keys and examine unexpected values. Break those findings down by contributing system, branch or period where relevant. A sudden gap in delivery dates from one branch is a more actionable finding than a single completeness score for the whole business.

Record the starting measurements. Otherwise, a pipeline can appear to improve quality simply by discarding the records that were hardest to process.

3. Standardise formats with Nigerian context

For monetary values, separate the amount from the currency. In our example, “₦15,000” can become an amount of 15000.00 with currency NGN, because the symbol establishes the currency. The bare value “15000” needs source metadata before the same assumption is justified. Document whether an export stores naira, kobo or another unit; a valid number in the wrong unit remains wrong.

For dates, 04/05/2026 is ambiguous without a declared convention. Use the source’s documented format, retain the original value, and send unresolved cases for review. Store an agreed standard representation once the meaning is established. Do not invent a time or timezone for a date-only field.

Treat phone numbers and business identifiers as text. Preserve leading zeros and keep the raw value alongside any normalised version. A phone number with a plausible format is not proof that the number is active or belongs to a particular customer.

For addresses, separate state, local government area, locality, street details and delivery instructions where the evidence permits. Retain useful landmark descriptions. A place name alone may not establish a unique location, and a rough geocoding result should not be labelled a verified building.

NIPOST’s current FAQ describes its Digital Postcode as an 11-character building reference that can be used alongside a written address. It also explains that multiple people or organisations in a building can share it. That makes it useful location context, but unsuitable as a unique customer identifier. Store the reference type and verification status rather than forcing every address code through one assumed numeric format. [1]

4. Resolve duplicates and joins deliberately

Separate a repeated export row from a second legitimate transaction. Prefer stable source identifiers and documented business keys. When identity is uncertain, create a candidate match for review instead of silently combining records.

Similar names are evidence to investigate, not enough to prove that two businesses are the same. OpenRefine illustrates this distinction: its clustering tools suggest similar text values, while the user approves which merges are applied. A reviewer should be able to keep two similar values separate. [2]

Check joins just as carefully. If one order has three payment events, joining the two tables can repeat the order total three times. Define whether the relationship should be one-to-one, one-to-many or many-to-one, then compare row counts and relevant totals before and after the join.

Tools can enforce part of this contract. pandas provides join validation and an indicator for unmatched records; its documentation also warns that null join keys can match each other. Missing identifiers therefore need explicit handling before a merge. Passing a join check still does not prove that the chosen key identifies the right entity. [3]

5. Keep missing values and local language meaningful

A missing payment amount is not zero revenue. A missing delivery timestamp is not automatically a failed delivery. Keep “unknown”, “not collected” and “not applicable” distinct when the source supports those distinctions. If a value is estimated, flag it and document the method rather than presenting it as an observation.

Apply the same care to text. Nigerian Pidgin and mixed-language messages should not be automatically rewritten into standard English and treated as corrected originals. Preserve the source wording; keep any translation or normalised form in a separate field with a clear purpose.

A 2024 study by Lin and colleagues examined Nigerian Pidgin spelling variation and found benefits from modelling that variation in translation and sentiment-analysis experiments. Those findings are specific to the study’s tasks, but they reinforce a practical point: variation can matter to model performance. Cleaning rules should be evaluated against the intended language task. [4]

For AI projects, protect the evaluation boundary too. Learn imputation values, scaling parameters and other fitted preprocessing from training data only, then apply the fitted transformations to held-out data. scikit-learn documents how learning these transformations from the full dataset can inflate evaluation results. [5]

A small example: what changes, what waits

The following records are invented. Assume the agreed output is one row per order, with a confirmed currency and an unambiguous order date required for release.

This produces three useful outputs: records that meet the release criteria, an exception queue with reasons, and a report explaining the changes. A record held for review has a visible status and owner; it has not simply disappeared.

6. Validate, review and release a reproducible version

Run checks after transformations as well as at intake. Verify required fields, key uniqueness, declared categories, join relationships and plausible business sequences. A refunded transaction may legitimately have a negative amount; the rule must reflect the transaction type rather than reject every negative number.

Set different responses for different failures. A missing required source identifier might block the batch. An optional-field gap might produce a warning. An uncertain match might enter a review queue. Great Expectations supports actions based on validation results, but the business still has to define which outcomes justify release. [6]

Record the input batch, code and rule versions, reviewer decisions and output version. A retry should not duplicate records or silently read a changed source. Apache Airflow’s guidance recommends repeatable task outcomes, fixed input partitions and avoiding partial outputs. Apply those principles whichever orchestration tool you use. [7]

At handover, include field definitions, coverage, transformation notes, unresolved limitations and a reconciliation of input records to released, excluded or held records. Compare quality measures across releases, and assign someone to investigate changes. A dashboard without a response process cannot repair a failing source.

What a buyer should ask for

1. Show the path from source to output. Include an example of a changed record and its reason.
2. Explain the exceptions. Report missing fields, unresolved matches and excluded records with clear denominators.
3. Show what was preserved. Keep original language, address context and source identifiers where relevant to the agreed scope.
4. Make the delivery repeatable. Identify the dataset version, cleaning rules, quality checks and update process.

Rust7’s view is that preparing Nigerian data requires both engineering discipline and local understanding. The useful result is a dataset whose changes can be explained and whose limits are visible to the next team.

Scoping a project? Explore data sourcing and preparation with Rust7. If you maintain a source, our data partnership page explains how to introduce it. Start with a description of the source and intended use.

Sources & research notes

Primary sources checked on 23 October 2026. The workflow and examples are Rust7’s recommendations, not claims about completed client work or the prevalence of errors across Nigeria. Tool documentation supports the mechanisms described; each project needs its own rules.

1. NIPOST · Digital Postcode FAQ

   Current service documentation, checked 23 October 2026. Defines a building-level reference, not a customer identifier or a guarantee of complete address coverage.

2. OpenRefine · Cell editing and clustering

   Primary tool documentation. Describes candidate clustering, user-approved merges and transformation limitations.

3. pandas · Merge validation

   Primary API documentation. Covers join cardinality checks, source indicators and null-key matching behavior.

4. Lin et al. · Modeling Orthographic Variation Improves NLP Performance for Nigerian Pidgin

   LREC-COLING 2024. Evidence from machine translation and sentiment analysis; not a benchmark of all Nigerian text or business datasets.

5. scikit-learn · Common pitfalls and recommended practices

   Primary documentation on consistent preprocessing and preventing training/evaluation leakage.

6. Great Expectations · Trigger actions based on results

   Primary documentation on validation-driven actions. Release gates and review policies remain implementation decisions.

7. Apache Airflow · Best practices

   Primary documentation on repeatable tasks, retries, stable input partitions and avoiding incomplete results.