Validate and deliver

With your CSV, MCF, and config.json ready, the last steps are to check the formatting, record what you changed, and package it all for submission.

Validate the files

Data Commons provides a Java tool that checks the formatting of the files before import. It validates that the CSV, MCF, and config.json follow the structure the pipeline expects, flagging errors in variable definitions, entities, or column mappings.

The only prerequisite is Java 11 or later. Download it from Java Downloads | Oracle. Confirm the installation with:

java -version

Then download the .jar from the Assets section of the latest release and run:

java -jar <path-to-jar> lint <list of mcf/tmcf/csv files>

The lint command can evaluate MCF files on their own. To evaluate CSV files, you must also provide a matching TMCF file. If it finds warnings or errors, the tool produces a JSON report with examples of the problems.

For more on the tool and its options, see the import tool usage docs.

Document the transformations

After adapting the data to the format Data Commons requires, document the transformations you applied during preparation. This is captured in a “Formatted data” template — its purpose is to transparently record the changes made to the original data (structural changes, column-name standardization, derived variables, and any other transformations needed for compatibility with the Data Commons schema).

General information

Field	Description
change description	Short description of the transformations relative to the original version
original file	Name of the source file before any modification
final file	Name of the file produced after processing, to be used for ingestion

Transformations applied

Field	Description
transformation type	Main type applied — column renaming, format change, aggregation, record filtering, or creation of derived variables
modified columns	Columns changed during processing
created columns	New columns or variables created during processing
conversions applied	Conversions such as unit changes, data-type transformations, or date standardization
missing-data handling	How missing or invalid values were handled

Final state of the dataset

Field	Description
number of rows	Total records in the final CSV
number of columns	Total columns in the final CSV
final column list	Columns present in the final file
period covered	Time span covered by the dataset after processing
entity type represented	Type of entity in the dataset (municipality, state, or country)
percentage of rows removed	Percentage of records removed during processing

Notes

Field	Description
additional notes	Free field for relevant information that doesn’t fit the fields above

Deliver the data

The final step is to submit the data. Upload a compressed file named <source>_<responsible>.zip containing:

a directory named after the source, holding the original data as extracted, with no changes or treatment;
a PDF describing the original data (from Document your source);
a directory named result, holding the JSON, CSV, and MCF files produced after processing;
a PDF describing the transformations applied to the data.

The archive should have a structure similar to this:

fonte_nome_do_responsavel.zip
├── source/
│   └── nome_da_fonte/
│       └── dados_originais.ext
├── result/
│   ├── dataset_final.csv
│   ├── variables.mcf
│   ├── config.json
│   └── outros_arquivos_necessarios.ext
├── descricao_dados_originais.pdf
└── descricao_transformacoes.pdf