indicators¶
The indicators
command calculates procurement indicators and red flags.
Run the help
command to read its description, output format and options:
$ ocdscardinal help indicators
Calculate procurement indicators from OCDS compiled releases in a line-delimited JSON file
The result is a JSON object, in which the keys are one of `OCID`, `Buyer`, `ProcuringEntity` or
`Tenderer`. The values are JSON objects, in which the keys are identifiers (e.g. ocid) and values
are results (of any indicators that returned a result).
Unless --no-meta is set, the result has a "Meta" key, with information about the quartiles and
fences used to calculate the results.
If --map is set, the result has a "Maps" key, with mappings from contracting processes to
organizations.
Usage: ocdscardinal[EXE] indicators [OPTIONS] <FILE>
Arguments:
<FILE>
The path to the file (or "-" for standard input), in which each line is a contracting
process as JSON text
Options:
-s, --settings <SETTINGS>
The path to the settings file
-v, --verbose...
Increase verbosity
-c, --count
Print the number of results per group to standard error
--no-meta
Exclude the "Meta" key from the results object
--map
Include the "Maps" key, mapping contracting processes to organizations
-h, --help
Print help (see a summary with '-h')
Workflow¶
Attention
Before following this command’s workflow, follow the earlier steps in the Overall workflow.
Select indicators. If you ran the init command when preparing your data, you already have a Settings file that enables all indicators. Enable or disable indicators as you wish.
Run the command. For example, if your settings are in
settings.ini
and your data is inprepared.jsonl
, this command writes the output toresults.json
:ocdscardinal indicators --settings settings.ini prepared.jsonl > results.json
Review the results. Read the demonstration to learn about the output format.
To do
This section will expand on the interpretation of results. See GitHub issue #40.
Edit the settings. Adjust the configuration of the indicators based on your review of the results, in order to reduce false positives.
Repeat the last three steps until you are satisfied with the results.
Note
Have questions, concerns, or feedback? Email James McKinney, OCP’s Head of Technology.
Demonstration¶
A procurement indicator or red flag can be about a contracting process, buyer, procuring entity or tenderer. For example, a contracting process might have a suspicious feature, like all bids except the winner’s being disqualified. Or, a buyer might exhibit suspicious behavior, like disqualifying a large number of bids across its contracting processes.
The JSON output is therefore organized as an object in which the key is a group: “OCID” (the unique identifier of a contracting process), “Buyer”, “ProcuringEntity” or “Tenderer”. For example:
{
"OCID": {},
"Buyer": {},
"ProcuringEntity": {},
"Tenderer": {}
}
Each value at the top level is an object representing the results within that group, in which the key is an identifier extracted from the input data:
Group |
Identifier |
---|---|
OCID |
|
Buyer |
|
ProcuringEntity |
|
Tenderer |
|
For example:
{
"OCID": {
"ocds-6550wx-JRFPFA-DAF-CM-2021-0012": {}
},
"Buyer": {
"DO-RPE-55216": {}
}
}
Each value at the second level is an object representing the results relating to that identifier, in which the key is the code for an indicator, and the value is the output of that indicator. For example:
{
"OCID": {
"ocds-6550wx-JRFPFA-DAF-CM-2021-0012": {
"R036": 1.0
}
},
"Buyer": {
"DO-RPE-55216": {
"R038": 0.8
}
}
}
The output of an indicator is always a decimal. If an indicator didn’t produce an output – either because it couldn’t be calculated, or because no red flag was raised – then its code won’t appear.
You can consult the codes for all indicators, read the description of their outputs and see a demonstration of their calculation.
The JSON output also has a Meta
key at the top level. Its value is an object with information about the quartiles and fences used to calculate the results, rounded to 4 decimals. For example:
{
// ...
"Meta": {
"R024": {
"q1": 66.6667,
"q3": 100.0,
"lower_fence": 16.6667
}
}
}
Configuration¶
The page for each indicator describes its individual settings.
All configuration is optional. Cardinal provides good defaults.
See also
An introduction to the Settings file.
Enable an indicator¶
To enable an indicator, start a section with its code, for example:
[R024]
You don’t need to set properties in this section. (Only if you want to!)
Disable an indicator¶
The disable an indicator, either delete its section and properties, or comment them out, for example:
; [R024]
; threshold = 0.05
Now, the indicators
command won’t run this indicator.
Global configuration¶
You might want to exclude some procedures from all indicators, whether for methodological reasons or due to quality issues. To exclude procedures based on the value of /tender/procurementMethodDetails
, add to your settings file, as a pipe-separated list, for example:
[exclusions]
procurement_method_details = Random Selection|Sorteo de Obras
Some indicators compares bid prices. However, for some procedures, it is inappropriate to compare bid prices; for example, if the buyer predetermines the price, then the indicator for prices close to the winning bid will return a false positive.
Procedures can be included or excluded based on the value of /tender/procurementMethodDetails
.
To exclude procedures from price comparisons, add to the top of your settings file, as a pipe-separated list, for example:
no_price_comparison_procurement_methods = Random Selection|Sorteo de Obras
Alternatively, to include procedures in price comparisons, add to the top of your settings file, as a pipe-separated list, for example:
price_comparison_procurement_methods = Reverse Auction
A procedure is excluded if either:
no_price_comparison_procurement_methods
is set, and/tender/procurementMethodDetails
is set and matchesprice_comparison_procurement_methods
is set, and/tender/procurementMethodDetails
isn’t set or doesn’t match
Glossary¶
- bid¶
An offer made by an economic operator as part of a contracting process. Also known as a tender.
- bidder¶
Synonym of tenderer.
- buyer¶
The organization aiming to conclude a contract with an economic operator or to use the goods, services or works resulting from the contract, as part of a contracting process.
- contracting process¶
All the actions aimed at implementing one or more contracts. This covers tendering, awarding, contracting and implementation. Also known as a procedure.
- economic operator¶
A person or organization – or group of people or organizations – that offers goods, services or works. Also known as a business, contractor, service provider, supplier or undertaking.
- procuring entity¶
The organization managing the contracting process. An organization can be both a buyer and a procuring entity (like in a simple contracting process).
- submitted¶
A bid is submitted if its status is pending (i.e. not evaluated yet), valid (i.e. qualified), or disqualified. It is not submitted if its status is invited or withdrawn.
- tenderer¶
An economic operator that submitted one or more bids as part of a contracting process.
- winner¶
A tenderer that is awarded a contract.
Indicators¶
Indicators are assigned codes for easy reference: for example, R001
. The first letter indicates the category: Red flag or Use case.
The page for each indicator describes its individual methodology. For all indicators, a contracting process is excluded if:
The
ocid
isn’t a string.The contracting process (
/tender/status
) is cancelled.The relevant organization references don’t set an
id
.Monetary values, where relevant, don’t use the main currency.
To configure the main currency, add to the top of your settings file:
currency = USD
Otherwise, the main currency is set to the first observed currency.
Note
Do you want to eliminate this exclusion? Please contributed to GitHub issue #11.
Red flags¶
Code |
Title |
Description |
---|---|---|
The submission period is too short. |
||
Only one tenderer submitted a bid. |
||
The percentage difference between the winning bid and the second-lowest valid bid is a low outlier. |
||
The ratio of winning bids to submitted bids for a top tenderer is a low outlier. |
||
Different tenderers submitted bids with the same price. |
||
The winning bid was received after the submission deadline. |
||
Bids are disqualified if not submitted by the single tenderer of the winning bid. |
||
The lowest submitted bid is disqualified, while the award criterion is price only. |
||
The ratio of disqualified bids to submitted bids is a high outlier per buyer, procuring entity or tenderer. |
||
The variety of items supplied by a tenderer is a high outlier. |
||
The percentage difference between the winning bid and the second-lowest valid bid is a high outlier. |