indicators

The indicators command calculates procurement indicators and red flags.

Run the help command to read its description, output format and options:

$ ocdscardinal help indicators
Calculate procurement indicators from OCDS compiled releases in a line-delimited JSON file

The result is a JSON object, in which the keys are one of `OCID`, `Buyer`, `ProcuringEntity` or
`Tenderer`. The values are JSON objects, in which the keys are identifiers (e.g. ocid) and values
are results (of any indicators that returned a result).

Unless --no-meta is set, the result has a "Meta" key, with information about the quartiles and
fences used to calculate the results.

If --map is set, the result has a "Maps" key, with mappings from contracting processes to
organizations.

Usage: ocdscardinal[EXE] indicators [OPTIONS] <FILE>

Arguments:
  <FILE>
          The path to the file (or "-" for standard input), in which each line is a contracting
          process as JSON text

Options:
  -s, --settings <SETTINGS>
          The path to the settings file

  -v, --verbose...
          Increase verbosity

  -c, --count
          Print the number of results per group to standard error

      --no-meta
          Exclude the "Meta" key from the results object

      --map
          Include the "Maps" key, mapping contracting processes to organizations

  -h, --help
          Print help (see a summary with '-h')

Workflow

Attention

Before following this command’s workflow, follow the earlier steps in the Overall workflow.

  1. Select indicators. If you ran the init command when preparing your data, you already have a Settings file that enables all indicators. Enable or disable indicators as you wish.

  2. Run the command. For example, if your settings are in settings.ini and your data is in prepared.jsonl, this command writes the output to results.json:

    ocdscardinal indicators --settings settings.ini prepared.jsonl > results.json
    
  3. Review the results. Read the demonstration to learn about the output format.

    To do

    This section will expand on the interpretation of results. See GitHub issue #40.

  4. Edit the settings. Adjust the configuration of the indicators based on your review of the results, in order to reduce false positives.

Repeat the last three steps until you are satisfied with the results.

Note

Have questions, concerns, or feedback? Email James McKinney, OCP’s Head of Technology.

Demonstration

A procurement indicator or red flag can be about a contracting process, buyer, procuring entity or tenderer. For example, a contracting process might have a suspicious feature, like all bids except the winner’s being disqualified. Or, a buyer might exhibit suspicious behavior, like disqualifying a large number of bids across its contracting processes.

The JSON output is therefore organized as an object in which the key is a group: “OCID” (the unique identifier of a contracting process), “Buyer”, “ProcuringEntity” or “Tenderer”. For example:

{
  "OCID": {},
  "Buyer": {},
  "ProcuringEntity": {},
  "Tenderer": {}
}

Each value at the top level is an object representing the results within that group, in which the key is an identifier extracted from the input data:

Group

Identifier

OCID

/ocid

Buyer

/buyer/id

ProcuringEntity

/tender/procuringEntity/id

Tenderer

/bids/details[]/tenderers[]/id

For example:

{
  "OCID": {
    "ocds-6550wx-JRFPFA-DAF-CM-2021-0012": {}
  },
  "Buyer": {
    "DO-RPE-55216": {}
  }
}

Each value at the second level is an object representing the results relating to that identifier, in which the key is the code for an indicator, and the value is the output of that indicator. For example:

{
  "OCID": {
    "ocds-6550wx-JRFPFA-DAF-CM-2021-0012": {
      "R036": 1.0
    }
  },
  "Buyer": {
    "DO-RPE-55216": {
      "R038": 0.8
    }
  }
}

The output of an indicator is always a decimal. If an indicator didn’t produce an output – either because it couldn’t be calculated, or because no red flag was raised – then its code won’t appear.

You can consult the codes for all indicators, read the description of their outputs and see a demonstration of their calculation.

The JSON output also has a Meta key at the top level. Its value is an object with information about the quartiles and fences used to calculate the results, rounded to 4 decimals. For example:

{
  // ...
  "Meta": {
    "R024": {
      "q1": 66.6667,
      "q3": 100.0,
      "lower_fence": 16.6667
    }
  }
}

Configuration

The page for each indicator describes its individual settings.

All configuration is optional. Cardinal provides good defaults.

See also

An introduction to the Settings file.

Enable an indicator

To enable an indicator, start a section with its code, for example:

[R024]

You don’t need to set properties in this section. (Only if you want to!)

Disable an indicator

The disable an indicator, either delete its section and properties, or comment them out, for example:

; [R024]
; threshold = 0.05

Now, the indicators command won’t run this indicator.

Global configuration

You might want to exclude some procedures from all indicators, whether for methodological reasons or due to quality issues. To exclude procedures based on the value of /tender/procurementMethodDetails, add to your settings file, as a pipe-separated list, for example:

[exclusions]
procurement_method_details = Random Selection|Sorteo de Obras

Some indicators compares bid prices. However, for some procedures, it is inappropriate to compare bid prices; for example, if the buyer predetermines the price, then the indicator for prices close to the winning bid will return a false positive.

Procedures can be included or excluded based on the value of /tender/procurementMethodDetails.

To exclude procedures from price comparisons, add to the top of your settings file, as a pipe-separated list, for example:

no_price_comparison_procurement_methods = Random Selection|Sorteo de Obras

Alternatively, to include procedures in price comparisons, add to the top of your settings file, as a pipe-separated list, for example:

price_comparison_procurement_methods = Reverse Auction

A procedure is excluded if either:

  • no_price_comparison_procurement_methods is set, and /tender/procurementMethodDetails is set and matches

  • price_comparison_procurement_methods is set, and /tender/procurementMethodDetails isn’t set or doesn’t match

Glossary

bid

An offer made by an economic operator as part of a contracting process. Also known as a tender.

bidder

Synonym of tenderer.

buyer

The organization aiming to conclude a contract with an economic operator or to use the goods, services or works resulting from the contract, as part of a contracting process.

contracting process

All the actions aimed at implementing one or more contracts. This covers tendering, awarding, contracting and implementation. Also known as a procedure.

economic operator

A person or organization – or group of people or organizations – that offers goods, services or works. Also known as a business, contractor, service provider, supplier or undertaking.

procuring entity

The organization managing the contracting process. An organization can be both a buyer and a procuring entity (like in a simple contracting process).

submitted

A bid is submitted if its status is pending (i.e. not evaluated yet), valid (i.e. qualified), or disqualified. It is not submitted if its status is invited or withdrawn.

tenderer

An economic operator that submitted one or more bids as part of a contracting process.

winner

A tenderer that is awarded a contract.

Indicators

Indicators are assigned codes for easy reference: for example, R001. The first letter indicates the category: Red flag or Use case.

The page for each indicator describes its individual methodology. For all indicators, a contracting process is excluded if:

  • The ocid isn’t a string.

  • The contracting process (/tender/status) is cancelled.

  • The relevant organization references don’t set an id.

  • Monetary values, where relevant, don’t use the main currency.

    To configure the main currency, add to the top of your settings file:

    currency = USD
    

    Otherwise, the main currency is set to the first observed currency.

    Note

    Do you want to eliminate this exclusion? Please contributed to GitHub issue #11.

Red flags

Code

Title

Description

R003

Short submission period

The submission period is too short.

R018

Single bid received

Only one tenderer submitted a bid.

R024

Price close to winning bid

The percentage difference between the winning bid and the second-lowest valid bid is a low outlier.

R025

Excessive unsuccessful bids

The ratio of winning bids to submitted bids for a top tenderer is a low outlier.

R028

Identical bid prices

Different tenderers submitted bids with the same price.

R030

Late bid won

The winning bid was received after the submission deadline.

R035

All except winning bid disqualified

Bids are disqualified if not submitted by the single tenderer of the winning bid.

R036

Lowest bid disqualified

The lowest submitted bid is disqualified, while the award criterion is price only.

R038

Excessive disqualified bids

The ratio of disqualified bids to submitted bids is a high outlier per buyer, procuring entity or tenderer.

R048

Heterogeneous supplier

The variety of items supplied by a tenderer is a high outlier.

R058

Heavily discounted bid

The percentage difference between the winning bid and the second-lowest valid bid is a high outlier.