# indicators The `indicators` command calculates procurement indicators and red flags. Run the `help` command to read its description, output format and options: ```console $ ocdscardinal help indicators Calculate procurement indicators from OCDS compiled releases in a line-delimited JSON file The result is a JSON object, in which the keys are one of `OCID`, `Buyer`, `ProcuringEntity` or `Tenderer`. The values are JSON objects, in which the keys are identifiers (e.g. ocid) and values are results (of any indicators that returned a result). Unless --no-meta is set, the result has a "Meta" key, with information about the quartiles and fences used to calculate the results. If --map is set, the result has a "Maps" key, with mappings from contracting processes to organizations. Usage: ocdscardinal[EXE] indicators [OPTIONS] Arguments: The path to the file (or "-" for standard input), in which each line is a contracting process as JSON text Options: -s, --settings The path to the settings file -v, --verbose... Increase verbosity -c, --count Print the number of results per group to standard error --no-meta Exclude the "Meta" key from the results object --map Include the "Maps" key, mapping contracting processes to organizations -h, --help Print help (see a summary with '-h') ``` (indicators-workflow)= ## Workflow :::{attention} Before following this command's workflow, follow the earlier steps in the {doc}`../../topics/workflow`. ::: 1. **Select indicators**. If you ran the {doc}`../init` command when preparing your data, you already have a {doc}`../../topics/settings` that enables all indicators. [Enable](#enable-an-indicator) or [disable](#disable-an-indicator) indicators as you wish. 1. **Run the command**. For example, if your settings are in `settings.ini` and your data is in `prepared.jsonl`, this command writes the output to `results.json`: ```bash ocdscardinal indicators --settings settings.ini prepared.jsonl > results.json ``` 1. **Review the results**. Read the [demonstration](#demonstration) to learn about the output format. :::{admonition} To do :class: caution This section will expand on the interpretation of results. See GitHub issue [#40](https://github.com/open-contracting/cardinal-rs/issues/40). ::: 1. **Edit the settings**. Adjust the configuration of the [indicators](#list) based on your review of the results, in order to reduce false positives. Repeat the last three steps until you are satisfied with the results. :::{note} Have questions, concerns, or feedback? [Email James McKinney](mailto:jmckinney@open-contracting.org), OCP's Head of Technology. ::: (indicators-demo)= ## Demonstration A procurement indicator or red flag can be about a contracting process, buyer, procuring entity or tenderer. For example, a contracting process might have a suspicious feature, like all bids except the winner's being disqualified. Or, a buyer might exhibit suspicious behavior, like disqualifying a large number of bids across its contracting processes. The JSON output is therefore organized as an object in which the key is a **group**: "OCID" (the unique identifier of a contracting process), "Buyer", "ProcuringEntity" or "Tenderer". For example: ```json { "OCID": {}, "Buyer": {}, "ProcuringEntity": {}, "Tenderer": {} } ``` Each value at the *top* level is an object representing the results within that **group**, in which the key is an **identifier** extracted from the input data: | Group | Identifier | | - | - | | OCID | `/ocid` | | Buyer | `/buyer/id` | | ProcuringEntity | `/tender/procuringEntity/id` | | Tenderer | `/bids/details[]/tenderers[]/id` | For example: ```json { "OCID": { "ocds-6550wx-JRFPFA-DAF-CM-2021-0012": {} }, "Buyer": { "DO-RPE-55216": {} } } ``` Each value at the *second* level is an object representing the results relating to that **identifier**, in which the key is the **code** for an indicator, and the value is the **output** of that indicator. For example: ```json { "OCID": { "ocds-6550wx-JRFPFA-DAF-CM-2021-0012": { "R036": 1.0 } }, "Buyer": { "DO-RPE-55216": { "R038": 0.8 } } } ``` The **output** of an indicator is always a decimal. If an indicator didn't produce an output – either because it couldn't be calculated, or because no red flag was raised – then its code won't appear. You can [consult](#list) the codes for all indicators, read the description of their outputs and see a demonstration of their calculation. The JSON output also has a ``Meta`` key at the top level. Its value is an object with information about the quartiles and fences used to calculate the results, rounded to 4 decimals. For example: ```json { // ... "Meta": { "R024": { "q1": 66.6667, "q3": 100.0, "lower_fence": 16.6667 } } } ``` (indicators-config)= ## Configuration The page for each [indicator](#list) describes its individual settings. All configuration is optional. Cardinal provides good defaults. :::{seealso} An introduction to the {doc}`../../topics/settings`. ::: (enable-an-indicator)= ### Enable an indicator To enable an indicator, start a section with its code, for example: ```ini [R024] ``` You don't need to set properties in this section. (Only if you want to!) ### Disable an indicator The disable an indicator, either delete its section and properties, or comment them out, for example: ```ini ; [R024] ; threshold = 0.05 ``` Now, the `indicators` command won't run this indicator. (global-configuration)= ### Global configuration You might want to exclude some procedures from all indicators, whether for methodological reasons or due to quality issues. To exclude procedures based on the value of `/tender/procurementMethodDetails`, add to your settings file, as a pipe-separated list, for example: ```ini [exclusions] procurement_method_details = Random Selection|Sorteo de Obras ``` Some indicators compares bid prices. However, for some procedures, it is inappropriate to compare bid prices; for example, if the buyer predetermines the price, then the indicator for prices close to the winning bid will return a false positive. Procedures can be included or excluded based on the value of `/tender/procurementMethodDetails`. To exclude procedures from price comparisons, add to the top of your settings file, as a pipe-separated list, for example: ```ini no_price_comparison_procurement_methods = Random Selection|Sorteo de Obras ``` Alternatively, to include procedures in price comparisons, add to the top of your settings file, as a pipe-separated list, for example: ```ini price_comparison_procurement_methods = Reverse Auction ``` A procedure is excluded if either: - `no_price_comparison_procurement_methods` is set, and `/tender/procurementMethodDetails` is set and matches - `price_comparison_procurement_methods` is set, and `/tender/procurementMethodDetails` isn't set or doesn't match ## Glossary % Do not add terms to the glossary that are not used in the documentation! :::{glossary} bid An offer made by an {term}`economic operator` as part of a {term}`contracting process`. Also known as a *tender*. bidder Synonym of {term}`tenderer`. buyer The organization aiming to conclude a contract with an {term}`economic operator` or to use the goods, services or works resulting from the contract, as part of a {term}`contracting process`. contracting process All the actions aimed at implementing one or more contracts. This covers tendering, awarding, contracting and implementation. Also known as a *procedure*. economic operator A person or organization – or group of people or organizations – that offers goods, services or works. Also known as a *business*, *contractor*, *service provider*, *supplier* or *undertaking*. procuring entity The organization managing the {term}`contracting process`. An organization can be both a {term}`buyer` and a procuring entity (like in a simple contracting process). submitted A {term}`bid` is submitted if its status is pending (i.e. not evaluated yet), valid (i.e. qualified), or disqualified. It is not submitted if its status is invited or withdrawn. tenderer An {term}`economic operator` that {term}`submitted` one or more {term}`bids` as part of a {term}`contracting process`. winner A {term}`tenderer` that is awarded a contract. ::: (list)= ## Indicators Indicators are assigned codes for easy reference: for example, `R001`. The first letter indicates the category: **R**ed flag or **U**se case. The page for each indicator describes its individual methodology. For all indicators, a contracting process is excluded if: - The `ocid` isn't a string. - The contracting process (`/tender/status`) is cancelled. - The relevant organization references don't set an `id`. - Monetary values, where relevant, don't use the main currency. To configure the main currency, add to the top of your settings file: ```ini currency = USD ``` Otherwise, the main currency is set to the first observed currency. :::{note} Do you want to eliminate this exclusion? Please contributed to [GitHub issue #11](https://github.com/open-contracting/cardinal-rs/issues/11). ::: :::{toctree} :hidden: true R/index ::: ### Red flags :::{list-table} :header-rows: 1 * - Code - Title - Description * - [R003](R/003) - [Short submission period](R/003) - The submission period is too short. * - [R018](R/018) - [Single bid received](R/018) - Only one tenderer submitted a bid. * - [R024](R/024) - [Price close to winning bid](R/024) - The percentage difference between the winning bid and the second-lowest valid bid is a low outlier. * - [R025](R/025) - [Excessive unsuccessful bids](R/025) - The ratio of winning bids to submitted bids for a top tenderer is a low outlier. * - [R028](R/028) - [Identical bid prices](R/028) - Different tenderers submitted bids with the same price. * - [R030](R/030) - [Late bid won](R/030) - The winning bid was received after the submission deadline. * - [R035](R/035) - [All except winning bid disqualified](R/035) - Bids are disqualified if not submitted by the single tenderer of the winning bid. * - [R036](R/036) - [Lowest bid disqualified](R/036) - The lowest submitted bid is disqualified, while the award criterion is price only. * - [R038](R/038) - [Excessive disqualified bids](R/038) - The ratio of disqualified bids to submitted bids is a high outlier per buyer, procuring entity or tenderer. * - [R048](R/048) - [Heterogeneous supplier](R/048) - The variety of items supplied by a tenderer is a high outlier. * - [R058](R/058) - [Heavily discounted bid](R/058) - The percentage difference between the winning bid and the second-lowest valid bid is a high outlier. :::