Skip to main content

NIM

Filter Statistics

Filters are a core foundation of how processing works in NIM, serving an essential component that manages and manipulate data streams. In NIM, filters are used to apply transformations, extract specific data, and perform various operations that are crucial for the efficient functioning of the system. Understanding the duration it takes for a filter to process data is vital for optimizing performance and ensuring smooth data flow. This knowledge allows developers to identify potential bottlenecks, balance workloads, and enhance the overall responsiveness of the system. By carefully monitoring and adjusting filter processing times, NIM can maintain high efficiency and reliability, crucial for handling complex data processing tasks.

Every time a filter executes, a log message is generated. All times listed on the log entry in milliseconds (ms). Below is an example

2022-08-08T12:25:44.093Z verbose: Filter stats fManagerEmployees (ms): total:5, query:4, resolve:0, docs:1, sort:0, ordering:0, exclude:0, lookup:0, append:0 #21/0/1 (initialization) {  "fields": [    {      "fieldName": "ID",      "fieldType": "STRING",      "readAccessType": "ADMINS_AND_SELF",      "multiValued": false,      "displayName": "ID"    },    {      "fieldName": "Type",      "fieldType": "STRING",      "readAccessType": "ADMINS_AND_SELF",      "multiValued": false,      "displayName": "Type"    },    {      "fieldName": "DeleteDate",      "fieldType": "STRING",      "readAccessType": "ADMINS_AND_SELF",      "multiValued": false,      "displayName": "DeleteDate"    }  ],  "schemaName": "Tools4ever",  "displayName": "Tools4ever"}
  • total: The total time to execute the filter

  • query: Time to initialize parameters and variables and calculate the raw query result

  • resolve: Time to calculate the result document references based on the raw query result and the dataset

  • docs: Time to compose the resulting documents

  • sort: Time to sort the resulting documents

  • ordering: Time to determine the included and excluded documents

  • exclude: Time to process excluded documents based on specific column

  • lookup: Time to process the lookup specification, if any

  • append: Time to append results of other specified filter.

  • #21/0/1:

    • Number of included documents (21)

    • Number of documents removed by lookup operation (0)

    • Number of documents excluded (1)