Skip to main content
  • BigPanda Documentation
  • Integrations
  • BigPanda Unified Data Connector

BigPanda Unified Data Connector

The BigPanda Unified Data Connector syncs data from your ITSM platforms to provide context and insight for the AI Incident Assistant, AI Incident Prevention, and AI Detection and Response. Ingested data is securely stored and made available in the IT Knowledge Graph, powering accurate answers, deep analytics, trend analysis, and advanced capabilities.

The Unified Data Connector has a standard connector for many tools, or a custom connection can be created for the tool of your choice.

Data Redaction

You can redact sensitive information ingested via the Unified Data Connector, including PII, PHI, and PCI, giving you confidence that your confidential data remains exclusively within your approved channels. See the Data Redaction documentation for details on managing sensitive data.

Streaming Modes

You can set up both historical and incremental streaming modes for each data connector. Most organizations will need to use both modes.

Historical mode

Historical modes sync data one time from a selected start date and after. This works well for one-time data loads or when you need to backfill data that was previously missed. Historical mode is the default, recommended method of initially syncing data. 

Historical mode

Historical streams can take minutes or days, depending on the amount of data sent.

Re-pulling historical data

If you need to re-ingest a specific historical window, such as after a ServiceNow data correction, a major migration, or a gap caused by an upstream outage, your BigPanda account team can trigger a targeted date-range backfill against an existing pipeline without rebuilding the connector.

Backfills run in chunks (typically per day) and execute in parallel with your incremental sync without disrupting it. Large ranges can be scheduled to run only during off-peak windows in your time zone.

To request a date-range backfill, reach out to your account team with:

  • The connector and pipeline name

  • The exact start and end date of the window to re-pull

  • Any preferred execution window (for example, midnight–6 AM EST)

While the backfill is running, you can view per-table row counts and chunk completion status in the BigPanda support response or in the BigPanda monitoring views shared by your account team

Incremental mode

Incremental modes run periodically and sync all new or updated records. This mode works well for scheduled, recurring data loads. Depending on what functionality you are using within BigPanda, your incremental sync time period may vary.

No historical data

Incremental mode runs periodically and syncs only new or updated records since the previous successful run. Once an incremental pipeline is started, its cursor moves forward from that point and does not look back to the configured Start Date. To ingest historical records, run a one-time historical sync (or request a date-range backfill) on the same source. Most production deployments use both: one historical sync to seed the dataset, then incremental mode for ongoing updates.

Frequently Asked Questions

General

How does the Unified Data Connector protect sensitive data and PII?

You can create filters for your tables to ensure that sensitive data is not included in the sync. Additionally, the Data Redaction feature can automatically scan and remove PII, including an advanced optional redaction stream for context-aware removal of PII and PHI.

How long does the initial sync take?

It depends on the rate limit in place, the amount of data synced, and the sync start date. The initial sync can take anywhere between a few hours to multiple weeks.

How often does the data sync?

The default sync frequency is set to 2 minutes, but it can be modified. You can set the frequency from 1 minute to 24 hours.

ServiceNow

How do you ensure that APIs don’t bypass manual review or approval processes when writing back into ServiceNow and that audit trails are preserved?

The Unified Data Connector does not write anything to ServiceNow. It only uses GET APIs to pull data. Just like other user activity, you can track the user activity of the Unified Data Connector user in the ServiceNow system logs.

What controls are in place to ensure that ServiceNow performance is not affected by large data pulls or frequent polling?

The Unified Data connector has the following features in place to control and limit the number of calls and load on the server:

  • Batch size control: Larger batch sizes result in fewer overall calls to the server.

  • Max calls per minute: This setting controls the maximum number of calls sent to ServiceNow. After it reaches the maximum, it pauses for the configured amount of time.

  • Incremental loads: Every incremental load starts from the state that the last load had completed.

Will customizations break if ServiceNow upgrades or changes its APIs?

Only a small number of unique calls are used for the sync. Calls made to ServiceNow are core REST APIs that don’t change often, such as now/table/{table_name}. If ServiceNow makes a change, the fixes to the Unified Data Connector will be minor and happen quickly.

Can UDC support dot walking reference tables (for example: assigned_to.email, assigned_to.name)?

Yes. The ServiceNow connector treats field names as a string, even if a dot is included in the string.

Troubleshoot connection issues

401 Unauthorized when testing or running the connector

  1. Confirm the service account is still active and has not been rotated, locked, or had its password reset since the connector was configured.

  2. Re-test using the credentials directly against your source's REST API (for example,GET /api/now/table/incident?sysparm_limit=1for ServiceNow). If that call also returns 401, the issue is on the source side, not the connector.

  3. Special characters in secrets — particularly backticks (`), ampersands (&), and other shell-meaningful characters — can be URL-encoded inconsistently by some clients. If your password or client secret contains these, regenerate it without them and re-submit.

  4. For OAuth, double-check bothclient_idandclient_secret, and confirm the OAuth token endpoint URL matches your instance.

  5. If you connect through a reverse proxy with a static-IP requirement, confirm the proxy IP has not changed since the connector was set up.

Connector authenticates but returns zero records

A 0-record result usually means authentication succeeded but the service account lacks read access to the target table. Verify with:

  • ServiceNow:GET /api/now/table/<table>?sysparm_limit=1executed as the BigPanda service account. If this returns rows in ServiceNow Studio but zero through the API, the account is missing table-level read ACLs.

  • Jira / JSM / Confluence:confirm the service account has read permission on every project, board, or space referenced by the pipeline configuration (JQL,space_keys, etc.).

Grant table-, project-, or space-level read access to the BigPanda service account and re-test the connection.

Drop or gap in record volume

If a Confluence sync returns far fewer records than expected, verify that every space you intend to ingest is listed in Space keys and that the Confluence service account has read access to each of those spaces. The connector only ingests spaces that are both listed in the configuration and accessible to the service account.

For other connector types, open a support ticket including the affected connector, the date and time the change was first observed, and any source-side changes deployed in the same window (firewall, IP allow-list, ACL, password rotation, ServiceNow upgrade). Most volume regressions trace back to a source-side change of this kind.

Intermittent 502 Bad Gateway responses on specific dates

If a historical sync over a specific date range consistently returns 502 responses, the most common cause is an individual record exceeding the API gateway's response-size limit (typically because a free-text field such as work_notes contains a very large attachment or pasted log). Two mitigations:

  • Reduce the per-call batch size by lowering sysparm_limit (the Batch size setting). Smaller batches are less likely to combine into an oversized response payload.

  • At the ServiceNow side, restrict the BigPanda service account's read ACL on the offending free-text column, or omit that column from the connector's Fields list.

If 502s persist after both mitigations, contact BigPanda support with a sample affected date so we can investigate at the connector level.

In this section: