Skip to content

fix(selectors): fetch all pages for paginated dropdown list routes#4823

Merged
waleedlatif1 merged 5 commits into
stagingfrom
waleedlatif1/validate-dropdown-fetch-changes
May 31, 2026
Merged

fix(selectors): fetch all pages for paginated dropdown list routes#4823
waleedlatif1 merged 5 commits into
stagingfrom
waleedlatif1/validate-dropdown-fetch-changes

Conversation

@waleedlatif1
Copy link
Copy Markdown
Collaborator

Summary

  • Dropdown selectors (Slack channels was already fixed; this extends the same pattern) only fetched the first page of paginated provider APIs, silently hiding results past page one
  • Added bounded server-side pagination draining to the list routes across Microsoft Graph, Google, Notion, Atlassian (Jira/JSM), Linear, AWS CloudWatch, and offset/token REST APIs (Airtable, Asana, Pipedrive, Webflow, Wealthbox, Monday, Zoom) + slack.users
  • Added a shared client-side drain cap + truncated signal in the selector hook, and removed a dead duplicate pagination path in the Confluence selector
  • Each route was implemented and then independently re-validated against the providers' live API docs (mechanism, termination, page-size limits, error handling)
  • Bonus: fixed the Word file picker, which was searching for .xlsx files (the shared Microsoft files route couldn't distinguish Word from Excel)

What does NOT change

  • Tool/block execution and stored selection IDs are unchanged — only the dropdown option lists are now complete
  • Response JSON shapes are identical; contract additions (fileType, offset) are optional and backwards-compatible
  • Typical small accounts still resolve in a single request (no added latency)
  • The two CloudWatch list routes are shared with their tools; a caller-supplied limit is still honored as a hard cap

Type of Change

  • Bug fix

Testing

Tested manually. Each pagination mechanism was validated against the providers' live API docs by independent review.

Checklist

  • Code follows project style guidelines
  • Self-reviewed my changes
  • Tests added/updated and passing
  • No new warnings introduced
  • I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

Dropdown selectors fetched only the first page of paginated provider
APIs, silently hiding results past page one. Add bounded server-side
draining to the list routes across Microsoft Graph, Google, Notion,
Atlassian, Linear, AWS CloudWatch, and offset/token REST APIs, plus a
shared client-side drain cap in the selector hook. Response shapes,
stored values, and tool execution are unchanged; CloudWatch list tools
still honor a caller-supplied limit. Also fixes the Word file picker
that was searching for .xlsx files.
@vercel
Copy link
Copy Markdown

vercel Bot commented May 31, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated (UTC)
docs Skipped Skipped May 31, 2026 3:47am

Request Review

@cursor
Copy link
Copy Markdown

cursor Bot commented May 31, 2026

PR Summary

Medium Risk
Touches many OAuth-backed list routes (more upstream calls and latency for large accounts) but keeps auth paths and response contracts unchanged; incomplete lists only when page caps are hit, with warnings logged.

Overview
Fixes workflow dropdown selectors that only loaded the first page of provider list APIs, so options beyond page one were missing from the UI.

Server routes now drain pagination with per-integration caps and warn when a cap is hit (Microsoft Graph via @odata.nextLink + assertGraphNextPageUrl, Google via new drainGooglePagedList, AWS CloudWatch nextToken, and offset/cursor styles for Airtable, Asana, Jira/JSM, Linear, Notion, Slack users, Zoom, Webflow, etc.). Response shapes stay the same; optional query fields like fileType and knowledge offset are backward-compatible.

Client: loadAllSelectorOptions drains fetchPage for search/replace and value resolution; useSelectorOptions caps auto-drain at 50 pages and exposes truncated. Confluence/knowledge use fetchPage only (removed duplicate fetchList loops). Word file picker now passes fileType=word on the shared Microsoft files route (was incorrectly searching .xlsx).

Reviewed by Cursor Bugbot for commit abdaf2c. Configure here.

Comment thread apps/sim/app/api/tools/jsm/selector-requesttypes/route.ts Outdated
Comment thread apps/sim/app/api/tools/monday/boards/route.ts
- JSM service-desk/request-type drains advance `start` by the actual row
  count returned (not the fixed page size) and stop on an empty page, so a
  short non-final page can't skip items.
- Monday boards drain now checks `response.ok` per page, surfacing a
  mid-drain HTTP failure instead of treating it as an empty final page and
  returning a partial 200.
@greptile-apps
Copy link
Copy Markdown
Contributor

greptile-apps Bot commented May 31, 2026

Greptile Summary

This PR fixes a widespread first-page-only bug in dropdown selector routes by adding bounded server-side pagination draining across ~30 API routes (Microsoft Graph, Google APIs, Notion, Jira/JSM, Linear, Slack, Airtable, Asana, Zoom, Webflow, Wealthbox, Monday, Pipedrive, and more). It also introduces a shared drainGooglePagedList utility, adds a MAX_AUTO_DRAIN_PAGES cap and truncated signal to the selector hook, converts the Confluence and Knowledge selectors from fetchList to the progressive fetchPage path, and fixes the Word file picker which was incorrectly searching for .xlsx files.

  • Pagination draining is added to all affected routes using consistent bounded-loop patterns with logger.warn at the cap; every provider uses the correct continuation mechanism (nextToken, nextLink, cursor, offset, page-number).
  • Shared utilities (drainGooglePagedList, loadAllSelectorOptions, assertGraphNextPageUrl) centralise repeated logic across Google and Microsoft Graph callers.
  • Client-side guard: use-selector-query.ts now stops auto-draining at 50 pages and surfaces a truncated flag; resolve-values.ts and workflow-search-replace.ts are updated to use loadAllSelectorOptions so paginated selectors work outside the React Query hook.

Confidence Score: 5/5

Safe to merge. All pagination loops are bounded, warnings are emitted at caps, and the response shapes are backward-compatible.

The change is additive: each route gets a bounded drain loop, shared utilities consolidate repeated patterns, and the client hook gains a truncated guard. The Word/Excel file-type fix is isolated to a single route with a new optional query param. No execution paths, stored IDs, or response contracts are broken.

No files require special attention; the one suggestion in registry.ts is a quality improvement rather than a correctness issue.

Important Files Changed

Filename Overview
apps/sim/lib/oauth/google-pagination.ts New shared utility for draining token-paginated Google REST endpoints; correctly bounded, emits warnings at cap, throws typed GooglePageError for callers.
apps/sim/hooks/selectors/use-selector-query.ts Adds MAX_AUTO_DRAIN_PAGES cap to the auto-drain loop and exposes a truncated signal; dependency array updated correctly.
apps/sim/hooks/selectors/registry.ts Adds loadAllSelectorOptions helper that drains either fetchList or fetchPage, with a MAX_LOAD_ALL_PAGES cap for the paged path.
apps/sim/hooks/selectors/types.ts Makes fetchList optional in SelectorDefinition so selectors can provide only fetchPage; well-documented with a clear JSDoc on mutual exclusivity.
apps/sim/app/api/tools/jira/projects/route.ts Adds full offset-based pagination drain via fetchAllJiraProjects; lastResponse relies on a ! non-null assertion but is safe at runtime.
apps/sim/app/api/auth/oauth/microsoft/files/route.ts Fixes Word file picker (was searching .xlsx), adds fileType param, and drains Graph pagination; clean refactor with correct MIME type filtering.
apps/sim/app/api/tools/cloudwatch/utils.ts Refactors describeLogStreams to drain all pages; respects totalLimit cap correctly and warns at MAX_LOG_STREAMS_PAGES.
apps/sim/hooks/selectors/providers/confluence/selectors.ts Removes the duplicate fetchList (which internally drained pagination) in favour of the fetchPage path; extracts toSpaceOption helper to reduce duplication.
apps/sim/hooks/selectors/providers/knowledge/selectors.ts Migrates from fetchList to fetchPage with offset-based cursor, wiring pagination correctly through pagination.hasMore and pagination.offset + pagination.limit.
apps/sim/app/api/tools/notion/databases/route.ts Adds bounded cursor-based drain using has_more/next_cursor; capped at 20 pages (2000 items). Uses getErrorMessage in the catch block.
apps/sim/lib/workflows/comparison/resolve-values.ts Switches to loadAllSelectorOptions so paginated selectors (Confluence, Knowledge) are fully drained during value resolution.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Selector Dropdown] --> B{fetchPage defined?}
    B -- yes --> C[useSelectorOptions fetchPage path]
    B -- no --> D[useSelectorOptions fetchList path]
    C --> E{pageCount >= 50?}
    E -- no --> F[fetchNextPage auto-drain]
    F --> G{hasNextPage?}
    G -- yes --> F
    G -- no --> H[Full option list]
    E -- yes --> I[Stop + truncated=true + logger.warn]
    D --> J[Single fetchList call]
    J --> H
    H --> K[Render Dropdown]
    L[resolve-values / search-replace loadAllSelectorOptions] --> M{fetchList defined?}
    M -- yes --> N[Call fetchList once]
    M -- no --> O{fetchPage defined?}
    O -- yes --> P[Drain fetchPage up to 50 pages]
    O -- no --> Q[Return empty array]
    N --> R[All options]
    P --> R
Loading

Reviews (2): Last reviewed commit: "fix(selectors): drain fetchPage in direc..." | Re-trigger Greptile

Comment thread apps/sim/app/api/tools/jsm/selector-requesttypes/route.ts
Comment thread apps/sim/app/api/tools/jsm/selector-servicedesks/route.ts
The offset-advancement fix (advance `start` by the rows returned, not the
fixed page size) landed in 7b19788; update the TSDoc to match so it no
longer reads as advancing by `limit`.
Making `fetchList` optional left three direct callers (outside the
useSelectorOptions hook) calling it unguarded, which broke the build's
type check. Route them through a shared `loadAllSelectorOptions` helper
that uses `fetchList` when present and otherwise drains `fetchPage`.
This also prevents a regression: `confluence.spaces` / `knowledge.documents`
now paginate via `fetchPage` only, and these callers (search/replace,
value resolution) would otherwise have silently returned no options.
@waleedlatif1
Copy link
Copy Markdown
Collaborator Author

@greptile

@waleedlatif1
Copy link
Copy Markdown
Collaborator Author

@cursor review

Copy link
Copy Markdown

@cursor cursor Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit abdaf2c. Configure here.

@waleedlatif1 waleedlatif1 merged commit 5fa8416 into staging May 31, 2026
13 checks passed
@waleedlatif1 waleedlatif1 deleted the waleedlatif1/validate-dropdown-fetch-changes branch May 31, 2026 03:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant