Co-authored-by: James George <25279263+jamesgeorge007@users.noreply.github.com>
Co-authored-by: Nahid Hasan <52489202+nahidhasan94@users.noreply.github.com>
Refined network retry logic to distinguish between transient infrastructure
failures and intentional test errors, preventing incorrect test skips in
JUnit validation scenarios.
1. Network Error Detection (utils.ts)
- Renamed `hasNetworkError` → `hasLowLevelNetworkError` for clarity
- Removed REQUEST_ERROR from retry patterns (too generic, matches intentional bad URLs)
- Now only retries on unambiguous TCP/DNS errors: ECONNRESET, EAI_AGAIN,
ENOTFOUND, ETIMEDOUT, ECONNREFUSED
- Preserved TEST_SCRIPT_ERROR detection when concurrent with REQUEST_ERROR
(the actual CI failure mode from undefined response objects)
- Added comprehensive JSDoc explaining when to use vs plain runCLI
2. JUnit XML Validation (test.spec.ts, 4 locations)
- Removed REQUEST_ERROR and TEST_SCRIPT_ERROR from XML retry patterns
- Only retry when low-level errors corrupt XML structure
- Prevents skipping tests with intentional errors in collections
(test-junit-report-export-coll.json has intentional invalid-url and
script reference errors for validation)
3. Test Corrections
- Fixed: "Fails to display console logs..." test now uses plain runCLI
(test expects errors from legacy sandbox, shouldn't use retry)
- Added: Environment version tests (v0, v1, v2) now use runCLIWithNetworkRetry
(use echo.hoppscotch.io, expect success, benefit from retry)
- Removed: Obsolete SKIP_EXTERNAL_TESTS env var check (retry logic handles this)
Add comprehensive test coverage for unsupported Postman APIs and ensure
consistent error messages across pre-request and post-request contexts.
Test improvements:
- Expand coverage from 13 to 25 unsupported APIs (50 tests total)
- Add missing APIs: collectionVariables.set/unset/has/clear/toObject,
vault.set/unset, iterationData.set/unset/has/toJSON
- Fix assertions to match actual error format with prefix
- Add pre-request context test for pm.execution.location
Implementation fixes:
- Add missing pm.iterationData.toJSON() in pre-request.js
- Sync post-request.js collectionVariables error messages to match
pre-request.js ("use environment or request variables instead")
- CLI Collection runner enhancements:
- Support mixed versions - child collections based on different versions compared to the parent collection.
- Better support for incoming data via IDs conforming to older formats.
- Bump `verzod` across packages.
- Relevant updates to test suite.
- Bump `zod` under `selfhost-desktop`.
