Repository navigation
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## victor/33298-software-ingestion #33390 +/- ##
================================================================
Coverage 64.08% 64.09%
================================================================
Files 2059 2059
Lines 205256 205242 -14
Branches 6767 6767
================================================================
- Hits 131548 131547 -1
+ Misses 63332 63313 -19
- Partials 10376 10382 +6
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
|
@coderabbitai full review |
✅ Actions performedFull review triggered. |
WalkthroughRevises macOS software ingestion to handle renamed apps by deduplicating via bundle IDs and batch-renaming matched entries. Introduces a helper to update names and sources in bulk. Expands tests to cover rename scenarios mixed with new software, browser extensions handling, and updated expectations for name and source updates. Changes
Sequence Diagram(s)sequenceDiagram
autonumber
participant Host as Host
participant API as Ingest API
participant DS as Datastore (MySQL)
Host->>API: Report installed software (names, bundle IDs, versions)
API->>DS: Link by bundle ID
Note over DS: Identify existing entries matching bundle IDs
DS-->>API: Matched software IDs
API->>DS: Insert new software (no bundle ID match)
Note over API,DS: Build softwareRenames map for bundle-ID matches with differing names
API->>DS: updateTargetedBundleIDs(softwareRenames)
activate DS
DS->>DS: Batched UPDATE (CASE) name and name_source
deactivate DS
DS-->>API: Completed updates
API-->>Host: Ingestion result
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Suggested reviewers
Pre-merge checks and finishing touches❌ Failed checks (2 warnings)
✅ Passed checks (1 passed)
✨ Finishing touches
🧪 Generate unit tests
Tip 👮 Agentic pre-merge checks are now available in preview!Pro plan users can now enable pre-merge checks in their settings to enforce checklists before merging PRs.
Please see the documentation for more information. Example: reviews:
pre_merge_checks:
custom_checks:
- name: "Undocumented Breaking Changes"
mode: "warning"
instructions: |
Pass/fail criteria: All breaking changes to public APIs, CLI flags, environment variables, configuration keys, database schemas, or HTTP/GraphQL endpoints must be documented in the "Breaking Change" section of the PR description and in CHANGELOG.md. Exclude purely internal or private changes (e.g., code not exported from package entry points or explicitly marked as internal).Please share your feedback with us on this Discord post. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
server/datastore/mysql/software.go (1)
748-750: Bug: clobbers previously computed titles (map reinit wipes entries)Reinitializing incomingChecksumToTitle here erases mappings already populated for software without bundle_identifier, causing missing title_id assignment downstream.
Apply this diff:
- // no-op code change - incomingChecksumToTitle = make(map[string]fleet.SoftwareTitle, len(newSoftwareChecksums)) + // keep previously collected titles (do not reinitialize incomingChecksumToTitle)Please add/adjust a test to cover a mixed batch of software with and without bundle_identifier to prevent regressions.
🧹 Nitpick comments (5)
server/datastore/mysql/software.go (1)
483-524: Defensive CASE: add ELSE name to avoid accidental NULLsIf an id slips into the WHERE but not into the CASE (unexpected but possible under refactors), name would become NULL. Make the CASE default to the current value.
Apply this diff:
- updateStmt := fmt.Sprintf( - `UPDATE software SET name = CASE id %s END, name_source = 'bundle_4.67' WHERE id IN (%s)`, + updateStmt := fmt.Sprintf( + `UPDATE software SET name = CASE id %s ELSE name END, name_source = 'bundle_4.67' WHERE id IN (%s)`, strings.Join(updateCases, " "), strings.Join(updateIDs, ","), )server/datastore/mysql/software_test.go (4)
1882-1897: Avoid brittle assertion on name_source; assert prefix instead.Hard‑coding "bundle_4.67" makes the test fragile if the internal marker changes. Prefer checking it indicates a bundle-ID match.
Apply this diff:
- // Verify the name_source was updated - var nameSource string - err = ds.writer(ctx).GetContext(ctx, &nameSource, - `SELECT name_source FROM software WHERE id = ?`, originalSoftwareID) - require.NoError(t, err) - require.Equal(t, "bundle_4.67", nameSource, "Name source should indicate bundle ID match") + // Verify the name_source was updated (bundle-ID match) + var nameSource string + err = ds.writer(ctx).GetContext(ctx, &nameSource, + `SELECT name_source FROM software WHERE id = ?`, originalSoftwareID) + require.NoError(t, err) + require.True(t, strings.HasPrefix(nameSource, "bundle_"), "Name source should indicate bundle ID match, got %q", nameSource)
8987-9042: Strengthen assertions: verify names actually changed on the reused rows.You confirm ID reuse and absence of duplicates; add a quick check that the names were updated to the new values.
Apply this diff after verifying IDs (before the COUNT query):
// Verify the IDs are the same (software was reused, not recreated) for _, s := range host.Software { originalID, ok := originalIDs[s.BundleIdentifier] require.True(t, ok, "Bundle ID %s should exist", s.BundleIdentifier) require.Equal(t, originalID, s.ID, "Software ID should be reused for bundle ID %s", s.BundleIdentifier) } + // Verify names were updated on the reused rows + expectedNames := map[string]string{ + "com.example.app": "Renamed.app", + "com.example.another": "AlsoRenamed.app", + } + for _, s := range host.Software { + require.Equal(t, expectedNames[s.BundleIdentifier], s.Name) + }
9043-9134: Make the name_source check resilient; assert a bundle_ prefix.*Same brittleness as above; use HasPrefix so internal versioning doesn’t break tests.
Apply this diff:
- // Verify name_source was updated - var nameSource string - err = ds.writer(ctx).GetContext(ctx, &nameSource, - `SELECT name_source FROM software WHERE id = ?`, s.ID) - require.NoError(t, err) - require.Equal(t, "bundle_4.67", nameSource, "Name source should indicate bundle ID match") + // Verify name_source was updated (bundle-ID match) + var nameSource string + err = ds.writer(ctx).GetContext(ctx, &nameSource, + `SELECT name_source FROM software WHERE id = ?`, s.ID) + require.NoError(t, err) + require.True(t, strings.HasPrefix(nameSource, "bundle_"), "Name source should indicate bundle ID match, got %q", nameSource)Also, consider filtering title counts to what this test created (to avoid coupling to fixtures/migrations):
- err = ds.writer(ctx).GetContext(ctx, &titleCount, - `SELECT COUNT(DISTINCT id) FROM software_titles`) + err = ds.writer(ctx).GetContext(ctx, &titleCount, + `SELECT COUNT(DISTINCT id) FROM software_titles WHERE name IN ('Slack 2.app','Chrome.app','CustomTool')`)
9135-9259: Validate browser extension invariants and reduce name-based coupling.
- Assert ExtensionID and Browser are set and bundle_identifier is empty for all entries.
- Optional: prefer keying by extensionID+browser to avoid name ambiguity.
Apply this diff after loading host.Software in Test Case 1:
// Verify extensions were added correctly err = ds.LoadHostSoftware(ctx, host, false) require.NoError(t, err) require.Len(t, host.Software, 4, "Should have 4 browser extensions") + // Invariants for browser extensions + for _, s := range host.Software { + require.Equal(t, "browser_plugins", s.Source) + require.Empty(t, s.BundleIdentifier) + require.NotEmpty(t, s.ExtensionID) + require.NotEmpty(t, s.Browser) + }Please confirm that "browser_plugins" is the intended canonical source string across the ingestion pipeline (earlier tests use "chrome_extensions"). If not, update to the canonical value.
📜 Review details
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (3)
changes/29053-sw-names-on-ingest(1 hunks)server/datastore/mysql/software.go(2 hunks)server/datastore/mysql/software_test.go(4 hunks)
🧰 Additional context used
📓 Path-based instructions (1)
**/*.go
⚙️ CodeRabbit configuration file
When reviewing SQL queries that are added or modified, ensure that appropriate filtering criteria are applied—especially when a query is intended to return data for a specific entity (e.g., a single host). Check for missing WHERE clauses or incorrect filtering that could lead to incorrect or non-deterministic results (e.g., returning the first row instead of the correct one). Flag any queries that may return unintended results due to lack of precise scoping.
Files:
server/datastore/mysql/software.goserver/datastore/mysql/software_test.go
🔇 Additional comments (3)
server/datastore/mysql/software.go (2)
426-428: No action required (comment-only change).
434-459: LGTM: Consolidated rename planning before batch updateCollecting rename targets from both newly linked and pre-existing bundle-ID matches is sound and keeps the UPDATE minimal.
server/datastore/mysql/software_test.go (1)
57-59: Nice additions to the test matrix.Adding rename-only, rename-with-new, and browser extensions scenarios meaningfully expands coverage.
| @@ -0,0 +1 @@ | |||
| * Re-added and optimized fix for macos software ingestion to prevent duplicate software due to end user renaming software on host. | |||
There was a problem hiding this comment.
Nit: casing and wording
Prefer “macOS” and hyphenate “end‑user”.
Apply this diff:
-* Re-added and optimized fix for macos software ingestion to prevent duplicate software due to end user renaming software on host.
+* Re-added and optimized fix for macOS software ingestion to prevent duplicate software due to end-user renaming software on host.📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| * Re-added and optimized fix for macos software ingestion to prevent duplicate software due to end user renaming software on host. | |
| * Re-added and optimized fix for macOS software ingestion to prevent duplicate software due to end-user renaming software on host. |
🤖 Prompt for AI Agents
In changes/29053-sw-names-on-ingest around line 1, update the commit/description
text to use the correct casing and hyphenation: change “macos” to “macOS” and
“end user” to “end‑user” (with a hyphen), e.g. “Re-added and optimized fix for
macOS software ingestion to prevent duplicate software due to end‑user renaming
software on host.”
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Resolves #29053 Resolves #33298 For reference, the diffs for merging Konstantin's changes into my original PR are here: #33390 # Checklist for submitter - [x] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. ## Testing - [x] Added/updated automated tests - [x] Where appropriate, [automated tests simulate multiple hosts and test for host isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing) (updates to one hosts's records do not affect another) - [x] QA'd all new/changed functionality manually <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - Bug Fixes - Fixed duplicate macOS software entries caused by users renaming apps, ensuring accurate, consolidated inventory. - Documentation - Documented improved software ingestion performance by pre-inserting data in smaller batches to reduce database lock times during host check-ins. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Resolves #29053 Resolves #33298 For reference, the diffs for merging Konstantin's changes into my original PR are here: #33390 # Checklist for submitter - [x] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. ## Testing - [x] Added/updated automated tests - [x] Where appropriate, [automated tests simulate multiple hosts and test for host isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing) (updates to one hosts's records do not affect another) - [x] QA'd all new/changed functionality manually <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - Bug Fixes - Fixed duplicate macOS software entries caused by users renaming apps, ensuring accurate, consolidated inventory. - Documentation - Documented improved software ingestion performance by pre-inserting data in smaller batches to reduce database lock times during host check-ins. <!-- end of auto-generated comment: release notes by coderabbit.ai --> (cherry picked from commit 1ae9597)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Resolves #29053 Resolves #33298 For reference, the diffs for merging Konstantin's changes into my original PR are here: #33390 # Checklist for submitter - [x] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. ## Testing - [x] Added/updated automated tests - [x] Where appropriate, [automated tests simulate multiple hosts and test for host isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing) (updates to one hosts's records do not affect another) - [x] QA'd all new/changed functionality manually <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - Bug Fixes - Fixed duplicate macOS software entries caused by users renaming apps, ensuring accurate, consolidated inventory. - Documentation - Documented improved software ingestion performance by pre-inserting data in smaller batches to reduce database lock times during host check-ins. <!-- end of auto-generated comment: release notes by coderabbit.ai --> (cherry picked from commit 1ae9597)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Resolves #29053 Resolves #33298 For reference, the diffs for merging Konstantin's changes into my original PR are here: #33390 # Checklist for submitter - [x] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. ## Testing - [x] Added/updated automated tests - [x] Where appropriate, [automated tests simulate multiple hosts and test for host isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing) (updates to one hosts's records do not affect another) - [x] QA'd all new/changed functionality manually <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - Bug Fixes - Fixed duplicate macOS software entries caused by users renaming apps, ensuring accurate, consolidated inventory. - Documentation - Documented improved software ingestion performance by pre-inserting data in smaller batches to reduce database lock times during host check-ins. <!-- end of auto-generated comment: release notes by coderabbit.ai --> (cherry picked from commit 1ae9597)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Resolves #29053 Resolves #33298 For reference, the diffs for merging Konstantin's changes into my original PR are here: #33390 # Checklist for submitter - [x] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. ## Testing - [x] Added/updated automated tests - [x] Where appropriate, [automated tests simulate multiple hosts and test for host isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing) (updates to one hosts's records do not affect another) - [x] QA'd all new/changed functionality manually <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - Bug Fixes - Fixed duplicate macOS software entries caused by users renaming apps, ensuring accurate, consolidated inventory. - Documentation - Documented improved software ingestion performance by pre-inserting data in smaller batches to reduce database lock times during host check-ins. <!-- end of auto-generated comment: release notes by coderabbit.ai --> (cherry picked from commit 1ae9597)
Related issue: Resolves #
Checklist for submitter
If some of the following don't apply, delete the relevant line.
Changes file added for user-visible changes in
changes/,orbit/changes/oree/fleetd-chrome/changes.See Changes files for more information.
Input data is properly validated,
SELECT *is avoided, SQL injection is prevented (using placeholders for values in statements)If paths of existing endpoints are modified without backwards compatibility, checked the frontend/CLI for any necessary changes
Testing
Added/updated automated tests
Where appropriate, automated tests simulate multiple hosts and test for host isolation (updates to one hosts's records do not affect another)
QA'd all new/changed functionality manually
For unreleased bug fixes in a release candidate, one of:
Database migrations
COLLATE utf8mb4_unicode_ci).New Fleet configuration settings
If you didn't check the box above, follow this checklist for GitOps-enabled settings:
fleetctl generate-gitopsfleetd/orbit/Fleet Desktop
runtime.GOOSis used as needed to isolate changesSummary by CodeRabbit
Bug Fixes
Performance