镜像站点 · 本页由第三方 GitHub 只读镜像提供，非 GitHub 官方站点，不接受任何登录或凭据输入。前往 github.com
Skip to content

Linux and Linux-MUSL Package artifacts getting produced with duplicate RIDs #45027

New issue
New issue
Closed
Closed
Linux and Linux-MUSL Package artifacts getting produced with duplicate RIDs#45027
Assignees
safern
Labels
area-InfrastructureuntriagedNew issue has not been triaged by the area owner

Description

@MattGal
MattGal
opened on Nov 20, 2020

Initially investigated as an AzDO issue via https://github.com/dotnet/core-eng/issues/11472 (see for more context), it looks like linux and linux-musl are getting conflated in package naming.

Seems to have been introduced in #43804. @safern already has context, so assigning to him.

Activity

  1. MattGal
    assigned
    safern
    on Nov 20, 2020
  2. Dotnet-GitSync-Bot
    added
    area-PAL-coreclronly for closed issues
    untriagedNew issue has not been triaged by the area owner
    on Nov 20, 2020
  3. safern
    added
    area-Infrastructure
    and removed
    area-PAL-coreclronly for closed issues
    on Nov 20, 2020

  4. safern commented on Nov 20, 2020

    @safern
    safern
    on Nov 20, 2020
    Member

    Thanks, @MattGal -- I'll put up a fix soon.

    FYI: @am11 @ViktorHofer

  6. am11 commented on Nov 20, 2020

    @am11
    am11
    on Nov 20, 2020
    Member

    Thanks @safern. I cannot access the engineering repo link, but all CI checks were passing when PR was completed. It could be that some target does not run during the CI builds?

  7. safern commented on Nov 20, 2020

    @safern
    safern
    on Nov 20, 2020
    Member

    This was happening on an official build which is internal (Where we produce and sign our packages). Still investigating. I'll update the issue once I have more info.

  8. safern commented on Nov 20, 2020

    @safern
    safern
    on Nov 20, 2020
    Member

    So for more context on this. Some of the packages produced in the linux-musl legs contain the wrong RID when produced:

      Microsoft.NETCore.DotNetHostResolver -> /root/runtime/artifacts/packages/Release/Shipping/Microsoft.NETCore.DotNetHostResolver.6.0.0-alpha.1.20570.6.nupkg
  Microsoft.NETCore.DotNetHost -> /root/runtime/artifacts/packages/Release/Shipping/Microsoft.NETCore.DotNetHost.6.0.0-alpha.1.20570.6.nupkg
  Microsoft.NETCore.DotNetHostPolicy -> /root/runtime/artifacts/packages/Release/Shipping/Microsoft.NETCore.DotNetHostPolicy.6.0.0-alpha.1.20570.6.nupkg
  Microsoft.NETCore.DotNetHost -> /root/runtime/artifacts/packages/Release/Shipping/runtime.linux-arm64.Microsoft.NETCore.DotNetHost.6.0.0-alpha.1.20570.6.nupkg
  Microsoft.NETCore.DotNetHostPolicy -> /root/runtime/artifacts/packages/Release/Shipping/runtime.linux-arm64.Microsoft.NETCore.DotNetHostPolicy.6.0.0-alpha.1.20570.6.nupkg
  Microsoft.NETCore.DotNetHostResolver -> /root/runtime/artifacts/packages/Release/Shipping/runtime.linux-arm64.Microsoft.NETCore.DotNetHostResolver.6.0.0-alpha.1.20570.6.nupkg
  Microsoft.NETCore.DotNetAppHost -> /root/runtime/artifacts/packages/Release/Shipping/runtime.linux-arm64.Microsoft.NETCore.DotNetAppHost.6.0.0-alpha.1.20570.6.nupkg
  Microsoft.NETCore.DotNetAppHost -> /root/runtime/artifacts/packages/Release/Shipping/Microsoft.NETCore.DotNetAppHost.6.0.0-alpha.1.20570.6.nupkg
  Microsoft.NETCore.App.Ref -> 
  Successfully created package '/root/runtime/artifacts/packages/Release/Shipping/Microsoft.NETCore.App.Ref.6.0.0-alpha.1.20570.6.nupkg'.
  Successfully created package '/root/runtime/artifacts/packages/Release/Shipping/Microsoft.NETCore.App.Ref.6.0.0-alpha.1.20570.6.symbols.nupkg'.
  Microsoft.NETCore.App.Runtime -> 
  Successfully created package '/root/runtime/artifacts/packages/Release/Shipping/Microsoft.NETCore.App.Runtime.linux-musl-arm64.6.0.0-alpha.1.20570.6.nupkg'.
  Successfully created package '/root/runtime/artifacts/packages/Release/Shipping/Microsoft.NETCore.App.Runtime.linux-musl-arm64.6.0.0-alpha.1.20570.6.symbols.nupkg'.
  Microsoft.NETCore.App.Host -> 
  Successfully created package '/root/runtime/artifacts/packages/Release/Shipping/Microsoft.NETCore.App.Host.linux-musl-arm64.6.0.0-alpha.1.20570.6.nupkg'.
  Successfully created package '/root/runtime/artifacts/packages/Release/Shipping/Microsoft.NETCore.App.Host.linux-musl-arm64.6.0.0-alpha.1.20570.6.symbols.nupkg'.

    If you look at that, some do have linux-musl-arm64 RID and some linux-arm64 missing the -musl part. What is confusing me, is that we're specifying /p:OutputRid=linux-musl-arm64 globally.

    Some thing that I do see, is that this might be wrong: https://github.com/dotnet/runtime/pull/43804/files#diff-aec399247327507f8a921926dfa34e159e9786455234791d9895fe64185d91f2R155

    As, that is setting OutputRid when Portable=true to _outputRID, however _outputRID doesn't consider linux-musl at all. The only case where linux-musl is considered is: portableOS... https://github.com/dotnet/runtime/pull/43804/files#diff-aec399247327507f8a921926dfa34e159e9786455234791d9895fe64185d91f2R91 which is ignored completely for the PortableBuild output rid.

  9. am11 commented on Nov 20, 2020

    @am11
    am11
    on Nov 20, 2020
    Member

    That part seem to be aligned with the older version: https://github.com/dotnet/runtime/pull/43804/files#diff-c69fd8bc3ea99b48dac09936c142064d7e750d2c626a7e9286384d94ef31fa2dL60.

    Tested with ./build.sh -c Release -cross /p:OutputRid=linux-musl-arm64 /v:diag | grep " OutputRid =", we get

                       OutputRid = linux-musl-arm64
                   OutputRid = linux-musl-arm64
                   OutputRid = linux-musl-arm64
                   OutputRid = linux-musl-arm64
                   OutputRid = linux-musl-arm64
                   OutputRid = linux-musl-arm64
                   OutputRid = linux-musl-arm64
                   OutputRid = linux-musl-arm64

    could be somewhere else in signing target.

  10. safern commented on Nov 20, 2020

    @safern
    safern
    on Nov 20, 2020
    Member

    Yeah, I just validated this. OutputRid is correct. The problem is PackageRID.

    So basically, we have two models here under installer. The regular packages that are built by:

    runtime/src/installer/pkg/projects/host-packages.proj

    Line 9 in 07193b8

    <ProjectReference Include="@(ProjectReference)" AdditionalProperties="PackageTargetRuntime=$(PackageRID)" />

    That uses PackageRID to set the PackageTargetRuntime property which is what then calculates the output ID for those packages:
    https://github.com/dotnet/arcade/blob/4eff4ec03f24b6208d51a372842470a222fa373b/src/Microsoft.DotNet.Build.Tasks.Packaging/src/build/Packaging.targets#L28_L32

    We also have the framework packages which are driven by the shared framework SDK in the sfxproj's. Those use OutputRid that is why they're getting the right output RID. Which use RuntimeIdentifier: https://github.com/dotnet/arcade/blob/d1a58bc7ed005abe2d609245adefe3de4dff5c12/src/Microsoft.DotNet.SharedFramework.Sdk/targets/Microsoft.DotNet.SharedFramework.Sdk.targets#L93
    https://github.com/dotnet/arcade/blob/d1a58bc7ed005abe2d609245adefe3de4dff5c12/src/Microsoft.DotNet.SharedFramework.Sdk/targets/Microsoft.DotNet.SharedFramework.Sdk.targets#L98

    And RuntimeIdentifier is set to OutputRid:

    runtime/src/installer/Directory.Build.props

    Line 61 in 75c0b99

    <RuntimeIdentifier>$(OutputRid)</RuntimeIdentifier>

    PackageRID is used in other places, i.e:

    runtime/src/installer/pkg/projects/Directory.Build.props

    Line 103 in 75c0b99

    <_buildingOnRID Include="$(PackageRID)" Condition="'$(BuildOnUnknownPlatforms)' != 'false'">

    https://github.com/dotnet/runtime/search?q=PackageRID

    It seems like after your change, PackageRID will never have the linux-musl prefix, which it used to have in various cases: https://github.com/dotnet/runtime/pull/43804/files#diff-6952310dc69d6c06f01efd35042d5168dfd1faa9dcfd4b8b5dce9e01a828dc0dL148

    The reason why it worked before is because of this:
    https://github.com/dotnet/runtime/pull/43804/files#diff-1d3554b1a5d588d73d6b8b4fd8203e48c4ba0fba262ca4cc290e341b47593985L142

    It would get a linux-musl prefix here:
    https://github.com/dotnet/runtime/pull/43804/files#diff-1d3554b1a5d588d73d6b8b4fd8203e48c4ba0fba262ca4cc290e341b47593985L100

    I think there are 2 things we need to do here:

    1. update the host-packages.proj to use OutputRid instead
    2. fix PackageRID to have linux-musl when it must have it.

  11. am11 commented on Nov 20, 2020

    @am11
    am11
    on Nov 20, 2020
    Member

    Thanks for the investigation, @safern.
    (i just pulled ubuntu-18.04-cross-arm64-alpine container)

    I have understood the problem. Before the change; in non-installer components:

    • PackageRID is used for package restore, which in case of cross-compilation is the RID of host OS.
    • OutputRid (different casing) is used for packages, RID of target OS.

    In installer, PackageRID had mixed meanings (when used for restore, it is Host's RID, when used under pkg/ directories, it was OutputRid), while OutputRid had consistent meaning.

    PR aligned the meaning of these two properties.

    So I think your #1 is the better option.

  12. safern commented on Nov 20, 2020

    @safern
    safern
    on Nov 20, 2020
    Member

    Yeah, that is exactly what I was going to comment right now. PackageRID is correct and should stay as is. I found out that linux_musl_x64 is producing the right packages, because it is not a cross build, so the PackageRID is linux_musl_x64, because the host is an alpine container, however for arm builds of alpine the host is ubuntu so PackageRID is linux_x64.

    So yeah, the right fix is #1. Are you planning on putting a PR? If not I can put it out there now.

  13. am11 commented on Nov 20, 2020

    @am11
    am11
    on Nov 20, 2020
    Member

    Yup, I have just found from build logs how to repro it in local container. Testing it right now, will put a PR soon.

  14. safern commented on Nov 20, 2020

    @safern
    safern
    on Nov 20, 2020
    Member

    Testing it right now, will put a PR soon.

    Thanks 😄

  15. safern commented on Nov 21, 2020

    @safern
    safern
    on Nov 21, 2020
    Member

    Fixed by: #45041

  16. safern
    closed this as completedon Nov 21, 2020
  17. ghost locked as resolved and limited conversation to collaborators on Dec 21, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area-InfrastructureuntriagedNew issue has not been triaged by the area owner

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions