TVMOp doesn't work well with GPU builds #17840

Activity

1. apeforest

   added

   Bug

   TVM OPOperators implemented using TVMOperators implemented using TVM

   on Mar 15, 2020
2. leezu commented on Mar 18, 2020

   

   leezu

   on Mar 18, 2020

   Contributor

   More actions

   @apeforest do you mean this error:

   [2020-03-18T08:37:14.025Z] TVMError: Check failed: ret == 0 (-1 vs. 0) : Check failed: f != nullptr: Cannot find function less_scalar_gpufloat32_2bool_2_kernel0 in the imported modules or global registry
   
3. apeforest commented on Mar 18, 2020

   

   apeforest

   on Mar 18, 2020

   ContributorAuthor

   More actions

   Yes, but it seems to be fixed.
4. apeforest

   closed this as completedon Mar 18, 2020
5. leezu commented on Mar 18, 2020

   

   leezu

   on Mar 18, 2020

   Contributor

   More actions

   Why would it be fixed? I got this error 2020-03-18T08:37:14.025Z UTC
6. leezu

   reopened this on Mar 18, 2020
7. ChaiBapchya commented on Apr 28, 2020

   

   ChaiBapchya

   on Apr 28, 2020

   Last edited by ChaiBapchya

   Contributor

   More actions

   The issue persists. Upon trying unix-gpu build on G4 in CI Dev account.
   http://jenkins.mxnet-ci-dev.amazon-ml.com/blue/organizations/jenkins/mxnet-validation-bapac%2Funix-gpu/detail/update_gpu_toolchain/8/pipeline/414

   all 3 failed tests fail in a similar fashion
   they fail at 7 tests with the following error

   TVMError: Check failed: ret == 0 (-1 vs. 0) : Check failed: f != nullptr: Cannot find function  <x> in the imported modules or global registry
   

   Internal functions that can't be found

   greater_equal_gpufloat32_0float32_0bool_0_kernel0 (x2)
   logical_and_gpufloat32_1float32_1bool_1_kernel0 (x2)
   equal_gpufloat32_2float32_2bool_2_kernel0 (x2)
   sum_gpureduce1st_dim_1req_kWriteTobool_5float32_2float32_2_kernel0 (x3)
   cuda_rad2degfloat32_2float32_2_kernel0 (x2)
   

   7 Tests that fail as a result

   tests/python/unittest/test_numpy_interoperability.py:test_np_array_function_protocol
   tests/python/unittest/test_numpy_interoperability.py:test_np_array_ufunc_protocol
   
   tests/python/unittest/test_numpy_ndarray.py:test_np_ndarray_binary_element_wise_ops
   
   tests/python/unittest/test_numpy_op.py:test_np_sum
   tests/python/unittest/test_numpy_op.py:test_np_mean
   tests/python/unittest/test_numpy_op.py:test_np_unary_funcs
   tests/python/unittest/test_numpy_op.py:test_np_binary_funcs
   
8. leezu commented on Apr 29, 2020

   

   leezu

   on Apr 29, 2020

   Contributor

   More actions

   Reproducer
   Compile MXNet with USE_TVMOP=1.

   import mxnet as mx
   
   x = mx.np.array([[0, 1], [1, 1], [2, 2]], ctx=mx.gpu())
   idx = x < 2
   x[idx]
   
9. leezu

   added a commit that references this issue on May 1, 2020

   Disable -DUSE_TVM_OP on GPU builds (#18204)

   ...

   03fdfe0
10. leezu

    changed the title [-]CI unix-gpu build failure in TVM op[/-] [+]TVMOp doesn't work well with GPU builds[/+] on May 1, 2020
11. leezu commented on May 1, 2020

    

    leezu

    on May 1, 2020

    Contributor

    More actions

    Has been disabled on CI: #18204

    Let's track fixing TVMOp in this issue?
12. leezu

    assigned

    yzhliu

    on May 1, 2020
13. yzhliu

    added

    v2.0

    Numpy

    WIP

    on May 6, 2020
14. yzhliu commented on May 9, 2020

    

    yzhliu

    on May 9, 2020

    Member

    More actions

    @jinboci will be helping
15. jinboci

    added 3 commits that reference this issue on Jun 9, 2020

    fixing apache#17840

    087a100

    fixing issue apache#17840

    75f975b

    fixing issue apache#17840

    84c389a
16. AntiZpvoh

    added a commit that references this issue on Jul 6, 2020

    Disable -DUSE_TVM_OP on GPU builds (apache#18204)

    ...

    d5d762d
17. yzhliu

    added a commit that references this issue on Aug 22, 2020

    Fixing tvmgpu issue & not restoring tvmop checks (#18818)

    ...

    a847df7