Repository navigation
Inconsistent replacement of list column element with NULL in table with 1 row #5558
Description
In base R data.frame we can replace an element of a list column with NULL via:
> DF1=data.frame(L=I(list("A")),i=1)
> DF1$L=list(NULL)
> DF1
L i
1 NULL 1However in data.table, doing that results in deleting the list column entirely:
> DT1=data.table(L=list("A"),i=1)
> DT1$L=list(NULL)
> DT1
i
<num>
1: 1Request: can we make the above code do a replacement (like base R data.frame) instead of deleting the column?
The above issue is confusing because it is inconsistent with how data table handles replacement in tables with more than one row:
> DT2=data.table(L=list("B","C"),i=1)
> DT2$L <- list(NULL,NULL)
> DT2
L i
<list> <num>
1: 1
2: 1
> DF2=data.frame(L=I(list("B","C")),i=1)
> DF2$L <- list(NULL,NULL)
> DF2
L i
1 NULL 1
2 NULL 1Actually, we can do the replacement (inconsistently) in data.table via
> DT1=data.table(L=list("A"),i=1)
> DT1$L=list(list(NULL))
> DT1
L i
<list> <num>
1: 1
Activity
avimallu commentedon Dec 8, 2022
I can do this:
that works, but oddly not
which should be identical per
data.tabledocumentation.[-]can not replace list column element with NULL in table with 1 row[/-][+]Inconsistent replacement of list column element with NULL in table with 1 row[/+]MichaelChirico commentedon Jan 20, 2025
The fix here, #6167, caused a regression: #6740 / markfairbanks/tidytable#831.
In line with the GOVERNANCE rules I think we should revert #6167, at least for now and at least by default:
data.table/GOVERNANCE.md
Line 17 in 3a7ec2d
If we decide the new approach landed on in #6167 is preferable, or that it unlocks a behavior that was previously impossible, we should re-introduce #6167 gradually with a deprecation cycle. Here's my proposal:
1.17.0: Leave Consistent Replacement of List Column with NULL #6167, but hide it behindif (!isTRUE(getOption("datatable.old.list.assignment", FALSE))) { ... }, document option and intention to change defaults from the next release1.18.0: Change option default toTRUE1.19.0: Throw awarningforFALSEoption1.20.0: Throw anerrorforFALSEoption1.21.0: Ignore option, throw warning if it is set1.22.0: Ignore option, throw error if it is set1.23.0: Remove option totallytdhock commentedon Jan 21, 2025
related: #6732
tdhock commentedon Jan 21, 2025
sure let's revert.
and reopen this issue.
MichaelChirico commentedon Jan 21, 2025
FYI, I'll plan to do a cherry-pick style revert instead of a simple clean one -- I think the test suite in the PR is really great, and of utility even if the exact behavior tested needs to change slightly.
joshhwuu commentedon Oct 14, 2025
@tdhock @MichaelChirico bringing attention back to this in lieu of 1.18.0 😄, have we determined that this new approach is still worth pursuing?
tdhock commentedon Mar 31, 2026
so this is definitely impossible to do in a backwards compatible way.
and backwards compatible is more important than consistency, which suggests that we may just close and live with this inconsistency.
it seems to be an unusual use case, so I will close for now.