Add support for WebAssembly System Interface (wasm32-wasi)

@tiran

BPO	46315
Nosy	@tiran, @pmp-p, @kumaraditya303
PRs	bpo-46315: Add ifdef HAVE_ feature checks for WASI compatibility (GH-30507) #30507 bpo-46315: Use fopencookie() to avoid dup() in _PyTokenizer_FindEncodingFilename #32033 bpo-46315: Use fopencookie only on Emscripten 3.x and newer (GH-32266) #32266

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/tiran'
closed_at = None
created_at = <Date 2022-01-09.14:47:43.508>
labels = ['type-feature', 'build', '3.11']
title = 'Add support for WebAssembly System Interface (wasm32-wasi)'
updated_at = <Date 2022-04-02.21:11:46.918>
user = 'https://github.com/tiran'

bugs.python.org fields:

activity = <Date 2022-04-02.21:11:46.918>
actor = 'christian.heimes'
assignee = 'christian.heimes'
closed = False
closed_date = None
closer = None
components = ['Build']
creation = <Date 2022-01-09.14:47:43.508>
creator = 'christian.heimes'
dependencies = []
files = []
hgrepos = []
issue_num = 46315
keywords = ['patch']
message_count = 4.0
messages = ['410153', '410198', '410470', '416588']
nosy_count = 3.0
nosy_names = ['christian.heimes', 'pmpp', 'kumaraditya']
pr_nums = ['30507', '32033', '32266']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue46315'
versions = ['Python 3.11']

WASI is another WebAssembly platform similar to Emscripten (bpo-40280). Simply speaking Emscripten binaries (wasm32-emscripten) run inside a browser while WASI binaries target standalone runtimes like wasmtime [2][3] on the host. The lines are a bit blurry, as it is possible to run WASI binaries in the browser with help of JS-polyfills. WASI provides compile once, run anyway with JIT/AOT and sandboxing.

WASI is still under development and is lacking several core features:

threading support and pthread API
sockets
signals are emulated (_WASI_EMULATED_SIGNAL and -lwasi-emulated-signal)
DAC APIs like chmod, umask
user-related APIs like pwd, grp
dup(), dup2(), F_DUPFD

For 3.11 I plan to fix our use of #ifdef HAVE_FEATURE to make it easier to experiment with WASI. The pthread APIs need stubs, which I won't commit to 3.11 upstream yet.

[1] https://wasi.dev/
[2] https://github.com/bytecodealliance/wasmtime
[3] https://hacks.mozilla.org/2019/03/standardizing-wasi-a-webassembly-system-interface/

dup() is required by _PyTokenizer_FindEncodingFilename(). I came up with this hack:

// from wasi-libc libc-top-half/musl/src/internal/stdio_impl.h
struct _IO_FILE {
    unsigned flags;
    unsigned char *rpos, *rend;
    int (*close)(FILE *);
    unsigned char *wend, *wpos;
    // incomplete
};

static int
dummy_close(FILE *fp) {
    return 0;
};

static FILE *
_Py_fdopen_borrow(int fd, const char *mode) {
    FILE *fp = fdopen(fd, mode);
    ((struct _IO_FILE*)fp)->close = dummy_close;
    return fp;
}

keithw on #wasi pointed out that fopencookie() can archive the same outcome without resorting to ABI-specific hack. A trivial implementation is straight forward:

typedef union {
    void *cookie;
    int fd;
} borrowed;

static ssize_t
borrow_read(void *cookie, char *buf, size_t size)
{
borrowed b;
b.cookie = cookie;
return read(b.fd, (void *)buf, size);
}

static ssize_t
borrow_write(void *cookie, const char *buf, size_t size)
{
    errno = ENOTSUP;
    return -1;
}

static int
borrow_seek(void *cookie, off_t *off, int whence)
{
    borrowed b;
    b.cookie = cookie;
    off_t pos;
    pos = lseek(b.fd, *off, whence);
    if (pos == (off_t)-1) {
        return -1;
    } else {
        *off = pos;
        return 0;    
    }
}

static int
borrow_close(void *cookie)
{
    // does not close(fd)
    return 0;
}

FILE *
_Py_fdopen_borrow(int fd, const char *mode) {
    // only support read for now
    if (strcmp(mode, "r") != 0) {
        return NULL;
    }
    cookie_io_functions_t cookie_io = {
        borrow_read, borrow_write, borrow_seek, borrow_close
    };
  	// cookie is just the fd
    borrowed b;
    b.fd = fd;
    return fopencookie(b.cookie, "r", cookie_io);
}

New changeset a6ca8ee by Christian Heimes in branch 'main':
bpo-46315: Add ifdef HAVE_ feature checks for WASI compatibility (GH-30507)
a6ca8ee

New changeset 3df0e63 by Christian Heimes in branch 'main':
bpo-46315: Use fopencookie only on Emscripten 3.x and newer (GH-32266)
3df0e63

pythongh-90473: Fix more tests on platforms without umask

pythongh-90473: Check for HAVE_KILL in main.c:exit_sigint

gh-90473: Fix more tests on platforms without umask (GH-95164)

pythongh-90473: Fix more tests on platforms without umask (pythonGH-9…

gh-90473: Check for HAVE_KILL in main.c:exit_sigint (GH-95165)

pythongh-90473: Check for HAVE_KILL in main.c:exit_sigint (pythonGH-9…

gh-90473: Fix more tests on platforms without umask (GH-95164)

gh-90473: Check for HAVE_KILL in main.c:exit_sigint (GH-95165)

Since we can build for WASI without patches now, I'm closing this as finished!

Navigation Menu

Repository navigation

Add support for WebAssembly System Interface (wasm32-wasi) #90473

81 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Add support for WebAssembly System Interface (wasm32-wasi) #90473

Description

Activity

tiran commented on Jan 9, 2022

tiran commented on Jan 10, 2022

tiran commented on Jan 13, 2022

tiran commented on Apr 2, 2022

81 remaining items

brettcannon commented on Mar 28, 2023

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions