镜像站点 · 本页由第三方 GitHub 只读镜像提供,非 GitHub 官方站点,不接受任何登录或凭据输入。前往 github.com
Skip to content

Feat/extraction workers - #116

Open
ajuvercr wants to merge 27 commits into
mainfrom
feat/extraction-workers
Open

ajuvercr wants to merge 27 commits into
mainfrom
feat/extraction-workers

Conversation

@ajuvercr

Copy link
Copy Markdown
Collaborator

Extract the members extraction to workers, this might help network requests handling.
This helps with LDES's like "https://www.pieter.pm/dcat/sweden/feed.ttl" as these have some very large pages that can clump up the v8 scheduler, which doesn't find time to handle HTTP connections resulting in dropped connections and failed replications.

This broke some tests as starting a worker is more intense then just loading some code, so it is unclear what the threshhold should be, but not something like 150ms.

Comment thread lib/fetcher/extractionPool.ts Outdated
Comment thread bin/cli.ts Outdated
Comment thread lib/fetcher/extractionPool.ts Outdated
Comment thread lib/utils/exitHandler.ts Outdated
- make sure client ends when finished
- harmonize worker variable name
- Add it to the README
- Make sure default config parameters are kept
- Fix race condition at end checking in ordered strategy
- Fix unstable test conditions for client interruptions
- Remove console.log calls in relation tests
- Fix unstable test conditions for unrodered tests
# Conflicts:
#	package-lock.json
#	package.json
- Extend worker pool to dynamically support Web Workers when in the browser
- Pass Quad objects from and to workers using rdf-string
- Add unit tests for individual worker implementations
- Setup browser test environment with vitest and playright
- Use alternative for os when in the browser
- Dynamically load winston when not in the browser
- Add github action job for browser tests
# Conflicts:
#	package-lock.json
#	package.json
# Conflicts:
#	package-lock.json
#	package.json
@julianrojas87

Copy link
Copy Markdown
Collaborator

Okay to merge this @ajuvercr?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants