Repository navigation
microcule + x-ray + docker streaming error #34
Description
While trying to implement a microcule powered scraper I've run in to what looks like a broken stream problem, but only if the whole thing is hosted in a docker container (I know, there's a few deps here that could be the root cause but I figured asking for help can't hurt).
To reproduce clone the gist from above:
git clone https://github.com/proxy/gist.github.com/d10ee3863a6f0773c5386b77abea77fa.git
Build the docker image:
docker build -t microcule-docker
Run the docker image in interactive terminal:
docker run -it -p 3000:3000 microcule-docker
After this hit the following URL's:
http://localhost:3000/request/hook
http://localhost:3000/scrape/hook
http://localhost:3000/scrape/middleware
The second URL breaks in the middle of the JSON. However if you just try a simple npm start and repeat it all runs fine.
One thing to note here is the /request/hook URL works fine in both native/docker so I guess I'm wondering what x-ray does to the stream internally to just close in the middle.
Part of the problem is that debugging this with docker + microcule is a bit tricky and since I suspect that @Marak know a thing or two about hosting hooks inside docker I decided it could be worth filing an issue.
Activity
Marak commentedon Feb 22, 2017
I don't really use Docker at all.
What is the broken stream error?
Is the service timing out? Maybe increase timeout limits?
Either way, if it's working outside docker, it must be a configuration setting related to docker?
mikaelkaron commentedon Feb 22, 2017
The stream error is just that the output is cut in the middle of the response - nothing in the terminal to show any errors so I'm not sure what's going on inside.
Timeout is already set to maximum.
There's a
/scrape/middlewarethat is the exact same code as in the hook but not executed viaspawnand that runs fine both native/docker.Marak commentedon Feb 22, 2017
Not sure, try testing more if you can. See if you can replicate the issue without Docker.
Marak commentedon Feb 25, 2017
@mikaelkaron - Any updates on this?
Are you able to reproduce this with a smaller case?
I'd like to understand why this isn't working in case we need to fix anything in
microcule.Thank you for all your feedback and contributions.
mikaelkaron commentedon Feb 27, 2017
Not yet. I'm working on an alternative implementation that uses a different scraper and so far so good. It looks like it's something in x-ray or it's does that is the problem in combo with microcule.
Any advice on how one can debug a hook remotely (as we spawn a new process it's a bit hard to track at the moment)