Repository navigation
Create a script to remove dead, deprecated or invalid links from planet #233
Description
- pysoy hasn't been updated in about 5 years
- Cherrypy link is dead
- Turbo gears hasn't been updated for over a year and the project is marginal
- These could or should be replaced with links to the pyramid, flask, pandas, or jupyter feeds
Activity
[-]Remove dead or deprecated links for sidebar[/-][+]Remove dead or deprecated links from sidebar[/+]rochacbruno commentedon Jun 13, 2017
I suggested to create a validator script to run monthly #124 (comment)
That script would take each URL check its status and validate it in RSS validator.
[-]Remove dead or deprecated links from sidebar[/-][+]Create a script to remove dead, deprecated or invalid links from planet[/+]rochacbruno commentedon Jun 13, 2017
@pydanny I changed the title of your issue to address the automation of this task.
rochacbruno commentedon Jun 13, 2017
Script should do
config.iniand alsopython librariesandpython planetslistCheck return code is 200
Check data of last update is newer than one year
Check feed is valid using feedvalidator/podcastvalidator/othervalidator
If not valid, store the URL in a simple database (text file, sqlite or something)
bad status\invalid3 times, remove from planetpydanny commentedon Jun 13, 2017
Pysoy and the others I mentioned aren't in the general feeds. They are in the "python libraries" and "python planets" lists. Will these be covered by your script or is that just for RSS feeds?
As for your script, go ahead and remove my blog from your list already. I say that because it's going to fail your W3C validator check and I don't have the luxury of time to fix it. Do keep in mind that this aggregator is the ONLY one I know of that insists on W3C validation.
rochacbruno commentedon Jun 13, 2017
@pydanny yeah we must discuss if using w3c validator is a good choice or not, we only need to check if the feed doesn't break the planet as it happened in the past. Maybe we can write our own simple validator so we do not need to rely on w3c one.
tjguk commentedon Jun 13, 2017
pydanny commentedon Jun 13, 2017
Yes, it's too strict. No other aggregator I'm on blocks my RSS feed.
pybites commentedon Jul 19, 2017
The cleanup and validation could be nice PyBites challenges -https://pybit.es/pages/challenges.html
What do you think?
pybites commentedon Apr 2, 2018
We asked our community to give this a crack https://pybit.es/codechallenge49.html
mridubhatnagar commentedon Jul 2, 2018
Hi
Can I be assigned this issue to work on. I don't really know the complexity of the issue. But, I am curious to work on the same.
Based on the above discussion what I could understand is
Links which are no more working should be removed.
I am currently tied up with some other stuff. But, after a month shall start working on it.
Can I please be assigned this. It looks interesting.
Thanks
mridubhatnagar commentedon Jul 2, 2018
Also, are we validating a RSS feed based on the W3 validator?
Or we consider a RSS feed to be valid if the feedparser is giving us no error?
Or we are relying on none, And instead create one of our own and validate?
rochacbruno commentedon Jul 3, 2018
@mridubhatnagar the only problem with W3 is that it considers podcast feeds invalid, so we need to have a podcast feed validator.
mridubhatnagar commentedon Jul 3, 2018
@rochacbruno Fair enough. I would like to give it a shot.
Also, I think some RSS feeds are valid. But, still not showing up on planet. Like @pybites live feed.
I am not sure though if feedparser works for podcast feeds or not.
I will code podcast feed validator.