First off, big props to @coder8 as this web app is based on his idea of scraping script. I’m just not a python guy to contribute to his repo, and wanted to expand it to other makes, so this is implemented in .NET around similar scrapping library ScrapySharp. I built core library that does the scrapping, and and quickly threw together rudimentary UI in ASP.net on top of that.
-
App is hosted in Asure, feel free to browse and test
http://carscrapper.azurewebsites.net/ -
Code is shared on GitHub, feel free to contribute, especially in need of some fancy shmancy pretty and responsive modern UI.
https://github.com/clasys/CarScrapper/
App usage:
- Currently configured for dozen or so dealers, mainly BMW and Mercedes. I will be adding more as I go. This is the current dealer list; https://github.com/clasys/CarScrapper/blob/master/CarScrapper.Core/Config/Dealers.json
- Enter make, e.g. “BMW”, model, e.g. “X7” and hit “Search”. You can check both dealer types, DealerOn and DealerInspire, although including DealerInspire makes search much slower.
- Result grid is sortable and car and dealer links are clickable.
Problems:
- Most glaring one is that resultset is limited per dealer, since most have paging on their websites. I either need to figure out how to grab full data per website call, avoiding paging or implement multiple calls to support it.
TODOs:
- Need to implement 3rd type of dealer website providers, forgot the name. Am planning to do it next.
Anyone who’s willing to contribute, are most welcome to. I’m planning to continue working on this here and there, time permitted.
My Azure account is free to some extent, if this blows up and I start to incur charges, I might change the hosting. If anyone knows free ASP.net 4.7 hosting, hit me up.