@coder8 I ran your crawler and built a little UI for it I saw all x3’s that returned are 2020 except a 2012 showed up in the result (just curious not meant to be critical at all).
6 Likes
If you guys wanted to monetize this, at least as a side hustle, I’m sure more than a few folks would give you a few bucks to search for cars much like the freemium EV-Hunter site.
It both does and doesn’t surprise me the OEM’s don’t have a boilerplate dealer site service that automatically ties into their own national search engine as well as API feed the major marketing sites like Car.com and Auto Trader. So dealers are one & done managing inventory and data is consistent across channels.
Until they ever get their act together, there’s space for someone to make a buck “Kayaking” it.
1 Like
Very very interesting. Unfortunately, I’m not a python guy, wasn’t even able to make it run in VS code… However, there is a similar .NET library, ScrapySharp, which works really well on DealerOn websites, regardless of make. Spent about a day creating core functionality that can be wired to UI afterwards. Still a bit rough with DealerInspire sites, and haven’t integrated third one. Just some rich data examples that could be had. Simply takes a model name, e.g. “X1” to search for BMW stock, “GLC” for MB stock etc.
Mercedes:
BMW:
Planning to spend few more days (time permitted) tuning it for most compatibility with websites, figuring out github and such for further sharing if anyone is interested.
Pushed it to GitHub if someone wants to get the code. I’m really a noob with Git, so please let me know if there is something with that repo.
6 Likes
I have nothing to contribute (for now at least) other than to say I f***ing love this forum and the knowledge people share here.
2 Likes
Don’t be so hard on yourself! Everyone has their strengths and expertise in different things.
Can’t wait for the gardening and landscaping thread to really explode!
9 Likes
My wife: glances at my laptop screen “What are you reading now on that site?”
Me: “Just how people are managing their lawns and what they pay and stuff.”
My wife: “Isn’t that a car forum?”
Me: “not always, babe”
13 Likes
Been thinking about doing something like this since I’ve joined this forum.
I’m a noob leaser but got good experience with scripting, Python, AWS (lambda, EC2, etc) and containers (docker).
I’d be happy to assist on this effort !
3 Likes
First off, big props to @coder8 as this web app is based on his idea of scraping script. I’m just not a python guy to contribute to his repo, and wanted to expand it to other makes, so this is implemented in .NET around similar scrapping library ScrapySharp. I built core library that does the scrapping, and and quickly threw together rudimentary UI in ASP.net on top of that.
-
App is hosted in Asure, feel free to browse and test
http://carscrapper.azurewebsites.net/ -
Code is shared on GitHub, feel free to contribute, especially in need of some fancy shmancy pretty and responsive modern UI.
https://github.com/clasys/CarScrapper/
App usage:
- Currently configured for dozen or so dealers, mainly BMW and Mercedes. I will be adding more as I go. This is the current dealer list; https://github.com/clasys/CarScrapper/blob/master/CarScrapper.Core/Config/Dealers.json
- Enter make, e.g. “BMW”, model, e.g. “X7” and hit “Search”. You can check both dealer types, DealerOn and DealerInspire, although including DealerInspire makes search much slower.
- Result grid is sortable and car and dealer links are clickable.
Problems:
- Most glaring one is that resultset is limited per dealer, since most have paging on their websites. I either need to figure out how to grab full data per website call, avoiding paging or implement multiple calls to support it.
TODOs:
- Need to implement 3rd type of dealer website providers, forgot the name. Am planning to do it next.
Anyone who’s willing to contribute, are most welcome to. I’m planning to continue working on this here and there, time permitted.
My Azure account is free to some extent, if this blows up and I start to incur charges, I might change the hosting. If anyone knows free ASP.net 4.7 hosting, hit me up.
Edit: lots has changed in 10 days, make sure you read till the end of the thread on how to run latest search
17 Likes
Great stuff! Happy I inspired you! The app looks super neat!
Thanks! It’s very interesting exercise, I haven’t had side project in so long
love the initiative and interested in helping out
1 Like
Would be awesome to have you on board. 
I started logging issues/TODOs on GitHub:
Feel free to get code and take a look. Let me know if I can be of any help.
1 Like
RustyDaemom, meme lord, keyboard enthusiast, coder, and all around MVP.
If you ever have some switches or mechs you need to sell, hit me up.
2 Likes
Lol, will do! No time for memes now, furiously slapping dat mech brb
1 Like
Added few DealerOn dealers for different makes, added list of makes/dealers that could currently be searched on:
https://carscrapper.azurewebsites.net/
DealerOn compatibility is great, basically just plug new dealer info into config and it works fine… If anyone could help me expand the list (any make really) for DealerOn websites, I could quickly add it to search.
DealerOn website is basically www.websitename.com/searchnew.aspx?Model={model}
1 Like
Who are these: [dealer].com/new-inventory/index.htm?model=XC90?
Not sure, this doesn’t look like DealerOn or DealerInspire default search URLs.
Just added few more dealers, now we have east coast Kia, Ford and Honda in the list. I gotta step away for few hours, pesky work stuff is piling up, not as nearly entertaining as scraping car info, but heh
1 Like
This:
Oh, I think that’s the 3rd provider @coder8 had implemented. I still need to create implementation for this one, it’s on the list…
1 Like
Added quick Dealer.com implementation, with one dealer for now, Ramey Auto Group. Fortunately they cell many makes, Chevy, Dodge, Caddy etc. Don’t forget to check “include DealerCom sites” checkbox, its unchecked by default.
1 Like