Multi-brand new/used car inventory search: an evolution story of python script to a web scraper to an API driven webapp

No worries. Let me know if you need any help!

Yup, just quickly tested it on one Volvo site I have. Itā€™s different URL than new cars, even easier. No need to look for image, and can be searched via query string parameters just like new inventory:

https://www.fallschurchdonbeyervolvo.com/demo-loaner-inventory.htm?model=XC40

Iā€™ll add that next. I wonā€™t probably have time during this weekend, going on family fishing trip, so sometime next week.

Just took a look at some local dealers, and it seems quite a few have a similar URL. If anybody knows dealers with similar, a detailed list can probably be useful for the integration.

Have fun fishing!

Thanks and I second that, if someone can come up with dealer list that support this format, Iā€™d just add them to config and these can be searched on. Minimum (if any) code tuning should be required.

Very few dealers have separate links for loaners. This link will only pull their storeā€™s loaner inventory vs. all 3 Don Beyer Volvo stores inventory for new cars. So you need 3 separate loaner scrapers for each store.

Yeah, V60 or V60CC search will not return any V60CC. Has to be V60 Cross Country

I hear what youā€™re saying, but from feature delivery time perspective, if everything else is equal, itā€™s easier for me to just add 3 config entries for DealerCom sites vs adding one DealerInspire superstore. DealerCom routine is much more generic and stable at the moment compared to DealerInspire, which is slowest and least reliable.

All 3 individual Don Beyer Volvo stores have dedicated links for loaners:
https://www.fallschurchdonbeyervolvo.com/demo-loaner-inventory.htm?model=XC40
https://www.dullesdonbeyervolvo.com/demo-loaner-inventory.htm?model=XC40
https://www.winchesterdonbeyervolvo.com/demo-loaner-inventory.htm?model=XC40

Making all that new inventory available to srapper engine is just adding 3 separate config entries for new inventory, same for loaners, when I add code for loaner support.

Yeah, I know. Whatever is easier, Iā€™m all for it.

mbusa also has an api for dealer inventory here:
https://nafta-service.mbusa.com/api/inv/en_us/new/vehicles/search?model=GLC63C4&dealerId={some 5 digit code}&start=0&year=2020&sortBy=availability&currentDealerOnly=false
you need to use another api to get all dealer id though

Iā€™m in. Not very good but time to put my CS degree to use

3 Likes

Had some time to quickly add loaner search for DealerCom Volvo sites. It only works on those for now.

Thatā€™s awesome! The more the merrier. What languages/frameworks youā€™d be able to help with coding?

So my fishing trip went kaput, but on a bright side Iā€™d probably have some time over the weekend to sink into improving scrapper.

Volvo individual stores are pleasure to work with, they all seem to use same exact version of dealerCom platform, and have very consistent structure across all dealers I tried. I changed loaner detection from specific URL (not all dealers have it, just like @Ursus said), to a loaner image presence detection (as @volvo1 suggested, thanks!) and this detection is very reliable.

Added bunch of Volvo dealers, and currently weā€™re at 85 loaners/400 total new stock for XC90 for example:

image

image

These volvo dealers are basically plug and play, add entry to a config, get dealerā€™s listings for loaners and new stock on UI.

2 Likes

Awesome :+1: It is usually a pain to find Volvo loaners on dealers sites (though they tend to be listed at the end, so one can start looking from last page)

That is awesome! Glad I could help out. Let me know if you need a list of dealers to add.

can you build a scraper for used cars?

You can. Same code, different URLs

Are you looking specifically for used cars at dealerships or everything? Though this is useful, there are a lot more used car dealerships with different website templates than new car dealers that use similar website templates.

I want to add dealers. How do I know if a dealer is Dealer.com, DealerOn or DealerInspire?

UPDATE: I believe I figured it out and submitted a pull request with a bunch of dealers.

I found 3 dealers that didnā€™t seem to belong to either Dealer.com, DealerOn or DealerInspire though. Are there other types out there? eg: Nabthat, DealerFIre

Hey thatā€™s greatā€¦ Iā€™ve seen your pull request, Iā€™ll review it ASAP. Gotta host 4th of July party, so probably early tomorrow. Happy 4th of July everyone.

1 Like

Hey thanks for putting time and effort in helping with dealers. I looked at it quickly and everything looks good. I approved your pull and merged to master. Iā€™ll double check everything tomorrow, to make sure weā€™re not missing anything, but so far it looks great. :+1: :+1:

Sorry, I should have posted earlier on how to identify DealerOn vs DealerInspire vs Dealercom website.

Go to a dealer website, run a search for new inventory for specific model and compare URL to those below. That should tell you which type of platform it is. If itā€™s none of those, we donā€™t support it yet.

www.dealerwebsite.com/new-inventory/index.htm?model={model} - thatā€™s dealerCom

www.dealerwebsite.com/searchnew.aspx?Model={model} - thatā€™s DealerOn

www.dealerwebsite.com//new-vehicles/{model}/#action=im_ajax_call&perform=get_results&vrp_view=listview&page=1 - thatā€™s dealerInspire

Substitute {model} with ā€œX3ā€ for BMW for example.

Next morning edit:
So I reviewed all config changes, everything is top notch. Moved one dealer from DealerInspire to DealerOn, and thatā€™s pretty much itā€¦ Donā€™t see results from Pacific BMW, but Iā€™ll look at it later. Never mind, they are there, too much vodka last night lol.

Now we have huge inventory results for BMW, both east coast and west coast dealers across all 3 types:

image

768 new X3s. Search is becoming slower and slower, as expected, just under 3 minutes for example above. I guess we need to start thinking about adding regional filter. Should I go with the way regions are broken down here, on LH?

  • [California]
  • [Northeast]
  • [Midwest]
  • [South]
  • [West]

Thatā€™s good question, probably, or some just have their own custom built websites altogether. Adding new type of dealer for generic processing would require brand new crap engine routine though, and is not quick.

Ughā€¦ Now with so many new BMW dealers, If you search for BMW across all 3 platforms at once, the search might timeout., depending on model. Apparently Azure has 230 seconds max hard timeout, that can not be changed. If operation takes longer than that, request will timeout. For now, donā€™t use all 3 dealer types at once for BMW searches, while Iā€™m figuring out workaround. Try two, if it still times out, go one by one.

Gotta love Microsoft. Nothing is ever easy or straightforward :grin: