Multi-brand new/used car inventory search: an evolution story of python script to a web scraper to an API driven webapp

Probably better way to pull Volvo from the group’s site. And also there are other makes there.
https://www.beyerauto.com/
image

I guess individual URLs will also work (group’s site is on DealerInspire, individual stores are on Dealer dot com

{
“make”: “Volvo”,
“name”: “Don Beyer”,
“url”: “https://www.beyerauto.com/new-vehicles/volvo”,
“dealertype”: “DealerInspire”
}

Group site might work with DealerInspire scrape, but I’d have to look. Individual stores though look like they are fine just like that to be processed via DealerCom code. Just need URLs like that https://www.fallschurchdonbeyervolvo.com/new-inventory/index.htm

Setup like this:
{
“make”: “Volvo”,
“name”: “Beyer Volvo Cars of Falls Church”,
“url”: “https://www.fallschurchdonbeyervolvo.com/”,
“dealertype”: “DealerCom”
}

Yup, just added Falls Church volvo, search for it with “dealerCom sites” checked

DealerCom implementation needs to be tuned more, volvo results are missing VIN and there is something with links

I was thinking about group site since you can pull inventory from all of their Volvo stores with one link (https://www.beyerauto.com/new-vehicles/volvo) rather than coding each store individually.

It’s just matter of adding individual stores to config. No coding required, except maybe tuning generic dealerCom engine a bit to account for slight differences in HTML around different dealers. And that engine tuning will positively apply to every future dealerCom site as well.

Honestly, its much faster, since DealerInspire implementation is most complicated/flaky at the moment. Also, still there is this problem with paging, I’m currently not supporting it. So unless group site shows all results on single page, app won’t show all inventory.

I’m planning to spend some time on that paging problem, this is the biggest one so far.

Yeah, that’s what I meant :slightly_smiling_face:

They go by 12/page on dealer com for Volvo that I looked at.

That’s a good idea, scraping group site rather than individual stores… Far more complicated to properly implement though. I’ll put it on TODO list

Should we combine both threads? Probably better to have everything in one place.

2 Likes

I second combining the two threads

1 Like

Sure thing.

I’ve been seriously neglecting my work duties in lieu of this lol, don’t remember when was the last time I had so much fun coding…

Added :fire: :fire:PAGING SUPPORT :fire: :fire: to dealerCom sites, now it returns all (I hope) results. @Ursus, want to run search for Volvo and confirm?

http://carscrapper.azurewebsites.net/

2 Likes

Did from my phone, nice. Some links don’t have vehicle parameters in them. The ones that do have double // (still works, of course)

https://www.fallschurchdonbeyervolvo.com**//**new/Volvo/2020-Volvo-XC90-…

Yea, I just fixed some misfires for URL identifications. On a single site, their URLs for new cars can point to 2 diff locations… lol Thanks for catching that

1 Like

Search by dealerCom sites now should return all available inventory. Added some Genesis dealers, and I think some others too… Latest version is published to Azure. I’m off, gotta recharge for tomorrow.

3 Likes

Thanks @RustyDaemon @coder8 for the effort! Love the energy here!

@michael and I have been brainstorming on how we can leverage the power of the community to integrate more tech features for the benefit of the community. In the early days of Leasehackr, we had outside help to build a web scrapper for RV/MF, which enabled us to create a master sheet for RV/MF each month, but due to the lack of maintenance, the feature did not stay. We fiddled with many other feature ideas within our tech capabilities since; most did not work out for those who have been around long enough. We nevertheless enjoyed the process since we find building for the community rewarding. The organic growth of the forum prompted us to explore the idea of extending our crowdsourcing culture to tech development. We discussed creating a @tech_crew group so folks with tech background on the forum can find each other and collaborate to spin up prototypes suggested by the community. Leasehackr is happy to credit, host, and provide a funding model for tech features beneficial to the community! It’s a wild thought, but hey, @michael and I never thought we could gather so many people to help others when we first started sharing tips on the internet. :stuck_out_tongue_closed_eyes:

8 Likes

Sounds like an awesome idea! Unfortunately other commitments have prevented me from helping out with this project thus far, but I think this could definitely help LH be the one-stop-shop for all things leasing.

1 Like

Make/model are case sensitive, you may want to change that :slight_smile:

1 Like

I like it a lot! Count me in, be happy to contribute in anything tech related!

Yeah, I’m going to, eventually

1 Like

In for being help in tech. Can help in python…

1 Like

looks like they are storing all dealer infos locally for that autocomplete dealer locator. webpack:///./frontend_src/bmwlib/api-services/dealer-info.js may be a file of interest

edit: have you tried bmwusa’s api? Looks like it’s getting a very small area of the zip but you can definitely go through the zipcode list. https://www.bmwusa.com/api/inventory/filter/search/{zipcode}?page=1&resultsPerPage=24&sortBy=PRICE&sortOrder=ASC

edit2: this api is working great. no M cars though. nvm i’m doing it wrong. it does have M models

edit3: ok so that zipcode doesn’t matter since you can set locator radius to 9999. Got 239 x4m© in the US can anyone confirm?

What any programer says, besides, it’s not a bug, it’s a feature.