Multi-brand new/used car inventory search: an evolution story of python script to a web scraper to an API driven webapp

15 lines and you get this

2 Likes

Pretty cool. Is it from BMW API? Main thing is that its been reported their dealer inventory might not be up to date. Is there a link to individual cars on dealer store that can be extracted?

no, but you have vin numbers.
by not up-to-date you mean they donā€™t have the new ones or they have the sold ones or both?

Found it, @coder8 reported it when they were writing their scripts.

I have VIN numbers, true. If we to marry our results somehow, how would we match my VIN numbers to your records?

ye i mean I have VIN numbers. see the full output from bmwusa api

If only there were some way to persist the data, in some sort of rows and columns. :thinking:

1 Like

VIN number should be the uid here and everything else should be parsed

This is awesome! Can you add loaners to the list returned?

I meant there were no VIN numbers in his initial screenshot. Theyā€™re there, I see it now.

Not sure Iā€™m following, could you explain please?

Hitting an API for each manufacturer (if any other provide that too) would be preferable to scraping it from web sites in terms of data parsing ease and speed, however, there is still a question on how up to date the data is in those API responses is. Certainly something that can be explored.

Iā€™ve hit a snag with scraping. Looks like there are different versions of same platforms, like Dealer dot com for example. Some versions run bunch of async scripts to populate vehicle data after document is loaded, while other donā€™t. With old implementation, Iā€™d often grab content that is not fully populated. Iā€™m working on implementing a way to insure all scripts are loaded before grabbing siteā€™s content for parsing, but so far it really slowed down performance and results are inconsistent.

So, before releasing any updates, I want to tackle this problem once and for all, since itā€™s a core prerequisite for reliable and consistent scrapping across various sites. Hopefully in few days Iā€™ll have updates.

I meant to reply to jeisensc but the theory is to come up with your own naming and parse the data from api/site scrapping

are you able to get the version of that platform? Or a lazy solution is to wait for everything to be loaded

looks like bmw api returns new cars only. I assume OPā€™s script will work for loaners.

No, havenā€™t found reliable way to determine it yet. Unfortunately, this is only solution I see for now. Itā€™s either this or just keeping quick ā€œhit and runā€ concept of grabbing web site content and missing out on some of the dealerā€™s stock.

I know Volvo site inventory search has links to each individual carā€™s dealer site.
But loaners/demos are not listed on Volvo site, obviously.

Awesome! Adding you in!

I looked (albeit very quickly) into what manufacturers provide inventory APIs for public consumption, and havenā€™t found anything.

Thanks!

General update:
Iā€™ve decided to hold off on implementing new way of waiting for website to load all its scripts before grabbing content. Itā€™s too slow and unreliable. Performance suffers just too much. Maybe Iā€™ll revisit it later or someone else can look into reliable and fast way of doing it.

For now, I fixed few ā€œundocumented featuresā€ (lol @HersheySweet), and removed make/model input case sensitivity. Although, for some searches, due to how some sites treat their query strings, model needs to be in proper case for search to work. Seems to depend on make. For example, ā€œtoyota corollaā€ will work, but search for ā€œvolvo xc40ā€ will not find anything. Needs to be ā€œvolvo XC40ā€. And ā€œacura RDXā€. Seems to depend on original modelā€™s case. If its all uppercase, then it needs to entered as such. Will push new updates to Azure shortly.

Also, some model identifiers on dealer sites are setup to be multi-words. ā€œchevrolet silveradoā€ will not work, but ā€œchevrolet silverado 1500ā€ will. If youā€™re not finding anything for a make that is listed in the ā€œMakeā€ list, go to dealer website run manual search there and see how model is listed/spelled on UI. Then use same exact spelling for search.

Iā€™m planning to be adding more dealers now, and for next usable feature, Iā€™ve been thinking to group all dealers regionally and allow search by regions, as well as search-all, if user chooses so. Shouldā€™t be too complicated. What would be your next most useful feature?

That is so cool! Thank you!

Always happy to contribute to community! :+1:

Specifically for Volvo loaners, there is a low resolution image that is included on most to all of the listings. I would be happy to help add this, but having a separate category designating loaner based on the presence or absence of the image would probably be helpful for the people targeting discounts of 20% or higher :laughing:.

image (here it is)

Yup, thatā€™s a must feature. Will look into it next