Multi-brand new/used car inventory search: an evolution story of python script to a web scraper to an API driven webapp

Tech Lounge
142

Check out my new fancy fast UI for working with async REST services pattern, written in JQuery entirely client-side :fire: :fire: :fire: :fire:

image
image1909×477 57.6 KB

  • still totally real-time searches, no stale car data here lol
  • searches will never time out, no matter how long it takes to scrape all dealers for info
  • client-side paging, sorting and insanely useful FILTERING on all results that you get - lighting fast search on all columns in the grid, e.g. putting “xdr blue FWD” will filter down all results to only xDrive FWD cars in blue. Try it on all columns, VIN, stock, engine etc.
  • real-time search statistics showing how search is initiated, how it polls for results, and when it finishes
  • fancy spinning indicator, always my proudest accomplishment in any UI work, yeah baby!

It’s live now in place of old UI:

http://carscrapper.azurewebsites.net/

Give it a spin and let me know.

Edit: just noticed dealerCom BMW searches return unrelated models along with model that is searched on. I’ll look into it

9 Likes
143

not sure if all but some dealerinspire website has a xml site-map like this one
https://www.bmwofbloomington.com/dealer-inspire-inventory/inventory_sitemap
and you can simply parse the url to get year/model/vin and make another async request to get details if you want

144

Cool I’ll check it out. Thanks

Edit: just fixed BMW search. What’s interesting is that 2 existing dealercom dealers changed their websites:

Long beach BMW went from

https://www.longbeachbmw.com/new-bmw/long-beach.htm?superModel={0}

to

https://www.longbeachbmw.com/new-bmw/long-beach.htm?model={0}

Sterling BMW went from standard dealerCom URL structure

www.sterlingbmw.com/new-inventory/index.htm?model={0}

to custom URL

www.sterlingbmw.com/new-inventory/pageSizeChange/1/10/~/VehicleType_~Model_{0}~Trim_~Year_~Price1_~TransmissionGeneric_~ExteriorColorGeneric_~InteriorColor_~EPAHighway_/~/100

On plus side, sterling new URL can set page size via query parameters, which removes the need to programmatically determine paging (which is not that reliable) and crawl each page to collect all stock.

1 Like
145

I noticed one of my local MB sites did something similar. Seemed to be with the last update in their inventory and they moved couple other links around.

146

Yup, this is unfortunate if dealers do it often enough. With this change, which was broken filter at dealer’s, it was easy enough to spot, by seeing not relevant to search results. But if URL changes drastically, some sites might start erroring out and stop providing results completely. Those would be hard to spot. Gotta periodically check logs to catch offenders like that.

147

Added bunch of tri-state Hyundai dealers, 90% of those are plug and play DealerCom sites, excellent uniformity. Search for Sonata for example is sitting around 600 units.

1 Like
148

Few updates:

  • added dynamic hiding of empty grid columns to preserve space
  • added detection of packages to dealercom listings. Now there is an option to filter results down to specific package.
    image
    image2095×784 116 KB

    Since I’m scraping this info from dealer’s search page, not all packages might be there. You can click individual listing link to see all packages, if they are also listed on dealer details page.

I’m going on vacation tomorrow, so most likely there will be no updates for next week. Have fun with search everyone and let me know what else can we add to it.

1 Like
149

Enjoy vacation. I am going on vacation this week too. But ironically, that means more hacking time for me, not less :grinning:

150

Thanks and enjoy yours too. Honestly, if it’d be up to me, my vacation would be sitting home coding/sleeping/gaming… :sob:

151

Update, probably last for the next week or so:

added 30 tri-state and PA Audi dealers for all your east coast audi searching needs :laughing:

Line 3: Audi Allentown
Line 4: Audi Bridgewater
Line 5: Audi Brooklyn
Line 6: Audi Devon
Line 7: Audi Eatontown
Line 8: Audi Fort Washington
Line 9: Audi Freehold
Line 10: Audi Greenwich
Line 11: Audi Hawthorne
Line 12: Audi Hunt Valley
Line 13: Audi Mechanicsburg
Line 14: Audi Mendham
Line 15: Audi Newton
Line 16: Audi of Huntington
Line 17: Audi Princeton
Line 18: Audi Reading
Line 19: Audi State College
Line 20: Audi Warrington
Line 21: Audi West Chester
Line 22: Audi Wynnewood
Line 23: Audi Wyoming Valley
Line 26: Bell Audi
Line 31: Biener Audi
Line 68: DCH Millburn Audi
Line 70: Fiore Audi
Line 71: Flemington Audi
Line 87: Jack Daniels Audi of Paramus
Line 88: Jack Daniels Audi of Upper Saddle River
Line 96: Mohegan Lake Audi
Line 116: Town Audi
2 Likes
152

A post was merged into an existing topic: Off Topic Landfill

153

Back from vacation, back to the grind. Added ability to search by specific region, or all listings nationally, if desired:

image

Going forward each dealer that is added to the config needs his region to be specified at that time.

1 Like
154
155

Is there a link where I can run this or do I need to put the code in notepad ++ and run the code? Very little coding experience but know a little. I have been looking for ppl like you guys for 6 months now and this freaking awesome. Any assistance helps and I appreciate all you motor heads.

156

There is web app to run the search, if that’s what you mean:

http://carscrapper.azurewebsites.net/

If you mean to run code locally on your machine, you’ll need Visual Studio 2019 to compile and run it, but the code is available here:

2 Likes
157

Thank you so much, like I said I took one year of graphic design in college 5 years ago so very little experiance. So I click on the web app above and then do i copy and past any code before putting model and make in? Like I feel something simple and I hope not pissing you off with dumb questions and I apperciate you!

158

It’s just regular web site, no need for any coding or such on there. Just put make/model and search away. Make sure model is case sensitive though, e.g. for BMW X3 you’re entering “X3”, not “x3”.

159

I got it figured out. I am having trouble with Toyota Tacoma, and 4 Runner. Also this is useful in the context you are searching all availble models and makes on the internet so your getting a great deal? Or is there any other perks to this awesome you made?

160

What kind of trouble? No results at all? There is only 1 Toyota dealer currently that is searchable, returning 5 results for Tacoma and 6 for “4Runner”. I haven’t gotten to adding more Toyota dealers yet, have been focusing on luxury makes.

This tool is scraping live listings from limited set of dealer websites. Those dealer sites need to be compatible with the way tool processes data, and manually added to config. You can see the list of currently available dealers at the bottom on the page. So, in other words, if I want to add new make, or expand existing one with new dealers, I need to examine dealer websites to make sure it’ll work and manually add those to config. Good percentage of dealer websites are compatible, so it’s just matter of time for someone to examine those and add them. I can explain how to determine if website is compatible if you want to help expanding dealer list.

Edit: just added several WA Toyota dealerships. At least some inventory for users searching for Toyotas in the West.

Line 24: AutoNation Toyota Spokane Valley
Line 63: Burien Toyota
Line 119: Town Toyota
Line 120: Toyota Of Olympia
Line 123: Vancouver Toyota
161