Multi-brand new/used car inventory search: an evolution story of python script to a web scraper to an API driven webapp

[muc2phx]

Great work @RustyDaemon. However, Make: BMW Model: M340i doesn’t give me any results. Am I doing something wrong?

[RustyDaemon]

Try entering model as “3 series” and when search results appear, filter it down further by entering “m340” in search box at top right of results grid.

[edmcman]

Any trick to getting a search for Volvo S60s to work?

[RustyDaemon]

Actually, it looks like any Volvo search is not returning results anymore. I’ll look into it.

[Ursus]

I think they switched sites design.

[RustyDaemon]

I decided to revisit tired old scraper, as most of dealers’ websites it was scraping were updated or changed since it was developed, and don’t really work anymore.

While revisiting, I stumbled upon goldmine of an API that dealers who use DealerInspire software are using for their own listings. Figured out how to consume it and tried with handful of dealers so far, but it looks very promising:

• this is the API dealerInspire websites call for their own stock listings, not a centralized manufacturer’s repository
• literally hundreds of hits for most single dealers
• lightning fast - several hundreds of results under a second
• new, used, and CPO listings

I’m going to work on tuning it for as many dealerInpire websites as I previously identified, but this looks very promising. For example, 166 new listings for single BMW dealer:

Stay tuned.

[RustyDaemon]

Check it out: new and improved scraper that doesn’t crap but pulls the data from the source

http://carscrapper.azurewebsites.net/

• 1K Mercedes nationwide listings under 3 seconds
• 70 EQSs
• used cars (hey, they’re there, why not?)
• Check “used/CPO cars only”, sort by year descending - you got your loaners
• Growing number of Audi/Honda/Land rover/Lexus/whatever dealers next
• Free-text search by model: knock yourself out with any combination of engine/trim/whatever else queries

I’ll keep adding dealers and brands. Which brands should be next?

Enjoy

Edit: I’m gonna do much loved here lately Jeep next… Getcha all 4xe stocks infos here!

Edit 2: Jeep is live. Quickly threw several dealers together with combined national inventory of 1K.

[RustyDaemon]

Bentleys anyone?

[dukez]

UGH why did I pick up a mortgage when I could have picked up a Continental?

[RustyDaemon]

New things and improvements so far:

• Lightning speed, hundreds of listings under a second:
  

• Full list of exterior, interior options and main features for each car:
  

  

• Pictures!
  

  

• Days on a lot and location:
  

• Growing list of brands and dealers:
  

• Thousands of listings nationwide:
  

[RustyDaemon]

Added bunch of new dealers, carfax link for used listings that have one:

[RustyDaemon]

Restyled UI a bit, looks cleaner and more modern. God, I hate CSS

[RustyDaemon]

Added several dealers, removed nationwide search option for region: my API was aggregating so much data the RAM quota in Azure was triggering just after few such searches and taking the app offline.

[RustyDaemon]

The milestone!

It’s in fact more than that, about half of that 100 are dealer groups with multiple dealers of various brands. So it’s more like 150 individual dealers.

Also,

Edit:
So I’m basically running out of huge auto groups to add that offer multiple brands under one roof. Gotta switch to adding brand-specific dealers.

Added dozen or so Genesis dealers, since Genesis count was under-represented. What brand should I focus on next?

[Rsantoro12]

late here but this is fantastic, well done @RustyDaemon

[RustyDaemon]

So AutoNation has sneaky little API that supports nationwide search by all their dealers. Added that as a supplement to main search, it generally brings couple hundreds additional listings for bunch of brands fairly quickly, with no significant performance drop. Lets just hope that their auth token does not expire anytime soon.

@Rsantoro12

late here but this is fantastic, well done @RustyDaemon

Always a pleasure!

[RustyDaemon]

Finetuned AutoNation search.

It was wonky with matching regions before, now it searches their dealers in states of that specific region only, as opposed to getting nationwide results and filtering it down to a zipcode radius, which proved to be less than accurate.

Counting all AutoNation stores, dealer count jumped to 435.

[RustyDaemon]

Figured out how to get data in a generic way from dealers using DealerOn API. Excellent performance with hundreds of listings, and pretty simple to add new individual dealers or dealer groups to the list. Also works for used inventory.

If anyone wants to help finding dealers or large auto groups using DealerOn API, it’s easy to do:

1. Google “dealeron auto group” or “dealeron dealer”

2. Open each result in new tab, hit F12 (in Chrome) for dev tools

3. Click on “new listings” or “all inventory” or similar menu options that shows all new inventory.

4. Inventory search page usually looks very close to this:
   

   

5. In Dev tools window, select “network” tab, put “?filter” query in search box and look for request that has URL that looks like this: /api/inventory/query/?filter=VHlwZT0nTic%3D&encoded=true
   

   

6. If you see it, then this is the site that can be added. Check against “available dealers” list, to make sure it’s a new dealer/group that is not on the list already: http://carscrapper.azurewebsites.net/

7. Get all makes from left-hand menu of the search page. These will be all searchable. If there is no make filter there, then it’s one brand store. Note the brand.
   

   

8. Note the geographical region of this store: northeast, midwest, west, southeast, southwest
   I’m using this for state to region map:

{
“Midwest” :[
“Wisconsin”, “North Dakota”, “Minnesota”, “South Dakota”, “Nebraska”,
“Iowa”, “Kansas”, “Missouri”, “Illinois”, “Michigan”, “Indiana”, “Ohio”
],
“West”:[
“Washington”, “Idaho”, “Montana”, “Oregon”, “Wyoming”, “California”,
“Nevada”, “Utah”, “Arizona”, “Colorado”, “New Mexico”
],
“Southwest” :[
“Texas”, “Oklahoma”, “Arkansas”, “Louisiana”
],
“Southeast”:[
“Kentucky”, “West Virginia”, “Maryland”, “Delaware”, “Virginia”,
“Tennessee”, “North Carolina”, “South Carolina”, “Georgia”,
“Alabama”, “Mississippi”, “Florida”
],
“Northeast”:[
“Connecticut”, “Maine”, “Massachusetts”, “New Hampshire”, “New Jersey”,
“New York”, “Pennsylvania”, “Vermont”, “Rhode Island”
]
}

8. Send me the info. 4 pieces that are needed to add new store to the config are region, brands (makes), dealer name and main URL:
   
   
9. I’ll add it to the app’s config and the inventory should start showing up.

[RustyDaemon]

Added some dealers, the milestone:

[JD81]

This is great, you should beach out into other desirable products and put it behind a paywall.