Web scraping is shitty by design but I was referring to Selenium. Having structured data on an API beats waiting & finding elements and then getting the data from the DOM. With Tesla, we have access to the entire listing data which is a ton more than what the website lets you see.
As for the time comment, it definitely varies between websites & Tesla inventory site is notoriously slow. Here’s a report I ran with Google’s LIghthouse / PageSpeed Insights. To make this efficient, let’s say we poll for our elements every 50ms, we still need to wait for the page to load completely (which could take anywhere between 6 seconds to 15 or more). Then you get all elements that match a listing and iterate through them all getting the necessary information. The more listings you have, the longer it will take. Maybe 20 seconds, 30 seconds or longer.
But definitely not 200ms. Our bots check every 3 seconds for an update and you could never achieve that with web scraping. Also, why would you load all of the images and web site elements – its like carrying a luggage with all of your belongings when all you needed was a charger from it.
I would assume that any in-inventory units that will drop from this point forward will reflect the new MSRP price increases? Also did the Model 3 LR AWD MSRP stay the same?
You realize that the web server on the other end can handle more than one request / response right? I understand that you are taking the easiest route with Selenium, but you can multi-process / multi-threaded scraper that will do this soo much faster. Granted you would be looking at whatever protection the target site is utilizing, but so far I haven’t seen one that I couldn’t trick into thinking my crawlers are not a threat.
I was talking about it to provide an example to someone who asked how to build a bot … not using it in this case.
but why though? The Tesla website is built in such a way that their UI layer is separate from the data layer (e.g. they do not serve the pages with the data inside of it). There is 1 single call made to this endpoint that provides the data to the UI layer to populate. So, we just simply make the same 1 call to get the data and boom - that’s it. With a beefed up crawler, you would still be crawling aka hitting this same API endpoint along with all the other assets of that site and still come up short because it did extra work it did not have to.
In our case, we make over 55 API calls simultaneously spread out over multiple threads and that’s how we are able to provide timely notifications for these alerts all across the United States & Canada vs just 1 search area.
Every website is different and one has to assess it individually to see what works best for it. For Tesla, API is the way to go. For others, a multi-threaded crawler could work better. Can’t be making blanket statements that cover all scenarios.
In the beginning, yes. A lot has changed on the implementation with storing data in the cloud and other things, but I kept it up if anyone wanted do something on a small scale for themselves.
aka hitting this same API endpoint along with all the other assets of that site and still come up short because it did extra work it did not have to.
