midlife crisis wrote:Derek -- this is great news. Thanks for all the work you've put into it so far. One thing I liked about Drinkvine was that, even though I used it primarily for beer, one could also add the odd whisky or bottle of wine (after an irresistible Belgian review). Any plans to expand beyond beer at any point?
I knew it was just a matter of time before I got this (very reasonable) request.
Short answer: Yes, I'll give it a shot and see what happens.
Long answer: It's absolutely possible to include all LCBO products. In fact, limiting it to just beer was a lot more difficult than a "full scan" (my original scraper included all products and I had to scale it back).
Here are a few of my concerns:
- A full scan generates a LOT of requests to the LCBO website. My current setup spreads these requests out and (tries to) keep my impact on LCBO.com to a minimum. I want to make sure that I don't cause any problems for their system administrators.
- A full scan also generates a TON of data! When I finished my scraper I let it run once and my database quickly became huge. I'll work on keeping the database efficient, but I don't want it to get sluggish.
- I'm hosting this project on my company's server. I'm not in a position where I can pay for hosting right now, so I have to make sure it doesn't impact their business. I have permission to use their server, but I have to be reasonable...
I should be able to keep my requests to LCBO.com spread out by tracking less popular products and updating them less frequently. If nobody is watching a particular product then I can update inventory weekly, etc. The obvious downside to this would be that fans of Palm Bay Key Lime Cherry may see out-of-date results. Doing most of the scanning late at night will reduce the impact on both the LCBO and my company servers.
My plan is to first improve the functionality of CaskHunter and remove bugs first. (It's only been up for about 5 days) I have a huge list of bugs and features that need to be addressed. Once it's running properly I will switch it back to full scan and see what happens.