09 Aug 2019

Build a scheduled web scraper with lambda

AWS Lambda is one of my favorite cloud services. The only limits are the imagination and skillset of the developers using it. Common categories of use include event driven processing, batch data processing, log ingest and analytics, and automation of the AWS environment. I've seen Lambda function as the backend for websites, as an engine to customize email delivery campaigns, as a monitor and control for other systems, and as the brain behind trendy online chatbots and Slack bots. For this example we will use Lambda to build a scheduled task and automatically scrape top artists information from the iTunes website. Here are the services we'll be using:

Lambda (code execution)

S3 (static file hosting)

CloudWatch (scheduling)

A note on cost: Lambda offers the first 1 million requests per month for free. Beyond that it adheres to a standard AWS pricing model, with the cost coming out to $0.0000002 per request in US East.

Identifying Our Source Data

We'll be grabbing our data from this site: https://www.apple.com/itunes/charts/songs. It contains a list of the top 100 songs and the list changes daily. To suit our imaginary objective, we want to save this data to our own server as a snapshot in time so we can retrieve the list at any point and potentially do something interesting with it later.
Web Scraping Overview
For our task we'll be using a popular Python library called Beautiful Soup which is built to parse HTML pages. The documentation can be found here. As it parses the page it creates a Python dictionary to represent the nested structure of the page. So if given the following HTML:

<html><head><title>This is a title</title></head><body><p>This is a paragraph</p></body></html>

You can retrieve the title with:
```
object.title.string
# This is a title
```

Create an S3 bucket

Before we get to our function code, we want to create an S3 bucket to serve as a destination for the file we will produce. Go to the S3 Console and either choose an existing bucket or create a new bucket. For our purposes we don't care too much about the bucket permissions, so feel free to keep the default settings or use any existing settings on the bucket of your choice. Make note of the bucket name. You'll need it in an upcoming step.

Setting up our Lambda Function

Log into the Lambda Console and create a new function. On the Create function screen we'll choose Author from scratch. Provide any reasonable name to your function, and for language we'll choose Python 3.7. Press Create function when ready.
Execution Role
Before we do anything else, we will set the permissions of our bucket. By default Lambda does not have access to write a file to S3. To fix this, click the link to view the [role-name] on the IAM console which will take you to the Identity Access Management (IAM) console. Once there, click Attach Policies, search for AmazonS3FullAccess and click Attach policy. When you're done, return to your Lambda function and reload the page. You should now see Amazon S3 listed as a target.
Inside the Identity Access Management (IAM) console, click Attach Policies, search for AmazonS3FullAccess and click Attach policy. When you're done, return to your Lambda function and reload the page. You should now see Amazon S3 listed as a target.

Working With The Lambda Code

We've already jumped through a variety of services. To proceed, we need to equip our lambda function with Beautiful Soup and a few other dependencies to properly handle our task. To keep things simple, I've provided a zip file with all the dependencies and code necessary to setup your web scraper. You can download it here.

To upload the zip, find the Function code area within the Lambda Console. In the menu for "Code entry type", select Upload a .zip file and then save your changes. Once you have the code uploaded, change the value of the variable s3bucketname to be the name of your s3 bucket.
The code itself is sprinkled with commented out print statements which you can uncomment to get a better idea of what each section is doing. If your S3 bucket is configured properly, what will happen when you press Test is a new file will be created named "songs" followed by today's date. It's named in this manner so each new entry will not override the previous. That's it for the code! Next up is scheduling.

Scheduling Our Function

Now that our code works, we can go ahead and schedule our function. To do this, we'll go to the CloudWatch Console. From there, click on Rules, and Create Rule.
Set it for a Schedule with a fixed rate of 24 hours. There are other, more granular ways to schedule this, but for us "once a day" is good enough.
On the right side of the screen, press add target and select your Lambda function by name. Press Configure details, provide a name for your rule and press create rule.

Congratulations. You're now the proud owner of an automated Python scraping script!