The Bolg

Back to coding blogs

Its time for the inaugural rewrite of the webapp to show if its time to break fast lol

2022 version

server side on heroku

scrape using pupeteer, then persist in mysql db hosted on planetscale

2023 version

fully on nextjs hosted on vercel

scraping done with cheerio

persistence layer using redis

query on getServerSideProps will be reading from redis cache everytime

in this edition, i replaced pupeteer with cheerio js after realising i dont need to click or perform actions on the site, i just fetch the page, and find the id i need

the newest and shinest thing in this version is Vercel Cron jobs. So its pretty straightforward, a normal handler in the api folder, and it is triggered in accordance to a cron command that is defined in vercel.json

So in this sample, i created a cron folder and it has scrapetime.js that has the handler to do cheerio scraping on muis site, which will then update the redis (hosted on upstash which is integrated onto vercel platform (lets gooo vendor lock in)

on the frontend, whenver user accesses the site, it will do a redis get on maghrib and subuh timing

so yeap thats all folks, happy ramadan

here is the github repo: https://github.com/shaikzhafir/when-eat

{
    "crons": [
      {
        "path": "/api/cron/scrapetime",
        "schedule": "0 0 * * *"
      }
    ]
  }

// cron job api handler 

export default async function handler(req, res) {
    // scrape using cheerio
    try {
        const response = await axios.get('https://www.muis.gov.sg/')
        const $ = cheerio.load(response.data);
        const scrapedSubuh = $('#PrayerTimeControl1_Subuh').text();
        const scrapedMaghrib = $('#PrayerTimeControl1_Maghrib').text();
        const subuhTime = convertToTime(scrapedSubuh, false)
        const maghribTime = convertToTime(scrapedMaghrib, true)
        console.log(subuhTime); // Output: the text content of the div with id "PrayerTimeControl1_Subuh"
        console.log(maghribTime); // Output: the text content of the div with id "PrayerTimeControl1_Maghrib"

        let client = new Redis(process.env.REDIS_URL);
        client.on("error", function (err) {
            throw err;
        });
        await client.set('maghrib', maghribTime, Redis.print);
        await client.set('subuh', subuhTime, Redis.print);
        res.status(200).json({ subuh: subuhTime, maghrib: maghribTime })

    } catch (error) {
        console.log(error);
        res.status(503).json({ error: error })
    }
}

// getServerSideProps on client page
export async function getServerSideProps() {
  try {
    const Redis = require("ioredis");
    require("dotenv").config();

    let client = new Redis(process.env.REDIS_URL);
    client.on("error", function (err) {
      throw err;
    });
    const subuh = await client.get("subuh");
    const maghrib = await client.get("maghrib");

    return {
      props: {
        data: {
          subuh: subuh,
          maghrib: maghrib,
        },
      },
    };
	// in case of error we return a padded time lol
  } catch (error) {
    return {
      props: {
        data: {
          subuh: "5:30",
          maghrib: "19:30",
        },
      },
    };
  }
}